How to insert into DynamoDB with DynamoDBoutput?

Four Stars

How to insert into DynamoDB with DynamoDBoutput?

Hi, All

 

I'm using Talend Open Studio Big Data 6.3, and I wanna use DynamoDB in my job.

 

image.png

 

image (1).png

 

My job is pretty simple.

 

Read file, and extract it, then insert to DynamoDB.

 

My AWS DynamoDB upload capacity is 2000.

 

So I think this job can insert 2000 rows /s. But It's actually 31 rows / s.

 

 Is there anyway past way to inset into Dynamo?

 

Thank you in advance..

Moderator

Re: How to insert into DynamoDB with DynamoDBoutput?

Hello,

In advanced setting of tDynamoDBoutput component, there are " Read Capacity Unite" and " Write Capacity Unite", which specify the number of read/write capacity units. For more information, please refer to

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.ht...

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: How to insert into DynamoDB with DynamoDBoutput?

Hello

 

I set it up 2,000 for each DynamoDB and tDynamoDBOutput component... and it's same...

 

image (2).png

 

Thank You for Reply.

Highlighted
Four Stars

Re: How to insert into DynamoDB with DynamoDBoutput?

Hi,

I'm not sure this issue is resolved or not. I have similar issue and I believe it is a problem with Talend tDynamoDBOutout component. It gives a strange error with "Action on Data" is set as "Insert". 

 

Try to use the "Action on Data" value as "Update". It works for me and I'm able to use this setting to Update as well as Insert new records in my DynamoDB table.

 

Thanks,

Siva

Eight Stars

Re: How to insert into DynamoDB with DynamoDBoutput?

When  "Action on Data" is set as "Insert",

 

Starting job BD_Job at 13:53 28/04/2019.

java.lang.VerifyError: Inconsistent stackmap frames at branch target 1601
Exception Details:
Location:
local_project/bd_job_0_1/BD_Job.tRowGenerator_1Process(Ljava/util/MapSmiley WinkV @1601: ldc_w
Reason:
Type top (current frame, locals[17]) is not assignable to 'com/amazonaws/services/dynamodbv2/document/spec/PutItemSpec' (stack map, locals[17])
Current Frame:
bci: @788
flags: { }
locals: { 'local_project/bd_job_0_1/BD_Job', 'java/util/Map', integer, 'java/lang/String', 'java/lang/String', 'java/util/Map', integer, 'local_project/bd_job_0_1/BD_Job$row1Struct', 'local_project/bd_job_0_1/BD_Job$dynamodb_outStruct', 'local_project/bd_job_0_1/BD_Job$dynamodb_outStruct', integer, integer, 'com/amazonaws/auth/AWSCredentials', 'com/amazonaws/services/dynamodbv2/AmazonDynamoDBClient', 'com/amazonaws/services/dynamodbv2/document/DynamoDB', 'com/amazonaws/services/dynamodbv2/document/Table', 'com/amazonaws/services/dynamodbv2/document/Item', top, integer, 'local_project/bd_job_0_1/BD_Job$1Util_tLogRow_1', 'java/lang/StringBuilder', integer, integer, 'local_project/bd_job_0_1/BD_Job$1Var__tMap_1__Struct', 'local_project/bd_job_0_1/BD_Job$dynamodb_outStruct', integer, integer, integer, 'local_project/bd_job_0_1/BD_Job$1tRowGenerator_1Randomizer', integer, integer, integer, integer, 'local_project/bd_job_0_1/BD_Job$1Var__tMap_1__Struct' }
stack: { 'local_project/bd_job_0_1/BD_Job$dynamodb_outStruct' }
Stackmap Frame:
bci: @1601
flags: { }
locals: { 'local_project/bd_job_0_1/BD_Job', 'java/util/Map', integer, 'java/lang/String', 'java/lang/String', 'java/util/Map', integer, 'local_project/bd_job_0_1/BD_Job$row1Struct', 'local_project/bd_job_0_1/BD_Job$dynamodb_outStruct', 'local_project/bd_job_0_1/BD_Job$dynamodb_outStruct', integer, integer, 'com/amazonaws/auth/AWSCredentials', 'com/amazonaws/services/dynamodbv2/AmazonDynamoDBClient', 'com/amazonaws/services/dynamodbv2/document/DynamoDB', 'com/amazonaws/services/dynamodbv2/document/Table', 'com/amazonaws/services/dynamodbv2/document/Item', 'com/amazonaws/services/dynamodbv2/document/spec/PutItemSpec', integer, 'local_project/bd_job_0_1/BD_Job$1Util_tLogRow_1', 'java/lang/StringBuilder', integer, integer, 'local_project/bd_job_0_1/BD_Job$1Var__tMap_1__Struct', 'local_project/bd_job_0_1/BD_Job$dynamodb_outStruct', integer, integer, integer, 'local_project/bd_job_0_1/BD_Job$1tRowGenerator_1Randomizer', integer, integer, integer, integer }
stack: { }
Bytecode:
0x0000000: 2b13 01b8 03b8 01ba b901 5003 0057 2ab4
0x0000010: 00dd 3d12 8e4e 128e 3a04 bb00 6c59 b700
0x0000020: 6e3a 0504 3606 2ab4 00bb 9a00 262a b400
0x0000030: b9c6 001f bb01 a959 b701 bfb6 01c0 0332
0x0000040: b601 c43a 072a b400 b919 07b6 0168 3606
0x0000050: 1506 9a00 0a2a b400 bb99 0822 2a04 b500
0x0000060: bbbb 01c9 59b7 01cb 3a07 bb01 cc59 b701
0x0000070: ce3a 0819 083a 092a b400 9813 01cf 03b8
0x0000080: 01d1 b901 5003 0057 2ab4 0094 1301 cfb8
0x0000090: 017c b801 80b9 0150 0300 5713 01cf 3a04
0x00000a0: 1c99 002a 1905 1301 d6b9 01d8 0200 c700
0x00000b0: 1d2a b400 a2bb 018f 5913 01dc b701 932d
0x00000c0: b601 95b6 0199 0303 b601 de03 360a 0336
0x00000d0: 0bbb 01e2 5913 01e4 1301 e6b7 01e8 3a0c
0x00000e0: bb01 eb59 190c b801 ed13 01f3 b601 f5b7
0x00000f0: 01fb 3a0d 190d 1301 feb6 0200 c001 eb3a
0x0000100: 0dbb 0204 5919 0db7 0206 3a0e 190e 1302
0x0000110: 09b6 020b 3a0f bb02 0f59 b702 113a 102a
0x0000120: b400 9813 0212 03b8 01d1 b901 5003 0057
0x0000130: 2ab4 0094 1302 12b8 017c b801 80b9 0150
0x0000140: 0300 5713 0212 3a04 1c99 002a 1905 1301
0x0000150: d6b9 01d8 0200 c700 1d2a b400 a2bb 018f
0x0000160: 5913 0214 b701 932d b601 95b6 0199 0303
0x0000170: b601 de03 3612 bb02 1659 2ab7 0218 3a13
0x0000180: 1913 1302 12b6 0219 1913 1006 bd00 ef59
0x0000190: 0313 021c 5359 0413 021e 5359 0513 0220
0x00001a0: 5359 0613 0222 5359 0713 0224 5359 0813
0x00001b0: 0226 53b6 0228 013a 1403 3615 2ab4 0098
0x00001c0: 1302 2c03 b801 d1b9 0150 0300 572a b400
0x00001d0: 9413 022c b801 7cb8 0180 b901 5003 0057
0x00001e0: 1302 2c3a 041c 9900 2a19 0513 01d6 b901
0x00001f0: d802 00c7 001d 2ab4 00a2 bb01 8f59 1302
0x0000200: 2eb7 0193 2db6 0195 b601 9903 03b6 01de
0x0000210: 0336 16bb 0230 592a b702 323a 17bb 01cc
0x0000220: 59b7 01ce 3a18 2ab4 0098 1302 3303 b801
0x0000230: d1b9 0150 0300 572a b400 9413 0233 b801
0x0000240: 7cb8 0180 b901 5003 0057 1302 333a 0403
0x0000250: 3619 0336 1a08 361b bb02 3559 2ab7 0237
0x0000260: 3a1c 0336 1da7 03ee 1907 191c b602 38b5
0x0000270: 023b 1907 191c b602 3eb5 0241 1907 191c
0x0000280: b602 44b5 0247 1907 191c b602 4ab5 024d
0x0000290: 841a 0113 0233 3a04 8419 0113 0233 3a04
0x00002a0: 1302 2c3a 041c 9900 1d2a b400 a2bb 018f
0x00002b0: 5913 022e b701 932d b601 95b6 0199 0404
0x00002c0: b601 de03 361e 0336 1f03 3620 1917 3a21
0x00002d0: 013a 0819 1819 07b4 023b b502 5019 1819
0x00002e0: 07b4 0241 b502 5219 1819 07b4 0247 b502
0x00002f0: 5419 1801 b502 5619 1801 b502 5819 1801
0x0000300: b502 5a19 183a 0803 361f 8416 0113 022c
0x0000310: 3a04 1908 c603 2d13 0212 3a04 1c99 001d
0x0000320: 2ab4 00a2 bb01 8f59 1302 14b7 0193 2db6
0x0000330: 0195 b601 9904 04b6 01de 1006 bd00 ef3a
0x0000340: 2119 08b4 0250 c600 0f19 2103 1908 b402
0x0000350: 50b8 025c 5319 08b4 0252 c600 0f19 2104
0x0000360: 1908 b402 52b8 025c 5319 08b4 0254 c600
0x0000370: 0f19 2105 1908 b402 54b8 025c 5319 08b4
0x0000380: 0256 c600 0f19 2106 1908 b402 56b8 025c
0x0000390: 5319 08b4 0258 c600 0f19 2107 1908 b402
0x00003a0: 58b8 025c 5319 08b4 025a c600 0f19 2108
0x00003b0: 1908 b402 5ab8 025c 5319 1319 21b6 0228
0x00003c0: 8415 0119 083a 0984 1201 1302 123a 0413
0x00003d0: 01cf 3a04 1c99 001d 2ab4 00a2 bb01 8f59
0x00003e0: 1301 dcb7 0193 2db6 0195 b601 9904 04b6
0x00003f0: 01de 013a 2201 3a23 bb00 6c59 b700 6e3a
0x0000400: 24bb 006c 59b7 006e 3a25 1909 b402 503a
0x0000410: 2219 09b4 0252 c600 0e19 09b4 0252 b602
0x0000420: 5f99 0012 1924 1302 1e01 b901 5003 0057
0x0000430: a700 1319 2413 021e 1909 b402 52b9 0150
0x0000440: 0300 5719 09b4 0254 c600 0e19 09b4 0254
0x0000450: b602 5f99 0012 1924 1302 2001 b901 5003
0x0000460: 0057 a700 1319 2413 0220 1909 b402 54b9
0x0000470: 0150 0300 5719 09b4 0256 c600 0e19 09b4
0x0000480: 0256 b602 5f99 0012 1924 1302 2201 b901
0x0000490: 5003 0057 a700 1319 2413 0222 1909 b402
0x00004a0: 56b9 0150 0300 5719 09b4 0258 c600 0e19
0x00004b0: 09b4 0258 b602 5f99 0012 1924 1302 2401
0x00004c0: b901 5003 0057 a700 1319 2413 0224 1909
0x00004d0: b402 58b9 0150 0300 5719 09b4 025a c600
0x00004e0: 0e19 09b4 025a b602 5f99 0012 1924 1302
0x00004f0: 2601 b901 5003 0057 a700 1319 2413 0226
0x0000500: 1909 b402 5ab9 0150 0300 5719 2513 021c
0x0000510: b202 62b9 0150 0300 5719 2513 021e b202
0x0000520: 62b9 0150 0300 5719 2513 0220 b202 62b9
0x0000530: 0150 0300 5719 2513 0222 b202 62b9 0150
0x0000540: 0300 5719 2513 0224 b202 62b9 0150 0300
0x0000550: 5719 2513 0226 b202 62b9 0150 0300 5719
0x0000560: 1013 021c 1922 b602 663a 1019 24b9 0130
0x0000570: 0100 b901 3601 003a 27a7 0093 1927 b901
0x0000580: 3c01 00c0 0141 3a26 1926 b901 4801 00c7
0x0000590: 0017 1910 1926 b901 4301 00c0 00ef b602
0x00005a0: 6a3a 10a7 0069 1925 1926 b901 4301 00b9
0x00005b0: 01d8 0200 c001 d2b6 026e 9900 3a19 26b9
0x00005c0: 0148 0100 b602 71b6 0272 3a28 1928 b602
0x00005d0: 5f99 0009 1302 75a7 0005 1928 3a28 1910
0x00005e0: 1926 b901 4301 00c0 00ef 1928 b602 773a
0x00005f0: 10a7 001b 1910 1926 b901 4301 00c0 00ef
0x0000600: 1926 b901 4801 00b6 027b 3a10 1927 b901
0x0000610: 5401 009a ff69 bb02 7e59 b702 8019 10b6
0x0000620: 0281 3a11 190f 1911 b602 8557 840b 0184
0x0000630: 0a01 1301 cf3a 0413 01cf 3a04 1302 123a
0x0000640: 0413 022c 3a04 1302 333a 0413 0233 3a04
0x0000650: 841d 0115 1d15 1ba1 fc11 2b13 028b 151a
0x0000660: b801 bab9 0150 0300 572a b400 9813 0233
0x0000670: 04b8 01d1 b901 5003 0057 2ab4 0096 1302
0x0000680: 33b8 017c b801 80b9 0150 0300 5713 022c
0x0000690: 3a04 1c99 003d 1905 1301 d6b9 01d8 0200
0x00006a0: c600 1619 0513 01d6 b901 d802 00c0 01d2
0x00006b0: b602 6e9a 001d 2ab4 00a2 bb01 8f59 1302
0x00006c0: 2eb7 0193 2db6 0195 b601 9905 03b6 01de
0x00006d0: 2ab4 0098 1302 2c04 b801 d1b9 0150 0300
0x00006e0: 572a b400 9613 022c b801 7cb8 0180 b901
0x00006f0: 5003 0057 1302 123a 0401 3a1d 2b13 028d
0x0000700: b901 d802 00c6 0014 2b13 028d b901 d802
0x0000710: 00c0 00a9 3a1d a700 22bb 00a9 59bb 00ab
0x0000720: 59b2 028f b700 adb7 00b0 3a1d 2b13 028d
0x0000730: 191d b901 5003 0057 191d 1913 b602 92b6
0x0000740: 0199 b601 1a19 1db6 016c 2b13 0296 1515
0x0000750: b801 bab9 0150 0300 571c 9900 3d19 0513
0x0000760: 01d6 b901 d802 00c6 0016 1905 1301 d6b9
0x0000770: 01d8 0200 c001 d2b6 026e 9a00 1d2a b400
0x0000780: a2bb 018f 5913 0214 b701 932d b601 95b6
0x0000790: 0199 0503 b601 de2a b400 9813 0212 04b8
0x00007a0: 01d1 b901 5003 0057 2ab4 0096 1302 12b8
0x00007b0: 017c b801 80b9 0150 0300 5713 01cf 3a04
0x00007c0: 2b13 0298 150b b801 bab9 0150 0300 571c
0x00007d0: 9900 3d19 0513 01d6 b901 d802 00c6 0016
0x00007e0: 1905 1301 d6b9 01d8 0200 c001 d2b6 026e
0x00007f0: 9a00 1d2a b400 a2bb 018f 5913 01dc b701
0x0000800: 932d b601 95b6 0199 0503 b601 de2a b400
0x0000810: 9813 01cf 04b8 01d1 b901 5003 0057 2ab4
0x0000820: 0096 1301 cfb8 017c b801 80b9 0150 0300
0x0000830: 57a7 004a 3a06 bb01 7959 2a19 0619 042b
0x0000840: 01b7 029a 3a07 1907 bf3a 062a b400 a2b6
0x0000850: 029d 1906 bf3a 2913 0233 3a04 1302 2c3a
0x0000860: 0413 0212 3a04 1301 cf3a 04a7 000a 3a2a
0x0000870: a700 053a 2a01 3a05 1929 bf13 0233 3a04
0x0000880: 1302 2c3a 0413 0212 3a04 1301 cf3a 04a7
0x0000890: 000a 3a2a a700 053a 2a01 3a05 2b13 01b8
0x00008a0: 04b8 01ba b901 5003 0057 b1
Exception Handler Table:
bci [35, 2097] => handler: 2100
bci [35, 2097] => handler: 2121
bci [35, 2133] => handler: 2133
bci [2135, 2155] => handler: 2158
bci [2135, 2155] => handler: 2163
bci [2171, 2191] => handler: 2194
bci [2171, 2191] => handler: 2199
Stackmap Table:
full_frame(@80,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer},{})
same_frame(@92)
append_frame(@203,Object[#457],Object[#460],Object[#460])
full_frame(@371,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer,Object[#457],Object[#460],Object[#460],Integer,Integer,Object[#728],Object[#491],Object[#516],Object[#646],Object[#527]},{})
full_frame(@528,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer,Object[#457],Object[#460],Object[#460],Integer,Integer,Object[#728],Object[#491],Object[#516],Object[#646],Object[#527],Top,Integer,Object[#534],Object[#399],Integer},{})
full_frame(@616,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer,Object[#457],Object[#460],Object[#460],Integer,Integer,Object[#728],Object[#491],Object[#516],Object[#646],Object[#527],Top,Integer,Object[#534],Object[#399],Integer,Integer,Object[#560],Object[#460],Integer,Integer,Integer,Object[#565],Integer},{})
same_frame_extended(@707)
append_frame(@826,Integer,Integer,Integer)
append_frame(@853,Object[#237])
same_frame(@873)
same_frame(@893)
same_frame(@913)
same_frame(@933)
same_frame(@953)
same_frame(@1010)
full_frame(@1060,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer,Object[#457],Object[#460],Object[#460],Integer,Integer,Object[#728],Object[#491],Object[#516],Object[#646],Object[#527],Top,Integer,Object[#534],Object[#399],Integer,Integer,Object[#560],Object[#460],Integer,Integer,Integer,Object[#565],Integer,Integer,Integer,Integer,Object[#237],Object[#3],Object[#3],Object[#305],Object[#305]},{})
same_frame(@1075)
same_frame(@1091)
same_frame(@1110)
same_frame(@1125)
same_frame(@1141)
same_frame(@1160)
same_frame(@1175)
same_frame(@1191)
same_frame(@1210)
same_frame(@1225)
same_frame(@1241)
same_frame(@1260)
same_frame(@1275)
same_frame(@1291)
full_frame(@1404,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer,Object[#457],Object[#460],Object[#460],Integer,Integer,Object[#728],Object[#491],Object[#516],Object[#646],Object[#527],Top,Integer,Object[#534],Object[#399],Integer,Integer,Object[#560],Object[#460],Integer,Integer,Integer,Object[#565],Integer,Integer,Integer,Integer,Object[#237],Object[#3],Object[#3],Object[#305],Object[#305],Top,Object[#317]},{})
full_frame(@1446,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer,Object[#457],Object[#460],Object[#460],Integer,Integer,Object[#728],Object[#491],Object[#516],Object[#646],Object[#527],Top,Integer,Object[#534],Object[#399],Integer,Integer,Object[#560],Object[#460],Integer,Integer,Integer,Object[#565],Integer,Integer,Integer,Integer,Object[#237],Object[#3],Object[#3],Object[#305],Object[#305],Object[#321],Object[#317]},{})
append_frame(@1498,Object[#239])
same_locals_1_stack_item_frame(@1500,Object[#239])
chop_frame(@1524,1)
full_frame(@1548,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer,Object[#457],Object[#460],Object[#460],Integer,Integer,Object[#728],Object[#491],Object[#516],Object[#646],Object[#527],Top,Integer,Object[#534],Object[#399],Integer,Integer,Object[#560],Object[#460],Integer,Integer,Integer,Object[#565],Integer,Integer,Integer,Integer,Object[#237],Object[#3],Object[#3],Object[#305],Object[#305],Top,Object[#317]},{})
full_frame(@1601,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer,Object[#457],Object[#460],Object[#460],Integer,Integer,Object[#728],Object[#491],Object[#516],Object[#646],Object[#527],Object[#638],Integer,Object[#534],Object[#399],Integer,Integer,Object[#560],Object[#460],Integer,Integer,Integer,Object[#565],Integer,Integer,Integer,Integer},{})
full_frame(@1619,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Integer,Object[#457],Object[#460],Object[#460],Integer,Integer,Object[#728],Object[#491],Object[#516],Object[#646],Object[#527],Top,Integer,Object[#534],Object[#399],Integer,Integer,Object[#560],Object[#460],Integer,Integer,Integer,Object[#565],Integer},{})
chop_frame(@1718,1)
same_frame(@1744)
append_frame(@1817,Object[#169])
same_frame(@1848)
same_frame_extended(@1917)
same_frame(@1943)
same_frame_extended(@2035)
same_frame(@2061)
full_frame(@2100,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305]},{Object[#425]})
same_locals_1_stack_item_frame(@2121,Object[#672])
same_locals_1_stack_item_frame(@2133,Object[#287])
full_frame(@2158,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305],Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Top,Object[#287]},{Object[#425]})
same_locals_1_stack_item_frame(@2163,Object[#672])
same_frame(@2165)
full_frame(@2171,{Object[#1],Object[#305],Integer,Object[#239],Object[#239],Object[#305]},{})
same_locals_1_stack_item_frame(@2194,Object[#425])
same_locals_1_stack_item_frame(@2199,Object[#672])
same_frame(@2201)

at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.privateGetMethodRecursive(Unknown Source)
at java.lang.Class.getMethod0(Unknown Source)
at java.lang.Class.getMethod(Unknown Source)
at sun.launcher.LauncherHelper.validateMainClass(Unknown Source)
at sun.launcher.LauncherHelper.checkAndLoadMain(Unknown Source)
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main"

Job BD_Job ended at 13:53 28/04/2019. [exit code=1]

 

 

. When the Action on data is set to update, it is working fine. What's wrong with tDynamoDBOutput component. 

 

@xdshi  , can you please check on this.  I am using Talend Open Studio for Big Data (Version: 7.1.1)

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now