Four Stars

tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

Hello everyone,

 

i found this thread https://community.talend.com/t5/Design-and-Development/Getting-error-while-running-tMysqlOutputBulkE... where is sadly not a follow-up from the OP. So lets not make this a https://xkcd.com/979/ situation and find an answer :-)

I encountered the same problem while trying to insert 32.000.000 rows from a XML-File to a MariaDB database.

The job runes fine with a "normal" tMysqlOutput - but takes a really, really long time. There are no problems with schemata or datatypes whatsoever

 

I have included

 - View of Job (really simple - just passing data from xml to DB, no transformation needed)

 - Advanced Settings of XML Component

- Basic Settings of tMysqlOutputBulkExec

 - Advanced Settings of tMysqlOutputBulkExec 

 - Advanced Settings of Job itself (8GB JVM Argument)

 

 My Version is 6.4.1.

 

Is this a Talend Settings Problem or a MySQL(/MariaDB) Settings Problem?

If there are any additional ressources / information needed to answer, pls tell me

 

Greetings from Germany

Stefan

1 REPLY
Twelve Stars

Re: tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

this is MySQL/MariaDB problem

 

in theory You can try play with max_allowed_packets, network buffer and other settings (no warranty for success)

or just split file for 3 chunk by 10M each and load file in 3 iterations

-----------