tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

Highlighted
Five Stars

tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

Hello everyone,

 

i found this thread https://community.talend.com/t5/Design-and-Development/Getting-error-while-running-tMysqlOutputBulkE... where is sadly not a follow-up from the OP. So lets not make this a https://xkcd.com/979/ situation and find an answer :-)

I encountered the same problem while trying to insert 32.000.000 rows from a XML-File to a MariaDB database.

The job runes fine with a "normal" tMysqlOutput - but takes a really, really long time. There are no problems with schemata or datatypes whatsoever

 

I have included

 - View of Job (really simple - just passing data from xml to DB, no transformation needed)

 - Advanced Settings of XML Component

- Basic Settings of tMysqlOutputBulkExec

 - Advanced Settings of tMysqlOutputBulkExec 

 - Advanced Settings of Job itself (8GB JVM Argument)

 

 My Version is 6.4.1.

 

Is this a Talend Settings Problem or a MySQL(/MariaDB) Settings Problem?

If there are any additional ressources / information needed to answer, pls tell me

 

Greetings from Germany

Stefan


Accepted Solutions
Five Stars

Re: tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

So,

 

i would like to reply to my own question. I have actually redone my Job in Talend Open Studio for Data Integration 7.01 (previous try was on 6.4.1)

 

And it works just like that! Just remember to set your XML-File to "Less Memory Usage (SAX)" in the Advanced Settings.

 

Hope this helps someone in the future!

 

Greetings from Germany

Stefan


All Replies
Forteen Stars

Re: tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

this is MySQL/MariaDB problem

 

in theory You can try play with max_allowed_packets, network buffer and other settings (no warranty for success)

or just split file for 3 chunk by 10M each and load file in 3 iterations

-----------
Five Stars

Re: tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

So,

 

i would like to reply to my own question. I have actually redone my Job in Talend Open Studio for Data Integration 7.01 (previous try was on 6.4.1)

 

And it works just like that! Just remember to set your XML-File to "Less Memory Usage (SAX)" in the Advanced Settings.

 

Hope this helps someone in the future!

 

Greetings from Germany

Stefan

Moderator

Re: tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

Hello stefan,

Thanks for your feedback and sharing your solution with us.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now