MySQL protocol limit reached using data from CSV file with bulk load

Four Stars

MySQL protocol limit reached using data from CSV file with bulk load

Hello,

 

I am working in Open Studio for MDM v6.3.2.

I have a job where I am extracting data from a CSV file that has a size of 2GB with 18,808,016 rows and 14 columns and attempting to write the data to a table in a MariaDB database.

 

My job is set up as: tFileInputDelimited --> tMysqlOutputBulkExec

 

When trying to run this I get:

java.sql.SQLNonTransientConnectionException: Could not read resultset: MySQL protocol limit reached, you cannot send more than 4GB of data

 

I've been unable to find a reason why this error can be happening if my file is only 2GB.

I created a temporary workaround where I split the file into several and load the data that way, but I would like to avoid doing it this way.

 

Any help would be greatly appreciated.

Thank you.

Moderator

Re: MySQL protocol limit reached using data from CSV file with bulk load

Hello,

Does this issue repro when you use tMysqlOutput without bulk load? Could you please post your tMysqlOutputBulkExec component setting screenshot on forum?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: MySQL protocol limit reached using data from CSV file with bulk load

Hi Sabrina,

 

Thank you for the reply.

I don't think the issue would reproduce using a tMysqlOutput, but the time to write all of this data would take too long.

 

Here are the basic settings of my tMysqlOutputBulkExec component:

tMysqlOutputBulkExec_settings.PNG

And the advanced settings:

tMysqlOutputBulkExec_advancedSettings.PNG

The full error is:

Exception in component tMysqlOutputBulkExec_1_tMBE
java.sql.SQLNonTransientConnectionException: Could not read resultset: MySQL protocol limit reached, you cannot send more than 4GB of data
    at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136)
    at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
    at org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264)
    at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288)
    at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:338)
    at test_project.staging_dump_poc_0_1.STAGING_DUMP_POC.tFileInputDelimited_1Process(STAGING_DUMP_POC.java:1626)
    at test_project.staging_dump_poc_0_1.STAGING_DUMP_POC.runJobInTOS(STAGING_DUMP_POC.java:2639)
    at test_project.staging_dump_poc_0_1.STAGING_DUMP_POC.main(STAGING_DUMP_POC.java:2398)
Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not read resultset: MySQL protocol limit reached, you cannot send more than 4GB of data
[statistics] disconnected
    at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:926)
    at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:991)
    at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281)
    ... 4 more
Caused by: java.io.IOException: MySQL protocol limit reached, you cannot send more than 4GB of data
    at org.mariadb.jdbc.internal.common.packet.PacketOutputStream.write(PacketOutputStream.java:84)
    at org.mariadb.jdbc.internal.common.packet.PacketOutputStream.sendFile(PacketOutputStream.java:63)
    at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:919)
    ... 6 more

 

 

Thank you for your help,

George

Moderator

Re: MySQL protocol limit reached using data from CSV file with bulk load

Hello,

Could you please try to use V 7.0 to see if this issue still repro on it?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.