topic Re: MySQL protocol limit reached using data from CSV file with bulk load in Talend Data Catalog

MySQL protocol limit reached using data from CSV file with bulk load

ggraf — Tue, 27 Feb 2018 16:47:21 GMT

Hello,

I am working in Open Studio for MDM v6.3.2.

I have a job where I am extracting data from a CSV file that has a size of 2GB with 18,808,016 rows and 14 columns and attempting to write the data to a table in a MariaDB database.

My job is set up as: tFileInputDelimited --> tMysqlOutputBulkExec

When trying to run this I get:

java.sql.SQLNonTransientConnectionException: Could not read resultset: MySQL protocol limit reached, you cannot send more than 4GB of data

I've been unable to find a reason why this error can be happening if my file is only 2GB.

I created a temporary workaround where I split the file into several and load the data that way, but I would like to avoid doing it this way.

Any help would be greatly appreciated.

Thank you.

Re: MySQL protocol limit reached using data from CSV file with bulk load

Anonymous — Thu, 01 Mar 2018 07:34:24 GMT

Hello,

Does this issue repro when you use tMysqlOutput without bulk load? Could you please post your tMysqlOutputBulkExec component setting screenshot on forum?

Best regards

Sabrina

Re: MySQL protocol limit reached using data from CSV file with bulk load

ggraf — Thu, 01 Mar 2018 14:28:45 GMT

Hi Sabrina,

Thank you for the reply.

I don't think the issue would reproduce using a tMysqlOutput, but the time to write all of this data would take too long.

Here are the basic settings of my tMysqlOutputBulkExec component:

And the advanced settings:

The full error is:

Exception in component tMysqlOutputBulkExec_1_tMBE
java.sql.SQLNonTransientConnectionException: Could not read resultset: MySQL protocol limit reached, you cannot send more than 4GB of data
   at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136)
   at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
   at org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264)
   at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288)
   at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:338)
   at test_project.staging_dump_poc_0_1.STAGING_DUMP_POC.tFileInputDelimited_1Process(STAGING_DUMP_POC.java:1626)
   at test_project.staging_dump_poc_0_1.STAGING_DUMP_POC.runJobInTOS(STAGING_DUMP_POC.java:2639)
   at test_project.staging_dump_poc_0_1.STAGING_DUMP_POC.main(STAGING_DUMP_POC.java:2398)
Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not read resultset: MySQL protocol limit reached, you cannot send more than 4GB of data
[statistics] disconnected
   at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:926)
   at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:991)
   at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281)
   ... 4 more
Caused by: java.io.IOException: MySQL protocol limit reached, you cannot send more than 4GB of data
   at org.mariadb.jdbc.internal.common.packet.PacketOutputStream.write(PacketOutputStream.java:84)
   at org.mariadb.jdbc.internal.common.packet.PacketOutputStream.sendFile(PacketOutputStream.java:63)
   at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:919)
   ... 6 more

Thank you for your help,

George

tMysqlOutputBulkExec_advancedSettings.PNG

Re: MySQL protocol limit reached using data from CSV file with bulk load

Anonymous — Wed, 25 Apr 2018 08:17:04 GMT

Hello,

Could you please try to use V 7.0 to see if this issue still repro on it?

Best regards

Sabrina