tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

[Anonymous]

Hello everyone,

i found this thread https://community.talend.com/t5/Design-and-Development/Getting-error-while-running-tMysqlOutputBulkE... where is sadly not a follow-up from the OP. So lets not make this a https://xkcd.com/979/ situation and find an answer 🙂

I encountered the same problem while trying to insert 32.000.000 rows from a XML-File to a MariaDB database.

The job runes fine with a "normal" tMysqlOutput - but takes a really, really long time. There are no problems with schemata or datatypes whatsoever

I have included

 - View of Job (really simple - just passing data from xml to DB, no transformation needed)

 - Advanced Settings of XML Component

- Basic Settings of tMysqlOutputBulkExec

 - Advanced Settings of tMysqlOutputBulkExec 

 - Advanced Settings of Job itself (8GB JVM Argument)

 My Version is 6.4.1.

Is this a Talend Settings Problem or a MySQL(/MariaDB) Settings Problem?

If there are any additional ressources / information needed to answer, pls tell me

Greetings from Germany

Stefan

[vapukov]

this is MySQL/MariaDB problem

in theory You can try play with max_allowed_packets, network buffer and other settings (no warranty for success)

or just split file for 3 chunk by 10M each and load file in 3 iterations

[Anonymous · Accepted solution]

So,

i would like to reply to my own question. I have actually redone my Job in Talend Open Studio for Data Integration 7.01 (previous try was on 6.4.1)

And it works just like that! Just remember to set your XML-File to "Less Memory Usage (SAX)" in the Advanced Settings.

Hope this helps someone in the future!

Greetings from Germany

Stefan

[Anonymous]

Hello stefan,

Thanks for your feedback and sharing your solution with us.

Best regards

Sabrina