topic Re: tMysqlOutputBulkExec Error "you cannot send more than 4GB of data" in Talend Studio

tMysqlOutputBulkExec Error "you cannot send more than 4GB of data"

Anonymous — Thu, 12 Jul 2018 15:50:26 GMT

Hello everyone,

I encountered the same problem while trying to insert 32.000.000 rows from a XML-File to a MariaDB database.

The job runes fine with a "normal" tMysqlOutput - but takes a really, really long time. There are no problems with schemata or datatypes whatsoever

I have included

- View of Job (really simple - just passing data from xml to DB, no transformation needed)

- Advanced Settings of XML Component

- Basic Settings of tMysqlOutputBulkExec

- Advanced Settings of tMysqlOutputBulkExec

- Advanced Settings of Job itself (8GB JVM Argument)

My Version is 6.4.1.

Is this a Talend Settings Problem or a MySQL(/MariaDB) Settings Problem?

If there are any additional ressources / information needed to answer, pls tell me

Greetings from Germany

Stefan

vapukov — Fri, 13 Jul 2018 13:11:57 GMT

this is MySQL/MariaDB problem

in theory You can try play with max_allowed_packets, network buffer and other settings (no warranty for success)

or just split file for 3 chunk by 10M each and load file in 3 iterations

Anonymous — Tue, 31 Jul 2018 14:52:55 GMT

So,

i would like to reply to my own question. I have actually redone my Job in Talend Open Studio for Data Integration 7.01 (previous try was on 6.4.1)

And it works just like that! Just remember to set your XML-File to "Less Memory Usage (SAX)" in the Advanced Settings.

Hope this helps someone in the future!

Greetings from Germany

Stefan

Anonymous — Tue, 31 Jul 2018 17:43:12 GMT

Hello stefan,

Thanks for your feedback and sharing your solution with us.

Best regards

Sabrina