Communications Link Failure Broken Pipe in tMysqlOutputBulkExec

Anonymous — Sat, 16 Nov 2024 12:47:17 GMT

Hi Talend,
We are using TIS 4.0.1
We have been running into the following error intermittently (but frequent enough) when using tMysqlOutputBulkExec. When processing a large file, we sometimes get the 'Communications Link Failure' - root cause Broken pipe. It always failed at executing 'LOAD DATA LOCAL INFILE...'. This suggests that the file is created properly but when pushing the file to load into database is when it hit the error. The file sits in one server and the database in another. Here are the details:
What we have tried:
* Set autoReconnect for the connection to 'true'
* Set flush buffer size in the component to 10k - although it should matter
Logic tells me that breaking the files into manageable chunks would help. Although looking at the component, it doesn't seem to have the option to easily handle that. Any suggestion? The job is quite simple - tMysqlInput -> tMap -> tMysqlOutputBulkExec.
File Size:
* ~14 million records (although it had failed as small as 1mil)
Mysql Variables Settings
? Wait-timeout is not explicitly defined. The default value is 8 hours
? Max-allowed-packet is 64M
? Net-read-timeout is not explicitly defined. The default value is 30 seconds
Stack trace:
Root cause: Broken Pipe
Exception in component tMysqlOutputBulkExec_2_tMBE
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Last packet sent to the server was 84874 ms ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2720)
at com.mysql.jdbc.MysqlIO.sendFileToServer(MysqlIO.java:2798)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2033)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1421)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1772)
at com.mysql.jdbc.Connection.execSQL(Connection.java:2430)
at com.mysql.jdbc.Connection.execSQL(Connection.java:2359)
at com.mysql.jdbc.Statement.execute(Statement.java:719)
Thanks
Sarah

Re: Communications Link Failure Broken Pipe in tMysqlOutputBulkExec

Anonymous — Fri, 26 Aug 2011 20:58:02 GMT

Hi Talend/Chong,
Do you have any update or suggestion with this?
Thanks
Sarah

topic Re: Communications Link Failure Broken Pipe in tMysqlOutputBulkExec in Talend Studio

Communications Link Failure Broken Pipe in tMysqlOutputBulkExec

Re: Communications Link Failure Broken Pipe in tMysqlOutputBulkExec