topic Re: Very Slow Performance in Talend Studio

Very Slow Performance

Anonymous — Sat, 16 Nov 2024 13:51:40 GMT

I will preface this e-mail by saying that I am brand new to Talend.
I am importing a CSV file from the metadata repository on one box to a MySQL box in the metadata repository. I have a filter on the text file. So it goes CSVInput>tFilterRow>tMap>MySQLOutput. It is extremely, extremely slow. The CSV is only ~17k rows, and the file is importing at a rate of a little less than 1 record per second. At this rate it will take more than about 5 hours. I have 6 more CSV files in exactly the same format with many, many more records. I need a way to speed this up. 17k records was nothing in DTS in MSSQL, so I must be doing something wrong. I just do not know what.
??? Any ideas where I should look. Please advise.
--Harry

Re: Very Slow Performance

_AnonymousUser — Wed, 22 Jul 2009 21:59:05 GMT

use extended insert in the mysqloutput component (under advanced settings)

Re: Very Slow Performance

Anonymous — Wed, 22 Jul 2009 22:13:44 GMT

Take care that you do not use the debug mode in TOS. This will slow down the speed to one row a second.

Re: Very Slow Performance

Anonymous — Wed, 22 Jul 2009 23:07:57 GMT

Thx Volker !
Just a precision : Volker is speaking about "The trace mode"
HTH,

Re: Very Slow Performance

Anonymous — Thu, 23 Jul 2009 14:54:00 GMT

Thank you simony. The extended insert button was already checked.
Volker and mhirt, I had the traces button checked. When I unchecked it, that completely solved the problem. I understand the need for traces now. I just kind of figured traces was like an explain in MSSQL. I t does not make that much of a difference. Night and day. This baby really moves now. 24.61 seconds on nearly 18k rows.
I figured it was something simple and bone headed I was doing. Thanks!
--Harry