topic Re: tSortRow runs very slow with EXCEL Input in Talend Studio

tSortRow runs very slow with EXCEL Input

Anonymous — Sat, 16 Nov 2024 14:25:15 GMT

Hi!
I am trying to sort about 90K rows coming from a multisheet excel file. If I use Excel Input and Sort (I am sorting on 4 keys out of about 10 columns), tSortRow runs very slow (2 rows/sec). If I remove the excel file and put a CSV file created from the same excel, the performance improves many folds.
A workaround to this is, of course, to convert this XL file to CSV and then use that as an input. I am just trying to figure out why the response for tSortRow be different for different inputs.
Note that I am using the new XL enhancement that allows me to read multiple worksheets without using tUnite.
Thanks.
Regards,
Sean

Re: tSortRow runs very slow with EXCEL Input

Anonymous — Sat, 22 Mar 2008 04:58:27 GMT

Additional information for the above query:
I tried running a job that extracted XL to the CSV and in a sub job (on job OK), I read from the same CSV that was just created. The result is the same low sorter speed. When I just use the CSV file in a standalone job, I get very fast speeds.
My workaround right now is to have one job to create the CSV and another one to read the CSV and do the processing. I am trying to keep everything to one PERL script but this will result into at least 2 scripts.
Thanks
Sean

Re: tSortRow runs very slow with EXCEL Input

Anonymous — Sat, 22 Mar 2008 09:00:43 GMT

Hi Sean,
Have you tried with tExternalSortRow ? This component writes the incoming data flow to a temporary file before sorting it with gnu sort. It avoids data bufferization and thus reduces memory consumption.
Hope it helps.
Richard

Re: tSortRow runs very slow with EXCEL Input

Anonymous — Sat, 22 Mar 2008 14:37:26 GMT

I think there is no other problem that the way the "real time statistics" performance rate (rows/s above each row link) is calculated. Don't take it into account, just read the total execution time and you'll see the input doesn't affect tSortRow performances.
Talend Open Studio generated code model was designed so that components are independant.

Re: tSortRow runs very slow with EXCEL Input

Anonymous — Sat, 22 Mar 2008 15:22:25 GMT

Well. You are right. I am doing something wrong to slow this job down. When I created a simple job with EXCEL input, a sort and then the CSV output, it was fast. So I need to look more as to why that particular job is slowing down. I'll get back to you on that.
Regards,
Sean