topic Re: How to optimize tSortRow with 10M rows ? in Talend Studio

How to optimize tSortRow with 10M rows ?

Mayot — Fri, 24 Aug 2018 08:52:11 GMT

Hello,

Do you have tips to optimize a 10 million lines processing with a tSortRow before inserting it into the database?

I have good performance at the beginning (~ 6600rows / s), the more the number of treated lines increases, the more the performances decrease. Arrived at 600 000 lines, I have the error OutOfMemoryError: GC overhead limit exceeded (I could increase the memory of the JVM for the job, but I think it's not optimal)

Thanks.

Re: How to optimize tSortRow with 10M rows ?

Jesperrekuh — Fri, 24 Aug 2018 10:49:40 GMT

In your tSQLinput query add : order by <column>

Alternatively, it sounds like a load once to this table? say from a multiple file source, first store them in smaller fragments, write output based on some logic... like a file for each week of the year / data you want to sort by. Then process these smaller files and sort them before writing to db.

Alternatively, write to a tmp table, and next write a tsql : insert into finaltable as select... from tmptable order by your columns.

Re: How to optimize tSortRow with 10M rows ?

Mayot — Fri, 24 Aug 2018 13:34:25 GMT

Thanks, i will try this