topic Re: howa to optimise tUniqRow and tSortRow in Talend Studio

howa to optimise tUniqRow and tSortRow

Anonymous — Mon, 05 Feb 2018 15:27:46 GMT

Is it better to put tSortRow before tUniqRow or vice versa for the best perfermence? Or to use tAgrregateSortedRow instead of tUniqRow? if not how to optimize tUniqRow? knowing that I use "disk option", and how the work crash. I am working on a file of 3 million lines

Re: howa to optimise tUniqRow and tSortRow

Anonymous — Tue, 06 Feb 2018 09:06:49 GMT

Hello,

Performance issue is usually caused by the DB connection or the job design, could you please upload some screenshots of your current job design?

Best regards

Sabrina

Re: howa to optimise tUniqRow and tSortRow

Anonymous — Tue, 06 Feb 2018 20:37:10 GMT

Usually you would want to put a tSortRow before a tUnique row. It makes more sense logically to sort your data before finding the unique values, especially if you are using tAggregateSortedRow. In terms of using tAggregateSortedRow or tUniqRow, it would depend on what you are trying to do. tAggregateSortedRow is meant to find metrics based on values and calculations, while tUniqRow is more for finding duplicates.

Re: howa to optimise tUniqRow and tSortRow

Anonymous — Wed, 07 Feb 2018 08:25:24 GMT

below the job schema ..... i used sorting on disk in tSortRow and the problem comes from tUniqRow

tOracleInput
|
|
tFileInputPosionnel ---- tMap
|
|
tUnit -------------------------->tSortRow------------>tUniqRow------------------>tFileOutputPosinnel
|
|
tOracleInput--------- tMap

Re: howa to optimise tUniqRow and tSortRow

Anonymous — Mon, 12 Feb 2018 19:31:45 GMT

If you are running into a Java Heap Space issue you could increase the JVM so you can process more records or you could also use the "Use of Disk" setting on the tUniqRow advanced settings as well. This will process the data with files, making it use less system memory to process data.