topic Re: sort data of large size in parallel in Talend Studio

sort data of large size in parallel

CChemali1616516071 — Tue, 23 Mar 2021 16:25:12 GMT

Hello,

I am using Talend Data Integration Studio and looking to sort large data set using components

tFileInputPositional, tSortRow and tUniqRow

The run is slow so i am exploring enabling parallelization but i cannot find the "Set

parallelization" option

How can i integrate this option in Talend Data Integration Studio?

Thanks

Re: sort data of large size in parallel

Anonymous — Wed, 24 Mar 2021 06:01:51 GMT

Hello,

If you set the job running in parallel it means you can have different start points and they will start in different threads in parallel.

With talend subscription solution, you will have the component tParallel in which you can trigger multiple sub jobs in parallel and wait for the end of all of them.

In your job, there are some cache component consuming two much memory. such as tUniqRow and tSortRow. For a large set of data, try to store the data on disk instead of memory on tUniqRow and tSortRow. Also, allocate more memory to execute the job.

Best regards

Sabrina

Re: sort data of large size in parallel

CChemali1616516071 — Wed, 24 Mar 2021 13:25:27 GMT

Hi Sabrina,

Thanks for your reply!.

I did use the disk and allocated more memory but the run is still slow

I am trying to implement the parallelization described in the Talend Data integration guide link below

https://help.talend.com/r/9bBURCEt_t~lUHE3DeE2LA/kOE3mfYu2ConQETvTlcb0Q

in this guide there is bullet below about activating parallelization

"Right-click the start component of the Job, tFileInputDelimited in the scenario, and from the contextual menu, select Set parallelization.Then the parallelization is automatically implemented."

I do not have this option with the Talend i downloaded

you mentioned about " talend subscription solution"

How can i subscribe?

Thanks!

-Chadi

Re: sort data of large size in parallel

Anonymous — Thu, 25 Mar 2021 06:50:24 GMT

Hello,

The Parallelization tab is available in talend subscription solution(paid version) not open source.

Best regards

Sabrina

Re: sort data of large size in parallel

CChemali1616516071 — Thu, 25 Mar 2021 13:14:50 GMT

I see. thanks !

Re: sort data of large size in parallel

Anonymous — Mon, 29 Mar 2021 06:36:29 GMT

Hello,

Feel free to let us know if there is any further help we can give.

Best regards

Sabrina