topic Re: Multithread, iterate and tFlowIterate in Talend Studio

Multithread, iterate and tFlowIterate

aagustoni — Sat, 16 Nov 2024 04:43:16 GMT

Hello

I am using Talend TOS (V7), I don't have access to Enterprise version, unluckily

I have a job with the following design (see the attached screenshot)

First a query that select among the other info, n couples colorID/sizeID

Then a tFlowToIterate to make colorID and sizeID available in an iteration

On component OK the colorID/sizeID are used as filters to retrieve the data from an Oracle Table

The job works fine, but when the number of colorId/SizeId are many, it is quite slow to complete.

I have tried enabling iterate parallel parameters and things run much better (the execution time halves, of course).

The problem is that reading the documentation I have found that globalMap is not thread safe so I feel there are risks in making my design multithread (a mix up of colorIds or sizeIds during the executions would cause serious problems)

Is there a way to modify my design in order to make it thread safe?

Thanks for your help

Alessandro

Basicall

Re: Multithread, iterate and tFlowIterate

fdenis — Tue, 10 Sep 2019 13:18:22 GMT

yes, you may iterate to trunjob using context value in place of global.
with that configuration thread have there own variables.

be careful tjava is not explicite so we do not know what is the objectif of your job. not easy for maintenance.

Re: Multithread, iterate and tFlowIterate

aagustoni — Tue, 10 Sep 2019 13:26:14 GMT

The tJava does some simple log, no biggies

So if I understand well I

- keep the iterate with parallel execution enabled

- use the globalMap values JUST to populate subjob context Parameters

Should I check the "use an independent process to run subjob" too?

Many thanks for your help

Alessandro

Re: Multithread, iterate and tFlowIterate

fdenis — Tue, 10 Sep 2019 13:52:33 GMT

please to help.
validate the solution on your side and check as solved.
By

Re: Multithread, iterate and tFlowIterate

aagustoni — Tue, 10 Sep 2019 13:56:07 GMT

Hello

I will for sure, as soon as I tested...

Just a detail : Should I check the "use an independent process to run subjob" too or the check in the iterate flow is enough?

Re: Multithread, iterate and tFlowIterate

fdenis — Tue, 10 Sep 2019 14:02:43 GMT

yes check it then you will have independent java thread.