Same subjob executed n-times in parallel

[Lorenzo5]

Hi,

is there any way to execute the same subjob, n-times ('n' configurable dynamically, i.e. from context) in parallel, _without_ using tRunJob (and executing a Job created with that subjob)?

Regards,

Lorenzo

[TRF · Accepted solution]

Shortly and to strickly answer your question, no.

[Dezzsoke]

If you generate N rows then do a tFlowToIterate -> (Execute x N) -> {rest of your job}

Then you basically execute the same job on dynamic number of threads in parallel.

[Lorenzo5]

@Dezzsoke, in your solution they are not executed in parallel but serially, correct?

[Lorenzo5]

Hi @TRF, Hi @xdshi

then, please, help to determine how to run the same job, multiple time, in parallel, using tRunJob but in an asynchronously way. Here is the scenario:

I have multiple log files to load in our DB, and I would load them in parallel.

I don't know in advance how many of then I have to load, so I can only read the directory content and loop on the file list. 

When I loop on the file list, how can I execute tRunJob for each list item, in an asynchronously mode, which mean multiple time without waiting for the end of each execution?

And how can I be aware about the end of all the executions?

Regards,

Lorenzo

[Lorenzo5]

I have an idea, please let me know if it sounds good.

Problem:

a dynamic list of files to be loaded in parallel into a DB.

Design:

2 jobs, a main one and a recursive one.

- The main one read the list of files and, thanks with the tRunJob, execute the "Recursive" one, passing it the list of files.

- The "Recursive" one decides: if the list contains only 1 file, then it reads and loads the file into the DB; if the list contains two or more files, then it splits the list in two sublists and pass them to two jRunJob components in parallel (tParallelize), which run two separate instances of the same "Recursive" job.

What do you think about this solution?

[Lorenzo5]

I have an idea, please let me know if it sounds good.

Problem:

a dynamic list of files to be loaded in parallel into a DB.

Design:

2 jobs, a main one and a recursive one.

- The main one read the list of files and, thanks with the tRunJob, execute the "Recursive" one, passing it the list of files.

- The "Recursive" one decides: if the list contains only 1 file, then it reads and loads the file into the DB; if the list contains two or more files, then it splits the list in two sublists and pass them to two jRunJob components in parallel (tParallelize), which run two separate instances of the same "Recursive" job.

What do you think about this solution?

[Dezzsoke]

Use enterprise version of talend, then on the iterate link you can set up how many threads you want.

The Async solution you want is just way too risky.

[Lorenzo5]

Very interesting! I didn't know about that! And yes, I have the Enterprise version and the parallel feature available.

Thanks!