topic Re: Dynamic Jobs in parallel - How to handle logs in console in Talend Studio

Dynamic Jobs in parallel - How to handle logs in console

Anonymous — Fri, 24 Jul 2015 16:46:07 GMT

Hi all,

We want to make an existing projet "multithreading compatible".

For now, most of the jobs are executed through tRunJob component with "Use dynamic job" option enabled and it runs fine. Those child jobs return information to Master Job through console using system.out.println() in tJava component.
To illustrate my words, let's say my project is built that way :
Master Job
|__Child Job A
|__Child Job B

Master Job runs through tRunJob component Child Job A then sequentially Child Job B.

Child Job A returns in console :
[list=*]

Child Job B returns in console :
[list=*]

In my Master Job, console should display :
[list=*]

Now, the parallel scenario.

We successfully enabled the parallel execution of Child Job A & Child Job B, but console now displays something like :
[list=*]

This display is not easily readable and not compatible with our requirements.

Do you have any clue on how to get the initial display ? Namely :
[list=*]

Changing the way the child jobs return information to master job is hardly an option as dozen of child jobs are concerned.

Many thanks in advance for any help !

Regards,

Sylvain

Re: Dynamic Jobs in parallel - How to handle logs in console

Anonymous — Tue, 11 Aug 2015 04:02:09 GMT

Hi Sylvain,

What's the workflow in your current master job? How did you setting the parallel execution of Child Job A & Child Job B (tParallelize or 'enable parallel execution')?
Screenshots will be preferred.

Best regards
Sabrina

Re: Dynamic Jobs in parallel - How to handle logs in console

Anonymous — Tue, 11 Aug 2015 09:07:10 GMT

Sylvian, if you have two people (A and B) counting to 4 one after the other, you get the following output....
A - 1
A - 2
A - 3
A - 4
B - 1
B - 2
B - 3
B - 4
If they count in parallel, you get something like the following if you have to log the instant they utter the number...

A - 1
A - 2
B - 1
A - 3
B - 2
A - 4
B - 3
B - 4
Basically, what you are asking for will need a rework so that ALL of the output from each job happens at once. You will not be able to guarantee the order in which the jobs are logged, but you will be able to force the output in one continuous chunk. However, this will require a significant amount of rework to store the messages in memory to output them all at the end of the job.