Main job with jobs in sequence keeps running

[Anonymous]

Hi

I have two jobs and each of these jobs are getting messages from Kafka topic and pushing into DB. These jobs terminate when there are no messages on kafka topic after 10 secs. Individually both these jobs works as expected. Now I have created a main job that simply calls these jobs in sequence tJob1 - tJob2. When I run this main job I expect the job to terminate automatically after 20 secs if no messages on topic. But this main job keeps running for 6 mins before terminating. Any pointers?

Starting job I900_901_Plan_Main at 11:35 11/12/2019.

[statistics] connecting to socket on port 3815
[statistics] connected
tRunJob_1 in I900_901_Plan_Main call mars_project.test_consume_job_0_1.test_consume_job with:

[--father_pid=enEB33, --root_pid=enEB33, --father_node=tRunJob_1, --context=Default, --stat_port=3815, --parent_part_launcher=JOB:I900_901_Plan_Main/NODE:tRunJob_1]

tRunJob_2 in I900_901_Plan_Main call mars_project.test_consume_job2_0_1.test_consume_job2 with:

[--father_pid=enEB33, --root_pid=enEB33, --father_node=tRunJob_2, --context=Default, --stat_port=3815, --parent_part_launcher=JOB:I900_901_Plan_Main/NODE:tRunJob_2]

312957 milliseconds
[statistics] disconnected

Job I900_901_Plan_Main ended at 11:41 11/12/2019. [exit code=0]

[Raghuram_Puram]

The condition you put in job1 and job2 to check messages from kafka topic, put that in the main job instead. See if this approach works.

Any screenshots of the job would be more helpful.

Cheers

[Anonymous]

How can I include topic specific setting in main job. See screenshot below

Main Job:

Child Job1 (Job2 is exact clone of job1 but listens to different topic):

[Anonymous]

In fact what I noticed is that Sub Job is not at all executed in Main Job. Its same behavior when I have the Subjob components entirely in  the same job. Is it a bug that tKafkaInput does not work when used in conjunction?

See below second subjob just gets stuck waiting to be executed

[Anonymous]

Hi,

   I could see that the first subjob got executed successfully.

   Could you please check whether the job is running fine when we are trying to print it in Console instead of trying to write to DB? Please also check whether Die on Error option is selected in tDBOutput component.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

Yes tDbOutput die on error is checked.

I have tried with just tKafkaInput and logging the output to screen but its same result (i.e. gets stuck in second subjob).

If I remove tKafkaInput keeping the tDBOutput component along with tJava it works fine.

This indicates that somehow use of tKafkaInput subsequently causes program to wait even when there are messages to be picked up in second topic.

[Anonymous]

I saw similar topic but without solution

https://www.talendforge.org/forum/viewtopic.php?id=45987

Any clue if it could be a bug in Talend?

[Anonymous]

Any help in this regard is appreciated.

Thanks.

[Anonymous]

Hi,

   I could see that the first Kafka input seems to be working for you. Could you please double check whether there is any difference between connection details? Could you please also check whether the subjobs are working fine when you are trying to run them as separate Talend jobs?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

Yes its same connection in both subjobs and as I provided my comment earlier

These jobs terminate when there are no messages on kafka topic after 10 secs. Individually both these jobs works as expected.