Context Variables (sometimes) not properly transmitted to Dynamic Subjob

[Anonymous]

Hi,

I want to build an ETL project with quite a lot of jobs. To be able to organize these jobs, I decided to go with a dynamic job design, which you can see in the attached file. In tFixedFlowInput I define a list of jobs, that should be run. These get executed in the tRunJob, with the settings shown in the second attached file.

As you can see I checked the box "Transmit whole context". Now there are some context variables that get set or changed during runtime. For example I need to determine the time, the very same job ran successfully last. The result (start time of last successfull run) is then stored in a predefined context variable that belongs to a context group and transmitted to the subjob.

Now my problem: For some reason one of these context variables that get set on runtime isn't transmitted to the subjob. If I change the Dynamic Job to a "normal" job (right now there is only one job in the job list, so this is quite simple to achieve) all of the context variables (there are around fifty of them) are transmitted as expected.
Is this a known issue with dynamic jobs or even better, is there a fix to this problem other than changing the design to "non dynamic"?

Thank you very much in advance for your help.

[Anonymous]

Hi,

    Could you please show us how you are allocating value to this problematic context variable? Did you check the value in the parent job by printing the variable to console and see the results?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[tchandu]

Hello,

        Your assigning the value to the context variable inside the sub job but you do not return that variable back to the main job. Even it is from the context group it's scoop is with in the sub job. Return value back to main job and try.

[Anonymous]

Hi there,

thank you very much for your fast reply. As my ETL project already consists of quite a lot of jobs, I hoped, I could avoid explaining the structure, but I will try 🙂

As you can see in the file extractOrchestration.jpg the values are derived from another Subjob (tRunJob_3). This Subjob determines three values that are transmitted to a tJavaFlex component. In that component the context variables of the job "extractOrchestration" are set, as you can see in the file contextVariablesAssignment.jpg. The context variables then get transmitted to the other two subjobs (tRunJob1 and tRunjob2) with the setting "Transmit whole context" which works perfectly fine. 

When running the tRunJob_2 from the extractMgmtDbOrchestration in dynamic mode though, one of the context variables "context.root_job_end_unix_time" isn't transmitted. Instead the default value is shown.

All the values are printed to the console in every subjob. And, as I metioned before, when running tRunJob_2 in extractMgmtDbOrchestration as "normal" job (not dynamic) everything works perfectly fine as well. So it seems to me that something went wrong when the context variables are transmitted to the new JVM. It's very weird though that only one of the context variables seems to get lost...

Any ideas? 

Thank you very much in advance!

ExtractMgmtDbOrchestration.jpg
contextVariablesAssignment.jpg
extractOrchestration.jpg

[Anonymous]

Hi,

    I would recommend you to create a support case since I am assuming that it could be due to code changes for dynamic job. If you are using open source version, could you please create a JIRA ticket in the below link.

https://jira.talendforge.org

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved