Pass context variable or value from child job to parent job

[Anonymous]

Hi All,

I have a screnario where im doing file validation and moving the file from local to hdfs and creating external hive table in one job "job1" (trunjob1) , i am calculating the tot_rec_cnt in that subjob. Upon success or failure of subjob i want to trigger another subjob "job2" (trunjob) and want to pass the tot_rec_cnt from job1 to job2.

I have captured the tot_rec_cnt as a context variable and i want to pass the context to main job so that i can pass the same to my audit job(job2).

My job flow is like:

trunjob1(job1-filevalidation)  ----onsubjobok---- trunjob2(job2 audit job)

 |

onsubjoberror

|

trunjob2(job2 audit job)

I am calculating the tot_rec_cnt in trunjob1(job1) and want to pass the value to audit job. And adding the this, i want my audit job to called either for the success or failure of my file validation job

Kindly help to resolve this.

[Anonymous]

hi rhall,

will this work from child DI job to Parent Spark job?

I need to set the max value of seq column in DI job to a variable and access that in Spark Parent job.

when I follow above steps, I am getting null value when I read that static variable. routines.MyStaticVariables.maxval;

[Anonymous]

aarthi, did that work from DI job to Spark job?

[Anonymous]

Given how Spark works I am not entirely sure that this method will work. Can you not identify the seq number within the parent job? Remember that Spark jobs are not processed in the same way as DI jobs. A DI job is processed by one machine in a single JVM.