How to set and use globalVariable/context variable in Big Data Batch job

[Anonymous]

Hi All,

I am not able to set globalVariable/context variable in Big Data Batch job.  While, I can do so in standard job.

Is there any other way to set and use globalVariable/context?

[Anonymous]

Hello,

For bigdata batch job, the talend recommendation is to use a DI job 'launcher' that loads the "dynamic" context.

So far, there is also no tContextLoad component for Big Data Batch and Spark Streaming.

Here is a related article about:https://community.talend.com/t5/Architecture-Best-Practices-and/Spark-Dynamic-Context/ta-p/33038/jum...

Best regards

Sabrina

[Anonymous]

Thanks Sabrina.

We were able to set context variable from DI job and pass it to Spark job.

But we are not able to reset that variable during runtime in spark job.

For e.g. if we want to capture status of certain operation in spark job, we
are not able to set variable as "success" or "failed".