topic Re: Shared variable between 2 jobs in Talend Studio

Shared variable between 2 jobs

Anonymous — Mon, 02 May 2016 18:56:24 GMT

Why does this simple thing seem impossible?
a) I have a PARENT job that calls SUBJOB1, then SUBJOB2 after SUBJOB1 is done

b) All 3 jobs share the same context with a context variable called "listOfThings" initialized to blank

c) SUBJOB1 appends data to the "listOfThings" context variable.

d) When SUBJOB2 reads "listOfThings", it is blank

e) I tried this with "globalMap" as well to no avail, it appears "globalMap" is only global to the current job and re-created for every job and hence has no data in it on SUBJOB2

f) If context var "listOfThings" is initialized to "1" this value is available to all subjobs, HOWEVER if any subjob mutates this context variable, that mutation does not carry over to subsequent subjobs that are executed.

This just seems like a very simple thing to me, to allow multiple jobs to interact with one another via some sort of thread-local variable or true "global" variables that span a job and any subjobs it invokes.
Confused and in need of a solution to this.

Re: Shared variable between 2 jobs

Anonymous — Tue, 03 May 2016 09:58:50 GMT

Here's one approach that you can take.
http://www.talendbyexample.com/talend-returning-values-from-subjobs.html
See the section "Returning Values through Context"

Re: Shared variable between 2 jobs

Anonymous — Tue, 03 May 2016 14:58:20 GMT

Simplest solution seems to be just create a "routine" backed with a ConcurrentHashMap that exposes get/put methods. Since an entire job is in the same Classloader this static instance is available to all jobs in the process.

Re: Shared variable between 2 jobs

Anonymous — Tue, 03 May 2016 17:57:17 GMT

Context variables and globalMap variables are only available to components inside individual jobs automatically. You can send then to child jobs (as I guess you are aware) pretty easily, but sending the values back to the parent is not so nice. The way they work makes sense (ie changing them in child jobs not automatically changing them in parent jobs), but it is annoying having to manually pass the values back if you want the values to change with the child jobs.
Your solution (the ConcurrentHashMap) is what I use if I need to do this. I think it could be handled in a better way by having an option on the tRunJob component to allow the contexts that are sent to the job to be passed back to the parent. Maybe this could be an enhancement Talend?

Re: Shared variable between 2 jobs

Anonymous — Tue, 03 May 2016 18:28:45 GMT

Yes the whole thing is annoying and completely non-intuitive the way they have designed it.
Create yourself a Routine with a static map member and just read/write from that. Shared classloader will ensure that all jobs/subjobs in one job access the same data. No clicking on buttons and moving variables around etc. i.e. you just do: MyRoutine.set(varname,value) and MyRoutine.get(varname)

Re: Shared variable between 2 jobs

swambulkar — Tue, 12 Jun 2018 23:14:33 GMT

Hello:

Can you provide example of this concurrentHashMap routine with get / set methods?

Thanks

Re: Shared variable between 2 jobs

Anonymous — Fri, 12 Jul 2019 08:26:15 GMT

Create a new Routine (under code in the repository list), call it UniversalMap, and paste this in there:

package routines;

import java.util.concurrent.ConcurrentHashMap;

public class UniversalMap {

    private static ConcurrentHashMap<String, Object> map = new ConcurrentHashMap<>();

    public static <T> T put(String key, T value) {
        return (T) map.put(key, value);
    }
    
    public static <T> T get(String key) {
    	return (T) map.get(key);
    }
}

Then you can use it like this:

UniversalMap.put("hello","world");
String who = UniversalMap.get("hello");
System.out.println(who);

Re: Shared variable between 2 jobs

PhilHibbs — Fri, 12 Jul 2019 15:42:32 GMT

This is my version of the sharedMap infrastructure.

1. Add a context variable sharedMap to each job

2. Create a joblet:

3. Add this joblet to each job, and link it using an onComponentOK link at the start of the job

4. In the tRunJob components, pass the sharedMap context variable the value context.sharedMap

5. In PARENT, put the following code in a tJava:

((java.util.map) context.sharedMap).put( "listOfThings", "" );

6. In SUBJOB1, put the following code:

((java.util.map) context.sharedMap).put( "listOfThings", (String)(((java.util.map) context.sharedMap).get("listOfThings")) + my.thing );

...where my.thing is whatever you want to add

7. In SUBJOB2, use the following wherever you want to use the listOfThings:

(String)(((java.util.map) context.sharedMap).get( "listOfThings" ))