topic Re: Run a job looping through all defined contexts? in Talend Studio

Run a job looping through all defined contexts?

Anonymous — Thu, 13 Jul 2017 19:39:32 GMT

I have a job where I am grabbing a most recent file in Box, unarchiving it and then appending data from the file (.csv) to a Google Sheet.

For ease of set up all the reports for different clients are going in one directory. I have defined using contexts filenames to use as a mask as well as google sheet file IDs and Sheet Names.

So when I want to update a particular report I select its context and run the job which works fine and as intended.

What I want to do is to automate running my job with all the contexts I've defined - I'm quite new with Talend so struggling to find how to do this?

Re: Run a job looping through all defined contexts?

Anonymous — Thu, 13 Jul 2017 22:53:32 GMT

A Context Group is a functional element in Talend to group a set of context variable values. The actual context group name is not passed down into the job, but instead the context variable values are.

Hence, instead of passing the values to the job, you need to think of loading the values dynamically at runtime. You can use a tFileInputDelimited to read a file with 2 columns, i.e. Key, Value, and then push that row into a tContextLoad component. This will override the values of the context variables in the current job. You will have as many properties as context groups.

At runtime, you will loop and dynamically load context variables from 1 properties file, do your processing, and then load the next set of context variable values and do your processing.

The 2 components you need to look at are tLoop and tContextLoad.

Re: Run a job looping through all defined contexts?

Anonymous — Thu, 13 Jul 2017 23:11:46 GMT

Thanks for the swift reply.

I did try tContextLoad with tFileInputDelimited but couldn't think how to iterate through the rows and do my processing one at a time.

Also a bit confused with the context groups. Should I create a group per variable I want to dynamically change and have a tContextLoad for each or can I have one for the 4 variables I am looping through (1 for client in file name, 1 for campaign in file name, 1 for gsheet file ID and one for gsheet sheet name)?

Re: Run a job looping through all defined contexts?

Anonymous — Thu, 13 Jul 2017 23:19:01 GMT

The Context and Context Groups are just logical grouping of your context variables at design time. They do not impact the way the code are generated.

If you have 10 variables across all contexts, then you need 10 variables in the file.

I generally create a file like this:

variable1=value1

variable2=value2

Just like a properties file.

I can use tFileInputDelimited to load the data in 2 columns key, value, with column separator = and new line separator.

If you have a key in your file and it matches a context variable, the value is set. If the variable does not exist, you get either a warning or error depending on your tContextLoad settings. If you do not specifiy a value for a variable in the file, the value of the variable in the job is not overridden.

Re: Run a job looping through all defined contexts?

Anonymous — Thu, 13 Jul 2017 23:23:40 GMT

Thanks for the tips, I'll have a play around tomorrow as hasn't still quite all fit together in my head.
Specifically with the context file, does it mean I need a file for each of the Contexts I want to load? I.e. I can't have more than one value per variable in the same file and iterate through them one at a time?

Re: Run a job looping through all defined contexts?

Anonymous — Thu, 13 Jul 2017 23:26:15 GMT

It is up to you to decide how to store the values. You can use a normal CSV format with 3 columns and do data integration basically. You load all the data, filter on a column, and then push the key, value to the tContextLoad component.

tFileInputDelimited -> tMap - > tContextLoad

Re: Run a job looping through all defined contexts?

Anonymous — Fri, 14 Jul 2017 19:28:22 GMT

Hi there, can you kindly help me a bit more?

After lots of different non working options with different components (tLoop, tJavaFlex) I think I've got the closest I've been to a working solution with the below.

You can see the code in the tJava that is running my loop below.

It basically works fine and loads my first set of context variables as intended but then when the child job connected to this one finishes the whole job ends.

I tried using a trigger from component/s in the child job to connect to tJava_3 and make it run again but that's not allowed.

Any tips?

System.out.println("tJava_3: Start code");
for(int i = 1; i < 2 && i<((Integer)globalMap.get("doneIteration")); i++)
{globalMap.put("currentIteration", i);
Integer currentIteration = i;
System.out.println("Current Iteration is: " +currentIteration);
System.out.println("tJava_3: Main code: i=" + i);
    }
System.out.println("tJava_3: End code");