topic Re: Are globalMap Variables specific to a row pass or do they interfere? in Talend Studio

Are globalMap Variables specific to a row pass or do they interfere?

Anonymous — Sat, 16 Nov 2024 02:20:25 GMT

Hello,

I need something clarified:

Are globalMap Variables specific to each single row pass (1 input flow / document) ?

Or do they interfere with each other ?

Background to the Question:

I am expecting alot of input documents and I will need to use if conditions to control which part of my job handles which document.

My Problem is now: It's very likely that another (if not many others) documents get processed (and arrive at the first if condition) while a document is still being processed (and still depends on variables for decisionmaking).

If I use globalMap Variables for this, will the incoming documents / "rows" interfere with eachother's variables?

If yes, is there anything I can use instead?

I haven't been able to find anything regarding this in the documentation so I apologize if this has been answered somewhere already.

Re: Are globalMap Variables specific to a row pass or do they interfere?

TRF — Mon, 25 May 2020 16:31:44 GMT

Do you have parts of your job which are parallelized?

If so, take to not reuse the same variable on these parts.

Also, if you want to "secure" your global variable you may have dynamic names.

For example, suppose you have a variable called "myVar" and you have a field called id on your document.

When you create the global var, you can use the following expression to make its name unique:

row1.id + "_myVar" // instead of "myVar"

Does this help?

Re: Are globalMap Variables specific to a row pass or do they interfere?

Anonymous — Mon, 25 May 2020 16:36:08 GMT

The Job is not parallelized, no.

Dynamic Variables might be a solution but that would massively swamp the memory would it not?

I'm expecting the job to run non-stop with several hundreds of incoming documents per day.

Furthermore: How would I use such a dynamic Variable in an "Execute when" connection?

Reading between the lines, there are not row pass specific flags/variables we can use?

Or can we somehow free up these dynamic variables ?

Re: Are globalMap Variables specific to a row pass or do they interfere?

Anonymous — Tue, 26 May 2020 08:57:01 GMT

Is there maybe a variable available in Talend that identifies a specific iteration / row pass?

That would solve my Issue hopefully, the dynamic variables sadly do not (not every subjob accesses the document and so won't be able to access the dynamic variable).

I'd be really grateful for any info / help on this.

Re: Are globalMap Variables specific to a row pass or do they interfere?

TRF — Tue, 26 May 2020 09:01:17 GMT

Can you share your job design?

Re: Are globalMap Variables specific to a row pass or do they interfere?

Anonymous — Tue, 26 May 2020 09:26:01 GMT

I cannot share the actual Job design, but I've attached a example with a similar Structure.

The gist is:

First each incoming document gets analyzed and various data is extracted.
Based on this flags are set to determine how this document needs to be processed (multiple flags can be true / 1 document can be processed in more than 1 subjob).

After that there are "Execute if" connections to the processing subjobs.

Until recently I reset the flags inside these subjobs, but our input documents changed and now the job has to account for the possibility that 1 document needs to be processed in more than 1 way / subjob.

This in return prevents me from resetting the flags inside the respective subJobs and am now afraid that the flags will get messed up by following documents while a respective document is still awaiting processing.