topic Re: Global Map and Parallelization and Iteration in Talend Studio

Global Map and Parallelization and Iteration

mborlo15 — Wed, 05 Jan 2022 03:28:20 GMT

I noticed in this documentation link

that there apparently is some unreliability with global Map when parallelization and/or iterate components are used. I wanted to better understand why this is the case. I have two scenarios that use parallelization, global variables, and iterate connections. Would these uses of the features cause issues with the global map?

Re: Global Map and Parallelization and Iteration

gjeremy1617088143 — Thu, 06 Jan 2022 09:10:52 GMT

Hi, I think it's when you use iteration with parallel execution activated, so iteration are processed in parallel and because you set a globalVar in the iteration it can be overwritten by an other iteration in parallel.

eg: you have one iteration executed in paralllel, a first iteration with a globalVar set to 1 an other with the same globalVar set to 2 maybe 2 will be used in the first one or 1 will be used in the second one, because they are not processed one after one but in the same time.

Send me love and kudos

Re: Global Map and Parallelization and Iteration

mborlo15 — Fri, 07 Jan 2022 06:18:46 GMT

Ah ok, so my examples above would be unaffected because each thread would have it's own isolated instance of the child job, right?

Re: Global Map and Parallelization and Iteration

gjeremy1617088143 — Fri, 07 Jan 2022 16:02:30 GMT

I think Yes

Re: Global Map and Parallelization and Iteration

mborlo15 — Sat, 08 Jan 2022 01:21:08 GMT

Can Talend employees confirm this? It’s key for me to understand this so that the right data is sent to the right places. @Xiaodi Shi

Re: Global Map and Parallelization and Iteration

Anonymous — Mon, 10 Jan 2022 07:03:07 GMT

Hello,

We will check it with our R&D team and come back to you as soon as we can.

Best regards

Sabrina

Re: Global Map and Parallelization and Iteration

mborlo15 — Thu, 17 Feb 2022 16:45:40 GMT

See @Richard Hall 's response on this matter. It is a result of globalMap being an instance of java.util.HashMap, which is not thread safe.

https://community.talend.com/s/feed/0D55b00006viq18CAA?language=en_US

for more details see

https://stackoverflow.com/questions/18542037/how-to-prove-that-hashmap-in-java-is-not-thread-safe