topic Re: Dynamic Variable - iterate over list in Talend Studio

Dynamic Variable - iterate over list

talendstar — Sat, 16 Nov 2024 05:06:34 GMT

I have a scenario where I need to iterate over a list(table column) and execute python code for whatever value variable holds at that moment and then move of the next value in the list - till all the items in the list have been processed.

Here is some background:

I have a table with a column that contains branch code.

TableA

BranchCode

123

124

125

126

127

128

129

130

Minimal Solution

What I would like my Talend flow is to iterate over the values in the column - for example for branch 123, set the variable, pass the variable to python code and execute python code. After that move on to next value in the column 124, set the variable, pass the variable to python and execute python code.. and so on.

Ideal Solution:

Ideally what I would like to do is split long list of values in the column between a parallelized workflow where each flow within the parallelized job take a set of values (let's say 25% each) and iterates over that. But this is more complex than minimal solution. So flow1 will get 123,124 and flow2 will get 125,125 and so on.

How can I accomplish this in Talend?

Thanks

Re: Dynamic Variable - iterate over list

Anonymous — Tue, 30 Jul 2019 22:52:39 GMT

Hi @talendstar ,

You can use the TalendDBComponent ->tFlowToIterate->tRunJob (in tRunJob settings add the globalVaribale of flow to Iterate as context)

Your tRunJob will have the code for python.

Thanks and Regards,

Subhadip

Re: Dynamic Variable - iterate over list

talendstar — Wed, 31 Jul 2019 14:13:20 GMT

Is it possible to pass the variable to a Python script that is run with tRunJob? or is there any other way to pass the variable to a python script?

thanks

Re: Dynamic Variable - iterate over list

Anonymous — Wed, 31 Jul 2019 14:59:47 GMT

Hi,

You cannot run the Python code directly using Talend components. The trick is to call them using Talend terminal components like tSystem, tSSH etc. where you can pass the value to your Python script through Windows commands or Linux Shell Scripting.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Dynamic Variable - iterate over list

talendstar — Wed, 31 Jul 2019 15:47:56 GMT

I usually run python scripts via .bat files and call the .bat file via tsystem.

Is there an example of passing variables to tsystem and then onto .bat file or python script that I can refer to?

thanks

Re: Dynamic Variable - iterate over list

Anonymous — Wed, 31 Jul 2019 16:03:05 GMT

Hi,

Could you please check the below link? Its not exactly same but will give you an idea.

https://community.talend.com/t5/Design-and-Development/passing-convext-parameter-value-to-the-shell-scripts-in-tSSH-to/td-p/41065

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved