topic Re: tinfiniteloop and global variable in Talend Studio

tinfiniteloop and global variable

Pariniranjan — Sun, 07 Jun 2020 14:20:12 GMT

I want to run a subjob in a loop. My design would go something like below

tinfiniteloop -> trunsubjob

Subjob:
Oracle sp call ->
On subjobok ->
Oracle input -> tmap -> tsetglobalvar -> twritejsonfield -> trestclient -> tdbrow

To describe above, I would need to call a stored procedure which inserts data from table A to table B based on a date condition . Then onsubjob ok , extract data from table B and map it and set one of the column which is unique into global variable and then form a json request for each of the record retrieved from table B and post to rest API. Upon successful response, retrieve the global variable of the current record and update a column "processed flag" on table B from "no" to "yes"

Here are my questions,

1. Would set the wait for each iteration on tinfiniteloop to 15 minutes. If a current iteration exceeds 15 minutes, would tinfiniteloop wait for it to complete or would it trigger anyway the next iteration?

2. In this scenario, would global variable be thread safe? Meaning I'm taking each database record and making it a json for rest API and updating this record again upon successful response

3. Is this design for the use case feasible or any suggestions are appreciated

Re: tinfiniteloop and global variable

Anonymous — Mon, 08 Jun 2020 03:05:50 GMT

Hi
tInfiniteLoop will trigger anyway the next iteration when the wait time is up. If you need to wait for the last iteration finishes, and the next iteration will be executed only when all the steps in last iteration executed successfully. You can use tLoop component with While type, define the Condition field with a context variable (boolean type, the default value is true). Set the condition variable as true in the end of iteration if all the steps execute successfully, otherwise, set it as false.

Regards
Shong

Re: tinfiniteloop and global variable

Pariniranjan — Tue, 09 Jun 2020 20:00:28 GMT

Thanks Shong. I went with tinfiniteloop anyway and incorporated tfiletouch and tfiledelete to have lock file over current iteration so subsequent steps are not executed on the next timed trigger when the current iteration is still in progress

Also I've a follow up question. I'm currently using tflowtoiterate from a database to iterate over each record and pass it on to rest api. The api returns generic response. So I need to keep track of each record and update one of the column named 'processed flag' after a successful response from the api. Is it advisable to use tflowtoiterate this way? Say in one of my run, there are close to 100k records extracted, would tflowtoiterate hold all these records as key value pairs or is there any limitations?
tmp_1591728413025.jpg

Re: tinfiniteloop and global variable

Anonymous — Wed, 10 Jun 2020 03:33:28 GMT

Yes, I will use tFlowToterate if I need to iterate each row for next business processing, and I will consider enabling parallel execution for the iteration to improve the performance.

If you read the generated code of job, you will see it is tOralceInput query all data and cache them into memory, and then iterate each record, tFlowToterate does not hold all these records.

Re: tinfiniteloop and global variable

Pariniranjan — Wed, 10 Jun 2020 18:09:12 GMT

Thanks again. This helps