topic Re: how to use loop condition in talend in Talend Studio

how to use loop condition in talend

akumar2 — Wed, 25 Oct 2023 08:52:45 GMT

Hi team,

i am new to talend.

I am using source and target as postgresql Db ( i.e productiin db and product db). my production db will get new data daily and some will get modified. combined both will be nearly 5lakh records. now i need to do migration from production to product on daily basis. while migration i am getting connection timeout error. so i will am trying to implement loop condition with batch size. but dont know how to implement. Can any one help me.

Thnks in advance.

Re: how to use loop condition in talend

Anonymous — Thu, 26 Oct 2023 08:02:46 GMT

Hello

If you want to do a loop and migrate part of data each time until all data are migrated. tLoop component might help you, here is the component documentation and scenario.

https://help.talend.com/r/en-US/8.0/orchestration/tloop-standard-properties

https://help.talend.com/r/en-US/8.0/orchestration/trunjob-tloop-tlogrow-procedure-standard-component-create

Please learn this component and let me know if you have any issues/questions.

Regards

Shong

Re: how to use loop condition in talend

jlolling — Thu, 26 Oct 2023 15:36:09 GMT

If you get connection aborts the solution is NOT looping!

It would be helpful to see your job design. Sometime in large jobs a connection is opened right at the start and is used later but get lost in the meantime.

Other problems could be iterating over database table records and the processing for one record takes to long and this cause also sometimes problems because idle connections will often killed by the database itself.

So finally: share a screenshot of your job design.

Re: how to use loop condition in talend

akumar2 — Tue, 31 Oct 2023 04:56:50 GMT

@Jan Lolling please find the attached ss of my job. here i am fetching data from source for 1 day and pushing to target for inserting and for updating. my source table will get new data daily and few records which are already present will get updated. i need to push this new data nd updated data to my target on daily basis.