how to use loop condition in talend

[akumar2]

Hi team,

i am new to talend.

I am using source and target as postgresql Db ( i.e productiin db and product db). my production db will get new data daily and some will get modified. combined both will be nearly 5lakh records. now i need to do migration from production to product on daily basis. while migration i am getting connection timeout error. so i will am trying to implement loop condition with batch size. but dont know how to implement. Can any one help me.

Thnks in advance.

[Anonymous]

Hello

If you want to do a loop and migrate part of data each time until all data are migrated. tLoop component might help you, here is the component documentation and scenario.

https://help.talend.com/r/en-US/8.0/orchestration/tloop-standard-properties

https://help.talend.com/r/en-US/8.0/orchestration/trunjob-tloop-tlogrow-procedure-standard-component-create

Please learn this component and let me know if you have any issues/questions.

Regards

Shong

[jlolling]

If you get connection aborts the solution is NOT looping!

It would be helpful to see your job design. Sometime in large jobs a connection is opened right at the start and is used later but get lost in the meantime.

Other problems could be iterating over database table records and the processing for one record takes to long and this cause also sometimes problems because idle connections will often killed by the database itself.

So finally: share a screenshot of your job design.

[akumar2]

@Jan Lolling​  please find the attached ss of my job. here i am fetching data from source for 1 day and pushing to target for inserting and for updating. my source table will get new data daily and few records which are already present will get updated. i need to push this new data nd updated data to my target on daily basis.