I would like to implement re-startability feature in talend jobs. Basically the goal is that if a job has been executed then in the subsequent runs it should pick only those records that have been not processed in the previous run. I do not want Talend to start executing from the start again. How can I achieve this?
The most flexible way to do this is to capture the state of your process. I have done this before where I will have a staging/state table where I will keep the record Id, and an Action/State flag.
For example, I can have the following state:
0 - Not processed
1 - Insert Record
2 - Update Record
3 - Update Dependencies or something else
4 - Delete Record
5 - etc etc
Then I process all the records and add/update the state table. Then I read from the state table to do further processing. In this way, each time I re-run the job, the job knows where it has reached in processing of records, and can re-start where it left. You can even force a record by switching the Action/State flag.
