how to relaunch the job without re-read the file ?
Hello
My title isn't very clear.
Well, I have a job consists in :
1) reading a file in a directory
2) feed the database
3) Many tMap to check the "business rules" (in french : règles de gestion, i don't know to translate in english, sorry) with :
- If the data are correct, they are accepted and loaded in the database
- If the data have an error, there are rejected and loaded in the database with an error message
Actually, when there is an error, we must correct the error directly in the file and relaunch the job. (and delete the lines in the database before relaunch the job).
But now, i have a new need :
-No more to correct the error in the file BUT directly in the database.
But the matter, when i relaunch the job, Talend start all over again : Reread the file, reload the database, etc.
The correction of the error in the database is useless....
I don't know how to correct the error in the database and to start again where it stopped and not since the beginning.
(Ps : I am not the expert of Talend....Not yet....)
If you want to ignore lines which has been loaded by previous run, store the current line number as a context variable (see tContextDump / tContextLoad) and use it in the tFileInputDelimited "header lines" parameter.
You need to keep a logging table for each step to do this. For example, the first part of the process is loading the data from a file. Once that is successfully completed for that filename (I assume the filename is unique....it kind of has to be), you can move to the second step, which I gather is validating the data. Once that is done, then you can load the data into your success table. So there are 3 steps. If you keep a table to log the success or failure of each step (maybe handled by an individual job?), then when your process fails, you can restart from the job(process) that did not finish successfully.
For example, your logging table might look something like this....
File
Batch
Process
Start
End
Success/Failure
File.xml
1
1
10:31
10:32
Success
File.xml
1
2
10:32
10:33
Failure
File.xml
1
2
10:34
10:35
Success
File.xml
1
3
10:35
10:36
Success
File2.xml
2
1
11:00
11:01
Success
File2.xml
2
2
11:01
11:02
Success
File2.xml
2
3
11:02
11:03
Failure
File2.xml
2
3
11:03
11:04
Success
You would drive the process from this table and have conditional logic in your job (assuming a Main job with 3 child jobs....one for each process). So when you restart, if the file has previously failed, it will restart the process that failed and continue from there.
