If while reading records, job get failed due to some error on 11th records. After fixing in file/table. then if you rerun it read from 11th records. how to achieve this both case if source if file and table.
2.
1:1 load
Target is file/table.
If while writing into table/file. Job get failed on 1100th record due to invalid records.
after fixing record in source. then rerun the jobs. its load from 1100th records.
case 1: Load from 11th records
case 2: if tables get commited on 1000th records. then need to roll back all records and load from 1st record to last records.
case 3: if tables get commited on 1000th records. then need to load from 1001 to last records.
3. If you define context variable in job level and TAC parameter level.
if Job run from TAC: which parameter it is consider TAC parameter value or job level context variable value.
I would never make a job fail for the data problem in one record especially in the case of large files. If its schema related problem, you can do the check by using tschemacompliancecheck component and all the records which fails in the check can be send to reject flow for further analysis.
This will make sure that data is loaded correctly eventhough you have some problematic records. Now during insert, if you want to reject duplicate records, you can select the key column in target DB schema and select Action on table as result. If you right click the DB and and use reject link path, you can pass the rejected records to a file or reject table for further analysis.
I hope you got the method to process the data quickly instead of making the job fail and trying to restart from the point of failure.
Warm Regards, Nikhil Thampi
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂
The recommended approach is to capture the rejected records to a different file. You can do all the data fixes in that file and then use it to reprocess. Else you can send the rejected data to Talend Data Stewardship so that Data Stewards can correct the data and then send to the target area based on the work flow created in TDS.
Either way, there should be a traceability of the changes made. If you are changing the data in source file itself, and if the cosumer contested the change at a later stage, there will not be any track where the change was made (for example, the job failed due to invalid DoB and support team member put a random 01-01-2000 to process the file). But its a serious data privacy issue and the rejected data should not be processed in that manner.
Warm Regards, Nikhil Thampi
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂
