Best Practices for Error Handling, tLogCatcher, and tDie in Multi-Level Job Design
Hi all,
I’d appreciate some guidance on error handling in Talend, especially regarding the correct use of tLogCatcher and tDie across multiple job levels.
Here’s my current setup:
I have a MAIN job that handles high-level orchestration. It writes to a HEAD audit table at the start and end of the process. If an error occurs, it updates the corresponding audit row with error info.
The MAIN job calls a STAGE subjob. STAGE loops through multiple source systems. For each source, it transfers all tables to a destination DB.
For each table transfer, there’s an audit row written in a DETAIL audit table. This row indicates success or failure per table.
My questions:
Where should tLogCatcher be used?
Only in the MAIN job?
In all jobs (MAIN, STAGE, and subjobs)?
Where should tDie be used?
Only in subjobs when something critical fails?
Should it be used to bubble up errors to the MAIN job?
What’s the best practice for implementing error handling in this type of architecture?
I want detailed tracking at the table level, but also proper bubbling up of errors to reflect them in the HEAD audit row.
Any recommendations or patterns you follow in similar setups would be very helpful.
1) Use a tLogCatcher in every job -> It's always good to be able to log anything that could happen during your execution. You can choose for each job you're using the component to either load the logs into a file, or into a SQL Table for example. In your actual way of functionning you could log everything to a table and update the rows needed with logs collected from tDie and tWarn through the tLogCatcher.
2) Use the tDie for the STAGE subjob or other subjobs. Only use it if you decide an error is critical and should "notify" the MAIN job. That way you can manage errors encountered during STAGE at the MAIN level.
3) If you want to track correctly you can also use tWarn components if an error is not critical, this way you avoid stopping the STAGE job with a tDie and you can continue the execution of STAGE.
1) Use a tLogCatcher in every job -> It's always good to be able to log anything that could happen during your execution. You can choose for each job you're using the component to either load the logs into a file, or into a SQL Table for example. In your actual way of functionning you could log everything to a table and update the rows needed with logs collected from tDie and tWarn through the tLogCatcher.
2) Use the tDie for the STAGE subjob or other subjobs. Only use it if you decide an error is critical and should "notify" the MAIN job. That way you can manage errors encountered during STAGE at the MAIN level.
3) If you want to track correctly you can also use tWarn components if an error is not critical, this way you avoid stopping the STAGE job with a tDie and you can continue the execution of STAGE.
