Add logging & error handling to existing job (with child jobs)

[Anonymous]

I created a (main) job that calls about 15 child jobs (using tRunJob). At the moment there is no logging. Now, I'm looking for an approach to add comprehensive logging for the existing jobs and the child jobs. I would like to write all the information to files.
Is it necessary to active logging per component? Per child job? Or is it sufficient to activate "Use statistcs", "Use logs", "Use volumetrics" in the project settings? Choosing the project settings variant is everything done automatically then or what has to be considered? How can I format the output to the log files using the third variant?
What is the difference between e.g. "Use statistics (tStatCatcher)" in project settings and tStatCatcher Statistics used for examle in component tFTPFileList. I would expect the second configuration option to disappear if project settings are set if it means the same.
Do the project settings replace the strategy using tAssertCatcher described here: http://bekwam.blogspot.de/2011/04/three-error-handling-strategies-in.html or in which case would I combine project settings and logging components in the GUI of the jobs?

[Anonymous]

Hi,
The purpose of tLogCatcher(log) is operating as a log function triggered by one of the three: Java exception, tDie or tWarn, to collect and transfer log data.(prerequisite: there must be an error in your job).
Stat&log in project setting is working on the whole project (Parent job and child jobs, and output them on file/DB but no context setting).
Stat&log in job view is working on the current job (you should set Stat&log of job view in both parent job and child jobs and output them on file/DB with context setting)
The option "tStatCatcher Statistic" of component don't support for outputting Stat&log into file/DB just print out Stat&log data on console.
In addition, there are also tStatCatcher, tLogCatcher and tAssertCatcher log components(you can do mapping or other formatting, transforming operation, such as tStatCatcher-->tMap-->tlogrow...). however, if you have a great need of log, statistics and other measurement of your data flows, you are facing the issue of having too many log-related components loading your Job Designs.
Hope it will be helpful for you.
Best regards
Sabrina

[willm1]

ohofrichter - this is a common strategic question for Talend. While it's quite easy to log information in jobs, the question usually, how to best achieve this option?
The tStatCatcher, tLogCatcher and tFlowmeterCatcher all have predefined schemas. In your job settings, you can chose to write these logs to console, file or database - or all 3 at the same time. The stats they gather are great because they give you an end-to-end view of your job run. In there, you'll see when each child job started and ended etc... without having to do any extra work. Just check all your components to spit out logs.
Now, I have the habit of writing log statements using tJava all over my job to tell me where I am. This is more of the flow the job. These usually print on the console. When the job is executed from a shell script of batch file, I can redirect standard out to a file by using >> myTalendlog.txt.
Alternatively, for the entire job run, you can chose to redirect the standard out to a file in Talend using the following. At the beginning of your job, place a tJava, and use the following code:
????????-
java.io.File file = new java.io.File(?C:/temp/myTalendLog.txt?);
java.io.PrintStream ps = new java.io.PrintStream(new java.io.FileOutputStream(file));
System.setOut(ps);
????????-
Doing this will redirect all console output to file, including all your custom logging using tJava.
If these are not sufficient (using standard Talend logging to tStatCatcher, tLogCatcher, tFlowmeterCatcher + StdOut in a file), you can also consider log4j to achieve even more granular control of how you generate your log info. Whereas you can redirect standard out to a file, with log4j you can do even more things like inject standard identifiers into the log lines to be able to segregate and find information quicker. You can find an implementation of log4j for Talend as a component on Exchange - http://www.talendforge.org/exchange/index.php?eid=214&product=tos&action=view&nav=1,1,1

[Anonymous]

Hi ohofrichter,
Is there any update for your issue?
Best regards
Sabrina

[Anonymous]

Thank you, the different variants are now clearer to me.

[aanchal_s]

Hi,

If I only choose these setting from Project settings and not add these components in my job, then also it will capture all logs for Project and Jobs,right?

Also the log files generated don't have header,can you please suggest how can I get that.

Regards

Aanchal