topic Re: How to Log everything my job does? in Talend Studio

How to Log everything my job does?

Anonymous — Tue, 16 Sep 2014 13:33:16 GMT

Hi,
I want to log everything my talend job does.
When running my job within talend I can see on the gui where the flow currently is, what its doing and it shows whether a flow was true, false, ok or error.
When deploying the executables of my job to a server and running it automated there I do not geht this information any more.
Of course I am able to do logs at several points but I dont want to create a log componenten after each single working-component, thats too much overhead.
Of course I could catch errors, but in my case the problem is, the process does not end its only reproducable on deployment not local with my open studio.
Thats why I want to catch as much information about all components and subjobs as possible.
We often get the case to have a look at any unknown behavior and it were much more comfortable to have A big log to look after.
Is there a way to get a predefined all logging-log like I am looking for?
Any suggestions else for easily creating something like this? (it mus be then a generic solution, not like creat a log componentent after every single working-component).
Thanks in advance.

Re: How to Log everything my job does?

Anonymous — Tue, 16 Sep 2014 14:58:19 GMT

In every component Advanced setting select tStatCatcher Statistic option and use tStatCatcher component at the last
Please look attached screenshot for reference

Re: How to Log everything my job does?

Anonymous — Wed, 17 Sep 2014 09:18:39 GMT

Hi,
Talend provides three types of logs to be catched using different components
- Statistics
- Error/warning
- Meter
You can enable these logging at project level or at job level. All the logs could be redirected to the file or database.
Even after doing these logging, you can't catch complete trace information available on console. In order to catch this information, you will have to use different technique to redirect consol trace information to flat file or database.
Vaibhav

Re: How to Log everything my job does?

Anonymous — Wed, 17 Sep 2014 09:38:40 GMT

Hi,
And please refer to related documents: TalendHelpCentertats & Logs, TalendHelpCenter:Logs & Errors components.
Best regards
Sabrina

Re: How to Log everything my job does?

Anonymous — Wed, 17 Sep 2014 13:35:14 GMT

Hey,
unfortunately the talend documentation seem to expect several prio knowledge.
In your linked documentation it says: "Select the Use Statistics, Use Logs and Use Volumetrics check boxes where relevant, to select the type of log information you want to set the path for."
Well I do not know which checkbox now is relevant and what effect it then will have.
Because of this I simple tested it: I hit the Checkbox for statistics, but console log doesnt change. Even the checkbox to use component-statistics wont make any change, no additionally logs.
Only when hitting the checkbox for component-statistics at every single component (in combination with component-statistic checkbox at job-level) will creat logs which indicates start and end of the specific component.
Well so I dont know what non-components-statistic-checkbox on job-level does, but it were great if it would just do what hitting every single components-checkbox does.
I could imagine that the non-components-statistic-checkbox on job-level would just catch all statistics caught by tStatCatcherbut if it were like this I would expect such a hint in documention on TalendHelpCentertats & Logs.
I now will make the setting at every component but this is not comfortable ob many and large mappings. Is there may a way to batch-activate thos checkboxes for all components?
Thanks for replies.

Re: How to Log everything my job does?

Anonymous — Thu, 18 Sep 2014 09:15:01 GMT

Hi Kaspatoo,
You can use Job Setting Stats & Logs from job review. This option is working on current job.
See my screenshot.
Best regards
Sabrina

Re: How to Log everything my job does?

Anonymous — Thu, 18 Sep 2014 09:18:32 GMT

Hey, well thanks but ur last post delivers the same information as your post before.
So that my worries of my last post are still there.

Re: How to Log everything my job does?

Anonymous — Thu, 18 Sep 2014 09:28:54 GMT

Hi,
Projecting setting Stats & Logs is working on my side without checking out any component-statistics at every single component.
Could you please indicate the build version you are using?
Best regards
Sabrina

Re: How to Log everything my job does?

Anonymous — Sun, 19 Apr 2020 04:34:14 GMT

@xdshi: Is there a way to get detail level logs of a job into a file. I want to see the number of records processed or the record and reason why it failed. Can I capture console level logs into a txt file?
I am triggering my job in a Linux server by a script.

Re: How to Log everything my job does?

Anonymous — Sun, 19 Apr 2020 23:22:36 GMT

Hello,
Is there an solution? I am also searching for a solution to get my job status and description.
If there is any solution can you please share? Thanks

Re: How to Log everything my job does?

billimmer — Mon, 20 Apr 2020 15:02:30 GMT

For input components I use the NB_LINE variable and write it with tWarn. The when I load data into the DB, the Talend DB component will error if the records cannot be written, such as a tDBbulkExec or tDbOutput.

For the "tDbRow" components, I wrap my sql to return a count. Then I use a tParseRecordSet to get the value of "RecCnt" and write it into the log with tWarn. For example:

WITH rows AS (
DELETE FROM mydatabase.myschema.mytable
WHERE "YYYYMM" >= '202004'
RETURNING 1
) SELECT count(*) AS "RecCnt" FROM rows;

If there is an easier way to do this I'd also like to know. I don't think tStatCatcher writes record counts. I wish it did.