topic Re: How to create a log file for a job running on a Linux server via python job in Talend Studio

How to create a log file for a job running on a Linux server via python job

Anonymous — Sat, 16 Nov 2024 02:40:10 GMT

I am using Open source Talend Open Studio for Data Integration, Version: 6.3.1.

I have a set of jobs which I am exporting and keeping in a Linux server. Post that I am triggering these jobs, using my Python script.

I do not have Talend installed in the server. I want to log the information of the job, step by step, in a flat file.

Is there a way by which I can collect all the logs of my job (the records inserted or the issues encountered)

The flow of the job is:

PostgresInput --> TMAP --> PostgresOptput.

Please help.

Re: How to create a log file for a job running on a Linux server via python job

billimmer — Thu, 16 Apr 2020 16:44:31 GMT

In your Talend job, on the "Job" tab of your (tabs are Job, context, component, run)

Then click the "Stats & Logs" and setup your logging as needed.

Re: How to create a log file for a job running on a Linux server via python job

Anonymous — Thu, 16 Apr 2020 20:55:15 GMT

@lli Thanks for your reply. I tried the options. I am only able to generate the stats log but the log file is not getting generated. I am pretty new to Talend so need some guidance. Thanks in advance for your help.

Attaching the screenshot of the job. Do I have to link it to output or do some other setting changes?

Log Catcher in Talend.png

Re: How to create a log file for a job running on a Linux server via python job

billimmer — Thu, 16 Apr 2020 21:19:55 GMT

You send information to the log file using the twarn and tdie components.

Re: How to create a log file for a job running on a Linux server via python job

Anonymous — Fri, 17 Apr 2020 13:35:26 GMT

Hi, You have to make a little change in your job and you can easily log your job status in a file.

Changed flow:

twarn-- (onSubJobOk)--> PostgresInput --> twarn(row) --> TMAP --> twarn(row) -->PostgresOptput --(onSubJobOk)--> twarn

tlogCatcher --> tFileOutputDelimeted

1. In tWarn, you can write your log message.

2. tlogCatcher will append it into the filename specified in tFileOutputDelimeted

Re: How to create a log file for a job running on a Linux server via python job

Anonymous — Sun, 19 Apr 2020 03:35:42 GMT

@rockujju Thanks for the reply. I tried your option but the type of log getting generated is of no use. I wanted the record level information the success records and if any failed or rejected records.

Is there a way to do that.

Log File Data example:

2020-04-18 16:59:24;UkjfS2;UkjfS2;UkjfS2;OSQAR;logfile_Copy;Default;4;tWarn;tWarn_1;this is a warning;42

Stats file data:

2020-04-18 16:59:24;UkjfS2;UkjfS2;UkjfS2;136184;OSQAR;logfile_Copy;_hogLIIGeEeqK8-2cXJU_4w;0.1;Default;;begin;;

Re: How to create a log file for a job running on a Linux server via python job

Anonymous — Sun, 19 Apr 2020 03:41:09 GMT

@lli can you please help in sending the steps on how to do it. I tried below flow:

Twarn ->Input->TWARN->TMAP->TWARN->TARGET

Logacatcher->DelimitedFile

My log file data looks like below:

2020-04-18 16:59:24;UkjfS2;UkjfS2;UkjfS2;OSQAR;logfile_Copy;Default;4;tWarn;tWarn_1;this is a warning;42

I have also set the STATS and LOG at the job level to collect the data in file, but the data I am getting is of no use. I am looking for actual record level log information, number of records successfully loaded and if failed then the reason.

If possible can you please help in setting the actual flow and settings please. I really appreciate you help.

Re: How to create a log file for a job running on a Linux server via python job

billimmer — Mon, 20 Apr 2020 15:06:57 GMT

I agree that tStatCatcher really only gives you duration. See my reply to this here:

https://community.talend.com/t5/Design-and-Development/How-to-Log-everything-my-job-does/m-p/57293#M20059