How to create a log file for a job running on a Linux server via python job

[Anonymous]

I am using Open source Talend Open Studio for Data Integration, Version: 6.3.1. 

I have a set of jobs which I am exporting and keeping in a Linux server. Post that I am triggering these jobs, using my Python script.

I do not have Talend installed in the server. I want to log the information of the job, step by step, in a flat file.

Is there a way by which I can collect all the logs of my job (the records inserted or the issues encountered)

The flow of the job is:

PostgresInput --> TMAP --> PostgresOptput.

Please help.

[billimmer]

In your Talend job, on the "Job" tab of your (tabs are Job, context, component, run)

Then click the "Stats & Logs" and setup your logging as needed.

[Anonymous]

@lli Thanks for your reply. I tried the options. I am only able to generate the stats log but the log file is not getting generated. I am pretty new to Talend so need some guidance. Thanks in advance for your help.

Attaching the screenshot of the job. Do I have to link it to output or do some other setting changes?

Log Catcher in Talend.png

[billimmer]

You send information to the log file using the twarn and tdie components.  

[Anonymous]

Hi, You have to make a little change in your job and you can easily log your job status in a file.

Changed flow:

twarn-- (onSubJobOk)--> PostgresInput --> twarn(row) --> TMAP --> twarn(row) -->PostgresOptput --(onSubJobOk)--> twarn

tlogCatcher --> tFileOutputDelimeted

1. In tWarn, you can write your log message.

2. tlogCatcher will append it into the filename specified in tFileOutputDelimeted

[Anonymous]

@rockujju Thanks for the reply. I tried your option but the type of log getting generated is of no use. I wanted the record level information the success records and if any failed or rejected records.

Is there a way to do that.

Log File Data example:

2020-04-18 16:59:24;UkjfS2;UkjfS2;UkjfS2;OSQAR;logfile_Copy;Default;4;tWarn;tWarn_1;this is a warning;42

Stats file data:

2020-04-18 16:59:24;UkjfS2;UkjfS2;UkjfS2;136184;OSQAR;logfile_Copy;_hogLIIGeEeqK8-2cXJU_4w;0.1;Default;;begin;;

[Anonymous]

@lli can you please help in sending the steps on how to do it. I tried below flow:

Twarn ->Input->TWARN->TMAP->TWARN->TARGET

Logacatcher->DelimitedFile

My log file data looks like below:

2020-04-18 16:59:24;UkjfS2;UkjfS2;UkjfS2;OSQAR;logfile_Copy;Default;4;tWarn;tWarn_1;this is a warning;42

I have also set the STATS and LOG at the job level to collect the data in file, but the data I am getting is of no use. I am looking for actual record level log information, number of records successfully loaded and if failed then the reason.

If possible can you please help in setting the actual flow and settings please. I really appreciate you help.

[billimmer]

I agree that tStatCatcher really only gives you duration.  See my reply to this here:

https://community.talend.com/t5/Design-and-Development/How-to-Log-everything-my-job-does/m-p/57293#M...