topic Re: Creating detailed log file in Talend Studio

Creating detailed log file

Anonymous — Sat, 16 Nov 2024 05:02:19 GMT

Hi All,

I want to create a detailed log/stats file with details such as

Job Name, Source file name, Source file count, Target file name, Target insert count, Time taken for job, start date, end date, Job status, failed status etc and store it either in file or table.

Can anyone help me with the same pls.

Thanks!

Re: Creating detailed log file

Anonymous — Thu, 08 Aug 2019 15:02:04 GMT

Hi,

You can use the logging and error components of Talend.

https://help.talend.com/reader/KxVIhxtXBBFymmkkWJ~O4Q/exEndmX5MGHMto5BeyIMhg

Please also go through below link.

https://help.talend.com/reader/SAZUuM203k6SNl8zrDfj5Q/GRS3dWpaYoK_QdY~Mvck5Q

I believe you would also be interested in AMC.

https://talendpnp.github.io/architecture-center/reference-architecture/summer-2018-7.0/talend-components/talend-activity-monitoring-console

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Creating detailed log file

Anonymous — Fri, 09 Aug 2019 08:36:54 GMT

Appreciate your response @nthampi .

All these components give some columns required.

Is there a way we can capture all at once. Instead of adding so many components.

The reason being, i want to make it reusable across multiple jobs or all jobs.

Re: Creating detailed log file

Anonymous — Fri, 09 Aug 2019 14:56:54 GMT

Hi,

There is an interesting blog related to this subject.

https://www.talend.com/blog/2018/10/09/how-to-implement-a-job-metadata-framework-using-talend/

Why don't you add it as a Joblet so that it can be reused easily with other jobs?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Creating detailed log file

Anonymous — Sat, 10 Aug 2019 13:10:06 GMT

Thanks, @nthampi.

I was referring to the same link. However, I'm not sure how to access tables such as talend_file_tracker,talend_jobs, talend_hwm_tracker, etc.

Or if I have to populate these tables, not sure of the components.

Appreciate your help.

Re: Creating detailed log file

Anonymous — Sun, 11 Aug 2019 04:19:26 GMT

Hi,

You will have to setup the necessary tables based on the data extracted from Talend error and log components. The flowchart is depicting the method in which you will have to pass the data from the log components.

Unfortunately I do not have the code handy with me. I would suggest you to build the code after verifying the elements I have already mentioned in the previous posts. If you are facing any specific error, we can guide you during that point of time.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Creating detailed log file

Anonymous — Mon, 12 Aug 2019 14:44:47 GMT

Thanks @nthampi.

I did try using multiple components and putting it in one table.

However each component offers different feature and can't connect all component and even if i do i can't make it generic.

And i'm still not able to generate those tables, since i'm not sure how to generate some of the column values.

Could you please guide me.

Many thanks!