topic Re: tStatsCatcher - Working with subjobs in Talend Studio

tStatsCatcher - Working with subjobs

Anonymous — Sat, 16 Nov 2024 03:01:21 GMT

Hello,

I'm facing an unexpected behavior from the tStatsCatcher while using it in subjob components. To be updated with the context, I have 3 jobs (1 to read data from the file, 1 to receive data to insert/get records from dimension tables and the last one to receive data and insert in the fact table). My database is in a star schema architecture.

Follow below my execution flow:

What I'm doing is using the tStatsCatcher in the job that call the others two, but I checked that this doesn't catch the stats from the subjob flows, only the component subjob stats. So, as a workaround, I inserted a tStatsCatcher in the two subjobs, to catch the stats inside each one. However, the log information will get all the interactions and will slow down my job performance dramatically.

Do anyone knows how can I get the stats of my subjobs in my parent job?

Thank you!

Re: tStatsCatcher - Working with subjobs

ricksherman — Tue, 10 Mar 2020 23:04:50 GMT

You can turn on stats at the project level or for individual components in a job with a checkbox. You can also turn on log4j

What we have done is create a joblet that that contains tStatsCatcher and tLogCatcher with the actions we want to get triggered. We then have as a standard using that joblet in all jobs (parent & children). Using the joblet and customizing the actions triggered is a great way to manage the job & workflow along with reducing the verbose interaction that the default stats runs.

Re: tStatsCatcher - Working with subjobs

Anonymous — Wed, 11 Mar 2020 13:08:54 GMT

Thank you @prof_sherman,

I'm sorry, but I don't specified the tools that I'm using. Follow below:

TOS 7.3

With this tool, I can't create joblets (unfortunately). I've created exactly a tStatsCatcher and tLogCatcher and inserted the same execution flow in all job (parents and children). See my screenshot below.

The problem is like my first post, I'm reading a file that contains at least 10000 records; I'm passing the data of each record to my subjobs, so if a had 10000 records, I will call my two subjobs 10000 times for each one. So with this, the tStatsCatcher will get this number of interations, wich will slow down dramatically my job performance. Exist a way that I can listen to the child jobs execution equal as the parent job (only one time per component)?

Luiz Ramos