topic Re: Capture job metrics (processed counts) for multiple sub processes in Talend Studio

Capture job metrics (processed counts) for multiple sub processes

sushantV — Thu, 20 Sep 2018 20:48:53 GMT

I've a complex job and I intend to capture few critical counts in that job. Essentially the job is reading a raw file, processing the data and pushing it into a MySQL dataset. The key counts are - number of rows read from the raw file, number of rejected/processed raw records, number of rows pushed into the database and number of rows processed/rejected during the database entry.

Here is how the relevant section of the job currently looks like:

I tried to utilize the tFlowMeter and tFlowCapture components however I could only fetch one value out of it. Do I repeat it four times (four tFlowMeter components??) or is there a more graceful way of doing it?

Re: Capture job metrics (processed counts) for multiple sub processes

Anonymous — Thu, 20 Sep 2018 21:02:29 GMT

This is actually quite easy and Talend provide everything you need for this. Look at the bottom left of your Studio. You will see a window entitled "Outline". Inside there you have all of your components that you are using. You can expand these components to see globalMap variables related to the components (counts and errors, etc). To use these (this is just an example that you should be able to extrapolate from), drop a tJava into your job at the end. Now go to the "Outline" window and drag a variable to the tJava component's Java window. You will see it formats the Java code for you. Now you can put a System.out.println() around these to print these at the end of the job. These variables can be used anywhere in the job, the only thing you will need to consider is timing. You cannot use them before the component has fired and most cannot be used until the component has finished.

Re: Capture job metrics (processed counts) for multiple sub processes

sushantV — Thu, 20 Sep 2018 21:28:50 GMT

I am not sure I am following this entirely. e.g. for my file read component, I can see the component variables in the outline section. The two variables (ERROR_MESSAGE, NB_LINE) that I see don't seem to fulfill my requirement.

And I am fine with waiting till the job concludes. I need those counts for the job summary email. @rhall

Re: Capture job metrics (processed counts) for multiple sub processes

Anonymous — Thu, 20 Sep 2018 22:43:00 GMT

I'm assuming you wanted the number of rows from that component. That would be the NB_LINE variable. I can't tell you precisely which ones to use, but you can experiment easily with the method I gave you.

Re: Capture job metrics (processed counts) for multiple sub processes

sushantV — Fri, 21 Sep 2018 17:47:39 GMT

Super. It worked for me. Had to do quick search and what I needed was all there. Thank you.