topic Re: How to capture each component processed time in Job in Talend Studio

How to capture each component processed time in Job

manodwhb — Mon, 18 Jun 2018 06:45:15 GMT

Hi,

could you please let me know how to capture each component processed time in Job?

Thanks,

Manohar

Re: How to capture each component processed time in Job

vapukov — Mon, 18 Jun 2018 07:48:59 GMT

You can check

Stats and Logs (tStatsCatcher)
and
How to automate the use of statistics & logs

Re: How to capture each component processed time in Job

manodwhb — Mon, 18 Jun 2018 09:41:02 GMT

Since i tried to get the component statistics but not able to get. are you able to get?

Re: How to capture each component processed time in Job

vapukov — Mon, 18 Jun 2018 11:10:57 GMT

may be we talking about different, but I can not understand - what You are looking for

.-------------------+------+--------+----------+--------+-------------+-------+----------+-------+----------+-----------------------+-----------+------------+--------.
|                                                                              tLogRow_1                                                                              |
|=------------------+------+--------+----------+--------+-------------+-------+----------+-------+----------+-----------------------+-----------+------------+-------=|
|moment             |pid   |root_pid|father_pid|project |job          |context|origin    |message|system_pid|job_repository_id      |job_version|message_type|duration|
|=------------------+------+--------+----------+--------+-------------+-------+----------+-------+----------+-----------------------+-----------+------------+-------=|
|2018-06-18 22:07:58|YjJ08T|YjJ08T  |YjJ08T    |BD_DEMOS|tStatCatcher1|demo   |tUniqRow_1|success|4614      |_CAQbYOdtEdu08_wAFUaozw|0.1        |end         |6       |
'-------------------+------+--------+----------+--------+-------------+-------+----------+-------+----------+-----------------------+-----------+------------+--------'

.-------------------+------+--------+----------+--------+-------------+-------+----------------------+-------+----------+-----------------------+-----------+------------+--------.
|                                                                                    tLogRow_1                                                                                    |
|=------------------+------+--------+----------+--------+-------------+-------+----------------------+-------+----------+-----------------------+-----------+------------+-------=|
|moment             |pid   |root_pid|father_pid|project |job          |context|origin                |message|system_pid|job_repository_id      |job_version|message_type|duration|
|=------------------+------+--------+----------+--------+-------------+-------+----------------------+-------+----------+-----------------------+-----------+------------+-------=|
|2018-06-18 22:07:58|YjJ08T|YjJ08T  |YjJ08T    |BD_DEMOS|tStatCatcher1|demo   |tFileOutputDelimited_1|success|4614      |_CAQbYOdtEdu08_wAFUaozw|0.1        |end         |9       |
'-------------------+------+--------+----------+--------+-------------+-------+----------------------+-------+----------+-----------------------+-----------+------------+--------'

this is just simple text example, but You can store stats in database and make analysis after

Re: How to capture each component processed time in Job

manodwhb — Mon, 18 Jun 2018 11:47:40 GMT

I am looking for something to capture each component taking time in a job.

Re: How to capture each component processed time in Job

vapukov — Mon, 18 Jun 2018 12:05:20 GMT

Stats & Logs
When you execute a Job, you can monitor the execution through the tStatCatcher Statistics option or through using a log component. This will enable you to store the collected log data in .csv files or in a database.

You can then set up the path to the log file and/or database once for good in the [Project Settings] dialog box so that the log data get always stored in this location.

Duration: Time for the execution of a Job or a component with the tStatCatcher Statistics check box selected.

this is cover - most of cases, if You have something more important - use tChronometerStart/Stop

Re: How to capture each component processed time in Job

Anonymous — Tue, 19 Jun 2018 08:20:13 GMT

Hello,

Are you looking for detailed logging of the components? The INFO messages shows the progress of the component? Those detailed logging you were getting with older versions where more of a DEBUG level, but were incorrectly logged with INFO level. So changes have been made to rectify this in newer releases.

Either use Debug level or consider using AMC if you want to track statistics on components.

Best regards

Sabrina

Re: How to capture each component processed time in Job

Anonymous — Fri, 22 Jun 2018 01:15:53 GMT

hello, i think this article should help you
-https://www.talend.com/blog/2016/10/05/talend-job-design-patterns-best-practices-part-3/
good day!

Re: How to capture each component processed time in Job

Anonymous — Fri, 22 Jun 2018 15:05:03 GMT

hello,

I am using tStatCatcher for capturing the log in a flat file.

best regards

Re: How to capture each component processed time in Job

Anonymous — Wed, 04 Jul 2018 14:06:52 GMT

I used tChronometerStart/Stop to measure the time taken by my job and it works perfectly fine but is it there a way I can store it in a flat file?
Thanks.

Re: How to capture each component processed time in Job

Anonymous — Thu, 05 Jul 2018 04:06:29 GMT

Hello,

Use tFixedFlowInput to create a row(s) containing your output and link it to tFileOutputDelimite.

((String)globalMap.get("tChronometerStop_1_DURATION"))
((String)globalMap.get("tChronometerStart_1_STARTTIME"))

Let us know if it is OK with you.

Best regards

Sabrina

Re: How to capture each component processed time in Job

Anonymous — Mon, 13 Aug 2018 11:17:42 GMT

i tried to get the component statistics but not able to get. are you able to get? Manohar B Natasha Jain

Re: How to capture each component processed time in Job

manodwhb — Thu, 31 Oct 2019 06:45:04 GMT

@sienna2506 ,

tStatCatcher - captures data sent from tStatCatcher Statistics check box on individual components.