topic Re: Catch processed row's count in Talend Studio

Catch processed row's count

Anonymous — Thu, 31 Aug 2017 14:59:30 GMT

Hi there,

I'm using TDI open studio. I have designed and run the job, whose screenshot is :

As you can see, there we are getting count of processed rows in green colour. I want to catch these count in separate file (prefer excel file).

So for above job, I expect below output :

After researching for a while, I come to know that there are 3 components that I feel one of these 3 can fulfill the requirement.

tStatCatcher

tLogCatcher

tFlowMeterCatcher

But I don't know which one to use and how to use it for my requirement. Please, help me out guys.

Thank you

Re: Catch processed row's count

Anonymous — Fri, 01 Sep 2017 04:25:34 GMT

Hello,

The tFlowMeterCatcher component catches the processing volumetric from the tFlowMeter component and passes them onto the output component.

For example:
tFileInput--->tFlowMeter--->tMysqlOutput_1
tFlowMeterCatcher---tLogRow
There is one column called "count" in the schema of tFlowMeterCatcher which counts the no. of records pass by the specify flow.
Let me know if it is OK with you.

Best regards

Sabrina

Re: Catch processed row's count

Anonymous — Fri, 01 Sep 2017 07:03:08 GMT

Hi @xdshi

Thank you so much. Your solution worked !

I tried 2 options :

1. Suggested by you : using tFlowMeterCatcher.- easy to record statistics in csv or excel file as it has output row.

2. Without using tFlowMeterCatcher - tried stats&log under job setting tab, where we check the box in front of 'Use volumetrics' and specify the file path and name. When we use this, we don't need to use tFlowMeterCatcher component. But its giving output file with values separated by ' ; ' (semicolon).

How can we get the the output that can be view properly as csv or excel using 2nd option ?

(might be comma separated, or using encoding, etc.)

Thanks

Re: Catch processed row's count

Anonymous — Fri, 01 Sep 2017 07:07:42 GMT

Hello,

We see that you have saved your 'Meter file name' as meter_file.csv

Could you please show us your actual output result with values separated by ' ; ' (semicolon)?

Please mask your sensitive data.

Best regards

Sabrina

Re: Catch processed row's count

Anonymous — Fri, 01 Sep 2017 07:33:41 GMT

Please find below the screenshots of the output result :

opened in ms excel :

opened in notepad :

Re: Catch processed row's count

Anonymous — Fri, 01 Sep 2017 07:57:58 GMT

Hello,

It should be field separator to separate fields for the transferred data when set the flow meter log to output the count record into a file.

Best regards

Sabrina

Re: Catch processed row's count

Anonymous — Fri, 01 Sep 2017 08:40:01 GMT

Where is the option to set field separator ?
I can see only encoding option there in job setting-> stats&log -> use volumetrics -> on files.
is there anything by which we can get comma instead of semicolon ?

Re: Catch processed row's count

Anonymous — Fri, 01 Sep 2017 09:33:37 GMT

Hello,

We will make an investigation on it and then come back to you as soon as we can.

Best regards

Sabrina

Re: Catch processed row's count

Anonymous — Sat, 02 Sep 2017 07:08:41 GMT

Hi,

In tFlowMeterCatcher, we are not getting the duration of processing.
I come to know that there is component called tChronometerStop to get a duration. But its logging the time on console. How can I get this time in file ? Or is there workaround for the same ?

Re: Catch processed row's count

Anonymous — Mon, 04 Sep 2017 04:35:37 GMT

@AkshayNG wrote:
Hi,

In tFlowMeterCatcher, we are not getting the duration of processing.
I come to know that there is component called tFlowMeterCatcher to get a duration. But its logging the time on console. How can I get this time in file ? Or is there workaround for the same ?

If I understand your issue clearly, do you want to use component tStatCatcher in your work flow and log the output into file? If so, you can design your work flow like: tStatCatcher-->tFileOutPutxxx(your target output file).

Best regards

Sabrina

Re: Catch processed row's count

Anonymous — Mon, 04 Sep 2017 07:48:14 GMT

Thanks @xdshi

I've tried tStatCatcher, and got the duration as well. But then I've to apply some lookup logic using tMap to get result of both in one output table containg row count (from tFlowMeterCatcher) and duration(using tStatCatcher).

I was basically looking for a way to add one more column for TIME in result of tFlowMeterCatcher. This column must contain values as duration of the specified component.

Re: Catch processed row's count

Anonymous — Tue, 05 Sep 2017 03:47:14 GMT

Hello,

So far, you cannot edit the schema setting of tFlowMeterCatcher component.

Best regards

Sabrina

Re: Catch processed row's count

hangel — Wed, 11 Dec 2019 06:47:30 GMT

Hello xdshi,

Good Day

I want to achieve similar outcome from talend.

I'm extracting data from Salesforce component and injecting the data in to influxdb and building dashboard in Grafana. I have a use case to get the count of status wrt location.

Eg:
Location New Pending Approved Total
CA 1 3 5 9
HE 4 6 8 18

Let me know which component to use in talend to achieve this case.