topic Re: unique count of rows in a file in Talend Studio

unique count of rows in a file

Aish123 — Thu, 23 Apr 2020 17:08:09 GMT

i have a text file which has 2 key elements, depending upon on the key elements i have to fetch the unique row count and store it in a file.

Please let me know how this can be acheived.

Thank you.

Re: unique count of rows in a file

billimmer — Thu, 23 Apr 2020 17:39:13 GMT

use tUniqRow to get the count.

Re: unique count of rows in a file

Aish123 — Fri, 24 Apr 2020 07:35:23 GMT

Hi,

Thanks for the reply.

I tried with the tuniqrow component.

Tfileinputpostional-->tuniqrow-->tlogrow

In this, i am getting the unique rows data on the console..i need the unique count and want to store the same in an file.

I also tried with tfileinputpositional->tuniqrow->tflowmeter->tflowmetercatcher-->tfileoutputdelimited.

In this, there are 0 rows processed at the tfileoutputdelimited and there is no data present in it.

Re: unique count of rows in a file

Anonymous — Fri, 24 Apr 2020 07:52:41 GMT

You don't use tFlowMeterCatcher component correctly.
There is a pre-defined global variable on tFileOutputDelimited that counts the total number of rows are output into file, you can use this variable on other subjob.
((Integer)globalMap.get("tFileOutputDelimited_1_NB_LINE"))
eg:
tfileinputpositional->tuniqrow-->tfileoutputdelimited.
|onsubjobok
tJava

on tJava:
System.out.println("the number of unique rows is "+(Integer)globalMap.get("tFileOutputDelimited_1_NB_LINE"))

Regards
Shong

Re: unique count of rows in a file

Aish123 — Fri, 24 Apr 2020 08:35:39 GMT

Hi Shong,

I tried the approach suggested by you, but it is throwing an error.

Please find the attached screen print for the same.

Thank you.

error.PNG
job1.PNG

Re: unique count of rows in a file

Anonymous — Fri, 24 Apr 2020 08:39:33 GMT

It is a compilation error, maybe you need to synchronize the schema after you modify the job, check the schema and basic settings of tFileOutputDelimited.

BTW, please insert images into your post body instead of attaching them!

Re: unique count of rows in a file

Aish123 — Fri, 24 Apr 2020 09:25:02 GMT

Hi Shong,

I am getting an unique data as an output in the delimited file instead i want to store the count of unique rows in the file.

I want to store the count "2" as seen on the console in the file.

Re: unique count of rows in a file

Anonymous — Fri, 24 Apr 2020 09:35:10 GMT

OK, just use a tJavaFlex instead of tFileOutputDelilimited, define a counter on tJavaFlex, eg:
begin part:
int counter=0;
main part:
counter++;
end part:
globalMap.put("counter",counter);

In next subjob, get the value of this global variable and output it to a file.
|onsubjobok
tFixedFlowInput--main--tFileOutputDelimited

on tFixedFlowInput, add one column with int type, and set its value as:
(Integer)globalMap.get("counter")

Re: unique count of rows in a file

billimmer — Fri, 24 Apr 2020 14:17:00 GMT

You don't need all that. The tUniqRow already returns:

NB_UNIQUES: the number of unique rows. This is an After variable and it returns an integer.

NB_DUPLICATES: the number of duplicate rows. This is an After variable and it returns an integer.

You can write out the value of NB_UNIQUES to a file or DB table with tFixedFlowInput.

Re: unique count of rows in a file

Aish123 — Fri, 24 Apr 2020 14:21:05 GMT

Hi Shong,

Thanks it worked successfully.