topic Re: [resolved] How to insert a counter to evaluate correct and incorrect rows in Talend Studio

[resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Mon, 01 Oct 2012 14:22:39 GMT

Hi !
here's the context :
- i have a txt file with more than 200k lines.
- it is a positional file, correctly entered in talend (74 columns)
- each line has a fixed string content, then a variable one (depends on entries before).
In addition, some lines are not properly entered, and we don't need to include these lines in our datamining project.
I want to count the number of correct line and incorrect line, so after the positionalFile, i put a tJava initiliazing the 2 counters (int correct and int incorrect) , then i put a tJavaRow where i increment the variables when the conditions are met, then i put a tJava to println the variables.
The problem is that the last tJava doesn't recognize the variables.
I tried to make them global, but it is not working.
I tried to insert it in the tJavaRow but, they print in each line read the values... it works but it is not clean..
If anyone have a good way to do so, it would help me a lot !
Thank you !

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Mon, 01 Oct 2012 17:23:02 GMT

hi
first try to use dedicated Talend component and not tjava or tjava row (if you can of course).
use for example tSchemaComplianceCheck to conforme your data structure.
you can also use filter row
you could also use the "reject flow" from those differents components.
Try the reject flow from your tfileInputPositionnal for ex.
And use values that Talend 's already stored for you (hava a look at "outline" - bottom left
hope it helps
regards
laurent

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Mon, 01 Oct 2012 17:27:49 GMT

Thank you, i will check this first time tomorrow and let you know.

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

alevy — Tue, 02 Oct 2012 09:48:25 GMT

And don't ever use tJava connected by rows as the code is executed before the rows even start.

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Tue, 02 Oct 2012 10:21:47 GMT

Hi,
If I understood this right, you want to count the number of lines rejected and the number of lines accepted(which pass schema check) right? If that is the case, your job should look like this:
You will need 2 context variables - one for the number of rows rejected and one for the number of lines accepted
tFileInputDelimited_1 --> tjava
onsubjobok
|
tFileOutputDelimited_2(rejects)
In the tjava component, you fill in the context variables with the values as fallows:
context.rejected_number_of_lines=((Integer)globalMap.get("tFileOutputDelimited_1_NB_LINE"));
context.acc_number_of_lines=((Integer)globalMap.get("tFileInputDelimited_1_NB_LINE"))-context.rejected_number_of_lines;
Those above are some system variables defined automatically for those 2 components(file input and fileoutput). You can find context variables, system variables defined for components, etc by pressing ctrl+space in editable fields.
Hope this helps.
Regards.

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Tue, 02 Oct 2012 10:44:37 GMT

Hello again,
today i 've added a tFilterRow after the inputFilePositionnal, then 2 tFileRowCount to count the filtered and rejected rows.
(schema attached images)
In the tFilterRow, i've added the following code to check if the lenght of the variable part is equal to it's supposed lenght by calculating the lenght of animals+cats+dogs part.

((input_row.variableContent.length())==
(40*Integer.parseInt(input_row.nb_animals)
+6*Integer.parseInt(input_row.nb_cats)
+22*Integer.parseInt(input_row.nb_dogs)))

But i have a failure in generating the code from Talend
I think the interpreter can't generate such complicated condition.

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Tue, 02 Oct 2012 10:51:10 GMT

Hi,
If I understood this right, you want to count the number of lines rejected and the number of lines accepted(which pass schema check) right?

it's not completely right, the lines can pass the schema check, but can be wrong depends on its real value.
You can see the condition in the above post.

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Tue, 02 Oct 2012 11:21:25 GMT

And don't ever use tJava connected by rows as the code is executed before the rows even start.

The first tJava was to initialize the counter (int correct / incorrect) , then the tJavaRow to increment, and the last tJava to println .
Well, it was very messy

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Tue, 02 Oct 2012 13:45:39 GMT

omg that was stupid to use the tRowCount TT sorry !!
i'm correcting it using what Neinfrantu said, brb !

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Tue, 02 Oct 2012 14:12:12 GMT

I don`t understand exactly what is the condition that must be satisfied by the rows. But I can think of a solution as fallows:
tjavrow -> main ->tfilterrow ->main -> outcomponent
|rej
outcomponent
edit schema of tjavarow and add a boolean type column and set the flag to 1 if condition/conditions are met and verify the condition:
If(condition)
out_row.flag=1;
else
out_row.flag=0;
(you can also use a tfiltercolumns if you want to get rid of this new column afterwards)
(also don`t forget to set all the other columns: out_row.col1=in_row.col1; etc)
Tfilterrow also has system defined variables which retain the number of rows which pass the filter and of those which didn`t.
(optional tfiltercolumns)
So the whole job could look like:
tFileInputDelimited_1 -->main-->tjavrow -> main ->tfilterrow ->main ->(tfiltercolumns)-> outcomponent
|rej |rej
outcomponent outcomp
|| onsubjobok
V
tjava (to set those context variables with the predefined values from components)

Regards,
Razvan

Re: [resolved] How to insert a counter to evaluate correct and incorrect rows

Anonymous — Tue, 02 Oct 2012 15:54:35 GMT

tjavrow -> main ->tfilterrow ->main -> outcomponent
|rej
outcomponent
edit schema of tjavarow and add a boolean type column and set the flag to 1 if condition/conditions are met and verify the condition:
If(condition)
out_row.flag=1;
else
out_row.flag=0;

exactly the right thing to do ! working perfectly and i find the same results as with my messy way
You can find a screenshot showing the work in attached.
thank you , you rock !