Log number of rows processed at each stage into database table for further data processing reports

[Anonymous]

Dear Experts,

I am new to talend, i have a requirement were the ETL should capture the number of rows processed by each stage and insert into table for further data processing report.

i have done something but couldn't move forward. please advise.

Note: i know that there are log components we can use but it seems too complex and i dont see what i want.

following things i have done so far.

tJava_3 -> i am assigning the extraction count to global variable "Extraction"

globalMap.put("ExtractionCount", ((Integer)globalMap.get("tFileRowCount_1_COUNT")));

tJava_1 -> 

globalMap.put("AggregatedCount", ((Integer)globalMap.get("tJavaRow_2_NB_LINE")));

tJava_2 ->

globalMap.put("Inserted", ((Integer)globalMap.get("tDBOutput_1_NB_LINE_INSERTED")));

[Anonymous · Accepted solution]

An easier way of doing this is to do it after the first subjob has finished. Simply connect a tJava to your tFileInputDelimited using an OnSubJobOK link. Then use the same code you have used in each of your tJava components in the new tJava. One change, you can get the tFileInputDelimited component's row count using this code....

((Integer)globalMap.get("tFileInputDelimited_1_NB_LINE"))

[Anonymous]

These global variables mentioned in your post have been already defined by component and you can use them directly in other subjobs, don't need to define new variables. eg:
tFileInputDelimited......tDBOutput
|onsubjobok
tJava1

on tJava1:
System.out.println("ExtractionCount",(Integer)globalMap.get("tFileInputDelimited_1_NB_LINE"));


Regards
Shong

[Anonymous]

thank you rhall

[Anonymous]

thank you Shong