topic Re: How to log progress statistics in Talend Studio

How to log progress statistics

Anonymous — Thu, 13 Jun 2013 12:22:00 GMT

Scenario : extracting data form one database and inserting it into another.
I try to use flowMeter for progress log purpose but it just displays a message (which is also very ugly) at the end :
2013-06-13 13:39:49;8dkrU5;ElLlYx;ElLlYx;4520;LEARN;update_tables;_W_buENKxEeKRePtjf-zDjA;0.1;Default;tFlowMeter_3;MY_CUSTOM_NAME;1541211;;
The enabled statsCatcher gives me another ugly log about the time of execution :
2013-06-13 13:39:49;8dkrU5;ElLlYx;ElLlYx;4520;LEARN;update_tables;_W_buENKxEeKRePtjf-zDjA;0.1;Default;tMSSqlOutput_1;end;success;513827
513 is the number of seconds it has spent to finish.
1. How can I tweak flowMetter (or use something else) to see a real progress log :e.g. 100 k inserted, 200 k inserted ?
2. How can I tweak statsCatcher to display the 'MY_CUSTOM_NAME' (name you give to the component on the design workspace) instead of 'tMSSqlOutput_1'.
3. How can I get rid of all those useless identifiers (e.g. '_W_buENKxEeKRePtjf-zDjA')

Re: How to log progress statistics

Anonymous — Thu, 13 Jun 2013 22:48:44 GMT

You can use the AMC functionality or use the tFlowCatcher component in conjunction with a file output or database output component.
tFlowCatcher -> tMySqlOutput

Re: How to log progress statistics

Anonymous — Fri, 14 Jun 2013 04:10:43 GMT

1. How can I tweak flowMetter (or use something else) to see a real progress log :e.g. 100 k inserted, 200 k inserted ?
2. How can I tweak statsCatcher to display the 'MY_CUSTOM_NAME' (name you give to the component on the design workspace) instead of 'tMSSqlOutput_1'.
3. How can I get rid of all those useless identifiers (e.g. '_W_buENKxEeKRePtjf-zDjA')

The schema of tFlowMeterCatcherand tStatsCatcher are read only, you can link it to a tMap to remove the columns that you do not want and do some transform, before log them to DB or file.
Shong

Re: How to log progress statistics

Anonymous — Fri, 14 Jun 2013 14:03:08 GMT

what about 1 (log progress) and 2(statCatcher schema does not have a label field in his schema) ?

Re: How to log progress statistics

Anonymous — Fri, 14 Jun 2013 18:20:04 GMT

what about 1 (log progress) and 2(statCatcher schema does not have a label field in his schema) ?

In order to avoid duplication of labels in a job, the tStatCatcher and tLogCather always log the real name of the component, not the label. If you want the label of the component to be logged, link it to a tMap and add a new column, let's call it "label", and set its expression like this:
row1.origin.equals("tMysqlInput_1")?"label1" row1.origin.equals("tMysqlInput_2")?"label2":"label3")
Shong

Re: How to log progress statistics

Anonymous — Mon, 17 Jun 2013 07:50:42 GMT

Thank you.
What about progress log ?
After running a job it is critical to have some logs about how much data was migrated, e.g. :
100.000
200.000
300.000
etc.
By now it just displays only one entry at the end of some job-part (statsCatcher - the overall time, flowMeter - the overral record count)