TFlowMeter did not collect the data to database

[WSyahirah21]

I've created a simple pipeline to run a spark job using Tsystem component. The pipeline is:

tSystem (spark job) -> tJava (to exit the pipeline)

Now, I need to consolidate the collected activity monitoring information from that job in Talend using AMC. So far I've set up the AMC in my TOS and TAC. However, as I checked the amc database, I able to collect the stats and logs , however seems like the flowmeter is empty (as if it is not being collected to database). I also able to see the analysis in AMC except for threshold chart as it fetch the data from tflowmeter.

I've tried 2 ways to collect the data: -

1. Configure Stats and Logs tab in talend job by connecting created stats, logs and tflowmeter tables in amc database to job.
2. Add tFlowMeterCatcher -> tDbOutput(MySQL) components in the job pipeline.

My question:

1. Did I missed any config for tFlowMeter? I thought , just like stats and logs that collect the data directly from the amc database connection (in Stats and Logs tab) , tflowMeter also acts that way.
2. Read some example that it use the tFlowMeter by includes it in the pipeline. By when I tried to do:

tSystem -> tFlowMeter component . It couldnt connect both components.

What should I do to ensure my job able to collect the tflowmeter? Please assist. Thanks.

[Anonymous]

Hi

tFlowMeterCatcher is used to catches the processing volumetric of data flow, you should click the row which you want to monitor and open the Advanced settings panel, followed by checking the 'Monitor this connection' box (this option is equal to tFlowMeter component). In your job, there is no a data flow.

 

Regards

Shong

[WSyahirah21]

Hi Shong. question. As my job is just basically run a spark job using tSystem and exit the job, so I yea I think this pipeline doesnt pass any data flows. Thus, tSystem just do Trigger connection to next component instead of row. Do you have any idea what should I add in the pipeline to enable the monitored connection in workflow?

Thanks.

[Anonymous]

In your job, only has trigger connector like oncomponentOK or onSubjobOK, no row type that can be captured by tFlowMeterCatcher, so it is normal that the flow table is empty.