topic Re: TFlowMeter did not collect the data to database in Talend Studio

TFlowMeter did not collect the data to database

WSyahirah21 — Fri, 15 Nov 2024 23:14:14 GMT

I've created a simple pipeline to run a spark job using Tsystem component. The pipeline is:

tSystem (spark job) -> tJava (to exit the pipeline)

Now, I need to consolidate the collected activity monitoring information from that job in Talend using AMC. So far I've set up the AMC in my TOS and TAC. However, as I checked the amc database, I able to collect the stats and logs , however seems like the flowmeter is empty (as if it is not being collected to database). I also able to see the analysis in AMC except for threshold chart as it fetch the data from tflowmeter.

I've tried 2 ways to collect the data: -

Configure Stats and Logs tab in talend job by connecting created stats, logs and tflowmeter tables in amc database to job.
Add tFlowMeterCatcher -> tDbOutput(MySQL) components in the job pipeline.

My question:

Did I missed any config for tFlowMeter? I thought , just like stats and logs that collect the data directly from the amc database connection (in Stats and Logs tab) , tflowMeter also acts that way.
Read some example that it use the tFlowMeter by includes it in the pipeline. By when I tried to do:

tSystem -> tFlowMeter component . It couldnt connect both components.

What should I do to ensure my job able to collect the tflowmeter? Please assist. Thanks.

Re: TFlowMeter did not collect the data to database

Anonymous — Tue, 08 Feb 2022 06:12:26 GMT

tFlowMeterCatcher is used to catches the processing volumetric of data flow, you should click the row which you want to monitor and open the Advanced settings panel, followed by checking the 'Monitor this connection' box (this option is equal to tFlowMeter component). In your job, there is no a data flow.

Regards

Shong

Re: TFlowMeter did not collect the data to database

WSyahirah21 — Tue, 08 Feb 2022 06:59:39 GMT

Hi Shong. question. As my job is just basically run a spark job using tSystem and exit the job, so I yea I think this pipeline doesnt pass any data flows. Thus, tSystem just do Trigger connection to next component instead of row. Do you have any idea what should I add in the pipeline to enable the monitored connection in workflow?

Thanks.

Re: TFlowMeter did not collect the data to database

Anonymous — Tue, 08 Feb 2022 08:09:26 GMT

In your job, only has trigger connector like oncomponentOK or onSubjobOK, no row type that can be captured by tFlowMeterCatcher, so it is normal that the flow table is empty.