How to configure observability metrics with Talend Cloud and Remote Engine

[TalendSolutionExpert]

You can observe your Data Integration Jobs running on Talend Remote Engines if your Jobs are scheduled to run on Talend Remote Engine version 2.9.2 or later.

This is a step-by-step guide on how Talend Cloud Management Console can provide the data needed to build your own customized dashboards, with an example of how to ingest and consume data from Microsoft Azure Monitor.

Once you have set up the metric and log collection system in Talend Remote Engine and your Application Performance Monitoring (APM) tool, you can design and organize your dashboards thanks to the information sent from Talend Cloud Management Console to APM through the engine.

Content:

• Prerequisites
• Configuring and starting the remote engine
• Get the metric details from the API
• Push metric to Azure logs workspace
• Observable metrics
• Queries and dashboard example
• Report One: A sample report for the number of rows processed by a component within the overall run
• Report Two: A sample report to showcase the average time taken by each component grouped by Job
• Report Three: A sample report to showcase OS memory and file storage available in MB
• Report Four: A sample report to showcase all jvm_process_cpu_load events counted in the last two day...
• Azure dashboard
• Pin the reports to the Azure dashboard
• Sample Talend Data Integration Job
• Job structure
• Job details

Prerequisites

This document has been tested on the following products and versions running in a Talend Cloud environment:

• Remote Engine 2.9.2 #182 – downloaded and installed following the instructions in Installing Talend Remote Engine in the Talend Help Center.

Optional requirements for obtaining detailed Job statistics:

• Studio 7.3.1 R2020-07 – downloaded from the Talend Cloud Portal and updated with the appropriate monthly patch
• Republish your Jobs from the new version of Studio to Talend Cloud Management Console

 

Configuring and starting the remote engine

To configure the files and check that the Remote Engine is running, navigate to the Monitoring Job runs on Remote Engines section of the Talend Remote Engine User Guide for Linux. 

Get the metric details from the API

Use any REST client, such as Talend API Tester or Postman, and use the endpoint as explained below.

1. Endpoint:

   GET http://ip_where_RE_is_installed:8043/metrics/json

   8043 is the default http port of Remote Engines. Replace it with the port you used when installing the Remote Engine.
2. Add a header: Authorization Bearer {token}.
   This token is defined in the etc/org.talend.observability.http.security.cfg file as endpointToken={token}.
3. Example:

   GET http://localhost:8043/metrics/json
           Authorization Bearer F7VvcRAC6T7aArU

   
   

Push metric to Azure logs workspace

There are numerous ways to push the metric results to any analytics and visualization tool. This document shows how to use the Azure monitor HTTP data collector API to push the metrics to an Azure log workspace. Python code is also used to send the logs in batch mode at frequent intervals. Alternatively, you can create a Talend Job as a service for real-time metric extraction. For more information, see the attached Job and Python Code.zip file.

The logs are pushed to the Azure Log Analytics workspace as “custom logs”.

Observable metrics

Talend Cloud Management Console provides metrics through Talend Remote Engine. They can be integrated in your APM tool to observe your Jobs.  

For the list of available metrics, see Available metrics for monitoring in the Talend Remote Engine User Guide for Linux. 

Queries and dashboard example

Report One: A sample report for the number of rows processed by a component within the overall run

Query:

Remote_Engine_OBS_CL 
|where TimeGenerated > ago(2d)
|where name_s=='component_connection_rows_total' 
|summarize sum(value_d) by context_target_connector_type_s
|render piechart

Chart:

Report Two: A sample report to showcase the average time taken by each component grouped by Job

Query:

Remote_Engine_OBS_CL 
|where TimeGenerated > ago(2d)
|where name_s=='component_execution_duration_seconds'
|summarize count(), avg(value_d) by context_artifact_name_s,context_connector_label_s

Chart:

 

Report Three: A sample report to showcase OS memory and file storage available in MB

Query:

Remote_Engine_OBS_CL 
|where name_s=='os_memory_bytes_available' or name_s =='os_filestore_bytes_available'
|summarize sum(value_d)/1000000 by name_s

Chart:

Report Four: A sample report to showcase all jvm_process_cpu_load events counted in the last two days, in 15-minute intervals

Query: 

Remote_Engine_OBS_CL 
|where TimeGenerated > ago(2d)
|where name_s =='jvm_process_cpu_load'
|summarize events_count=sum(value_d) by bin(TimeGenerated, 15m), context_artifact_name_s
|render timechart

 Chart:

Azure dashboard

Pin the reports to the Azure dashboard

Sample Talend Data Integration Job

This section explains the sample Job used to send the metric logs to the Azure log workspace. This Job is available in the attached Job and Python Code.zip file.

Job structure

Job details

The components used and their detailed configurations are explained below.

tREST

Component to make a REST API Get call.

tJavaRow

The component used to print the response from the API call.

tFileOutputRaw

The component used to create a JSON file with the API response body.

tSystem

Component to call the Python code.

tJava

 

Related Content

Log management and monitoring in Talend Cloud.

[test1233]

Hello Team, The job attachments are missing. Please advise. Thank you 

[Sonja_Bauernfeind]

@test1233 

Thank you for highlighting this. The attachment is now available.

All the best,
Sonja