Configuring Talend JobServer and Talend ESB Runtime with Filebeat

TalendSolutionExpert — Tue, 23 Jan 2024 02:35:30 GMT

This article introduces the best practices that Talend suggests you follow when working with Filebeat in Talend JobServer and Talend ESB Runtime.

This is a continuation of the Talend Community Knowledge Base article, Talend logging using Filebeat.

JobServer

The JobServer installation doesn’t include the Filebeat component, so you'll need to copy a Filebeat installation folder from another installation folder (for example the one installed with Talend logging using Filebeat) and configure a dedicated service.

Configuring TAC portal

TAC uses two folders dedicated to the Job generation and execution. Configure these folders on TAC by following these steps.

Log in to TAC.
Change the location of the Job conductor logs on the TAC Configuration page, as follows:
- Generated jobs folder: %JOBSERVER_installation%/Administrator/generatedJobs
- Task logs folder: %JOBSERVER_installation%/Administrator/executionLogs
These folders should be created manually on the machine, and the TAC service user should have permission on it.

Configuring Studio Log4j

Before enabling Filebeat and redirecting the logs to Log Server, you need to review the Log4j configuration for Job execution. The Job execution log doesn’t include a timestamp for each line, and this may cause Filebeat not to recognize and map all log details.

For this reason, you need to change the Log4j configuration to generate a timestamp into the log.

Open the project in Talend Studio, then select File > Edit Project properties > Log4j.
Scroll to  and change the <param name="ConversionPattern" value="[%-5p]: %c - %m%n" /> value to:
```
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p: %c - %m%n" />
```
Click Apply, then click OK.
Redeploy the Job.

Configuring Filebeat

Configure Filebeat, on the installation location, to permit Log Server to redirect logs to a specific logstash instance, by performing the following steps:

Open the %filebeat_installation%/filebeat.yml file and change output.logstash: > hosts: with the location of your Log Server, as shown below:
```
output.logstash:
 # The Logstash hosts
 hosts: ["logserver_ip:5044"]
 #hosts: ["localhost:5044"]
```

Change the filebeat.prospectors: section as shown below:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - %JOBSERVER_installation_path%/Administrator/executionLogs/*/*.log
  fields:
    app_id: JOBSERVER
  fields_under_root: true
  multiline.pattern: '([12]\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]))'
  multiline.negate: true
  multiline.match: after
  exclude_lines: ['###']

If you want to send the entire log file to one entry, use the following command:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - %JOBSERVER_installation_path%/Administrator/executionLogs/*/*.log
  fields:
    app_id: JOBSERVER
  fields_under_root: true
  multiline.pattern: '([### Job STARTED'
  multiline.negate: true
  multiline.match: after

Restart the Filebeat service.

Kibana portal results

The following Kibana portal screenshots show logs related to Job execution:

(filter - application.keyword: “JOB_SERVER”)

ESB Runtime

On Talend 7.1, the event logging solution has been deprecated and replaced by Event Monitoring.

Event Monitoring is an implementation of Apache Karaf Decanter, monitoring solution running in Apache Karaf.

The ESB Runtime installation doesn’t include the Filebeat component, so you'll need to copy a Filebeat installation folder from another installation folder.

Configuring containers

ESB Containers need to be configured to redirect logs to logstash. To configure and redirect all logs to the centralized Log Server, perform the following steps.

Open the %ESB_runtime_installation%/esb/%container_name%/etc/filebeat_em.yml file.

Change paths with the right %container_name_folder%.
Change output.logstash: > hosts with the location of your Log Server.

filebeat.prospectors:

- type: log
  enabled: true
  paths:
   - ${LOG_PATH:/opt/Talend-
7.1.1/esb/%container_name_folder%/data/eventmonitoring/em*.log*}
 fields:
   app_id: ${APP_NAME:ESB-app}
 fields_under_root: true
 json.keys_under_root: true
 json.message_key: log

logging.metrics.enabled: false
#----------------------------- Logstash output --------------------------------
output.logstash:
 # The Logstash hosts
 hosts: ["${LOGSTASH_HOSTS:logserver_ip}:${LOGSTASH_PORT:5044}"]
 #hosts: ["${LOGSTASH_HOSTS:localhost}:${LOGSTASH_PORT:5044}"]

ESB container uses different log collectors, which permits sending different log categories to Log Server. Usually, not all collectors are used at the same time.
1. Open the following %ESB_runtime_installation%/esb/%container_name%/etc/ org.apache.karaf.decanter.collector.* files:
  - org.apache.karaf.decanter.collector.log.cfg
  - org.apache.karaf.decanter.collector.jms.cfg
  - org.apache.karaf.decanter.collector.eventadmin-sam.cfg
  - org.apache.karaf.decanter.collector.eventadmin-karaf.cfg
  - org.apache.karaf.decanter.collector.eventadmin-framework.cfg
2. In each file:
  - Change “fields.add.agentId” = agent_%container_name%_%log_type% (all collectors)
  - Change “fields.rename.renderedMessage logMessage” to “fields.rename.renderedMessage message” (only collector-log)
  - Change “fields.rename.level severity” to “fields.rename.level priority” (only for collector-log)
  - Change “fields.remove.message = true” to “#fields.remove.message = true” (all collectors)
  - Change “fields.rename.event.topics = logMessage” to “fields.rename.event.topics = message” (except collector-log)
  - Add “fields.rename.loggerName logger_name” (only collector-log)
Connect in ssh to the Karaf container, by using the following command:
```
#ssh karaf@runtime_ip -p port
```
Run the following command:
```
#tesb:start-em-default
```
This command starts to collect all log category to event monitoring log file (under data/eventmonitoring/em.log). This file is the one referred by Filebeat.
Start Filebeat by using the following command (or create a service):
```
filebeat -e -c {KARAF_HOME}/etc/filebeat_em.yml
```

Kibana portal results

The following Kibana portal screenshots show logs related to ESB Runtime:

Container logs:

filter - agentid: “agent_app_log”

Karaf login logs:

 filter - agentid: “agent_app_karaf”

ESB service deploy logs:

filter - agentid: “agent_app_framework”

For more information on configuring Talend components with Filebeat, see the following Talend Community Knowledge Base (KB) articles:

article Configuring Talend JobServer and Talend ESB Runtime with Filebeat in Official Support Articles

Configuring Talend JobServer and Talend ESB Runtime with Filebeat

JobServer

Configuring TAC portal

Configuring Studio Log4j

Configuring Filebeat

Kibana portal results

ESB Runtime

Configuring containers

Kibana portal results