Configuring Talend Administration Center with Filebeat

TalendSolutionExpert — Tue, 23 Jan 2024 02:35:30 GMT

This is a continuation of the Talend Community Knowledge Base article, Talend logging using Filebeat.

Depending on your Talend Administration Center (TAC) installation (single component or Talend bundle component), TAC can be installed alone or with Filebeat.

If Filebeat is not part of the installation, copy a Filebeat installation folder from another installation folder (for example, the one installed with Talend logging using Filebeat), and configure a dedicated service.

Configuring Talend Administration Center portal

Talend Administration Center uses two files to collect technical and business logs. Configure these files on Talend Administration Center (TAC) by performing the following steps.

Log in to TAC.
Change the location of the Technical and Business logs in the Logs section on the TAC Configuration page, as follows:
- Technical file appender: %TAC_installation%/tac/logs/technical.log
- Business log file path: %TAC_installation%/tac/logs/business.log
These files and folders should be created manually on the machine, and the TAC service user should have permission on it.

Configuring Filebeat

Configure Filebeat, on the installation location, to permit Log Server to redirect logs to a specific logstash instance.

Open the %TAC_installation%/filebeat/filebeat.yml file and change output.logstash: > hosts: with the location of your Log Server, as shown below:
```
output.logstash:
 # The Logstash hosts
 hosts: ["logserver_ip:5044"]
 #hosts: ["localhost:5044"]
```

Change the filebeat.prospectors: section. By default, there are already two sub-sections, identified by fields > app_id: TAC and DQ (out of scope, not reported).

filebeat.prospectors:
- type: log 
  enabled: true 
  paths: 
    - %TAC_installation_path%/tac/logs/*.log 
  fields: 
    app_id: TAC 
  fields_under_root: true

Ensure that the paths in the TAC section point to the same location of the logs you configured on TAC portal.

Note: It is also possible to redirect the apache-tomcat logs by creating a new sub-section, as shown below:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - %TAC_installation_path%/tac/logs/*.log
  fields:
    app_id: TAC
  fields_under_root: true
- type: log
  enabled: true
  paths:
    - %TAC_installation_path%/tac/apache-tomcat/logs/*.log
  fields:
    app_id: TAC_APACHE_TOMCAT
  fields_under_root: true

Restart the Filebeat service.

Kibana portal results

The following Kibana portal screenshots show two log entries related to logout and login:

(filter - application.keyword: “TAC”)

For more information on configuring Talend components with Filebeat, see the following Talend Community Knowledge Base (KB) articles:

article Configuring Talend Administration Center with Filebeat in Official Support Articles

Configuring Talend Administration Center with Filebeat

Configuring Talend Administration Center portal

Configuring Filebeat

Kibana portal results