Talend logging using Filebeat

TalendSolutionExpert — Tue, 23 Jan 2024 02:35:30 GMT

Filebeat, a part of the ELK stack, is a lightweight shipper for forwarding and centralizing log data. This article introduces the best practices that Talend suggests you follow when working with Filebeat.

Click the following links for the recommended configuration when using Filebeat with the following Talend components:

Note: that all components are separate and can be installed on different machines.

The article shows you how to configure and redirect all logs to the centralized logging solution, Log Server.

Configuring Log Server (ELK stack)

The Log Server installation provides the following components:

Kibana portal
Elasticsearch
Logstash
Filebeat

These components are already integrated and don’t require a deep configuration.

Usually, in the DEV/TEST environment, Log Server is installed on the same machine as the other Talend components (TAC, Runtime, and others). Thus, the configuration related to remote connections is not necessary.

However, in the PREPROD or PROD environments, Talend recommends installing Log Server on dedicated machines and clustered.

The configuration covered in this article is only for a single node installation (not a cluster). In Production, Talend recommends a cluster solution, and depending on the load Elasticsearch, Logstash, and Kibana; it can be installed on a dedicated machine.

Firewall ports

Firewall configuration is necessary to permit remote communication to Log Server. Thus, it’s necessary to open the following ports:

5601 - Kibana portal
9200 - Elasticsearch
5044 - Logstash

Elasticsearch

By default, Elasticsearch only accepts a local connection. Edit the configuration to permit Elasticsearch to receive a remote connection using port 9200:

Edit the %LogServer_installation%/logserv/elasticsearch-6.1.2/config/elasticsearch.yml file.
Change network.host to accept the remote connection:
- network.host: ip_address - binding to accept a remote connection to specific IPs
  
  Or
- network.host: 0.0.0.0 - binding to accept all remote connections
Optional: To avoid the bootstrap check (only for single node), add the following parameter:
```
discovery.type: single-node
```

Talend Administration Center portal

If Log Server is installed on a different machine respective of Talend Administration Center (TAC), it is necessary to set the Kibana URL in TAC.

Connect to TAC.
Navigate to Configuration > Monitoring, and configure the Kibana URL by replacing the localhost with the Log Server IP address. This configuration permits Log Server to see the Kibana portal from the TAC portal under Monitoring > Logging.

article Talend logging using Filebeat in Official Support Articles

Talend logging using Filebeat

Configuring Log Server (ELK stack)

Firewall ports

Elasticsearch

Talend Administration Center portal