Qlik Talend Integration: How to Delete Elasticsearch Logs Automatically?

[Xiaodi_Shi]

Elasticsearch logs are generated in the Logserver/elasticsearch-1.5.2/log directory. If the log files are not moved or deleted, disk space can fill up. Can Elasticsearch log files be purged automatically?

Resolution

Update the Elasticsearch log configuration, and use the MaxBackupIndex option to determine how many backup files are kept before the oldest is erased.

The Elasticsearch logs configuration can be found in Logserver/elasticsearch-1.5.2/config/logging.yml.

The default configuration is:

"
file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
"

With this configuration a file will be created every day in order to save the previous day's log. As a result, the number of log files will increase, and it can lead to the disk full problem.

To restrict the amount of backup files, set MaxBackupIndex:

1. Stop these Talend services:

     Talend Log Server search engine

     Talend Log Server analytics and visualization platform

2. Go to the <Talend_logserv_installation>elasticsearch-2.4.0\config\ folder.
3. Open the logging.yml file and modify it as follows, adding maxFileSize and maxBackupIndex:
   

   "
   file:
   type: rollingFile
   file: ${path.logs}/${cluster.name}.log
   maxFileSize : 200KB
   maxBackupIndex: 4
   layout:
   type: pattern
   conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
   "

4. Save the file and restart the services.

Related Content:

Qlik Talend Product: Reducing the logging threshold for Elasticsearch in Talend Log Server  

 

Environment

Talend Data Integration