Hi!
Over the last weeks I am constantly running into memory issues.
These are caused by a high number of files in the following directories:
Job Server installation on Talend Server:
1. ...\JobServer\repository (data is more than 1 year old)
2. ...\JobServer\jobexecutions\logs
Scheduler settings from TAC
3. ...Administrator\archiveJobs
4. ...Administrator\executionLogs
The settings in the JobServer.properties are like this that files older than 3 months are automatically deleted. Obviously this is not working. Also I am wonderung whether the parameter org.talend.remote.jobserver.server.JobServerConfiguration.MAX_OLD_EXECUTIONS_LOGS refers to all logs at all or to all logs per job / task?
Since the content of those 4 directories sums up to 50GB (of 80GB total) I definitely need to delete files.
Could somebody please explain where and how to change settings to automatically decrease size of folders?
Old files from deleted tasks are never used again so they don't need to be kept.
Many thanks,
Reiko
TIS-version: 4.1.2r53616
Hi Reiko
The cleaning parameter from TAC are configurated in <Tomcat installation-installation-path>/webapp/org.talend.administrator/WEB-INF/classes/configuration.properties, for example:
# Time in seconds between each cleaning action, set with 0 to disable all cleanings scheduler.conf.cleaning.frequencyCleaningAction=600 # Time in seconds before cleaning executions logs from the current date, 3 months=3*30*24*60*60=7776000, set with 0 to disable delete action with this method scheduler.conf.cleaning.maxDurationBeforeCleaningOldExecutionsLogs=0 # Max executions logs to keep, set with 0 to disable delete action with this method. scheduler.conf.cleaning.maxOldExecutionsLogs=1000 # Time in seconds before cleaning archives jobs from the current date, 3 months=*30*24*60*60=7776000, set with 0 to disable delete action with this method scheduler.conf.cleaning.maxDurationBeforeCleaningOldJobs=0 # Max archive jobs to keep, set with 0 to disable delete action with this method. scheduler.conf.cleaning.maxOldJobs=3
Can you show us your settings from Job server and TAC?
Hi Shong,
settings in tomcat..../configuration.properties are exactly what you stated. Guess these are the standard settings.
in TalendJobServer.properties settings are like this:
###############################################################
# Temporary data cleaning parameters
###############################################################
# Time in seconds between each cleaning action, set with 0 to disable all cleanings
org.talend.remote.jobserver.server.JobServerConfiguration.FREQUENCY_CLEAN_ACTION=600
# Time in seconds before cleaning executions logs from the current date, 3 months=3*30*24*60*60=7776000, set with 0 to disable delete action with this method
org.talend.remote.jobserver.server.JobServerConfiguration.MAX_DURATION_BEFORE_CLEANING_OLD_EXECUTIONS_LOGS=2592000
# Max executions logs to keep, set with 0 to disable delete action with this method.
org.talend.remote.jobserver.server.JobServerConfiguration.MAX_OLD_EXECUTIONS_LOGS=100
# Time in seconds before cleaning archives and deployed jobs from the current date, 3 months=3*30*24*60*60=7776000, set with 0 to disable delete action with this method
org.talend.remote.jobserver.server.JobServerConfiguration.MAX_DURATION_BEFORE_CLEANING_OLD_JOBS=2592000
# Max archive and deployed jobs to keep, set with 0 to disable delete action with this method.
org.talend.remote.jobserver.server.JobServerConfiguration.MAX_OLD_JOBS=10
# Time in seconds before cleaning cached libs from the current date, 3 months=3*30*24*60*60=7776000, set with 0 to disable delete action with this method
org.talend.remote.jobserver.server.JobServerConfiguration.MAX_DURATION_BEFORE_CLEANING_OLD_LIBS=7776000
# Max cached libs to keep, set with 0 to disable delete action with this method.
org.talend.remote.jobserver.server.JobServerConfiguration.MAX_OLD_LIBS=300
I would change the settings in TAC-properties to fit our environment.
However one question is still open: Does each of the settings apply to one job always or to all jobs? Meaning, if we decide to keep 100 logs files - do we keep 100 files for each tasks or overall?
Thanks, Reiko
Hi
The settings in TAC-properties are the defaut one, you need to change them to fit your environment. Don't forget to restart Tomcat or Job Server to apply the changes whenever you modify the settings.
These settings are for all jobs, not only one job.
Hi Shong,
even though I changed the settings and restarted the server files are not deleted.
Is there a "main switch" I need to turn on in order to let Tomcat use the settings in the properties-file?
Thanks, Reiko
In our company Environment we have set both file but it doesn't clear archive.
Example with file this setting in configuration.propreties:
### Temporary data cleaning parameters ###
# Time in seconds between each cleaning action, set with 0 to disable all cleanings
scheduler.conf.cleaning.frequencyCleaningAction=600
# Time in seconds before cleaning executions logs from the current date, 3 months=3*30*24*60*60=7776000, set with 0 to disable delete action with this method
scheduler.conf.cleaning.maxDurationBeforeCleaningOldExecutionsLogs=2592000
# Max executions logs to keep, set with 0 to disable delete action with this method.
scheduler.conf.cleaning.maxOldExecutionsLogs=1000
# Time in seconds before cleaning archives jobs from the current date, 3 months=3*30*24*60*60=7776000, set with 0 to disable delete action with this method
scheduler.conf.cleaning.maxDurationBeforeCleaningOldJobs=2592000
# Max archive jobs to keep, set with 0 to disable delete action with this method.
scheduler.conf.cleaning.maxOldJobs=3
We have file in opt/Talend-5.0.2/tac/archive/cmdline 1 year old..
It's a problem already present?
We have Talend 5.0.2.
thanks
Hi, which is the parameter for cleaning the commandLine job generation folder? Thanks and Best Regards, Francesco
There is no parameter for cleaning the cmdline job generation folder, you have to delete manually delete the old archive files right now, however, it could be a new feature.
Shong
