Talend Remote Engines houekeeping

[Stephen-Elves]

So it would appear that the Remote Engines do not out of the box clean-up after themselves leaving large amounts of data in <install_dir>/TalendJobServersFiles/repository/, including the temp directory.

There are 2 major problems with this:

1) If your jobs stage data to disk you rapidly run out of space.

2) Data, possibly including sensitive data, is left "at rest" with all the GDPR, etc. implications of this.

So is there a switch or setting that I can change to get the Remote Engine to properly clean up after itself or will I have to work out a way to do it using CRON, etc?

[Anonymous]

Hello,

Do you want to delete the files in tmp folder (TalendRemoteEngine\data\tmp) without affecting the operation of Talend Integration?

We can delete the content when the Remote Engine is ON but we would better suggest to stop the Remote Engine, DELETE the content in tmp folder and Restart the Remote Engine.

Best regards

Sabrina

[sajal23]

Is there a way to delete the temp files automatically after the job finishes? like a cleanup step post job completion?

[Stephen-Elves]

No I ideally want to delete the content of /TalendJobServersFiles/repository/<job_name and version>/<artefact Name>/temp automatically when a job ends or is terminated (ideally the behaviour should be switchable).

It's all very well having a tPostJob procedure that cleans up the files, but this won't run if the job crashes or is terminated leading to not only running out of disc space but also data protection compliance issues.