topic Re: Talend Remote Engines houekeeping in Talend Studio

Talend Remote Engines houekeeping

Stephen-Elves — Sat, 16 Nov 2024 07:03:53 GMT

So it would appear that the Remote Engines do not out of the box clean-up after themselves leaving large amounts of data in <install_dir>/TalendJobServersFiles/repository/, including the temp directory.

There are 2 major problems with this:

1) If your jobs stage data to disk you rapidly run out of space.

2) Data, possibly including sensitive data, is left "at rest" with all the GDPR, etc. implications of this.

So is there a switch or setting that I can change to get the Remote Engine to properly clean up after itself or will I have to work out a way to do it using CRON, etc?

Re: Talend Remote Engines houekeeping

Anonymous — Wed, 26 Dec 2018 07:27:11 GMT

Hello,

Do you want to delete the files in tmp folder (TalendRemoteEngine\data\tmp) without affecting the operation of Talend Integration?

We can delete the content when the Remote Engine is ON but we would better suggest to stop the Remote Engine, DELETE the content in tmp folder and Restart the Remote Engine.

Best regards

Sabrina

Re: Talend Remote Engines houekeeping

sajal23 — Sat, 29 Dec 2018 01:53:38 GMT

Is there a way to delete the temp files automatically after the job finishes? like a cleanup step post job completion?

Re: Talend Remote Engines houekeeping

Stephen-Elves — Wed, 30 Jan 2019 16:16:18 GMT

No I ideally want to delete the content of /TalendJobServersFiles/repository/<job_name and version>/<artefact Name>/temp automatically when a job ends or is terminated (ideally the behaviour should be switchable).

It's all very well having a tPostJob procedure that cleans up the files, but this won't run if the job crashes or is terminated leading to not only running out of disc space but also data protection compliance issues.