Remote Engine \data\tmp folder filling up

[David_Beaty]

Hi,

Is there any way, as part of the Remote Engine configuration, to clear down old files from the <Remote Engine>\data\tmp folder.

It very quickly fills up with files "download-{big number}.tmp".

When this number gets very large (80,000 or more) we observe odd behavior in the jobs that run on the server, things like creation of temporary files via tCreateTemporaryFile, tFileInput{thing} or tFileOutput{thing} all getting access denied.

When we stop the remote engine and clear all of the content from this folder and start the Remote Engine the problem goes away, until the dross left here builds back up again.

I've seen from a Community Knowledge page, that the recommended approach is as I describe above (stop engine, clear folder, start engine), but it should really handle this itself, like it does with the contents of "\TalendJobServersFiles\jobexecutions\logs", which seems to be configured in the file org.talend.remote.jobserver.server.cfg

Thanks

David

[kurzkano]

Good hint! Had a look in that folder on the remote engine and found more than 305.000 items, rapidly increasing!

Lots of:

• {big HEX nmber}{bignumber}_log4j2.xml
• download-{big number}.tmp

and a bunch of

• jansi-1.17.1-{big number}.dll

Stopping the engine is not an option!

[kurzkano]

Forgot to mention the version: Talend-RemoteEngine-V2.8.1-165 running on Windows Server 2016.

[David_Beaty]

Yup, checked again this morning, and is building up again as you describe.

Ours Remote Engine is v2.8.0-165 on Windows Server 2016 Datacenter

[Anonymous]

Hello,

Here is a workaround. Could you please let me know if it helps?

it’s not the cleanest solution of this world but you can implement a job that deletes the files in the folder and schedule it to run periodically (if you don’t need to stop the RE first), or you can create a bash script (if you are on Linux) that does everything and then using cron to schedule it.

Thanks for your time.

Best regards

Sabrina

[kurzkano]

Running on Windows we created a scheduled task calling ForFiles... which deletes *.tmp and *.xml files older than 1 day running at 5:00 AM.

Not the cleanest solution as you said, Sabrina, but it works.

[Anonymous]

TalendRemoteEngine\data\tmp: this folder is for internal use of Apache Karaf. We can't, or I would say it's dangerous to delete something from here while the engine is running.

If you run jobs in Remote Engine, all jobs are saved in TalendRemoteEngine\TalendJobServersFiles folder. 

Instead I would recommend updating the following property to "true" to do a /data/ directory cleanup, but this will require RE to be restarted:

Property file name: \RemoteEngine\etc\system.properties

karaf.clean.all = true

####Line 70-72 of \etc\system.properties####

# Deletes the entire karaf.data directory at every start

#

karaf.clean.all = false

This would require the engine to be restarted periodically but it would clean the file automatically on restart.

[Anonymous]

Hi, we have create the jira-bug TMC-24629 to log this issue and they will look how this can be fixed.

For now, you can remove those temporary files (download-{N-digits}.tmp) and you do not need to stop/start RE for this.

You just need to be sure that you are not deleting the current tasks executions. So if no task is running for more than x days, you can delete all those files older than those x days.