Talend Remote Engine Job start failed

[XRocq1696001865]

Hello,

On Talend Remote Engine 2.13.9 we have this random error 1% of the time:

Failure type: UNDEFINED_ERROR

Failure message: org.talend.remote.commons.exception.ClientServerException: Remote server has encountered an error: , messageType=SERVER_ERROR, idRequestResponsePair=null, message=Job start failed because of parallel cleanup or deployment jobId

This error is completely random, never the same task never the same time and only between 4 and 6 times a day.

We are not able to reproduce it because its random and around 1% of our runs

Let's see the logs for trying to understand step by step: 

"Action 'deployJob' executed with success." => Deployment step OK

"starting job execution with 30 minutes timeout under task execution 65798xb7f803c2469667ae29:2.3/31f2e220-aea3-4f8b-b5e7-94878c38d2b7 " =>  Execution step OK

"starting job dx7f803c2469667ae2923 for task execution 31f2e220-aea3-4f8b-b5x7-94878c38d2b7"

"Remote server has encountered an error:
, messageType=SERVER_ERROR, idRequestResponsePair=null, message=Job start failed because of parallel cleanup or deployment jobId = dx7f803c2469667ae2923" => Starting step KO

We can see it can be : 

1) "Parallel cleanup" => i look into the documentation and i didnt find anything about what is Parallel cleanup and how we can configure it...

2) "Parallel deployment jobId" => The job of this example hasnt have any another instance at the same time, its launching only 1 time a week and the last execution 7 days ago was finished ok with 1 minute duration...

More over, we don't want a same task be able to run in parallel and when this scenario is happening we have a "misfired task" launch status which is ok for us so its not about a parallel task execution

Does someone have an idea about the cause of this error ?

Thanks

[XRocq1696001865 · Accepted solution]

Hello,

After trying some settings on my side, the temporary solution is to deactivate the general cleanup of the remote engine (FREQUENCY_CLEAN_ACTION to 0) :

https://help.qlik.com/talend/en-US/installation-guide-linux/Cloud/understanding-the-talend-remote-en...

After that, the talend support confirmed that :

"Hello

Your issue seems to be related indeed with 2.13.7 2.13.8 and 2.13.9
this was intoduced by capacity based pricing.

issue will be solved with 2.13.10 released end of April now, the solution consist on downgrading to 2.13.6"

[MadGluck]

Unfortunately, it seems that updating to version 2.13.11 doesn't resolve this issue

[Kishor]

Hello,

have you made any configuration changes to TAC 

The same error msg we found in my case after doing some configuration changes in TAC

Please let me know if you have done any configuration changes 

Thanks,

Kishor 

[XRocq1696001865]

Hi kishor, unfortunately i'm not using the TAC, i'm using the TMC (cloud)

But i suggest you to stop your remote engine then update the FREQUENCY_CLEAN_ACTION value to 0 (in the org.talend.remote.jobserver.commons.config.JobServerConfiguration fil)

following this official way => https://help.qlik.com/talend/en-US/installation-guide-linux/Cloud/understanding-the-talend-remote-en...

[MadGluck]

Why is the issue marked as resolved if disabling cleanup will lead to the server’s free space being filled up?  And the promise to fix the issue in newer versions of Talend Remote Engine was not fulfilled.