Error - The remote server has closed the connection, if the remote JobServer is 5.x the monitoring port is required

[maayandermer]

Hi,

Info about our setup:

Talend Cloud Data Integration, Talend Studio 7.2.1, Using Remote Engines upgraded from 2.4 to 2.7.

Last week we saw a failure that we haven't seen before in one of our UAT environments after upgrading our Remote Engines from 2.4 to 2.7 following Summer '19 upgrade.

We assume it was just a one-off or a network glitch but we started to see intermittent failures on our Prod. I assume it got to do with the Remote Engines upgrade since it was done 2 days before the errors started to occur. 

Task XXXX/2.3 failed unexpectedly.org.talend.ipaas.rt.flow.controller.impl.FlowExecutionException: Step xxxxxxxxxxxxx failed with code -1 and error The remote server has closed the connection, if the remote JobServer is 5.x the monitoring port is required, please ensure you the monitoring port configured for host '127.0.0.1' is reachable, else upgrade your JobServer 5.x to 6.x could avoid this issue. at org.talend.ipaas.rt.flow.controller.impl.JobControllerImpl.doRun(JobControllerImpl.java:103) ~[?:?] at org.talend.ipaas.rt.flow.controller.impl.AbstractControllerImpl.lambda$run$0(AbstractControllerImpl.java:106) ~[?:?] at java.lang.Thread.run(Unknown Source) [?:?]Caused by: java.io.IOException: The remote server has closed the connection, if the remote JobServer is 5.x the monitoring port is required, please ensure you the monitoring port configured for host '127.0.0.1' is reachable, else upgrade your JobServer 5.x to 6.x could avoid this issue. at org.talend.remote.client.AbstractObjectClient.sendCommand(AbstractObjectClient.java:258) ~[?:?] at org.talend.remote.jobserver.client.CommandClient.requestStartJob(CommandClient.java:405) ~[?:?] at org.talend.ipaas.rt.jobserver.client.JobServerClientFactoryImpl.executeJob(JobServerClientFactoryImpl.java:131) ~[?:?] at org.talend.ipaas.rt.flow.controller.impl.JobControllerImpl.doRun(JobControllerImpl.java:84) ~[?:?] ... 2 moreCaused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?] at java.net.SocketInputStream.socketRead(Unknown Source) ~[?:?] at java.net.SocketInputStream.read(Unknown Source) ~[?:?] at java.net.SocketInputStream.read(Unknown Source) ~[?:?] at java.net.SocketInputStream.read(Unknown Source) ~[?:?] at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source) ~[?:?] at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source) ~[?:?] at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source) ~[?:?] at java.io.ObjectInputStream.readObject0(Unknown Source) ~[?:?] at java.io.ObjectInputStream.readObject(Unknown Source) ~[?:?] at org.talend.remote.jobserver.client.CommandClient.readResponseMessage(CommandClient.java:188) ~[?:?] at org.talend.remote.client.AbstractObjectClient.sendCommand(AbstractObjectClient.java:196) ~[?:?] at org.talend.remote.jobserver.client.CommandClient.requestStartJob(CommandClient.java:405) ~[?:?] at org.talend.ipaas.rt.jobserver.client.JobServerClientFactoryImpl.executeJob(JobServerClientFactoryImpl.java:131) ~[?:?] at org.talend.ipaas.rt.flow.controller.impl.JobControllerImpl.doRun(JobControllerImpl.java:84) ~[?:?] ... 2 more

Did anyone encounter that kind of error before? I couldn't find any info on it anywhere

Screenshot attached. 

Cheers

Maayan

[Anonymous]

Hi Maayan,

     I have not seen this issue personally for the remote server. Since its critical to address a production issue quickly, I would recommend to create a support ticket so that support team can check the details remotely using a Webex session.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[maayandermer]

Thanks Nikhil,

I created a ticket immediately when it happened, yet, support sometimes takes time and I was hoped for a quick insight before I'm rolling back all our environments back to RE2.4 

Maayan

[kkryczka]

Hi, 

what was the resolution of this issue? was it issue or network glitch? 

we started to see this issue a week ago. We have Talendf Remote Engines 2.8 and talend batch jobs developed using Talend studio 7.2.1. We are on the Talend Cloud Data Services subscription. 

Thanks

Kris

[Samith_Shetty]

Please go to /etc folder where you're remote engine installed and look for org.talend.remote.jobserver.server.cfg file

and set below parameter to 0

org.talend.remote.server.MultiSocketServer.SERVER_SOCKET_LIFETIME=0

and reboot the server. It works fine.

[Piyushs]

---

@Samith_Shetty wrote:

Please go to /etc folder where you're remote engine installed and look for org.talend.remote.jobserver.server.cfg file

and set below parameter to 0

org.talend.remote.server.MultiSocketServer.SERVER_SOCKET_LIFETIME=0

and reboot the server. It works fine.

---

Hey Samith

A couple of questions around the suggested solution:

The configuration file says the value can not be lower than client configuration value; setting the value to 0 has an impact on other areas/components?

# Cannot be lower than client configuration value 3600

By setting lifetime to 0, the process will keep the connection open indefinitely - can it cause a hung connection if the process is extremely time-consuming?

# Set the maximum life time of a server socket in seconds

# Cannot be lower than client configuration value 3600

org.talend.remote.server.MultiSocketServer.SERVER_SOCKET_LIFETIME=7200

Piyush Srivastava

I am new to Talend