topic Re: Unreliable results from tRESTClient? in Talend Studio

Unreliable results from tRESTClient?

Anonymous — Wed, 23 Apr 2014 02:35:19 GMT

I have tRESTClient in a subjob. The parent job loops through pages on a web service (changing the ?Page= parameter on the URL) and everything works fine -- until it doesn't.
I have had errors:
WARNING: Interceptor for {<url>?Page=115}tRESTClient has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message
WARNING: Interceptor for {<different url endpoint>?Page=89}tRESTClient has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message
<page 136 end | success>
Exception in component tRESTClient_1
javax.ws.rs.ServiceUnavailableException
all of which I suspect are caused by problems with the server I am attempting to access.
All requests work perfectly WHEN THEY ARE RERUN.
How can I get more reliable results from a (possibly) unreliable REST service?

Re: Unreliable results from tRESTClient?

Anonymous — Fri, 25 Apr 2014 09:33:47 GMT

Hi,

all of which I suspect are caused by problems with the server I am attempting to access.
All requests work perfectly WHEN THEY ARE RERUN.

This exception is random? Is there any time-out setting on this server?
Best regards
Sabrina

Re: Unreliable results from tRESTClient?

Anonymous — Sun, 27 Apr 2014 21:40:46 GMT

Yes. The fault appears at random. A response from this server normally completes a 13MB download in 90 seconds. I had used Advanced Settings in my tRESTClient to set a Connection timeout of 600 seconds and a Receive timeout of 600 seconds. There is no indication the problems result due to a Talend timeout as the errors are thrown well within these 10 minute windows.
To work around this problem, I have used a tJava node connected to the tRESTClient using an OnComponentError trigger and written a separate bash script wrapper for the .sh script exported from Talend. The tJava node writes the failed page number to a file (page_to_restrart) and exits with status code of 1:
BufferedWriter writer = null;
try {
File logFile = new File("page_to_restart");
// This will output the full path where the file will be written to...
System.out.println(logFile.getCanonicalPath());
writer = new BufferedWriter(new FileWriter(logFile));
writer.write(Integer.toString(context.page)+"\n");
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
// Close the writer regardless of what happens...
writer.close();
} catch (Exception e) {
}
}
System.exit(1);
The bash script tests for the status_code being 1 and restarts on the failed page. Over the course of downloading 222 pages, the server failed 7 times in my latest run.

Re: Unreliable results from tRESTClient?

Anonymous — Mon, 28 Apr 2014 04:33:36 GMT

Hi,
Could you please open a work item jira issue of DI project on TalendBugTracker so that our component developer will check it to see if there is a solution for your issue.
Best regards
Sabrina