Unreliable results from tRESTClient?

One Star

Unreliable results from tRESTClient?

I have tRESTClient in a subjob. The parent job loops through pages on a web service (changing the ?Page= parameter on the URL) and everything works fine -- until it doesn't.
I have had errors:
WARNING: Interceptor for {<url>?Page=115}tRESTClient has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message
WARNING: Interceptor for {<different url endpoint>?Page=89}tRESTClient has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message
<page 136 end | success>
Exception in component tRESTClient_1
javax.ws.rs.ServiceUnavailableException
all of which I suspect are caused by problems with the server I am attempting to access.
All requests work perfectly WHEN THEY ARE RERUN.
How can I get more reliable results from a (possibly) unreliable REST service?
Moderator

Re: Unreliable results from tRESTClient?

Hi,
all of which I suspect are caused by problems with the server I am attempting to access.
All requests work perfectly WHEN THEY ARE RERUN.

This exception is random? Is there any time-out setting on this server?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Unreliable results from tRESTClient?

Yes. The fault appears at random. A response from this server normally completes a 13MB download in 90 seconds. I had used Advanced Settings in my tRESTClient to set a Connection timeout of 600 seconds and a Receive timeout of 600 seconds. There is no indication the problems result due to a Talend timeout as the errors are thrown well within these 10 minute windows.
To work around this problem, I have used a tJava node connected to the tRESTClient using an OnComponentError trigger and written a separate bash script wrapper for the .sh script exported from Talend. The tJava node writes the failed page number to a file (page_to_restrart) and exits with status code of 1:
BufferedWriter writer = null;
try {
File logFile = new File("page_to_restart");
// This will output the full path where the file will be written to...
System.out.println(logFile.getCanonicalPath());
writer = new BufferedWriter(new FileWriter(logFile));
writer.write(Integer.toString(context.page)+"\n");
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
// Close the writer regardless of what happens...
writer.close();
} catch (Exception e) {
}
}
System.exit(1);
The bash script tests for the status_code being 1 and restarts on the failed page. Over the course of downloading 222 pages, the server failed 7 times in my latest run.
Moderator

Re: Unreliable results from tRESTClient?

Hi,
Could you please open a work item jira issue of DI project on TalendBugTracker so that our component developer will check it to see if there is a solution for your issue.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch