Component tHttpRequest hangs when the JSON response is too big

Highlighted
Sixteen Stars
Sixteen Stars

Component tHttpRequest hangs when the JSON response is too big

Hello guys,

 

I've a service developed using TOS ESB 6.4.1 which must call an other service exposed by a data provider to get informations about some companies (I can't give more details).

Anyway, sometimes the JSON response size is "normal" and everything works fine but sometimes, depending on the company size, the response is really huge: for example, 26 Mb Mo with a formatted JSON composed of 565,750 lines!

In this case, tHttpRequest never ends.

I've also tried with tRESTClient with the same problem.

When I call this service using Postman everything works fine, so I can say the called service is not the problem, only the response size is the problem.

 

Any idea of how I can deal with such a huge JSON response?


TRF

Accepted Solutions
Highlighted
Sixteen Stars
Sixteen Stars

Re: Component tHttpRequest hangs when the JSON response is too big

Problem has been solved by the data provider himself by reducing the response size.

Thank's guys.


TRF

View solution in original post


All Replies
Highlighted
Nine Stars

Re: Component tHttpRequest hangs when the JSON response is too big

Did you try streaming the HTTP response?  If you write it in Java you can stream it so the response is not read into memory fully.  That might be your issue, not sure.

https://stackoverflow.com/questions/11710683/stream-a-http-response-in-java

--
Please give Kudos and mark topics as solved where appropriate.
Highlighted
Sixteen Stars
Sixteen Stars

Re: Component tHttpRequest hangs when the JSON response is too big

The idea is to use components available within TOS. However if they all have this kind of limitations, I'll do it by myself using a piece of Java code.

TRF
Highlighted
Sixteen Stars
Sixteen Stars

Re: Component tHttpRequest hangs when the JSON response is too big

Problem has been solved by the data provider himself by reducing the response size.

Thank's guys.


TRF

View solution in original post

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Talend API Designer – Technical Overview

Take a look at this technical overview video of Talend API Designer

Watch Now

Getting Started with APIs

Find out how to get started with APIs

Read