Component tHttpRequest hangs when the JSON response is too big

Highlighted
Fifteen Stars TRF
Fifteen Stars

Component tHttpRequest hangs when the JSON response is too big

Hello guys,

 

I've a service developed using TOS ESB 6.4.1 which must call an other service exposed by a data provider to get informations about some companies (I can't give more details).

Anyway, sometimes the JSON response size is "normal" and everything works fine but sometimes, depending on the company size, the response is really huge: for example, 26 Mb Mo with a formatted JSON composed of 565,750 lines!

In this case, tHttpRequest never ends.

I've also tried with tRESTClient with the same problem.

When I call this service using Postman everything works fine, so I can say the called service is not the problem, only the response size is the problem.

 

Any idea of how I can deal with such a huge JSON response?


TRF

Accepted Solutions
Fifteen Stars TRF
Fifteen Stars

Re: Component tHttpRequest hangs when the JSON response is too big

Problem has been solved by the data provider himself by reducing the response size.

Thank's guys.


TRF

All Replies
Nine Stars

Re: Component tHttpRequest hangs when the JSON response is too big

Did you try streaming the HTTP response?  If you write it in Java you can stream it so the response is not read into memory fully.  That might be your issue, not sure.

https://stackoverflow.com/questions/11710683/stream-a-http-response-in-java

--
Please give Kudos and mark topics as solved where appropriate.
Fifteen Stars TRF
Fifteen Stars

Re: Component tHttpRequest hangs when the JSON response is too big

The idea is to use components available within TOS. However if they all have this kind of limitations, I'll do it by myself using a piece of Java code.

TRF
Fifteen Stars TRF
Fifteen Stars

Re: Component tHttpRequest hangs when the JSON response is too big

Problem has been solved by the data provider himself by reducing the response size.

Thank's guys.


TRF

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

An API-First Approach to Modernizing Applications

Learn how to use an API-First Approach to Modernize your Applications

Watch Now

Talend API Designer – Technical Overview

Take a look at this technical overview video of Talend API Designer

Watch Now

Getting Started with APIs

Find out how to get started with APIs

Read