topic Re: Component tHttpRequest hangs when the JSON response is too big in Talend Studio

Component tHttpRequest hangs when the JSON response is too big

TRF — Sat, 16 Nov 2024 05:20:11 GMT

Hello guys,

I've a service developed using TOS ESB 6.4.1 which must call an other service exposed by a data provider to get informations about some companies (I can't give more details).

Anyway, sometimes the JSON response size is "normal" and everything works fine but sometimes, depending on the company size, the response is really huge: for example, 26 Mb Mo with a formatted JSON composed of 565,750 lines!

In this case, tHttpRequest never ends.

I've also tried with tRESTClient with the same problem.

When I call this service using Postman everything works fine, so I can say the called service is not the problem, only the response size is the problem.

Any idea of how I can deal with such a huge JSON response?

Re: Component tHttpRequest hangs when the JSON response is too big

nfz11 — Tue, 09 Jul 2019 23:34:01 GMT

Did you try streaming the HTTP response? If you write it in Java you can stream it so the response is not read into memory fully. That might be your issue, not sure.

https://stackoverflow.com/questions/11710683/stream-a-http-response-in-java

Re: Component tHttpRequest hangs when the JSON response is too big

TRF — Wed, 10 Jul 2019 07:01:55 GMT

The idea is to use components available within TOS. However if they all have this kind of limitations, I'll do it by myself using a piece of Java code.

Re: Component tHttpRequest hangs when the JSON response is too big

TRF — Thu, 11 Jul 2019 09:12:38 GMT

Problem has been solved by the data provider himself by reducing the response size.

Thank's guys.