topic Out of memory issue with tREST API call in Talend Studio

Out of memory issue with tREST API call

numutesi — Tue, 03 Dec 2024 19:19:21 GMT

Hi,

I am trying to call a tREST API that returns huge amount of data (more than 50k records) and getting "Out of Memory" error message. I tried increasing the JVM parameters (-Xms = 2048M and -Xmx=6144M) which worked for a few days but now the same issue reappeared.

I am storing the data into a file in /tmp folder.

The call to the API looks like this:

What else should I do to make this work? I believe our API does not allow pagination.

Thank you!

Re: Out of memory issue with tREST API call

Sonal_Reddy_18 — Tue, 10 Dec 2024 19:55:31 GMT

Can you try using the tHTTPClient component.

Re: Out of memory issue with tREST API call

numutesi — Wed, 11 Dec 2024 14:20:51 GMT

I have never used it before but I can try. But what the difference that is going to make?

Re: Out of memory issue with tREST API call

numutesi — Tue, 17 Dec 2024 13:17:16 GMT

Hi,

So I did try to use the tHTTPClient component instead of tREST but the issue of insufficient memory is still there. The job works fine when I try to retrieve only one record!

So, now I really dont know what to do here!

Re: Out of memory issue with tREST API call

Dave_Simo — Tue, 17 Dec 2024 15:51:45 GMT

Hello,

Try with the tHttprequest component maybe that will give a better result

And if that still doesn't work you should perhaps think about recovering the data by page (pagination) rather than trying to recover everything at once.

Best Regards,