Out of memory issue with tREST API call

[numutesi]

Hi,

I am trying to call a tREST API that returns huge amount of data (more than 50k records) and getting "Out of Memory" error message. I tried increasing the JVM parameters (-Xms = 2048M and -Xmx=6144M) which worked for a few days but now the same issue reappeared.

I am storing the data into a file in /tmp folder.

The call to the API looks like this:

What else should I do to make this work? I believe our API does not allow pagination.

Thank you!

[Sonal_Reddy_18 · Accepted solution]

Can you try using the tHTTPClient component.

[numutesi · Accepted solution]

I have never used it before but I can try. But what the difference that is going to make? 

[numutesi · Accepted solution]

Hi,

So I did try to use the tHTTPClient component instead of tREST but the issue of insufficient memory is still there. The job works fine when I try to retrieve only one record!

So, now I really dont know what to do here!

[Dave_Simo]

Hello,

Try with the tHttprequest component maybe that will give a better result

And if that still doesn't work you should perhaps think about recovering the data by page (pagination) rather than trying to recover everything at once.

Best Regards,