topic Re: How can i avoid the 429 response from API Call in Talend Studio

How can i avoid the 429 response from API Call

kakooo16 — Mon, 20 Apr 2020 17:39:15 GMT

Hello ,

Im facing a problem that i couldn't solve .

Here is my Job

Eventually , i know what is causing the problem (mylist) --> because im iterating 22000 items : these items are stocked in mylist

The question is how could i break down my list which is composite of 22000 to 200 items so that i could iterate the first 200 items until the list of 22000 items is over .

Thanks in advance .

Re: How can i avoid the 429 response from API Call

Anonymous — Tue, 21 Apr 2020 02:39:53 GMT

Hi
My idea is to split the items to several files (eg: 200 items in one file) on disk, and then iterate each file. eg:
bababa---tfileOutputDelimited
|onsubjobok
tFileList--iterate--tfileinputDelimited--tFlowToIterate--iterate---tRest...bababa--oncomponento--tSleep( if needed)
|onsubjobok
tFileDelete

On tFileOutputDelimited: check the 'split the output to several files' box in the advanced settings panel.

tFileDelete: delete the file folder in the end.

Can you try if it could avoid the 429 response from API Call.

Regards
Shong

Re: How can i avoid the 429 response from API Call

kakooo16 — Tue, 21 Apr 2020 08:24:38 GMT

Hello Shong ,

I really don't get how to put every 200 in a tfileOutputDelimited when we are talking about 22000 items .

I've been reading about ''OFFSET" yesterday and if i m not wrong this will create a sort of pagination (which seems a great idea) .

So i've changed my query to be like this :

Im no longer getting the 429 for a list of 50 items .

Now how could i iterate the other items . here is my main workflow job .

However , do i have to add a tloop before my tpostgresInput query to iterate the rest of the items .

Hope that will be a great idea .

And im waiting for a reply or an other idea because im still stuck in this .

Re: How can i avoid the 429 response from API Call

kakooo16 — Tue, 21 Apr 2020 14:53:51 GMT

Is there any suggestions ?