topic Re: Iterate API calls based on count divided by pagesize in Talend Studio

Iterate API calls based on count divided by pagesize

Serendipity — Sun, 15 Aug 2021 04:56:21 GMT

I'm looking for some help on a strategy here - I've got an API that offers pagination by using a separate endpoint to return a count of records, which you then use to calculate how many pages based on the pagesize, which is a max of 100.

So, I'm needing to make a call to one endpoint /records/count and then take that result and divide by 100 for the page size. So if the count is 347, I need to add the page argument to the /records endpoint for each page.

I'm not sure how to take the count value from the API result and then iterate through to use it as a variable in the URL Query parameters.

Re: Iterate API calls based on count divided by pagesize

Serendipity — Mon, 16 Aug 2021 01:49:20 GMT

Made some progress on at least getting the count value, but I feel like this flow is overcomplicated:

Here is the result of the tREST:

<?xml version="1.0" encoding="UTF-8"?>

<resource><count>482</count></resource>|

I get having to convert the string that returns to a document with tConvert, but do I really have to ExtractXML and then XMLMap, I tried so many variations on just going from tConvert to tXML map, but I never got the value.

Re: Iterate API calls based on count divided by pagesize

Serendipity — Mon, 16 Aug 2021 17:34:55 GMT

Thanks for that info, that sounds like it would be very helpful.

However - I'm not seeing any links attached, but I'll see if I can search them out in your other posts.

Re: Iterate API calls based on count divided by pagesize

Anonymous — Tue, 17 Aug 2021 01:30:25 GMT

Why you convert the string result to a document? You can extract the count value from the string result using tExtractXMLField.

Re: Iterate API calls based on count divided by pagesize

Serendipity — Tue, 17 Aug 2021 22:36:47 GMT

I'm struggling to get all the schema aligned and proper, so that helps alleviate the errors I kept getting until I can better understand that setup.

When I connect without the convert node, I get an error about converting an integer to a string, even though the field I'm looking at isn't an integer, or I'll get an error that body cannot be resolved or is not a field. So, basically through trial and error I found something that worked.

Between trying to get this project working, I'm going through tutorials so hopefully I'll get a better understanding of what I'm doing wrong with all the schema settings.

Re: Iterate API calls based on count divided by pagesize

Serendipity — Wed, 18 Aug 2021 01:34:42 GMT

So, I've got my count from the API /count endpoint

I've converted it to an integer so I can calculate total pages needed.

I'm thinking I take the calculated value, set a global variable, then use a tLoop to iterate through? However, I'm not figuring out how to reference the global variable within the tLoop, then how in turn to increment until the total page count is done.

I'm trying this loop setup:

In this workflow:

So, the value I set in the tXMLMap_8 pageCount output is what I want to use as the variable in the loop. In this instance, pageCount is 4, and I want the loop to start at 1, and then increase until 4, and the variable I'm using in the URL would then need to be updated with each iteration.

Here's the tSetGlobalVar:

Re: Iterate API calls based on count divided by pagesize

Anonymous — Wed, 18 Aug 2021 04:31:30 GMT

one quick question, is tLoop used to iterate another API call? i see one issue about getting the value of global variable for query parameter "page", the double quotes should remove.

(String)globalMap.get("page")

Regards

Shong

Re: Iterate API calls based on count divided by pagesize

Serendipity — Sun, 22 Aug 2021 22:28:41 GMT

I am trying to use tLoop to take the "pageCount" variable that I'm calculating, and use that as the upper limit of the For loop.

I have the "pageCount" value showing in a table in a log to verify it's there:

.-----+---------.

| tLogRow_15 |

|=----+--------=|

|value|key |

|=----+--------=|

|4 |pageCount|

'-----+---------'

But when run the tLoop I get

Detail Message: The operator <= is undefined for the argument type(s) int, Object

Re: Iterate API calls based on count divided by pagesize

Anonymous — Mon, 23 Aug 2021 07:35:38 GMT

change the To expression to:

(Integer)globalMap.get("pageCount.value")