Loop through different api calls with pagination

MBI1667841504 — Fri, 15 Nov 2024 22:23:23 GMT

I'm trying to retrieve data out of an api calls with different tokens (credentials), each token represent a different division. The api has a limit of row per page which is why is paginated. So far my job extract the data out of the API looping through the pages until record count is 0, but is only working for 1 token and I have 6 different tokens, so if I add another token it should retrieve all records for token 1 and then continue to token 2, etc... So the only thing remaining is loop through each token but making sure it retrieve all records for each token.

Below is my job:

Tjava_1 code:

globalMap.put("page",new Integer("1"));

globalMap.put("enddata", new Boolean(true));

Tloop_1:

tFixedFlowInput_1, here is where I put all different tokens but like I explained above if I use 1 it works perfect but if I put more than 1 it doesn't work:

tJavaRow_1:

globalMap.put("Token", new String(row1.Token));

globalMap.put("Division", new String(row1.Division));

System.out.println("Token: " + globalMap.get("Token"));

System.out.println("Division: " + globalMap.get("Division"));

tFlowToIterate_2:

tREST_2:

tExtractJSONFields_6:

tJavaRow_6:

System.out.println(row21.count);

globalMap.put("page",(Integer)globalMap.get("page")+1);

if(row21.count == null || row21.count == 0)

{

globalMap.put("enddata", new Boolean(false));

}

This is the result I get when I use 1 token in the tFixedFlowInput_1, which works as expected:

This is the result if I add 2 or more tokens, which is what I want to resolve. Instead of doing this it should loop all pages for token 1 until there are no records and then continue with token 2 and so on...

Any help will be greatly appreciated!!

Re: Loop through different api calls with pagination

Anonymous — Tue, 08 Nov 2022 11:01:25 GMT

OK, you are practically there with this, but I think you need to make a change to how you deal with your tokens.

If you have a tLoop iterating over the pagination, that should be done for each token to completion before switching token. For this, I think you should set up a tFixedFlowInput (with your token data), connect that to a tFlowToIterate and connect that to your tJava_1 which initiates your variables for the tLoop. What this will do is cycle through your tokens and for each token it will run the tLoop to completion.

As a quick example of what I meant, see below...

Re: Loop through different api calls with pagination

MBI1667841504 — Tue, 08 Nov 2022 13:44:10 GMT

This approach is working.

Thank you!

topic Re: Loop through different api calls with pagination in Talend Studio

Loop through different api calls with pagination

Re: Loop through different api calls with pagination

Re: Loop through different api calls with pagination