topic Re: Loop within a loop in Talend Studio

Loop within a loop

MattE — Fri, 15 Nov 2024 23:04:33 GMT

Hi, i have the requirement where i have to build and execute a number of tREST API calls, the calls are built dynamically in SQL and the number of them could change. However, the API I'm using also uses pagination so for each API call i also have to loop and call it multiple times incrementing the page number until there are no results left.

So the scenario is

Build call www.apicall1.com

loop

call page 1 www.apicall1.com&page1

call page 2 www.apicall1.com&page2

.......

Build call www.apicall2.com

loop

call page 1 www.apicall2.com&page1

call page 2 www.apicall2.com&page2

.......

and so on

I can't figure a way to do this with Talend Studio's components as tLoop has to be a start component but i essentially have to start a tLoop for each call.

Any one any ideas without resorting to Java as i'd like to make this easier to support with native Talend components?

The attached is what I've tried but because tLoop is part of the subjob it is only called once which is not the behaviour i need.

Thanks

Re: Loop within a loop

gjeremy1617088143 — Tue, 29 Mar 2022 09:25:05 GMT

Hi does the first call return the number of page ?

if yes you could do something like this :

first call send the value of page to a globalVar

then use a tRowgenerator with one field :

name "pageNumber"

as type String

value : String.valueOf(Numeric.sequence("s1",1,1))

in the row number field you call your globalMap :

you rely this to a tflowToiterate then you iterate on your tRest component for the loop call

and in the url you add the globalvar (String)globalMap.get("row1.pageNumber")

here row1 is the row linked to the tFlowToIterate

Re: Loop within a loop

MattE — Tue, 29 Mar 2022 09:41:53 GMT

Hi @guenneguez jeremy thanks that would work for the pagination on one call but the issue is that once i have looped through the pages of that first call i then need to build and loop a second and maybe 3rd of 4th call and loop through their pages as well. Essentially there would be an outer loop which is where my issue lies. The ultimate goal is to combine all these into a single file.

Re: Loop within a loop

gjeremy1617088143 — Tue, 29 Mar 2022 09:47:07 GMT

maybe try to separate loop in distinct job for a clearer vision

Re: Loop within a loop

Anonymous — Tue, 29 Mar 2022 10:03:47 GMT

yes, I agree with @guenneguez jeremy , loop different API calls in the main job, and do the page loop for current API in child job.

Re: Loop within a loop

MattE — Tue, 29 Mar 2022 12:35:39 GMT

Ok so this can't be done in one job. Thanks

Re: Loop within a loop

gjeremy1617088143 — Tue, 29 Mar 2022 12:52:17 GMT

it can be done in one job , but definitively i suggest you to partition this in smaller jobs to keep control of each part.