topic How to loop a trest job? in Talend Studio

How to loop a trest job?

rp2018 — Fri, 15 Nov 2024 21:27:08 GMT

We have a limitation to extract 3000 records per call from this site and we need to implement a loop job for it.

We've tried to create a job but without any success. Can someone point me where we should add this to loop 3000 records till all of the records from the site are extracted.

NOTE: We are passing start and count variables in the HTTP Body.

"json={\"start\": "+context.start+", \"count\" :"+context.count+"}"

start and count are the parameters required by the site.

start = which record to start from

count = total # of records to extract

This is the Entire Job design:

Re: How to loop a trest job?

Anonymous — Mon, 28 Aug 2023 03:06:46 GMT

@Ranjan Patel , if I understand your requirement well, you should increase the value of context.start for next API calling, and exit the loop if there are no records are extracted. To to, add a tJava component after tJavaFlex_2,

...tJavaFlex_2--OnComponentOK--tJava

and add these code:

context.start=context.start+3000;

if((Integer)globalMap.get("tExtractJSONFields_3_NB_LINE")==null){

globalMap.put("V_LOOP", false);

}

//move these code from tJavaFlex_1 to tJava component)

Please try and let me know if it works.

Regards

Shong