topic Re: [resolved] How to use tRest (JSON URL) in a loop? in Talend Studio

[resolved] How to use tRest (JSON URL) in a loop?

Remco1 — Fri, 15 Apr 2016 08:31:27 GMT

I've the following job wich works ok: tREST_1 > tXMLMap_1 > tExtractJSONFields_1 > tLog
The problem I'm facing is that the tRest URL will return a body in which 100 records are returned (in the data section) but also a new URL (in the pagination section) to retrieve the next 100 records (if resultset contains more then 100 records). How can I loop this so I retrieve all the records I'm looking for?
The order of the job should be:

retrieve data of first tRest (tExtractJSONFields Loop Jsonpath query: "$.data
"
extract the fields
evaluate if body contains column "next" in the pagination section (tExtractJSONFields Loop Jsonpath query: "$.pagination"
if true load this URL and retrieve the next 100 records
loop while pagination section contains column "next"

Thnx Remco

Re: [resolved] How to use tRest (JSON URL) in a loop?

Anonymous — Fri, 15 Apr 2016 11:01:06 GMT

This isn't exactly what you want to do, but it is close enough. I have written a tutorial on getting spotify data from Facebook. This requires pagination and looping through REST calls. The tutorial is here. The section you want starts with the title "The GetMySpotifyListeningHistory Job".

Re: [resolved] How to use tRest (JSON URL) in a loop?

Remco1 — Mon, 18 Apr 2016 09:52:08 GMT

Thnx for your reply, your tutorial guided me to the right direction!!

Re: [resolved] How to use tRest (JSON URL) in a loop?

arpiitv — Sun, 29 Jul 2018 13:18:22 GMT

Page unable to open.