topic Re: how to impliment pagination while extracting data form API in Talend Studio

how to impliment pagination while extracting data form API

Anonymous — Sat, 03 May 2014 07:27:18 GMT

I need to implement pagination in relation to API, as on a given URL there is Offset and it will return only those no of records and i want to fetch complete Records.

Re: how to impliment pagination while extracting data form API

Anonymous — Sat, 03 May 2014 09:21:09 GMT

At first you have to know when the end of your data realm is reached (probably if you get less then the maximum amount of data sets).
Next if you do the next request you need to know how to setup the request to get the next bunch of data.
In every job run you must keep the amount of data you have got or the last index row you got and start in every request with the next one.
In Talend we do not have a build-in mechanism to support that, it is almost an individual solution.
You could take a look at the components tJobInstanceStart+ tJobInstanceEnd + tJobDataRangeScanner.
Theses components from Talend exchange provides the capabilities to keep metadata for every job run and to get key figures from the previous run. It is well documented.
http://www.talendforge.org/exchange/index.php?eid=987&product=tos&action=view&nav=1,1,1

Re: how to impliment pagination while extracting data form API

Anonymous — Sun, 04 May 2014 10:32:58 GMT

Hi,
What i want is when i hit the api it should return me complete results and not the limited set of result as there is a variable named as limit in api which is set to 100 , it's because of that i get 100 reords at a time. i need to pass that variable dynamically and set it's value dynamically, the challenge here is i don't know the no of records in API. and i need to iterate all of them

Re: how to impliment pagination while extracting data form API

arpiitv — Sun, 29 Jul 2018 13:14:42 GMT

Please Specify more