One Star

how to impliment pagination while extracting data form API

I need to implement pagination in relation to API, as on a given URL there is Offset and it will return only those no of records and i want to fetch complete Records.
2 REPLIES
Seventeen Stars

Re: how to impliment pagination while extracting data form API

At first you have to know when the end of your data realm is reached (probably if you get less then the maximum amount of data sets).
Next if you do the next request you need to know how to setup the request to get the next bunch of data.
In every job run you must keep the amount of data you have got or the last index row you got and start in every request with the next one.
In Talend we do not have a build-in mechanism to support that, it is almost an individual solution.
You could take a look at the components tJobInstanceStart+ tJobInstanceEnd + tJobDataRangeScanner.
Theses components from Talend exchange provides the capabilities to keep metadata for every job run and to get key figures from the previous run. It is well documented.
http://www.talendforge.org/exchange/index.php?eid=987&product=tos&action=view&nav=1,1,1
One Star

Re: how to impliment pagination while extracting data form API

Hi,
What i want is when i hit the api it should return me complete results and not the limited set of result as there is a variable named as limit in api which is set to 100 , it's because of that i get 100 reords at a time. i need to pass that variable dynamically and set it's value dynamically, the challenge here is i don't know the no of records in API. and i need to iterate all of them