Handling Pagination in tHttprequest GET method

Highlighted
Six Stars

Handling Pagination in tHttprequest GET method

I am using tHttpRequest GET method to get a json file from Kinvey. But there seems to be some restriction limit of 1k (or 10k) records that can only be processed using GET Method.
May I know what is the alternative way to extract the records without having this restriction. Also, I heard a concept about pagination; May I know how can this pagination be implemented in thttp request component in Talend.

Highlighted
Employee

Re: Handling Pagination in tHttprequest GET method

The limitation will be with the Kinvey API.  You need to check with them.  The same for the pagination. You will need to figure how to provide the parameters to your GET Method request.  Sometimes the API provider may have a bulk API too to get high volume of data.

 

If they offer a Rest API, then you can use tRestClient component which has all the operations for Rest.  

Highlighted
Seventeen Stars

Re: Handling Pagination in tHttprequest GET method

There is no build-in pagination method in RESTful web services. Every service have to establish it individually.

Typically such methods like offset and limit as request query parameters  are used to do this. 

In all cases the client and the server have to set it up for the service implementation.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog