How to do paginaton for Rest api

Highlighted
Five Stars

How to do paginaton for Rest api

Hey Guys,
My requiremnt is I need to get Data from Rest api per pages and one page contain 250 records. I know I need to also iteration for this. But I dont how store value in variable and make counter. Can someone help on this.

Any help much appricated.
Community Manager

Re: How to do paginaton for Rest api

Essentially you need to follow this process.....

 

1) Create a globalMap variable to hold your webservice call URL. Do this in a tJava.

2) Connect the tJava to a tLoop using an OnSubJobOk link.

3) Set the loop to be based on the presence of a URL value in globalMap. While globalMap.get("url")!=null, for example.

4) Connect your tLoop to whatever web service component you want to use.

5) After the webservice call, use a tExtractXMLField component to retrieve your data and the pagination URL. Put the pagination URL into the globalMap variable. If there is no pagination URL (because it is the last page), set it to null.

 

This is a high level explanation of how to achieve this with A LOT of assumptions as to how your web service works. You should be able to tailor this to your web service.

Six Stars

Re: How to do paginaton for Rest api

 
Six Stars

Re: How to do paginaton for Rest api

 
Community Manager

Re: How to do paginaton for Rest api

How many records the page could have? Is it always in a range such as 1-500? or uncertain?
----------------------------------------------------------
Talend | Data Agility for Modern Business
Six Stars

Re: How to do paginaton for Rest api

The page has always 250 records, Its fix.

Six Stars

Re: How to do paginaton for Rest api

 
Six Stars

Re: How to do paginaton for Rest api

 
Six Stars

Re: How to do paginaton for Rest api

 
Community Manager

Re: How to do paginaton for Rest api

The step by step method I suggested will work. You just need to change step 5 to calculate how many records you have received and use that to change your URL or stop the tLoop. 

Community Manager

Re: How to do paginaton for Rest api

I have knocked up an example job. This does not use a real REST service, because I couldn't find one suitable (for pagination), but it should give you an idea. Look at the code in the tJava components (it is described there) and look at the tLoop config. You will have to work some of the logic out for yourself with your service (working out how many records are returned, how many are left, etc), but the logic shown in this example should easily be able to be extrapolated from. 

 

The example was written in Talend v6.5.1. If you are using a younger version, download anything above 6.5.1 and you can look at it there.

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog