From Thursday, July, 9, 3:00 PM Pacific,
our Community site will be in
read-only mode
through Sunday, July 12th.
Thank you for your patience.

How to do paginaton for Rest api

Highlighted
Five Stars

How to do paginaton for Rest api

Hey Guys,
My requiremnt is I need to get Data from Rest api per pages and one page contain 250 records. I know I need to also iteration for this. But I dont how store value in variable and make counter. Can someone help on this.

Any help much appricated.
Highlighted
Community Manager

Re: How to do paginaton for Rest api

Essentially you need to follow this process.....

 

1) Create a globalMap variable to hold your webservice call URL. Do this in a tJava.

2) Connect the tJava to a tLoop using an OnSubJobOk link.

3) Set the loop to be based on the presence of a URL value in globalMap. While globalMap.get("url")!=null, for example.

4) Connect your tLoop to whatever web service component you want to use.

5) After the webservice call, use a tExtractXMLField component to retrieve your data and the pagination URL. Put the pagination URL into the globalMap variable. If there is no pagination URL (because it is the last page), set it to null.

 

This is a high level explanation of how to achieve this with A LOT of assumptions as to how your web service works. You should be able to tailor this to your web service.

Highlighted
Six Stars

Re: How to do paginaton for Rest api

 
Highlighted
Six Stars

Re: How to do paginaton for Rest api

 
Highlighted
Community Manager

Re: How to do paginaton for Rest api

How many records the page could have? Is it always in a range such as 1-500? or uncertain?
----------------------------------------------------------
Talend | Data Agility for Modern Business
Highlighted
Six Stars

Re: How to do paginaton for Rest api

The page has always 250 records, Its fix.

Highlighted
Six Stars

Re: How to do paginaton for Rest api

 
Highlighted
Six Stars

Re: How to do paginaton for Rest api

 
Highlighted
Six Stars

Re: How to do paginaton for Rest api

 
Highlighted
Community Manager

Re: How to do paginaton for Rest api

The step by step method I suggested will work. You just need to change step 5 to calculate how many records you have received and use that to change your URL or stop the tLoop. 

Highlighted
Community Manager

Re: How to do paginaton for Rest api

I have knocked up an example job. This does not use a real REST service, because I couldn't find one suitable (for pagination), but it should give you an idea. Look at the code in the tJava components (it is described there) and look at the tLoop config. You will have to work some of the logic out for yourself with your service (working out how many records are returned, how many are left, etc), but the logic shown in this example should easily be able to be extrapolated from. 

 

The example was written in Talend v6.5.1. If you are using a younger version, download anything above 6.5.1 and you can look at it there.

Five Stars

Re: How to do paginaton for Rest api

Hi,

I am doing the same thing in one job, but when i am extracting json data from tExtractJson using Xpath and there was no response from the RestAPI link, it should extract null values, and my loop should end after that. Please see the Below image - 

 

JsonExtract.PNG

 

This is how i am extracting values. The response from postman is below :- 

 

Postman_Response.PNG

And ModifyURL( tJavaflex) component i am modifying my URL and storing the offset value coming from tExtractJson. offset value is used to end the loop.

 

ModifyURL.PNG

 

But as i am not getting any values, offset should be null and loop should end. But in this case loop is not getting end. It's keep on iterating, i think because V_loop value is not getting set as false.

 

Please see the below response in tlogRow : -

 

logRowValues.PNG

tlogRow_6 is the values i am getting after tExtractJson.

 

Setting_loop.PNG

This is how i am setting my loop.

 

Also i am not able to see, my tJavaflex println values on the console. I don't know why, please help me in this.

 

Thanks,

Abhishek

Highlighted
Community Manager

Re: How to do paginaton for Rest api

Why do you have this code....

if(context.offset==null){
......

....it looks like it should be more like....

if(row18.offset==null){
.....

This might not be the only issue. If your code compiles and runs, it means that you have a context variable called "offset". What is that supposed to do and where is it set?

Highlighted
Four Stars

Re: How to do paginaton for Rest api

Hhi @rhall_2_0 , 

 

Your job has been very useful to me and I've tried to emulate it with a real REST API: 

 

Captura de pantalla 2020-06-18 a las 13.57.56.png

However, I'm facing with a strange error:

 

In my tJava_4 component I've configured the URL and the initial condition ("records") to get into the loop:

 Captura de pantalla 2020-06-18 a las 14.05.04.png

In tLoop_2 I set the condition to stay in the loop: 

Captura de pantalla 2020-06-18 a las 14.07.50.png

 

Next, I set the REST API configuration:

 Captura de pantalla 2020-06-18 a las 14.13.06.png

 

In tJava_5, at last, I'm doing some debugging: 

Captura de pantalla 2020-06-18 a las 14.17.47.png

 

 

As you can see, I've commented the last condition because it is causing an error, because row14.total is null in the first iteration. I don't know why this is happening, but when I execute the whole job I realize that this is the real behaviour: 

Captura de pantalla 2020-06-18 a las 14.19.49 1.png

 

As you can see, in the first iteration the Body returned is null, and in the second iteration I start receiving data. I've also checked the URL in the first iteration with Postman and, however, I see results: 

Captura de pantalla 2020-06-18 a las 14.22.30.png

 

So, am I not taking something into account when designing this job? In the first iteration, when I get null value for row14.total, Talend generates an error and the job breaks. I don't understand why I'm starting to get data in the second iteration instead of the first one.

Highlighted
Community Manager

Re: How to do paginaton for Rest api

I *think* that this might be caused by you using the tJava and not a tJavaFlex. The tJava is not intended to receive or send out "rows". If you take a look at your second service call and look at the page index, it says 1. I *think* this might be the response from the first the call. If you replace the tJava with a tJavaFlex and configure it's schema appropriately, you can use the same code. 

 

Another thing to point out, in the code you commented out it looks like you are referring to section of the JSON body. Before you can make use of the "total" value, you will need to extract it from the JSON using a tExtractJSONFields component.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog