[resolved] How to use tRest (JSON URL) in a loop?

Highlighted
Six Stars

[resolved] How to use tRest (JSON URL) in a loop?

I've the following job wich works ok: tREST_1 > tXMLMap_1 > tExtractJSONFields_1 > tLog
The problem I'm facing is that the tRest URL will return a body in which 100 records are returned (in the data section) but also a new URL (in the pagination section) to retrieve the next 100 records (if resultset contains more then 100 records). How can I loop this so I retrieve all the records I'm looking for?
The order of the job should be:

retrieve data of first tRest (tExtractJSONFields Loop Jsonpath query: "$.data
"
extract the fields
evaluate if body contains column "next" in the pagination section (tExtractJSONFields Loop Jsonpath query: "$.pagination"
if true load this URL and retrieve the next 100 records
loop while pagination section contains column "next"

Thnx Remco

Accepted Solutions
Community Manager

Re: [resolved] How to use tRest (JSON URL) in a loop?

This isn't exactly what you want to do, but it is close enough. I have written a tutorial on getting spotify data from Facebook. This requires pagination and looping through REST calls. The tutorial is here. The section you want starts with the title "The GetMySpotifyListeningHistory Job". 

All Replies
Community Manager

Re: [resolved] How to use tRest (JSON URL) in a loop?

This isn't exactly what you want to do, but it is close enough. I have written a tutorial on getting spotify data from Facebook. This requires pagination and looping through REST calls. The tutorial is here. The section you want starts with the title "The GetMySpotifyListeningHistory Job". 
Six Stars

Re: [resolved] How to use tRest (JSON URL) in a loop?

Thnx for your reply, your tutorial guided me to the right direction!!
Five Stars

Re: [resolved] How to use tRest (JSON URL) in a loop?

Page unable to open.

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads