[resolved] How to use tRest (JSON URL) in a loop?

Six Stars

[resolved] How to use tRest (JSON URL) in a loop?

I've the following job wich works ok: tREST_1 > tXMLMap_1 > tExtractJSONFields_1 > tLog
The problem I'm facing is that the tRest URL will return a body in which 100 records are returned (in the data section) but also a new URL (in the pagination section) to retrieve the next 100 records (if resultset contains more then 100 records). How can I loop this so I retrieve all the records I'm looking for?
The order of the job should be:

retrieve data of first tRest (tExtractJSONFields Loop Jsonpath query: "$.data
"
extract the fields
evaluate if body contains column "next" in the pagination section (tExtractJSONFields Loop Jsonpath query: "$.pagination"
if true load this URL and retrieve the next 100 records
loop while pagination section contains column "next"

Thnx Remco

Accepted Solutions
Community Manager

Re: [resolved] How to use tRest (JSON URL) in a loop?

This isn't exactly what you want to do, but it is close enough. I have written a tutorial on getting spotify data from Facebook. This requires pagination and looping through REST calls. The tutorial is here. The section you want starts with the title "The GetMySpotifyListeningHistory Job". 

All Replies
Community Manager

Re: [resolved] How to use tRest (JSON URL) in a loop?

This isn't exactly what you want to do, but it is close enough. I have written a tutorial on getting spotify data from Facebook. This requires pagination and looping through REST calls. The tutorial is here. The section you want starts with the title "The GetMySpotifyListeningHistory Job". 
Six Stars

Re: [resolved] How to use tRest (JSON URL) in a loop?

Thnx for your reply, your tutorial guided me to the right direction!!
Five Stars

Re: [resolved] How to use tRest (JSON URL) in a loop?

Page unable to open.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now