Accessing the extacted JSON Values with java to paginate REST call/end Loop

Highlighted
Five Stars

Accessing the extacted JSON Values with java to paginate REST call/end Loop

Hi There,

 

my first real day with Talend Opern Studio, so please be patient with me :-)

 

The Task:

Call a REST API with three parameters: starte_date and end_date (current_date) as well as an id for requesting the next rows if there are more than 500. The API will only give me max. 500 datasets as a JSON, so i should pass the id of the last row for the next call. (it will then give me the rows starting with the next id after the provided one).

At the end i want to add all received rows to a mysql table.

 

What i have so far:

JSON:

tos_json

 

 

Design:

tos_design

tJava_1: i set the start_date and end_date parameters to the todays date via Java code, resulting in e.g. "2018-12-10" for this day. I also want to set two values  id and continueLoop. The id variable to be set to the last received id to be used with the next call (loop).

globalMap.put("start_date",currentDate);
globalMap.put("end_date",currentDate);
globalMap.put("id","0");
globalMap.put("continueLoop",true);

 

 

tRESTClient1: The API Call with the three query parameters

 tos_rest_client

 

 

tExtractJsonFields: To map the results (see screenshot). Im unsure if the count value can be received this way (a level outside the given "Loop Jsonpath query")

 toc_extrac_json

 

 

tJavaFlex_1: 

This is where i'm currently lost. The plan was to store the id of the last received dataset for a possible next call and to set the "continueLoop" variable to false if the count value is unequal to 500.

Main Code:

// here is the main part of the component,
// a piece of code executed in the row
// loop
globalMap.put("id",row3.id);
System.out.println(row3.count);
System.out.println(row3.id);

End Code:

// end of the component, outside/closing the loop
 System.out.println("Last Row"+ row3.id);
 if(!row3.count.equals("500")) globalMap.put("continueLoop",false);

 

Currently i receive the error shown in the corresponding screenshot.

 toc_error

 

I appreciate any help from you guys; Thanks in advance for your time!

Five Stars

Re: Accessing the extacted JSON Values with java to paginate REST call/end Loop

A short update on this. I managed to solve the error. It was caused by the Mapping of the "count" variable in tExtractJSONFields_1.

So i know did manage to receive all the rows within a loop.

 

Now: How can i get the rows of data into a Database (e.g. MySQL or BigQuery?).

Thanks again

 

 

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Downloads and Trials

Test drive Talend's enterprise products.

Downloads