Multiple paginated APIs in one job

Highlighted
Six Stars

Multiple paginated APIs in one job

I have a job in mind that I'm not sure how to execute. I need to call two separate REST APIs then join their outputs before loading to my database. The problem is that both of these API's only return a few records per page and need a page parameter for subsequent calls. When I have done similar jobs with only one API call, I have used a flow meter to tell when I have reached the last page. Is it possible to have two separate pairs of tFlowMeter and tFlowMeterCatcher in the same job, without interfering with one another? If not, how should I accomplish this?


Accepted Solutions
Eight Stars

Re: Multiple paginated APIs in one job

Maybe try creating two sub jobs and calling them from one parent job?
OR
use tIterate component after fetching total records and then call the remaining records?

Never worked with tFlowMeter

All Replies
Eight Stars

Re: Multiple paginated APIs in one job

Maybe try creating two sub jobs and calling them from one parent job?
OR
use tIterate component after fetching total records and then call the remaining records?

Never worked with tFlowMeter
Six Stars

Re: Multiple paginated APIs in one job

I used your idea of making separate child jobs, and it worked wonderfully.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download