Multiple paginated APIs in one job

Highlighted
Six Stars

Multiple paginated APIs in one job

I have a job in mind that I'm not sure how to execute. I need to call two separate REST APIs then join their outputs before loading to my database. The problem is that both of these API's only return a few records per page and need a page parameter for subsequent calls. When I have done similar jobs with only one API call, I have used a flow meter to tell when I have reached the last page. Is it possible to have two separate pairs of tFlowMeter and tFlowMeterCatcher in the same job, without interfering with one another? If not, how should I accomplish this?


Accepted Solutions
Highlighted
Eight Stars

Re: Multiple paginated APIs in one job

Maybe try creating two sub jobs and calling them from one parent job?
OR
use tIterate component after fetching total records and then call the remaining records?

Never worked with tFlowMeter

View solution in original post


All Replies
Highlighted
Eight Stars

Re: Multiple paginated APIs in one job

Maybe try creating two sub jobs and calling them from one parent job?
OR
use tIterate component after fetching total records and then call the remaining records?

Never worked with tFlowMeter

View solution in original post

Highlighted
Six Stars

Re: Multiple paginated APIs in one job

I used your idea of making separate child jobs, and it worked wonderfully.

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog