Extract Hubspot API JSON data using tRESTClient and tLoop

Six Stars

Extract Hubspot API JSON data using tRESTClient and tLoop

Hello,

I have another thread open for this but I'm still looking for a solution.

I need to extract data from a Hubspot API using tRESTClient -> tMap -> tExtractJSONFields -> tAmazonAuroraOutput. This part I have working well. This challenge I'm trying to solve now is how to iterate through API calls based on data from the API prior call. There are 2 fields in the JSON output that can be used to determine if there are more records (API limits calls to 100 records): An 'offset' that indicates where the next call should start and 'hasmore' that indicates if there are more records available (true/false). 

I believe I need 2 jobs (parent & child). In the parent job initiate a loop (tLoop) and iterate over the subjob, which calls the API and extracts the JSON data, until in the subjob it encounters the hasmore = 'false'.

I have tried a variety of different configurations and it seems to all point to setting a gloabalMap variable that contains a java.util.concurrent.ConcurrentHashMap() and being able to modify that variable in the subjob and pass back to the parent to decide if another iteration should be fired off (hasmore = 'true) or stop processing (hasmore = 'false').

If anyone has done anything similar and can provide some guidance I would greatly appreciate it. It obviously doesn't need to be a Hubspot APi...just and API that needs to be called multiple times based on data from the API.

Here is an example of the demo data on Hubspot:

https://api.hubapi.com/contacts/v1/lists/all/contacts/all?hapikey=demo

Thanks

Moderator

Re: Extract Hubspot API JSON data using tRESTClient and tLoop

Hello,

Could you please post your current work flow screenshots on forum which will be helpful for us to understand your job requirement?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch