I have another thread open for this but I'm still looking for a solution.
I need to extract data from a Hubspot API using tRESTClient -> tMap -> tExtractJSONFields -> tAmazonAuroraOutput. This part I have working well. This challenge I'm trying to solve now is how to iterate through API calls based on data from the API prior call. There are 2 fields in the JSON output that can be used to determine if there are more records (API limits calls to 100 records): An 'offset' that indicates where the next call should start and 'hasmore' that indicates if there are more records available (true/false).
I believe I need 2 jobs (parent & child). In the parent job initiate a loop (tLoop) and iterate over the subjob, which calls the API and extracts the JSON data, until in the subjob it encounters the hasmore = 'false'.
I have tried a variety of different configurations and it seems to all point to setting a gloabalMap variable that contains a java.util.concurrent.ConcurrentHashMap() and being able to modify that variable in the subjob and pass back to the parent to decide if another iteration should be fired off (hasmore = 'true) or stop processing (hasmore = 'false').
If anyone has done anything similar and can provide some guidance I would greatly appreciate it. It obviously doesn't need to be a Hubspot APi...just and API that needs to be called multiple times based on data from the API.
Here is an example of the demo data on Hubspot:
Could you please post your current work flow screenshots on forum which will be helpful for us to understand your job requirement?
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Introduction to Talend Open Studio for Data Integration.