I have another thread open for this but I'm still looking for a solution.
I need to extract data from a Hubspot API using tRESTClient -> tMap -> tExtractJSONFields -> tAmazonAuroraOutput. This part I have working well. This challenge I'm trying to solve now is how to iterate through API calls based on data from the API prior call. There are 2 fields in the JSON output that can be used to determine if there are more records (API limits calls to 100 records): An 'offset' that indicates where the next call should start and 'hasmore' that indicates if there are more records available (true/false).
I believe I need 2 jobs (parent & child). In the parent job initiate a loop (tLoop) and iterate over the subjob, which calls the API and extracts the JSON data, until in the subjob it encounters the hasmore = 'false'.
I have tried a variety of different configurations and it seems to all point to setting a gloabalMap variable that contains a java.util.concurrent.ConcurrentHashMap() and being able to modify that variable in the subjob and pass back to the parent to decide if another iteration should be fired off (hasmore = 'true) or stop processing (hasmore = 'false').
If anyone has done anything similar and can provide some guidance I would greatly appreciate it. It obviously doesn't need to be a Hubspot APi...just and API that needs to be called multiple times based on data from the API.
Here is an example of the demo data on Hubspot:
Could you please post your current work flow screenshots on forum which will be helpful for us to understand your job requirement?
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.