Extract Hubspot API JSON data using tRESTClient and tLoop
Hello,
I have another thread open for this but I'm still looking for a solution.
I need to extract data from a Hubspot API using tRESTClient -> tMap -> tExtractJSONFields -> tAmazonAuroraOutput. This part I have working well. This challenge I'm trying to solve now is how to iterate through API calls based on data from the API prior call. There are 2 fields in the JSON output that can be used to determine if there are more records (API limits calls to 100 records): An 'offset' that indicates where the next call should start and 'hasmore' that indicates if there are more records available (true/false).
I believe I need 2 jobs (parent & child). In the parent job initiate a loop (tLoop) and iterate over the subjob, which calls the API and extracts the JSON data, until in the subjob it encounters the hasmore = 'false'.
I have tried a variety of different configurations and it seems to all point to setting a gloabalMap variable that contains a java.util.concurrent.ConcurrentHashMap() and being able to modify that variable in the subjob and pass back to the parent to decide if another iteration should be fired off (hasmore = 'true) or stop processing (hasmore = 'false').
If anyone has done anything similar and can provide some guidance I would greatly appreciate it. It obviously doesn't need to be a Hubspot APi...just and API that needs to be called multiple times based on data from the API.
