Extract Hubspot API JSON data using tRESTClient and tLoop

Anonymous — Thu, 02 Nov 2017 21:18:33 GMT

Hello,

I have another thread open for this but I'm still looking for a solution.

I need to extract data from a Hubspot API using tRESTClient -> tMap -> tExtractJSONFields -> tAmazonAuroraOutput. This part I have working well. This challenge I'm trying to solve now is how to iterate through API calls based on data from the API prior call. There are 2 fields in the JSON output that can be used to determine if there are more records (API limits calls to 100 records): An 'offset' that indicates where the next call should start and 'hasmore' that indicates if there are more records available (true/false).

I believe I need 2 jobs (parent & child). In the parent job initiate a loop (tLoop) and iterate over the subjob, which calls the API and extracts the JSON data, until in the subjob it encounters the hasmore = 'false'.

I have tried a variety of different configurations and it seems to all point to setting a gloabalMap variable that contains a java.util.concurrent.ConcurrentHashMap() and being able to modify that variable in the subjob and pass back to the parent to decide if another iteration should be fired off (hasmore = 'true) or stop processing (hasmore = 'false').

If anyone has done anything similar and can provide some guidance I would greatly appreciate it. It obviously doesn't need to be a Hubspot APi...just and API that needs to be called multiple times based on data from the API.

Here is an example of the demo data on Hubspot:

https://api.hubapi.com/contacts/v1/lists/all/contacts/all?hapikey=demo

Thanks

Re: Extract Hubspot API JSON data using tRESTClient and tLoop

Anonymous — Fri, 10 Nov 2017 07:47:04 GMT

Hello,

Could you please post your current work flow screenshots on forum which will be helpful for us to understand your job requirement?

Best regards

Sabrina

topic Extract Hubspot API JSON data using tRESTClient and tLoop in Talend Studio

Extract Hubspot API JSON data using tRESTClient and tLoop

Re: Extract Hubspot API JSON data using tRESTClient and tLoop