topic How can I read multiple pages from Harvest API in Talend Studio

How can I read multiple pages from Harvest API

MarkM1 — Thu, 20 Jul 2023 18:59:52 GMT

The Harvest API is a REST api. The results of a GET is a page of up to 2000 rows of data, plus some paging information including the total number of pages available, and a URL to retrieve the next page of data. That URL is null if there is not a next page. In order to use this API, I somehow have to feed that URL in the output schema back to the input of the tRest component so it can retrieve the next page. So how do I do that?

I can get the first page with tRest, and extract the URL for the next page with tExtractJSONFields, but I can't determine how to tell Talend to get the next page. I don't know how many pages of data there are until I have read the first page.

Here is the link to Harvest API documentation: https://help.getharvest.com/api-v2/

Re: How can I read multiple pages from Harvest API

anselmopeixoto — Thu, 20 Jul 2023 19:38:35 GMT

Hello @Mark Murphy

I suggest you use a tLoop component with "While" loop type.

Here's an brief description of how to do it:

First, declare a global variable using tSetGlobalVar, let's say you call it "total_pages" and start it with 1 value.

On tLoop component condition write i < ((Integer)globalMap.get("total_pages")) , this way you'll guarantee that the loop will iterate at least once.

Connect tLoop to tRESTClient and map the API response total number of pages available to another tSetGlobalVar or a tJavaRow so you can update the "total_pages" variable on each execution.

This is just a brief example, you'll probably need to add other conditions o tLoop component to control the job flow.