Pagination on tHTTPClient

[ebacon]

Hi,

I have an API that I'm trying to load via the component tHttpClient. I've got it so that the call I need is working, but I need to cycle through pages to load the full data set as the call is limited to 100 entries per page. For this I'm attempting to use the Pagination option on Advanced Settings, but I'm struggling to see how to work with this.

The call I have uses Offset and Limit  (e.g. https://-url-here-/search?limit=100&offset=0), and the start of the return looks like the below image. I've tried the Box pre-set option on Pagination with the offset and limit set to 100 and 0 respectively, but I am only getting errors returned.

My question is - has anyone successfully used Pagination on an output that looks like the above (using the "next" output)?

Separately, if Pagination isn't possible for this type of API page-functionality, does anyone have a good method of looping through the results of the tHttpClient, as a tLoop component won't link to httpClient, and I'm having to use tJavaFlex as an interrim component to loop through. It'll probably work, but just wondering if anyone has encountered this issue before!

Thanks, Ellie

[Denis_Segard]

Hello,

What error do you get ?

Do the HTTP request returns an array ?

https://help.qlik.com/talend/en-US/components/8.0/http/thttpclient
Path to elements: Set the path to the JSON array that contains the desired elements. It must be a JSON array.

Kind regards
Denis

[ebacon]

Hi Denis,

Thanks for getting back to me. I've tried a few things so didn't put the error in - below are the two attempts I think I was closest with!

For info, this is how I'm getting my data from the API. Not sure whether that means it's returning an array or not, but it's definitely json!

Attempt 1 - Error: org.talend.sdk.component.api.exception.ComponentException: Exception raised during HTTP call: (class java.lang.ClassCastException) Cannot cast org.apache.johnzon.core.JsonObjectImpl to javax.json.JsonArray'

Input into pagination:

Attempt 2 - Error: org.talend.sdk.component.api.exception.ComponentException: Exception raised during HTTP call: (class java.lang.NullPointerException) Cannot invoke "javax.json.JsonObject.getJsonArray(String)" because "current" is null

Input into pagination:

Thanks, Ellie

[Denis_Segard]

Hello,

The error:
Cannot cast org.apache.johnzon.core.JsonObjectImpl to javax.json.JsonArray'
is raised because the Path to Elements is not an array.
The component needs this to be an array in order to compute the number of entries returned.
If the output returned by the API called using tHTTPClient does not contain this array , you'll need to "loop" on the component tHTTPClient :

For example :
tLoop -> tFixedFlowInput -> tHTTPCLient 

Kind regards
Denis

[ebacon]

Thanks Denis - I'm not sure I follow the problem I'm afraid! Is there an example of an output that would work anywhere? I'm struggling to see how the API input I'm using doesn't fit the criteria.

[Denis_Segard]

Hello,
If you have a Talend Cloud account , the following API "works" fine with tHTTPClient with Path to Elements : .items

https://api.eu.cloud.talend.com/processing/executables/tasks/executions

https://talend.qlik.dev/apis/processing/2021-03/#operation_get-available-task-executions

Kind regards
Denis

[ebacon]

Thanks for those links, unfortunately I don't have Talend Cloud so can't access the first one! From what I can see the problem is that our output doesn't have specific offset and limit elements, does that seem correct to you?

Is it just the case that the Pagination isn't built for full page links such as the ones we've got being output in the API we're using?

[Denis_Segard · Accepted solution]

I found a "free" REST API supporting pagination
https://openlibrary.org/search.json?q=title:flammable&offset=0&limit=1
The following settings works fine :

Kind regards
Denis

[ebacon]

Thanks for all your help on this Denis, I was able to use the free API example to make a bit of progress. I now have different issues where the output isn't consistent so I need to add some entries to the JSON normalisation advanced settings, so just need to make the decision around which is simpler - looping manually or adding all the normalisation.

But seems like if these issues didn't exist, the pagination might have worked!

Thanks, Ellie