tRest response with pages

[Anonymous]

Hi Talend Community !

I'm using a tRest to communicate with an API. It returns me a JSON response like that : 

{
"response": {

                  "infos": {
                  "nbperpage": "5000",
                  "nbpages": 2,
                  "pagenum": "2",
                  "nbtotal": "5539"
                   },
"result": {
              "876603": {
                               "id": "876603",
                               "corpid": "51634",

...etc

The number limit for the field nbperpage is 5000.

Actually i'm using 2 tRest with 5000 nbperpage. It's working because I have 5539 elements.

But i would like to automate this, with some kind of loop, that send as much request with a tRest to the API as long as there are still elements to get.

If you need more information, ask me

[Anonymous]

There is a very simple way of achieving this using a tLoop. You've not quite given enough information as I assume that your URL will hold the page information in it. But I am assuming you can extrapolate from what I will describe.

Essentially what you need to do is create a globalMap variable which holds the next page to consume before your tLoop. Then base your tLoop logic on the existence of this value. After the tLoop, prepare your URL (maybe in a tJava) using the globalMap variable and use that in your tRest. After the tRest, analyse the returned value and update your globalMap variable with either the next page or null (if no next page exists). When the tLoop tries to fire again, it will check the globalMap variable. If it is null, the loop will end. If it is a number, the process will repeat.

[Anonymous]

I'm sorry i didn't understand well what you mean.

More details on my job :

The api i'm using requires an Oauth authentification, so i'm building the authentification header in the tJava before the tRest(context.url).

Then i write in the post body which method i want to call (here ->Timetracking.getList) with some parameters to filter("search" json object").

And as you can see there is also a pagination json object where i can tell how many elements i want per page (nbperpage) and which page number.

5000 is actually the maximum amount of elements per page.

I tried to do what you said (around)  :

I created a globalVar with the nbtotal after the first tRest to know how many request more i need to do but it wasn't working as i expected.

" If it is null, the loop will end. If it is a number, the process will repeat."

Like a flag ? 

[Anonymous]

It looks like you are almost there. Your tJava generating your authentication header can be used after your tLoop component. You will want to change the code that sets the Page No. This should be your globalMap variable. In the tLoop, you will want to use a while loop. Set the condition to check whether your globalMap variable is not null. Essentially setting a "while is not null" loop. The only bit left is to figure out whether another page is required after your service has run. You will need to see how your API indicates this. But once you know this status, you simply need to set the globalMap to the next page number or null.

[Anonymous]

I understand what you'r saying but the problem is to know how many pages are left and this information we get it after calling one time the API. 

And if i put the tLoop just before the tJava. The latter will loop but not my tRest, am i wrong ?

What i tough of is to do a while with something like :

int pageNo=1;pageNo<=nbPages;i++

But as i said, we only have access to nbpages after calling the api once.

I'm sorry i'm having difficulties to exactly understand what you want me to do.

Thank you in advance,

[Anonymous · Accepted solution]

The flow is like this.....

1. tJava ----> 2. tLoop ----> 3. tJava -----> 4. tRest ----->5. {result analysis} ----> 6. tJava

1. tJava

This component sets the initial value of the globalMap variable. I assume for the first call it will be 1.

2. tLoop

This component will check the value of the globalMap and if "true", will fire the tJava that follows.

3. tJava

This component prepares the URL using the globalMap variable.

4. tRest

This component is called and retrieves the result of the URL

5. {result analysis}

Process the data and see if another call is required

6. tJava

if another page is required, set the globalMap to the next value. If not, set it to null.

2. tLoop

The tLoop is called again to test the WHERE clause. If "true" steps 3,4,5 and 6 are called again. If "false", the next part of the job is started.

Regarding your globalMap and your tLoop condition. If you set your globalMap as below....

globalMap.put("page", 1);

Then your WHERE clause condition can be as below...

globalMap.get("page")!=null

FYI you do not need to worry about the declaration or iteration fields for the WHERE clause if you condition does not use the declared value. Just leave them as defaults.