topic Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient in Talend Studio

Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Gourav_King_of_DataLand — Wed, 22 Aug 2018 11:36:36 GMT

Hi All,

Need you expert help.

The requirement is to pull all chat data from REST API (one time full data dump) and then pull chat on daily basis.The output is spread across 180K pages with each page giving URL to next and previous (except first page which have only 'nex_url' and last page with have only 'prev_url').

So Far I have been able to use the API/URL to extract information from first page first page

tRestClient ->tJavaRow->tJsonExtract->tOracleOut

How do I modify the job to

1) Pull all data for one time data dump, 180k pages

2) Pull data on daily basis for current day or extract data until the timestamp is current day.

Example output from API

Page 1 gives
{
"chats": [

all chat related attributes that needs to imported

],
"count": 179451,
"next_url": "next_url_here"

}

Page2 gives

{
"chats": [

all chat related attributes that needs to imported

],
"count": 179451
"prev_url": "previous_url_here"
"next_url": "next_url_here"

}

Page 3 gives ......next page

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Anonymous — Wed, 22 Aug 2018 11:56:28 GMT

My assumption is that the "next_url" element will not be supplied if there are no other pages after that one. If that is the case, you can do it like this.....

1) Set up a tLoop using the "while" loop functionality. Use a globalMap variable holding your initial URL (set in a tJava preceding the tLoop) as your test on your while clause. "While globalMap value is not null" for example.

2) Use the globalMap value in your tRestClient

3) Retrieve your data for each service call and also retrieve the next_url. Set the globalMap value to be that of the next_url. If it is not present, then this will be null.

The tLoop will fire for each url supplied and will stop when the next_url value is not supplied.

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Gourav_King_of_DataLand — Wed, 22 Aug 2018 14:06:25 GMT

I am trying to do what you suggested, I am quite new to Talend hence sometimes it's bit difficult to achieve small and simple things as well.

I used tsetglobal to set the initial URL and then passed it to tRestClient. Then extracting the 'next_url' from tExtractJson and till here things are good. I looked up result in tLogRow and can see the next_url. However I am not able to ficure out how to assign the next_url from tExtractJson to a global variable in tJava.

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Anonymous — Wed, 22 Aug 2018 14:36:56 GMT

You need to have this sort of layout...

You set the initial globalMap in the "Set initial globalMap" tJava. Then set the "Where" clause logic in the tLoop component. The "Dummy" component is just to allow you to link to the tRestClient. I've included the "Modify JSON" tJavaFlex following on from your last question. Then you can set the next url in the "Set globalMap" tJavaFlex.

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Gourav_King_of_DataLand — Wed, 22 Aug 2018 15:55:25 GMT

Hi rhall_2_0, thank you for replying.

I am unable to get the attribute value from tExtractJson to tJava. In this case I am extracting "next_url" in tExtractJson but when I link it to tJava I have no clue how to assign globalMap variable. Below is the code in the tJava, this is giving error 'input_row cannot be resolved'

globalMap.put("next_url",input_row.next_url);
System.out.println("Value Of GlobalVar: "+globalMap.get("next_url"));

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Anonymous — Wed, 22 Aug 2018 16:23:50 GMT

LInk the tExtractJson component to a tJavaFlex not a tJava. The code for the tJavaFlex should be in the Main Code section.

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Gourav_King_of_DataLand — Thu, 23 Aug 2018 07:40:32 GMT

thank you for replying. The below mentioned code in tJavaFlex is giving error

globalMap.put("v_next_url",input_row.next_url);

'input_row cannot be resolved'

tExtractJson to tJavaFlex (what code I have to write in tJAvaFlex to get next_url value from tExtractJson to a global variable in tJavaFlex. The one I somehow manage to find from internet doesn't work 😞 )

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Gourav_King_of_DataLand — Thu, 23 Aug 2018 13:23:07 GMT

resolved it, had to use
globalMap.put("V_API_URL",row2.next_url);
instead of
globalMap.put("v_next_url",input_row.next_url);

row2 is the output from tJsonExtract

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Anonymous — Thu, 23 Aug 2018 19:21:14 GMT

Ah yes, sorry I have been away from my machine all day. The tJavaRow uses the input_row and output_row row names ( for some strange reason) the tJavaFlex uses the actual row names.

Did this work for you?

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Gourav_King_of_DataLand — Fri, 24 Aug 2018 08:46:57 GMT

Yes this did work for me, the final solution looks like this.

Global Variable

"V_API_URL" - Holds initial URL and then is updated with next URL that I get from tExtractJson.

"V_LOOP" - Defined Boolean, hold value true to start with and is set to false when the V_API_URL is null

TLoop

Used While loop, without Declaration and Iteration

tRestClient

tJavaFlex

For each loop Global Variable V_API_URL is set to the next_url got from tJsonExtract

Check for if the V_API_URL has any url or not, if it does the loop continues as the value for the V_LOOP is still true and if there is no url then V_LOOP is set to False. This would exit the loop when it's end page.

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Anonymous — Fri, 24 Aug 2018 14:29:58 GMT

Glad you got it to work and a good explanation of the steps you took. However, it is good form to award the "Accepted Solution" to those who have provided you with the answer. In this case, to use the tLoop and a globalMap to hold the value of the next url. You can award several "Accepted Solutions" if you feel multiple people have contributed towards it (you may feel that you deserve some credit for writing this up, and I would agree), but it is a little frustrating when you spend time building an example solution only to have it used but not accepted.

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Gourav_King_of_DataLand — Fri, 24 Aug 2018 16:07:34 GMT

My apologies rhall_2_0, accepting solution to my post was not intended towards taking any credit. The context was more towards summarizing everything and accepting it as a solution so that it would help others to quickly find the solution rather than reading whole conversation. I wouldn’t have had solution to my problem if you didn’t help. This is the second problem you helped me with consecutively and I am truly greatful to get help and support from an expert like yourself.

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Gourav_King_of_DataLand — Fri, 24 Aug 2018 16:14:47 GMT

Also I wasn’t aware that multiple answers can selected as accepted solution 😕

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Anonymous — Fri, 24 Aug 2018 16:46:51 GMT

No problem at all and thanks for sorting it. There are plenty of people who just leave after having their questions answered and it can get a little irritating when all it takes is a click on "Solution Accepted". I happen to use this forum as a method of promoting my business. So "Kudos" and "Accepted Solutions" are of value. Once again, thanks for sorting this and thanks for writing your solution up to demonstrate to others who might have the same question 🙂

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Anonymous — Fri, 19 Oct 2018 22:38:52 GMT

Just wanted to thank both of you guys. I was also in a similar problem and after modifying your solution it works for me up to some extent. Will post if there is something i need help with

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Parikhharshal — Thu, 01 Nov 2018 22:35:40 GMT

Hi @gr44 and @rhall:

I am trying to follow what exactly you had been doing but I seem to stuck at TRestClient where it keeps giving me nullpointerexception Java error.

Not sure if the problem is with this component though as I can see that in Tjava component I had println but nothing is getting printed there.

Is there anything I should be looking at?

In Tjava I got below code:

globalMap.put("VApiUrl",row4.NextUrl);

System.out.println("Value of Globar Var: "+globalMap.get("VApiUrl"));

In TrestClient I have got config like this:

((String)globalMap.get("VApiUrl"))

Can you please share your job code if possible?

Thanks

Harshal.

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Parikhharshal — Fri, 02 Nov 2018 07:12:23 GMT

Hi guys

Would really appreciate if you can respond to this or share some more details on your solution please.

Thanks

Harshal.

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Gourav_King_of_DataLand — Fri, 02 Nov 2018 10:41:30 GMT

Hi @Parikhharshal,

As far as I can see with the screenshot you posted, you have not assigned any value to the global variable. If you are using a global variable in the URL for the tRestClient_1 then make sure that the value is assigned in tSetGlobalVar_1

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Anonymous — Fri, 02 Nov 2018 12:22:32 GMT

I suspect that @gr44 is correct. The process should be.....

1) Set the first URL in the tSetGlobalVar_1

2) The first iteration of the loop will use the URL set in step 1

3) The JSON returned is then inspected for your data and another URL

4) The tJavaFlex at the end will set a new value to the globalMap variable for the URL

5) The loop goes to the next iteration, back to step 2

Re: Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

Parikhharshal — Fri, 02 Nov 2018 12:34:49 GMT

Hi @rhall and @gr44

The way you guys did it, I am following that but not sure what I have missed out.

This is what I have set in global var.

This is what set in tloop.

This is what mentioned in tjava.

This is what is mentioned in trestclient

Am I missing anything or doing anything wrong here?

Guys....As mentioned if you could share screenshot of these values that would be fantastic!

p.s.: You guys are awesome as always and thanks a lot for replying .

Thanks

Harshal.