Iterative Data extraction (Pagination and Polling) from REST API using tRestClient

[Gourav_King_of_DataLand]

Hi All,

Need you expert help.

The requirement is to pull all chat data from REST API (one time full data dump) and then pull chat on daily basis.The output is spread across 180K pages with each page giving URL to next and previous (except first page which have only 'nex_url' and last page with have only 'prev_url').

So Far I have been able to use the API/URL to extract information from first page first page

tRestClient ->tJavaRow->tJsonExtract->tOracleOut

How do I modify the job to

1) Pull all data for one time data dump, 180k pages

2) Pull data on daily basis for current day or extract data until the timestamp is current day.

Example output from API

Page 1 gives
{
    "chats": [

                  all chat related attributes that needs to imported

                 ],
    "count": 179451,
    "next_url": "next_url_here"

}

Page2 gives

{
    "chats": [

                all chat related attributes that needs to imported

                 ],
    "count": 179451
    "prev_url": "previous_url_here"
    "next_url": "next_url_here"

}

Page 3 gives ......next page 

[Anonymous]

Glad you got it to work and a good explanation of the steps you took. However, it is good form to award the "Accepted Solution" to those who have provided you with the answer. In this case, to use the tLoop and a globalMap to hold the value of the next url. You can award several "Accepted Solutions" if you feel multiple people have contributed towards it (you may feel that you deserve some credit for writing this up, and I would agree), but it is a little frustrating when you spend time building an example solution only to have it used but not accepted. 

[Gourav_King_of_DataLand]

My apologies rhall_2_0, accepting solution to my post was not intended towards taking any credit. The context was more towards summarizing everything and accepting it as a solution so that it would help others to quickly find the solution rather than reading whole conversation. I wouldn’t have had solution to my problem if you didn’t help. This is the second problem you helped me with consecutively and I am truly greatful to get help and support from an expert like yourself.

[Gourav_King_of_DataLand]

Also I wasn’t aware that multiple answers can selected as accepted solution 😕

[Anonymous]

No problem at all and thanks for sorting it. There are plenty of people who just leave after having their questions answered and it can get a little irritating when all it takes is a click on "Solution Accepted". I happen to use this forum as a method of promoting my business. So "Kudos" and "Accepted Solutions" are of value. Once again, thanks for sorting this and thanks for writing your solution up to demonstrate to others who might have the same question 🙂 

[Anonymous]

Just wanted to thank both of you guys. I was also in a similar problem and after modifying your solution it works for me up to some extent. Will post if there is something i need help with

[Parikhharshal]

Hi @gr44 and @rhall:

I am trying to follow what exactly you had been doing but I seem to stuck at TRestClient where it keeps giving me nullpointerexception Java error.

Not sure if the problem is with this component though as I can see that in Tjava component I had println but nothing is getting printed there.

Is there anything I should be looking at?

In Tjava I got below code:

globalMap.put("VApiUrl",row4.NextUrl);

 System.out.println("Value of Globar Var: "+globalMap.get("VApiUrl"));

In TrestClient I have got config like this:

((String)globalMap.get("VApiUrl"))

Can you please share your job code if possible?

Thanks

Harshal.

[Parikhharshal]

Hi guys

Would really appreciate if you can respond to this or share some more details on your solution please.

Thanks

Harshal.

[Gourav_King_of_DataLand]

Hi @Parikhharshal,

As far as I can see with the screenshot you posted, you have not assigned any value to the global variable. If you are using a global variable in the URL for the tRestClient_1 then make sure that the value is assigned in tSetGlobalVar_1

[Anonymous]

I suspect that @gr44 is correct. The process should be.....

1) Set the first URL in the tSetGlobalVar_1

2) The first iteration of the loop will use the URL set in step 1

3) The JSON returned is then inspected for your data and another URL

4) The tJavaFlex at the end will set a new value to the globalMap variable for the URL

5) The loop goes to the next iteration, back to step 2

[Parikhharshal]

Hi @rhall and @gr44

The way you guys did it, I am following that but not sure what I have missed out.

This is what I have set in global var.

This is what set in tloop.

This is what mentioned in tjava.

This is what is mentioned in trestclient

Am I missing anything or doing anything wrong here?

Guys....As mentioned if you could share screenshot of these values that would be fantastic!

p.s.: You guys are awesome as always and thanks a lot for replying .

Thanks

Harshal.