topic Re: Paginated REST API call and writing to DB in Talend Studio

Paginated REST API call and writing to DB

Anonymous — Sat, 16 Nov 2024 03:58:44 GMT

Hi,

I am doing the pagination while reading data from RestApi , but when i am extracting json data from tExtractJson using Xpath and there was no response from the RestAPI link, it should extract null values, and my loop should end after that. Please see the Below image -

This is how i am extracting values. The response from postman is below :-

And ModifyURL( tJavaflex) component i am modifying my URL and storing the offset value coming from tExtractJson. offset value is used to end the loop.

But as i am not getting any values, offset should be null and loop should end. But in this case loop is not getting end. It's keep on iterating, i think because V_loop value is not getting set as false.

Please see the below response in tlogRow : -

tlogRow_6 is the values i am getting after tExtractJson.

This is how i am setting my loop.

Also i am not able to see, my tJavaflex println values on the console. I don't know why.

Can anyone help me in this!

Thanks,

Abhishek

Re: Paginated REST API call and writing to DB

Anonymous — Fri, 29 Nov 2019 11:25:36 GMT

Hi, I see you have asked this same question on another post. I think it makes sense to continue this discussion in its own thread, so please ignore my other response. I will add it here.....

Why do you have this code....

if(context.offset==null){
......

....it looks like it should be more like....

if(row18.offset==null){
.....

This might not be the only issue. If your code compiles and runs, it means that you have a context variable called "offset". What is that supposed to do and where is it set?

Re: Paginated REST API call and writing to DB

Anonymous — Sat, 30 Nov 2019 13:27:39 GMT

Because i am saving row18.offset value into context.offset and updating offset value with every run.

this is my logic, it is working for other RespAPI because it has some response values. But here as we are not getting any response.

Means Channels are blank and offset value is also. Hence offset value is not going into tJvaflex.

Please check the below image -

I have used logRow to check on the console what is going on. Hence you can see, tlogrow_6 after extracting offset value, ans tlogrow_7 after going into channels.

Re: Paginated REST API call and writing to DB

Anonymous — Mon, 02 Dec 2019 06:34:35 GMT

Hi,

Is this there any solution for the same. As you can see in the below image we are getting zero rows from second tExtractJson, hence the offset is not getting modified in the tjavaflex.

Re: Paginated REST API call and writing to DB

Anonymous — Mon, 02 Dec 2019 07:51:49 GMT

I got the solution, i have put my tjavaflex after first tExtractJson, then using second. Because of this, my offset value is getting updated .

Thanks!

Re: Paginated REST API call and writing to DB

Anonymous — Mon, 02 Dec 2019 09:14:33 GMT

Well done for getting it sorted. Unfortunately I did not log in over the weekend so I did not see your posts.

Re: Paginated REST API call and writing to DB

Anonymous — Wed, 04 Dec 2019 13:04:35 GMT

Hi,

I have another issue in that, in offset value i have to only get data after July 10, 2019. As offset is the unix timestamp i am first converting it into string ( in date format) and need to use in if condition so that my loop gets stop if offset is less than July 10, 2019. But getting compile type error , please see the below image :-

Is this the right way doing, or i am missing something!

Thanks!

Re: Paginated REST API call and writing to DB

Anonymous — Wed, 04 Dec 2019 14:56:43 GMT

Can you raise another post for this please? This, while connected to your problem, is a new question. Somebody else may have this issue not connected to your main issue. It can make answers very difficult to find if solutions get included with unrelated questions.