topic Re: Paginated API call and writing to DB in Talend Studio

Paginated API call and writing to DB

Parikhharshal — Sat, 16 Nov 2024 07:21:51 GMT

Hi there

I want to achieve something like this:

1. Read data from API using paginated URL (check all pages and keep reading)

2. Make URL dynamic by reading IDs (https://abc.com/api/v1/courses/<Course_id>/analytics/student_summaries)

3. Write to Redshift DB/File

I know there have been few threads regarding same but I want 2 loops. And would appreciate if someone can share some design code on this.

Urgent response is really appreciated!

Thanks

Harshal.

Re: Paginated API call and writing to DB

Parikhharshal — Fri, 02 Nov 2018 07:37:28 GMT

Hi there

I continued doing what was mentioned in the earlier post:

https://community.talend.com/t5/Design-and-Development/Iterative-Data-extraction-Pagination-and-Polling-from-REST-API/m-p/131209#M81857

However, I seem to stuck at TRestClient where it keeps giving me nullpointerexception Java error.

Not sure if the problem is with this component though as I can see that in Tjava component I had println but nothing is getting printed there.

Is there anything I should be looking at?

In Tjava I got below code:

globalMap.put("VApiUrl",row4.NextUrl);

System.out.println("Value of Globar Var: "+globalMap.get("VApiUrl"));

In TrestClient I have got config like this:

((String)globalMap.get("VApiUrl"))

Can you please share your job code if possible?

Thanks

Harshal.

Re: Paginated API call and writing to DB

root — Fri, 02 Nov 2018 10:18:17 GMT

Can u post the job? Otherwise its difficult to pinpoint the issue

Re: Paginated API call and writing to DB

Parikhharshal — Fri, 02 Nov 2018 12:35:56 GMT

@mbocquet: I have already posted job picture above. Please have a look. Thanks.

Re: Paginated API call and writing to DB

Anonymous — Fri, 02 Nov 2018 12:50:43 GMT

You are setting this key before ever calling the api.

globalMap.put("VApiUrl",row4.NextUrl);

It's always going to be null for the first call if your job is designed in this fashion.

Re: Paginated API call and writing to DB

Parikhharshal — Fri, 02 Nov 2018 12:53:18 GMT

@evansdar: Thanks a lot for your reply. So can you please suggest how it should be? Would appreciate if you can guide me on this or share few screenshots/examples you might have.