topic Re: Load Data by using Rest API in Connectivity & Data Prep

Load Data by using Rest API

vishavb007 — Tue, 12 Oct 2021 05:06:30 GMT

Hi All,

I am loading data from Elastic Search by using Rest API in Qlik Sense SaaS. I am able to get only 2 lac records from Elastic Search instance but there are 2 crore records are available in the ES. I have seen many solution to load all the data, none of that is working. The use of scroll API might work but I don't know how to use scroll API to get data from ES. I don't have ready reference script of scroll API. The pagination which I am trying in connection string is offset where I am able to load data only 2 lac records.

Pls, help me out to get all the records from ES to Qlik Sense.

Warm Regards,

Vishav Bhatia

M/W: +919034572560

Re: Load Data by using Rest API

Vinay_B — Tue, 12 Oct 2021 05:34:23 GMT

Hi @vishavb007

I believe the first step for us is to know how the REST API being connected to handles paging, and it is really important to know where and how the API endpoints send the signal to get the next page. Also, Just to add some context, Qlik SaaS itself does pagination. Please refer to the image that I am attaching here.

Re: Load Data by using Rest API

vishavb007 — Tue, 12 Oct 2021 05:51:58 GMT

Hi @Vinay_B

Sure, I am also using the Pagination supports by Qlik SaaS in connection string but here itself the limit to load the records is 2 lac.

I am using pagination type as offset as Elastic Search supports offset pagination.

Pls find attached image for your reference.

It is working for up to 2 lac records but for more than 2 lac records qlik is suggesting to use scroll API, pls refer below image.

Re: Load Data by using Rest API

Vinay_B — Tue, 12 Oct 2021 06:03:35 GMT

@vishavb007 I will check in my environment and update you on my findings. Thanks!

Re: Load Data by using Rest API

Akshesh_Patel — Tue, 12 Oct 2021 18:57:40 GMT

@vishavb007

https://www.elastic.co/guide/en/elasticsearch/reference/7.5/search-request-body.html#request-body-se...

After each post to the .../_search/scroll endpoint, you need to check if you received any data so you can stop when you've retrieved all the data.

Best

Akki

Re: Load Data by using Rest API

vishavb007 — Wed, 20 Oct 2021 04:44:54 GMT

Hi Akki,

Can you please share the Qlik load editor script for the same as it will help me to understand it more effectively.

Regards,

Vishav Bhatia

+919034572560

Re: Load Data by using Rest API

Saurabh_manish — Wed, 08 Jun 2022 11:29:08 GMT

Hi Vishav

I am facing issue Qlik rest connector for elasticsearch using custom pagination. Currently I am able to fetch only 10 records.

Thanks

Saurabh