Load Data by using Rest API

[vishavb007]

Hi All,

I am loading data from Elastic Search by using Rest API in Qlik Sense SaaS. I am able to get only 2 lac records from Elastic Search instance but there are 2 crore records are available in the ES. I have seen many solution to load all the data, none of that is working. The use of scroll API might work but I don't know how to use scroll API to get data from ES. I don't have ready reference script of scroll API. The pagination which I am trying in connection string is offset where I am able to load data only 2 lac records.

Pls, help me out to get all the records from ES to Qlik Sense.

Warm Regards,

Vishav Bhatia

M/W: +919034572560

[Vinay_B]

Hi @vishavb007 

I believe the first step for us is to know how the REST API being connected to handles paging, and it is really important to know where and how the API endpoints send the signal to get the next page. Also, Just to add some context, Qlik SaaS itself does pagination. Please refer to the image that I am attaching here.

[vishavb007]

Hi @Vinay_B 

Sure, I am also using the Pagination supports by Qlik SaaS in connection string but here itself the limit to load the records is 2 lac.

I am using pagination type as offset as Elastic Search supports offset pagination.

Pls find attached image for your reference.

It is working for up to 2 lac records but for more than 2 lac records qlik is suggesting to use scroll API, pls refer below image.

[Vinay_B]

@vishavb007 I will check in my environment and update you on my findings. Thanks!

[Akshesh_Patel]

@vishavb007 

 Yes, you have to use the scroll API.

 https://www.elastic.co/guide/en/elasticsearch/reference/7.5/search-request-body.html#request-body-s...

After each post to the .../_search/scroll endpoint, you need to check if you received any data so you can stop when you've retrieved all the data.

Best

Akki

[Akshesh_Patel]

 https://www.elastic.co/guide/en/elasticsearch/reference/7.5/search-request-body.html#request-body-s...

After each post to the .../_search/scroll endpoint, you need to check if you received any data so you can stop when you've retrieved all the data.

Best

Akki

[vishavb007]

Hi Akki,

Can you please share the Qlik load editor script for the same as it will help me to understand it more effectively.

Regards,

Vishav Bhatia

+919034572560

[Saurabh_manish]

Hi Vishav

I am facing issue Qlik rest connector for elasticsearch using custom pagination. Currently I am able to fetch only 10 records.

Thanks

Saurabh