Issue in Extracting Data from Elastic Search through rest connector

fashid — Thu, 23 Dec 2021 13:12:34 GMT

Hi Experts,

I have been trying to extract data from the Elastic search through rest connector but i have been able to fetch only 10 records .

Below is the script that i am using

let url = "http://10.35.12.5:9200/prem_reg_final_4_test_1_201903/_search?";

LIB CONNECT TO 'testconnection_heah (conn_qlikdv1)';

NoConcatenate

scrollTable:

SQL SELECT

"_scroll_id" AS "scrollid",

(SELECT

"total" AS "total_u1"

FROM "hits")

FROM JSON (wrap on) "root"
WITH CONNECTION(Url "$(url)");

Let total = Peek('total_u1',0,'scrollTable');
Let scrollid = Peek('scrollid',1,'scrollTable');

for startAt = 0 to total step pageSize
RestConnectorMasterTable:
SQL SELECT
"__KEY_root",
(SELECT
"__FK_hits",
"__KEY_hits",
(SELECT
"__FK_hits_u0",
"__KEY_hits_u0",
(SELECT
"NUM_REFERENCE_NO",
"__FK__source"
FROM "_source" FK "__FK__source")
FROM "hits" PK "__KEY_hits_u0" FK "__FK_hits_u0")
FROM "hits" PK "__KEY_hits" FK "__FK_hits")
FROM JSON (wrap on) "root" PK "__KEY_root"
WITH CONNECTION(url "$(url)");

scrollid = lookup('_scroll_id', '__KEY_root', 1,'RestConnectorMasterTable');

Trace $(scrollid);

url = 'http://10.35.12.5:9200/prem_reg_final_4_test_1_201903/_search/scroll?scroll=1m&size=100&scroll_id=$(scrollid)';

NEXT startAt;

When i am trying to run this script i am getting the below error

The following error occurred:

QVX_UNEXPECTED_END_OF_DATA: HTTP protocol error 400 (Bad Request): {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/prem_reg_final_4_test_1_201903/_search/scroll] contains unrecognized parameters: [scroll], [scroll_id], [size]"}],"type":"illegal_argument_exception","reason":"request [/prem_reg_final_4_test_1_201903/_search/scroll]

Also if i just load the hits table to get total and scroll id then my scroll id field is blank . i do have values in the total field .

If anybody could guide or throw some light into this .

Regards,

Nadeem Shaikh

Re: Issue in Extracting Data from Elastic Search through rest connector

Anil_Babu_Samineni — Fri, 04 Oct 2019 13:14:00 GMT

For me this sounds, there might be some filter condition in the URL white creating the connection.

Re: Issue in Extracting Data from Elastic Search through rest connector

fashid — Mon, 07 Oct 2019 05:09:22 GMT

Hi,

thank you for replying could you please help out in understanding what you mean by filter condition in the URL

Regards,

Nadeem

Re: Issue in Extracting Data from Elastic Search through rest connector

fashid — Fri, 11 Oct 2019 05:32:48 GMT

Any Experts out there who can help on this

Re: Issue in Extracting Data from Elastic Search through rest connector

fashid — Wed, 16 Oct 2019 06:57:12 GMT

Bump.....

Re: Issue in Extracting Data from Elastic Search through rest connector

martinpohl — Wed, 16 Oct 2019 07:32:13 GMT

maybe this helps?

https://www.youtube.com/watch?v=QlCT55_712I

Regards

topic Re: Issue in Extracting Data from Elastic Search through rest connector in Connectivity & Data Prep

Issue in Extracting Data from Elastic Search through rest connector

Re: Issue in Extracting Data from Elastic Search through rest connector

Re: Issue in Extracting Data from Elastic Search through rest connector

Re: Issue in Extracting Data from Elastic Search through rest connector

Re: Issue in Extracting Data from Elastic Search through rest connector

Re: Issue in Extracting Data from Elastic Search through rest connector