topic Read records more than 10000 from Elastic Search URL in Talend Studio

Read records more than 10000 from Elastic Search URL

dineshkumare — Sun, 04 Apr 2021 17:19:21 GMT

Hi Everyone

I am having a requirement in

Talend Open Studio DI

to read the huge amount of data(e.g: around 12000 records) from elastic search URL Using the below URL, I have read the first 10000 records using trestclient component GET request.

http://localhost:9200/action-2021.03.31/_search?scroll=60m&size=10000

But I don't know how to read the remaining 2000 records in talend.

I can able to achieve the same scenario in postman tool by doing the following steps.

step 1: http://localhost:9200/action-2021.03.31/_search/?scroll=60m&size=10000

As a result of this I accessed the fist 10000 records and scroll_id is generated as part of this.

Step 2: Passing the scroll_id in the body section of the URL as below

URL:

http://localhost:9200/_search/scroll

Body:

{

"scroll" : "10m",

"scroll_id": "DIN1ZXJ5QW5kRmV0Y2gBAAAAAAJtbIoMMwN0Ui1YY0VTQXFIY3NGUjVZMTN3Zw=="

}

But I don't know to implement this in talend like, how to pass the body section in the URL and which component has to be used for achieving this scenario.

Any experts can help on this scenario.

Regards,

Dinesh

Re: Read records more than 10000 from Elastic Search URL

gjeremy1617088143 — Tue, 06 Apr 2021 07:01:28 GMT

Hi Dinesh, you can use a twritejsonfields to write the body section and connect it to the trestclient to send it via a post request.

Re: Read records more than 10000 from Elastic Search URL

dineshkumare — Tue, 06 Apr 2021 11:01:14 GMT

Hi gjeremy,

Thanks for your response.

Since I am new to Talend, can you please help me here by sharing the screenshots on how to achieve this scenario.

Your help is highly appreciated.

Re: Read records more than 10000 from Elastic Search URL

gjeremy1617088143 — Tue, 06 Apr 2021 17:56:54 GMT

https://help.talend.com/r/sgE1qM_4~9kOIJyS0XV1zA/yqePgSMBe4i1p2166jF6RQ

Re: Read records more than 10000 from Elastic Search URL

dineshkumare — Fri, 09 Apr 2021 13:11:10 GMT

Hi gjeremy,

Thanks for your inputs.

We have achieved this by the following steps

step 1:

Used trestclientcomponent and passed the URL with the below format

http://localhost:9200/action-2021.03.31/_search/?scroll=60m&size=10000

As a result of this we accessed the fist 10000 records and scroll_id.

Step2:

Saved the generatd scroll_id in the context variable.

Step 3:

Used another trestclientcomponent and passed the URL in the below format and passed the generated scroll_id in the query parameter.

After performing this we successfully accessed the next 10000 records

URL:

http://localhost:9200/_search/scroll