Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi Everyone
I am having a requirement in
Talend Open Studio DI
to read the huge amount of data(e.g: around 12000 records) from elastic search URL Using the below URL, I have read the first 10000 records using trestclient component GET request.
http://localhost:9200/action-2021.03.31/_search?scroll=60m&size=10000
But I don't know how to read the remaining 2000 records in talend.
I can able to achieve the same scenario in postman tool by doing the following steps.
step 1: http://localhost:9200/action-2021.03.31/_search/?scroll=60m&size=10000
As a result of this I accessed the fist 10000 records and scroll_id is generated as part of this.
Step 2: Passing the scroll_id in the body section of the URL as below
URL:
http://localhost:9200/_search/scroll
Body:
{
"scroll" : "10m",
"scroll_id": "DIN1ZXJ5QW5kRmV0Y2gBAAAAAAJtbIoMMwN0Ui1YY0VTQXFIY3NGUjVZMTN3Zw=="
}
But I don't know to implement this in talend like, how to pass the body section in the URL and which component has to be used for achieving this scenario.
Any experts can help on this scenario.
Regards,
Dinesh
Hi Dinesh, you can use a twritejsonfields to write the body section and connect it to the trestclient to send it via a post request.
Hi gjeremy,
Thanks for your response.
Since I am new to Talend, can you please help me here by sharing the screenshots on how to achieve this scenario.
Your help is highly appreciated.
Hi gjeremy,
Thanks for your inputs.
We have achieved this by the following steps
step 1:
Used trestclientcomponent and passed the URL with the below format
http://localhost:9200/action-2021.03.31/_search/?scroll=60m&size=10000
As a result of this we accessed the fist 10000 records and scroll_id.
Step2:
Saved the generatd scroll_id in the context variable.
Step 3:
Used another trestclientcomponent and passed the URL in the below format and passed the generated scroll_id in the query parameter.
After performing this we successfully accessed the next 10000 records
URL:
http://localhost:9200/_search/scroll