topic Re: elasticsearch component return only 10000 characters in Talend Studio

elasticsearch component return only 10000 characters

ADe_Pinto1673864968 — Mon, 21 Oct 2024 05:17:16 GMT

I designed a route to get all the document with level "ERROR" or "WARN" from an elastic index but when I save the result in a file, the route saves only the first 10.000 characters, so I cannot see the rest of the docs with errors. I looked into the configuration also from Camel, but I didn't find any parameter who helps me. If I try to log the message the result is the same. How can I get more data within a call?

This is the query :

{
"track_total_hits": true,
"query":
{ "bool":
{ "must":
[
{ "match":
{ "level":
{
"query" : "ERROR WARN",
"operator": "or"
}
}
}
],

"must_not":
[
{ "match":
{ "tags": "_jsonparsefailure" }
}
],

"filter":
[
{ "range":
{ "@timestamp":
{ "gte": "now-24h", "lte" : "now" }
}
}
]
}
}
}

and I attach a screenshot of the process.

Thanks in advance,

Andrea De Pinto

Re: elasticsearch component return only 10000 characters

Xiaodi_Shi — Mon, 21 Oct 2024 07:46:41 GMT

Hello,

For search operation, could you please try to modify the threadpool.search.queue_size: 10000 in elasticsearch.yml file to see if it helps?

Best regards

Sabrina

Re: elasticsearch component return only 10000 characters

ADe_Pinto1673864968 — Mon, 21 Oct 2024 12:15:34 GMT

Hello Sabdrina,

thanks for your reply. It's not a problem of elastic. I did the same query with a normal cRest component and it works perfectly fine, so I suppose it's embedded somehow in that specific component. I will change the route and use the cRest component.

Re: elasticsearch component return only 10000 characters

Xiaodi_Shi — Tue, 22 Oct 2024 02:25:28 GMT

Hello,

Thanks for your feedback and feel free to let me know if cRest component is OK with it.

Best regards

Sabrina

Re: elasticsearch component return only 10000 characters

fcolagiacomo — Tue, 22 Oct 2024 11:51:40 GMT

Hi,

if your query returns more than 10000 documents then you have to use Elasticsearch scroll query.

In example:

POST /my-index-000001/_search?scroll=1m
{
  "size": 100,
  "query": {
    "match": {
      "message": "foo"
    }
  }
}

The result from the above request includes a _scroll_id, which should be passed to the scroll API in order to retrieve the next batch of results.

POST /_search/scroll                                                               
{
  "scroll" : "1m",                                                                 
  "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAD4WYm9laVYtZndUQlNsdDcwakFMNjU1QQ==" 
}

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/paginate-search-results.html#scroll-search-results

Re: elasticsearch component return only 10000 characters

ADe_Pinto1673864968 — Thu, 24 Oct 2024 06:40:38 GMT

Thanks for your reply, I'll try it