Upload .json file in req body as a binary file

[heshkaru]

curl -XPOST -u 'xxxxx:xxxxxx' 'https://xxxxxxxxxxxxxxxxxxxx/etl_staging/_bulk' --data-binary @etl-staging-job1.json -H 'Content-Type: application/json'

How to upload a json file and use it in a POST API

[Anonymous]

Set a tRestClient's Http method to "POST", the Content Type to "JSON" and read the JSON into the component's "string" column.

[heshkaru]

Hi,

My target is to send and API call to ElasticSearch index

curl -XPOST localhost:9200/index_local/my_doc_type/_bulk --data-binary @/home/data1.json

Is there a way to do rather than reading the whole json body.(Its a quite a bit huge json)

[heshkaru]

Actually data is in below format.

{ "index" : { } }

{ "ENV":"PROD", "BATCH_ID": "2021-10-10 08:27:16","TEST_CASE": "test_case3", "IS_TEST_CASE_MATCHED" : "FALSE", "COMPARISON": "POL", "OLD_PURCHASE_ORDER_NO": "ADDQYU5B", "NEW_PURCHASE_ORDER_NO":"100018718", "OLD_OC_NUMBER":"U0056A", "NEW_OC_NUMBER":"GIMEDN", "SEQUENCE":"1000", "JSON_PATH":"transmissionStatus", "IS_MATCHED":"TRUE", "EXPECTED_VALUE":"ADDQYU5B", "ACTUAL_VALUE":"ADDQYU5B" }

{ "index" : { } }

{ "ENV":"PROD", "BATCH_ID": "2021-10-10 08:27:16","TEST_CASE": "test_case2", "IS_TEST_CASE_MATCHED" : "FALSE", "COMPARISON": "POL", "OLD_PURCHASE_ORDER_NO": "ADDQYU5B", "NEW_PURCHASE_ORDER_NO":"100018718", "OLD_OC_NUMBER":"U0056A", "NEW_OC_NUMBER":"GIMEDN", "SEQUENCE":"1000", "JSON_PATH":"transmissionStatus", "IS_MATCHED":"TRUE", "EXPECTED_VALUE":"100018718", "ACTUAL_VALUE":"100018718" }

{ "index" : { } }

{ "ENV":"PROD", "BATCH_ID": "2021-10-10 08:27:16","TEST_CASE": "test_case1", "IS_TEST_CASE_MATCHED" : "FALSE", "COMPARISON": "POL", "OLD_PURCHASE_ORDER_NO": "ADDQYU5B", "NEW_PURCHASE_ORDER_NO":"100018718", "OLD_OC_NUMBER":"U0056A", "NEW_OC_NUMBER":"GIMEDN", "SEQUENCE":"1000", "JSON_PATH":"transmissionStatus", "IS_MATCHED":"FALSE", "EXPECTED_VALUE":"ADDQYU5B", "ACTUAL_VALUE":"100018718" }

somewhat different from json, but this is the way that elasticsearch identifies index and data

[Anonymous]

What you appear to have there is multiple JSON documents. One "command" line followed by a "data" line. I have had to follow this process before with ElasticSearch.....but a while ago.

How huge is the file? Reading it shouldn't cause any issues and I believe that the body of the API is intended to contain the JSON as String. If the file is huge, it would actually make sense to break it up into several posts than just one large one....although you can try that I guess. But as I said, the ElasticSearch API requires the data in the body. The Curl --data-binary command (this is Curl and not related to ElasticSearch, so shouldn't be use with Talend) simply reads the content of the file into the body. So even though it looks like you are sending a packages file with Curl, you are just reading the contents.

[heshkaru]

 

Im currently creating the above mentioned .json file in tFileOutputDelimited_2.

How do I proceed to get the answer in your way

Thank you very much

[Anonymous]

Is that creating the JSON you showed me above? Has that JSON been tested with CUrl? If so, I guess "if it ain't broke, don't fix it" comes to mind. But this does not look like the best method of generating the JSON.

With regard to sending it with the tRestClient, you just need to read it from the file (maybe with a tFileInputRaw) and make sure the JSON text is passed to the "string" input schema column of the tRestClient.

[heshkaru]

Above job creates the .json file and curl is working fine. Need to sort this TRestClient task. Thank you for your time. I will try to apply this