topic Re: Upload .json file in req body as a binary file in Talend Studio

Upload .json file in req body as a binary file

heshkaru — Thu, 17 Feb 2022 08:50:20 GMT

curl -XPOST -u 'xxxxx:xxxxxx' 'https://xxxxxxxxxxxxxxxxxxxx/etl_staging/_bulk' --data-binary @etl-staging-job1.json -H 'Content-Type: application/json'

How to upload a json file and use it in a POST API

Re: Upload .json file in req body as a binary file

Anonymous — Thu, 17 Feb 2022 16:10:41 GMT

Set a tRestClient's Http method to "POST", the Content Type to "JSON" and read the JSON into the component's "string" column.

Re: Upload .json file in req body as a binary file

heshkaru — Thu, 17 Feb 2022 16:21:57 GMT

Hi,

My target is to send and API call to ElasticSearch index

curl -XPOST localhost:9200/index_local/my_doc_type/_bulk --data-binary @/home/data1.json

Is there a way to do rather than reading the whole json body.(Its a quite a bit huge json)

Re: Upload .json file in req body as a binary file

heshkaru — Thu, 17 Feb 2022 17:27:32 GMT

Actually data is in below format.

{ "index" : { } }

{ "ENV":"PROD", "BATCH_ID": "2021-10-10 08:27:16","TEST_CASE": "test_case3", "IS_TEST_CASE_MATCHED" : "FALSE", "COMPARISON": "POL", "OLD_PURCHASE_ORDER_NO": "ADDQYU5B", "NEW_PURCHASE_ORDER_NO":"100018718", "OLD_OC_NUMBER":"U0056A", "NEW_OC_NUMBER":"GIMEDN", "SEQUENCE":"1000", "JSON_PATH":"transmissionStatus", "IS_MATCHED":"TRUE", "EXPECTED_VALUE":"ADDQYU5B", "ACTUAL_VALUE":"ADDQYU5B" }

{ "index" : { } }

{ "ENV":"PROD", "BATCH_ID": "2021-10-10 08:27:16","TEST_CASE": "test_case2", "IS_TEST_CASE_MATCHED" : "FALSE", "COMPARISON": "POL", "OLD_PURCHASE_ORDER_NO": "ADDQYU5B", "NEW_PURCHASE_ORDER_NO":"100018718", "OLD_OC_NUMBER":"U0056A", "NEW_OC_NUMBER":"GIMEDN", "SEQUENCE":"1000", "JSON_PATH":"transmissionStatus", "IS_MATCHED":"TRUE", "EXPECTED_VALUE":"100018718", "ACTUAL_VALUE":"100018718" }

{ "index" : { } }

{ "ENV":"PROD", "BATCH_ID": "2021-10-10 08:27:16","TEST_CASE": "test_case1", "IS_TEST_CASE_MATCHED" : "FALSE", "COMPARISON": "POL", "OLD_PURCHASE_ORDER_NO": "ADDQYU5B", "NEW_PURCHASE_ORDER_NO":"100018718", "OLD_OC_NUMBER":"U0056A", "NEW_OC_NUMBER":"GIMEDN", "SEQUENCE":"1000", "JSON_PATH":"transmissionStatus", "IS_MATCHED":"FALSE", "EXPECTED_VALUE":"ADDQYU5B", "ACTUAL_VALUE":"100018718" }

somewhat different from json, but this is the way that elasticsearch identifies index and data

Re: Upload .json file in req body as a binary file

Anonymous — Thu, 17 Feb 2022 18:24:36 GMT

What you appear to have there is multiple JSON documents. One "command" line followed by a "data" line. I have had to follow this process before with ElasticSearch.....but a while ago.

How huge is the file? Reading it shouldn't cause any issues and I believe that the body of the API is intended to contain the JSON as String. If the file is huge, it would actually make sense to break it up into several posts than just one large one....although you can try that I guess. But as I said, the ElasticSearch API requires the data in the body. The Curl --data-binary command (this is Curl and not related to ElasticSearch, so shouldn't be use with Talend) simply reads the content of the file into the body. So even though it looks like you are sending a packages file with Curl, you are just reading the contents.

Re: Upload .json file in req body as a binary file

heshkaru — Fri, 18 Feb 2022 05:53:23 GMT

Im currently creating the above mentioned .json file in tFileOutputDelimited_2.

How do I proceed to get the answer in your way

Thank you very much

Re: Upload .json file in req body as a binary file

Anonymous — Fri, 18 Feb 2022 11:33:40 GMT

Is that creating the JSON you showed me above? Has that JSON been tested with CUrl? If so, I guess "if it ain't broke, don't fix it" comes to mind. But this does not look like the best method of generating the JSON.

With regard to sending it with the tRestClient, you just need to read it from the file (maybe with a tFileInputRaw) and make sure the JSON text is passed to the "string" input schema column of the tRestClient.

Re: Upload .json file in req body as a binary file

heshkaru — Fri, 18 Feb 2022 12:05:16 GMT

Above job creates the .json file and curl is working fine. Need to sort this TRestClient task. Thank you for your time. I will try to apply this