topic Http Request POST to CKAN API in Talend Studio

Http Request POST to CKAN API

Anonymous — Sat, 16 Nov 2024 11:07:38 GMT

Hi all,
I have to make an Http Request method POST to CKAN API. The CKAN API is ckan.logic.action.create. package_create. Here the link http://docs.ckan.org/en/ckan-2.0.2/ckan.logic.action.create.html
I just tried the request with Advanced Rest Client, as shown in the figure and it works. Now, I want to make the request using Talend Studio. I am using the component tHttpRequest, as shown in the figure, but I don't know how the JSON fields should be set in the file "Post parameters". In any manner I write the fields in the file, I get "400 bad request". Which is the correct way to set the JSON fields?
Thanks all.

Re: Http Request POST to CKAN API

Anonymous — Tue, 28 Jul 2015 08:20:11 GMT

Hi, perhaps one option is to use tRESTClient with JSON posted as a String - that would be closest to the example with AdvancedRestClient
Sergey

Re: Http Request POST to CKAN API

Anonymous — Tue, 28 Jul 2015 09:34:47 GMT

Hi,
I am trying with tRestClient, as shown in figure. I use tFileInputRaw in order to read the JSON from file as string.
I use tXMLMap in order to connect tFileInputRaw and tRestCLient, as shown in figure. By default the schema that accept tRestClient as input is:
body type Document,
string type String.
So, I map the content file to string. I don't know if it is correct. Then, I set tRestCLient as shown in figure.
But, when I run the Job, I get this error:

Exception in component tRESTClient_1
javax.ws.rs.InternalServerErrorException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Unknown Source)
	at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:463)
	at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:881)
	at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:852)
	at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:410)
	at org.apache.cxf.jaxrs.client.WebClient.post(WebClient.java:441)
	at scrape.restclient_0_1.restclient.tFileInputRaw_1Process(restclient.java:1181)
	at scrape.restclient_0_1.restclient.runJobInTOS(restclient.java:1637)
	at scrape.restclient_0_1.restclient.main(restclient.java:1496)

Where is the mistake?
Thanks.

Re: Http Request POST to CKAN API

Anonymous — Tue, 28 Jul 2015 10:33:38 GMT

Have you tried going to tRESTClient > Advanced Settings, and ticking the "Log messages" option? This will log the exact content of the REST request & response messages in the Console window, which should hopefully help debug the cause of the javax.ws.rs.InternalServerErrorException you're getting.

Re: Http Request POST to CKAN API

Anonymous — Tue, 28 Jul 2015 10:49:09 GMT

Hi,
Maybe I understand the problem. The output of tXMLMap is (body+string):

<?xml version="1.0" encoding="UTF-8"?>
<root/>|{ 	"name":"dataset-test",
  	"author":"vincentdj",
	"title":"Dataset Test",
	"author_email":"vincenzodj.it"
}

I do not know how it happened that by run several times the Job, the output has became like this (without the body):

{ 	"name":"dataset-test",
  	"author":"vincentdj",
	"title":"Dataset Test",
	"author_email":"vincenzodj.it"
}

and it works correctly. But, If I delete voluntarily the body from tXMLMap, I get error on tRestClient schema that want body field, as shown in figure.

Re: Http Request POST to CKAN API

Anonymous — Tue, 28 Jul 2015 15:38:12 GMT

tRESTClient expects both body (Document) and string (String) columns, so you can't delete them.
The way around this is to make sure that the body column is completely empty in the input to the tRESTClient. If body is empty, tRESTClient will use the content of string when sending a POST request to the REST service.
Remove the tXMLMap because it's initialising the body column with a Document containing a single <root> element, which you don't want.
Instead, remove the tXMLMap. Replace it with a tConvertType, to move the content of the content column from tFileInputRaw into the string column for tRESTClient. So your flow looks something like this:

tFileInputRaw --> tConvertType --> tRESTClient --> tLogRow
Settings on tConvertType should be: Auto Cast unticked, Input column "content" goes to output column "string".
I would post a screenshot but I don't have enough posts yet