[resolved] Managing Talend jobs from outside Talend

[Anonymous]

As the title states, I'm looking for a way to manage Talend jobs from outside any Talend product. Basically I'm looking for an API (REST or SOAP) which let me
- start a job (with parameters)
- stop/cancel a job
- check whether a job is running (since when, last successful run, ...)
- get a list of all jobs defined
- <any 'normal' automation stuff>
I have been looking around, but I'm not able to find anything useful in the forum or documentation so far.
I did take a look at the AdministratorWebService (http://localhost:8080/org.talend.administrator/services/AdministratorWebService?wsdl) but this contains only 3 very basic functions: initSession, scheduleTaskForInstantRun, and closeSession. This allows me to schedule a job, but as this is an asynchronous operation, I don't get any feedback on its status. Also, it would require me to know task ids in advance somehow.
I also looked into the 'Deploying or exposing a Job as a Web service' KB article and the similar tutorial and wiki entry on TalendForge, but that doesn't provide anything useful either. I would basically have to know the web service before I can execute it. Moreover, there seems to be no security when calling such a service which makes its usage unacceptable in a production environment. Also, there only seems to be a runmethod, nothing about cancellation or requests of stats.
Is there any other resource / API I might haven't found yet?

[Anonymous]

Hi John,
All what you are looking for is possible using TOS.
start a job (with parameters)
>> use property file to read the parameter values...
- stop/cancel a job
>> You will have to track the java class instance for corresponding job and then kill that instance after specific request...
- Use delay at appropriate place to stop, this could be activated using real time parameter in the value or database
- check whether a job is running (since when, last successful run, ...)
>> You need to modify your job design to enable efficient custom logging, you can get all this
- get a list of all jobs defined
>> You can get list of jobs inside your current project from the talend repository...
What else you are looking for?
Vaibhav

[Anonymous]

I don't want to use TOS, I want to do it from outside any Talend application.
After some more digging, I found the metaServlet service. I believe this is what I'm looking for though I cannot yet get the service to send me a proper reply. I always get the same answer that the query is empty, independt of the actionName I'm using:
I'm sending:

GET http://localhost:8080/org.talend.administrator/metaServlet HTTP/1.1
Host: localhost:8080
Content-Type: application/json; charset=utf-8
Content-Length: 68
{ "actionName": "listTasks","authPass": "pass","authUser": "admin" }

And I receive:

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Length: 41
Date: Tue, 01 Jul 2014 13:56:44 GMT
{"error":"Query is empty","returnCode":2}

[Anonymous]

In the documentation, I found a single example:

For users who want to use metaServlet as a Web API, note that the typical URL generated by your requests reads
as follows:
localhost:8080/<ApplicationPath>/metaServlet?<UserRequest>, for example
http://localhost:8080/org.talend.administrator/metaServlet?
eyJhY3Rpb25OYW1lIjoiaGVscCIsImNvbW1hbmROYW1lIjoidXNlckV4aXN0In0=

There is no further explaination about this format anywhere, but on a wild guess, I put this through a base64 decoder and this translates to

{"actionName":"help","commandName":"userExist"}

So the JSON format specified in the help of metaServlet needs to be base64 encoded as a query string in the HTTP request.
I hope this helps someone else, because the documentation is rather unclear in my opinion.

[Anonymous]

Hi johndoe2,

There is no further explaination about this format anywhere, but on a wild guess, I put this through a base64 decoder and this translates to
{"actionName":"help","commandName":"userExist"}
So the JSON format specified in the help of metaServlet needs to be base64 encoded as a query string in the HTTP request.
I hope this helps someone else, because the documentation is rather unclear in my opinion.

Thanks for sharing your soution with us. It's really helpful. Are you referring to document https://help.talend.com/search/all?query=Calling+metaServlet&content-lang=en which is unclear?
Best regards
Sabrina

[Anonymous]

Yes.
More specifically, metaServlet is "sold" as a REST web service throughout the documentation. If you use the metaServletCaller.cmd you clearly see that all requests are made in JSON format. As I could not find any other information or real world example on how to contact the service I tried to craft my own HTTP packet with json as the content. That clearly didn't work. By accidentally stumbling about the piece of information there, I found out that contacting the metaServlet service is done using an ordinary base64 encoded querystring of the JSON formatted request.
What would really help the documentation is an example on how to call the metaServlet service from any other programming language or from outside Talend. I would prefer a demo, showing
1) Creating the request in JSON format
2) Translation of the JSON string into its base64 encoded variant
3) Call of the web service using the result of 2) as query string
4) Receive and process the result
The metaServletCaller.cmd is a nice tool, but it doesn't illustrate the usage of the REST web service.

[Anonymous · Accepted solution]

Hi all,
In my own investigations, I ended up with the same conclusions about base64 encoding and I also think that this REST stuff is very powerful and should be well documented.
BR

[Anonymous]

thanks john.  that was very helpful for me as well.

[Anonymous]

Thanks John for your clarification. It helped me a lot, and I agree with you that the use of "base64" encoding must be clearly stated in the page.
Talend should take more care of the documentation:
That's not a good example of documentation.