[resolved] Remote start of Talend job deployed on Job Server from external Java

[Anonymous]

Hi,
I found few of threads which discuss this idea, but none of it is work against my current requirement.
We use Talend 5.4.1 Enterprise Edition. I want to execute the job deployed on job server, available in Job Conductor in TAC, remotely from external Java application.
I know already 2 remote job start scenarios supported by Talend:
1. Start your job from Java
Calling a Talend Job from an external Java application
Problem with this approach is that the standalone build version needs to be directly referenced on the CLASSPATH, so each change of the job you do, you need to re-reference in Java/
2. Start your job remotely from Talend Studio
Use Distant Run from Talend Enterprise Data Integration to execute remote Jobs
Problem with this approach is that I can execute the job only from within Talend, so there is required on each run, local build
Anyway both scenarios are handy, but actually is there any communication interface against TAC available, where I can call it, tell TAC now run this job and the job gets executed?
Best regards,
Ladislav

[Anonymous · Accepted solution]

Yes. The TAC provides a restful web service.
I have made a component tRunTask (similar to tRunJob) which allows you to write a job which can start a task in the TAC.
We often build such jobs to create highly complex job start scenarios or to connect external job scheduler with the TAC by the help of communication job (simply job exported as standalone java program embedded in the external application).
Here the link to Talend Exchange:
https://exchange.talend.com/#marketplacesearch:gallery=marketplace%252F1&ob=releaseDate&o=0&c=20&d=t...

[Anonymous]

Hi,
very good Jan! :-), thank you.
I see that you handle all parameters except one, the project name, imagine you have 2 labels (job name visible in TAC) for 2 or more different projects, what will happen, colision? in the TAC database schema there is EXECUTIONTASK referencing PROJECT as parent, so not only the LABEL, but the project is what makes the job unique, am I right?
I finally did a workaround following way yesterday just by exploring other way around.
There is TalendTaskExec connector in the BonitaSoft BPM. I extracted the Java code from it and it is very simple.
You need to find following jar files (when searching for them, make search for jar within a jar as soon as some of them are packed within other jars):

talend-tac-5.10.1.jar
bonita-client-5.3.jar
log4j.1.2.12.jar
talend-json.jar
xstream-1.3.1.jar
commons-codec-1.4

Once those jars loaded to classpath, you can write simple call in Java class:

TalendTaskExec tte = new TalendTaskExec();
// Usually the context is to set the environment variables like user name and password, connection type, etc.
// Not required to use
//tte.setContext(tte);
// This is standard TAC url you use in browser, remember to load SSL certificate in JDK trusted store if
// you use HTTPS connection
tte.setTacURL("http://hostname:8080/org.talend.administrator/");
tte.setTacUsername("TACUser");
tte.setTacPassword("TACPassword");
// column ID in table EXECUTIONTASK
tte.setTaskId(new Long("9244"));
// I don't know about other modes, but assume there might be asynchronous as well
tte.setMode("synchronous");
// Execute the job and this will print the result, there is as well method .getResult() returning String
// so you can process later in error handler or audit framework
tte.execute();

Now, what about that setTaskId( Long ) method, how to get the name?
Well the way I did it is that as input on web service I expose the application need to define:

Project name - this should be mandatory, if optional, you need to raise error if more than one record found
Job Label - mandatory
Job version - optional, if not passed, you can default to latest version available
And here how to retrieve the data from database tables:

SELECT ET.ID,MAX(ET.JOBVERSION)
FROM EXECUTIONTASK ET
INNER JOIN PROJECT P
ON ET.PROJECT_PROJECT_ID = P.ID
AND P.LABEL = 'PROJECT_NAME_IN_TAC'
AND ET.LABEL = 'JOB_NAME_IN_TAC'
GROUP BY ET.ID;

So this solution works as well, but the component by Jan is probably better. As soon as I expose the service on Oracle Service Bus level, it will be better for me to utilize the rest service way instead of the Java way, so I am preventing creation of Java dependencies on OSB.
Thanks Jan again, I will change your component in the way that is supports the Project.
Best regards,
Ladislav

[Anonymous]

Hi Ladislav,
I am pretty sure the Bonita Connector use the same API as I do, there is no other API in the TAC.
If you use the release 5.6.x you will experience a stable run. All previous releases of the TAC handle the run command very weak. In the asynchronous run the TAC did not provides a id of the current run request and therefore I had to track the changes of the status of the task in the TAC. If you use such older Talend releases I strongly suggest you stick with the synchronous mode!
In my component you have the possibility to use the label (you are right, it could be ambiguous, but usually we do not use a label twice).
The web service of the TAC does not allow the usage of the project name, it is not a matter of necessary changes of my component.
I enabled this by the help of another web service returning the task ID for the label.
If you want to add such functionality in your application (outside Talend) feel free to use the library of the component and build your own wrapper code around this. The component source code is open source and published. I have just added the link to the code in Talend Exchange.

[Anonymous]

Hi Jan,
well we are already managing about 4 projects in TAC and others are upcoming soon... If I imagine our number of jobs and the fact we follow some standard naming conventions, the situation might occur in our scenario very easily... hm, I will think about it.
Still the solution with select over the TAC tables will prevent work without issue, on the other hand I do direct db access with mixture of standard TAC API.
I see that you follow what is implemented in Talend, so your component is aligned just fine with current TAC API implementation.
So this is sorted for me, we have to ways, the standard, but not secure from "only my" perspective, and the mixed where the secure way is done via direct db access.
Thanks for discussion on this.
Ladislav

[Anonymous]

Just small update on this, source of dependency Java jar library file for tRunTask is available at for those who need to change it for their needs and call it directly from within external Java application:
https://github.com/archenroot/talend-administration-center-remote#talend-administration-center-remot...
Ladislav

[Anonymous]

Because Talend changes the database structure sometimes I would not rely on a database access.
Next, in some cases you will fail because if the database is e.g. a PostgreSQL database the upper case table names will not work.
I suggest you add an option to lower case the identifiers if necessary.

[Anonymous]

Of course that is good point I am aware of, I will always be here on Oracle, I am also aware about possible database structure changes with new releases....
I didn't decided yet if I go with the REST or combined way yet, I decompiled your code as you see and put it on github, hope you don't mind, will slightly change it and document it little bit.
I already prepared WSDL and XSD schemas for SOAP service which I will expose on our enterprise service bus (Oracle one). This service will reference the the jar file and will let any application to start the any Talend job remotely. So even in ETL processing I will apply SOA practice, as soon as the source application knows when data are ready to transport by Talend, it will prevent scheduled polling to just waste resources.
Here is XSD to capture required parameters in the moment:

<xs:simpleType name="String20Type">
		<xs:restriction base="xs:string">
			<xs:maxLength value="20"/>
		</xs:restriction>
	</xs:simpleType>
	<xs:simpleType name="String100Type">
		<xs:restriction base="xs:string">
			<xs:maxLength value="100"/>
		</xs:restriction>
	</xs:simpleType>
	<xs:simpleType name="VersionType">
		<xs:restriction base="xs:token">
			<xs:pattern value="{1,2}.{1}{1,2}"/>
		</xs:restriction>
	</xs:simpleType>
	<xs:simpleType name="String4000Type">
		<xs:restriction base="xs:string">
			<xs:maxLength value="4000"/>
		</xs:restriction>
	</xs:simpleType>
	<xs:complexType name="ContextVariableListOfValuesType">
		<xs:sequence>
			<xs:element name="Value" type="tns:String4000Type" maxOccurs="unbounded"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="ContextVariableType">
		<xs:sequence>
			<xs:element name="ContextVariableName" type="tns:String4000Type"/>
			<xs:element name="ContextVariableListOfValues"
				type="tns:ContextVariableListOfValuesType"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="ContextVariableListType">
		<xs:sequence>
			<xs:element name="ContextVariable" type="tns:ContextVariableType" maxOccurs="unbounded"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="TalendJobType">
		<xs:sequence>
			<xs:element name="TalendProject" type="tns:String20Type" nillable="true">
				<xs:annotation>
					<xs:appinfo/>
					<xs:documentation source="https://www.talendforge.org/tutorials/tutorial.php?idTuto=54#step813" xml:lang="en">Represents project name defined in TAC to which the job requested to be started is related to.</xs:documentation>
				</xs:annotation>
			</xs:element>
			<xs:choice>
				<xs:element name="TalendJobLabel" type="tns:String100Type"/>
				<xs:element name="TalendJobId" type="xs:int"/>
			</xs:choice>
			<xs:element name="TalendJobVersion" type="tns:VersionType" nillable="true"/>
			<xs:element name="ContextVariableList" type="tns:ContextVariableListType"
				nillable="true"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="TalendJobListType">
		<xs:sequence>
			<xs:element name="TalendJob" type="tns:TalendJobType" minOccurs="1"
				maxOccurs="unbounded"/>
		</xs:sequence>
	</xs:complexType>
	<xs:element name="TriggerTalendJobRequest">
		<xs:complexType>
			<xs:complexContent>
				<xs:extension base="meta:MessageType">
					<xs:sequence>
						<xs:element name="TalendJobList" type="tns:TalendJobListType"/>
					</xs:sequence>
				</xs:extension>
			</xs:complexContent>
		</xs:complexType>
	</xs:element>
	<xs:element name="TriggerTalendJobResponse">
		<xs:complexType>
			<xs:complexContent>
				<xs:extension base="meta:MessageType">
					<xs:sequence>
						<xs:element name="ResultMessage" type="xs:string"/>
					</xs:sequence>
				</xs:extension>
			</xs:complexContent>
		</xs:complexType>
	</xs:element>
	<xs:element name="TalendJobNotTriggerFault">
		<xs:complexType>
			<xs:complexContent>
				<xs:extension base="meta:FaultMessageType"> </xs:extension>
			</xs:complexContent>
		</xs:complexType>
	</xs:element>

[Anonymous]

Actually it was not necessary to decompile the code. The code is freely available at source forge.
At the start I was also thinking about my own service on the TAC database to get better information and have more possibilities. At the moment I would not do this because some things cannot be done only by manipulating the database.
I am staying in contact with Talend and in the past release 5.6 they have resolved an feature request from me to have the job return code in the web service response of the run task action. I am sure if you file such requests they will not ignore it. 
http://sourceforge.net/projects/talend-user-components/

[Anonymous]

I didn't know where the source code is.. still decompilation is 1 minute task...
I am thinking about feature request already, yes. I was handling new components like tHTTPRequest and other changes which are in main stream already and also about 20-30 bugs back in 2009-2011 with Talend, the communication is quite good from my experience.
In my case SOAP on enterprise service bus is standard we go, so I still need to use it as wrapper service as interface to that jar/REST interface. I of course agree the right way when missing some functionality is to extend existing API instead of jumping into direct database communication..
Will post here the request once created.