TIS and integration with Tidal Enterprise Scheduler

[Anonymous]

Hello.
Long story short.
I work for a company that I would like to see replace their current ETL tool with TIS (I have used TIS at another company). Currently, we use Tidal for ALL enterprise scheduling and so Tidal is the scheduler that kicks off our current ETL jobs (even though the ETL tool has its own scheduler, it goes unused). I cannot say that I agree with this implementation, but there may not be much I can do about it.
I am not that familiar with Tidal and the different mechanisms that it uses to kick off various types of jobs, so I did some searching for Tidal adapters that integrate with Talend and also for some other people who may be using Tidal to kick of TIS jobs that have been set up in the Talend Administration Console. I have found nothing.
So, my question is this: Is there a way that an application (such as Tidal) can kick off a job that exists in TIS Admin Console so that the Admin Console will still control what JobServer is running the job, etc., and also allow TIS to send a return code of some sort back to the calling application.
I imagine a world where the job is scheduled in Tidal, and then Tidal tells the Admin Console (via web service or something) to run some job that has been set up by our release engineer, etc.
Comments, suggestions?
Thanks in advance.

[Anonymous]

Hello smackedus,
The Talend Integration Suite Jobs can be scheduled by Tidal.
Our exported Jobs are fully compliant with every third party scheduler of the Marketplace Autosys, ControlM, Tidal, ...
The easiest and first integration you can perform is using the "export job script" function from your Talend Integration Suite studio and then to deploy and schedule in Tidal manually your process. I know that's not the full integration that you expect but that's a good start and it helps you to use the Tidal console for the return code and other error management. With this integration you didn't lose the capacity of Talend to monitor your process through the Activity Monitoring Console and the Monitoring Dashboard.
Indeed every Talend Jobs are pushing some exceptions, statistics and activities logs via Flat files or a Log table. Then you can visualize these information into 2 different consoles : the Acitivity Monitoring Console and the Monitoring Dashboard.

Then we do offer a CommandLine module with a specific API interface in Talend Integration Suite to be used by your third party Scheduler. Your own scheduler will call some command to generate, and execute with automatic deployment on your favorite execution server.
Finally our Talend Admin Center and our CommandLine; two component for the generation, the deployment and the execution of our Jobs can be intergate with your Tidal by using a WebService call. The issue with this integration is that the main part of the enterprise SCHEDULER don't have the capabilities to call a WebService call.
So to sum up on the topics : 3 different ways :
- easiest : export job script and manual deploy and integration into your Scheduler
- commandLine : use the CommandLine interface to call the export, the deploy and execution of your Jobs into your Scheduler.
- webservices : use the WebSerivces API to adress this need in your Scheduler.
Best regards;

[Anonymous]

Hi cantoine,
Thanks for the quick reply.
Ah, shell script land. So, one of things that TIS Enterprise is doing for you (I think) is kicking off a job to run on one of many JobServers in your execution grid. In the Admin Console, you'd create a job and assign it to a Virtual Server (been awhile, I forget the terminology now that I do not have TIS anymore) and the Admin console will run that job on the most bored server in your grid. Super cool. But, would I not lose this functionality if I were to "export job script" and stick it on a specific server for Tidal to call? Is there a way around this loss of functionality? (I'm going to guess CommandLine.)
As for CommandLine, this is exactly what the Admin Console is doing anyway and I'd rather not reinvent the wheel, if at all possible. But I think I get what you're saying and I'm surprised. I have been operating under the assumption that the CommandLine module doesn't know anything about the various JobServers, and especially the execution grid. I had thought that the AdminConsole uses CommandLine to compile the job into a deployable package but it is the AdminConsole which actually deploys the package to a JobServer instance to run based on what the JobServers are telling the AdminConsole about resource usage (I say this because the CommandLine doesn't have to be installed on the same box as the AdminConsole, therefore wouldn't know about resource utilization across your grid). Are you saying that I could write a script that calls CommandLine to deploy a TIS package to a random (least utilized) JobServer for execution?

[Anonymous]

Hi smackedus,
Yes, I'm saying : you can leverage the CPU/Execution balancer from the Virtual server through the CommandLine (this is available in Talend Integration Suite Enterprise Edition only).
There is a command called : executeJobOnServer with many input parameters such as JobName, JobVersion, JobContext and ServerName (physical or virutal).
The webservices API (mentioned into my previous thread) is also providing the same level of integration from your own corporate scheduler to execute the Talend Job via the Execution balancer features.
Best regards;