topic Re: Running multiple interdependent jobs one after othe in Talend Studio

Running multiple interdependent jobs one after othe

Anonymous — Wed, 12 Jul 2017 07:58:22 GMT

Hi,

I have a case where a Job has to be broken down to two sub jobs as I have to take the output of one job, do some modification (in the output table) and use it as an input to the next job. So, how to do it ? If i try to do something like in the screenshot below then it runs both the jobs simultaneously which I don't want.

Re: Running multiple interdependent jobs one after othe

Anonymous — Wed, 12 Jul 2017 09:29:06 GMT

Right click on TEST1, select Trigger and select "On Subjob OK". Drag this link to the TEMP_STAGING _TABLE component. Done.

Re: Running multiple interdependent jobs one after othe

Anonymous — Wed, 12 Jul 2017 09:47:19 GMT

No, I think you didn't picked my question properly. What I want is that on running the job only first part should run, i.e from TEST to temp_staging_stable, then I have to do some modifications in the temp_staging_stable, then when I run it gain, now the second part of job should run ,i.e from temp_staging_stable to Employee.

Any suggestions how to achieve this?

Regards

Pankaj

Re: Running multiple interdependent jobs one after othe

Anonymous — Wed, 12 Jul 2017 10:08:48 GMT

Check for a controlling state, i.e. either a variable, or some data, to identify that the first step is complete and you have done your modifications in the temp_staging_table, then depending on that controlling state, you either run the first subjob or the second subjob.

What you will do is as follows:

tJava -> Run If -> (Your First Subjob)

-> Run If -> (Your Second Subjob)

Yes, they are linked to the same tJava which should not contain any Java code in its body. It is just a stub to allow you to run to If triggers from 1 point. In each if trigger, you are checking for the correct controlling state. You can use context variables, or actually query the DB for some of the modifications you did, before you reach the tJava to decide what is the controlling state for the If condition.

Re: Running multiple interdependent jobs one after othe

Anonymous — Wed, 12 Jul 2017 11:32:01 GMT

Its a bit difficult for me to design and also I want to avoid any Java coding as my client doesn't have any coding skills.

Isn't it possible that I select my firstjob, execute it, then do some modifications in the resultant table, then select my second Job and execute it!

Re: Running multiple interdependent jobs one after othe

Anonymous — Wed, 12 Jul 2017 12:12:47 GMT

@iburtally was not talking about using any Java. The tJava is just used as a dummy component to drive the conditional RunIfs from.

Why are you designing this as one job? If you have to perform some manual steps between the two processes, why not just write two jobs? Don't get me wrong, it is easily achieved by Talend, but there is no point struggling for ages when two separate jobs will do.

Re: Running multiple interdependent jobs one after othe

Anonymous — Wed, 12 Jul 2017 12:20:21 GMT

Well that had to be the last resort, i.e. to have two separate jobs for this task as there is a manual intervention involved between tow sub jobs. All I was thinking of a way to possible 'select' one the two subjobs and execute and then repeat it for the next. But that doesn't seems to be the case so I shall go with the option of having two jobs for this.

Thanks anyways.

Re: Running multiple interdependent jobs one after othe

Anonymous — Wed, 12 Jul 2017 12:25:57 GMT

What @iburtally described would be fine for this. You can also use context variables to do this. You can use the state of your data in your db to do this. You can use the presence of data in your db to do this. You can do this in SO many ways. You need to give more information about what you have to play with before anyone can be more specific I'm afraid.

Re: Running multiple interdependent jobs one after othe

Anonymous — Fri, 14 Jul 2017 09:38:37 GMT

Hi @iburtally

Is it possible that at the end of step1 or first subjob I can see the result in a GUI(i.e all the columns) and can update some value of a column which updates my Database table accordingly and then I can use it as an Input to my second Job?

The purpose is that my client wants some manual approval process at the end of first job i.e. where we have applied some rule and he can see the changes and put a tick mark against the columns which he actually wants for the main table.

Re: Running multiple interdependent jobs one after othe

Anonymous — Fri, 14 Jul 2017 09:58:54 GMT

Hi,

We have a product called Talend Data Stewardship for this which provides GUI to curate the data.

If you use Talend Data Stewardship, you send your data to the Data Stewardship, then you can login into the UI for Data Stewardship, curate and fix your data, then your second job read the curated data from Data Stewardship and do further processing.

If you want to build your own UI, then you can do that too. But Talend ETL does not provide a UI. It is supposed to be batch running without a UI.

Re: Running multiple interdependent jobs one after othe

Anonymous — Fri, 14 Jul 2017 16:15:08 GMT

Is Talend Data Stewardship also an Open Source or we need to buy it.

Re: Running multiple interdependent jobs one after othe

Anonymous — Fri, 14 Jul 2017 17:53:38 GMT

Talend Data Stewardship is only available as a paid product. However, we include 2 free licenses of it with a Talend Platform product.

Re: Running multiple interdependent jobs one after othe

Anonymous — Sat, 15 Jul 2017 06:14:40 GMT

Ok.

here i want to ask a couple of questions:

1) If Talend provides the option to customize jobs/ components, where are the options to do it.?

2) If for this job, which we have been discussing long time now(two interdependent jobs), I want to write a customized routine, which will lets's say allow me open up a GUI with the ability to load database table in it(in a tabular structure may be) and I look at the columns and have the ability to change values and click on 'Save' button to make changes in database.

This will do my job. So how difficult it's going to be, if feasible?

Kindly let me know all the options.

Re: Running multiple interdependent jobs one after othe

Anonymous — Sat, 15 Jul 2017 13:50:47 GMT

1) You build the jobs yourself, so you therefore customise them while building them. If you are talking about runtime dynamic functionality, you have to build this as well. You can use Context variables for this for example. Components come with configuration options and you can create your own components. Creating components is too difficult to explain how to do in a scenario like this. You will need to Google how to do this and then experiment. There are no easy answers to that.

2) I am not sure why you would do this. Talend jobs should not be considered GUI enabled. The jobs are not meant to work with GUIs. You *can* do this of course, but it is not something that can be done out of the box. You will need to build it in. You can emulate a very basic version of the stewardship console by yourself by loading data to a db table, setting up a JSP page (maybe using Tomcat) to display and allow editing of the data, and then load the edited data from the table onwards.

Re: Running multiple interdependent jobs one after othe

Anonymous — Mon, 17 Jul 2017 06:24:29 GMT

Ok. Thanks for the detailed reply.

Talking about point no. 2 below, the reason we wanna do it is that our client wants it that way

Anyways if I were to choose that option, then on completion of my first job, can I have a 'button' on UI on click of which I can invoke my webpage which loads the data in UI? Or how do I go about it if I want to link that web page?

Re: Running multiple interdependent jobs one after othe

Anonymous — Tue, 18 Jul 2017 05:16:39 GMT

Any replies on this?

Re: Running multiple interdependent jobs one after othe

Anonymous — Tue, 18 Jul 2017 09:22:50 GMT

Your UI can have a submit button that call a rest service. The rest service could trigger your DI job, or even actually do the actual processing, depending on whether your job is processing 1 record on each button submit, or hundreds/thousands of records on button submit.

You can use any web i.e. HTML, JavaScript etc, to call you rest service.

Re: Running multiple interdependent jobs one after othe

Anonymous — Tue, 18 Jul 2017 10:02:12 GMT

No but this not clear. My question is how to connect the JSP page which I make, through Talend. The jsp page should be invoked when my first job is finished and I click let's say a 'Review' button on my Talend screen.

How to do this?

Re: Running multiple interdependent jobs one after othe

Anonymous — Tue, 18 Jul 2017 10:12:35 GMT

What @iburtally was suggesting was that you could use a Talend web service to either trigger your job or carry out the processing. Calling a web service would require simply firing a call to a URL (www.myserver.co.uk:8040/myService or localhost:8040/myService).

I think you are making this a little more complicated than it needs to be though.

Re: Running multiple interdependent jobs one after othe

Anonymous — Tue, 18 Jul 2017 10:15:22 GMT

Talend has products like Talend Data Stewardship for this kind of problem. Or you can use a BPM workflow engine with custom UI.

However, You can have your Talend Job call a tSystem with

chrome.exe google.com command. Change the URL to the one you built and it will open the webpage.

Calling a Talend Job from a UI is straight forward since it is just a java process to call with Java.exe.