Six Stars

Running multiple interdependent jobs one after othe

Hi,

I have a case where a Job has to be broken down to two sub jobs as I have to take the output of one job, do some modification (in the output table) and use it as an input to the next job. So, how to do it ? If i try to do something like in the screenshot below then it runs both the jobs simultaneously which I don't want.

multi.jpg

  • Data Integration
22 REPLIES
Ten Stars

Re: Running multiple interdependent jobs one after othe

Right click on TEST1, select Trigger and select "On Subjob OK". Drag this link to the TEMP_STAGING _TABLE component. Done.

Rilhia Solutions
Six Stars

Re: Running multiple interdependent jobs one after othe

No, I think you didn't picked my question properly. What I want is that on running the job only first part should run, i.e from TEST to temp_staging_stable, then I have to do some modifications in the temp_staging_stable, then when I run it gain, now the second part of job should run ,i.e from temp_staging_stable to Employee.

Any suggestions how to achieve this?

Regards

Pankaj

Employee

Re: Running multiple interdependent jobs one after othe

Check for a controlling state, i.e. either a variable, or some data, to identify that the first step is complete and you have done your modifications in the temp_staging_table, then depending on that controlling state, you either run the first subjob or the second subjob. 

 

What you will do is as follows:

tJava -> Run If -> (Your First Subjob)

          -> Run If -> (Your Second Subjob)

 

Yes, they are linked to the same tJava which should not contain any Java code in its body.  It is just a stub to allow you to run to If triggers from 1 point.  In each if trigger, you are checking for the correct controlling state.  You can use context variables, or actually query the DB for some of the modifications you did, before you reach the tJava to decide what is the controlling state for the If condition.

Six Stars

Re: Running multiple interdependent jobs one after othe

Its a bit difficult for me to design and also I want to avoid any Java coding as my client doesn't have any coding skills.

Isn't it possible that I select my firstjob, execute it, then do some modifications in the resultant table, then select my second Job and execute it!

Ten Stars

Re: Running multiple interdependent jobs one after othe

@iburtally was not talking about using any Java. The tJava is just used as a dummy component to drive the conditional RunIfs from. 

 

Why are you designing this as one job? If you have to perform some manual steps between the two processes, why not just write two jobs? Don't get me wrong, it is easily achieved by Talend, but there is no point struggling for ages when two separate jobs will do.

Rilhia Solutions
Six Stars

Re: Running multiple interdependent jobs one after othe

Well that had to be the last resort, i.e. to have two separate jobs for this task as there is a manual intervention involved between tow sub jobs. All I was thinking of a way to possible 'select' one the two subjobs and execute and then repeat it for the next. But that doesn't seems to be the case so I shall go with the option of having two jobs for this.

Thanks anyways. 

Ten Stars

Re: Running multiple interdependent jobs one after othe

What @iburtally described would be fine for this. You can also use context variables to do this. You can use the state of your data in your db to do this. You can use the presence of data in your db to do this. You can do this in SO many ways. You need to give more information about what you have to play with before anyone can be more specific I'm afraid. 

Rilhia Solutions
Six Stars

Re: Running multiple interdependent jobs one after othe

Hi @iburtally

Is it possible that at the end of step1 or first subjob I can see the result in a GUI(i.e all the columns) and can update some value of a column which updates my Database table accordingly and then I can use it as an Input to my second Job?

The purpose is that my client wants some manual approval process at the end of first job i.e. where we have applied some rule and he can see the changes and put a tick mark against the columns which he actually wants for the main table.

Employee

Re: Running multiple interdependent jobs one after othe

Hi,

 

We have a product called Talend Data Stewardship for this which provides GUI to curate the data.  

If you use Talend Data Stewardship, you send your data to the Data Stewardship, then you can login into the UI for Data Stewardship, curate and fix your data, then your second job read the curated data from Data Stewardship and do further processing.

 

If you want to build your own UI, then you can do that too.  But Talend ETL does not provide a UI.  It is supposed to be batch running without a UI.  

 

Six Stars

Re: Running multiple interdependent jobs one after othe

Is Talend Data Stewardship also an Open Source or we need to buy it.

Employee

Re: Running multiple interdependent jobs one after othe

Talend Data Stewardship is only available as a paid product.  However, we include 2 free licenses of it with a Talend Platform product.

Six Stars

Re: Running multiple interdependent jobs one after othe

 Ok.

here i want to ask a couple of questions:

1) If Talend provides the option to customize jobs/ components, where are the options to do it.?

2) If for this job, which we have been discussing long time now(two interdependent jobs), I want to write a customized routine, which will lets's say allow me open up a GUI with the ability to load database table in it(in a tabular structure may be) and I look at the columns and have the ability to change values and click on 'Save' button to make changes in database.

This will do my job. So how difficult it's going to be, if feasible?

Kindly let me know all the options.

Ten Stars

Re: Running multiple interdependent jobs one after othe

1) You build the jobs yourself, so you therefore customise them while building them. If you are talking about runtime dynamic functionality, you have to build this as well. You can use Context variables for this for example. Components come with configuration options and you can create your own components. Creating components is too difficult to explain how to do in a scenario like this. You will need to Google how to do this and then experiment. There are no easy answers to that.

 

2) I am not sure why you would do this. Talend jobs should not be considered GUI enabled. The jobs are not meant to work with GUIs. You *can* do this of course, but it is not something that can be done out of the box. You will need to build it in. You can emulate a very basic version of the stewardship console by yourself by loading data to a db table, setting up a JSP page (maybe using Tomcat) to display and allow editing of the data, and then load the edited data from the table onwards.

 

 

 

 

Rilhia Solutions
Six Stars

Re: Running multiple interdependent jobs one after othe

Ok. Thanks for the detailed reply.

Talking about point no. 2 below, the reason we wanna do it is that our client wants it that way Smiley Happy

Anyways if I were to choose that option, then on completion of my first job, can I have a 'button' on UI on click of which I can invoke my webpage which loads the data in UI? Or how do I go about it if I want to link that web page?

Six Stars

Re: Running multiple interdependent jobs one after othe

Any replies on this?

Employee

Re: Running multiple interdependent jobs one after othe

Your UI can have a submit button that call a rest service. The rest service could trigger your DI job, or even actually do the actual processing, depending on whether your job is processing 1 record on each button submit, or hundreds/thousands of records on button submit.   

 

You can use any web i.e. HTML, JavaScript etc, to call you rest service.

Six Stars

Re: Running multiple interdependent jobs one after othe

No but this not clear. My question is how to connect the JSP page which I make, through Talend. The jsp page should be invoked when my first job is finished and I click let's say a 'Review' button on my Talend screen.

How to do this?

Ten Stars

Re: Running multiple interdependent jobs one after othe

What @iburtally was suggesting was that you could use a Talend web service to either trigger your job or carry out the processing. Calling a web service would require simply firing a call to a URL (www.myserver.co.uk:8040/myService or localhost:8040/myService). 

 

I think you are making this a little more complicated than it needs to be though. 

Rilhia Solutions
Employee

Re: Running multiple interdependent jobs one after othe

Talend has products like Talend Data Stewardship for this kind of problem.  Or you can use a BPM workflow engine with custom UI.  

However, You can have your Talend Job call a tSystem with

 

chrome.exe google.com  command.  Change the URL to the one you built and it will open the webpage.  

 

Calling a Talend Job from a UI is straight forward since it is just a java process to call with Java.exe.   

 

Six Stars

Re: Running multiple interdependent jobs one after othe

Hi @rhall_2_0,

I have one jsp page which shows my data in the temporary table and allow my client to do some processing in it and save the data in DB. So. I am not properly clear how can Talend help me launch this page by using some internet components.

I am not having any 'Web Service' at this moment. Need I first create a Web service for my one jsp or what?

And please, if possible help me with a sample job expalining your statement - "Calling a web service would require simply firing a call to a URL (www.myserver.co.uk:8040/myService or localhost:8040/myService). "

Thanks.

 

Ten Stars

Re: Running multiple interdependent jobs one after othe

If you want to launch this from a Talend job you will either need to have the job running on the machine that the user is using OR you will need to prompt the user to open a URL themself. @iburtally showed you how you can open a web page using the tSystem component.

 


For tutorials on Web Services in Talend, there are plenty online and on here. Take a look at some of these and get familiar with Web Services and it will help you figure this out.

Rilhia Solutions
Six Stars

Re: Running multiple interdependent jobs one after othe

Hi @iburtally,

When I try to run the command chrome.exe google.com on a tSystem component, I am getting the following errors:

google.jpggoogle1.jpg