One Star

Talend as part of another application

Hey I have been using talend a little and I want to use it as part of another Eclipse application. Pretty much I want to take a job and call it from another java program. I could use system to do this but, that is not very portable and I need my application to be portable. Is there any good way to use a talend job inside another java program?

33 REPLIES
Community Manager

Re: Talend as part of another application

Hello lotu
Thanks for your interesting in Talend.
You can not only call a Java program in a Talend job, but also call a Talend job in a Java program. You can simply follow the steps below:
1) Create a simple job in TOS.(see screenshot1)
2) Export the job Java script.(see screenshot 2 and 3)
3) Unzip the Java script file on current folder.
4) Launch Eclipse and create a new Java project, then create a class 'Talend.java'.(see screenshot4)
5) Right click on test and select 'Build Path'-->'Configure Build Path'. (see screenshot 5)
6) Click on 'Add External Jars' button to import the jar file 'myjob_0_1.jar' (under myJob_0.1\myJob_0.1\myJob) and all the jar files (under myJob_0.1\myJob_0.1\lib) (see screenshot 6 and 7)
7) Type in the Java code in class 'Talend.java' (see screenshot 8)
public class Talend
{
public static void main(String[] args)
{
shong.myjob_0_1.myJob.main(new String[]{});
}
}

Here you will notice that:
shong: Talend project name
myjob_0_1: jar file name
myJob: Talend job name
8) Finally, run the Java program.(see the screenshot 9)
Best regards
shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Talend as part of another application

Hi everybody !
I'm very interisting by this solution but I'm interisting by two other things.
The first is I would like give argument for the context param
And the second my job finish by a tBuffertOutput component and I would like to recup the data.
How can I do ?
Thank you !
One Star

Re: Talend as part of another application

Hi togethter,
I wrote a integration for a talend job to is it in the Job Scheduler.
One part of the integration is a java wrapper to easily call a talend job (because additional you have to solve (depending of your configuration) some problems with the class path.
tBuffetOuput is not possible but may be worth a closer look.
If you would like to have more information you can contact me by PM. I'll publish the code (as open source) as soon I get time to do it ...
Bye
Volker
One Star

Re: Talend as part of another application

I published a first version of the code at sourceforge http://sourceforge.net/projects/talendjobschedu/.
If you have any question please contact me (until I wrote a documentation).
Bye
Volker
One Star

Re: Talend as part of another application

Hi Volker
Nice work you have done there.
I had an analog Wrapper to implements and found your post, and downloaded your source code.
To go further, did you think about how Talend could notify progress to your application.
Some "callback" interface allowing the app to display job progress for instance.
I will try something around tSocketOutput and a RMI listener in app or in the Wrapper.

Regards
One Star

Re: Talend as part of another application

Hi there,
I'am just evaluating the job scheduler Volker was takling about and I was starting to write a wrapper class too.
I didn't test yours yet, but does it works with the new version 3.1.x of Talend, I notice some stuffs changed.
If you have some documentation about it, it would help me too to get started (conventions about context variables and sos params).
I'm deeply interested in any integration of talend job, with other opensource plateforms.
I worked on an GUI TalendJobLauncher based on the .item xml source file to get parameters.
I don't know if there is some neat way to split a talend job in steps a third party product could control, (apart from creating distinct jobs of course).
Regards
One Star

Re: Talend as part of another application

Did anybody tried an integration with http://community.intalio.com/ an open source BPMS ?
One Star

Re: Talend as part of another application

Hi Morbo,
I didn't test the scheduler with 3.1. But I set it just now on my ToDo list. I can't believe that this will be a big effort. But actual there much work to do at my job and that means less spare time :-|
If you have any detailed question, just ask. I'll monitor this thread (I actual don't manage to read all) but you could also send me a pm.
Bye
Volker
One Star

Re: Talend as part of another application

7) Type in the Java code in class 'Talend.java' (see screenshot 8)
public class Talend
{
public static void main(String[] args)
{
shong.myjob_0_1.myJob.main(new String[]{});
}
}

shong

Thanks for that Smiley Happy
How does one also specify the path to any relevant context files that might be required using this approach? I have seen a few threads about this but nothing from those seems to be helping me very much (they appear to work within batch files but I've not seen mention of them regarding an integrated project).
For example - Talend Open Studio will deliver a properties file such as job/jobname/context/Default.properties. I copy those properties into my project but can't seem to get the job to pick them up using the switch '--context_param file=/properties/Default.properties'
Any help appreicated.
cheers,
neil
One Star

Re: Talend as part of another application

Hi neil,
sorry for my late answer. I never used --context_param option. But your job will search inside the classpath for the file. So you must ensure that even your properties files are in the cp.
Bye
Volker
One Star

Re: Talend as part of another application

Hi,
How can I compile/modify and run my job outside Talend?
I'm fairly new to Java but what I want do to is to modify my job so that it prompts the user to enter dates to be used in the tGoogleAnalytics component. And I think I can only do this outside Talend(?).

Thanks.

Regards,
VL
One Star

Re: Talend as part of another application

Hello,
U could use a tJava with :
context.myValue = javax.swing.JOptionPane.showInputDialog("Please input a value");
One Star

Re: Talend as part of another application

Thanks Morbo.
That might do the trick, however, how do I make the components wait for the user input?

(sorry if this is getting a bit off topic, I still want to know how to use the generated code in another tool)
One Star

Re: Talend as part of another application

The prompt will block the process until the user confirm the dialog.
I've made an Talend job GUI launcher for exported jobs (with sources included) which asks for context values marked as "prompt". It's a eclipse RCP project, I might publish it someday.
One Star

Re: Talend as part of another application

OK, so maybe my question should be how to rearrange the flow of the components(?).
The tGoogleAnalytics is being run first and, obviously, fires an error.
Again, thank you.


Edit: managed to control the flow with the "OnComponentOk" flow. Is this the right way?
One Star

Re: Talend as part of another application

Place a tjava as the first component, it should be green, then use a trigger "onSubJob ok" to link to the rest of the job.
One Star

Re: Talend as part of another application

Can anybody tell us the way to launch a TOS job which uses Default.properties outside of TOS?
One Star

Re: Talend as part of another application

I made a GUI job launcher a while ago.
http://www.assembla.com/wiki/show/inserm-sirh-talend-joblauncher
I didn't test it with the newest version of TOS though.

Re: Talend as part of another application

How does one also specify the path to any relevant context files that might be required using this approach? I have seen a few threads about this but nothing from those seems to be helping me very much (they appear to work within batch files but I've not seen mention of them regarding an integrated project).
For example - Talend Open Studio will deliver a properties file such as job/jobname/context/Default.properties. I copy those properties into my project but can't seem to get the job to pick them up using the switch '--context_param file=/properties/Default.properties'
Any help appreicated.
cheers,
neil

i try to use project 'Implicity Loading' and set the properties file to be collected from the OS variable (System.getenv("TALEND_INITIAL_CONTEXT")
the variable has to exist on the server running the job and contain the path to a file like: c:/somefolder/talend_properties/context_default.properties
this contains file variables for a database connection; from this DB i can collect all other variables that i need.
it is handy since all variables can be changed in the DB without editing Talend Jobs.
you have to make sure that any Context that you use are loaded from the DB.
ris it clear?
Nicolas

Re: Talend as part of another application

I published a first version of the code at sourceforge http://sourceforge.net/projects/talendjobschedu/.
If you have any question please contact me (until I wrote a documentation).

hi Volker,
i had a look on sf.net and i can not find any file related to this project. could you please post a link to the location of these files?
many thanks,
Nicolas
One Star

Re: Talend as part of another application

Hi
I nedd call Talend inside an java program, but couldn't use.
Can give me more explanation.
I followed all the steps on slides but I can't running after the slides 8 and 9. What
can I be doing wrong?
My java programming don't call the talend.
Thanks!
One Star

Re: Talend as part of another application

I wrote this article on my blog : http://www.tchogy.fr/2011/03/talend-integration-dans-une-application-j2ee/
It explains how to launch dynamically a job from a J2EE application.
One Star

Re: Talend as part of another application

Nice article!
Thinking of an enhancement for a future version of TOS related to the generated java code of the job... It would be nice to be able to provide hooks to the job (regular property with getter/setter) in order to use dependency injection (and be able to load a Talend job from an application context provided by the Spring Framework). -- More on this in the comment I left on your article.
I like the direction where this is going...
Regards,
Daniel
One Star

Re: Talend as part of another application

I wrote this article on my blog : http://www.tchogy.fr/2011/03/talend-integration-dans-une-application-j2ee/
It explains how to launch dynamically a job from a J2EE application.

Shame it's not in English.
One Star

Re: Talend as part of another application

Hi thank you for your topic
i have a problem doing a task with java
indeed i expotred the job in java and when i execute it it gives me the following error
Exception in component tMysqlOutput_1
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Cannot load connection class because of underlying exception: 'java.lang.NumberFormatException: For input string: "null"'.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.Util.getInstance(Util.java:384)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:929)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:298)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at importationsql.job1_0_1.job1.tFileInputDelimited_1Process(job1.java:712)
at importationsql.job1_0_1.job1.runJobInTOS(job1.java:1212)
at importationsql.job1_0_1.job1.main(job1.java:1066)
at Talend.main(Talend.java:9)
Caused by: java.lang.NumberFormatException: For input string: "null"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at com.mysql.jdbc.NonRegisteringDriver.port(NonRegisteringDriver.java:814)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:289)
... 6 more

Help me please
One Star

Re: Talend as part of another application

HI,
I tried this solution and it works but not with Jobs that use contexts.
I want to develop an interface in java with eclipse where it will be easier to call jobs with defaults contexts (context can be alterable).
I use contexts for the connection to the data base, to define a paths....
So i would know if someone find how to call jobs with contexts with this code?
(I already know how to execute then in command line in windows terminal)
Thanks
Hello lotu
Thanks for your interesting in Talend.
You can not only call a Java program in a Talend job, but also call a Talend job in a Java program. You can simply follow the steps below:
1) Create a simple job in TOS.(see screenshot1)
2) Export the job Java script.(see screenshot 2 and 3)
3) Unzip the Java script file on current folder.
4) Launch Eclipse and create a new Java project, then create a class 'Talend.java'.(see screenshot4)
5) Right click on test and select 'Build Path'-->'Configure Build Path'. (see screenshot 5)
6) Click on 'Add External Jars' button to import the jar file 'myjob_0_1.jar' (under myJob_0.1\myJob_0.1\myJob) and all the jar files (under myJob_0.1\myJob_0.1\lib) (see screenshot 6 and 7)
7) Type in the Java code in class 'Talend.java' (see screenshot 8)
public class Talend
{
public static void main(String[] args)
{
shong.myjob_0_1.myJob.main(new String[]{});
}
}

Here you will notice that:
shong: Talend project name
myjob_0_1: jar file name
myJob: Talend job name
8) Finally, run the Java program.(see the screenshot 9)
Best regards
shong
Community Manager

Re: Talend as part of another application

hi zanka131
Take a look at this topic to know how to pass value to context variable when calling Talend job in Eclipse.
http://www.talendforge.org/forum/viewtopic.php?id=10915
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Talend as part of another application

Thanks, Smiley Very Happy
Until now, I was able to load file.properties of the Job with the java.util.properties but after that I didn't know how to use it in the main which launch the Job.
Thanks again, it works very well and with less encoding line. Smiley Wink
One Star

Re: Talend as part of another application

Thanks for the nice post! I actually was able to run a talend job from within Java application.
However, I have a question. When we pass context variables to a job which is run from withing
another java app, can we pass variables other than String? For example Java objects.. like File, or List?