One Star

Run Job continuously with Java input

How can I create a job which is actually called from Java and the data gets passed from Java as well. The job would need to run continuously and shouldn't stop and I should be able to pass the data from Java to Talend.
Basically I'm asking for an equivalent to the "Injector" step in Pentahos Data Integration Tool.
http://wiki.pentaho.com/display/EAI/Injector
Thanks
7 REPLIES
Community Manager

Re: Run Job continuously with Java input

Hi
Yes, it is possible, you are able to call Talend job from external Java application and pass data to Talend job, see
http://www.talendforge.org/forum/viewtopic.php?id=10915
To continue the job forever, you can use a tInfiniteLoop component in the job or write similar code to fire the call of Talend job in Java. It totally depends on your request.
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Run Job continuously with Java input

Thanks for the links. I've read through them but wasn't quite able to find the answer to my requirements. Let me be a little more specific:
1. An existing Java application reads data (Objects, XML ..) from a message queue in real-time.
2. The data then gets transformed into a useful format in the Java application. (whatever can be used to be passed to Talend, maybe a String array?).
3. The application then passes the transformed data to a Talend job which does the rest (lookups and loading of dimensional and fact tables).
So basically what I need to do is start a Talend job which is continuously running and ready to receive data at any second. This might be 1 record or
a 1000 records. So it's not gonna scale if I run a new Talend job every time I get new data. So the job should be running all the time and whenever
data gets passed to the job it should be processed. This means that while the job is currently loading tables with some data, new data can already be
processed by the first components of the same job.
So the trigger to pass the data is the Java application, which means Talend doesn't know when it gets the data.
- So how do I continuously pass in data to a Talend job running from Java?
- What happens when the "main" method of a job gets called from Java and what other methods are available?
- How do I receive or process data in Talend that has been passed in by Java?
Community Manager

Re: Run Job continuously with Java input

Hi
The current situation is that the data are always passed to the job once the job is called in the Java application, the data are passed via string array and call the method runJob(String[]), see
http://talendforge.org/forum/viewtopic.php?id=5932
We are able to schedule the job and make job run all the time, this is not a problem!
How do pass data to Talend job and fires next processing once it receive data? Just a thinking, we hold a socket listener in the job using tSocketxxx component and the job is running all the time, send message to the socket in the Java application, the job will fires next processing once the socket receive message.
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Run Job continuously with Java input

But what happens if I keep on calling runJob(String[]) with every row I want to pass to Talend?
Will it create independent job runs all the time which means different instances of the job run at the same time and are not aware of each other?
If so, that wouldn't be a solution for my problem since I will have to use the same job all the time because the cache being used in dimensional lookup/insert.
The functionality I'm looking for is kind of similar to what a tJMSInput does, isn't it? The same job listens to new messages and processes them when they arrive, correct? So in theory it should be possible to pass data to a talend job from java without having to start it more than once.
It's getting more and more confusing ... I'm sorry Smiley Happy
Community Manager

Re: Run Job continuously with Java input

Hi
The job will be executed many times with different instances of the job if you call runJob(String[]) with every row. For me, you should schedule the job, only start the job one time and run all the time. As you said, the job holds a message listener and process next subjob once a new message arrives.
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Run Job continuously with Java input

Hi ,
I have 2 Query
1. Is their any component in talend that supported all the data base. i found tdbinput , tjdbcinput but which one is best tdbinput required odbc driver & tjdbcinput required jar file.
2. I want to load mysql data into cache file . I am using thashoutput it stored data in cache memory and tcacheoutput it stored data in cache memory - global buffer variable name.
any way to make a file in cache memory(we want to create a file in memory not in physical path).
because i want to upload the data in ts3bucket .
Thanks,
Naveen Dewase
Moderator

Re: Run Job continuously with Java input

Hi Naveen Dewase,
I have replied your related forum Forum 29990, hope it will helpful for you.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.