Find talend job process name at operating system level

Six Stars

Find talend job process name at operating system level

Hi

I have a job which has to run throughout the day, if it aborts for any reason, it has to be resubmitted automatically, In other ETL tools like Datastage, I would be able to find the ETL job process name at OS level using ps-ef, when i tried that for talend in windows environment using tasklist command, i'm not able to find that Talend job name process which runs at the backed. I know I can create another job which checks if a process is running  but I'm trying to create a non Talend based solution and i don't want to export the job and run it outside TAC.. so how can find at OS level if a job is running. 

 

Thanks

MJ

 


Accepted Solutions
Sixteen Stars

Re: Find talend job process name at operating system level

The answer to your question is the following command on the machine that is running your job....

 

jps -v

On another note, I think you should reconsider your process here. A Talend job is not really suitable to be running all day non-stop. They are meant for batch. You can run a Talend job 1000s of times a day, no problem. If you try to run one continuously for the whole day you will likely end up with memory issues after a while. It is not what Talend DI jobs have been designed for. Your problem sounds to me to be more aligned with a real time (or near to real time) solution. It sounds like a Talend Route (ESB) would be a better fit.

Sixteen Stars

Re: Find talend job process name at operating system level

The first way that springs to mind would not be much of a change to your job structure, but would make it a little more resistant to issues. You could use a Windows scheduler (or Cron on Linux) to run your orchestration job every hour. I assume that the child jobs need to run strictly sequentially (with maybe some interaction with data processed?). As I said, this would not be a massive change for you. The big issue here is solely having a Talend job running as a scheduling device. I assume you have had issues with doing this, hence the original question about monitoring whether it is still running.


All Replies
Sixteen Stars

Re: Find talend job process name at operating system level

The answer to your question is the following command on the machine that is running your job....

 

jps -v

On another note, I think you should reconsider your process here. A Talend job is not really suitable to be running all day non-stop. They are meant for batch. You can run a Talend job 1000s of times a day, no problem. If you try to run one continuously for the whole day you will likely end up with memory issues after a while. It is not what Talend DI jobs have been designed for. Your problem sounds to me to be more aligned with a real time (or near to real time) solution. It sounds like a Talend Route (ESB) would be a better fit.

Six Stars

Re: Find talend job process name at operating system level

Hi
Thanks for responding, to be more specific,I have a main job (Orchestrate job) that has to run from 5.00 AM to 4.30 PM, That orchestrate job triggers a set of 5 jobs sequentially for every 1 hour. The thought behind the question was, I don't want any manual intervention if the main orchestrate job abort, so how do you think I can better design this without the memory issue? please

thanks
MJ

Sixteen Stars

Re: Find talend job process name at operating system level

The first way that springs to mind would not be much of a change to your job structure, but would make it a little more resistant to issues. You could use a Windows scheduler (or Cron on Linux) to run your orchestration job every hour. I assume that the child jobs need to run strictly sequentially (with maybe some interaction with data processed?). As I said, this would not be a massive change for you. The big issue here is solely having a Talend job running as a scheduling device. I assume you have had issues with doing this, hence the original question about monitoring whether it is still running.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.