How to Stop a job when Execution Time is too long

Six Stars

How to Stop a job when Execution Time is too long

I have a job who make something, and i want to add the rule :

 

If the job duration > 10seconds, stop it.

 

To get the start duration, i think tChronometerStart can be a good choice because he have : (Long)globalMap.get("tChronometerStart_1_STARTIME")

 

The problem is the tChronometerStop, because this component help when we want to get the duration of the full job AFTER execution.

I don't know how to write correctly the "stop" duration because after running 10seconds, i want to stop the job.

To stop the job, i think about a tDie with a "if" condition like this : ((Long)globalMap.get("tChronometerStop_2_STARTIME") - (Long)globalMap.get("CURRENT_TIME???"))/1000 > 10 secondes

 

thanks

 

 

 

 


Accepted Solutions
Sixteen Stars

Re: How to Stop a job when Execution Time is too long

OK, what I would do is the following....

1) Use a tPreJob to link to a tJava component. Calculate the current time and set it to the globalMap

2) In your tMap create a tMap boolean variable (between the input and output tables) and use the logic similar to what you have shown in your description. Use your start time globalMap value and compare to the current time (use the TalendDate routine). If the time difference is within 10 secs, set to True, else set to False.

3) Create another tMap output for when the time has expired. Use the tMap output filtering to decide which output to use based on the tMap variable result. For the output when you have run out of time, add any columns you may wish to report have not been done in time and connect that output to whatever component will fulfill your logging. Then connect that component to a tDie using a RunIf with some logic to identify that you do indeed want to kill the process (this is required because outputs from tMap components are always triggered even if there is no data).

 

By the way, I am assuming you are using the Open Source Edition and cannot use parallel processing. If you are using the Enterprise Edition you can use a tParallelize component and carry out some check based on a tLoop. The reason I have shown the way above is that you need to build the check into the main body of your processing if you want it actioned when essentially running on a single thread


All Replies
Sixteen Stars

Re: How to Stop a job when Execution Time is too long

You are going to have to thin about this in a different way. Can you show us a screenshot of your job? What you will likely need to do is add checks to the job processing so that the elapsed time can be checked in the flow of the job. This is because of the flow of processing within Talend Jobs. 

 

Show us your job and we can suggest where and how to implement this

Six Stars

Re: How to Stop a job when Execution Time is too long

my job :

Caps.PNG

if condition : ((Long)globalMap.get("tChronometerStart_1_STARTIME") - (Long)globalMap.get("tChronometerStop_1_DURATION"))/1000 > 10

 

how can i add checks to the job processing?

Sixteen Stars

Re: How to Stop a job when Execution Time is too long

OK, what I would do is the following....

1) Use a tPreJob to link to a tJava component. Calculate the current time and set it to the globalMap

2) In your tMap create a tMap boolean variable (between the input and output tables) and use the logic similar to what you have shown in your description. Use your start time globalMap value and compare to the current time (use the TalendDate routine). If the time difference is within 10 secs, set to True, else set to False.

3) Create another tMap output for when the time has expired. Use the tMap output filtering to decide which output to use based on the tMap variable result. For the output when you have run out of time, add any columns you may wish to report have not been done in time and connect that output to whatever component will fulfill your logging. Then connect that component to a tDie using a RunIf with some logic to identify that you do indeed want to kill the process (this is required because outputs from tMap components are always triggered even if there is no data).

 

By the way, I am assuming you are using the Open Source Edition and cannot use parallel processing. If you are using the Enterprise Edition you can use a tParallelize component and carry out some check based on a tLoop. The reason I have shown the way above is that you need to build the check into the main body of your processing if you want it actioned when essentially running on a single thread

Six Stars

Re: How to Stop a job when Execution Time is too long

it works, thanks

Sixteen Stars

Re: How to Stop a job when Execution Time is too long

No problem. Glad to help

Five Stars

Re: How to Stop a job when Execution Time is too long

Why not just use a tSleep that triggers the tDie when it times out?

Six Stars

Re: How to Stop a job when Execution Time is too long

for tSleep, i think it's a solution, but only if you have "tParrallelize" component : one "Thread" who work, one "Thread" who tSleep and stop.
but this component is not available in open studio talend version.
Six Stars

Re: How to Stop a job when Execution Time is too long

I had a similar scenario where i need to end a job automatically when its execution goes beyond 2 minutes ,

Add a tjavaflex in the main flow before tmap and put the condition (as per your time limit) as in the screenshot  ,

Tjavaflex details :

start code - Initialize a variable that stores the starting time .

Main code - An IF Condition checking whether the difference between the stored variable time and the current time is greater than the time limit (5 sec in my case for testing) , at one point of the main flow this becomes true and the break statement inside it is called which stops the code flow. Also have a context flag say , context.timeout for which flag value 1 is assigned inside the IF condition 

Run-IF - use the condition context.timeout==1 , to trigger tDIE so that u can display a timeout error message

Test Scenario :

In the below attached job screenshot Row generator generates 1 crore records. Since i have given a limit of 5 seconds in tjavaflex , execution stops in the middle (at 45 lakh records) and tdie is triggered.

 

Capture20.PNGRow generator generates 1 crore records. Since i have given a limit of 5 seconds in tjavaflex , execution stops in the middle and tdie is triggered.
Capture25.PNGTjavaFlex Code

Five Stars

Re: How to Stop a job when Execution Time is too long

I do that in a scheduler I wrote for open DI.

 

Allow parallel processing in your job. Then include a tSleep component that calls tDie. You will also need a tDie at the end of the actual process since if it ends before the timeout you don't want to wait on the tSleep.

 

Five Stars

Re: How to Stop a job when Execution Time is too long

how could it handle multiple input data sources? I can't link multiple data to the tJavaFlex.

Five Stars

Re: How to Stop a job when Execution Time is too long

The solution I posted runs in parallel. No tJavaFlex.

Six Stars

Re: How to Stop a job when Execution Time is too long

Even in multi link processing the java flex code will break the entire jobflow and end the job , you just have to give that code in any one of your multiple links , quite easy to end the job by just adding two lines of code.

Five Stars

Re: How to Stop a job when Execution Time is too long

But what if parallel sequences are running and the one with the JavaFlex component ends before the timeout and the others continue to run? Won't this model fail in that example?

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.