[resolved] tFlowMeterCatcher gets duration of the job?

One Star

[resolved] tFlowMeterCatcher gets duration of the job?

Hi,
I am wondering which component I used in order to obtain the duration of the job? I used tflowMeterCatcher, I am able to obtain the job name, the start date & time, the total number of rows processed; but cannot obtain the duration of the job.
Below is my workflow:
tMySQLInput --> Main --> tMap --> tFlowMeter --> tOracleOutput --> Reject --> tFileOutputDelimited --> if --> tSendMail
tflowMeterCatcher --> tMap --> tOracleOuptut
Any help is really appreciated! Thanks!!

Accepted Solutions
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

klmc,
I think I know why you're getting the null exception. It might be that when a log event is triggered, the durations in your tChronometerStart and tChronometerStop are not yet set. Keep in mind that those are set after the component finished executing.
Try accessing them after a tPostJob component:
tPostJob ------onComponentOk-----> tFixedFlowInput ------main---------> tOracleOutput
In your tFixedFlowInput, you access the tChronometerStart and tChronometerStop. This will make sure that they finished executing.
Hope this helps!
Rabih

All Replies
Community Manager

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hello
I am wondering which component I used in order to obtain the duration of the job?

tChronometerStart and tChronometerStop fit your need.
Best regards
shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hi Shong,
Thanks again!! I used tChronometerStart & tChronometerStop; it gives the duration of a job. However, how do I link tStatsCatcher & tFlowMeterCatcher to obtain the following information on my job and output the data into a table in the database:
Job Name
Job Start Time
Job End Time
Row Processed
I tried to use tFlowMeterCatcher with tStatCatcher as a lookup with the tMap and join them with the job name. However, when executed only job name and the number of row processed are saved in the database. I am not able to obtain the Job Start Time & Job End Time? Is the tStatCatcher moment refer to Job Start Time or Job End Time?
Thanks again!
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hi klmc,
One possibility of doing this is that in your tMap, set the startTime and endTime values in the output row which you can get from tChronometerStart and tChronometerStop.
However, I have previously raised a bug on those two components since I wasn't able to extract their values in a Java. But Shong suggested a workaround to get their values. For more information on this, check the forum topic http://talendforge.org/forum/viewtopic.php?id=9645
Hope this helps!
Rabih Dagher
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hi rdagher,
Thanks for your help! I created the flow as
tStatCatcher_1 --> Main --> tMap2 --> Main --> tOracleOutput_2
In tMap2, I created 2 variable expression:
globalMap.get("tChronometer_Start1")
globalMap.get("tChronometer_Stop1")
After running the process, the following data is saved in the database:
Job Name Start Time Stop Time Message Type Message
j_rating 1267235883244 1267235883603 end success
Can you or anyone please help. How can I generate the Start Time & Stop Time in Date format? In addition, I really don't know to join the tFlowMeterCatcher with tStatCatcher to get the total number of rows processed to the database.
Thanks again!!!
Community Manager

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hello
How can I generate the Start Time & Stop Time in Date format?

Convert long type to date type on tMap, for example:
new java.util.Date(row1.startTime)
Best regards
shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hi Shong,
I tried the solution you mentioned in the thread; however, I encountered the following java exception error:
Starting job j_rating at 13:28 04/03/2010.
connecting to socket on port 4119
connected
Exception in component tMap_2
java.lang.NullPointerException
at etl_gne.j_rating_0_1.j_rating.tStatCatcher_1Process(j_rating.java:6671)
at etl_gne.j_rating_0_1.j_rating.runJobInTOS(j_rating.java:7099)
at etl_gne.j_rating_0_1.j_rating.main(j_rating.java:6943)
the end is near
Job j_rating ended at 13:28 04/03/2010.
Can you please help me again!
Thanks!
Community Manager

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hello
In tMap2, I created 2 variable expression:
globalMap.get("tChronometer_Start1")
globalMap.get("tChronometer_Stop1")

1)The correct expression should be:
globalMap.get("tChronometerStart_1")
globalMap.get("tChronometerStop_1")
Please check it!
2)Why don't you define the data type as long/Long replace Ojbect when you define the two vars? and set the expression as:
(Long)globalMap.get("tChronometerStart_1")
(Long)globalMap.get("tChronometerStop_1")
then conver long to Date on output table:
new java.util.Date(Var.start_Time)

Best regards
shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hi Shong,
I did what you suggested; however, I still cannot obtain the tChronometer Start & Stop time in DATE format. When I run the job, I encountered the following error:
Starting job j_rating at 11:52 08/03/2010.
connecting to socket on port 3992
connected
Exception in component tMap_2
java.lang.NullPointerException
at etl_gne.j_rating_0_1.j_rating.tStatCatcher_1Process(j_rating.java:6681)
at etl_gne.j_rating_0_1.j_rating.runJobInTOS(j_rating.java:7124)
at etl_gne.j_rating_0_1.j_rating.main(j_rating.java:6968)
the end is near
Job j_rating ended at 11:52 08/03/2010.
Did I miss anything? Can you please help!
Thanks a bunch!
Community Manager

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hello
Does the tChronometerStart_1 and tChronometerStop_1 components exist in your job? For example:
tChronometerStart_1
|
onSubjobok
|
tJava( or any subJobs)
|
onsubJobok
|
tChronometerStop_1
tStatCatcher_1 --> Main --> tMap2 --> Main --> tOracleOutput_2
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hi Shong,
The tChronometerStart_1, tChronometerStop_1 and tStatCatcher_1 component exist in my job. Did I miss anything?
Thanks again!
Community Manager

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hello
Don't define the vars on tMap, use the global var directly on expression, for example:
new java.util.Date((Long)globalMap.get("tChronometerStart_1"))
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hi Shong,
Thanks for the quick response! I tried as what you suggested and add the coding in the expression; however when I ran the job, it still gave the following error:
Starting job j_rating at 09:44 09/03/2010.
connecting to socket on port 4209
connected
Exception in component tMap_2
java.lang.NullPointerException
at etl_gne.j_rating_0_1.j_rating.tStatCatcher_1Process(j_rating.java:6683)
at etl_gne.j_rating_0_1.j_rating.runJobInTOS(j_rating.java:7114)
at etl_gne.j_rating_0_1.j_rating.main(j_rating.java:6958)
the end is near
Job j_rating ended at 09:44 09/03/2010.
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

klmc,
I think I know why you're getting the null exception. It might be that when a log event is triggered, the durations in your tChronometerStart and tChronometerStop are not yet set. Keep in mind that those are set after the component finished executing.
Try accessing them after a tPostJob component:
tPostJob ------onComponentOk-----> tFixedFlowInput ------main---------> tOracleOutput
In your tFixedFlowInput, you access the tChronometerStart and tChronometerStop. This will make sure that they finished executing.
Hope this helps!
Rabih
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

Hi Rabih,
Thanks for your help! I will definitely try the method you suggested; however, I have a question regarding the tPostJob component! Does tPostJob component trigger only when a job complete without any errors?
Really appreciate everyone who help me in this thread!!
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

klmc, the tPostJob component is triggered after the main job completes, no matter if there was or not an error.
One Star

Re: [resolved] tFlowMeterCatcher gets duration of the job?

I need to extract basic information about a job every time its run and output this into a DB.
I need information such as the following:
Username - of the person running the Job
Project Name
Job Name
Start Time
End Time
Row Count
Result ? Success or Failure.
What would I create this Job very simply?