logging Talend job execution on a Database

hi,
could you tell me if it is possible to log Talend job as they process in a database table?
or to a file that can later be imported into a db?
thanks
7 REPLIES
Employee

Re: logging Talend job execution on a Database

Hello,

Yes, you can use the tStatCatcher and link it to a DBG Output component.
Regards,
Michaël
One Star

Re: logging Talend job execution on a Database

What is the structure of the talend_logs, talend_stats and talend_meters tables to log execution events inside a database ?
How is it possible to use the "Stats & Logs" tabs in the properties panel of each job ?
Is it related to the option in the Talend/preferences ?
One Star

Re: logging Talend job execution on a Database

I need to extract basic information about a job every time its run and output this into a DB.
I need information such as the following:
Username - of the person running the Job
Project Name
Job Name
Start Time
End Time
Row Count
Result ? Success or Failure.
What would I create this Job very simply?
Community Manager

Re: logging Talend job execution on a Database

Hi dna1718
Username: the job don't know who will run the job, you can use a tMsgbox to ask for the username at the beginning of job.
Project Name: there is a global variable called projectName, you can use this variable.
Job Name: there is a global variable called jobName!
Start Time and End Time: use tChronometerStart and tChronometerStop components to populate the job start time and end time. (see demo in topic: http://www.talendforge.org/forum/viewtopic.php?id=10032)
Row Count: use tFlowMeter and tFlowMeterCapture to populate the number of specific row.
At the end of job, use a tFlowInputFlow to generate one row including these columns username, project name, job name, start time, end time, and row count.
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: logging Talend job execution on a Database

Thanks for that. Much appreciated.
Do you know the variable that holds the row count?
Community Manager

Re: logging Talend job execution on a Database

Thanks for that. Much appreciated.
Do you know the variable that holds the row count?

Please learn tFlowMeter and tFlowMeterCapture components in the user documentation.
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: logging Talend job execution on a Database

Several questions about logging:
(1) What is the best way to log events that aren't considered component failures by Talend --- for instance apply functional validations to input data that wouldn't be considered a schema failure per se, or check to make sure the expected number of output records was generated based on the input, etc. ? Should I use tJava or tJavaRec components to manually enforce those edits, and if so how would I get my error messages to tLogCatcher ?
(2) What is best way to have multiple levels of verbosity in my log output ? Is there an input parameter to explicitly tell the job to generate "Debug" level warning messages vs. just regular ones? Is there any way to code the job such that it automatically generates Debug level messages if any errors are encountered, or do I have to code my shell script to re-run the job in Debug mode if an error was encountered the first time?
(3) In general, is it best practices to rely on tWarn and tDie components to generate ALL of my log output, or is there a more elegant/efficient way to code that ? I know that sounds like a strange question given all the documentation on logging, but I thought I heard a Talend consultant once say something that made me think that those should not be over-utilized...
Thanks!
Chris