Best way to report a job's results

One Star

Best way to report a job's results

Hi,
I'm looking for the best way to log and report how a specific run of a job went.

For example :

Job 1:
- Schema check for input file xxxxx failed : did not produce output (14 lines could .

Job 2 :
- Produced output for 1925 lines out of 1940 lines in input.
- Error getting mapping for the field "AAA" in 6 lines - did not produce output. check specific errors log file.
- Error getting mapping for the field "BBB" in 9 lines - did not produce output. check specific errors log file.




I looked at various ways but I did not find any best-case solution :
[list=*]

  • tWarn : this will output many debugging data. I can filter them with a tMap, but it's a little complicated to output my statistics no?

 

  • tFileOutputPDF : this will output a table, not logging information as I wish to do.

 

  • tFileOutputOdt : could not make it work, I had Null Pointer exceptions every time.

 

  • tJava : this may be the final solution, but it seems a little too "low level" and I'd prefer a more graphical way, without the need to open the file and so on...


Thank you for your advices.

Moderator

Re: Best way to report a job's results

Hi,
Do you want to log a talend job result?
If you want to store the result information into a file or DB table, it is very convenient to set the log in job view to output the record into file or table.
Statistics table records details like begin Time, End Time which is generated by any job.
Logs table records the messages from Java Exception, tDie and/or tWarn (prerequisite: there must be an error in your job).
Flow Meter table records the number of rows processed in the defined flow(prerequisite: you have to use tFlowMeter in your job).

Best regards
Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Best way to report a job's results

Sweet, thanks a lot.
I updated my project to output volumetrics in a file, and added tflowMeter items in my jobs : "input lines #" vs "output lines #", various error types encountered along the way (in the lookups of the tMaps I have), ...
Now everything goes to the meter file.
To ease the reading, I created a job to process this CSV file and only output the metering label and the count to an Excel File, sorting the lines and removing all lines with a meter value of 0 (if there is no lookup error encountered, I don't need to see a message in my final file). 
So that satisfies my logging needs perfectly, as I can check the correct execution of my 15+ jobs at a glance in the resulting Excel file Smiley Very Happy
The only "issue" I found using this method, is that catching the volumetrics and logs globally at the project level instead of doing it so at the job level (using a tFlowMeterCatcher), there is no header outputted to the resulting CSV file... Maybe it would be nice in a future version of Talend to let the user choose to add the header or not in the menu you showed me.
Regards,
Philippe.
Moderator

Re: Best way to report a job's results

Hi,
The only "issue" I found using this method, is that catching the volumetrics and logs globally at the project level instead of doing it so at the job level (using a tFlowMeterCatcher), there is no header outputted to the resulting CSV file... Maybe it would be nice in a future version of Talend to let the user choose to add the header or not in the menu you showed me.

You can do it at the job level. Go to job review-->Stat&Logs, you will find it.


Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Best way to report a job's results

Hi,
What I meant is that if I set the "global" log/metering catching at the project or at the job level, no header is present in the file output.
This is because the only control over the resulting CSV file is the name of it. There is no "Include Header" option.
Using the project/job settings, I obtain this file for example:

2016-04-21 09:18:03;xPvETY;xPvETY;xPvETY;3008;testProject;test;_qVyhMOa0EeWKMJszPPuHvA;0.1;testContext;tFlowMeter_7;Input lines;64;;



On the other hand, if I do it more locally, and use a tFlowMeterCatcher, I have to use a tFileOutputDelimited (or Excel, or whatever), and in this component I do have the "Include Header" option...
Using the tFlowMeterCatcher and tFileOutput[...], I have this file for instance :

moment;pid;father_pid;root_pid;system_pid;project;job;job_repository_id;job_version;context;origin;label;count;reference;thresholds
2016-04-21 09:18:15;xPvETY;xPvETY;xPvETY;3008;testProject;test;_qVyhMOa0EeWKMJszPPuHvA;0.1;testContext;tFlowMeter_7;Input lines;64;;


As I said, it would be nice to be able to "Include Header" and maybe other options at the project/job level in the Stats & Logs tab.

I don't know if I explained myself :-p

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog