Creating detailed log file

Highlighted
Six Stars

Creating detailed log file

Hi All,

I want to create a detailed log/stats file with details such as

Job Name, Source file name, Source file count, Target file name, Target insert count, Time taken for job, start date, end date, Job status, failed status etc and store it either in file or table.

Can anyone help me with the same pls.

 

Thanks!

Employee

Re: Creating detailed log file

Hi,

 

    You can use the logging and error components of Talend.

 

https://help.talend.com/reader/KxVIhxtXBBFymmkkWJ~O4Q/exEndmX5MGHMto5BeyIMhg

 

    Please also go through below link.

https://help.talend.com/reader/SAZUuM203k6SNl8zrDfj5Q/GRS3dWpaYoK_QdY~Mvck5Q

 

    I believe you would also be interested in AMC.

 

https://talendpnp.github.io/architecture-center/reference-architecture/summer-2018-7.0/talend-compon...

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Six Stars

Re: Creating detailed log file

Appreciate your response @nikhilthampi .

All these components give some columns required.

Is there a way we can capture all at once. Instead of adding so many components. 

The reason being, i want to make it reusable across multiple jobs or all jobs.

Employee

Re: Creating detailed log file

Hi,

 

    There is an interesting blog related to this subject.

 

https://www.talend.com/blog/2018/10/09/how-to-implement-a-job-metadata-framework-using-talend/

 

    Why don't you add it as a Joblet so that it can be reused easily with other jobs?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Six Stars

Re: Creating detailed log file

Thanks, @nikhilthampi.

I was referring to the same link. However, I'm not sure how to access tables such as talend_file_tracker,talend_jobs, talend_hwm_tracker, etc.

 

Or if I have to populate these tables, not sure of the components.

 

Appreciate your help.

Employee

Re: Creating detailed log file

Hi,

 

    You will have to setup the necessary tables based on the data extracted from Talend error and log components. The flowchart is depicting the method in which you will have to pass the data from the log components.

 

     Unfortunately I do not have the code handy with me. I would suggest you to build the code after verifying the elements I have already mentioned in the previous posts. If you are facing any specific error, we can guide you during that point of time.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Six Stars

Re: Creating detailed log file

Thanks @nikhilthampi.

 

I did try using multiple components and putting it in one table.

However each component offers different feature and can't connect all component and even if i do i can't make it generic.

And i'm still not able to generate those tables, since i'm not sure how to generate some of the column values.

Could you please guide me.

 

Many thanks!

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog