Five Stars

How do I get job specific logs? (Beginner)

Hello everyone,

I am a beginner in Talend, I currently work on Talend Open Studio for Data Integration.

 

I need to get all informations from my job / subjob and insert them into a table:

JOB_NAME character # Job Name
LOAD_TYPE character # Loading status (Insert / Update)
STATUS character # Job status / under job (OK / KO)
JOB_STIME date # Start date of job / subjob
JOB_ETIME date # End date of job / subjob

 

I managed to get the JOB_NAME thanks to a tMap connected to the tLogCatcher (field : job) but I do not know how to get the other fields despite my research.

In thanking you and apologizing for this trivial question for Talend experts.

 

Regards

  • Data Integration
3 ACCEPTED SOLUTIONS

Accepted Solutions
Eleven Stars

Re: How do I get job specific logs? (Beginner)

This tutorial on the Talend AMC might help you.....

 

https://www.rilhia.com/tutorials/talend-activity-monitoring-console-amc

Rilhia Solutions
Eleven Stars

Re: How do I get job specific logs? (Beginner)

Did you look at the linked tutorials? This one shows you how to switch the AMC on per job (https://www.rilhia.com/tutorials/switch-activity-management-console-amc-functionality-talend-job). Essentially this Talend functionality just needs to be switched on and it will carry out a low level of logging for you automatically. The tutorial I sent you talks about the database tables you need to set up and the tutorial I have posted here shows how to switch it on for your job. Once you have done both, you can see all of the stats generated for your job without having to manually code anything

Rilhia Solutions
Eleven Stars

Re: How do I get job specific logs? (Beginner)

This is easy. Just query your database for the last status. The use the value to decide what to do.

 

Connect your database component to a tFlowToIterate. The tFlowToIterate will produce a globalMap variable holding your status data returned from your query. It will be in the format....

 

((String)globalMap.get("row.column"))

 

Please note "row.column" refers to the name of the row feeding the tFlowToIterate and the name of the column holding the status value. This will need to be changed accordingly. 

From the tFlowToIterate use RunIf connectors to join to a component to inform your users that it cannot be run (maybe a tJava with a System.out command) and to the start component of this job. In the success link use logic like this....

 

((String)globalMap.get("row.column")).compareToIgnoreCase("success")==0

 

....in the failed link use logic like this.....

 

((String)globalMap.get("row.column")).compareToIgnoreCase("failed")==0

 

Then, when you run the job it will check the result of your last run and if it was a success, it will run the main body of the job. If it was a failure it will run the mechanism to to inform of an issue

 

Rilhia Solutions
22 REPLIES
Eleven Stars

Re: How do I get job specific logs? (Beginner)

This tutorial on the Talend AMC might help you.....

 

https://www.rilhia.com/tutorials/talend-activity-monitoring-console-amc

Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

Hello rhall_2_0, 

 

Thanks for your help but I really don't understand your solution. 

I need something in my job like in tMap ou tLogCatcher to log my data no ? 

Eleven Stars

Re: How do I get job specific logs? (Beginner)

Did you look at the linked tutorials? This one shows you how to switch the AMC on per job (https://www.rilhia.com/tutorials/switch-activity-management-console-amc-functionality-talend-job). Essentially this Talend functionality just needs to be switched on and it will carry out a low level of logging for you automatically. The tutorial I sent you talks about the database tables you need to set up and the tutorial I have posted here shows how to switch it on for your job. Once you have done both, you can see all of the stats generated for your job without having to manually code anything

Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

Ok but I already do these tutorials and succeed to have basic infos. 

 

But I need these particular infos : 

LOAD_TYPE character # Loading status (Insert / Update)
STATUS character # Job status / under job (OK / KO)
JOB_STIME date # Start date of job / subjob
JOB_ETIME date # End date of job / subjob

 

How should I do it ? Thank you 

Eleven Stars

Re: How do I get job specific logs? (Beginner)

The only data you have given as an example that cannot be gained from the AMC is the LOAD TYPE. The rest of the information is available. To get this you have several options. This will help you extend the AMC.

 

The AMC uses process ids to log your results (PID). In Talend you have access to the PID in Java code. If you use  a tJava component you can use the code below to print your PID to the output window......

System.out.println("My pid is "+pid);

 

What you can do is add a tPostJob component to your job and attach some components to provide extra logging using this PID. You will need to create your own database table for this. Now if you make the pid a key in your database table, you can link whatever extra information to your AMC logging you want. So you can add your LOAD TYPE data. So long as it is linked to the PID, you can get all of the other standard AMC data really easily.

Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

Ok well, I have succeeded to have my infos but I don't understand how can I get the LOAD_TYPE. 

I understand your explanation (tPostJob component, creation of db ...) 

 

But where the Update / Insert status is stored ? 

 

I would like to have the number of update / insert lines too ? 

 

Thank you for your help. Regards

 

Eleven Stars

Re: How do I get job specific logs? (Beginner)

Look at the Global Variables section of this document (https://help.talend.com/reader/hfF27SBEP9Avc0ofq7IDjw/E6MTIua23KeeVBOs_R0iOA) as an example of how you can use database components (and other components reading and writing data). 

 

If you look at the Studio and look in the bottom left hand corner, you will see a tab called "Outline". Inside there you will see a list of component names (of components used in the job). If you expand these you will see all of the global variables that can be used for each component. If you take a tJava component and drag the variable from that box into the tJava code, you can use it there in code. It will look something like this.....

 

((Integer)globalMap.get("tMSSqlOutput_1_NB_LINE_INSERTED"))

 

 

Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

Hello rhall,

 

I don't have the Outline tab. I am using Talend Open Studio, it can explain that, I don't know. 

I haven't succeeded to define Global variable in my mapping. Can you help me please. 

 

Joined : A part of my job with postgresql database

 

Regards

Eleven Stars

Re: How do I get job specific logs? (Beginner)

A globalMap variable is essentially a value stored in the globalMap HashMap (a data structure). To use this value all you need to know is the name of the component. In the example below, the component is called "tMSSqlOutput_1" the rest of the code below is exactly what you want to use in order to get number of rows inserted.

 

((Integer)globalMap.get("tMSSqlOutput_1_NB_LINE_INSERTED"))
Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

Ok well, is it this mapping which i must do ? (Joined in my message)

1  : Input database

2 : Output database 

tPostgresqlOutput_5 : Output log database 

 

And I put in the tJava : 

((Integer)globalMap.get("2_NB_LINE_INSERTED"));

 

i'm not sure of the name of my component, where can I see it ? Because "2" is an alias I think and when i put 2 in the formula, it doesn't work. 

 Thank you

Eleven Stars

Re: How do I get job specific logs? (Beginner)

To get the name of the component, click on it and you will see it in brackets in the Component tab.

Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

Yes but in my case the name of my table in brackets, doesn't work with the formula : 

 

((Integer)globalMap.get("fact_table_NB_LINE_INSERTED"))

 

 

Eleven Stars

Re: How do I get job specific logs? (Beginner)

Click on your output component and take a screen shot of your component tab. Then post it.

Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

Join in my message. 

Thank you

Eleven Stars

Re: How do I get job specific logs? (Beginner)

You need to use....

 

((Integer)globalMap.get("tPostgresqlOutput_1_NB_LINE_INSERTED"))

Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

YEAAAAAAAH, it's work Smiley Happy

 

Last question : 

I would like to run my job in function of the last return status. 

For exemple, in my AMC table, my last job finished with "success" status, so I can run another time my job. 

But when my last status is "failed", I would like to "block" the launching and inform the users. 

 

Best regards

Eleven Stars

Re: How do I get job specific logs? (Beginner)

This is easy. Just query your database for the last status. The use the value to decide what to do.

 

Connect your database component to a tFlowToIterate. The tFlowToIterate will produce a globalMap variable holding your status data returned from your query. It will be in the format....

 

((String)globalMap.get("row.column"))

 

Please note "row.column" refers to the name of the row feeding the tFlowToIterate and the name of the column holding the status value. This will need to be changed accordingly. 

From the tFlowToIterate use RunIf connectors to join to a component to inform your users that it cannot be run (maybe a tJava with a System.out command) and to the start component of this job. In the success link use logic like this....

 

((String)globalMap.get("row.column")).compareToIgnoreCase("success")==0

 

....in the failed link use logic like this.....

 

((String)globalMap.get("row.column")).compareToIgnoreCase("failed")==0

 

Then, when you run the job it will check the result of your last run and if it was a success, it will run the main body of the job. If it was a failure it will run the mechanism to to inform of an issue

 

Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

Ok perfect, I will check that later. 

 

Concerning my previous question, how can I get the PID in a tFixedFlowInput ? 

Is there a standard variable to get it ? 

 

Thank you Smiley Happy

Eleven Stars

Re: How do I get job specific logs? (Beginner)

You can get the PID from any Talend job using the variable pid. In a tJava component you can print it out to the output using the following code....

 

System.out.println("My PID is "+pid);
Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

I would like to use in tFixedFlowInput because tJava doesn't work sorry.

I have always this error : Syntax error, insert "AssignmentOperator Expression" to complete Expression". 

 

EDIT : I've found, I just write pib Smiley Happy 

 

Thank you

Eleven Stars

Re: How do I get job specific logs? (Beginner)

Sorry, the reason I was using the extra code is that a tJava requires that you do something with the variable.....otherwise you do not see anything. With other Talend components (as you discovered) you can just use the variable.

Rilhia Solutions
Five Stars

Re: How do I get job specific logs? (Beginner)

Ok well, I've succeeded everything thanks to you. 

Very very thank you for your help and explanations.

Best regards.