Studio: How to log Job output to a file as well as the console

Problem Description

Studio logs output from the tLogRow component to the Run tab console, as shown below. But if you log too many statements, for example, more than 1,000 statements, Studio can slow down or hang.

 

studio-console-shows-job-output.png

 

Solution

Log the Job output to a separate log file, by navigating to Studio > File > Edit Project properties > Log4j. Your Studio will look like this:

 

studio-log4j-configuration.png

 

  1. Create a FILE appender section and add the following lines:

    <!-- FILE appender -->
    <appender name="FILE" class="org.apache.log4j.FileAppender">
      <param name="file" value="c:/temp/talend_job.log" />
      <param name="append" value="false" />
      <param name="threshold" value="debug" />
      <layout class="org.apache.log4j.PatternLayout">
        <!-- 'p' is Priority: INFO, DEBUG, TRACE, etc. -->
        <!-- 'c' is Category: fully qualified project and job name -->
        <!-- 'm' is Message: the message the component is logging -->
        <!-- 'n' is Line Separator: e.g. the newline character -->
        <param name="ConversionPattern" value="[%-5p]: %c - %m%n" />
        <!-- <param name="ConversionPattern" value="%m%n" /> -->
      </layout>
    </appender>

    Notice that you can customize the output path, "file", for example, "c:/temp/talend_job.log".

     

  2. Create an <appender-ref ref="FILE"/> section under the 'Root logger' section at the bottom:

    <!-- Root Logger --> 
     <root>
     <priority value ="warn"/>
     <appender-ref ref="CONSOLE"/>
     <appender-ref ref="FILE"/>
     <!-- <appender-ref ref="LOGSTASH"/> -->
    </root>

    You will already have the CONSOLE appender section.

     

    These sections of your file should look similar to this:

    <!-- CONSOLE appender -->
    <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
    <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="[%-5p]: %c - %m%n" />
    </layout>
    </appender>
    ...
    <!-- FILE appender -->
    <appender name="FILE" class="org.apache.log4j.FileAppender">
      <param name="file" value="c:/temp/talend_job.log" />
      <param name="append" value="false" />
      <param name="threshold" value="debug" />
      <layout class="org.apache.log4j.PatternLayout">
        <!-- 'p' is Priority: INFO, DEBUG, TRACE, etc. -->
        <!-- 'c' is Category: fully qualified project and job name -->
        <!-- 'm' is Message: the message the component is logging -->
        <!-- 'n' is Line Separator: e.g. the newline character -->
        <param name="ConversionPattern" value="[%-5p]: %c - %m%n" />
        <!-- <param name="ConversionPattern" value="%m%n" /> -->
      </layout>
    </appender>
    ...
    <!-- Root Logger --> 
     <root>
     <priority value ="warn"/>
     <appender-ref ref="CONSOLE"/>
     <appender-ref ref="FILE"/>
     <!-- <appender-ref ref="LOGSTASH"/> -->
    </root>

     

  3. Open your Job. In the Run view, select the Advanced settings tab. Select the log4jLevel check box, and from the drop-down list, set the value to Debug.

    set-log4jLevel-logging-studio-run-advanced-settings.png

 

Alternatives

The following alternates redirect the console output to other places instead of creating a new place to output the information.

 

Alternate solution 1

As shown in this Community discussion forum thread, you can do this with a little Java code in a tJava component at the beginning of your Job. For example:

java.io.File outputFile = new java.io.File("c:/temp/my_job_output.txt");
System.setOut(new java.io.PrintStream(new java.io.FileOutputStream(outputFile, true), true));

 

Alternate solution 2

If you do not want pure Java, tRedirectOutput is a custom component that is used to redirect all the messages printed on the console to a file.

 

Alternate solution 3

Uncomment and configure the logstash section of the Log4j settings by navigating to Studio > File > Edit Project properties > Log4j:

 <!-- logstash appender -->
 <!-- <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
 <param name="RemoteHost" value="localhost"/>
 <param name="Port" value="8052"/>
 </appender> -->
Version history
Revision #:
17 of 17
Last update:
‎04-13-2019 12:49 PM
Updated by:
 
Tags (1)