Qlik Talend Data Integration: How to log Job output to a file as well as the console

[TalendSolutionExpert]

Talend Studio logs output from the tLogRow component to the Run tab console, as demonstrated below. However, if you log an excessive number of statements, such as exceeding 1,000, Talend Studio may experience slowdowns or hangups.

Solution

To log the Job output to a separate log file, navigate to Talend Studio > File > Edit Project properties > Log4j. Your Studio will look like this:

1. Create a FILE appender section and add the following lines:

   <!-- FILE appender -->
   <appender name="FILE" class="org.apache.log4j.FileAppender">
     <param name="file" value="c:/temp/talend_job.log" />
     <param name="append" value="false" />
     <param name="threshold" value="debug" />
     <layout class="org.apache.log4j.PatternLayout">
       <!-- 'p' is Priority: INFO, DEBUG, TRACE, etc. -->
       <!-- 'c' is Category: fully qualified project and job name -->
       <!-- 'm' is Message: the message the component is logging -->
       <!-- 'n' is Line Separator: e.g. the newline character -->
       <param name="ConversionPattern" value="[%-5p]: %c - %m%n" />
       <!-- <param name="ConversionPattern" value="%m%n" /> -->
     </layout>
   </appender>

   Notice that you can customize the output path, "file", for example, "c:/temp/talend_job.log".

2. Create an <appender-ref ref="FILE"/> section under the 'Root logger' section at the bottom:

   <!-- Root Logger --> 
    <root>
    <priority value ="warn"/>
    <appender-ref ref="CONSOLE"/>
    <appender-ref ref="FILE"/>
    <!-- <appender-ref ref="LOGSTASH"/> -->
   </root>

   You will already have the CONSOLE appender section.

   These sections of your file should look similar to this:

   <!-- CONSOLE appender -->
   <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
   <layout class="org.apache.log4j.PatternLayout">
   <param name="ConversionPattern" value="[%-5p]: %c - %m%n" />
   </layout>
   </appender>
   ...
   <!-- FILE appender -->
   <appender name="FILE" class="org.apache.log4j.FileAppender">
     <param name="file" value="c:/temp/talend_job.log" />
     <param name="append" value="false" />
     <param name="threshold" value="debug" />
     <layout class="org.apache.log4j.PatternLayout">
       <!-- 'p' is Priority: INFO, DEBUG, TRACE, etc. -->
       <!-- 'c' is Category: fully qualified project and job name -->
       <!-- 'm' is Message: the message the component is logging -->
       <!-- 'n' is Line Separator: e.g. the newline character -->
       <param name="ConversionPattern" value="[%-5p]: %c - %m%n" />
       <!-- <param name="ConversionPattern" value="%m%n" /> -->
     </layout>
   </appender>
   ...
   <!-- Root Logger --> 
    <root>
    <priority value ="warn"/>
    <appender-ref ref="CONSOLE"/>
    <appender-ref ref="FILE"/>
    <!-- <appender-ref ref="LOGSTASH"/> -->
   </root>

3. Open your Job. In the Run view, select the Advanced settings tab. Select the log4jLevel check box, and from the drop-down list, set the value to Debug.

   

Alternatives

The following alternates redirect the console output to other places instead of creating a new place to output the information.

Alternate solution 1

You can log the output to a file by using a small amount of Java code in a tJava component at the start of your Job. For instance:

java.io.File outputFile = new java.io.File("c:/temp/my_job_output.txt");
System.setOut(new java.io.PrintStream(new java.io.FileOutputStream(outputFile, true), true));

Alternate solution 2

If you do not want pure Java, tRedirectOutput is a custom component that is used to redirect all the messages printed on the console to a file.

Alternate solution 3

Uncomment and configure the logstash section of the Log4j settings by navigating to Studio > File > Edit Project properties > Log4j:

 <!-- logstash appender -->
 <!-- <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
 <param name="RemoteHost" value="localhost"/>
 <param name="Port" value="8052"/>
 </appender> -->