unable to use log4j2 in Talend

[MKAPOOR1596038160]

I am using Talend 7.3 data integration open source version and trying to generate logs in talend flow during my various steps in design. The idea is to better capture data like errors, rejected records, number of DB records in/out, component errors etc, so that the operations team whom i am going to handover the flow after design be able to measure the errors, progress of flow etc.

I explored stat, metrics, and logs in talend but the information i get out of them is not very relevant to pin point or track an operational issue. Then i also find that there is log4j2 method of logging that exist in new talend versions of Talend however there is very little documentation and examples given to understand how log4j2 can be used with Talend 7.3. There is a very simple below xml present but no information on how to use it.

When i run my job from Talend console, i get all the data( printed by logrow component) on the console but when i run the same exported job from linux command line using xxxx_run.sh in background, i am unable to log it appropriately in a log file. Also i want to print errors, num of records parsed by each component in a common log file.

log4j2.xml

<?xml version="1.0" encoding="UTF-8"?>

<Configuration >

  <Appenders>

    <Console name="Console" target="SYSTEM_OUT">

      <PatternLayout pattern="[%-5level] %d{HH:mm:ss} %logger{36}- %msg%n" />

    </Console>

  </Appenders>

  <Loggers>

    <Root level="WARN">

      <AppenderRef ref="Console" />

    </Root>

  </Loggers>

</Configuration>

Can someone please help here? An example of same would be appreciated..

[Anonymous]

Hi

Refer to the following documentation and KB articles, you can output the logs to file.

https://community.talend.com/s/article/Log-j-tips-and-tricks-I8730

https://help.talend.com/reader/gMvNEA2uv25kZtDW1I9Tsw/tHrq84riybZZNhoRyvZcOQ

About the number of processed records, tlog4j can't capture this kind of information, we usually use the global variables which are pre-defined on the input/output components, such as

((Integer)globalMap.get("tDBInput_1_NB_LINE"))

((Integer)globalMap.get("tDBOutput_1_NB_LINE_INSERTED"))

or use tFlowMeterCather to capture the number of records in the data flow after one component.

Regards

Shong

[MKAPOOR1596038160]

Thanks Shong for a quick reply, however i already had a look at the Hyperlink resources that you shared. These were not easy to setup especially the case of my design where i am using some StringHandling regex in tMap to filter the rows and at times it gives parsing errors. I would like to print those parsing errors in the log file. So my few questions are :

1. Which is best way to print success/fail/errors of each component of flow in a log file?
2. What is default level of logging that Talend provides? Is log4j2 enabled by default for all components? if yes, where are the logs stored? There is none in my case?
3. Can i keep stat, metrics, logs and log4j2 all simultaneously enabled for my given design?