topic Help on Log4j setting for different LogStash servers in Talend Studio

Help on Log4j setting for different LogStash servers

Sanjay5 — Sat, 16 Nov 2024 09:55:45 GMT

Hi All,
I need help on setting Log4j setting for Different Logstash servers.
Scenario:
I have 3 different Logstash servers, and 3 different Job servers.
Logstash server is different than Job Server.
I have added Logstash appender in Log4j setting of my Studio. (See below screen shot)

Issue:
I have set log4j in my studio only for 1 logstash server not for all 3.
I am running the job in 3 different job servers.
I don't want to change the setting for each of the servers (DEV, QA and PROD)
If i set up 3 logstash apenders for each logstash servers, logs will be going to (DEV, QA and PROD) together, which i dont want.

Question:
Is there any way to setup LOG4J so that i get the logs when i run the job in any of the job servers.
I want the DEV logs, when i run the job in DEV
QA logs when i run the job in QA, like wise for PROD.

Note:
Logging works fine if i setup 1 Logstash appender in my LOG4J setting.
I can't use "Localhost" as Logstash and Job servers are different.
Please let me know if you have any idea on this.

Re: Help on Log4j setting for different LogStash servers

vapukov — Thu, 06 Apr 2017 02:58:59 GMT

first of all - screenshot not attached
but in any case - You can define context variable for any settings, and create 3 context group - 1 per Job server
depending from server You choose proper context group for Job

Re: Help on Log4j setting for different LogStash servers

Sanjay5 — Thu, 06 Apr 2017 04:39:11 GMT

Hi,
Thanks for the reply,
Please find below screen shot

I tried using context variable, but if i provide context variable as below, It is not working in LOG4J setting.

    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value="context.logstash_hostname"/>
        <param name="Port" value="8052"/>
    </appender>

Re: Help on Log4j setting for different LogStash servers

vapukov — Thu, 06 Apr 2017 04:58:59 GMT

try change:

<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value="context.logstash_hostname"/>
        <param name="Port" value="8052"/>
</appender>

<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value=context.logstash_hostname/>
        <param name="Port" value="8052"/>
</appender>

I have only 1 log stash at the moment, so it hard to test, but at least it not return errors

Re: Help on Log4j setting for different LogStash servers

Sanjay5 — Thu, 06 Apr 2017 06:09:06 GMT

I tried appender using context variable as below:

    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value=context.logstash_hostname/>
        <param name="Port" value="8052"/>
</appender>

It is not accepting context variable in Studio log4j setting.
It is not giving error but logs are not getting logged.

Re: Help on Log4j setting for different LogStash servers

Sanjay5 — Thu, 06 Apr 2017 17:27:39 GMT

Few things i wanted to confirm:
1. Are you able to get component level logs using context variable in log4j setting.
2. I am giving logstash appender in Studio log4j setting (See attached), SO is this the only way to get component level logs or is there any other way i can add logstash appender to get component level logs.

Re: Help on Log4j setting for different LogStash servers

Sanjay5 — Fri, 07 Apr 2017 21:35:52 GMT

I got the solution for this issue:
For each of the 3 job servers, we need to include Logserver IP address in the "hosts file" located at "/ect/hosts"
What i did is as below

Added logstash IP address in Job Server 1 "hosts file"
XX.XX.XXX.XXX talend_logstash.aexp.com

Added logstash IP address in Job Server 2 "hosts file"
YY.YY.YYY.YYY talend_logstash.aexp.com

Added logstash IP address in Job Server 3 "hosts file"
ZZ.ZZ.ZZZ.ZZZ talend_logstash.aexp.com

Now i am using below details in logstash appender:

    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value="talend_logstash.aexp.com"/>
        <param name="Port" value="8052"/>
    </appender>