Help on Log4j setting for different LogStash servers

Highlighted
Six Stars

Help on Log4j setting for different LogStash servers

Hi All,
I need help on setting Log4j setting for Different Logstash servers.
Scenario:
I have 3 different Logstash servers, and 3 different Job servers.
Logstash server is different than Job Server.
I have added Logstash appender in Log4j setting of my Studio. (See below screen shot)

Issue: 
I have set log4j in my studio only for 1 logstash server not for all 3.
I am running the job in 3 different job servers. 
I don't want to change the setting for each of the servers (DEV, QA and PROD)
If i set up 3 logstash apenders for each logstash servers, logs will be going to (DEV, QA and PROD) together, which i dont want.

Question:
Is there any way to setup LOG4J so that i get the logs when i run the job in any of the job servers.
I want the DEV logs, when i run the job in DEV
QA logs when i run the job in QA, like wise for PROD. 

Note: 
Logging works fine if i setup 1 Logstash appender in my LOG4J setting.
I can't use "Localhost" as Logstash and Job servers are different.
Please let me know if you have any idea on this.
Thirteen Stars

Re: Help on Log4j setting for different LogStash servers

first of all - screenshot not attached
but in any case - You can define context variable for any settings, and create 3 context group - 1 per Job server
depending from server You choose proper context group for Job
-----------
Six Stars

Re: Help on Log4j setting for different LogStash servers

Hi,
Thanks for the reply,
Please find below screen shot


I tried using context variable, but if i provide context variable as below, It is not working in LOG4J setting.
<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value="context.logstash_hostname"/>
        <param name="Port" value="8052"/>
    </appender>
Thirteen Stars

Re: Help on Log4j setting for different LogStash servers

try change:
<!-- logstash appender -->
<appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
<param name="RemoteHost" value="context.logstash_hostname"/>
<param name="Port" value="8052"/>
</appender>

to 
<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value=context.logstash_hostname/>
        <param name="Port" value="8052"/>
</appender>


I have only 1 log stash at the moment, so it hard to test, but at least it not return errors
-----------
Six Stars

Re: Help on Log4j setting for different LogStash servers

I tried appender using context variable as below:
<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value=context.logstash_hostname/>
        <param name="Port" value="8052"/>
</appender>


It is not accepting context variable in Studio log4j setting.
It is not giving error but logs are not getting logged.
Six Stars

Re: Help on Log4j setting for different LogStash servers

Few things i wanted to confirm:
1. Are you able to get component level logs using context variable in log4j setting.
2. I am giving logstash appender in Studio log4j setting (See attached), SO is this the only way to get component level logs or is there any other way i can add logstash appender to get component level logs.
Six Stars

Re: Help on Log4j setting for different LogStash servers

I got the solution for this issue:
For each of the 3 job servers, we need to include Logserver IP address in the "hosts file" located at "/ect/hosts"
What i did is as below

Added logstash IP address in Job Server 1 "hosts file"
XX.XX.XXX.XXX    talend_logstash.aexp.com


Added logstash IP address in Job Server 2 "hosts file"
YY.YY.YYY.YYY    talend_logstash.aexp.com


Added logstash IP address in Job Server 3 "hosts file"
ZZ.ZZ.ZZZ.ZZZ    talend_logstash.aexp.com

Now i am using below details in logstash appender:
<!-- logstash appender -->
    <appender name="LOGSTASH" class="org.apache.log4j.net.SocketAppender">
        <param name="RemoteHost" value="talend_logstash.aexp.com"/>
        <param name="Port" value="8052"/>
    </appender>

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.