About Us

Citrus Consulting Services is the Consulting and the Transformation Services arm of Redington Gulf.

Sunday – Thursday: 9:00AM–6:00PM (Sales), Sunday – Saturday: 24×7 / 365 (Support) E.O#3, Ground Floor, Building 01 Dubai Internet City, P.O Box 501 761 Dubai, UAE (+971) 04 516 1500
(+966) 11 462 5323
info@citrusconsulting.com

Citrus Consulting Services

How to use Syslog-ng along with Splunk for ingesting Syslog Data

How to use Syslog-ng along with Splunk for ingesting Syslog Data?

 

Syslog-ng an open-source Linux utility is one of the most preferred and easy way to listen and write logs from variety of network and security Devices and write them to a human readable format in text files. Once you have setup your syslog-ng system in place it becomes piece of cake to ingest those logs into Splunk by using a small weight universal forwarder deployed on the syslog-server. In this post we will walk you through the steps, best practices and sample configuration file for syslog-ng and Splunk Universal forwarder inputs.conf.

 

Note: Please note the steps and suggestions are mentioned for nix* OS only.

 

Install Syslog-ng

Install the syslog-ng utility in case it’s not already present on OS.

Check if syslog-ng service is present/running on the system:

systemctl status syslog-ng

 

If the command gives an error that no such service is present install the service by using one of the below commands depending on the type of nix* OS you are using:

yum install syslog-ng
apt-get install syslog-ng

This will download and install the syslog-ng service on the Linux OS. By default, now if you will run the systemctl status command it should show the status of the service as running.

 

Configure Custom Syslog Ports on each Device

For collecting data from multiple syslog devices and subsequently ingesting into Splunk it’s easy if you configure each device to send logs a unique port to your syslog server instead of the default port 514. For example:

Palo Alto Firewall > Syslog_Server:5515/TCP

Fortinet Firewall > Syslong_Server:5516/TCP

CISCO Router > Sysog_Server:5517/TCP

 

This approach has multiple advantages:

  1. Simplicity of configuration required on Syslog-ng – Syslog-ng can be configured to listen to multiple ports. As each port is unique to a device type, further configurations are simpler to write all the logs listened from that port to a directory specific for each device.
  2. Ease of Troubleshooting – As each device is sending on unique port it makes the network troubleshooting also easier in case syslog is not receiving logs from a particular device. You can easily do a tcpdump on the syslog server for that port and see if there’s any traffic hitting at all or not:

tcpdump -i <interface> | grep <portnumber>

 

 

Syslog-ng File Configuration and Troubleshooting

Following are the default location for relevant syslog-ng configuration files:

/etc/syslog-ng/syslog-ng.conf – This is the master config file which ships along with syslog-ng installation

/etc/syslog-ng/conf.d – Any .conf file created under this directory is processed along with syslog-ng.conf file

As a best practice we create our custom configuration file under the conf.d directory as it’s a more modular approach and allows creating multiple files for different configuration without effecting the overall functionality.

Following is a sample configuration file created under conf.d directory:
syslog_custom_ports.conf

 

#Listen on port 5515/tcp for logs being sent by Palo Alto Firewall and write it to corresponding folder having date as filename
source palo_alto {

tcp(ip(0.0.0.0) port(5515));

};
destination d_palo_alto {

file(“/opt/syslog-ng/palo_alto/$HOST/$MONTH$DAY.log”);

};

log     {

source(palo_alto);

destination(d_palo_alto);

};

#Listen on port 5516/tcp for logs being sent by Fortinet firewall and write it to corresponding folder having date as filename

source fortinet {

tcp(ip(0.0.0.0) port(5516));

};

destination d_fortinet  {

file(“/opt/syslog-ng/fortinet/$HOST/$MONTH$DAY.log”);

};

log     {

source(fortinet);

destination(d_fortinet);

};

 

#Listen on port 5517/tcp for logs being sent by Cisco router and write it to corresponding folder having date as filename

source cisco_router {

tcp(ip(0.0.0.0) port(5517));

};

destination d_cisco_router  {

file(“/opt/syslog-ng/cisco_router/$HOST/$MONTH$DAY.log”);

};

log     {

source(cisco_router);

destination(d_cisco_router);

};

 

Once you save the file you need to reload the syslog-ng service for the new changes to take effect:

# Reloading configuration file without restarting syslog-ng
systemctl reload syslog-ng

 

#Restarting syslog-ng

systemctl restart syslog-ng

 

If the configuration file is correct and there are no permissions service should reload correctly post which you can run the following commands to see which all ports are now active on the OS where syslog-ng is listening on

netstat -an

 

Also, if you have devices sending logs in, navigate to respective directory and verify the logs are being written correcty.

vi /opt/syslog-ng/PaloAlto/<host>/<date in ddmm format>.log

 

In case there is any issue with syntax of the file run following command to see the errors:

syslog-ng –syntax-only

 

For other issues further troubleshooting might be required, often analyzing the logs in/var/log/messages is found to be quite helpful. Also, at times SELinux might be causing issues in which case try disabling the SELinux on the OS level.

 

vi /etc/selinux/config

SELINUX=disabled

# reboot OS for changes to take effect
restart

 

Read the logs by Splunk

Depending whether you are managing the configurations through Splunk Deployment server or managing them locally on the instance you need to place inputs.conf file on the syslog-server running Universal Forwarder or Heavy Forwarder. In the inputs.conf you need to define the monitor stanzas for each device/log source you configured in the Syslog-ng. For the above syslog-ng following is a sample inputs.conf file:

 

[monitor:///opt/syslog-ng/palo_alto/*/*.log]

host_segment = 4

index = palo_alto

#sourcetype as per add-on installed or custom sourcetype defined

sourcetype = pan:firewall

no_appending_timestamp = true

 

[monitor:///opt/syslog-ng/fortinet/*/*.log]

host_segment = 4

index = fortinet

#sourcetype as per add-on installed or custom sourcetype defined

sourcetype = fgt

 

[monitor:///opt/syslog-ng/cisco_router/*/*.log]

host_segment = 4

index = cisco_router

#sourcetype as per add-on installed or custom sourcetype defined

sourcetype = syslog

 

For the final check log in to you Splunk GUI and run query to see if the logs are being read and ingested correctly.

index= palo_alto

 

If you have any feedback, suggestions feel free to drop a note to author of this series – bharat.jindal@citrusconsulting.com

 

 

Bharat Jindal is an experienced senior consultant with expertise in network and information security domain both on-premise and cloud infrastructure. He is currently leading the security team at Citrus and delivering end to end security consulting services starting from solutions design to managed support services to clients across MEA on number of security solutions including but not limited to cloud Security, SIEM, Firewalls, PIM/PAM, EDR, IPS, IDS and others. He is also a passionate learner with 20+ certifications from leading enterprise security vendors including AWS Certified Security Specialty."

Post a Comment

10 + fourteen =