How to use Syslog-ng along with Splunk for ingesting Syslog Data
How to use Syslog-ng along with Splunk for ingesting Syslog Data?
Syslog-ng an open-source Linux utility is one of the most preferred and easy way to listen and write logs from variety of network and security Devices and write them to a human readable format in text files. Once you have setup your syslog-ng system in place it becomes piece of cake to ingest those logs into Splunk by using a small weight universal forwarder deployed on the syslog-server. In this post we will walk you through the steps, best practices and sample configuration file for syslog-ng and Splunk Universal forwarder inputs.conf.
Note: Please note the steps and suggestions are mentioned for nix* OS only.
Install Syslog-ng
Install the syslog-ng utility in case it’s not already present on OS.
Check if syslog-ng service is present/running on the system:
systemctl status syslog-ng
If the command gives an error that no such service is present install the service by using one of the below commands depending on the type of nix* OS you are using:
yum install syslog-ng
apt-get install syslog-ng
This will download and install the syslog-ng service on the Linux OS. By default, now if you will run the systemctl status command it should show the status of the service as running.
Configure Custom Syslog Ports on each Device
For collecting data from multiple syslog devices and subsequently ingesting into Splunk it’s easy if you configure each device to send logs a unique port to your syslog server instead of the default port 514. For example:
Palo Alto Firewall > Syslog_Server:5515/TCP
Fortinet Firewall > Syslong_Server:5516/TCP
CISCO Router > Sysog_Server:5517/TCP
This approach has multiple advantages:
- Simplicity of configuration required on Syslog-ng – Syslog-ng can be configured to listen to multiple ports. As each port is unique to a device type, further configurations are simpler to write all the logs listened from that port to a directory specific for each device.
- Ease of Troubleshooting – As each device is sending on unique port it makes the network troubleshooting also easier in case syslog is not receiving logs from a particular device. You can easily do a tcpdump on the syslog server for that port and see if there’s any traffic hitting at all or not:
tcpdump -i <interface> | grep <portnumber>
Syslog-ng File Configuration and Troubleshooting
Following are the default location for relevant syslog-ng configuration files:
/etc/syslog-ng/syslog-ng.conf – This is the master config file which ships along with syslog-ng installation
/etc/syslog-ng/conf.d – Any .conf file created under this directory is processed along with syslog-ng.conf file
As a best practice we create our custom configuration file under the conf.d directory as it’s a more modular approach and allows creating multiple files for different configuration without effecting the overall functionality.
Following is a sample configuration file created under conf.d directory:
syslog_custom_ports.conf
#Listen on port 5515/tcp for logs being sent by Palo Alto Firewall and write it to corresponding folder having date as filename
source palo_alto {
tcp(ip(0.0.0.0) port(5515));
};
destination d_palo_alto {
file(“/opt/syslog-ng/palo_alto/$HOST/$MONTH$DAY.log”);
};
log {
source(palo_alto);
destination(d_palo_alto);
};
#Listen on port 5516/tcp for logs being sent by Fortinet firewall and write it to corresponding folder having date as filename
source fortinet {
tcp(ip(0.0.0.0) port(5516));
};
destination d_fortinet {
file(“/opt/syslog-ng/fortinet/$HOST/$MONTH$DAY.log”);
};
log {
source(fortinet);
destination(d_fortinet);
};
#Listen on port 5517/tcp for logs being sent by Cisco router and write it to corresponding folder having date as filename
source cisco_router {
tcp(ip(0.0.0.0) port(5517));
};
destination d_cisco_router {
file(“/opt/syslog-ng/cisco_router/$HOST/$MONTH$DAY.log”);
};
log {
source(cisco_router);
destination(d_cisco_router);
};
Once you save the file you need to reload the syslog-ng service for the new changes to take effect:
# Reloading configuration file without restarting syslog-ng
systemctl reload syslog-ng
#Restarting syslog-ng
systemctl restart syslog-ng
If the configuration file is correct and there are no permissions service should reload correctly post which you can run the following commands to see which all ports are now active on the OS where syslog-ng is listening on
netstat -an
Also, if you have devices sending logs in, navigate to respective directory and verify the logs are being written correcty.
vi /opt/syslog-ng/PaloAlto/<host>/<date in ddmm format>.log
In case there is any issue with syntax of the file run following command to see the errors:
syslog-ng –syntax-only
For other issues further troubleshooting might be required, often analyzing the logs in/var/log/messages is found to be quite helpful. Also, at times SELinux might be causing issues in which case try disabling the SELinux on the OS level.
vi /etc/selinux/config
SELINUX=disabled
# reboot OS for changes to take effect
restart
Read the logs by Splunk
Depending whether you are managing the configurations through Splunk Deployment server or managing them locally on the instance you need to place inputs.conf file on the syslog-server running Universal Forwarder or Heavy Forwarder. In the inputs.conf you need to define the monitor stanzas for each device/log source you configured in the Syslog-ng. For the above syslog-ng following is a sample inputs.conf file:
[monitor:///opt/syslog-ng/palo_alto/*/*.log]
host_segment = 4
index = palo_alto
#sourcetype as per add-on installed or custom sourcetype defined
sourcetype = pan:firewall
no_appending_timestamp = true
[monitor:///opt/syslog-ng/fortinet/*/*.log]
host_segment = 4
index = fortinet
#sourcetype as per add-on installed or custom sourcetype defined
sourcetype = fgt
[monitor:///opt/syslog-ng/cisco_router/*/*.log]
host_segment = 4
index = cisco_router
#sourcetype as per add-on installed or custom sourcetype defined
sourcetype = syslog
For the final check log in to you Splunk GUI and run query to see if the logs are being read and ingested correctly.
index= palo_alto
If you have any feedback, suggestions feel free to drop a note to author of this series – bharat.jindal@citrusconsulting.com
