Citrus Consulting Services

Splunk Enterprise Security Configuration Best Practices

Splunk Enterprise Security is a premium and paid app from Splunk which is installed on top of Splunk Core i.e. Splunk Enterprise. Splunk Enterprise Security is the app which power the Splunk to work as full-fledged industry leading Security Incident and Event Management i.e. SIEM solution. It’s so powerful that Splunk + Splunk Enterprise Security have been leading the Gartner quadrant for SIEM solutions continuously for past many years, bypassing the traditional SIEM solution such as HPE ArcSight, IBM Qradar and the more recent solutions like Logrythm, Exabeam, ELK etc. Further Splunk Enterprise Security now comes with built in Use Case library. Splunk Enterprise Security along with Use case library provides an impressive 1000+ use cases which includes reports, dashboards and alerts.

However, to leverage the value and power from this impressive solution it’s prudent to setup and configure the Splunk Enterprise Security properly. In the subsequent sections we will cover the most important aspects which needs to be taken care of while configuring Splunk Enterprise Security following industry best practices

Proper Hardware Configuration

Splunk Enterprise Security documentation recommends minimum hardware specifications as below:

As mentioned in the table these are minimum Specifications mentioned. In case there is heavy search load and/or huge data volume it’s recommended to further provision more resources. Having enough CPU cores and RAM is very important for Splunk Enterprise security to run it’s numerous correlation searches and data model acceleration and summarization searches. Not enough resource can have serious impact for ES performance

Common Information model (CIM)

CIM can be thought of as a model or set of guidelines or instructions regarding what field names should be used and what tags, eventtypes should be assigned to events when the data is being ingested and parsed by Splunk. CIM is required so that logs coming from different devices can be normalized in same format and subsequently single search referring standard field names can be written to search and query data across various data sources. For example

Device A –  Uses field name “Source_address” for IP of the machine which initiated the traffic.

Device B – Uses field name “src_address” for the same

CIM – Cim suggests field name “src_ip” should be used for the same

In such case for both Device A and Device B we need to create field alias or create a new field “src_ip” which contains the value of “Source_address” and “src_address” for device A and Device B respectively.

This is very critical that all log sources have been made CIM compliant, as all the use cases Splunk enterprise security ships with use the fields from CIM only. Without making the data CIM compliant the ES use cases will simply not work as it would e looking for field names which are not there in the events.

Now, good thing is most of the recommendation and requirements of CIM can be fulfilled at search time i.e. irrespective the format in which the device sent the data, Splunk can rename the fields, create new fields, extract new fields, assign tags, event types, create macros during the search time itself.

Further Splunkbase contains numerous add-ons for data parsing which make the data CIM compliant. You can check the same by navigating to details of the add-on:

Where possible you should prefer to use add-ons built and supported by Splunk. In case a relevant add-on is not present or even the Splunk add-on does not make the logs 100% CIM compliant it’s required to do this process manually following the guidelines provided by CIM add-on for each type of data. More details can be found on below link:

https://docs.splunk.com/Documentation/CIM/5.0.0/User/Howtousethesereferencetables

Data model acceleration and setup

Once your data is CIM compliant next step is to make sure the data models which are shipped with Splunk ES are setup properly.

Navigate to “CIM setup” from Configuration menu in Splunk Enterprise Security. There for each data model you get various configuration options, below we mentioned the best practices and guidelines for the same:

• Summary Range – This specifies the data range of data a data model will store summaries of. For example if you specify summary range of 3 months, DM will run searches and summarize results on data for last 3 months, as a result next when you run searches referring Data model for data in last 3 months the results are lightning fast. Now it’s noteworthy bigger the summary range, more time and resources it will take for summarization, on the other hand if the summary range is too small you will not be able to search summarized data using data model beyond that range. As such it’s recommended to configure this as per the size of the corresponding data and also based on the expected timeframe that SOC Analysts might need to search this data. In our experience usually summary range of 1 month to 6 months is pretty good enough.
• Backfill Range – This is required when you are setting up Splunk ES on an existing Splunk core setup. Here you can specify when you enable the DM acceleration how back in past ES should consider the data to start summarizing. Usually backfill of 10 days to 30 days is good enough for most customers
• Accelerate – This setting is the one which enabled data model to start running searches in background for data summarization. For more data models it should be enabled, as most searches OOTB ES refer to only summarized data. Hence if any data model is not accelerated, the corresponding data model search might not work. Further you can check if “summariesonly=true” has been mentioned in the search to verify the same.
• Indexes Whitelist -This is one of the most important parameter to setup as it can hugely reduce or increase the duration of DM summarization searches. By default data model searches across all indexes which could drastically increase the summarization time unnecessarily. As such we recommend to take the search of each data model, run it over all indexes for 1 week to 1 month for example and see which indexes are the ones which have the relevant events. Thereafter updates the indexes under CIM setup accordingly.

Data Model Audit and Skipped Searches           

Once the above steps are done, we need to monitor and tune the performance of Splunk Enterprise security as a whole. For this there are two dashboards we recommend to monitor:

• Data Model Audit – Under ES navigate to Audit > Data model audit. Review the duration, size percent complete for each data model. If any data model is too big or taking too long to complete, it’s recommended to go back to CIM Setup as mentioned in previous section and tune the settings.
• Skipped Searches – This refers to the ratio of searches including DM summarization searches which got skipped due to various reasons. You can check this by Navigating to Monitoring Console>Searching>Schedule Activity>Scheduled Activity Instance. There if you the skipped search ratio more than 30%, you should navigate below in the dashboards for the “Reasons – contributing to skipped searches” and accordingly take actions. Usually following the most common reasons for skipped searches is search limit being exceeded for various searches. This can be remediated by below:
  1. Add additional CPU cores to Splunk Search head and indexers
  2. Increase the search limits in limits.conf for parameters such as
     1. base_max_searches
     2. max_searches_per_cpu

• batch_search_max_pipeline

1. max_searches_perc

If you have any feedback, suggestions feel free to drop a note to author of this series – bharat.jindal@citrusconsulting.com