Splunk can be installed in Windows and Linux for Production environments, there are some tweaks that you can configure to make your environment run better and with no issues, this class makes few recommendations that are very new to me so I will list them here to keep adding to my notes.
Linux Settings Recommendations
Ulimit
The class recommends the use of ulimit -ato view settings, and then increase the parameters on indexers and search heads. This one seems a bit off, but here is a quick informational link about the ulimit command
The ulimit command is used to identify the resources a current logged in user can access in such system, I’m running the command in my MAC to see what is the deal with the command.
So fo example on my computer when I run it I get the following
asarmiento-mbp:~ asarmiento$ ulimit
unlimited
!!!!!!!!!!!!!!!!! THIS MEANS THAT I HAVE UNLIMITED ACCESS TO THE RESOURCES OF THE COMPUTER - YOU BETTER!
Now there is a command that kind of goes over what the “unlimited” means to me, so let’s drill into it, the command you run is the following:
asarmiento-mbp:~ asarmiento$ ulimit -a
This really doesn’t mean much to me and my computer, but it was definitely a nice diverssion for me to look at, now lets get back to Splunk, which is what we care for right now
For the purposes of Splunk and the installation it is recommended to use th following settings:
For open files you will need > = 64K –> Check results by running ulimit -n
For max user processes you need > = 16K –> Check results by running ulimit -u
The way you set these files is by going to the following directory
/etc/security/
If you’d like to modify the limit numbers, you can use nano or bi to make your changes
cat /etc/security/limits.conf
A nice read and document that goes over what the
ulimit feature covers, is here –> https://linuxhint.com/linux_ulimit_command/
** Also it is important to understand how to identify the errors, here by the Splunk DOcumentation Portal –> Troubleshooting ulimit Errors
** One more, which I have previously shared — Splunk System Requirements
Transparent Huge Pages
It looks like Splunk does not like the use of THP, which was a nice enhancement introduced a while ago, THP has the reputation to have a negative impact on performance, as stated in this link, seems like more than a myth than a fact, however Splunk knows better in this case, hence why they recommend turning of the feature.
But hey, I’m not a DB expert nor a Splunk guru so I will give this one to Splunk, and follow the “FM” – In the same thought Splunk goes in very great detail about THP and the reasons they recommend turning it off, here are a few:
- The implementation is too aggressive at coalescing memory pages for short-lived processes (such as many Splunk searches)
- It can prevent the jemalloc memory allocation implementation from releasing memory back to the operating system after use. The jemalloc implementation is more scalable version of the malloc implementation and has been used in newer distributions of Linux
- For some workloads, it can cause I/O regressions surrounding swapping of huge pages
That is all great information, but lets see how to disable it
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
I was able to find a nice forum response on this one that goes beyond the basics on making sure this service is disabled, such as disabling it from the Startup as making sure its not running either.
Time Synchronization
Because logs mean nothing if you don’t have a way to know when events happened, make sure your Splunk installation as well as anything you install on your network synchronizes with an NTP server. NTP servers are HUGE part of any implementation.
Startup Account and Recommendations
Lets go with the regular and the always recommended, never use the root account to perform regular user operations.
Last but not least, Splunk has a very comprehensive installation guide and I think it will be horrible from my end not to share with you – Enjoy!
Windows
Windows (from the command line)
Linux
What is next?
Let me tell you that that part was only 3 slides and I wanted to cover more tonight, however this will be all for now, the next post should be all related to the Splunk Directory Structure, which should be quick and easy, but a very heavy part of the classes I took las week
About the Author:
Andres Sarmiento, CCIE # 53520 (Collaboration)
With more than 15 years of experience, Andres is specialized in Unified Communications and Collaboration technologies. Consulted for several companies in South Florida, also Financial Institutions on behalf of Cisco Systems. Andres has been involved in high-profile implementations including Cisco technologies; such as Data Center, UC & Collaboration, Contact Center Express, Routing & Switching, Security and Hosted IPT Service provider infrastructures.