Splunk Series III: System Administrator Class (File Structure, Settings and CLI)

Awesome, I was able to move from those 2 slides that took a lot of content and energy our of my brain! – The slides I’m referring to are the ones you get when you assist Splunk’s class for System Administration, basically the posts are my notes (This blog) which is an extraction of the most important content.

The next post will be about few things that I consider very important, like the settings, the directory structure and so on, so lets get at it

Splunk Directory Structure

In Splunk’s documentation the you will see that the installation location is referenced as $SPLUNK_HOME – But note that it could be any folder of your preference, not sure if this will affect any of the installations or any dependancies for some Applications hardcoded with the default path

The directory structure is very well documented, which I will post as an image as part of the post.

Main directory



Main Sub Directories

/opt/splunk/bin !!! This is the directory that holds all executables
/opt/splunk/etc !!! This one holds all Licenses and Configuration Files
/opt/splunk/var !!! This one is the one that holds all Indexes



Running Splunk at Boot

Splunk is very particular and by default it does not start at boot time, unless instructed to do it

Not Recommended Way

So this command will do the trick ** However, you should know and as mentioned in my previous post, Splunk should be started with a user that is not Root

$SPLUNK_HOME/bin/splunk enable boot-start


Recommended way to start from boot

Here is the recommended way, and off course the way that seems a bit more complex – Security first right?

[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user bob


Make sure bob has ownership over the Splunk Home Folder

[sudo] chown -R bob $SPLUNK_HOME


But now you have to make sure that your user, in this case bob can start/stop splunk interactively right? – The next thing is to modify the following file

!!! Use nano or vi !!!


!!! The user variable at the top of the file will make sure that bob can complete the commands mentioned as part of the config file 

. /etc/init.d/functions

splunk_start() {
  echo Starting Splunk...
  su - ${USER} -c '"$SPLUNK_HOME/bin/splunk" start --no-prompt --answer-yes'
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
splunk_stop() {
  echo Stopping Splunk...
  su - ${USER} -c '"$SPLUNK_HOME/bin/splunk" stop'
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/splunk
splunk_restart() {
  echo Restarting Splunk...
  su - ${USER} -c '"$SPLUNK_HOME/bin/splunk" restart'
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
splunk_status() {
  echo Splunk status:
  su - ${USER} -c '"$SPLUNK_HOME/bin/splunk" status'
case "$1" in


Now guess what, Splunk has documented the process very well, maybe better than I have, which is why I’m sharing this link with you – Splunk BootStart

The Splunkd Process

This process runs by default using SSL * Nice!! * This process is the one in charge of the Splunk Web Proxy, KV Store and Introspection services. As well as each search, any scripted input or alert. This process is important because it handles all search requests and returns results.

Splunk Web

I feel a bit sad because by default Splunk runs on HTTP port 8000, if you want to run SSL on your server you will have to modify your Settings. SO no need to be sad anymore! To access the settings so you can change it to use SSL, simply go to Settings –> Server Settings –> General Settings – Make sure you select the SSL Settings to yes!


Google Chrome Disclaimer

Google Chrome will not know how to get to your Server, well it will but it will not allow you for security reasons, one little trick is that you can simply type “thisisunsafe” and Chrome will allow you to continue… Trust me it works, here is the article –> Bypass NET::ERR_CERT_INVALID

Splunk CLI

Splunk is an executable that resides in SPLUNK_HOME/bin – The same sintax will be used on all supported platforms

splunk help
splunk [start | stop | restart]
splunk start --accept-license
splunk status
splunk show splunkd-port
splunk show web-port
splunk show servername
splunk show deafult-hostname 

What is next?

Moving to Indexes, and Index Management, there is still quite a lot to cover, see you on the next one!

About the Author:

Andres Sarmiento, CCIE # 53520 (Collaboration)
With more than 15 years of experience, Andres is specialized in Unified Communications and Collaboration technologies. Consulted for several companies in South Florida, also Financial Institutions on behalf of Cisco Systems. Andres has been involved in high-profile implementations including Cisco technologies; such as Data Center, UC & Collaboration, Contact Center Express, Routing & Switching, Security and Hosted IPT Service provider infrastructures.

You can follow Andres using Twitter, LinkedIn or Facebook

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top