Splunk Index Time Process
Data ingestion for Splunk is broken down into 3 different phases
- Input Phase – Data is handled at the source and is usually done by a forwarder
- Parsing Phase – Handled by the indexer or Heavy Forwarders, the data is broken into events
- Indexing Phase – The license meter runs as data is been written to disk, this happens before the compression of the data. After this process runs the data cannot be changed
What data can Splunk ingest?
There are many supported files for ingestion by Splunk, here is a quick list
- Files and Directories
- Network Data
- Scripts Output
- Windows Logs
- HTTP events
Metadata Settings
When indexing a data source, Splunk assigns metadata values to the source, applies defaults to visualize and organize de data. Visualization and defaults can be changed at the time of the input or later in the process
- Source
- Host
- Sourcetype
- Index
Sourcetype is what Splunk uses to determine how the data will be presented to Splunk and in further searches
Uploading data to Splunk
Splunk has a way to help ingest data by using APPs that are used for common types of data ingestion, like in the image below
Also, there are 3 other different ways to upload/present data to Splunk
- Upload – Local log files Local structured files (e.g. CSV)
- Monitor – Provides one time or continuous monitoring of files and directories
- Forward – This one is the main source of input in production environments, collected after the forwarder plugin is installed in the source server
Source Types
Splunk has default settings for many different types of data
What is next?
Basic Search
About the Author:
Andres Sarmiento, CCIE # 53520 (Collaboration)
With more than 15 years of experience, Andres is specialized in Unified Communications and Collaboration technologies. Consulted for several companies in South Florida, also Financial Institutions on behalf of Cisco Systems. Andres has been involved in high-profile implementations including Cisco technologies; such as Data Center, UC & Collaboration, Contact Center Express, Routing & Switching, Security and Hosted IPT Service provider infrastructures.