Part of being a responsible software engineer includes the systems engineering process of configuration management. Although backups and access controls are a necessary part of maintaining a secure IT system, relying on these processes for configuration management is inefficient and dangerous.
Example of git log showing Splunk configuration changes
Thankfully, the text-based configuration of Splunk makes it relatively easy to record configuration changes in a source control system. I prefer git, one of the most widely-used source control applications available. And yes, it's open source!
Git diff (in tig) showing changes made in a particular commit
Assuming you know how git works, here are a few tips and tricks that have worked for me:
- Initialize your git repo in the base Splunk installation directory (/opt/splunk by default)
- Add metadata files to gitignore. Refer to the attached gitignore file to get started.
- Use a separate git repo for apps that you install or create (Separaation of concerns / Single Responsibility Principal)
- Commit frequently, and use a good commit message. If a change isn't committed, or doesn't have a useful commit message, you don't really have a proper configuration management process.
- Use tig to page through the git log and diffs. Install with apt-get install tig or yum install tig.
- Follow this process on all of your Splunk nodes - indexers, forwarders, etc.