Identifying Splunk forwarders that phone home too frequently


As I've worked on many large-scale Splunk environments, a common problem I've noticed is that Splunk forwarders phone home to the deployment server (DS) too frequently. When a forwarder phones home to the DS more often than necessary, it wastes resources on the DS, and can prevent the DS from deploying apps to forwarders correctly.

Splunk saved search (and correlation search!) explorer


As a Splunk administrator, have you ever needed to list out your saved searches in some way? Perhaps you need to know which searches might be accelerated, scheduled, or even real-time scheduled? Here's a quick dashboard to show this information.

The dashboard is attached as an txt file.

saved search explorer screenshot


Quick tip when working with stats command


When you're working with the stats command, it's often nice to rename the fields to drop the aggregatation type:

| makeresults count=3 
| streamstats count
| eval foo="bar"
| stats latest(foo) as foo latest(count) as count

This gets annoying because of all the extra typing involved, and it violates the DRY principal.

Instead, try this next time:

Splunk Tip: Regex Extractions in Props.conf


When you create an extraction in props.conf (a search-time field), you specify the name of the new field through a named capture group in the regular expression. For example:

EXTRACT-foo =  (?P<foo>\w+)

This configuration will create a field "foo" with a value of the first word found in the _raw field. You can also specify a field besides _raw in your extraction:

EXTRACT-foo =  (?P<foo>\w+) in host

How to change the default search time range in Splunk 6


This is a simple request, but the official docs are out of date.

  1. Create a file: $SPLUNK_HOME/etc/apps/search/local/ui-prefs.conf
  2. Create a search stanza like this:

    dispatch.earliest_time = -7d@d
    dispatch.latest_time = now

Restart Splunk and you're good to go!

I Just Discovered Simple XML Dashboard Extensions!


I just discovered that Simple XML dashboards in Splunk can be customized with JavaScript! That's right, the point and click dashboards can be customized in a very powerful way. Here's the official docs, and an official example.

How to Make Faster Joins in Splunk


Although it's often possible--and recommended--to avoid the join command, sometimes it is necessary to use join. I was recently exploring the performance impact of the join command and I wanted to share my findings.

Splunk Configuration Management -- my progress so far


Part of being a responsible software engineer includes the systems engineering process of configuration management. Although backups and access controls are a necessary part of maintaining a secure IT system, relying on these processes for configuration management is inefficient and dangerous.

How to Delete Splunk Events When Using a Transforming Command


Recently, I needed to delete some events that matched certain summary conditions. For example, where the event count exceeds a certain threshold:

Example showing event search with stats criteria

Now, if you try to delete the events by appending | delete, you'll receive an error:

Error in 'delete' command: This command cannot be invoked after the non-streaming command 'stats'

How to Utilize Post-Process Searches in Splunk Simple XML and HTML


It took me a while to figure out how to use a Post-Process Search in a Splunk Dashboard, so I thought it would be a good idea to remind my future self how it's done.

This is a Simple XML dashboard. It is essentially the same as the example in my last post. The full source code is attached to this post.

In order to use a Post Process search, only three changes are needed:


Subscribe to RSS - splunk