splunk

When you create an extraction in props.conf (a search-time field), you specify the name of the new field through a named capture group in the regular expression. For example:

EXTRACT-foo =  (?P<foo>\w+)

This configuration will create a field "foo" with a value of the first word found in the _raw field. You can also specify a field besides _raw in your extraction:

EXTRACT-foo =  (?P<foo>\w+) in host

This is a simple request, but the official docs are out of date.

1. Create a file: $SPLUNK_HOME/etc/apps/search/local/ui-prefs.conf
2. Create a search stanza like this:
   
   [search]
   dispatch.earliest_time = -7d@d
   dispatch.latest_time = now

Restart Splunk and you're good to go!

I just discovered that Simple XML dashboards in Splunk can be customized with JavaScript! That's right, the point and click dashboards can be customized in a very powerful way. Here's the official docs, and an official example.

Although it's often possible--and recommended--to avoid the join command, sometimes it is necessary to use join. I was recently exploring the performance impact of the join command and I wanted to share my findings.

Part of being a responsible software engineer includes the systems engineering process of configuration management. Although backups and access controls are a necessary part of maintaining a secure IT system, relying on these processes for configuration management is inefficient and dangerous.

Recently, I needed to delete some events that matched certain summary conditions. For example, where the event count exceeds a certain threshold:

Now, if you try to delete the events by appending | delete, you'll receive an error:

Error in 'delete' command: This command cannot be invoked after the non-streaming command 'stats'

It took me a while to figure out how to use a Post-Process Search in a Splunk Dashboard, so I thought it would be a good idea to remind my future self how it's done.

This is a Simple XML dashboard. It is essentially the same as the example in my last post. The full source code is attached to this post.

In order to use a Post Process search, only three changes are needed:

Since it is so easy to search for data in Splunk, and then create a dashboard in just a couple of clicks, you might be tempted to do just that -- and release your dashboard into production. For some situations, that's absolutely fine. But as your organization becomes more reliant on Splunk dashboards, this approach can become unwieldy. And if there's anything we want, it's wieldy searches!

One of the most common scenarios I experience in Splunk is where I need to use data from two different indexes at once—typically in order to build management and reporting dashboards. With my background in developing applications on relational databases, my first attempts at this solution used the "join" command in Splunk. Once I realized that a combination of the "append" and "stats" commands can be a better choice, I started using those more. But today I will show an even better, faster approach!

I use Splunk to report on business objects moreso than typical security operation data. For instance, helpdesk tickets rather than firewall logs. I have created various Python scripts to import these business objects from various REST and SQL sources, and I want these import scripts to be idempotent. That is, I want to import helpdesk tickets every day, but no more than once per day, regardless of how many times the import script is called.