consecutive errors

How to alert on consecutive error counts per minute with Splunk? Say you want to alert when an error has occured five minutes consecutively.

I don’t necessarily even recommend handling this type of logic in Splunk. Just dump it to something like PagerDuty and make your choices there. Don’t limit your alerts to one monitoring product.

Anyway, here is one simple way:

| bin _time span=1m
| stats count by _time, thing
| stats count(thing) as minutes by thing
| where minutes==5

Now make your alert trigger whenever you get more than zero events.

This logic does not handle cases where there always would be 1 minute with no errors. You may want to smooth the count with trendline.

Back to posts