Kapacitor: Alert escalation based on time that has elapsed since status change

InfluxDB 1 Kapacitor
1

We have a case where we are monitoring metrics and looking out for a specific value (-99) that indicates a communication failure in the device.

We would like to generate an event with level “warning”, sending an e-mail, when the value has been -99 for > 1 hour, and escalate this to level “critical” with another e-mail, when the value has been -99 for => 24 hours.
So, the
anything else than -99 => normal
-99 > 1 hour => warning, email
-99 > 24 hours => critical, email

We have searched through the documentation and the community and cannot find any information that can help us figure this out. Is seems that the scenario should be a fairly common case, so there should be a solution out there. Or are we missing out on something?

2

You should be able to do this with the state duration node i think. state duration docs

I use it to trigger on CPU alerts.

|stateDuration(lambda: "Percent_Processor_Time" > threshold)
    .unit(1m)
    .as('state_duration')

var trigger = data
|alert()
    .crit(lambda: "state_duration" > crit)
    .warn(lambda: "state_duration" > warn)

threshold = the alert to value, in your case -99
warn = 1 hour
crit = 24 hour.

If the alert is reset or recovers before 24 hours then the timer will restart and the level will reset to OK. if it doesn’t then the alert duration will increase until the next threshold.

hope that helps