Notification when status changes from OK -> CRIT

Hi there, I’ve been playing around with the alerting in influx2.0-beta and “When status” “is equal to” works like a charm however I’m having issues to get the “When status” “changes from” OK -> Crit. I’ve changed the check around to make the status go from OK -> CRIT -> OK -> CRIT but the Notification rule doesn’t trigger which is a shame.
The when status is equal to works but it’ll send a message on every check which in my case is every minute so the state transition is really the way to go. Has anyone got this working?

Hello @aastroem,
Sorry in advance if this is a silly question, but are you hoping for a message when the alert goes from CRIT->OK? or is the problem that. you’re not getting any Notifications now?

Hello @Anaisdg!
Yes, that’s right. I’m getting alarms when the status is equal to but not when the status transitions between statuses.

Hello @aastroem,
I would recommend checking out stateChanges() then.

@Anaisdg I’m using the docker container running Influx 2.0 BETA so I’m configuring the notification rule through the UI. My understanding is that kapacitor is built into Influx 2.0? Is there any way that I can troubleshoot this to know what’s going on under the hood in the black box? I managed to get the rule to kick in by changing the check rule back and forth to force the status to change back and forth. I got quite inconsistent results and I have no clue in why it doesn’t work on normal operation? Is it a performance issue?

Hi again, I’ve been troubleshooting this for hours now and it’s not easy that’s for sure.
I’ve made sure that I’m only dealing with one series and I’ve tweaked the timings between:

  • Aggregation function window period = 1m (last)

  • Check is scheduled every 1m10s

  • I’ve verified the thresholds i.e. count > 0.9 = CRIT count < 1 = OK

  • The check in combination with the thresholds are spitting out status changes accordingly at Xm+10s

  • I’ve played around with the notification rules and managed to get the “when status equals to” working but I can’t get the "when status changes from OK to CRIT to work. I’ve tried with different schedule times i.e. 1m15s, 5m15s, 10m15s becase I figured that the notification rule check might use an aggregate function / bucket to see if there have been any state transitions during the time windows since the last check. Guess what, doesn’t work.

Can someone please point out what am I doing wrong here?

We’re seeing the same problem on our alerts as well with Influx 2.0.0 beta 8.

Opened https://github.com/influxdata/influxdb/issues/17809