Currently, I have deadman checks that run every minute and set to CRIT if there’s no data for 90s. I have 2 notification rules that check every minute to see whether status has changed from CRIT -> OK or OK -> CRIT and send a message to Slack.
The checks seem to be triggering properly, but the notification rules aren’t triggering. The notification rule does trigger when I change the rule to check if the status equals to CRIT or OK, but I only want to be notified when it changes from one status to the other (basically trying to mimic alerts in Grafana where you can shoot out a message if there’s been no data for some time period).
The OK -> CRIT and CRIT -> OK rules would occasionally fire before upgrading from 6 -> 9, but some would fail due to the queue limit being fixed at 10. I’ve upgraded to 9 and bumped up the queue limit to 50 so the tasks are no longer failing, but now the notification rules aren’t firing at all. I’ve added images of the rules I’ve set below. Is there anything wrong with the rules I’ve set that might prevent the notification rules from firing (race conditions, etc.)? If not, is there any alt. method you’d recommend for configuring alerts?
