I created a notification rule as follows:
The idea is to remind us every 24 hours if a check is still in “critical”
While this seemed to work, it also resulted in our notification channel getting completely spammed. We probably received close to 100 notifications at the same time for the same alert.
Seems like a bug.
Hello @philipb,
Can you please share the entire task for the notification rule?
How frequently are you getting the alert? Is your expectation to just get one alert every 24 hours if the level=“crit”?
It sounds like if you’re querying statuses data from the last 24 hours and you want a notification if the last status has a _level=“crit” then you might want to apply a last() function to your data or query a smaller amount of statuses data every 24 hours.
You might also be interested in applying a stateChange() function and only alerting when the level has changed from crit to another level.
Can you please share some demo data and your expectation on all the conditions when you want to alert so I can help you craft a custom notification rule in case the information above doesn’t help you?
Finally, have you seen InfluxDB's Checks and Notifications System | InfluxData ? It could be helpful if you haven’t.
