Is there a way to create stateless Kapacitor alerts? In other words, alerts which fire on state change from “no state” to “warning” or “critical”, but whose follow on state is not saved or kept in any way?
We have a use case where we’d like to alert on Docker container failures in a Mesos cluster. Since the container is never “fixed”, just re-launched, there’s no way to remediate a particular alert (its Mesos task ID changes upon re-launch). This would be fine, except we’d like these container alerts not to just accumulate on the Kapacitor host, stuck in a perpetual CRITICAL state.
I think from reading the documentation and a little bit of the code that this isn’t possible, but would appreciate any input.