Kapacitor - Stateless Alerts?

InfluxDB 1 Kapacitor
1

Is there a way to create stateless Kapacitor alerts? In other words, alerts which fire on state change from “no state” to “warning” or “critical”, but whose follow on state is not saved or kept in any way?

We have a use case where we’d like to alert on Docker container failures in a Mesos cluster. Since the container is never “fixed”, just re-launched, there’s no way to remediate a particular alert (its Mesos task ID changes upon re-launch). This would be fine, except we’d like these container alerts not to just accumulate on the Kapacitor host, stuck in a perpetual CRITICAL state.

I think from reading the documentation and a little bit of the code that this isn’t possible, but would appreciate any input.

2

Hmm, this isn’t possible with the alert node as its specifically for stateful alerts.

A different way to do this would be to use several where filters combined with a httpPost node. This method doesn’t allow you to use all the alert handlers since they are tied to the alert node but you could get an event via the httpPost node anytime the condition is met.

Something like:

var data = ...

data
   |where(lambda: <warn condition>)
   |httpPost()
      .endpoint('warn')

data
   |where(lambda: <crit condition>)
   |httpPost()
      .endpoint('crit')
3

Hmm, I’d like to re-use the AlertNode handlers, and also not have to write a custom HTTP endpoint.

Do you foresee any sort of problem if I use the alert node, but have a cron job that periodically deletes the alerts from Kapacitor? For example, if I do such a thing, could the alert be deleted prior to an alert handler firing? (I don’t want an alert firing, but no handlers executing).

Could this be considered a feature request? There’s already the NoRecoveries node, which sort of works for my use case – except that the alert state is still kept. Could additional behavior be added to this node (an option maybe)? My understanding is that as long as the saved alert state is deleted outside of the window of data points for a particular stream, that Kapacitor will not generate redundant alarms.

4

Yes, this seems like a reasonable feature request. Deleting the alert topics should simply blow away the state and history but not effect running alerts, so a cron job may be a simple stop gap for now. Please create a github issue requesting the stateless alerts feature so we can track it. Thanks