Kapacitor: Emit alert on startup if OK

1

Does anyone know a way to force kapacitor to emit an alert on startup? I am assuming that the default state for an alert node is to start at the “OK” level, so if the alert does not transition into “WARNING” or “CRITICAL”.

I guess this would require a 4th alert state of “UNKOWN” or “INIT” to allow it to transition

2

As of version 1.2 the alert will start in the state it was in before Kapacitor was shutdown. For new alerts they start in the OK state. Can you explain what you are trying to do by triggering an alert at startup? There maybe a way to achieve what you want a different way.

3

I am currently taking alert node state and pushing this back into influxdb, which allows me to create a dashboard of the current state of alerts. With this I can see how long we were in alert states, alert patterns over time etc. For example:

data_max
    |alert()
        .stateChangesOnly()
        .warn(lambda: "_max_connections" >= atg_concurrent_connections_warn)
        .crit(lambda: "_max_connections" >= atg_concurrent_connections_critical)
        .message(string(alert_header) + '
            Average COAP connection concurrency are {{ index .Fields "_max_connections" | printf "%0.2f"}} over a ' + string(atg_concurrent_connections_window_period) + ' window
            Expecting < {{ if eq .Level "CRITICAL" }}' + string(atg_concurrent_connections_critical) + '{{ else }}' + string(atg_concurrent_connections_warn) + '{{ end }}
            <' + string(atg_concurrent_connections_graph_url) + '|Grafana>')
        .slack()
          .channel(slack_channel)
        .levelField('level')
    |eval(lambda: if("level" == 'CRITICAL', 2, if("level" == 'WARNING' , 1, 0)))
        .as('state')
        .keep()
    |influxDBOut()
            .database(ops_alerts_db)
            .retentionPolicy(ops_alerts_rp)
            .measurement(taskname)
            .tag('stack', stack)
            .tag('deployment', deployment)

Since the state at startup is normally going from OK to OK, and I only trigger alerts on state changes, no state will be output back into the database, and I will not have a metric for that task. Only once a given task transitions state will I get a measurement point added to the database.

Second, this is more a nice to have, and only gives me a warm fuzzy feeling, knowing that kapacitor nodes is doing something, I have monitoring on them anyways. I am running kapacitor inside a docker container, so if the container restarts I will be starting without state. It would be a nice to have the alerts transition from a state of INIT or UKNOWN to its current state, and at this point a series of slack messages would be emitted from the alert nodes.

Does that help explain my use case?

4

Is it possible to restore the kapacitor.db state in the container? If you did I think your use case would be solved as if the alert was in a non-OK state when the container is stopped then with the next start the alert is in an OK state Kapacitor will see that as an alert transition from the non-OK to the OK state and fire the event which will in turn be recorded in the influxdb database.

If that won’t quite solve the issue for you then please open an issue on github as a feature request of the INIT state to be added.

5

Thanks Nathaniel,

It would solve one portion of my use-case but not the, first time ever running case. I will open a feature request for the INIT state.

Cheers

6

Opened an issue.

Sorry, I could make this change myself, but I would not be able submit it back to the project, as my legal department is still issues with me signing the CLA :wink: