Kapacitor not raising alerts correctly

,
1

I’m having some issues with getting a new instance of Chronograf+Kapacitor (v1.7.4) running with an existing InfluxDB server.

I’ve gone to Configuration and created a new connection to the remote InfluxDB server on port 8086. We also have the Kapacitor connection established on the localhost on port 9092. I’ve configured the Alert Endpoints for OpsGenie2 and I can see in the Atlassian OpsGenie logs that the test messages are coming in. For good measure, I also enabled the OpsGenie2 configuration in /etc/kapacitor/kapacitor.conf.

I can also create visualisations and dashboards via the Chronograf UI. When I try to create an alert, it doesn’t seem to be working. For example, in “Alert Rules” I have created an alert for the ping module based on “percent_packet_loss”. The conditions I’m setting is "send alert where [percent_packet_loss] is [equal or greater] 10. I can preview the data just fine in the conditions over the past 5m, 15m, etc so the data is fine.

In the Alert Handler I configured it to use opsGenie2 and the default “Don’t sent alert on condition recovery” is unchecked and “Send alert only when condition state changes” is checked. To test packet-loss, I modified some routes on our devices to break reachability, I can see packet-loss go from 0% to 100% in both our regular Grafana dashboards as well as in the preview data for the alert I’m trying to create. However, I don’t see anything in OpsGenie. When I click “Alerting → Alert History” it tells me there are no alerts to display.

I’ve confirmed both the alerts and TICKscripts are enabled (I haven’t made any changes to the TICKscripts as I’m not really sure these need to be changed). An example snippet of the TICKscript with some sensitive information tweaked slightly.

var db = 'influx_database'

var rp = 'autogen'

var measurement = 'ping'

var groupBy = []

var whereFilter = lambda: ("name" == 'wan-connection-1') AND isPresent("percent_packet_loss")

var name = 'Alert - ICMP Packet-Loss - WAN CONNECTION 1'

var idVar = name

var message = 'ID: {{.ID}}
Time: {{.Time}}
Name:  {{.Name}}
index.Tags value:  {{ index .Tags "value" }}
index .Fields value:   {{ index .Fields "value" }}'

var idTag = 'alertID'

var levelTag = 'level'

var messageField = 'message'

var durationField = 'duration'

var outputDB = 'chronograf'

var outputRP = 'autogen'

var outputMeasurement = 'alerts'

var triggerType = 'threshold'

var crit = 10

var data = stream
    |from()
        .database(db)
        .retentionPolicy(rp)
        .measurement(measurement)
        .groupBy(groupBy)
        .where(whereFilter)
    |eval(lambda: "percent_packet_loss")
        .as('value')

var trigger = data
    |alert()
        .crit(lambda: "value" >= crit)
        .message(message)
        .id(idVar)
        .idTag(idTag)
        .levelTag(levelTag)
        .messageField(messageField)
        .durationField(durationField)
        .stateChangesOnly()
        .opsGenie2()
        .teams('OPSGENIE-TEAM')
        .recipients()

trigger
    |eval(lambda: float("value"))
        .as('value')
        .keep()
    |influxDBOut()
        .create()
        .database(outputDB)
        .retentionPolicy(outputRP)
        .measurement(outputMeasurement)
        .tag('alertName', name)
        .tag('triggerType', triggerType)

trigger
    |httpOut('output')
2

Hello @Brad,
Unfortunately we don’t offer Kapacitor support here anymore.
You can use this sandbox to test configuration :slight_smile: :slight_smile:

Also here is a good resource for TICK scripts:

Hopefully another kapacitor user here can help.