Stream TICK script not alerting [solved]

InfluxDB 1 Kapacitor
1

Hi,

I can’t figure out why this script is not pushing anything out. It’s enabled, and using the “show” command I can see that it is processing points, yet nothing gets triggered. Nothing on the logs either.

The objective of the script is to constantly push a status to an external system, that’s why I’m using info level when the situation is normal (rejected_connections == 0). I do not expect any OK from this script, just CRITICAL or INFO.

It’s probably a simple thing, but I think I need a second pair of eyes :wink:

Thanks in advance.

dbrp "telegraf"."autogen"
var db = 'telegraf'
var rp = 'autogen'
var measurement = 'redis'
var groupBy = ['host', 'port']
var whereFilter = lambda: TRUE
var name = 'Redis: Rejected Connections'
var idVar = name + ':{{.Group}}'
var message = 'TESTING'
var idTag = 'alertID'
var levelTag = 'level'
var messageField = 'message'
var durationField = 'duration'
var outputDB = 'chronograf'
var outputRP = 'autogen'
var outputMeasurement = 'alerts'
var triggerType = 'threshold'
var crit = 0

var data = stream
    |from()
        .database(db)
        .retentionPolicy(rp)
        .measurement(measurement)
        .groupBy(groupBy)
        .where(whereFilter)
    |eval(lambda: "rejected_connections")
        .as('value')

var trigger = data
    |alert()
        .info(lambda: "value" == crit)
        .crit(lambda: "value" > crit)
        .message(message)
        .id(idVar)
        .idTag(idTag)
        .levelTag(levelTag)
        .messageField(messageField)
        .durationField(durationField)
        .log('/tmp/redis_rejected_connections.log')
        .exec('/etc/kapacitor/exec/alert_custom.py')

trigger
    |eval(lambda: float("value"))
        .as('value')
        .keep()
    |influxDBOut()
        .create()
        .database(outputDB)
        .retentionPolicy(outputRP)
        .measurement(outputMeasurement)
        .tag('alertName', name)
        .tag('triggerType', triggerType)

trigger
    |httpOut('output')
2

Here’s the show task output:

DOT:
digraph redis_rejected_connections {
graph [throughput="0.00 points/s"];

stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="153"];

from1 [avg_exec_time_ns="29.627µs" errors="0" working_cardinality="0" ];
from1 -> eval2 [processed="153"];

eval2 [avg_exec_time_ns="26.6µs" errors="0" working_cardinality="6" ];
eval2 -> alert3 [processed="153"];

alert3 [alerts_inhibited="0" alerts_triggered="0" avg_exec_time_ns="83.605µs" crits_triggered="0" errors="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" working_cardinality="6" ];
alert3 -> http_out6 [processed="0"];
alert3 -> eval4 [processed="0"];

http_out6 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];

eval4 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
eval4 -> influxdb_out5 [processed="0"];

influxdb_out5 [avg_exec_time_ns="0s" errors="0" points_written="0" working_cardinality="0" write_errors="0" ];

}

3

As it turns out, seems that Kapacitor was keeping state from an initial execution of this script. I had to invert the crit/warn conditions in order to make it alert again. However, it only alerts once, on each state change.

So now the question is, if I’m not using StateChangesOnly, why wouldn’t this script alert every single time?

4

The problem was in kapacitor.conf [smtp] section.
global was set to true, as well as state-changes-only…
Dangerous setting!