Kapacitor sends alerts twice

InfluxDB 1 Kapacitor
1

I am a bit puzzled why this TICK script sends every alert twice, what do I miss?

dbrp "telegraf"."autogen"

var cpu_temperature_warn_level = 58
var cpu_temperature_crit_level = 80

var regex_topic = /^OpenCH\/(.*)\/([0-9A-Fa-f]{16})\/PubSystemStats\/(.*)$/
var get_device_type = lambda: regexReplace(regex_topic, "topic", '$1')
var get_device_id = lambda: regexReplace(regex_topic, "topic", '$2')

stream
    |from()
        .measurement('mqtt_consumer')
        .where(lambda: ("topic" =~ regex_topic AND "name" == 'temperature'))

    |eval(get_device_type, get_device_id)
        .as('device_type', 'device_id')
        .keep('value')
        .tags('device_type', 'device_id')

    |alert()
        .id('{{ index .Tags "device_type" }} {{ index .Tags "device_id" }} CPU {{ index .Tags "name" }}')
        .message('{{ .Level }} {{ .Time }} {{ .ID }} {{ index .Fields "value" }}{{ index .Tags "units" }}')
        
        .warn(lambda: int("value") >  cpu_temperature_warn_level)
        .crit(lambda: int("value") >  cpu_temperature_crit_level)

        .log('/tmp/alerts.log')

        .slack()
        .channel('#debug')

It’s Kapacitor OSS 1.5.2 (git: HEAD 3086452d00830e01d932838d8c6d1df818648ad3)
running on Raspberry Pi 3B+.

A small piece of the /tmp/alerts.log with 2 alerts repeated twice each - with the same timestamp:

{"id":"RPiGateway ba27ebfffee8c620 CPU temperature","message":"WARNING 2019-06-01 13:47:01.389610437 +0000 UTC RPiGateway ba27ebfffee8c620 CPU temperature 62.8℃","details":"{\u0026#34;Name\u0026#34;:\u0026#34;mqtt_consumer\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;cpu_alert_temperature\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;nil\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device_id\u0026#34;:\u0026#34;ba27ebfffee8c620\u0026#34;,\u0026#34;device_type\u0026#34;:\u0026#34;RPiGateway\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;rpi-opench-data\u0026#34;,\u0026#34;name\u0026#34;:\u0026#34;temperature\u0026#34;,\u0026#34;topic\u0026#34;:\u0026#34;OpenCH/RPiGateway/ba27ebfffee8c620/PubSystemStats/json\u0026#34;,\u0026#34;units\u0026#34;:\u0026#34;℃\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;localhost\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;56f71fa5-5272-4305-8e00-7fd8955e4faf\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;54052fbb-a451-4323-91e2-4159b4f31294\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;RPiGateway ba27ebfffee8c620 CPU temperature\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;value\u0026#34;:62.8},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2019-06-01T13:47:01.389610437Z\u0026#34;,\u0026#34;Duration\u0026#34;:0,\u0026#34;Message\u0026#34;:\u0026#34;WARNING 2019-06-01 13:47:01.389610437 \u0026#43;0000 UTC RPiGateway ba27ebfffee8c620 CPU temperature 62.8℃\u0026#34;}\n","time":"2019-06-01T13:47:01.389610437Z","duration":0,"level":"WARNING","data":{"series":[{"name":"mqtt_consumer","tags":{"device_id":"ba27ebfffee8c620","device_type":"RPiGateway","host":"rpi-opench-data","name":"temperature","topic":"OpenCH/RPiGateway/ba27ebfffee8c620/PubSystemStats/json","units":"℃"},"columns":["time","value"],"values":[["2019-06-01T13:47:01.389610437Z",62.8]]}]},"previousLevel":"WARNING","recoverable":true}
{"id":"RPiGateway ba27ebfffee8c620 CPU temperature","message":"WARNING 2019-06-01 13:47:01.389610437 +0000 UTC RPiGateway ba27ebfffee8c620 CPU temperature 62.8℃","details":"{\u0026#34;Name\u0026#34;:\u0026#34;mqtt_consumer\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;cpu_alert_temperature\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;nil\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device_id\u0026#34;:\u0026#34;ba27ebfffee8c620\u0026#34;,\u0026#34;device_type\u0026#34;:\u0026#34;RPiGateway\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;rpi-opench-data\u0026#34;,\u0026#34;name\u0026#34;:\u0026#34;temperature\u0026#34;,\u0026#34;topic\u0026#34;:\u0026#34;OpenCH/RPiGateway/ba27ebfffee8c620/PubSystemStats/json\u0026#34;,\u0026#34;units\u0026#34;:\u0026#34;℃\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;localhost\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;56f71fa5-5272-4305-8e00-7fd8955e4faf\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;54052fbb-a451-4323-91e2-4159b4f31294\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;RPiGateway ba27ebfffee8c620 CPU temperature\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;value\u0026#34;:62.8},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2019-06-01T13:47:01.389610437Z\u0026#34;,\u0026#34;Duration\u0026#34;:0,\u0026#34;Message\u0026#34;:\u0026#34;WARNING 2019-06-01 13:47:01.389610437 \u0026#43;0000 UTC RPiGateway ba27ebfffee8c620 CPU temperature 62.8℃\u0026#34;}\n","time":"2019-06-01T13:47:01.389610437Z","duration":0,"level":"WARNING","data":{"series":[{"name":"mqtt_consumer","tags":{"device_id":"ba27ebfffee8c620","device_type":"RPiGateway","host":"rpi-opench-data","name":"temperature","topic":"OpenCH/RPiGateway/ba27ebfffee8c620/PubSystemStats/json","units":"℃"},"columns":["time","value"],"values":[["2019-06-01T13:47:01.389610437Z",62.8]]}]},"previousLevel":"WARNING","recoverable":true}
{"id":"RPiData ba27ebfffe755f19 CPU temperature","message":"OK 2019-06-01 13:45:02.004254415 +0000 UTC RPiData ba27ebfffe755f19 CPU temperature 44.5℃","details":"{\u0026#34;Name\u0026#34;:\u0026#34;mqtt_consumer\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;cpu_alert_temperature\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;nil\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device_id\u0026#34;:\u0026#34;ba27ebfffe755f19\u0026#34;,\u0026#34;device_type\u0026#34;:\u0026#34;RPiData\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;rpi-opench-data\u0026#34;,\u0026#34;name\u0026#34;:\u0026#34;temperature\u0026#34;,\u0026#34;topic\u0026#34;:\u0026#34;OpenCH/RPiData/ba27ebfffe755f19/PubSystemStats/json\u0026#34;,\u0026#34;units\u0026#34;:\u0026#34;℃\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;localhost\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;56f71fa5-5272-4305-8e00-7fd8955e4faf\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;54052fbb-a451-4323-91e2-4159b4f31294\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;RPiData ba27ebfffe755f19 CPU temperature\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;value\u0026#34;:44.5},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2019-06-01T13:45:02.004254415Z\u0026#34;,\u0026#34;Duration\u0026#34;:-119385356022,\u0026#34;Message\u0026#34;:\u0026#34;OK 2019-06-01 13:45:02.004254415 \u0026#43;0000 UTC RPiData ba27ebfffe755f19 CPU temperature 44.5℃\u0026#34;}\n","time":"2019-06-01T13:45:02.004254415Z","duration":-119385356022,"level":"OK","data":{"series":[{"name":"mqtt_consumer","tags":{"device_id":"ba27ebfffe755f19","device_type":"RPiData","host":"rpi-opench-data","name":"temperature","topic":"OpenCH/RPiData/ba27ebfffe755f19/PubSystemStats/json","units":"℃"},"columns":["time","value"],"values":[["2019-06-01T13:45:02.004254415Z",44.5]]}]},"previousLevel":"OK","recoverable":true}
{"id":"RPiData ba27ebfffe755f19 CPU temperature","message":"OK 2019-06-01 13:45:02.004254415 +0000 UTC RPiData ba27ebfffe755f19 CPU temperature 44.5℃","details":"{\u0026#34;Name\u0026#34;:\u0026#34;mqtt_consumer\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;cpu_alert_temperature\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;nil\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device_id\u0026#34;:\u0026#34;ba27ebfffe755f19\u0026#34;,\u0026#34;device_type\u0026#34;:\u0026#34;RPiData\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;rpi-opench-data\u0026#34;,\u0026#34;name\u0026#34;:\u0026#34;temperature\u0026#34;,\u0026#34;topic\u0026#34;:\u0026#34;OpenCH/RPiData/ba27ebfffe755f19/PubSystemStats/json\u0026#34;,\u0026#34;units\u0026#34;:\u0026#34;℃\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;localhost\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;56f71fa5-5272-4305-8e00-7fd8955e4faf\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;54052fbb-a451-4323-91e2-4159b4f31294\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;RPiData ba27ebfffe755f19 CPU temperature\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;value\u0026#34;:44.5},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2019-06-01T13:45:02.004254415Z\u0026#34;,\u0026#34;Duration\u0026#34;:-119385356022,\u0026#34;Message\u0026#34;:\u0026#34;OK 2019-06-01 13:45:02.004254415 \u0026#43;0000 UTC RPiData ba27ebfffe755f19 CPU temperature 44.5℃\u0026#34;}\n","time":"2019-06-01T13:45:02.004254415Z","duration":-119385356022,"level":"OK","data":{"series":[{"name":"mqtt_consumer","tags":{"device_id":"ba27ebfffe755f19","device_type":"RPiData","host":"rpi-opench-data","name":"temperature","topic":"OpenCH/RPiData/ba27ebfffe755f19/PubSystemStats/json","units":"℃"},"columns":["time","value"],"values":[["2019-06-01T13:45:02.004254415Z",44.5]]}]},"previousLevel":"OK","recoverable":true}
2

I am a bit puzzled why this TICK script sends every alert twice, what do I
miss? ```

Do you mean every alert, or just criticals?

    .warn(lambda: int("value") >  cpu_temperature_warn_level)
    .crit(lambda: int("value") >  cpu_temperature_crit_level)

Suppose the value is > crit, and I’m assuming that crit is > warn.

Then value > crit, so you get that alert, but value > warn as well, so you get
that one also.

If value is > warn but < crit, I’d expect you just to get the warning.

Antony.

3

Thanks Antony, I see your point - with the temperature above 80 degrees there will be both indeed - warning and critical. I’ll add extra condition in warning, thanks for your remark.

But my problem is that I see 2 identical alerts with the same timestamp for every single warning even when the temperature is below 80. Please take a look at the small snippet of my /tmp/alerts.log above in the original post.

I see duplicated alerts not only in the /tmp/alerts.log, but also in slack:

kapacitor APP [5:23 PM]
WARNING 2019-06-01 15:17:01.388172632 +0000 UTC RPiGateway ba27ebfffee8c620 CPU temperature 62.3℃
WARNING 2019-06-01 15:17:01.388172632 +0000 UTC RPiGateway ba27ebfffee8c620 CPU temperature 62.3℃
OK 2019-06-01 15:15:02.11081486 +0000 UTC RPiData ba27ebfffe755f19 CPU temperature 45.1℃
OK 2019-06-01 15:15:02.11081486 +0000 UTC RPiData ba27ebfffe755f19 CPU temperature 45.1℃
4

Hi @Yuriy_Olkhovyy ,

the alerts are warnings , with a temperature 62.3 ,
maybe it is the way the datapoints are gathered ?
Can you find the datapoints in the mqtt_consumer measurement that cause these alerts ?
Or it is the where clause in the streamNode that needs more “and”'s ?

best regards ,

5

I checked the database - there was only 1 record with the 2019-06-01 15:17:01.388172632 timestamp.

I think I’ve an idea why this happens - because there is no window node in the script, so all records are fetched from the database every time a new point is written into it. On every new point I get alerts on all previously written points too.

Like this:

1st point arrives - alert1
2nd point arrives - alert1 allert2 <— (there is no window, we get all points all over again)
etc…

This is not how I’d expect the stream work - I thought processing would be applied to the new points only - like stream has a position, new points arrives, you get the poin from the stream, and process it. Apparently it is not how kapacitor’s stream works - you need to specify window to exclude previously processed points, essentially you have to manage the position in the stream with the window.

6

Ok , that can indeed explain , in that case you could also use : statechangesonly