Alerts are not displaying in chronograf dashboard

[Gowrishankar]

I 'm having an issue of getting alerts in alert history in chronograf.

I have created alert rule for cpu with threshold > 5 and task is enabled without handlers below is my tick script , i have written log

var db = ‘telegraf’

var rp = ‘autogen’

var measurement = ‘cpu’

var groupBy =

var whereFilter = lambda: (“cpu” == ‘cpu2’) AND isPresent(“usage_idle”)

var name = ‘cpu_alerts2’

var idVar = name

var message = ’ {{.ID}} {{.Name}}’

var idTag = ‘alertID’

var levelTag = ‘level’

var messageField = ‘message’

var durationField = ‘duration’

var outputDB = ‘chronograf’

var outputRP = ‘autogen’

var outputMeasurement = ‘alerts’

var triggerType = ‘threshold’

var crit = 10

var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)
|eval(lambda: “usage_idle”)
.as(‘value’)
|log()

var trigger = data
|alert()
.crit(lambda: “value” > crit)
.stateChangesOnly()
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.stateChangesOnly()
.log(‘/tmp/alerts.log’)

trigger
|eval(lambda: float(“value”))
.as(‘value’)
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag(‘alertName’, name)
.tag(‘triggerType’, triggerType)

trigger
|httpOut(‘output’)

According to log data is receiving from influxdb to kapacitor , when kapacitor show no data points has been processed and also alerts are not gettting triggered please find below

DOT:
digraph chronograf-v1-b1d33782-2631-4218-8d45-f9a013dfa587 {
graph [throughput=“0.00 points/s”];

stream0 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
stream0 → from1 [processed=“0”];

from1 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
from1 → eval2 [processed=“0”];

eval2 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
eval2 → alert3 [processed=“0”];

alert3 [alerts_inhibited=“0” alerts_triggered=“0” avg_exec_time_ns=“0s” crits_triggered=“0” errors=“0” infos_triggered=“0” oks_triggered=“0” warns_triggered=“0” working_cardinality=“0” ];
alert3 → http_out6 [processed=“0”];
alert3 → eval4 [processed=“0”];

http_out6 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];

eval4 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
eval4 → influxdb_out5 [processed=“0”];

influxdb_out5 [avg_exec_time_ns=“0s” errors=“0” points_written=“0” working_cardinality=“0” write_errors=“0” ];
}

i have also checked Kapacitor stats ingress -
user@user-VivoBook-ASUSLaptop-X409JA-X409JA:~$ kapacitor stats ingress
Database Retention Policy Measurement Points Received
_kapacitor autogen edges 12461
_kapacitor autogen ingress 5125
_kapacitor autogen kafka 318
_kapacitor autogen kapacitor 688
_kapacitor autogen load 688
_kapacitor autogen nodes 11085
_kapacitor autogen runtime 688
_kapacitor autogen topics 1344

kindly help to resolve this issue to generate alerts
Thanks

[scott]

@Gowrishankar Have you connected Chronograf to your Kapacitor instance?

[Gowrishankar]

[Gowrishankar]

Yes ,I have connected to chronograf to kapacitor instance. I’m able to query data in chronograf to kapacitor also

[scott]

Chronograf writes Kapacitor alert events back to InfluxDB and stores them in the alerts measurement. Can you query that measurement to see if any alerts have been stored?

SELECT * FROM alerts

[Gowrishankar]

I’m using influxdb version 2.7.6 where create() db is not possible , hence i have created chronograf bucket in influxdb with autogen retention policy but yet i have no alerts measurement writing to influxdb from chronograf.

when i run kapacitor show it say no data points processed

DOT:
digraph chronograf-v1-b1d33782-2631-4218-8d45-f9a013dfa587 {
graph [throughput=“0.00 points/s”];

stream0 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
stream0 → from1 [processed=“0”];

from1 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
from1 → eval2 [processed=“0”];

eval2 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
eval2 → alert3 [processed=“0”];

alert3 [alerts_inhibited=“0” alerts_triggered=“0” avg_exec_time_ns=“0s” crits_triggered=“0” errors=“0” infos_triggered=“0” oks_triggered=“0” warns_triggered=“0” working_cardinality=“0” ];
alert3 → http_out6 [processed=“0”];
alert3 → eval4 [processed=“0”];

http_out6 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];

eval4 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
eval4 → influxdb_out5 [processed=“0”];

influxdb_out5 [avg_exec_time_ns=“0s” errors=“0” points_written=“0” working_cardinality=“0” write_errors=“0” ];
}

I’m able to create dead man alerts from chronograf and same can be seen in alert history and in influxdb as alert measurement but Threshold alerts are not been create or triggered .