Alerts are not displaying in chronograf dashboard

Gowrishankar · May 21, 2024, 7:10am

I 'm having an issue of getting alerts in alert history in chronograf.

I have created alert rule for cpu with threshold > 5 and task is enabled without handlers below is my tick script , i have written log

var db = ‘telegraf’

var rp = ‘autogen’

var measurement = ‘cpu’

var groupBy =

var whereFilter = lambda: (“cpu” == ‘cpu2’) AND isPresent(“usage_idle”)

var name = ‘cpu_alerts2’

var idVar = name

var message = ’ {{.ID}} {{.Name}}’

var idTag = ‘alertID’

var levelTag = ‘level’

var messageField = ‘message’

var durationField = ‘duration’

var outputDB = ‘chronograf’

var outputRP = ‘autogen’

var outputMeasurement = ‘alerts’

var triggerType = ‘threshold’

var crit = 10

var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)
|eval(lambda: “usage_idle”)
.as(‘value’)
|log()

var trigger = data
|alert()
.crit(lambda: “value” > crit)
.stateChangesOnly()
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.stateChangesOnly()
.log(‘/tmp/alerts.log’)

trigger
|eval(lambda: float(“value”))
.as(‘value’)
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag(‘alertName’, name)
.tag(‘triggerType’, triggerType)

trigger
|httpOut(‘output’)

According to log data is receiving from influxdb to kapacitor , when kapacitor show no data points has been processed and also alerts are not gettting triggered please find below

DOT:
digraph chronograf-v1-b1d33782-2631-4218-8d45-f9a013dfa587 {
graph [throughput=“0.00 points/s”];

stream0 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
stream0 → from1 [processed=“0”];

from1 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
from1 → eval2 [processed=“0”];

eval2 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
eval2 → alert3 [processed=“0”];

alert3 [alerts_inhibited=“0” alerts_triggered=“0” avg_exec_time_ns=“0s” crits_triggered=“0” errors=“0” infos_triggered=“0” oks_triggered=“0” warns_triggered=“0” working_cardinality=“0” ];
alert3 → http_out6 [processed=“0”];
alert3 → eval4 [processed=“0”];

http_out6 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];

eval4 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
eval4 → influxdb_out5 [processed=“0”];

influxdb_out5 [avg_exec_time_ns=“0s” errors=“0” points_written=“0” working_cardinality=“0” write_errors=“0” ];
}

i have also checked Kapacitor stats ingress -
user@user-VivoBook-ASUSLaptop-X409JA-X409JA:~$ kapacitor stats ingress
Database Retention Policy Measurement Points Received
_kapacitor autogen edges 12461
_kapacitor autogen ingress 5125
_kapacitor autogen kafka 318
_kapacitor autogen kapacitor 688
_kapacitor autogen load 688
_kapacitor autogen nodes 11085
_kapacitor autogen runtime 688
_kapacitor autogen topics 1344

kindly help to resolve this issue to generate alerts
Thanks

scott · May 21, 2024, 10:40pm

@Gowrishankar Have you connected Chronograf to your Kapacitor instance?

Gowrishankar · May 22, 2024, 5:00am

Gowrishankar · May 22, 2024, 5:02am

Yes ,I have connected to chronograf to kapacitor instance. I’m able to query data in chronograf to kapacitor also

scott · May 22, 2024, 2:12pm

Chronograf writes Kapacitor alert events back to InfluxDB and stores them in the alerts measurement. Can you query that measurement to see if any alerts have been stored?

SELECT * FROM alerts

Gowrishankar · May 23, 2024, 5:23am

I’m using influxdb version 2.7.6 where create() db is not possible , hence i have created chronograf bucket in influxdb with autogen retention policy but yet i have no alerts measurement writing to influxdb from chronograf.

when i run kapacitor show it say no data points processed