Hi, I’m trying to setup some simple kapacitor alerts for testing. If I create a deadman alert via chronograf it works fine (i.e. if I kill the host, I will receive an opsgenie alert and will see it in the chronograf history). However, I cannot for the life of me get “threshold” or “relative” alerts to work. They just never trigger. I’ve tried many, many different combinations of stats, and it’s clearly visible on the chronograf preview that the stats have passed the threshold, and yet they never trigger (nothing in kapacitor logs).
My guess is I’m making some mistake when selecting data, but since I’ve tried “group_by”, individual hosts, and many, many combinations, I really don’t know where to turn next. I’ve followed the chronograf alert tutorial and the kapacitor manual tutorial, but neither of them triggered.
Any information that might point me in the right direction would be greatly appreciated - but please remember, it’s not an issue with the alerting mechanism, since that was successfully tested with the deadman.
EDIT: I’m using Kapacitor 1.5.1 and Chronograf 1.7.3. I’ve also tried manual tick scripts, but still with no success.
EDIT2: screenshot showing the threshold is passed, and yet no alert triggered