We were exploring the Kapacitor for alerting. We have decided to do put all the alert in version control and have a CI which will configure the alert on the Kapacitor.
We are facing the following challenges:
- How to test these alert locally, so that I will be sure that my alert would work?
- Is there any way, we can generate InfluxQL query from tick script?
- Is there any we can send test notification from tick script.?
- If u r following the official tutorial, they use telegraf to collect the cpu usuage and log them into influx database. When ever an alert happens it will log it into influxdb. U can query influxdb about number of alerts. If u r not using any of the TICK stack, may be one way of testing it is just by creating a simple POST alert in kapacitor.
Example:
I created a simple node js server on my local machine that listen to a port and print the post requests on the terminal. And in TICK script u can add some thing like this.
|alert()
.crit(lambda: "value" > crit)
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.post('http://192.168.0.1:3123')
.header('content-type', 'application/json')
In this case, I am sending a post to 192.168.0.1(or localhost/127.0.0.1) on port 3123 and don’t forget to allow the port to listen for http request. I highly recommend chronograf user interface if u r a beginner, it will make life easy.
-
Yes you can create queries. Check these example codes
-
I don’t understand what’s test notification is !! but in general u can send notifications. There is plenty of documentation out there.