I’m afraid i don’t know how to configure alerts from within Grafana, we were unable to include some of the template tags from Grafana in our alerts so we only use Grafana purely for our dashboards.
Our alerts are handled by Kapacitor, the tick script itself is generated by Chronograf but the code would be the same whether generated from Chronograf or written inside an IDE.
A quick question, are you wanting to alert when a node with the TICK stack goes offline? if so, then the following won’t work as Kapacitor won’t be running if the node is offline. If thats the case, then you might need to ask on the Grafana forums about setting an alert up for this. Maybe based on throughput, I’m unsure.
Otherwise, if you montor multiple nodes with per your TICK servers (as in 3 or 4 nodes reporting into one influx server) and want to alert on one of those nodes going offline, the following should work.
Chronograf is just an easy way to get to grips with TICK script in general, however if you don’t want to install Chronograf in your stack then I’d recommend having a look through some of the sample TICK scripts
From there, you can create a base template to work from.
deadman script above, if you update the DB and RP variable to use your database and retention policy and upload this script to one of your servers you can enable it using the Kapacitor CLI.
Upload the script, you can leave the script wherever you choose. I move them into /etc/kapacitor/tick but running it from the home directory for now would suffice.
Once the scripts on the server, you can use the following to enable the script (assuming you are using linux)
sudo kapacitor define deadman_alert -type stream -tick /path/to/script -dbrp database.retention_policy
sudo kapacitor enable deadman_alert
When a node goes offline, the alert should trigger. But, as i say… If you want to monitor the TICK server itself then Kapacitor won’t be able to help you, unless you monitor your tick nodes with separate instance dedicated to “watching the watcher”.
We use an heartbeat API through OpsGenie to alert us if our TICK server is offline, that might be something worth looking into.
I think though maybe you are wanting to alert when your TICK node is offline. In this instance, you might need to ask on the Grafana forums about configuring this.
Let me know if theres anything else.