“I want to learn about the alert mechanism and how is the best way to set them. I know that I can not use alerts if I use variables. Yet, when you have 500+ node to monitor let’s say simply cpu usage 90% for 1 min, how do you set alerts? Of course there are many other metrics. Should I create different dashboards for alerts separated completely from the main monitoring dashboards?”
It’s not Influx related directly but as far as I see this community is more active so I take my chances.
Also, I don’t have go with grafana. I would like to hear your experiences with TICK stack. Should I try to implement kapacitor? Alerting is crucial since dashboards does not scale.
I would really like to know how you manage alerts and incidents in your environments.
Hello @Mert,
Thanks for your question. Yes we generally advise that you use Kapacitor to handle large amounts of alerts if you insist on using the TICK stack. I suggest you start creating a TICK Script with Chronograf first to start learning it.
@Mert,
If you are planning on setting threshold or deadman alerts, I would suggest also looking into using Influx 2.0 and integrated alerts. It’s super simple!
I haven’t tried influx 2.0 since it’s on alpha builds. I set TICK stack up yet been struggling with alert rules for hours. I’ve configured chronograf and kapacitor. Kapacitor is able to send test alerts via SMTP. I’ve created simple dummy alert rules, even though thresholds crossed no alerts arised.
@Mert,
That is very odd. Can you verify that Kapacitor is running? Can you run ````kapacitor logs``` and then please try recreating the following alert (without Alert Handlers to start).
My initial set up was on docker. I’ve installed another kapacitor instance on a linux vm and now everything looks fine. Still, couldn’t handle the docker setup. There was no errors in the logs. Also, chronograf alert history was stuck with loading animation.
Currently, Influx and chronograf are working on docker and kapacitor on linux. I can go with this setup. Yet, I’m wondering if docker setup is suggested for production. Docker TICK Stack in Production
I’ll keep digging and post back if I find where I’m doing wrong. (:
