We’ve just started experimenting and expanding things with a TICK stack and I was tasked with making some alerts based on our data. After I made a half dozen alerts for various services, we suddenly noticed one alert (a Deadman) wasn’t alerting anymore. It would still send a slack alert, but there would be nothing in Chronograf’s alert history.
We dog down and found that we were getting write errors on InfluxDBOut
influxdb_out7 [avg_exec_time_ns="5.877µs" errors="1" points_written="0" working_cardinality="0" write_errors="1" ];
influxdb_out7 [avg_exec_time_ns="0s" errors="1" points_written="0" working_cardinality="0" write_errors="1" ];
Since this was all test data, we wiped out the alerts and started over. Now the deadman is working but different alerts stopped. They all log and alert slack, but they can’t write to the database.
I’ve been googling around to try and find solutions since Monday, and at this point, any direction would be appreciated.