I run Telegraf on all my hosts with Docker containers, and use the Docker input on Telegraf. This does all seem to be set up and working, so I recently tried setting up an alert in Grafana to detect when containers go down. I’ve done this by setting a threshold for “running” containers, so if it drops below that value it should fire.
The problem is that the Docker input appears to be returning “no data” quite frequently. This is causing the alert to fire constantly. I can tweak the alert so that “no data” means “OK” but this negates some of the usefulness as ideally “no data” should suggest a host going down and I’d want to know.
Has anyone else had a similar issue or use the Docker input plugin? I will frequently see the values on the dashboard showing lower than they should be until I manually refresh the query.
