Is there a way for Telegraf to notify if it hasn't seen data from a source for some time?

Hello, I have only very recently been introduced to the Influx tools, so please bear with me!

I have several sensors which wake from a deep sleep to occasionally publish messages over MQTT, and my goal is to get the measurements into an InfluxDB database. It seems from a little reading that a reasonable way to do this (please feel free to suggest alternatives) would be to use Telegraf combined with the MQTT Consumer Input Plugin, which connects to the MQTT broker (Mosquitto).

My problem is that I would like to have some way to be notified if one of the sensors goes down. Something similar to the up{} metric in Prometheus . In that case, the metric contains a 1 if the scrape was successful, and 0 if it failed. Now, ofcourse it is easy in that case because Prometheus is pull-based, whereas Influx however is push-based.

I am wondering if there is some way to configure in Telegraf to create an event (just a 1 or 0) if a message has not been published to a topic for some time, say for a hour? Is there a way I can achieve this with the mentioned tools?

I cannot simply periodically poll the sensors in this case because they spend almost all of their time in deep sleep.

Thank you!

I can’t think of a way to do this easily today. In the next Telegraf release we are hoping to have a processor that will run a custom script, should be possible then. However, I think you will be best off setting up a deadman’s alert in InfluxDB or Kapacitor. This way you will also catch any issues with Telegraf reporting the sensor data as well.

Hi Daniel, thanks for the reply, I’ll keep an eye out for the new release. Could you briefly show / explain how I would implement this type of alert in InfluxDB? I assume you are referring to this https://docs.influxdata.com/kapacitor/v1.5/nodes/alert_node/. Forgive me, but I am not too familiar with Inlufx yet.

I see that this is something which should be implemented with the “Flux” scripting language, and that these Flux commands can be used in the Influx CLI tool. I can see in this doc for example https://v2.docs.influxdata.com/v2.0/reference/flux/stdlib/monitor/deadman/ . At the moment, I have InfluxDB running as a service in the background. Is there some configuration file or something where I should type these commands to set my monitor, and have it watching automatically when the service starts?

With InfluxDB 1.x I would use Kapacitor, you can start with this guide.

If you are using InfluxDB 2.x then you can be configured in the GUI. I’m unsure how to set it up from the CLI with Flux query, I haven’t tried this myself, but I believe you need to create a task to run the script.