I have a device that publishes both floating-point sensor values and “status” strings via MQTT. I have configured Telegraf to subscribe to these topics, however I’ve had to add two separate instances of the MQTT Consumer plugin in order to handle the different types of data formats:
This results in duplicate data in InfluxDB, because some topics are caught by both consumers and written to the database. The topics for values that are actual strings never appear in the mqtt_consumer measurement, but the topics for float values do appear in the mqtt_consumer_string measurement, as strings.
Apart from creating a list of explicit topics, is there a better way to do this? Or does the data agnostic nature of MQTT mean that there’s no way for Telegraf to know whether a topic is a float or a string (does MQTT not have a data type field?), and having two consumers (one for float topics, one for string topics) is the only way to deal with this?
Alternatively, what if I collected all my data as strings? Will InfluxDB (and downstream services like Grafana) interpret those values as numerical, or will I need to convert them (e.g. Kapacitor TICKscript)?
Is it possible to modify the format of the message in MQTT? If so you could send Line Protocol and use the influx data_format which is much more powerful.
Alternatively, what if I collected all my data as strings? Will InfluxDB (and downstream services like Grafana) interpret those values as numerical, or will I need to convert them (e.g. Kapacitor TICKscript)?
Sending Line Protocol sounds like it could work, at the cost of the published data being less friendly to non-InfluxDB subscribers (no longer a simple value). I’ll think about whether it will work in my application.
Do you know if there’s a way to convert a string to a float in a Grafana query? The query will know that the string is meant to be a float (so that’s the explicit conversion that is required), but I can’t find a way to actually make the conversion. Is there a hidden “cast” or “convert” function, either in Grafana or in the InfluxDB query language? Aggregators like Mode still work on string representations of floats because they treat each value as a member of a set of arbitrary values, but Mean does not work because it needs numerical values to calculate the mean, and if given strings it doesn’t automatically convert.
I tried sending Line Protocol to a single MQTT topic, using the data_format = "influx" setting for [[inputs.mqtt_consumer]] and it does work, but I did notice that the “measurement” part of the Line Protocol seems to be ignored, such that all tags and fields end up under the single “mqtt_consumer” measurement. However by filtering on publisher-supplied tags (rather than MQTT topics with the “value” method) I’m able to extract the right data, as string or float as needed, if I tag it sensibly.
It does feel a bit like an extra hoop to jump through though, and it does mean any other MQTT subscribers (like a mobile app or websocket app) don’t see nice per-topic values coming in, as everything arrives via a single topic. I’ll do some more testing and consider it as a possible solution though.
It’s a shame that the measurement name isn’t used by Telegraf when sending the data to InfluxDB. That would have been a really nice solution to another problem I have where everything via MQTT ends up in one “mqtt_consumer” measurement and certain operations are not possible unless they are split between measurements. The workaround for that is currently multiple instances of the mqtt_consumer input plugin, with name_override to set each measurement’s name.
I’ve narrowed it down to use of the name_override directive, which I need to use as I have multiple instances of the MQTT input plugin. It seems that this override also overrides the Line Protocol measurement names, which doesn’t seem right to me:
If I don’t use the name_override directive the measurement name is correctly handled. I can work around this by using the default instance for any LP handling, but it doesn’t seem quite right to me.
Ok, thanks, I was under the impression I needed distinct names for each instance of the plugin, but if they can happily co-exist then I can omit the name_override whenever I’m using the influx data format. Thanks for the pointers.
