Good day! I have recently updated my Telegraf and InfluxDB docker environments to v1.21.4 and v2.1.1. I am adding aggregate tasks in InfluxDB, and for that I want to transform the data that is sent by my Telegraf clients that use the MQTT_consumer input plugin. I would like to use the topic levels to determine the measurement and topic names, so I can aggregate based on the topic. I came across the topic parsing feature, which sounds like it does what I want. I could not get it to work, so I setup a separate test environment, with the following Telegraf configuration:
[agent] interval = "60s" round_interval = true metric_batch_size = 1000 metric_buffer_limit = 10000 flush_interval = "60s" debug = true omit_hostname = true [[outputs.influxdb_v2]] urls = ["$INFLUX_URL"] token = "$INFLUX_TOKEN" organization = "$INFLUX_ORG" bucket = "$INFLUX_BUCKET" timeout = "10s" [[inputs.mqtt_consumer]] servers = [ "tcp://$MQTT_IP:$MQTT_PORT", ] qos = 1 connection_timeout = "300s" topics = [ "homewizard/#", "parser/#" ] persistent_session = false client_id = "Telegraf_HomeWizard_Test" data_format = "json" [[inputs.mqtt_consumer.topic_parsing]] topic = "_/topic" measurement = "measurement/_"
I added the topic_parsing section. I only specified the topic and measurement parsings, as the field is in the json data. The parser/# topic was added for testing. I have a Docker container that sends MQTT messages to topic homewizard/dbtemperatureall for example, with a JSON message as contents. The message format is:
I also tried sending test messages interactively to topic parser/test with message:
I expected these messages to be put in measurements homewizard and parser, with topics being dbtemperatureall and test. So I expected the first level in the topic to become the measurement name, and the second level the topic. However, data is stored in measurement mqtt_consumer, and topic is the complete topic level, so homewizard/dbtemperatureall for example. Same for my test message. I know that the parser is active, because Telegraf won’t start if I just comment out the measurement line in the parser section.I’m also sending to an InfluxDB v2 environment setup specifically for parse testing, so I know the data is being received into InfluxDB. It looks like the messages are not parsed at all.
I read this article but do not see where my logic is failing. The feature is fairly new, so there is not a lot of information to be found.
Does anyone see where I go wrong?