Telegraf does not create a measurement called mqtt_consumer

Telegraf
, , ,
1

Hello,

I set up MQTT → Mosquitto → Telegraf → InfluxDB 1.8 → Grafana. All of this software is running on Docker on a Raspberry Pi with Raspberry PI OS.

Data is received from Mosquitto and taken by Telegraf to InfluxDB.

My telegraf.conf is the following:

[global_tags]

[agent]
  interval = "30s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = "BKW"
  omit_hostname = false
  debug = true

[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  database = "influx"
  timeout = "5s"
  username = "*****"
  password = "******"

[[inputs.mqtt_consumer]]
  servers = ["tcp://mosquitto:1883"]
  topics = [
     "inverter/+/+/+",
     "inverter/+/+",
  ]
  qos = 0
  connection_timeout = "30s"
  username = "*****"
  password = "******"
  data_format = "value"
  data_type = "float"

[[inputs.mqtt_consumer.topic_parsing]]
  topic = "inverter/+/+/+"
  measurement = "measurement/_/_/_"
  tags = "_/name/ch/field"

[[inputs.mqtt_consumer.topic_parsing]]
  topic = "inverter/+/+"
  measurement = "measurement/_/_"
  tags = "_/name/field"

[[processors.pivot]]
  tag_key = "field"
  value_key = "value"

This generates only the measurement inverter in the influxdb. But no measurement called mqtt_consumer.

The data in inverter is complete, but not useable by Grafana. The table contains for each value an own column. But the transformation in telegraf should actually output a smaller version which contains the field and the value.

In case I run mosquitto_sub with the given topic selection I receive what I need:

mosquitto_sub -h localhost -t inverter/+/+/+ -u -P
inverter/BKW/ch1/YieldDay 274
inverter/BKW/ch1/YieldTotal 12.814
inverter/BKW/ch2/YieldDay 840
inverter/BKW/ch2/YieldTotal 16.973
inverter/BKW/ch0/YieldDay 1114
inverter/BKW/ch0/YieldTotal 29.787

Running telegraf in test mode returned:

2023-04-23T19:20:57Z I! Loading config file: /etc/telegraf/telegraf.conf
2023-04-23T19:20:57Z I! Starting Telegraf 1.26.1
2023-04-23T19:20:57Z I! Available plugins: 235 inputs, 9 aggregators, 27 processors, 22 parsers, 57 outputs, 2 secret-stores
2023-04-23T19:20:57Z I! Loaded inputs: mqtt_consumer
2023-04-23T19:20:57Z I! Loaded aggregators: 
2023-04-23T19:20:57Z I! Loaded processors: pivot
2023-04-23T19:20:57Z I! Loaded secretstores: 
2023-04-23T19:20:57Z W! Outputs are not used in testing mode!
2023-04-23T19:20:57Z I! Tags enabled: host=BKW
2023-04-23T19:20:57Z D! [agent] Initializing plugins
2023-04-23T19:20:57Z D! [agent] Starting service inputs
2023-04-23T19:20:57Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2023-04-23T19:20:57Z D! [agent] Stopping service inputs
2023-04-23T19:20:57Z D! [inputs.mqtt_consumer] Disconnecting [tcp://mosquitto:1883]
2023-04-23T19:20:57Z D! [inputs.mqtt_consumer] Disconnected [tcp://mosquitto:1883]
2023-04-23T19:20:57Z D! [agent] Input channel closed
2023-04-23T19:20:57Z D! [agent] Processor channel closed
2023-04-23T19:20:57Z D! [agent] Stopped Successfully

Which looks actually normal.

Furthermore the log files of telegraf do not show any issues:

2023-04-23T17:58:13Z I! Loading config file: /etc/telegraf/telegraf.conf
2023-04-23T17:58:13Z I! Starting Telegraf 1.26.1
2023-04-23T17:58:13Z I! Available plugins: 235 inputs, 9 aggregators, 27 processors, 22 parsers, 57 outputs, 2 secret-stores
2023-04-23T17:58:13Z I! Loaded inputs: mqtt_consumer
2023-04-23T17:58:13Z I! Loaded aggregators: 
2023-04-23T17:58:13Z I! Loaded processors: pivot
2023-04-23T17:58:13Z I! Loaded secretstores: 
2023-04-23T17:58:13Z I! Loaded outputs: influxdb
2023-04-23T17:58:13Z I! Tags enabled: host=BKW
2023-04-23T17:58:13Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"BKW", Flush Interval:10s
2023-04-23T17:58:13Z D! [agent] Initializing plugins
2023-04-23T17:58:13Z D! [agent] Connecting outputs
2023-04-23T17:58:13Z D! [agent] Attempting connection to [outputs.influxdb]
2023-04-23T17:58:13Z D! [agent] Successfully connected to outputs.influxdb
2023-04-23T17:58:13Z D! [agent] Starting service inputs
2023-04-23T17:58:13Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2023-04-23T17:58:23Z D! [outputs.influxdb] Wrote batch of 10 metrics in 20.122597ms
2023-04-23T17:58:23Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-04-23T17:58:33Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-04-23T17:58:43Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-04-23T17:58:53Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics

What I do wrong in order to get the measurement mqtt_consumer with the data Grafana actually needs?

Thanks

2

Unfortunately, it does not. Test mode should print out any metrics that Telegraf has received and parsed. However, because you are using the mqtt consumer, a service input, metrics are only created when they are received, not at every interval. As such --test mode will not work with mqtt consumer.

2023-04-23T17:58:23Z D! [outputs.influxdb] Wrote batch of 10 metrics in 20.122597ms

It looks like Telegraf did write something based on this log message. Are you really really sure nothing got written? :slight_smile:

3

It writes data, but not in mqtt_consumer. But I could solve it by removing the sections inputs.mqtt_consumer.topic_parsing.