Using telegraf MQTT to bring Shelly Plus H&T into influxdb

Stefonaut · October 21, 2023, 1:11pm

Hi,

I have a Shelly Plus H&T temperature and humidity sensor that transmits its readings to an MQTT broker. I want to use telegraf to get these readings into my InfluxDB (v1.8). I have already looked at the MQTT Consumer Input Plugin and experimented with it.
However, the problem is that the Shelly splits its readings into different Topics. Each of these topics contains a JSON object with a different structure.

Here is an excerpt of the transmitted data:
grafik

I have compiled the following configuration, exemplary exclusively for the value tC. However, I get numerous error messages from Telegraf for all JSON objects of the other topics that this value cannot be found. This is understandable, the value only exists in one of the JSON objects.

[[inputs.mqtt_consumer]]
  servers = ["tcp://mosquitto:1883"]
  topics = [
    "shellyplusht-80XXXXXXXXXX/status/+"
  ]
  topic_tag = "topic"
  username = "telegraf"
  password = "well_its_a_password"
  data_format = "json_v2"

  [[inputs.mqtt_consumer.topic_parsing]]
    topic = "+/+/+"
    tags = "shelly/_/_"

  [[inputs.mqtt_consumer.json_v2]]
    measurement_name = "shellies_plusht"
    [[inputs.mqtt_consumer.json_v2.field]]
      path = "tC"
      type = "float"

telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [inputs.mqtt_consumer] No metrics were created from a message. Verify your parser settings. This message is only printed once.
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-21T12:28:51Z E! [inputs.mqtt_consumer] Error in plugin: the path tC doesn't exist
telegraf_mqtt  | shellies_plusht,shelly=shellyplusht-80XXXXXXXXXX,topic=shellyplusht-80XXXXXXXXXX/status/temperature:0 tC=26 1697891331440149159
telegraf_mqtt  | 2023-10-21T12:28:56Z D! [outputs.file] Wrote batch of 1 metrics in 97.54µs
telegraf_mqtt  | 2023-10-21T12:28:56Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

Is that intentional? Do I just have to ignore the error messages and add my other values? Can I parse this better?

I can provide more information if necessary.

Thank you!

Anaisdg · October 24, 2023, 6:10pm

Hello @Stefonaut,
This user had a similar problem:

I’m not sure if it’s intentional but it does make sense if it only exits in one of the JSON objs.

I think you might be okay.

@jpowers do you have any better feedback here?

jpowers · October 24, 2023, 9:28pm

@Stefonaut,

The JSON v2 config that you have added will apply to every topic that attempts to get read. In your case, only the temperature topic has this field, while everything else does not, hence the 9 messages about the field not existing.

What is your goal?

If you want to ignore the error messages you can. You can also set optional = true under the inputs.mqtt_consumer.json_v2.field option and it should suppress the additional errors as well.

Stefonaut · October 25, 2023, 3:17pm

@Anaisdg @jpowers,

thank your for your input.

I tried with this config now, also trying to get more than one value:

[[inputs.mqtt_consumer]]
  topics = [
    "shellyplusht-80XXXXXXXXXX/status/+"
  ]
  topic_tag = "topic"
  [[inputs.mqtt_consumer.topic_parsing]]
    topic = "+/+/+"
    tags = "shelly/_/_"
  [[inputs.mqtt_consumer.json_v2]]
      measurement_name = "shellies_plusht"
      [[inputs.mqtt_consumer.json_v2.field]]
        path = "battery.V"
        type = "float"
        optional = true
      [[inputs.mqtt_consumer.json_v2.field]]
        path = "battery.percent"
        type = "float"
        optional = true
      [[inputs.mqtt_consumer.json_v2.field]]
        path = "rh"
        type = "float"
        optional = true
      [[inputs.mqtt_consumer.json_v2.field]]
        path = "tC"
        type = "float"
        optional = true

I get the following output

telegraf_mqtt  | 2023-10-25T15:00:32Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf_mqtt  | 2023-10-25T15:00:37Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:37Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:37Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:37Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:37Z D! [inputs.mqtt_consumer] No metrics were created from a message. Verify your parser settings. This message is only printed once.
telegraf_mqtt  | 2023-10-25T15:00:37Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:37Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:37Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:37Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:38Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:42Z D! [outputs.file] Wrote batch of 3 metrics in 81.57µs
telegraf_mqtt  | 2023-10-25T15:00:42Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf_mqtt  | shellies_plusht,shelly=shellyplusht-80XXXXXXXXXX,topic=shellyplusht-80XXXXXXXXXX/status/devicepower:0 V=5.96,percent=98 1698246038741057241
telegraf_mqtt  | shellies_plusht,shelly=shellyplusht-80XXXXXXXXXX,topic=shellyplusht-80XXXXXXXXXX/status/humidity:0 rh=53.4 1698246038753424629
telegraf_mqtt  | shellies_plusht,shelly=shellyplusht-80XXXXXXXXXX,topic=shellyplusht-80XXXXXXXXXX/status/temperature:0 tC=25.4 1698246038766134957
telegraf_mqtt  | 2023-10-25T15:00:48Z D! [parsers.json_v2::mqtt_consumer] the path battery.V doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:48Z D! [parsers.json_v2::mqtt_consumer] the path battery.percent doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:48Z D! [parsers.json_v2::mqtt_consumer] the path rh doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:48Z D! [parsers.json_v2::mqtt_consumer] the path tC doesn't exist
telegraf_mqtt  | 2023-10-25T15:00:52Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

Unfortunately, optional=true does not seem to suppress the error messages, as the complete path cannot be found in 9 out of 10 topics. From the topics where the path is found, however, it correctly builds the measurement.
Can the search that [[inputs.mqtt_consumer.json_v2.field]] performs be restricted to certain Topics?

jpowers · October 25, 2023, 3:32pm

Unfortunately, optional=true does not seem to suppress the error messages,

Those are now debug messages not errors like before. Note the D vs E. As I mentioned, the optional config option suppresses the error messages.

Can the search that [[inputs.mqtt_consumer.json_v2.field]] performs be restricted to certain Topics?

You can by specifying a different mqtt_consumer plugin for each topic you want to monitor.

Stefonaut · October 26, 2023, 4:05pm

Those are now debug messages not errors like before. Note the D vs E.

You are right. I had debug set to true, so they of course will still be shown.

You can by specifying a different mqtt_consumer plugin for each topic you want to monitor.

Would i then still be able to combine all the information into one point put into the measurement? Right now i get this, following your previous advice:

telegraf_mqtt  | shellies_plusht,shelly=shellyplusht-80XXXXXXXXXX V=5.93,percent=96 1698335561434231399
telegraf_mqtt  | shellies_plusht,shelly=shellyplusht-80XXXXXXXXXX rh=57.9 1698335561450663939
telegraf_mqtt  | shellies_plusht,shelly=shellyplusht-80XXXXXXXXXX tC=24.9 1698335561468381481

I plan on using a pivot (https://www.influxdata.com/blog/pivot-mqtt-plugin/) to combine this into that:

telegraf_mqtt  | shellies_plusht,shelly=shellyplusht-80XXXXXXXXXX V=5.93,percent=96,rh=57.9,tC=24.9 1698335561434231399

Thanks again!

Stefonaut · October 26, 2023, 4:11pm

Scratch that, I must have misunderstood the pivot plugin. I’ll have to take a closer look.

ThiefMaster · January 24, 2024, 12:00am

Did you find a solution in the end? I think my problem is VERY similar to yours…

Stefonaut · January 24, 2024, 5:29pm

Unfortunately, I have not found a completely satisfactory solution.
I have settled for not being able to save all values in my InfluxDB. I also had to do without some tags.
For reference, below is my complete telegraf config.

[global_tags]
[agent]
  interval = "1s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  hostname = ""
  omit_hostname = true
[[outputs.influxdb]]
    urls = ["http://influxdb:8086"]
    database = "db_telegraf"
    skip_database_creation = true
    username = "telegraf"
    password = "GarbleGarbleGarble"
[[inputs.mqtt_consumer]]
  servers = ["tcp://mosquitto:1883"]
  topics = [
    "shellyplusht-123123123123/status/+"
  ]
  topic_tag = ""
  username = "telegraf"
  password = "GarbleGarbleGarble"
  data_format = "json_v2"
  [[inputs.mqtt_consumer.topic_parsing]]
    topic = "+/+/+"
    tags = "shelly/_/_"
  [[inputs.mqtt_consumer.json_v2]]
      measurement_name = "shellies_plusht"
      [[inputs.mqtt_consumer.json_v2.field]]
        path = "battery.V"
        rename = "batteryVoltage"
        type = "float"
        optional = true
      [[inputs.mqtt_consumer.json_v2.field]]
        path = "battery.percent"
        rename = "batteryPercent"
        type = "int"
        optional = true
      [[inputs.mqtt_consumer.json_v2.field]]
        path = "external.present"
        rename = "externalPresent"
        type = "bool"
        optional = true
      [[inputs.mqtt_consumer.json_v2.field]]
        path = "rh"
        type = "float"
        optional = true
      [[inputs.mqtt_consumer.json_v2.field]]
        path = "tC"
        type = "float"
        optional = true

Jason_Stirnaman · January 24, 2024, 6:10pm

If each point that you want to merge contained the same timestamp, then you could use the Merge aggregator plugin. However, if your timestamps are different (as in your example), how will you decide which points to merge and what the final timestamp should be?

Jason_Stirnaman · January 24, 2024, 6:12pm

IIUC, you might want telegraf/plugins/aggregators/final at master · influxdata/telegraf · GitHub

Jason_Stirnaman · January 25, 2024, 3:07pm

Sorry to mislead you, the final plugin won’t work for this.
See @ThiefMaster’s example.
If you don’t need nanosecond time precision, you can reduce your timestamp precision so that your points fall into the same series. Then, you can use Telegraf to merge them or you can write them to InfluxDB and let the database merge them.

Topic		Replies	Views
Basic JSON>MQTT>Telegraf issues Telegraf telegraf	2	7329	October 1, 2017
MQTT FLOAT AND PAYLOAD MIXED Help to Config Telegraf	5	476	May 17, 2023
Newbie question on parsing MQTT Telegraf influxdb , mqtt	10	694	December 15, 2022
Parsing data from MQTT to telegraf Telegraf	3	462	October 16, 2023
Not able to get Telegraf to write mqtt messages to influxDB Telegraf influxdb , telegraf , mqtt , json	1	312	October 30, 2023

Using telegraf MQTT to bring Shelly Plus H&T into influxdb

Related topics