Newbie question on parsing MQTT

Telegraf
,
1

Hello all, newbie on Telegraf here. I’m having a hard time understanding how to get measurements from mqtt to influxdb. I added my mqtt sub to Telegraf in inputs.mqtt_consumer and all the temperature and humidity readings are written into the influxDB just nicely. However, these are not;

/my/topic {"time" : "2022-12-12 16:27:01", "model" : "ShellyGarage", "power" : "861.68"}
/my/topic {"time" : "2022-12-12 16:27:02", "model" : "ShellyMitsuILP", "power" : "458.21"}
/my/topic {"time" : "2022-12-12 16:27:02", "model" : "ShellyPM4-0", "power" : "0"}
/my/topic {"time" : "2022-12-12 16:27:02", "model" : "ShellyPM4-1", "power" : "0"}
/my/topic {"time" : "2022-12-12 16:27:02", "model" : "ShellyPM4-2", "power" : "0"}
/my/topic {"time" : "2022-12-12 16:27:02", "model" : "ShellyPM4-3", "power" : "0"}

What do I need to do in order to get these measurements into the Influxdb?

Thanks

2

Am I correct in assuming that Telegraf didn’t know about “power”, so I have to parse that somehow or help Telegraf along the way? I really don’t know where to start, I have read on inputs.mqtt_consumer.topic_parsing but since the topic is the same for all my measurements I guess I have to look at something else. Please help.

3

Hi,

Can you share your telegraf config?

Can you also share an example of the temperature readings that you are parsing? Is temperature also reported as a string? Are these power readings read from a 2nd topic?

You may need to specify a specific topic, or tell telegraf how to parse the data you are getting differently, but I don’t know enough without some additional info.

Thanks

4 
# egrep -v '^($|#|  #)' /etc/telegraf/telegraf.conf
[global_tags]
[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  hostname = ""
  omit_hostname = false
[[outputs.influxdb]]
urls = ["http://127.0.0.1:8086"]
database = "mqtt_data"
username = "mqtt"
password = "password"
[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false
  core_tags = false
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.mqtt_consumer]]
servers = ["ssl://server.name:8883"]
topics = [
	"/my/topic"
]
client_id = "telegraf@grafana"
username = "harald"
password = "password"
data_format = "json_v2"
/my/topic {"time" : "2022-12-12 16:27:04", "model" : "Fineoffset-WH2", "id" : 167, "temperature_C" : 16.700, "humidity" : 43, "mic" : "CRC", "mod" : "ASK", "freq" : 433.922, "rssi" : -12.144, "snr" : 13.872, "noise" : -26.016}
/my/topic {"time" : "2022-12-12 16:27:04", "model" : "Fineoffset-WH2", "id" : 231, "temperature_C" : 5.900, "humidity" : 54, "mic" : "CRC", "mod" : "ASK", "freq" : 433.921, "rssi" : -12.144, "snr" : 13.665, "noise" : -25.810}
> select * from mqtt_consumer limit 10
name: mqtt_consumer
time                battery_ok button channel freq    gap  host    humidity id  impulses noise   rssi    snr    subtype temperature_C topic
----                ---------- ------ ------- ----    ---  ----    -------- --  -------- -----   ----    ---    ------- ------------- -----
1670853676581996622                           433.913      grafana 46       135          -26.016 -0.114  25.902         17            /my/topic
1670853678553051307                           433.911      grafana          136          -27.673 -0.136  27.537         -10.5         /my/topic
1670853680650901601                           433.926      grafana                       -28.165 -0.102  28.063         -20           /my/topic
1670853681174219163 1                 1       433.966      grafana          167 67867397 -27.231 -0.106  27.125                       /my/topic
1670853681699008291                   1       433.964 1361 grafana          167          -28.72  -0.121  28.598                       /my/topic
1670853689038213976                           433.921      grafana 43       167          -26.704 -12.144 14.559         16.7          /my/topic
1670853689044534972                           433.92       grafana 55       231          -25.612 -12.144 13.468         3.5           /my/topic
1670853692710018786 1          0      3       433.935      grafana 33       213          -28.165 -0.106  28.059 9       20.3          /my/topic
1670853698999954514                           433.916      grafana 72       215          -27.831 -12.131 15.699         -9.6          /my/topic
1670853708962160643                           433.925      grafana          199          -26.346 -0.13   26.217         17.2          /my/topic

Everything is in one topic.

Thank You, appreciate Your time.

5

Everything is in one topic.

Is power a 2nd message?

@srebhan what am I missing?

6

This is how the mqtt sub looks like;

/my/topic {"time" : "2022-12-15 10:49:36", "model" : "FT-004B", "temperature_C" : -19.800, "mod" : "ASK", "freq" : 433.930, "rssi" : -0.099, "snr" : 19.566, "noise" : -19.664}
/my/topic {"time" : "2022-12-15 10:49:38", "model" : "Norgo-NGE101", "id" : 167, "channel" : 1, "gap" : 1496, "mic" : "CRC", "mod" : "ASK", "freq" : 433.965, "rssi" : -0.130, "snr" : 20.083, "noise" : -20.213}
/my/topic {"time" : "2022-12-15 10:49:38", "model" : "Norgo-NGE101", "id" : 167, "channel" : 1, "gap" : 1496, "mic" : "CRC", "mod" : "ASK", "freq" : 433.962, "rssi" : -0.137, "snr" : 20.217, "noise" : -20.354}
/my/topic {"time" : "2022-12-15 10:49:45", "model" : "Fineoffset-WH2", "id" : 199, "temperature_C" : 16.600, "mic" : "CRC", "mod" : "ASK", "freq" : 433.927, "rssi" : -0.103, "snr" : 19.840, "noise" : -19.943}
/my/topic {"time" : "2022-12-15 10:49:47", "model" : "Fineoffset-WH2", "id" : 215, "temperature_C" : -8.600, "humidity" : 73, "mic" : "CRC", "mod" : "ASK", "freq" : 433.916, "rssi" : -7.439, "snr" : 12.557, "noise" : -19.996}
/my/topic {"time" : "2022-12-15 10:49:56", "model" : "Fineoffset-WH2", "id" : 183, "temperature_C" : -4.100, "humidity" : 82, "mic" : "CRC", "mod" : "ASK", "freq" : 433.915, "rssi" : -0.133, "snr" : 20.489, "noise" : -20.621}
/my/topic {"time" : "2022-12-15 10:50:00", "model" : "Fineoffset-WH2", "id" : 135, "temperature_C" : 17.200, "humidity" : 43, "mic" : "CRC", "mod" : "ASK", "freq" : 433.917, "rssi" : -0.135, "snr" : 20.836, "noise" : -20.971}
/my/topic {"time" : "2022-12-15 10:50:01", "model" : "ShellyIVTILP", "power" : "528.35"}
/my/topic {"time" : "2022-12-15 10:50:02", "model" : "ShellyGarage", "power" : "1.79"}
/my/topic {"time" : "2022-12-15 10:50:02", "model" : "ShellyMitsuILP", "power" : "754.66"}
/my/topic {"time" : "2022-12-15 10:50:02", "model" : "ShellyPM4-0", "power" : "0"}
/my/topic {"time" : "2022-12-15 10:50:02", "model" : "ShellyPM4-1", "power" : "0"}
/my/topic {"time" : "2022-12-15 10:50:02", "model" : "ShellyPM4-2", "power" : "0"}
/my/topic {"time" : "2022-12-15 10:50:02", "model" : "ShellyPM4-3", "power" : "0"}
/my/topic {"time" : "2022-12-15 10:50:02", "model" : "SumPower", "power" : "1284.80"}
/my/topic {"time" : "2022-12-15 10:50:05", "model" : "Fineoffset-WH2", "id" : 136, "temperature_C" : -8.500, "mic" : "CRC", "mod" : "ASK", "freq" : 433.913, "rssi" : -0.128, "snr" : 20.373, "noise" : -20.501}
/my/topic {"time" : "2022-12-15 10:50:09", "model" : "Prologue-TH", "subtype" : 9, "id" : 213, "channel" : 3, "battery_ok" : 1, "temperature_C" : 19.800, "humidity" : 33, "button" : 0, "mod" : "ASK", "freq" : 433.935, "rssi" : -0.123, "snr" : 20.146, "noise" : -20.269}

In my own scripts I use “model” and “id” to identify what I’m reading.

7

Thank You for this tip. It indeed looks like I have been posting “power” as a string and that’s why Telegraf didn’t omnomnom it. This now works as expected.

8

I now don’t have any identifer in influxdb though. So I am unable to identify which sensor has reported power.

Is there a way to store say “model” in influxdb as well? What is the key for measurements in influx?

edit: I started posting a numerical ID as well, that sorted that. I learned that telegraf and / or influxdb doesn’t really care about strings and only stores numerical data. Right?

9

Generally strings are tags, something you would want to filter on. When parsing your data, if there is a specific string value you want to store as a field you need to specify it explicitly.

10

Thanks. Is this specifying of a string value that I want to store in influxdb doable in telegraf.conf?

11

Yeah with the json_v2 parser you can specify it:

        [[inputs.mqtt_consumer.json_v2.field]]
            path = "" # A string with valid GJSON path syntax to a non-array/non-object value
            rename = "new name" # A string with a new name for the tag key
            type = "int" # A string specifying the type (int,uint,float,string,bool)
            ## Setting optional to true will suppress errors if the configured Path doesn't match the JSON
            optional = false

of course in your case it will be a “string” type.

1 Like