Cant figure out topic parsing from big mqtt input

Telegraf
1

So Ive tapped in to our energy meter that sends out data from an rj12 port.
Ive managed to send this to a mqtt broker and trying to ingest it into a influxDB with the help of Telegraf mqtt plugin.

I have 4 main topics that Im listening for
“esp32_204B18/Energy”,
“esp32_204B18/Power”,
“esp32_204B18/Voltage”,
“esp32_204B18/Current”,

The payload from mqtt broker:
esp32_204B18 #Device Main

Energy
	Active
		Positive
			value = 00002167.887
			unit = kWh
		Negative
			value = 00000000.000
			unit = kWh
	Reactive
		Positive
			value = 00000436.252
			unit = kVArh
		Negative
			value = 00000000.217
			unit = kVArh
Power 
	Instant
		Active
			Positive
				value = 0002.706
				unit = kW
				L1
					value = 0000.937
					unit = kW
				L2
					value = 0000.973
					unit = kW
				L3
					value = 0000.794
					unit = kW
			Negative
				value = 0000.000
				unit = kW
				L1
					value = 0000.000
					unit = kW
				L2
					value = 0000.000
					unit = kW
				L3
					value = 0000.000
					unit = kW
		Reactive
			Positive
				value = 0000.855
				unit = kVAr
				L1
					value = 0000.452
					unit = kVAr
				L2
					value = 0000.214
					unit = kVAr
				L3
					value = 0000.188
					unit = kVAr
			Negative
				value = 0000.000
				unit = kVAr
				L1
					value = 0000.000
					unit = kVAr
				L2
					value = 0000.000
					unit = kVAr
				L3
					value = 0000.000
					unit = kVAr
Voltage 
	Instant
		L1
			value = 227.7
			unit = V
		L2
			value = 227.9
			unit = V
		L3
			value = 232.8
			unit = V
Current 
	Instant 
		L1
			value = 004.5
			unit = A
		L2
			value = 004.3
			unit = A
		L3
			value = 003.5
			unit = A

Been trying parsing but simply cant get it to work…
Anyone feeling bored and willing to explain/show how I need to think?
Value is always a float as the numbers change for every broker payload update.
They come in every 10s from the broker.

Sorry for being such n00b :pensive:

2

Does the payload actually look exactly as shown in the code block? So only indented text?

3

I use a mqtt viewer and that’s what I see… :-/

4

Can you change the output format of your iot devices to a more common format (json, xml, …)?
This indented text format is quite unusual and there is no parser in Telegraf that can parse it easily.

You could try grok input format or the processors.regex plugin, but that would be a real pain.

I would probably solve it with a processors.starlark or processors.execd plugin and unfortunately write the parser myself, because as far as I know there are no libraries that can parse this format either.

5

I quickly wrote a parser in Starlark for your payload.
However, the parser only works with tabs or 4 spaces as indentations and the values may only be float.
Logging can be removed from the Starlark script if it is no longer needed.

[[inputs.mqtt_consumer]]
  # add this to your input plugin:
  name_override = "mqtt"
  data_format = "value"
  data_type = "string"

[[processors.starlark]]
  source = '''
load("logging.star", "log") # only needed for logging

def apply(metric):
  hierarchy = list()
  depth = 0
  if metric.name == "mqtt":
    value = metric.fields.pop("value")  # get payload
    lines = value.splitlines(False)  # split into lines
    for line in lines:
      if line.strip() == "":
        continue  # ignore empty lines in payload
      elif "unit" in line:
        continue  # ignore lines with unit
      elif "value" in line:
        floatvalue = float(line.strip().split(" = ")[1])
        value_name = "_".join(hierarchy)  # build field name
        metric.fields[value_name] = floatvalue  # write metric
        log.debug("{} : {}".format(value_name, floatvalue))
      else:
        depth = 0
        while line.startswith("\t"):  # remove leading tabs
          line = line.split("\t", 1)[1]
          depth += 1
        while line.startswith("    "):  # remove leading 4 spaces
          line = line.split("    ", 1)[1]
          depth += 1
        key_name = line.strip()
        # log.debug("{} : {}".format(key_name, depth))
        if depth == 0:
          hierarchy.clear()
          hierarchy.append(key_name)
        elif depth >= len(hierarchy):
          hierarchy.append(key_name)
        elif depth < len(hierarchy):
          hierarchy = hierarchy[:depth]
          hierarchy.append(key_name)
  return metric
'''
6

Perhaps some preprocessing/parsing in ‘Node-RED’ is easier?