MQTT topic and tag reference to a user defined table - possible?

Telegraf
1

Guys,

I have the following working telegraf.conf

[global_tags]

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  debug = false
  hostname = ""
  omit_hostname = false


[[outputs.file]]
  files = ["stdout"]

[[inputs.mqtt_consumer]]
  alias = "mqtt_consumer_value"
  client_id = "telegraf_value"
  servers = ["tcp://127.0.0.1:1883"]
  username = "UUUU"
  password = "pppp"
  name_override = "logdata"        # measurement
  topic_tag = "addr"            # tag, instead of "topic"
  topics = [
    "knx/2/2/2",
    "knx/2/0/3",
    "mtr/wasser",
    "mtr/hzg/temp_VL",
    "mtr/hzg/temp_RL",
    "mtr/hzg/temp_RLWW",
    "knx/4/5/7",
  ]
  data_format = "value"
  data_type = "string"

It reads topic / value pairs from mqtt alright. Ultimately, the output will go into InfluxDB but for testing and debugging the [[output.file]] is active, for the time being.

As you may notice, some topics are cryptic, like “knx/2/2/2”. It would require additional knowledge later-on to interpret what was written to InfluxDB. The approach I have in mind to solve it is like following:
a) create a array of tables (question: where? in the [global_tag] section?)
b) reference to this table in the [[inputs.mqtt_consumer]] section (question: how?)
c) and create an additional tag with the “user friendly name” (ufn). (question: how?)

The array of tables could look like:

topicmap = [
  { topic = "knx/2/2/2", ufn = "temperature" },
  { topic = "knx/2/0/3", ufn = "energy" },
]

and the [[inputs.mqtt_consumer]] may loke like

...
topics = [ topicmap.topic ]
tags = [ topicmap.ufn ]
...

However, while writing, I realize that a construct like that may require an adaptation of the [[inputs.mqtt_consumer]] plugin…

Does anyone have any insights here to be shared? Or an different approach to tackle my initial problem?

Thanks

2

Two options immediately come to mind:

  1. Use a starlark processor that has a dictionary of topics to more complex name. If the topic matches, add the new tag.

  2. Use the lookup processor with a file that has the mappings

3

The [[processors.lookup]] is actually what I was looking for. Thank you.
However, not being fluent with Go, I’m stuck with the following error message

2023-04-13T17:16:16Z I! Loading config file: /etc/telegraf/telegraf.conf
2023-04-13T17:16:16Z I! Starting Telegraf 1.26.0
2023-04-13T17:16:16Z I! Available plugins: 235 inputs, 9 aggregators, 27 processors, 22 parsers, 57 outputs, 2 secret-stores
2023-04-13T17:16:16Z I! Loaded inputs: mqtt_consumer (2x)
2023-04-13T17:16:16Z I! Loaded aggregators:
2023-04-13T17:16:16Z I! Loaded processors: enum lookup
2023-04-13T17:16:16Z I! Loaded secretstores:
2023-04-13T17:16:16Z I! Loaded outputs: file influxdb
2023-04-13T17:16:16Z I! Tags enabled: host=LeoDB
2023-04-13T17:16:16Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"LeoDB", Flush Interval:10s
2023-04-13T17:16:16Z D! [agent] Initializing plugins
2023-04-13T17:16:16Z D! [agent] Connecting outputs
2023-04-13T17:16:16Z D! [agent] Attempting connection to [outputs.influxdb]
2023-04-13T17:16:16Z D! [agent] Successfully connected to outputs.influxdb
2023-04-13T17:16:16Z D! [agent] Attempting connection to [outputs.file]
2023-04-13T17:16:16Z D! [agent] Successfully connected to outputs.file
2023-04-13T17:16:16Z D! [agent] Starting service inputs
2023-04-13T17:16:16Z I! [inputs.mqtt_consumer::mqtt_consumer_jsonv2] Connected [tcp://127.0.0.1:1883]
2023-04-13T17:16:16Z I! [inputs.mqtt_consumer::mqtt_consumer_value] Connected [tcp://127.0.0.1:1883]
2023-04-13T17:16:16Z **E! [processors.lookup] generating key failed: template: key:1:2: executing "key" at <.Tag>: can't evaluate field Tag in type *metric.trackingMetric**
2023-04-13T17:16:16Z D! [processors.lookup] metric was &{logdata_test map[addr:mtr/hzg/temp_RL host:LeoDB] map[value:24.8] 1681406176470229230 0xc00014e200}

The relevant part of telegraf.conf looks like

...
[[processors.lookup]]
  ## List of files containing the lookup-table
  files = [ "/etc/telegraf/ufn.csv" ]

  ## Format of the lookup file(s)
  ## Available formats are:
  ##    json               -- JSON file with 'key: {tag-key: tag-value, ...}' mapping
  ##    csv_key_name_value -- CSV file with 'key,tag-key,tag-value,...,tag-key,tag-value' mapping
  ##    csv_key_values     -- CSV file with a header containing tag-names and
  ##                          rows with 'key,tag-value,...,tag-value' mappings
  format = "csv_key_values"

  ## Template for generating the lookup-key from the metric.
  ## This is a Golang template (see https://pkg.go.dev/text/template) to
  ## access the metric name (`{{.Name}}`), a tag value (`{{.Tag "name"}}`) or
  ## a field value (`{{.Field "name"}}`).
  key = '{{.Tag "addr"}}'

with ufn.csv being like

# ignored,tag-name1,...,tag-valueN
ignored,                     ufn
knx/2/2/2,                   heat-bath-duration-(min)
knx/2/0/3,                   circulation-pomp-(min)
...

Any help is appreciated.

4

I attributed the issue I have been experiencing to my ignorance of GOLANG. With reference to the bug described here I will test with the next telegraf release.

5

With telegraf 1.26.2, the issue is solved.

1 Like
6

@universal-dilettant you are also aware of the fact that you can directly read from KNX with the corresponding input plugin, aren’t you!?

7

@srebhan No I wasn’t aware. Like I did mention in another thread, I’m relatively new to telegraf. But thx for the hint. :grinning:

1 Like