Telegraf [processors.starlark]: issues only one topic appeared

is there something wrong with this telegraf config ?

  urls = [""]
  token = ""
  organization = "."
  bucket = "MQTT input"
  exclude_bucket_tag = false
  timeout = "600s"

  servers = ["mqtt://", "mqtt://", "mqtt://"]
  topics = ["sensor/#","sensor/lokasi/+", 
  qos = 1
  connection_timeout = "600s"
  max_undelivered_messages = 1000
  persistent_session = true
  client_id = "telegraf_cf820687"
  username = "."
  password = "."
  data_format = "value"
  data_type = "float"
  source = '''
def apply(metric):
    topic = metric.tags.get("topic")
    if topic.startswith("sensor/"):
        parts = topic.split("/")
        if len(parts) >= 4:
            topik_ke3 = parts[2]
            if topik_ke3 == "kedalaman" :
              metric.fields["kedalaman"] = metric.fields["value"]
            elif topik_ke3 == "meter1":
              metric.fields["meter1"] = metric.fields["value"]
            elif topik_ke3 == "meter2":
              metric.fields["meter2"] = metric.fields["value"]
            elif topik_ke3 == "meter3":
              metric.fields["meter3"] = metric.fields["value"]

    return metric

Hello everyone, i have quite troubled for the past days regarding the problem above.
the code above supposedly make input with the specific 3rd part of topic which is “kedalaman”, “meter1”, “meter2”, or “meter3” appeared in influxdb cloudless. but went i tested by input values on that specified topic, only “kedalaman” topic that going in influx cloudless not the other 3. because i new to starlark code idk how to make it so other specified topic can be appeared in influx as well.


I would enable logging in your starlark configuration and ensure that the sensors are as you expected and the parts[2] is what you expected.

Then enable [[outputs.file]] to get the actual metrics generated by telegraf.


I saw your config you sent me. Nothing stands out as wrong, but can you share some metrics from [[outputs.file]]?

To add, what would really help me is this:

  1. Give me some example messages
  2. Show me what you want the messages to look like in line protocol.

We can probably work on something for you to get you going.

this is the metrics,
i does several test with another topic other than that one and apparently it didnt send any errors or some kind that tell me it didnt succesfully sent

here the history of me sending some input with different topics other than sensor/lokasi/kedalaman in 20:05 WIB to 20:08 Western Indonesian time

2024-06-26T20:04:25+07:00 D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:04:28+07:00 D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:04:38+07:00 D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:04:38+07:00 D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:04:50+07:00 D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:04:51+07:00 D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:05:01+07:00 D! [outputs.file] Wrote batch of 6 metrics in 40.4062ms
2024-06-26T20:05:01+07:00 D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:05:03+07:00 D! [outputs.influxdb_v2] Wrote batch of 6 metrics in 1.1699926s
2024-06-26T20:05:03+07:00 D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:05:13+07:00 D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:05:17+07:00 D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:07:48+07:00 D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-06-26T20:07:51+07:00 D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics


and for this one i want it simple

if i/or the sensor input 54 on topic sensor/lokasi/kedalaman, 64 on topic sensor/lokasi/meter1, and so on

i want it to be like this

mqtt_consumer,topic=sensor/lokasi/kedalaman kedalaman=54 1719407101413763000
mqtt_consumer,topic=sensor/lokasi/meter1 meter1=64 1719407101413763000

Here is how I would do it:

  1. First use the regex processor to extract out the sensor tag. This assumes that your topics are of the format /sensor/host/name. It will create a temporary tag called sensor with the value name.
  2. Then using starlark take the sensor tag value and replace the value field. So the field value=42 and tag sensor=meter2 becomes the field meter2=42.
  order = 1
    key = "topic"
    pattern = '.*\/(.*)'
    replacement = "${1}"
    result_key = "sensor"

  order = 2
  source = '''
def apply(metric):
    # Replace "value" field with sensor name
    if metric.tags.get("sensor") and metric.fields.get("value"):
        metric.fields[metric.tags["sensor"]] = metric.fields["value"]

    return metric

Let me know if that works for you. This transforms:

mqtt_consumer,topic=sensor/lokasi/kedalaman value=32
mqtt_consumer,topic=sensor/lokasi/meter1 value=64


mqtt_consumer,topic=sensor/lokasi/kedalaman kedalaman=32 1719413009000000000
mqtt_consumer,topic=sensor/lokasi/meter1 meter1=64 1719413009000000000

I will add, that doing this is not ideal for InfluxDB. You are creating what we call “sparse” metrics. Meaning that your metrics have a lot of different field names that are not always present. This means that your metrics have lots of null fields if you think about it like a spreadsheet or table:

Topic kedalaman meter1 meter2
sensor/lokasi/meter2 null null 33
sensor/lokasi/meter1 null 22 null
sensor/lokasi/kedalaman 11 null null

i try the code but somehow it give me this error in that above code

error parsing data: line 58: invalid TOML syntax

and also is it fine to add [processors.starlark] below the [[processors.regex.tags]]

we can see in this quote that the color between processors.starlark and processor.regex.tags is the same but went i try it on telegraf that in influxdb cloud the color of processor.starlark is the same as the rest as the line code. is it fine like that?

oh shoot I missed a quote above. I’ve updated the config.

Specifically, this was missing the closing quote:

pattern = '.*\/(.*)'

hi, the code work it transforms value field into desired field but here the catch
it only somehow work went the topic only sensor/lokasi/kedalaman

mqtt_consumer,topic=sensor/lokasi/kedalaman kedalaman=48 1719415960423360900
mqtt_consumer,topic=sensor/lokasi/kedalaman kedalaman=48 1719415960423360900
mqtt_consumer,topic=sensor/lokasi/kedalaman kedalaman=23 1719416187701302900
mqtt_consumer,topic=sensor/lokasi/kedalaman kedalaman=23 1719416187701302900

and also

i didnt quite get it on the commented code about changing “value” with sensor name am i needed to change that into sensor name or just let the code stay at it is

Are you sure you are getting other topics? Because I showed you that it is possible to parse out others.

Well went i test it with different topics (after i change the code like you suggested) like sensor/lokasi/depth, sensor/lokasi/length, sensor/lokasi/meter
It didn’t register to influx it like it ignored the different topics completely beside sensor/lokasi/kedalaman.
I already test it in different bucket the problem still persist

Nothing in those processors would prevent additional topics from getting landed. Are you seeing any topics actually delivered?

If you use outputs.file do you see other topics?

Not sure how to help you as the examples you have shown have only shown that one topic as well.

I use output.file and i didnt see other topics other than that specific topic so I’m not sure how to tell what’s wrong with it because there no error or indication that the message with other topics failed to sent.
I saw and input several topics getting delivered only on mqtt consumer but only that specific topic get processed by telegraf

Are you sure these are correctly specified?

after the changed i found that only “sensor/lokasi/kedalaman” and “sensor/lokasi/kedalaman/#” is going in my table. and with “sensor/lokasi/kedalaman/#” any topic after kedalaman work like in example

so i’m not gonna change that anytime soon

so since this code will have a lot of null values in the fields that i worked with is there a way to not show that null values went try to present that fields in influx cloud or any other visual tool? (sorry if the above question is confusing beforehand)

The correct way to handle this is to use the originally structured value field and the topic tag. This way you can filter on a topic tag and the value field will show you your values.

1 Like

I see thx for the answer