Write data despite missing tag value

Telegraf
, ,
1

Hi,
I’m using Telegraf with the MQTT-Input-Plugin to write data to an InfluxDB.
I’m parsing the input from the MQTT-source as JSON.

Here is a part of my configuration conf-file:

[[inputs.mqtt_consumer.json_v2]]
	[[inputs.mqtt_consumer.json_v2.tag]]
    	path = "deviceName"

    [[inputs.mqtt_consumer.json_v2.tag]]
    	path = "devEUI"		
    
	[[inputs.mqtt_consumer.json_v2.tag]]
    	path = "tags.location"		

	[[inputs.mqtt_consumer.json_v2.tag]]
    	path = "tags.municipality"		

	[[inputs.mqtt_consumer.json_v2.tag]]
    	path = "tags.sensortype"

The tags “tags.location”, “tags.municipality” and “tags.sensortype” in the source data are in the end from manual inputs when registering sensors. So they might be missing, when some forget to put in a value. Still the sensors will send data that I want to write into the database.
But Telegraf dismisses the complete data set, when one of the configured tags is missing.
Is there a way to force Telegraf to still write the data, though one of the tags is missing?
Would be fine for me to write the missing tag as “” or “null” or whatever.

2

To solve this, you can use a processor plugin in Telegraf to set default values for missing tags. The defaults processor is perfect for this.

Here’s how you can modify your configuration:

[[inputs.mqtt_consumer.json_v2]]
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "deviceName"
  
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "devEUI"
  
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "tags.location"
  
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "tags.municipality"
  
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "tags.sensortype"

# Add this processor to handle missing tags
[[processors.defaults]]
  [processors.defaults.tags]
    location = "unknown"
    municipality = "unknown" 
    sensortype = "unknown"

If some of these tags are actually required for your data to make sense, you might want to use more specific default values or set up alerting to notify you when tags are missing.

3

@skartikey , thank you very much for your reply! Sorry that it took a while before I became able to come back to this topic.
I tried your advice but for reasons I don’t know it still doesn’t work. I removed one of the tags (usecase) from the incoming data and telegraf stoped to write data into InfluxDB.

I looked the documentation for the plugin up and found everything exactly as you suggested.

Here is an excerpt from my conf-file:

[[processors.defaults]]
  [processors.defaults.tags]
    location = "-"
	municipality = "-"
	sensortype = "-"
	usecase = "-"

In the telegraf logs I found this:

2025-06-26T07:29:52Z E! [inputs.mqtt_consumer::hapysc_coop_garden_data] Error in plugin: the path "deviceInfo.tags.usecase" doesn't exist

Do I have to put the [[processors.defaults]]-part maybe before the [[inputs.mqtt_consumer]]-part, so that the tags are set before they are processed? :thinking:

4

Thinking about my own post it doesn’t make sense. Before the [[inputs.mqtt_consumer]]-part the tags doesn’t even exist.

As far as I understand it is not an issue that I have an empty tag, but the issue is that the MQTT-input-plugin does not find the path “deviceInfo.tags.usecase” and stops processing the JSON object. Maybe the [[processors.defaults]]-plugin doesn’t help in this case?

5

@cortlieb You can make tags optional.

[[inputs.mqtt_consumer.json_v2]]
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "deviceName"
  
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "devEUI"
  
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "deviceInfo.tags.location"
    optional = true
  
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "deviceInfo.tags.municipality"
    optional = true
  
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "deviceInfo.tags.sensortype"
    optional = true
    
  [[inputs.mqtt_consumer.json_v2.tag]]
    path = "deviceInfo.tags.usecase"
    optional = true

# Then use defaults processor to set default values for missing tags
[[processors.defaults]]
  [processors.defaults.tags]
    location = "-"
    municipality = "-"
    sensortype = "-"
    usecase = "-"
6

@skartikey , thank you for your quick answer :folded_hands: ! I promise I will be also more responsive this time!

With the optional-configuration the following happens:

  • data is written when one of the tags marked optional is missing
  • the value (“-”) I try to put to the missing usecase-tag by the defaults-processor is not written to the tag. The tag is still completely missing.
  • the tag-set is different between data sets with usecase-tag and without. Therefore data is for example drawn in the InfluxDB UI “Data Explorer” in two separate curves (different colors).
    I think we’re close, but actually having the missing tag with a default value would be the perfect solution!