Prevent json_v2 from splitting JSON field into separate fields

Telegraf
1

I have the following Influx Telegraf / InfluxDB Cloud 2.0 issue:

I’m sending an event MQTT message like this, containing a new event as well as the list of all currently active issues:

{
  "timestamp": "2025-03-07T14:00:00.000Z",
  "event": {
    "type": "CONNECTION_QUALITY",
    "state": "BAD",
    "metadata": {
      "failures": 59,
      "failureRate": "0.98"
    },
    "severity": "CRITICAL"
  },
  "active_issues": [
    {
      "type": "CONNECTION_QUALITY",
      "since": "2025-03-07T14:00:00.000Z",
      "last_update": "2025-03-07T14:00:00.000Z",
      "state": "BAD",
      "severity": "CRITICAL",
      "metadata": {
        "failures": 59,
        "failureRate": "0.98"
      }
    },
    {
      "type": "MEMORY_ALMOST_FULL",
      "since": "2025-03-07T12:00:00.000Z",
      "last_update": "2025-03-07T12:00:00.000Z",
      "status": "ACTIVE",
      "severity": "WARNING",
      "metadata": {
        "usedMemory": 123456
      }
    }
  ]
}

My goal would be to declare type a tag and put state, status, severity into separate fields. metadata is something I can’t set for all types now and I don’t want to customize this config every time a new type is added. So, I would like to declare metadata as a plain string because I assume Telegraf / InfluxDB cannot automatically create sub-jsons. The same is true for the array of active_issues.

Now, using that config:

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = true
  debug = false
  quiet = false


[[outputs.file]]
  files = ["stdout"]
  #tagexclude = ["topic"]
  
# Bucket 'site'
[[outputs.influxdb_v2]]
  urls = ["https://eu-central-1-1.aws.cloud2.influxdata.com"]
  token = "$INFLUX_TOKEN"
  organization = "MyOrg"
  bucket = "site"
  user_agent = "telegraf_bucket_site"
  namepass = [""events"]
  tagexclude = ["topic"]


# site / events
# Read system and component status events from MQTT
[[inputs.mqtt_consumer]]
  servers = ["ssl://123456789012-ats.iot.eu-west-1.amazonaws.com:8883"]
  tls_ca = "/tmp/telegraf/AmazonRootCA1.pem"
  tls_cert = "/tmp/telegraf/telegraf-certificate.pem"
  tls_key = "/tmp/telegraf/telegraf-key.pem"
  client_id = "telegraf_status"
  persistent_session = true
  qos = 1
  topics = ["controller/+/events"]
  name_override = "events"
  data_format = "json_v2"
  precision = "1s"

  # Extract the 'controller_id' from the topic:
  [[inputs.mqtt_consumer.topic_parsing]]
    topic = "controller/+/events"
    tags  = "_/controller_id/_"

  [[inputs.mqtt_consumer.json_v2]]
    timestamp_path   = "timestamp"
    timestamp_format = "2006-01-02T15:04:05Z07:00"

    # Turn event.code into a Telegraf "tag"
    [[inputs.mqtt_consumer.json_v2.tag]]
      path   = "event.type"
      rename = "event_type"

    # Turn event.status into a string field
    [[inputs.mqtt_consumer.json_v2.field]]
      path   = "event.status"
      type   = "string"
      rename = "status"
      optional = true

    # Turn event.state into a string field
    [[inputs.mqtt_consumer.json_v2.field]]
      path   = "event.state"
      type   = "string"
      rename = "state"
      optional = true

    # Turn event.severity into a string field
    [[inputs.mqtt_consumer.json_v2.field]]
      path   = "event.severity"
      type   = "string"
      rename = "severity"

    # Store the entire event.metadata object as one JSON string
    [[inputs.mqtt_consumer.json_v2.field]]
      path    = "event.metadata"
      rename  = "metadata"
      type    = "string"   # <- THIS IS IGNORED
      optional = true

    # Store the entire active_issues array (with all nested data) as one JSON string
    [[inputs.mqtt_consumer.json_v2.field]]
      path    = "active_issues"
      rename  = "active_issues"
      type    = "string"   # <- THIS IS IGNORED AS WELL
      optional = true

… I end up having this in the database:

#group,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false
#datatype,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,dateTime:RFC3339
#default,,,,,,,,,,,,,,,,,
,active_issues_code,active_issues_last_update,active_issues_metadata_failureRate,active_issues_metadata_failures,active_issues_metadata_meta1,active_issues_metadata_meta2,active_issues_metadata_usedMemory,active_issues_severity,active_issues_since,active_issues_state,active_issues_status,active_issues_type,controller_id,event_type,severity,state,time
,,2025-03-07T12:00:00.000Z,0.98,59,,,123456,WARNING,2025-03-07T12:45:00.000Z,BAD,ACTIVE,MEMORY_ALMOST_FULL,Controller_123456,CONNECTION_QUALITY,CRITICAL,BAD,2025-03-07T14:46:00Z

So, I try to tell Telegraf to handle metadata and active_issues like plain strings but it’s ignoring that part of the configuration.
Instead of having plain strings like {"failures": 59, "failureRate": "0.98"} it’s flattening the JSON inside metadata into 'active_issues_metadata_failures and active_issues_metadata_failureRate.
The same is happening with the JSON inside active_issues.

I couldn’t find a way to stop Telegraf to flatten my JSON. How can I do this, how can I declare a json to be treated like a string?

2

Does anyone have any ideas? :slight_smile:

3

Hello @NNNN,
You might consider using the execd input plugin or the xpath_json instead because when you specify a path to an object or array in a field configuration, the parser will automatically flatten the nested structure rather than treating it as a string, even when you specify with the type.

Another approach might be to use the object configuration in the JSON v2 parser with specific included/excluded keys to control what gets flattened.

But I’d take a look at