Json_v2 parsing multiple objects in same payload results in ignored objects

[whchung]

Hello!

I’m trying to parse a JSON payload using telegraph json_v2. I’d like to use multiple [[inputs.xxx.json_v2.object]] sections to parse different objects in the payload. However, only the first object is getting parsed and written to InfluxDB. If I add a json_v2.object section for the second object, then there are no error messages but nothing gets written. Same thing if I just have one json_v2.object section for the second object only. Any ideas why the parser seems to be ignoring anything beyond the first object?

My input JSON (made generic to avoid sensitive details):

{
  "object1": {
    "firstname": "John",
    "lastname": "Smith",
    "address": 10,
    "type": "residential"
  },
  "object2": {
    "lat": "0",
    "lon": "1"
  },
  "object3": {
    "Deliver": true,
    "Type": "0x1",
    "Version": "0x02",
    "location1": {
      "temp": "25",
      "minTemp": "20",
      "maxTemp": "30"
    },
    "location2": {
      "temp": "25",
      "minTemp": "20",
      "maxTemp": "30"
    },
    "AvgTemp": "20",
    "CurrentTemp": "21"
  }
}

My telegraf config snippet:

[[inputs.xxx.json_v2.object]]
path = "object1"
tags = ["firstname", "last name", "type"]

[[inputs.xxx.json_v2.object]]
path = "object2"
tags = ["lat", "lon"]

telegraf is happy to parse object1 and write it to InfluxDB. But object2 never gets parsed as far as I can tell. I thought maybe only one json_v2.object section was allowed? However, even if I just include the object definition for object2, it results in nothing written to InfluxDB.

Thank you for any suggestions about what may be wrong!

[phill]

Hi @whchung - Welcome to the Community Forum!

Check the indentation. The example below was copied from working code in my conf file.

 data_format = "json_v2"

  [[inputs.http.json_v2]]
    measurement_name = "something"

  # TOML subtables for JSON V2 parser
    [[inputs.http.json_v2.object]]
      path = "myPath"
      included_keys = ["myKey"]
      tags = ["myTag"]

    [[inputs.http.json_v2.object]]
      path = "anotherPath"
      included_keys = ["anotherKey"]

There is also a JSON Playground where you can verify that your path (the part inside the quotes) is picking off the correct name-value pairs.

Phill

[whchung]

Phill,

Thank you for your response! Yes, I used the JSON Playground to verify my paths.

Your example shows more than one json_v2.object section with more than one path. So it looks like using more than one json_v2.object section should work . I’ll try this again, following your example with regard to indentation, now that I know indentation matters.

whchung

[whchung]

Can each of the json_v2.object sections have a different tag definition?

Also, what does the indentation look like if I’m using mqtt_consumer with additional subsections, such as topic parsing? I tried the snippet below using the indentation from your example with no luck. The second json_v2.object section is ignored; I get no “anotherMyTag” despite the path “anotherPath” being correct.

data_format = "json_v2"
  [[inputs.mqtt_consumer.topic_parsing]]
    topic = "+/+/+"

  [[inputs.mqtt_consumer.json_v2]]
    measurement_name = "something"

  # TOML subtables for JSON V2 parser
    [[inputs.http.json_v2.object]]
      path = "myPath"
      included_keys = ["myKey"]
      tags = ["myTag"]

    [[inputs.http.json_v2.object]]
      path = "anotherPath"
      tags = ["anotherMyTag"]

[whchung]

Interestingly, using a single json_v2.object definition works. At least I get tags for both objects. I needed to specify the tags using a path prefix with _, not ., as the separator. I’m now wondering again if using two json_v2.object definitions is a problem, especially with the mqtt_consumer plugin vs http?

[[inputs.mqtt_consumer.json_v2.object]]
  path = "@this"
  tags = ["myPath_myKey", "anotherPath_anotherKey"]

[phill]

@whchung

@whchung

  [[inputs.mqtt_consumer.json_v2]]
    measurement_name = "something"

  # TOML subtables for JSON V2 parser
    [[inputs.http.json_v2.object]]

Be careful - This is mixing inputs.mqtt with inputs.http which won’t work. All TOML subtables below the initial [[inputs.whatever]] plugin declaration must start with the same [[inputs.whatever. . . .]]

@whchung

Can each of the json_v2.object sections have a different tag definition?

Yes

[whchung]

@phill Sorry, I’m using inputs.mqtt_consumer with everything, with no http. The mixing in my example was a copy/paste error . So I’m really using something like the snippet below. I never get tag “anotherMyTag” to show up unless I only use one [[inputs.mqtt_consumer.json_v2]] section and then define the tags using the object path prefix as I showed above.

data_format = "json_v2"
  [[inputs.mqtt_consumer.topic_parsing]]
    topic = "+/+/+"

  [[inputs.mqtt_consumer.json_v2]]
    measurement_name = "something"

  # TOML subtables for JSON V2 parser
    [[inputs.mqtt_consumer.json_v2.object]]
      path = "myPath"
      included_keys = ["myKey"]
      tags = ["myTag"]

    [[inputs.mqtt_consumer.json_v2.object]]
      path = "anotherPath"
      tags = ["anotherMyTag"]

[phill]

@whchung
Take a look at this post. If you can’t get everything to work in 1 plugin, you can split it up into more. Also mqtt_consumer parsing is discussed in more detail.

MQTT plugin not writing data to influxdb

Hi @jmo @phill, So it is more than acceptable to have two or even more of the same plugin type within your config. In some cases we even encourage it to spread the load with large datasets. Can I propose a config reshuffle: [[inputs.mqtt_consumer]] servers = ["tcp://192.168.50.36:1883"] topics = [ "perfscore" ] topic_tag = "topic" qos = 0 connection_timeout = "30s" max_undelivered_messages = 1000 persistent_session = false data_format = "value" data_type = "i…

[whchung]

@phill I appreciate the pointer. I looked at the post you referred to. It’s using two different MQTT brokers, so I can understand why using two different mqtt_consumer plugins makes sense.

The post also uses mqtt_consumer.topic_parsing to generate tags, not the json_v2 parser. It dumps the entire payload as keys using path = "@this". I was hoping to use the json_v2 parser to generate both tags and keys from the different JSON objects in my payload by referring to the different objects using different paths. You gave an example that shows this works with the http plugin . Did you try this with the mqtt_consumer plugin? Given the other post you referred to, I’m wondering if the json_v2 parser might behave differently with mqtt_consumer vs http.

Thank you!

[phill]

@whchung I use mqtt to relay sensor data to Home Assistant but I haven’t used the [[inputs.mqtt_consumer]] plugin. So there’s the JSON payload inside an mqtt wrapper (the topic) and I assume the topic_parsing is required to unwrap and get to the payload.

I think you could remove the tags = "_/_/_/network/_/device/_" statement from the [[inputs.mqtt_consumer.topic_parsing]] section and put the tags you need in the JSON parser. If this doesn’t work, there’s a tag rename (see below).

@whchung

It dumps the entire payload as keys using path = "@this"

Yes, path = "@this" selects the entire payload for that example, though you can change this to pick off specific name-value pairs, if desired (see below).

I did not include all the details of my http JSON parser earlier, but renaming tags is possible. Note in this example I have an array of JSON objects and use pattern matching to drill down into the array. Then the @this refers to the JSON fragment that remains after the drill down.

    [[inputs.http.json_v2.object]]
      path = "devices.#(SpecificDeviceMatching)#.@this"
      included_keys = ["MyKey"]
      tags = ["MyTag"]
      [inputs.http.json_v2.object.renames]
        MyTag = "MyNewTag"
        MyKey = "MyNewKey"
 

[whchung]

I gave up trying to define multiple [[inputs.mqtt_consumer.json_v2.object]] sections. It really does look like only one will work.

To define fields, I just wrote out multiple `[[inputs.mqtt_consumer.json_v2.field]] sections to define each of the fields I wanted. This was not as elegant as being to specify a path in an object definition, but it seems to work.

Thank you for your help @phill!

[phill]

@whchung Sometimes brute force wins over elegance. Glad to hear it’s working. Good luck with your project!