Inputs.http_listener_v2.json_v2 parser help

Telegraf
1

I was wondering if i could get help with my telegraf config.
I’ve tried this a bunch of different ways from the json_v2 testdata, but cant seem to figure out how to get the information cleanly into influxdb.
here is an example json i am trying to send to telegraf:

{
    "tenant": [
        {
            "Cu1": [
                {
                    "Cu1-Site1": {
                        "device-type": "vedge",
                        "interfaces": [
                            {
                                "GigabitEthernet1": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "GigabitEthernet2": {
                                    "rx_kbps": 17,
                                    "tx_kbps": 52
                                }
                            },
                            {
                                "GigabitEthernet3": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "GigabitEthernet4": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Loopback65528": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "NVI0": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Sdwan-system-intf": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Tunnel1": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Tunnel2": {
                                    "rx_kbps": 1,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "vmanage_system": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            }
                        ],
                        "reachability": "reachable"
                    }
                },
                {
                    "Cu1-Site2": {
                        "device-type": "vedge",
                        "interfaces": [
                            {
                                "GigabitEthernet1": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "GigabitEthernet2": {
                                    "rx_kbps": 15,
                                    "tx_kbps": 52
                                }
                            },
                            {
                                "GigabitEthernet3": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "GigabitEthernet4": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Loopback65528": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "NVI0": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Sdwan-system-intf": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Tunnel1": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Tunnel2": {
                                    "rx_kbps": 1,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "vmanage_system": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            }
                        ],
                        "reachability": "reachable"
                    }
                },
                {
                    "cu1-hub-internet": {
                        "device-type": "vedge",
                        "interfaces": [],
                        "reachability": "unreachable"
                    }
                }
            ]
        },
        {
            "cu2": [
                {
                    "Cu2-Site1": {
                        "device-type": "vedge",
                        "interfaces": [
                            {
                                "GigabitEthernet1": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "GigabitEthernet2": {
                                    "rx_kbps": 13,
                                    "tx_kbps": 30
                                }
                            },
                            {
                                "GigabitEthernet3": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "GigabitEthernet4": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Loopback65528": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "NVI0": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Sdwan-system-intf": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Tunnel1": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Tunnel2": {
                                    "rx_kbps": 1,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "vmanage_system": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            }
                        ],
                        "reachability": "reachable"
                    }
                },
                {
                    "Cu2-Site2": {
                        "device-type": "vedge",
                        "interfaces": [
                            {
                                "GigabitEthernet1": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "GigabitEthernet2": {
                                    "rx_kbps": 14,
                                    "tx_kbps": 52
                                }
                            },
                            {
                                "GigabitEthernet3": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "GigabitEthernet4": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Loopback65528": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "NVI0": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Sdwan-system-intf": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Tunnel1": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "Tunnel2": {
                                    "rx_kbps": 1,
                                    "tx_kbps": 0
                                }
                            },
                            {
                                "vmanage_system": {
                                    "rx_kbps": 0,
                                    "tx_kbps": 0
                                }
                            }
                        ],
                        "reachability": "reachable"
                    }
                }
            ]
        }
    ],
    "vManage Response Time": 57.96632242202759
}

and here is my current telegraf config:

   data_format = "json_v2"
        [[inputs.http_listener_v2.json_v2]]
          [[inputs.http_listener_v2.json_v2.object]]
            disable_prepend_keys = true
            path = "@this"
            included_keys = [
                "tenant",
                "interfaces"
            ]
            tags = ["interfaces"]

this bring in all the information i need, but not in a scalable format.
with influxdb, everything is listed by interface rx then interface tx, with no association to what device.
what im looking for with this is a way to parse the json with json_v2 so in influxdb i can sort by the following:

tenant -> device -> interfaces -> rx/tx stats

for example, from the above json:

tenant -> Cu1 -> Cu1-Site1 -> interfaces (list all interfaces) -> view the rx/tx stats

ive tried this a few different ways, but it always shows up like a mess in influx.
trying to learn how to filter it with https://gjson.dev/ has helped, but im not sure how to account for the possible variables. i.e. if i dont know what the tenant name, device name, or what interfaces it has
I am a few days into this, so i would really appreciate the help.
if there is an easier way to do this, let me know. or if i should format the json differently to make the filtering easy let me know. im flexible either way
Thanks

2

I don’t know if you can configure the json_v2 parser to achieve the desired result.
In my opinion, your json data is poorly structured. Can this be changed?
It is valid json, but it does not use a key-value structure.

EDIT:
One dirty solution would be to use disable_prepend_keys = false and write a custom processors.starlark plugin to pick apart the weird influx key names.

3

Here is a working example of the “dirty” solution:

[[inputs.file]]
  files = ["tenant.json"]
  data_format = "json_v2"
  [[inputs.file.json_v2]]
    measurement_name = "tenant"
    [[inputs.file.json_v2.object]]
      disable_prepend_keys = false
      path = "tenant"

[[processors.starlark]]
  namepass = ["tenant"]
  source = '''
def split_to_tags(key):
  fragments = key.split("_")
  tags = {}
  for i in range(len(fragments)):
    if i == 0:
      tags["tenant"] = fragments[i]
    elif i == 1:
      tags["site"] = fragments[i]
    elif i == 3:
      tags["interface"] = fragments[i]
    else:
      continue  # skip all other fragments
  return tags

def apply(metric):
  new_metric = Metric(metric.name)  # Create a new metric
  new_metric.time = metric.time  # copy time
  for key, value in metric.fields.items():
    if "rx_kbps" in key:
      new_metric.fields["rx_kbps"] = int(value)
      new_metric.tags.update(split_to_tags(key))
    elif "tx_kbps" in key:
      new_metric.fields["tx_kbps"] = int(value)
      new_metric.tags.update(split_to_tags(key))
    elif "device-type" in key:
      new_metric.tags["type"] = str(value)
      new_metric.tags.update(split_to_tags(key))
    elif "reachability" in key:
      new_metric.fields["reachability"] = str(value)
      new_metric.tags.update(split_to_tags(key))
    else:
      new_metric.fields[key] = value
  return new_metric
'''

[[outputs.file]]  # file output only for debugging
  files = ["tenant.out"]
  influx_sort_fields = true
4

It absolutely can be changed.
this is an aggregate of about 5 other API calls. i just used a python app to build them into one json to send to telegraf.
I had a feeling that might be my main issue.
ill look up the best practices and see what im missing here.

5

oh ok. i see what going on.
ya. ill see if i can clean up the json better so i dont need this.
since it will be going into production, i need it as clean as possible for people who might fall in after me.

6

If you already have a Python script running, I would save myself the diversions via json and send the metrics directly in the influx line protocol format to Telegraf (with point.to_line_protocol() method) or even directly to the InfluxDBv2 itself (with the write_api.write() method).

There is already a very good client library for this:

GitHub - influxdata/influxdb-client-python: InfluxDB 2.0 python client

InfluxDB 2.0 python client — influxdb_client 1.39.0 documentation