Parse JSON array composed of dissimilar objects

Telegraf
1

@sspaink or other parsing expert . . .

My JSON array consists of about 50 objects each having a different set of between 10-30 name-value pairs. I want to extract specific metrics across the whole array. These metrics are only found in some of the array elements - those that have one (or more) common key-values. The approach so far is to use the pattern matching query in [[inputs.file.json_v2.object]] (i.e., path = “target pattern”) to get only those elements that contain the measurement. Note: the pattern to match on is not the same as the measurement because that measurement might exist in other array elements that should not be included in this metric. E.G., There are blue and red socks, so I match on the color and then extract the count of each.

I can extract the raw measurement using ‘included_keys = [ ]’ or ‘tags = [ ]’, but having tried several variations of subtables, I can’t yet rename tags or change the type of values. And I need to do both.

If the pattern matching approach can do all this, then I welcome any help that can get me pointed towards a solution. However I’ve also been looking into the ‘optional=true’ examples which parse the whole array and suppress errors if the configured path doesn’t match a particular array element. I’d prefer the first approach because I know I’m getting only those array elements that have the measurement of interest.

2

It’s working and here’s what I learned . . .

  • A JSON array of objects with dissimilar name-value pairs is irrelevant
  • The pattern matching query is a powerful method for getting at specific array elements
  • It is very important to specify the path = " " correctly. This was the root cause of my problem.
  • The proper syntax is path = "<pattern matching query which returns an array.#.@this
  • Failure to get the path right will cause either no output, repeated output or even a horrible “influx could not serialize field” error

I studied

These are all great resources for someone who couldn’t spell Telegraf only 4 days ago! In hindsight an additional example or two of a complex query on a complex JSON array would be helpful. Perhaps something like this . . . (Note the escaped inner quotes on path! Thanks @sspaink).


    [[inputs.file.json_v2.object]]
      path = "root.#(JSON_NAME==\"JSON Value\")#.@this"
      included_keys = ["My_Key"]
      tags = ["My_Tag"]
      [inputs.file.json_v2.object.renames]
        My_Tag = "New Tag Name"
        My_Key = "New Key Name"
      [inputs.file.json_v2.object.fields]
        My_Key = "float"
1 Like
3

@phill
Thanks so much for sharing your solution! We really appreciate it.

Out of curiosity what are you trying to build with InfluxDB? I’d love to hear more about your project.
Thanks!