Is someone can share hint to generate line protocol from JSON file

Telegraf
1

Hi,

I tried to use the input file plugin with json_v2 parser but I did not find exactly how to configure it.

My json input looks like:

[
{
“benchmark” : “B1”,
“primaryMetric” : {
“score” : 109.57272003762402
}
},
{
“benchmark” : “B2”,
“primaryMetric” : {
“score” : 103.2295949532184
}
}
]

And I want to generate something like below:

micro-benchmark benchmark=“B1” score=109.57272003762402 1668873524000000000
micro-benchmark benchmark=“B2” score=103.2295949532184 1668873524000000000

The following configuration allow to generate something near to what I want but without tag benchmark, difficulty comes when I want to add the tag benchmark.

[[inputs.file.json_v2]]
measurement_name = “micro-benchmark”
timestamp_format = “unix_ns”
timestamp_timezone = “Europe/Paris”
[[inputs.file.json_v2.field]]
path = “@this.#.primaryMetric.score”

I also tried with json_v2.object but without getting the expected result.
If someone have some hints, thank to share them

2

For me the following config

[[inputs.file]]
  files = ["..."]
  data_format = "json_v2"
  [[inputs.file.json_v2]]
    measurement_name = "micro-benchmark"
    [[inputs.file.json_v2.object]]
      path = "@this"
      disable_prepend_keys = true

creates

> micro-benchmark benchmark="B1",score=109.57272003762402 1669117064000000000
> micro-benchmark benchmark="B2",score=103.2295949532184 1669117064000000000
1 Like
3

Thanks for your help.

My real input is a JMH Json result file
benchmark.txt (3.7 KB)

Currently I used the following configuration:

[[inputs.file]]
files = [“./benchmark.json”]
data_format = “json_v2”
[[inputs.file.json_v2]]
measurement_name = “micro-benchmark”
timestamp_format = “unix_ns”
timestamp_timezone = “Europe/Paris”
[[inputs.file.json_v2.object]]
path = “@this”
disable_prepend_keys = true
tags = [“benchmark”, “jmhVersion”, “mode”, “jdkVersion”, “vmName”, “vmVersion”, “primaryMetric_scoreUnit”]
included_keys = [“primaryMetric_score”, “primaryMetric_scoreError”]

It works as expected and produces:

micro-benchmark,benchmark=com.mp.jmh.benchmarks.StringBufferVsStringBuilderBench.stringBuffer,jdkVersion=1.8.0_345,jmhVersion=1.35,mode=avgt,scoreUnit=ns/op,vmName=OpenJDK\ 64-Bit\ Server\ VM,vmVersion=25.345-b01 score=109.57272003762402,scoreError=8.593461468517338 1669477302000000000
micro-benchmark,benchmark=com.mp.jmh.benchmarks.StringBufferVsStringBuilderBench.stringBuilder,jdkVersion=1.8.0_345,jmhVersion=1.35,mode=avgt,scoreUnit=ns/op,vmName=OpenJDK\ 64-Bit\ Server\ VM,vmVersion=25.345-b01 score=103.2295949532184,scoreError=5.352573860930696 1669477302000000000

Now, what I want is to add fields such as percentile_0_0=XXX, percentile_50_0=YYY, percentile_90_0=ZZZ and so on that could be generated from nested data named scorePercentiles, but I did not find a way to do it.

Have you an idea how to do it ?

Thanks

4

Ok, I did some progress, the following configuration allow me to have line protocol that seems good, except that I want to rename some fields.

  [[inputs.file.json_v2.object]]
    path = "@this"
    disable_prepend_keys = true
    tags = ["benchmark", "jmhVersion", "mode", "jdkVersion", "vmName", "vmVersion", "primaryMetric_scoreUnit"]
    included_keys = ["primaryMetric_score", 
      "primaryMetric_scoreError", 
      "primaryMetric_scorePercentiles_0.0",
      "primaryMetric_scorePercentiles_50.0",
      "primaryMetric_scorePercentiles_90.0",        
      "primaryMetric_scorePercentiles_95.0",
      "primaryMetric_scorePercentiles_99.0",
      "primaryMetric_scorePercentiles_99.9",
      "primaryMetric_scorePercentiles_99.99",
      "primaryMetric_scorePercentiles_99.999",
      "primaryMetric_scorePercentiles_99.9999",
      "primaryMetric_scorePercentiles_100.0",
      ]

One of the line protocol generated is

micro-benchmark,benchmark=com.mp.jmh.benchmarks.StringBufferVsStringBuilderBench.stringBuffer,jdkVersion=1.8.0_345,jmhVersion=1.35,mode=avgt,scoreUnit=ns/op,vmName=OpenJDK\ 64-Bit\ Server\ VM,vmVersion=25.345-b01 0.0=109.18695291585028,100.0=110.09766277034856,50.0=109.43354442667322,90.0=110.09766277034856,95.0=110.09766277034856,99.0=110.09766277034856,99.9=110.09766277034856,99.99=110.09766277034856,99.999=110.09766277034856,99.9999=110.09766277034856,score=109.57272003762402,scoreError=8.593461468517338 1669569429000000000

Is there a way to rename fields in bold into something such as percentile_0_0, percentile_50_0 and so on directly into the parser or is it possible to do it by using the rename processor ?

5

Hey @mike_35, to rename the fields you probably should use the regexp processor with a [[processors.regex.field_rename]] subsection. Like (untested)

[[processors.regex]]
  namepass = ["micro-benchmark"]

  # Rename metric fields
  [[processors.regex.field_rename]]
    ## Regular expression to match on a field name
    pattern = "^([0-9]+,([0-9]+)$"
    replacement = "percentile_${1}_${2}"

You might need to drop the original metric fields, not sure…

1 Like