Removing measurement name from json output

Telegraf
1

Basically I want to do this:

  [[processors.rename]]
  ## Specify one sub-table per rename operation.
  [[processors.rename.replace]]
    measurement = "kafka_consumer"
    dest = ""

So that the json output aren’t nested under the measurement names:

image
image1020×1026 62 KB

I would like to have name and age on the root of the document. Setting the name to “” in a starlark script triggers an error. Is there any way to make this happen?

Thanks!

2

By root, you mean not under _source or _source/kafka_consumer?

3

_source is what Elastic/opensearch received, so that’s basically the root. I would like to shape the document in such a way that tag/host is directly under the root (in this representation under _source) without the measurement_name.

4

You are using the elasticsearch output? Trying to figure out what is serializing the data this way in the first place to see if this is possible.

A processor probably is not going to help you as this looks to be the format the output you are using or whatever serializer you are using uses.

In fact if you are using the elasticsearch output here is where it is building that JSON. The fields are placed under the meaasurement name.

Is there a compelling reason to change this?

5

@ict-one-nl is that example an input you are trying to parse? What are your expected output? Which plugins are involved?

6

Basically I want control over the output to transform it in such a way that it’s usable in a SIEM. So basically it’s now

{
  "kafka_consumer" : " {
  "age" : 30,
  "name" : "John"
  }
}

I want to be able to transform it to

{
  "bla" : 30,
  "blergh" : "John"
}

This would give us the flexibility to transform the documents to fit in our own opensearch cluster (output plugin still missing btw), but with another telegraf instance to transform them to fit the needs of our Security Operations Center partner. So this is less about metrics (our usual use case for Telegraf), but still Telegraf could be very useful (thus dropping logstash that’s treating opensearch as a second class citizen).

7

@ict-one-nl I suggest to take a look at json_transformation of the json serializer telegraf/plugins/serializers/json at master · influxdata/telegraf · GitHub.