Is it possible to mass rename measurements with Telegraf?

guilhem.marchand · December 16, 2018, 12:53pm

Hello !

Is it possible to mass rename measurements with any of the processors plugins in Telegraf ?

My goal is to make Prometheus to match a specific naming convention format that the target requires, where naming rules of Prometheus will do not allow a dot in the metric name.

For example, I wish to rename:

kafka_controller_*

by:

kafka_controller.*

The following example with the rename processor allows me to rename on a per metric basis, but there are hundreds of them, and explicitly renaming them would not be realistic:

[[processors.rename]]

  [[processors.rename.replace]]
    measurement = "kafka_controller_ActiveControllerCount_Value"
    dest = "kafka_controller.ActiveControllerCount"

I did some experiment with the regex processor without success.

Thank you for your help !

Guilhem

Esity · December 16, 2018, 4:06pm

Yes it is

github.com

influxdata/telegraf/blob/master/docs/CONFIGURATION.md

<!-- markdownlint-disable MD024 -->

# Configuration

Telegraf's configuration file is written using [TOML][] and is composed of
three sections: [global tags][], [agent][] settings, and [plugins][].

View the default [telegraf.conf][] config file with all available plugins.

## Generating a Configuration File

A default config file can be generated by telegraf:

```sh
telegraf config > telegraf.conf
```

To generate a file with specific inputs and outputs, you can use the
--input-filter and --output-filter flags:

This file has been truncated. show original

name_override : Override the base name of the measurement. (Default is the name of the input).

So if you have your input
[[inputs.cpu]]
name_override = “processor_metrics”

guilhem.marchand · December 16, 2018, 4:42pm

Hi @Esity

Thanks for your reply, yes I get your point, however this does not work in the case of Prometheus inputs.

The name of the metrics are composed by several parts of the JMX beans depending on the configuration the Prometheus exporter, data looks like this for example:

kafka_controller_UncleanLeaderElectionEnableRateAndTimeMs_Mean;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962
kafka_controller_ControlledShutdownRateAndTimeMs_Min;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962
kafka_controller_AutoLeaderBalanceRateAndTimeMs_StdDev;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962
kafka_controller_LeaderAndIsrResponseReceivedRateAndTimeMs_75thPercentile;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962
kafka_controller_EventQueueTimeMs_50thPercentile;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 6 1544977962
kafka_controller_OfflinePartitionsCount_Value;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962

Ideally a rename processor with regex features would do the work, but a name override within the Prometheus inputs would only work if these were static and simple.

github.com

prometheus/jmx_exporter/blob/main/example_configs/kafka-2_0_0.yml

lowercaseOutputName: true

rules:
# Special cases and very specific rules
- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    topic: "$4"
    partition: "$5"
- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
  name: kafka_server_$1_$2
  type: GAUGE
  labels:
    clientId: "$3"
    broker: "$4:$5"
- pattern : kafka.coordinator.(\w+)<type=(.+), name=(.+)><>Value
  name: kafka_coordinator_$1_$2_$3
  type: GAUGE

This file has been truncated. show original

Guilhem

Esity · December 16, 2018, 8:33pm

My apologies, I missed the prometheus part which is clearly stated in the question.

How are you collecting the data? A Telegraf input or inside of Prometheus sending it out to InfluxDB?

guilhem.marchand · December 16, 2018, 9:24pm

No problem at all

Yes, Data is being collected via Telegraf Prometheus inputs.

Esity · December 18, 2018, 3:59pm

Okay yeah makes sense. I unfortunately don’t have an answer(other then the rewrite rule for each measurement name)

Maybe @daniel might know if this is possible? He always has answers to my Telegraf questions

daniel · December 18, 2018, 8:03pm

Is it always 3 parts: metric name (may contain underscore), field name (camelcase), metric type (camelcase)? If so, I think the regex processor could work if we added support for modifying the measurement. Could you open a github issue for this feature?

Perhaps it can be modified in the exporter configuration? Is it possible to modify name: kafka_server_$1_$2 to be name: kafka_server.$1_$2?

Finally, you could investigate switching from the exporter you are using for JMX to the jolokia2 plugin, you may be able to get better results this way.

guilhem.marchand · December 18, 2018, 9:06pm

Thank you very much @Esity !

guilhem.marchand · December 18, 2018, 9:17pm

Hi !

Sure thing, I will be happy to open a Git issue, I think in the absolute this is a use case that is interesting, explicit renames are great things but something with regex capabilities would be much more powerful and fill various potential requirements for users.

To reply to your question, no this is not necessary 3 segments that define the metric name as this relies on the naming convention used by the Java classes, which could probably be anything.

However, if the extended regex supports capturing groups, then one could have different rules to match his use case, and used the captured group to form the metric.

Regarding Jolokia2 plugin I like it very much and this is what I primary use and recommend to implement my apps, however my goal is the have both Jolokia and Prometheus JMX exporter be compatibles with my implementation, specially for people running Kafka deployment in Kubernetes.

In Kubernetes world, most Kafka deployments promote the usage of Prometheus exporter running as a side car within the pod, so it makes sense to comply with both options.

Thank you for your reply

guilhem.marchand · December 18, 2018, 9:34pm

Issue opened:

github.com/influxdata/telegraf

Extend the regex processor to allow renaming measurements based in regular expression and capturing groups extractions

opened 09:31PM - 18 Dec 18 UTC

closed 04:45PM - 04 Nov 21 UTC

guilhemmarchand

feature request

## Feature Request Allow mass renaming measurements using the regex processor… and one or more rules. ### Proposal: Some collectors may have specific metric rules and naming conventions, where the targets (which could be multiple) might require a different naming convention. Currently, on the rename processor allows renaming measurements explicitly on a per measurement basis, however with very large applications such as Java based application like Kafka, there are hundreds of potential metrics and a long explicit renaming would be not realistic. Proposal would be having the regex processor being able to rename measurements with capturing groups capabilities, such that a user could mass rename the metric generated by the source collector depending in its needs with one or more rules. ### Current behavior: Only the rename processor can statically rename a measurement, which is very limited. ### Desired behavior: Extend the regex processor with measurement renaming capabilities ### Use case: Providing advanced management of metric naming convention to allow an easier interfaces between foreign collectors and targets that do not rely on the same naming convention, or expect one that cannot be easily achieved due the huge number of metrics. A use case example is the ingestion of Prometheus JMX exporter metrics where Prometheus forbids the use of "." in the metric name. Some target system may require a different naming convention to categorise the metrics such as a ".". **For reference please read:** [](https://community.influxdata.com/t/is-it-possible-to-mass-rename-measurements-with-telegraf/7855 ) Currently the capabilities are limited to explicit renames using the rename processor, although this works well for specific use cases, this is slightly limited and could advantageously be extended by regex capabilities. Thank you

Thanks

daniel · December 18, 2018, 10:09pm

Thanks for opening the issue. I would like to get have this working well with the Prometheus JMX exporter as well, so please don’t take this as any indication that I don’t want to support this method, but you might also consider running Telegraf as a sidecar for the pod. We do this internally and have had good success with this method: Monitoring Kubernetes Architecture | InfluxData

The regex processor does support capture groups, so I think adding feature will be enough. My only concern is if you will always be able to segment the input measurement into it’s parts using a regular expression, it may be hard to determine where to split the components.

Topic		Replies	Views
Processor rename plugin not working as expected Telegraf windows	1	429	November 10, 2022
Telegraf as external scrapper Telegraf telegraf , prometheus	1	471	June 1, 2022
Rename Network Interface Telegraf telegraf	3	1136	May 27, 2020
Inputs.prometheus tags assign Telegraf	4	294	October 13, 2023
Own Telegraf Plugin -- Need to scrape metrics from Prometheus clients Telegraf telegraf , prometheus , client-libraries	2	1489	December 3, 2019

Is it possible to mass rename measurements with Telegraf?

Related Topics