Is it possible to mass rename measurements with Telegraf?

telegraf

#1

Hello !

Is it possible to mass rename measurements with any of the processors plugins in Telegraf ?

My goal is to make Prometheus to match a specific naming convention format that the target requires, where naming rules of Prometheus will do not allow a dot in the metric name.

For example, I wish to rename:

kafka_controller_*

by:

kafka_controller.*

The following example with the rename processor allows me to rename on a per metric basis, but there are hundreds of them, and explicitly renaming them would not be realistic:

[[processors.rename]]

  [[processors.rename.replace]]
    measurement = "kafka_controller_ActiveControllerCount_Value"
    dest = "kafka_controller.ActiveControllerCount"

I did some experiment with the regex processor without success.

Thank you for your help !

Guilhem


#2

Yes it is

  • name_override : Override the base name of the measurement. (Default is the name of the input).

So if you have your input
[[inputs.cpu]]
name_override = “processor_metrics”


#3

Hi @Esity

Thanks for your reply, yes I get your point, however this does not work in the case of Prometheus inputs.

The name of the metrics are composed by several parts of the JMX beans depending on the configuration the Prometheus exporter, data looks like this for example:

kafka_controller_UncleanLeaderElectionEnableRateAndTimeMs_Mean;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962
kafka_controller_ControlledShutdownRateAndTimeMs_Min;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962
kafka_controller_AutoLeaderBalanceRateAndTimeMs_StdDev;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962
kafka_controller_LeaderAndIsrResponseReceivedRateAndTimeMs_75thPercentile;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962
kafka_controller_EventQueueTimeMs_50thPercentile;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 6 1544977962
kafka_controller_OfflinePartitionsCount_Value;host=ip-10-0-0-26;url=http:--ip-10-0-0-26:18080-metrics 0 1544977962

Ideally a rename processor with regex features would do the work, but a name override within the Prometheus inputs would only work if these were static and simple.

Guilhem


#4

My apologies, I missed the prometheus part which is clearly stated in the question.

How are you collecting the data? A Telegraf input or inside of Prometheus sending it out to InfluxDB?


#5

No problem at all :wink:

Yes, Data is being collected via Telegraf Prometheus inputs.


#6

Okay yeah makes sense. I unfortunately don’t have an answer(other then the rewrite rule for each measurement name)

Maybe @daniel might know if this is possible? He always has answers to my Telegraf questions


#7

Is it always 3 parts: metric name (may contain underscore), field name (camelcase), metric type (camelcase)? If so, I think the regex processor could work if we added support for modifying the measurement. Could you open a github issue for this feature?

Perhaps it can be modified in the exporter configuration? Is it possible to modify name: kafka_server_$1_$2 to be name: kafka_server.$1_$2?

Finally, you could investigate switching from the exporter you are using for JMX to the jolokia2 plugin, you may be able to get better results this way.


#8

Thank you very much @Esity !


#9

Hi !

Sure thing, I will be happy to open a Git issue, I think in the absolute this is a use case that is interesting, explicit renames are great things but something with regex capabilities would be much more powerful and fill various potential requirements for users.

To reply to your question, no this is not necessary 3 segments that define the metric name as this relies on the naming convention used by the Java classes, which could probably be anything.

However, if the extended regex supports capturing groups, then one could have different rules to match his use case, and used the captured group to form the metric.

Regarding Jolokia2 plugin I like it very much and this is what I primary use and recommend to implement my apps, however my goal is the have both Jolokia and Prometheus JMX exporter be compatibles with my implementation, specially for people running Kafka deployment in Kubernetes.

In Kubernetes world, most Kafka deployments promote the usage of Prometheus exporter running as a side car within the pod, so it makes sense to comply with both options.

Thank you for your reply


#10

Issue opened:

Thanks


#11

Thanks for opening the issue. I would like to get have this working well with the Prometheus JMX exporter as well, so please don’t take this as any indication that I don’t want to support this method, but you might also consider running Telegraf as a sidecar for the pod. We do this internally and have had good success with this method: https://www.influxdata.com/blog/monitoring-kubernetes-architecture/

The regex processor does support capture groups, so I think adding feature will be enough. My only concern is if you will always be able to segment the input measurement into it’s parts using a regular expression, it may be hard to determine where to split the components.