Using opentelemetry histogram data written to influx

InfluxDB 2
,
1

Hi,

I have the following setup:

an application using opentelemetry to record histogram values (request duration)

this gets exported to a telegraf agent running the opentelemetry input service with prometheus_v2 configuration since this seems to be the format opentelemetry uses. The data I get look like the following in influx.

{A657F0DE-91F5-49CC-B170-CC8B899CF78D}
{A657F0DE-91F5-49CC-B170-CC8B899CF78D}468×347 21.4 KB

But I can’t wrap my head around how to work with the data in this format later in flux.

What I want is for example to plot the requests / s and the 99th percentile for the request duration.

I tried multiple ways to work with it but the graph was never really meaningful. Also I did not find much documentation regarding this.

I already tried the guide Work with Prometheus histograms | Flux Documentation but it does not really show how the data in influx should be structured.

Having a clear example for this would be quite helpful since otherwise it would render using influx for our application opentelemetry data useless.

2

When using the following query:

import "experimental/prometheus"

from(bucket: "mybucket")

  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)

  |> filter(fn: (r) => r["_measurement"] == "prometheus" and r["_field"] == "request.duration_bucket" and r["response.type"] == v.response_type)

  |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false)

  |> prometheus.histogramQuantile(quantile: 0.99)

  |> yield(name: "quantile")

I get the following:

{6C5418CE-CD16-4EF6-88EB-699757363FCB}
{6C5418CE-CD16-4EF6-88EB-699757363FCB}506×533 21.4 KB

3

When using the histogram max (blue line) with the following query:

from(bucket: "mybucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "prometheus" and (r["_field"] == "request.duration_min" or r["_field"] == "request.duration_max") and r["response.type"] == v.response_type)
  |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false)
  |> yield(name: "min max")

I see that the data aligns but the 99th percentile value is higher than the max which can’t be.

{910C0B0F-C197-4B8F-8703-F3DE4B9912C7}
{910C0B0F-C197-4B8F-8703-F3DE4B9912C7}451×521 28.1 KB

4

I found a possible explanation for the 99th percentile being higher than the max. If the Prometheus function in influx is implemented the same way as in Prometheus this would also apply for influx. promql: P99 created using histogram_quantile shows value greater than the maximum possible value in the series · Issue #12740 · prometheus/prometheus · GitHub