How to store OpenTelemetry histogram metrics

InfluxDB 2
1

I’m trying to write an InfluxDB sink for OpenTelemetry metrics for .NET. There are five metric types: integer and float gauges; integer and float sums; and histograms.

For histograms, you get a set of points with an ExplicitBound float and a BucketCount integer.

Now, I know you’re not supposed to write histograms into InfluxDB, but I have two choices:

  1. Say “histograms are not supported by InfluxDB”
  2. Find a way to do it that makes it meaningfully usable in dashboards.

Is there any way to do option 2? Like writing a pair of fields for each point, something like b0_bound=0.25,b0_count=178,b1_bound=0.5,b1_count=98

Would that enable creation of a useful chart in the dashboard or Grafana or whatever using Flux or InfluxQL?

2

Hello @mtr,
Welcome!
Thank you!!
Have you seen the following documentation?

Please give it a quick look through or lmk if you have specific questions about histograms in InfluxDB

You don’t need to write those values to InfluxDB to create histograms because you can use the histogram function and the histogram visualization type. So I wouldn’t say that InfluxDB doesn’t support histograms, rather the opposite.

3

Hi @Anaisdg

Thanks for replying. I know about the “dynamic” histograms you can do with InfluxDB, and they’re awesome. It’s more about OpenTelemetry Metrics having this Histogram metric type and I’m wondering what to do with it. It might just be “don’t use the Histogram metric type when you’re writing to InfluxDB” I guess, although OpenTelemetry doesn’t really output the kind of metrics that InfluxDB wants.

What I mean by that is, you’ve got Sum and Gauge, i.e. counters and values that go up and down over time, and that’s cool. But there’s no Timer metric for “just write the request duration for every request” which InfluxDB would then use to build a histogram at query time. Seems like the “standard” way to record timings is to pre-histogram them, which obviously cuts down on data storage too.

So if there were a way to store and query and visualize these pre-made histograms in InfluxDB, that might be useful. And having an InfluxDB sink for OTel Metrics sounds like a win.

Thanks,
Mark

4

Hmm @mtr,
You can definitely write it to InfluxDB.

I’m not really sure I understand what you mean, but imma try and give it a go.
You could maybe use stateDuration() to calculate the request duration?

Here’s my failed attempt at getting that data into the shape expected by the UI for histograms:

import "join"
import "experimental"
import "strings"
import "array"
import "regexp"
//b0_bound=0.25,b0_count=178,b1_bound=0.5,b1_count=98
// if your data above was written as fields your output would look like this 
// after you filter for all of those fields. 
data = array.from(rows: [
{"_time": 2022-06-17T17:52:18.681Z, "_field": "b0_bound", "_value": 0.25},
{"_time": 2022-06-17T17:52:18.681Z, "_field": "b0_count", "_value": 178.0},
{"_time": 2022-06-17T17:52:18.681Z, "_field": "b1_bound", "_value": 0.5},
{"_time": 2022-06-17T17:52:18.681Z, "_field": "b1_count", "_value": 98.0},
{"_time": 2022-06-18T17:52:18.681Z, "_field": "b0_bound", "_value": 0.25},
{"_time": 2022-06-18T17:52:18.681Z, "_field": "b0_count", "_value": 179.0},
{"_time": 2022-06-18T17:52:18.681Z, "_field": "b1_bound", "_value": 0.5},
{"_time": 2022-06-18T17:52:18.681Z, "_field": "b1_count", "_value": 99.0},
])
|> group(columns: ["_field", "_time"], mode:"by")

sum = data
|> filter( fn: (r) => r._field =~ regexp.compile(v: "count")) 
|> group(columns: ["_field"], mode:"by")
|> map(fn: (r) => ({ r with index: 1.0 }))
|> cumulativeSum(columns: ["index"])
|> drop(columns: ["_time"])
|> yield(name: "sum")


bounds = data
|> filter( fn: (r) => r._field =~ regexp.compile(v: "bound")) 
|> set(key: "_field", value: "le" )
|> map(fn: (r) => ({ r with index: 1.0 }))
|> cumulativeSum(columns: ["index"])
|> yield(name: "bounds")

join(
    tables: {bounds: bounds, sum: sum},
    on: ["index"],
    method: "inner",
)
|> yield(name: "join")