Need help to create an ever stacking histogram in InfluxDB with Kapacitor

Store
,
1

Hi,

I’ve started working on this when I saw that the histogram() function wasn’t implemented in InfluxQL, and before I discovered that it was in Flux.

I continued because we’re not planning to migrate to InfluxDB 1.7 soon (which is the first version where Flux is integrated if I’m correct), and also because I feel like I’m close to making it work.

I’m posting here because I need help to solve a data combination operation :thinking:

My requirements are:

  1. store histograms indefinitely
  2. compute and store histograms for multi-fields measurements
  3. and of course, compute percentiles from histogram data

I haven’t worked yet on point 3, which will be in the alerting part of the project, but for the other two points, I’ve created a set of two TICK scripts.

A “data-quantizer” stream script which:

  • defines the bins’ logarithmic scheme for the histogram, according to an estimation of data values range (the bin scheme extends dynamically if the value range has been underestimated)
  • computes the bin# in which falls each data point
  • and stores the “quantized” data points in a short-lived measurement

A “data-stacker” batch script which regularly:

  • extracts the quantized data points, and computes their distribution by bin#
  • extracts the last accumulated histogram data from the long duration measurement
  • adds the newly computed frequencies to the accumulated ones, bin# by bin#
  • stores the result as a new data point in the long duration measurement

Now I’m stuck on the adding operation described in the third dot above :slightly_frowning_face:

To be continued on next post…

2

I’ve inserted below a sample union of both dynamically quantized data and accumulated data:

time                activity   dyn_count dyn_bin# cumul_bin# cumul_count
----                --------   --------- -------- ---------- -----------
1548953580000000000 Background 1         17
1548953580000000000 Background 2         20
1548953580000000000 Background 1         29
1548953580000000000 Foreground 1         118
1548953580000000000 Foreground 2         125
1548953580000000000 Foreground 1         127
1548953640000000000 Background                    2          1
1548953640000000000 Background                    1          1
1548953640000000000 Background                    13         1
1548953640000000000 Background                    5          3
1548953640000000000 Foreground                    16         1
1548953640000000000 Foreground                    12         1
1548953640000000000 Foreground                    10         1
1548953640000000000 Foreground                    15         1
1548953640000000000 Foreground                    9          2

Notes:

  • dyn_bin# and cumul_bin# are tags, and the *_count are integer fields
  • my data also have a supplementary dimension, which is the “activity” tag
  • I’ve chosen bin# to be integers because such histograms’ goal is purely to allow percentile computing for alerting

The idea here is:

  • to add any two *_count values that intersect each other (same “activity” value AND dyn_bin# == cumul_bin#)
  • to store dyn_bin# and dyn_count respectively in cumul_bin# and cumul_count when there’s no intersection
  • and to leave all other values from accumulated set (cumul_*) untouched

The problem is that the two sets of data points:

  • have different timestamps (.tolerance() methods should be able to address this I think)
  • have unequal numbers of points
  • and have collections of *_bin# tags values uncorrelated between each set

I’ve tried all transformations I can think of, both on the union and the individual batch data before they are joined, but to no avail :cry:

Does anyone have an idea of how to make that combination?

Thanks.

3

If you use Telegraf, you could try the histogram aggregator plugin to see if it suits your needs. If it does not, you could start from its code and code a new plugin that works as you intend. You’ll write the logic in go so it will be easier than figuring out a query that process the data as intended.

4

Thanks @samaust for your suggestion, but I’m afraid I don’t have the Go skills -nor the time- to edit a Telegraf plugin’s code :blush:

One of my requirements is to keep the complete distribution of values ever collected for the whole life of my application (10-15 years), without having to keep so many years of history inside InfluxDB. Hence, I need perpetually stacking bins.

InfluxDB’s 64 bits signed integers allow to increment a number by 1 every second for more than 292 billion years, so it shouldn’t be a problem to stack forever.

From what I understand from the Telegraf plugin, the stacking only occurs in the runtime context of the Telegraf daemon, and gets reset whenever it’s restarted.

I’ve also checked Flux’s histogram() function, and in my understanding, it doesn’t allow to do that either. It builds a static histogram from a static input table.

I need a periodically refreshed histogram to perpetually store the distribution of incoming data!