Best practices for storing high-velocity data

InfluxDB 2
1

At work we’re working on migrating some logging tasks to InfluxDB 2.0. We have multiple machines logging many values.
Per machine we log
~80 fields at a rate of 500Hz or 1000Hz
~50 fields at a rate of 50Hz
~50 fields at a rate of 0.2Hz
We need the high granularity on those 80 fields, so downsampling is not an option.

The ‘schema’ I’m currently testing with has all data in one single bucket, using the machine name as the measurement and 4 different tags in each record. One of the tags being a ‘job id’, of which each machine generates a few per day.

Pushing data into the database seems to be fine and doesn’t require too many resources, but querying the database seems to sometimes be problematic from a performance/memory usage stand point. Some queries cause high memory usage, causing the occasional OOM killer to spring into action. I think the high cardinality might be a cause of this high memory usage, but I’m not sure what the best way is to avoid it.

What would be a more optimal schema to store this kind of high velocity data in?
Would it be advisable to maybe use that ‘job id’ tag as the measurement instead? Or maybe create one bucket per machine or even per job?

Any help would be greatly appreciated!

2

Hello @twim,
tags, measurements, and buckets are indexed. Fields are not.
What types of queries cause high memory usage?
What types of queries do you want to be able to perform?
In the meantime I can recommend this resource:

Also please be aware of pushdown patterns in Flux for optimizing queries. Outlined in the blog below:

1 Like
3

Thanks so much for the resources!

A typical example or a query has proven to be problematic would be:

from(bucket: "logger")
|> range(start: 2020-10-23T16:55:28.3905490Z, stop: 2020-10-23T16:55:38.3905490Z)
|> filter(fn: (r) => r.JobID == "#14011")
|> limit(n: 1, offset: 0)

Which is a ‘trick’ I use to determine which fields are present in a ‘JobID’ (JobID is a tag here), since we can guarantee that all used fields will be present in any 10 second interval. A query like this can take anywhere between a few seconds to tens of minutes, and sometimes take the server down due to out-of-memory issues.

Other queries that tend to take long are ones in which we use drop columns (Like drop(columns: [“SomeTag”]) or group(). These will sometimes cause near 10x slow down in the query time.