Best practices for storing high-velocity data

Thanks so much for the resources!

A typical example or a query has proven to be problematic would be:

from(bucket: "logger")
|> range(start: 2020-10-23T16:55:28.3905490Z, stop: 2020-10-23T16:55:38.3905490Z)
|> filter(fn: (r) => r.JobID == "#14011")
|> limit(n: 1, offset: 0)

Which is a ‘trick’ I use to determine which fields are present in a ‘JobID’ (JobID is a tag here), since we can guarantee that all used fields will be present in any 10 second interval. A query like this can take anywhere between a few seconds to tens of minutes, and sometimes take the server down due to out-of-memory issues.

Other queries that tend to take long are ones in which we use drop columns (Like drop(columns: [“SomeTag”]) or group(). These will sometimes cause near 10x slow down in the query time.