Is there a way to prevent users from consuming too much memory in an “aggregateWindow” operation in InfluxDB 2.1?
For example, with this config.toml on an instance with 16GB of RAM:
# Maximum total bytes of memory allowed for queries.
query-max-memory-bytes = 8590000009
# Number of queries allowed to execute concurrently.
query-concurrency = 1024
# Initial bytes of memory allocated for a query.
query-initial-memory-bytes = 8388671 # 8 MiB
# Maximum bytes of memory allowed for a single query.
query-memory-bytes = 8388671 # 8 MiB
# Maximum number of queries allowed in execution queue. When queue limit is reached, new queries are rejected.
query-queue-size = 1024
And this query looking at the last >= 7 days:
from(bucket: "metrics")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "cloudwatch_aws_lambda")
|> filter(fn: (r) => r["some_item"] == "my_data")
|> filter(fn: (r) => r["_field"] == "some_field_a" or r["_field"] == "some_field_b")
|> filter(fn: (r) => r["function_name"] == "my-lambda-function")
|> filter(fn: (r) => (exists r["resource"]) == false)
|> aggregateWindow(every: 1ms, fn: mean)
|> yield(name: "mean")
The 604800000 buckets/windows that would generate on 7 days is obviously unreasonable for the data store to handle, but how can I limit queries or memory usage to avoid a single bad query crashing the cluster?
