we are facing to issue with crashing database during start with logged memory issues. It can be “fatal error: runtime: cannot allocate memory”, “memory allocation of 672 bytes failed” or “out of memory allocating heap arena metadata”.
Action right before fatal error are different - usually it’s Compacting file, but we found also Starting query controller or Configuring InfluxQL statement executor. It seems not connected to specific action.
Influx is in version 2.7., running in official docker container (influxdb:2.7) . Memory available for container was 64 GB, but even increasing to 96GB didn’t help.
We also tried add/modify some parameters e.g. storage-cache-max-memory-size or storage-max-concurent-compactions, but with no effect.
Do you have any idea what we can try to fix the issue and run database?
Hello @Jindrich.strakos,
Welcome!
What is your query and ingest load like?
Did you always get this error or did something change in your throughput?
Hmm I do know that memory issues are a common issue with 2.x. Part of the reason why 3.x was created.
Especially clustered and dedicated which offers a lot more control over partitioning and memory:
Hi - there wasn’t any query, just start of influxdb. And then it crash by itself.
As I wrote, errors were different, but always related to memory and allocation.
Eventually we solved this issue by using a time when container was starting and we could check the buckets. We found one of the bucket, even retenton was set to 5 years has shard roup duration was short (1 day), so we updated shard group duration and database didn’t crash. Unfortunately it also changed retention immediatelly from 5 years to 30 days, we lost older data from specific bucket, but database (container) remain stable.
Eventually we decided to split buckets between two nodes and decrease amount of data in one container.
