At work, we run an (OSS) instance of influxDb 2.5. We generate an incredibly high amount of data (probably bordering around 500k-1M values per second). For a while now, we’ve been experiencing performance problems, mostly memory related (OOM-killer). The VM the instance is running on is has 12 Xeon Gold 5218 cores and ~90G of RAM.
We are looking to see if there are any experienced 3rd parties that can offer paid support to helping overcome these performance issues.
Just for some context, can you confirm you are using Flux? How are you ingesting the data and how frequently? When you say that you “generate” a high amount of data, does that mean you are constantly running queries to display or aggregate your data? (in the range of 500k to 1M values per second)?
Yes, we’re using Flux for our (read) queries. However, the performance issues we’re seeing seem to be mostly related to writing data to buckets (using the line protocol through the /api/v2/write endpoint).
The 500k-1M values referred to us writing data to the database. It looks like we might be writing faster than influx can compact the data. The reason for this suspicion is that, if I run influxdb without any read or write clients connected to it, it will run with a very high CPU usage and the debugging enpoints show that most of this CPU time is spent in compaction-related functions (I.E runPeriodicCompaction()).
