We were running InfluxDB 1.7.4 and seeing some high memory/cpu utilization. After reading some blogs about TSI I thought it would be the way to go.
We upgraded to 1.7.10 and enabled TSI. To my surprise things started going sideways and resource utilization started getting higher.
Our instances were running on 10k RAID arrays and it’s been fine in the past which I believe that might have been before the 1.7.x.
We are seeing iops of 2-4k/sec. Seeing those numbers, I’ve created a new instance on some Intel Optane p4800x. That’s the fatest nvme drive I could find.
The usage got worst. Memory usage climbed up to 65GB and eating all the CPU they could get.
My writes are at 2 Gbps per seconds (200MB/sec) and the iops are similar.
There is no queries sent to that server yet. Read IO is none.
What could be happening?
Am I missing some settings to prevent constant compaction?
Seems like we have about 16GB of data a day (shards are daily)
Thanks for the help