Lack of understanding of memory usage

Store
1

Hi,

I’m testing InfluxDB v1.5 and I’m sending 20M points distributed in 1M series by batch of 20000. I’m using on-disk indexing (index-version = “tsi1”) and autogen RP.
My hardware sizing is : 8GB RAM and 4 CPU core. I know I do not follow yet the recommended hardware sizing but I do not think this comes into account here and I will change soon.

With Grafana, I’m monitoring the VM InfluxDB is running on and InfluxDB itself through the _internal database.

Here is the evolution of the mem.ramUsed and mem.ramTotal metrics when loading my 20M points :

RamEvolution
RamEvolution.png1815×603 43.2 KB

I’m trying to understand how InfluxDB works under the hood and I can’t figure out why is the RAM decreasing about 5 to 10 minutes after the insertion is done.

I checked some metrics and tested some config modification :

  • There is no compaction (of any type) happening at this moment and the size of disk.used does not change.
  • The modification of the cache-snapshot-write-cold-duration config value does not affect the duration after which the RAM starts to decrease.
  • RAM cached (mem.Cached metric) size does does not change when mem.ramUsed size decreases.

I would like to understand why is the RAM behaving in such a way not to experience undesirable RAM evolution before changing my hardware sizing.

Thank you for you help,

Bastien

2

Hi guys ! No idea what’s going on there ?

Do you think this is an Influx process that “flushes” RAM or is it a Linux system feature ?

Thank you for your help!

Bastien

3

The initial increase is probably just the baseline memory usage to handle writes. The decrease is probably the Go runtime returning memory to the operating system. You could gather heap or goroutine profiles from the /debug/pprof endpoint if you wanted to get more insight into how the memory is being consumed.

4

Thank you ! As you said, decreases seems to be go heap released to the OS.

Moreover, as mentioned here and here, Garbage collector is executed every 2 minutes and then memory is given back to the OS about 5 minutes later if it has not been requested by the Go process. This corresponds to the behavior I notice.

Thank you !