Influx freezes VM, maxing out memory

I’m running a small influx database on a google g1-small instance with 1.7GB of RAM. Recently, I noticed that the instance freezes up every couple of days. Looking at go_memstats_sys_bytes, it appears it grows steadily until it hits about 1.3GB and then the VM freezes. Most likely because it’s running out of memory. Then if I restart influxd, go_memstats_sys_bytes goes back down to about 200MB and everything works normally until it reaches out of memory again. The question is why doesn’t influx clean up memory that’s no longer being used / was used by previous queries? Or am I misunderstanding how that’s supposed to work?

I’m on influx 2.0.3.

Hello @kostyan5,
What type of queries are you executing?
Are you running any tasks?

I have 1 task running every minute resampling data from previous hour but that’s a very low amount of data (50 points/hour or so).

For queries, I have my grafana dashboard, but my entire database is 20MB so it’s not a lot of data.

I also have a /metrics scraper running.

Anyway, it appears that influx is not running garbage collection as go_memstats_sys_bytes never seems to go down. It increases from 200MB to 1.3GB and then influx freezes the system until I restart the service. See attached plot. Every time the memory drops is after a restart.

@Anaisdg Any thoughts on this?

I upgraded my instance with more RAM and the memory usage just keeps growing. Sounds like a memory leak.

Hello @kostyan5,
Hmmm I don’t know. That is odd. Let me share with the team. Thanks for sharing all that info.

@Anaisdg Here’s some additional info:

I’m noticed that influxdb 2 keeps growing in memory usage until it hits about 2.6GB and then levels off. I was trying to run this on a machine with 1.7GB of RAM and it was regularly maxing it out in about 1 day and freezing up. I upped the RAM to 5GB and it appears to level off at 2.6GB. Is there a way I can configure influx to use less RAM? It can clearly run my workload as it is functional for about a day with 1.7GB of RAM. It appears that it keeps stuff in memory until it hits a pre-defined memory threshold and then starts cleaning it up. I just want to start the clean up sooner than 2.6GB.

purple = go_memstat_sys_bytes
blue = go_memstat_alloc_bytes

I’m not sure if this will work for you, and it probably will reduce performance, but should decrease memory usage:
Set the environmental variable

Blockquote
GOGC=20

Where influxdb can see it and restart influxdb.

Hey,
I’m struggling the something similar in a fresh install of 2.0.4 (dpkg) on Ubuntu 18.04.4 LTS. I have loaded some simple data into influx (a small subset of data from earlier influx install that has worked fine.) Currently no queries, no new data is written, no telegram services, etc.

Influxdb starts. Ramps up memory use. Reaches around 1.7/1.8 of the 2 gigs available. Then it remains there for a minute or two and crashes. Restarts. Cycle repeats 5-10 times every hour.

I tried adding GOGC=20 to sudo systemctl edit --full influxdb.service. But no change.

Update 30th April: Did clean install to version 2.0.6. Wrote minuscule amount of data. Size of /var/lib/influxdb/engine/data is 5 megabyte. Influxdb is using 30 % of memory, as reported by htop, without running any queries. This must be excessively large, even taking into account any overhead needed.