Sudden 100% swap usage on influxdb

, ,
1

We have a situation where the Swap has suddenly spiked up to 100%. When does such a situation occur and how do we debug this? During such scenario, the VM become unresponsive or slow.

If I check the free -lh I see as follows:

influxdb3/8b37314c-a9ff-47a9-9ccb-86e67f1fec1e:~# free -lh
               total        used        free      shared  buff/cache   available
Mem:            61Gi        40Gi       425Mi       614Mi        20Gi        22Gi
Low:            61Gi        61Gi       425Mi
High:             0B          0B          0B
Swap:            9Gi         9Gi       0.0Ki

Available memory is 22 GB (which I guess is free+shared+buff). But if 22Gb is free why is 100% swap used?
Also CPU during didn’t seem high.
The VM runs 16CPU and 64GB Memory. The load went up to 10. So I assume the system isn’t overloaded either.

When does influx prefer to take swap over free memory? and how do we debug if it’s a specific query or ingestion that is causing the issue

Running of influxdb 1.8 on 22.04.1-Ubuntu

2

Hello @vipinvkmenon,
Memory concerns are a long standing concern with InfluxDB. What version of InfluxDB are you using?
Do you have any:

  • Long-running queries or queries that require a lot of memory?
  • High write rates that could cause InfluxDB to consume more memory temporarily

Previous versions of InfluxDB are notorious for grabbing all available RAM and there isn’t operator control over that :frowning:

3

Hello, I am observing this issue in InfluxDB version 2 as well. Is it fixed in the latest influxDB version?

4

Hi Anaiadg,

Thanks for you response.

We’re using Influxdb 1.8.10 (influxdb-1.8.10_linux_amd64) on ubuntu 22.04.1
One of the major aspects that I’ve seen with vmstat is that the amount os si/so is very high indicating large amount of memory swaps.

My first question with regards to influxdb is it recommended to enable swap or disable swap and why as I’ve see contradicting statements on this about some telling to disable swap in production and some telling not to.

Additionally are there documentations on how to identify on what causes this swap…is it ingestion or some query. And if it is queries how can we identify which query is causing the swap?