Swapping with InfluxDB

Hi! From what I’ve read about the storage engine it seems that the read-only files in the FileStore are mmap'ed into memory and never explicitly munmap'ed until the file is closed (mmapAccessor only calls munmap when it is closed, and it is only ever closed when the TSMReader which has a reference to it is closed). Consequently, it seems it’s necessary to enable swapping so the OS can page memory in and out of the mmap’ed files when it needs to. However, most databases recommend disabling swapping, for example Elasticsearch and Cassandra, citing performance reasons. I was curious why InfluxDB took the opposite approach and what your experience has been? I was also wondering if there were any plans to change the storage engine in the future to remove the dependence on swapping?

Thanks!

Hi,
Enabling swap isn’t required for MMAP’d files to be paged to and from disk. You can run the DB just fine without swap space enabled. MMAPing a file allows the OS to manage paging to and from disk automatically via the filesystem page cache. Disk blocks that are accessed will be paged in automatically and accessed as if they as regular memory by the application. Pages that are not in use are paged out (if necessary) by the OS. If you have a lot of memory, the OS will use more of it to keep disk pages in memory. This is reported as cache memory by the OS.

While I can’t say for sure, my guess is that Elasticsearch and Cassandra recommend disabling swap to avoid the JVM process memory from being swapped to disk and causing large GC pause times. They may also rely on the process crashing to trigger failure detection and reroute writes and queries to live nodes. If it does not crash, it may be considered live but continuously slow and failing causing other issues.

As I said previously, running without swap is fine and swap is not required to run the database. You may want to enable swap to reduce the risk of the linux OOM killer kicking in and killing a process due to an errant query. The PostgresSQL docs have some tuning settings that maybe useful to explore. If your DB restarts fast enough, it may be fine to allow the process to crash in this case and restart quickly by your process supervisor.

1 Like

Hey,

Thanks for the detailed response! This is very helpful! I wasn’t aware that MMAP’ed files will still be paged to and from disk when swapping is turned off so this makes a lot more sense now. MMAP IO is, in fact, also the default IO method in Cassandra (and I’m sure a lot of other databases as well) as well, so I wasn’t sure how to reconcile the recommendation to disable swapping with their use of MMAP before your clarification.

It seems that one possible downside to MMAP is that if a needed page is not in memory, it will cause a page fault and the current thread will be put to sleep until the page is read from disk. Reading a file that is not memory will also cause the page to be read from disk, but from what I understand Go handles Read/Write file operations in their own thread so that the runtime will still have the same number of processor threads able to run goroutines as set by GOMAXPROCS. Consequently, it seems that in the case of MMAP there may be one less processor thread to run goroutines than in the case of a traditional file IO call when a page fault occurs. I was curious if you’ve ever seen this in practice? It, admittedly, certainly seems like it would be a very uncommon scenario though.

@jeromefroe I’ve found that when swap is enabled, under heavy load scenarios the server enters a swap-y death spiral:

  • Memory pressure (normally from increased write volume combined with an already high series count) causes the machine to activate swap
  • The additional CPU required to manage the swap space reduces the CPU available to handle and persist incoming writes
  • This uses more memory and forces more and more data into swap exacerbating the already constrained CPU
  • This quickly leads to an unrecoverable situation and the process goes down.

I would definitely suggest that you disable swap in production with Influx if that is something you wish to avoid.

1 Like