After some more investigation, this appears to be a WAL issue. Based on some other threads like InfluxDB 1.7.4 fails after 9 months without issues and [0.9.3] WAL gets progressively slower as DB size increases · Issue #3885 · influxdata/influxdb · GitHub, it seems like the WAL files get progressively larger as new data is added, and don’t get flushed. Then when the database starts again, it tries to load the entire WAL into memory to do compaction, which results in an OOM error.
I’ve tried altering all the storage configurations, but nothing seems to prevent Influx from reading all the data into memory at once. Any help with this?
