New connections are refused, WAL file grows too large to restart influxdb

InfluxDB 2
1

Hi,
I have X-Influxdb-Build: OSS, X-Influxdb-Version: 1.7.9. I’m currently doing ~10,000 writes to influxdb per minute all day. After several days go by, I noticed that influxdb will stop receiving new connections and if I then try to restart it, it will run out of memory and get stuck in a restart loop. This is only then fixed by clearing out my WAL directory. Prior to doing so, I always find one WAL file with a very high size like below:

4.0K    ./data/autogen/185
440K    ./data/autogen/44
440K    ./data/autogen/7
432K    ./data/autogen/48
416K    ./data/autogen/221
416K    ./data/autogen/16
476K    ./data/autogen/38
260K    ./data/autogen/194
440K    ./data/autogen/45
79M     ./data/autogen/252

Here is what the connection error looks like:

Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0xb5e5ae90>: Failed to establish a new connection: [Errno 111] Connection refused’)

While influxdb is running I can sometimes see one of the WAL files start to grow way bigger than the rest. Is there something I’m doing wrong? Any help in troubleshooting this would be greatly appreciated!

2

bumping this for visibility as this is still a reoccurring issue