I have X-Influxdb-Build: OSS, X-Influxdb-Version: 1.7.9. I’m currently doing ~10,000 writes to influxdb per minute all day. After several days go by, I noticed that influxdb will stop receiving new connections and if I then try to restart it, it will run out of memory and get stuck in a restart loop. This is only then fixed by clearing out my WAL directory. Prior to doing so, I always find one WAL file with a very high size like below:
4.0K ./data/autogen/185 440K ./data/autogen/44 440K ./data/autogen/7 432K ./data/autogen/48 416K ./data/autogen/221 416K ./data/autogen/16 476K ./data/autogen/38 260K ./data/autogen/194 440K ./data/autogen/45 79M ./data/autogen/252
Here is what the connection error looks like:
Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0xb5e5ae90>: Failed to establish a new connection: [Errno 111] Connection refused’)
While influxdb is running I can sometimes see one of the WAL files start to grow way bigger than the rest. Is there something I’m doing wrong? Any help in troubleshooting this would be greatly appreciated!