InfluxDB crashes with write failure error

1

I have InfluxDB 2.7.0 running as a service in a Ubuntu 22.04 server. Node-red writes to this instance and at the moment we only have a handful of buckets. Earlier today I could not connect the dashboard and Node-red had thousands of error messages indicating a failure to write messages. In syslog this appeared over 14,000 times:

WARN: Write to InfluxDB failed (attempt: 1). x [RequestTimedOutError]: Request timed out

And in the journal for influxdb.service this appeared once, within about 30 seconds of the bulk of the Node-red errors:

influxd-systemd-start.sh[640]: ts=2023-11-08T01:59:42.189785Z lvl=info msg="Write failed" log_id=0lEUcbgl000 service=storage-engine service=write shard=128 error="engine: context canceled"

The entire server was mostly unresponsive when I discovered the problem and had to be rebooted from the virtualisation control panel. Any places I should start looking?

2

Hello @phil-aznz,
Welcome!
Hello I’m not sure. You could check the InfluxDB logs. System resources. Also maybe Service Configuration. Check the configuration of the InfluxDB service. Look for any parameters that might limit connections or write capabilities (max-concurrent-writes, write-timeout, etc.).

3

Thanks. I didn’t notice anything out of the ordinary prior to the meltdown. It’s a lightly loaded VPS and we only have a few devices sending what I’d consider to be light traffic… 10K publish requests with an average payload of 4 bytes each over a 24-hour period is normal at the moment.

The service config is mostly unmodified from the distribution defaults with the exception of the storage paths and enabling SSL.

I’ll put some extra monitoring in place and see if I can get more information the next time it happens.