I installed InfluxDB, Node-RED and Grafana on a Raspberry Pi 3B more than a year ago, and it has been working absolutely flawlessly ever since - until last night. All my Grafana graphs show “No data”, and when I logged in and started influx, I got
$ influx
Failed to connect to http://localhost:8086: Get http://localhost:8086/ping: dial tcp [::1]:8086: connect: connection refused
Please check your connection settings and ensure 'influxd' is running.
After waiting for 20 seconds, the influx command suddenly worked! Then it didn’t. So it’s working on and off.
So I looked in /var/log/syslog, and it looks like this:
Jan 9 21:29:46 data influxd-systemd-start.sh[5123]: InfluxDB API unavailable after 46 attempts...
Jan 9 21:29:47 data influxd-systemd-start.sh[5123]: InfluxDB API unavailable after 47 attempts...
Jan 9 21:29:48 data influxd-systemd-start.sh[5123]: InfluxDB API unavailable after 48 attempts...
Jan 9 21:29:49 data influxd-systemd-start.sh[5123]: ts=2022-01-09T20:29:49.561766Z lvl=info msg="Reading file" log_id=0Yx8K1LW000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1784/_00071.wal size=1220887
Jan 9 21:29:49 data influxd-systemd-start.sh[5123]: InfluxDB API unavailable after 49 attempts...
Jan 9 21:29:49 data Node-RED[370]: 9 Jan 21:29:49 - [error] [influxdb out:9b0882b3.c88ff] Error: connect ECONNREFUSED 127.0.1.1:8086
Jan 9 21:29:50 data influxd-systemd-start.sh[5123]: ts=2022-01-09T20:29:50.235210Z lvl=info msg="Reading file" log_id=0Yx8K1LW000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1784/_00072.wal size=1219380
Jan 9 21:29:50 data influxd-systemd-start.sh[5123]: InfluxDB API unavailable after 50 attempts...
Jan 9 21:29:50 data influxd-systemd-start.sh[5123]: ts=2022-01-09T20:29:50.905023Z lvl=info msg="Reading file" log_id=0Yx8K1LW000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1784/_00073.wal size=1218962
Jan 9 21:29:51 data influxd-systemd-start.sh[5123]: ts=2022-01-09T20:29:51.567135Z lvl=info msg="Reading file" log_id=0Yx8K1LW000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1784/_00074.wal size=1220233
Jan 9 21:29:51 data influxd-systemd-start.sh[5123]: InfluxDB API unavailable after 51 attempts...
Jan 9 21:29:52 data Node-RED[370]: 9 Jan 21:29:52 - [error] [influxdb out:f14f0368.2762] Error: No host available
Jan 9 21:29:52 data Node-RED[370]: 9 Jan 21:29:52 - [error] [influxdb out:f14f0368.2762] Error: No host available
Jan 9 21:29:52 data Node-RED[370]: 9 Jan 21:29:52 - [error] [influxdb out:f14f0368.2762] Error: No host available
Jan 9 21:29:52 data Node-RED[370]: 9 Jan 21:29:52 - [error] [influxdb out:f14f0368.2762] Error: No host available
-just to show you a small excerpt.
SD card is 7.1 GB, 5.4 GB used, 1.4 GB free.
Once in a while I also get these:
Jan 9 01:51:31 data influxd-systemd-start.sh[2591]: runtime: out of memory: cannot allocate 16842752-byte block (530776064 in use)
Jan 9 01:51:31 data influxd-systemd-start.sh[2591]: fatal error: out of memory
Right now it looks like this:
$ free
total used free shared buff/cache available
Mem: 945368 470484 34488 14788 440396 428788
Swap: 102396 101912 484
Any suggestions?