I am using InfluxDB (1.7.3 and 1.8.3) and Grafana (5.1.4) since 2019 on two RPi Zero W only with the onboard SD card for logging and visualizing temperatures and humidity. One instance writes 6 temperatures every 20 seconds, the other one 1 temperature and 1 humidity every 30 seconds. Both databases keep data forever (no retention policy, 0s) and are stil below 100 MBytes now.
All started working fine, but eventually I remarked in Grafana missing measurements at about 05:10 local time (Europe/Zurich) for 1 to many minutes during influxd TSM full compactions. I tried to avoid this by changing CompactFullWriteColdDuration from 4h to 2h, but this only shifted full compactions to 03:10 - so this configruation figured out not to be a “duration” (I believed interval) but the UTC “time” to start a TSM full compaction. As I mainly observed the curves in Grafana during the day, I made no further investigations. After about 2 years I transferred the SD card image to a new one, believing that card wear out was the cause for more missing measurements and longer lasting TSM full compactions.
This autumn I took more time to investigate the logs, as missing measurements prolongated sometimes til 45 minutes. In syslog I found around 05:10 sometimes problems regarding RAM memory going out, time outs and failures for influxd services and kernel warnings about blocked tasks for many minutes in mmc_rescan etc. So there are problems in writing the SD card (bad blocks) or just too many reads and writes and missing free RAM memory (I logged every 5 minutes the result of free, going from 200k down to 10k). My first try to transfer system and database to a USB stick memory and using the SD card just for booting was of no success.
Then I regarded the size of my database and the _internal database of InfluxDB. The latter one is about 44 MBytes, half the size of my own database since 2019, but retention policy is at 7 days! So I made twice a “sudo ls -alsR /var/lib/influxdb/ > file_1.txt” into different files with a pause of ~7,5 minutes between them and regarded them with diff - oops, my database increased for 8,6 kBytes (in 1 .wal file), whereas into _internal were written 14,7 MBytes (in 2 .wal files), a ratio of ~1:1709! That’s the SD card killer…
I write 6 fields every 20 seconds, the _internal has 113 fields and writes every 10 seconds, that are per minute 6x3 = 18 entries against 113x6 = 678 entries, but that is only 1:37,6 and not 1:1709! Reading out one of my temperatures from InfluxDB I got exactly the numbers I wrote, i.e. 3 entries per minute, but reading out one value (currentSegmentDiskBytes FROM tsm1_wal) from _internal, I got 1248 entries per minute, 208 per 10 seconds! Instead of writing one value every 10 second, _internal database has 208 entries with the same timestamp, mainly value 0 and only 4 other numbers (e.g. 326073, 3174020, 10577078 and 4996451 at 2022-12-04,23:00:00 UTC). I didn’t check all 113 fields, but other fields from _internal tested showed all 208 entries every 10 seconds…
What is the meaning of writing in _internal database fields every 10 seconds not only one value but 208 values and thereof 204 times just 0 (zero)? Just a nasty bug? And I couldn’t find any description of the 113 fields.
Since about 2 years I have a Raspberry Pi 3A+ with InfluxDB (1.8.4) and Grafana (7.3.7) logging power meter values. This quad core processor is powerfull enough to do influxd TSM full compactions in a few seconds without obstructing other tasks, I couldn’t find problems in syslog. And I believe, a new Raspberry Pi Zero 2 W would also do the job easily. But that is not a solution, the intense writing to the SD card remains fatal.
My solution is keeping the Raspberry Pi Zero W and I disable the _internal database in influxdb.conf:
[monitor]
# Whether to record statistics internally.
# store-enabled = true
store-enabled = false
I don’t know, why this is enabled by default, as there is no description and I believe no tool available to monitor the InfluxDB application, and nobody does it on such small systems, it’s only obstruction.
