InfluxDB 1.8.10 accessing filesystem every ~5 seconds

Hello,

I need some help with an issue I recognized since I have migrated my RasPi 3B from running with a microSD to a system running from a SSD.

I migrated the RasPi and all works fine so far. The short migration path has been, that I added the following lines to the end of /boot/config.txt in tha ALL-section

program_usb_boot_mode=1
program_usb_timeout=1
dtparam=sd_poll_once

Then I powered it off (sudo shutdown -h now), copied an image from the microSD to a SSD (WDS250G1B0A) which is connected then to the USB-Port of the RasPi and after powering the device on it runs (now for three days).

The Raspi is used with InfluxDB, Node-Red, Moswuitto and grafana to monitor two tasmota based sockets (Nuos 1AT, reporting every 5 Minutes) and temperature + frequency of the Raspi itself (also every 5 Minutes).

The SSD is stored in an external case that has an acivity LED and I recognized an drive activity every 5-6 seconds wich has not been reported by the build-in activity LED of the Raspi as long as it runs from microSD. Therefore I checked with iotop for the reason and found:

$ iotop
TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
582 ?sys influxdb    0.00 B/s   19.26 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf

Now my question is, what can I do to reduce these operations as I don’t think they are needed (cause I have not seen them before switched from microSD to SSD) and I believe it will damage the SSD in a shorter time.

I tried to find solutions by myself but have not found helpful information although I am pretty sure that I am not the first human being facing this issue :slight_smile: Furthermore I try to provide any helpful details that might help to analyze the issue but I will give more details if needed of course. I have only very basic Linux-skills an my skills in InfluxDB, Red-Node, Mosquitto and grafana are about to 14 days now :wink: Just to have it said.

Thanks in advance for your time and help!

The RasPi is running as a headless system in my own home-network where I am the only user. Here are some more details:

$ cat /sys/firmware/devicetree/base/model
Raspberry Pi 3 Model B Rev 1.2

$ hostnamectl
   Static hostname: IOT_01
         Icon name: computer
        Machine ID: #
           Boot ID: #
  Operating System: Debian GNU/Linux 11 (bullseye)
            Kernel: Linux 5.15.84-v8+
      Architecture: arm64

$ influxd version
InfluxDB v1.8.10 (git: 1.8 688e697c51fd)

And this is the shortened influxdb.conf I use (I cut the comment-lines):

### Welcome to the InfluxDB configuration file.
[...]
[meta]
  # Where the metadata/raft database is stored
  dir = "/var/lib/influxdb/meta"


[data]
  # The directory where the TSM storage engine stores TSM files.
  dir = "/var/lib/influxdb/data"

  # The directory where the TSM storage engine stores WAL files.
  wal-dir = "/var/lib/influxdb/wal"

  series-id-set-cache-size = 100

[coordinator]

[retention]

[shard-precreation]

[monitor]

[http]
enabled = true
bind-address = ":8086"
auth-enabled = true
access-log-path = "/var/log/influxdb/access.log"
write-tracing = false
pprof-enabled = false
https-enabled = false
http-certificate = "/etc/ssl/influxdb.pem"

[subscriber]

[[graphite]]

[[collectd]]

[[opentsdb]]

[[udp]]

[continuous_queries]

[tls]

Hi,

no ideas anyone?

Here is a iotop-log of 5 minutes - it contains 33 times “influxd -config /etc/influxdb/influxdb.conf” :frowning:

$ sudo iotop -bktoqqq
22:33:10      98 ?sys root        0.00 K/s  113.70 K/s  ?unavailable?  [jbd2/sda2-8]
22:33:10     485 ?sys influxdb    0.00 K/s  397.93 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:33:16      98 ?sys root        0.00 K/s    3.67 K/s  ?unavailable?  [jbd2/sda2-8]
22:33:20     525 ?sys influxdb    0.00 K/s   22.78 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:33:30      98 ?sys root        0.00 K/s   15.19 K/s  ?unavailable?  [jbd2/sda2-8]
22:33:30     525 ?sys influxdb    0.00 K/s   22.79 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:33:40     520 ?sys influxdb    0.00 K/s   26.60 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:33:50     520 ?sys influxdb    0.00 K/s   22.79 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:34:01     551 ?sys influxdb    0.00 K/s   22.79 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:34:10      98 ?sys root        0.00 K/s   15.18 K/s  ?unavailable?  [jbd2/sda2-8]
22:34:10     519 ?sys influxdb    0.00 K/s   26.57 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:34:20     551 ?sys influxdb    0.00 K/s   22.78 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:34:30     815 ?sys influxdb    0.00 K/s   22.80 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:34:40     519 ?sys influxdb    0.00 K/s   22.80 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:34:50      98 ?sys root        0.00 K/s   15.20 K/s  ?unavailable?  [jbd2/sda2-8]
22:34:50     525 ?sys influxdb    0.00 K/s   26.59 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:35:00     555 ?sys influxdb    0.00 K/s   21.98 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:35:02      98 ?sys root        0.00 K/s   26.59 K/s  ?unavailable?  [jbd2/sda2-8]
22:35:02     136 ?sys root        0.00 K/s  212.75 K/s  ?unavailable?  systemd-journald
22:35:02     451 ?sys root        0.00 K/s    7.60 K/s  ?unavailable?  rsyslogd -n -iNONE [rs:main Q:Reg]
22:35:02     557 ?sys influxdb    0.00 K/s    7.60 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:35:02     494 ?sys mosquitt    0.00 K/s    3.80 K/s  ?unavailable?  mosquitto -c /etc/mosquitto/mosquitto.conf
22:35:08      98 ?sys root        0.00 K/s    3.80 K/s  ?unavailable?  [jbd2/sda2-8]
22:35:11     519 ?sys influxdb    0.00 K/s   22.68 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:35:20     525 ?sys influxdb    0.00 K/s   26.59 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:35:30      98 ?sys root        0.00 K/s   15.19 K/s  ?unavailable?  [jbd2/sda2-8]
22:35:30     557 ?sys influxdb    0.00 K/s   22.79 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:35:39      98 ?sys root        0.00 K/s    7.29 K/s  ?unavailable?  [jbd2/sda2-8]
22:35:39     556 ?sys influxdb    0.00 K/s    7.29 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:35:40      98 ?sys root        0.00 K/s    3.79 K/s  ?unavailable?  [jbd2/sda2-8]
22:35:40   44261 ?sys influxdb    0.00 K/s   22.73 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:35:50     556 ?sys influxdb    0.00 K/s   25.58 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:36:01     519 ?sys influxdb    0.00 K/s   22.06 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:36:10      98 ?sys root        0.00 K/s   15.21 K/s  ?unavailable?  [jbd2/sda2-8]
22:36:10     519 ?sys influxdb    0.00 K/s   22.82 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:36:20     519 ?sys influxdb    0.00 K/s   26.61 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:36:30   44261 ?sys influxdb    0.00 K/s   22.82 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:36:37      98 ?sys root        0.00 K/s   38.03 K/s  ?unavailable?  [jbd2/sda2-8]
22:36:37     136 ?sys root        0.00 K/s  110.28 K/s  ?unavailable?  systemd-journald
22:36:37     451 ?sys root        0.00 K/s   11.41 K/s  ?unavailable?  rsyslogd -n -iNONE [rs:main Q:Reg]
22:36:37     535 ?sys grafana     0.00 K/s    3.80 K/s  ?unavailable?  grafana server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
22:36:37     717 ?sys grafana     0.00 K/s   19.01 K/s  ?unavailable?  grafana server --config=/etc/grafana/grafana.ini --pidfile=/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
22:36:40      98 ?sys root        0.00 K/s    3.80 K/s  ?unavailable?  [jbd2/sda2-8]
22:36:40     551 ?sys influxdb    0.00 K/s   22.81 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:36:50      98 ?sys root        0.00 K/s   15.20 K/s  ?unavailable?  [jbd2/sda2-8]
22:36:50     551 ?sys influxdb    0.00 K/s   22.80 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:37:00     520 ?sys influxdb    0.00 K/s   25.79 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:37:11      98 ?sys root        0.00 K/s    7.60 K/s  ?unavailable?  [jbd2/sda2-8]
22:37:11   44261 ?sys influxdb    0.00 K/s   22.79 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:37:20     551 ?sys influxdb    0.00 K/s   22.80 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:37:30      98 ?sys root        0.00 K/s   15.19 K/s  ?unavailable?  [jbd2/sda2-8]
22:37:30   44261 ?sys influxdb    0.00 K/s   26.59 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:37:40     815 ?sys influxdb    0.00 K/s   22.80 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:37:50     554 ?sys influxdb    0.00 K/s   21.90 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:38:00     554 ?sys influxdb    0.00 K/s   25.66 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf
22:38:10      98 ?sys root        0.00 K/s   15.20 K/s  ?unavailable?  [jbd2/sda2-8]
22:38:10     554 ?sys influxdb    0.00 K/s   22.79 K/s  ?unavailable?  influxd -config /etc/influxdb/influxdb.conf

Hello @FrancisC,
I’m not sure. I’m asking around. Thank you for your patience.

I’m not sure why it would be different using SSD if you’re using the same config and workload but I wouldn’t worry too much about this size of load causing the drive to fail anytime soon. You may be able to reduce the load by disabling internal monitoring. To do that, set this in the config:

[monitor]
  store-enabled = false

You can read more about internal monitoring in the docs: InfluxDB _internal 1.x measurements and fields | InfluxData Platform Documentation

Hello @Anaisdg @dgorton,

thank you - I will give it a try and will let you know if this setting will change the behavior.

Just to clearify:
I can not say for sure if these operations have not been on the micro-sd-card already. But the LED have not shown that often any activities.

Regards

Hello,

the configuration-hack @dgnorton posted above did it for me. As it is very unlikely that I will need any internal monitoring information of this InfluxDB-system in the future I do not care too much in not saving these bits. After changing the the config and a restarting the Raspi the filesystem-operations decreased significantly.

Therefore I tagged the post with the “Solution”-tag.

So thank you very much for your assistance
:star: :star: :star: :star: :star: