Disk input not reporting metrics for all disks

Telegraf
1

This is the telegraf.toml that I’m using:

[[inputs.cpu]]
[[inputs.mem]]
[[inputs.disk]]
  mount_points = ["/", "/mnt/storage"]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

[[inputs.statsd]]
  protocol = "udp"
  service_address = ":8125"
  metric_separator = "."
  datadog_extensions = true
  datadog_distributions = true

[[outputs.postgresql]]
  connection = "postgresql://postgres:postgres@postgres:5432/metrics"
  tags_as_foreign_keys = true
  create_templates = [
      '''CREATE TABLE {{ .table }} ({{ .columns }})''',
      '''SELECT create_hypertable({{ .table|quoteLiteral }}, 'time', chunk_time_interval => INTERVAL '7d')''',
      '''ALTER TABLE {{ .table }} SET (timescaledb.compress, timescaledb.compress_segmentby = 'tag_id')''',
  ]

And here is the relevant Docker Compose configuration:

  telegraf:
    image: telegraf:1.30.2
    container_name: telegraf
    restart: unless-stopped
    privileged: true
    environment:
      - HOST_MOUNT_PREFIX=/hostfs
      - HOST_PROC=/hostfs/proc
      - HOST_SYS=/hostfs/sys
      - HOST_ETC=/hostfs/etc
    volumes:
      - ./telegraf.toml:/etc/telegraf/telegraf.conf:ro
      - /:/hostfs:ro
    ports:
      - "8125/udp"
    depends_on:
      - postgres

This works great for my root mountpoint:

metrics=# select time from disk inner join disk_tag on disk_tag.tag_id = disk.tag_id where disk_tag.path = '/' order by disk.time desc limit 10;
        time         
---------------------
 2024-09-13 15:43:00
 2024-09-13 15:42:50
 2024-09-13 15:42:40
 2024-09-13 15:42:30
 2024-09-13 15:42:20
 2024-09-13 15:42:10
 2024-09-13 15:42:00
 2024-09-13 15:41:50
 2024-09-13 15:41:40
 2024-09-13 15:41:30
(10 rows)

But metrics for /mnt/storage are very intermittent:

metrics=# select time from disk inner join disk_tag on disk_tag.tag_id = disk.tag_id where disk_tag.path = '/mnt/storage' order by disk.time desc limit 10;
        time         
---------------------
 2024-09-13 15:36:50
 2024-09-13 14:35:40
 2024-09-13 03:14:00
 2024-09-13 03:13:50
 2024-09-13 03:13:40
 2024-09-13 03:13:30
 2024-09-13 03:13:20
 2024-09-13 03:13:10
 2024-09-13 03:13:00
 2024-09-13 03:12:50
(10 rows)

Any ideas what I might have misconfigured?

2

@scttnlsn it would be best to create a config with only the inputs.disk plugin. Then run

telegraf --config <config only containing disk> --test --debug

on the host (i.e. not inside a docker container) to see if and why disks are omitted and ruling out container issues.

If everything works in this setup, do the same from within the container and check the output. Is it still valid? Then probably the issue is on the output side…