Disk_used_percent is missing for iscsi disks

Telegraf
1

I can’t get telegraf to include disk_used_percent for iscsi disks. The metrics for the mount point “/” are properly delivered. But the metrics for “/var/lib/docker/volumes” and “/usr/local/backup” are missing. The docker image used is the most recent available. The filesystem used on the disks is ext4. Any ideas what I forgot?

compose.yml

  telegraf:
    container_name: grafana-telegraf
    image: telegraf:alpine
    user: telegraf:989
    pid: host
    network_mode: host
    restart: unless-stopped
    depends_on:
      - prometheus
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
#      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/run:/var/run:ro
      - /proc/net:/hostfs/proc/net:ro
      - /dev:/dev:ro

telegraf.conf


[agent]
  interval = "15s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "15s"
  flush_jitter = "0s"
  precision = "0s"


[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false
  core_tags = false


[[inputs.mem]]


[[inputs.swap]]


[[inputs.system]]
  fieldexclude = ["uptime_format"]

[[inputs.net]]
  interfaces = ["enp*"]


[[inputs.disk]]
  mount_points = ["/","/var/lib/docker/volumes","/usr/local/backup"]

[[inputs.diskio]]
  skip_serial_number = true


[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  gather_services = false
  source_tag = false
  container_name_include = []
  container_name_exclude = []
  storage_objects = ["container", "image", "volume"]
  timeout = "5s"
  perdevice_include = ["cpu"]
  total_include = ["cpu", "blkio", "network"]
  docker_label_include = ["com_docker_compose_project", "container_name"]
  docker_label_exclude = ["container_id", "container_image", "container_status", "container_version", "server_version", "host"]
  namedrop = ["container_id", "container_image", "container_status", "container_version", "server_version", "host"]
  fieldexclude = ["container_id","health_status"]


[[outputs.http]]
  url = "http://10.4.0.3:9090/api/v1/write"
  data_format = "prometheusremotewrite"
  use_batch_format = true

  [outputs.http.headers]
    Content-Type = "application/x-protobuf"
    Content-Encoding = "snappy"
    X-Prometheus-Remote-Write-Version = "0.1.0"

Greetings!

2

Are you certain these are correct mount points for the disks or partitions? The plugin will not work on individual folders.

Additionally, are you getting any metrics from the two mount points, or are they entirely missing? If you are getting the metrics in some capacity but with some missing fields, what do they look like?

3

It works with the following compose.yml. I also tried to mount the host root folder to /hostfs, but that did not work. The docker volumes folder is mounted with a volume because the telegraf user doesn’t have access rights on the host volumes folder.

compose.yml

  telegraf:
    container_name: grafana-telegraf
    image: telegraf:alpine
    user: telegraf:989
    pid: host
    network_mode: host
    restart: unless-stopped
    depends_on:
      - prometheus
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
      #- /var/run/docker.sock:/var/run/docker.sock:ro
      - telegraf-volume:/var/lib/docker/volumes:ro
      - /usr/local/backup:/usr/local/backup:ro
      - /dev:/dev:ro
      - /var/run:/var/run:ro
      - /proc/net:/hostfs/proc/net:ro

Many Greetings!

1 Like