Incorrect hostname picked up by Telegraf (docker container)

Telegraf
, , ,
1

Hi everyone,

I have two servers running Ubuntu 20.04 that are connected to each other with Docker Swarm, and I’ve managed to setup Telegraf, InfluxDB and Grafana as containers to monitor them.

TL;DR

I want Grafana to display the name of the hosts instead of the Docker container ids.

Everything seems to work great save for one little detail: inside the Grafana dashboard, the hostname that appears is the container id rather than the actual host where the containers are deployed.

According to the documentation for the telegraf image, under “Monitoring the Docker Engine Host”, the recommended way is to mount the host filesystem into the container. And as per the telegraf agent configuration, the hostname can be left blank and it’ll default to use the function os.Hostname() to populate the field.

As I’ve mounted the host’s filesystem in the container, why is this not being picked up correctly?

Here’s an excerpt from the compose file:

telegraf:
    image: telegraf:1.21 # not the latest
    deploy:
      mode: global
    environment:
      HOST_ETC: /hostfs/etc
      HOST_PROC: /hostfs/proc
      HOST_SYS: /hostfs/sys
      HOST_VAR: /hostfs/var
      HOST_RUN: /hostfs/run
      HOST_MOUNT_PREFIX: /hostfs
    volumes:
      - ./files/telegraf.conf:/etc/telegraf/telegraf.conf
      - /:/hostfs:ro
    networks:
      - influxdb

And the configuration file that I’m mounting:

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  hostname = "" # <--- Left blank on purpose
  omit_hostname = false

[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  timeout = "5s"
  database = "telegraf"
  username = "telegraf"
  password = "metricsmetricsmetricsmetrics"

Thanks for the help!

2

Hi,

As I’ve mounted the host’s filesystem in the container, why is this not being picked up correctly?

The hostname is correctly reporting the hostname of the container. Telegraf will run go’s os.hostname which will read the hostname from here.

When you mount the host filesystem it ends up at /hostfs in the container. The container’s filesystem does not get overridden or updated to be identical to the host.

Thanks

3

Hi Josh,

Thank you for the help on this. I’m not familiar with Go but it seems that os.hostname tries to get this information from /proc/sys/kernel/hostname. However when I read from that file inside my container, as well as on the mounted filesystem, I get the same name:

root@722e7b0d7ee9:/# cat /proc/sys/kernel/hostname
722e7b0d7ee9

root@722e7b0d7ee9:/# cat /hostfs/proc/sys/kernel/hostname
722e7b0d7ee9

It’s almost like is overwriting this information somehow, though I’m not sure if it’s relevant. In any case, do you know of a way to have the host name show up instead of the container name?

EDIT: Seems like mounting /proc/sys/kernel/hostname from the host to the container does not really work either.

Thanks again!

4

Correct, /proc is a special filesystem and does not behave like bind-mounting a normal text file for example.

In any case, do you know of a way to have the host name show up instead of the container name?

I have seen others recommend using an environment variable with Docker e.g. (-e HOST_HOSTNAME='hostname') and then you could have your telegraf configuration possibly read that value.

5

That explains it, very good to know.

That seems to be the only option from the looks of it. As I’m deploying from a compose file, and the telegraf configuration is parsed when the container runs, I need to figure out a way to make this dynamic so that the environment variable matches each host it runs on. It would be great to have the ability to provide a file, as opposed to a value, similar to how it works when using Docker Secrets.

At this point I think it’s clear that the issue is no longer related to telegraf, and more so with Docker. I’ll come back and update this thread if I manage to figure something out.

Thanks the help!

6

The solution to this problem is to create the service using template parameters. You can specify the hostname of the container but using one of the ready to use variables, so the compose file would look something like this:

telegraf:
    image: telegraf:1.21 # not the latest
    hostname: "{{.Node.Hostname}}"

In case it helps someone :slight_smile:

2 Likes
7

Thanks for looping back and sharing what you used!