Inputs.docker - no cpu, network infos of container

Telegraf
1

Telegraf 1.28.5 is running locally in an Alpine system 3.19 running docker. All metrics from the host system itself including cpu, mem and running container are tracked. All metrics of the container itself (e.g. docker_container_cpu, docker_container_mem) are not tracked any more. A long time ago I saw them… I played with the settings but the result is always 0 for docker_container_cpu and docker_container_mem of any container.
Can someone help?

telegraf.conf
[[inputs.docker]]
endpoint = “unix:///var/run/docker.sock”
timeout = “5s”
perdevice = false
perdevice_include = [“cpu”, “network”]
total_include = [“cpu”, “network”]

2

Hmm so what changed? Was it from a previous version of telegraf? Old version of Docker? To ask the basic quesiton, are you sure you have containers running?

What do your logs show?

I took your config and ran it locally with an eclipse-mosquitoo container running:

docker,engine_host=ryzen,host=ryzen,server_version=25.0.3 n_listener_events=0i,n_used_file_descriptors=31i,n_containers=6i,n_containers_running=1i,n_containers_stopped=5i,n_containers_paused=0i,n_images=19i,n_goroutines=64i,n_cpus=32i 1709569311000000000
docker,engine_host=ryzen,host=ryzen,server_version=25.0.3 memory_total=67334135808i 1709569311000000000
docker_container_status,container_image=eclipse-mosquitto,container_name=crazy_austin,container_status=running,container_version=unknown,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=ryzen,host=ryzen,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=25.0.3 pid=85492i,exitcode=0i,restart_count=0i,container_id="d1b3f3c308d1ffb6bf753df65f77877ccb062be9a0201ca97a67c53b22bbfd1a",started_at=1709569307749544367i,uptime_ns=4481244991i,oomkilled=false 1709569312000000000
docker_container_mem,container_image=eclipse-mosquitto,container_name=crazy_austin,container_status=running,container_version=unknown,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=ryzen,host=ryzen,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=25.0.3 max_usage=0i,usage=4747264i,usage_percent=0.0070503080540553625,container_id="d1b3f3c308d1ffb6bf753df65f77877ccb062be9a0201ca97a67c53b22bbfd1a",active_anon=405504i,inactive_anon=0i,pgfault=960i,limit=67334135808i,active_file=3932160i,inactive_file=135168i,pgmajfault=5i,unevictable=0i 1709569312000000000
docker_container_cpu,container_image=eclipse-mosquitto,container_name=crazy_austin,container_status=running,container_version=unknown,cpu=cpu-total,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=ryzen,host=ryzen,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=25.0.3 usage_in_usermode=10977000i,usage_in_kernelmode=7313000i,throttling_periods=0i,throttling_throttled_periods=0i,container_id="d1b3f3c308d1ffb6bf753df65f77877ccb062be9a0201ca97a67c53b22bbfd1a",usage_total=18290000i,usage_system=330746070000000i,throttling_throttled_time=0i,usage_percent=0.008710888610763454 1709569312000000000
docker_container_net,container_image=eclipse-mosquitto,container_name=crazy_austin,container_status=running,container_version=unknown,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=ryzen,host=ryzen,maintainer=Roger\ Light\ <roger@atchoo.org>,network=eth0,server_version=25.0.3 tx_bytes=0i,rx_bytes=6246i,rx_errors=0i,tx_packets=0i,tx_dropped=0i,tx_errors=0i,rx_dropped=0i,rx_packets=47i,container_id="d1b3f3c308d1ffb6bf753df65f77877ccb062be9a0201ca97a67c53b22bbfd1a" 1709569312000000000
3

I knew that the question would appear, but I do not really know. I set up the dashboard for docker and I saw the values. I had two docker systems, perhaps the values came from the other deactivated one… I also changed to docker compose. And yes, I am sure, that container are running and using memory.

Current log

image
image1267×270 15.3 KB

Here are the results running with --test

> mem,host=myserver active=2295705600i,available=3075436544i,available_percent=71.60558700561523,buffered=0i,cached=1474510848i,commit_limit=16858673152i,committed_as=11122188288i,dirty=0i,free=1600925696i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,inactive=284532736i,low_free=0i,low_total=0i,mapped=0i,page_tables=91512832i,shared=19120128i,slab=0i,sreclaimable=0i,sunreclaim=0i,swap_cached=0i,swap_free=536870912i,swap_total=536870912i,total=4294967296i,used=1219530752i,used_percent=28.394412994384766,vmalloc_chunk=0i,vmalloc_total=35184372087808i,vmalloc_used=80584704i,write_back=0i,write_back_tmp=0i 1709572597000000000
> docker,engine_host=myserver,host=myserver,server_version=25.0.3 n_containers=13i,n_containers_paused=0i,n_containers_running=12i,n_containers_stopped=1i,n_cpus=2i,n_goroutines=154i,n_images=16i,n_listener_events=0i,n_used_file_descriptors=120i 1709572597000000000
> docker,engine_host=myserver,host=myserver,server_version=25.0.3 memory_total=4294967296i 1709572597000000000
> docker_container_status,com.docker.compose.config-hash=1c107cc22d921c7933bd37fa95afa4bb31b3a9660dbbb23486d0b394a9f3a0ba,com.docker.compose.container-number=1,com.docker.compose.depends_on=gitea-db:service_started:false,com.docker.compose.image=sha256:c24cd4cd2ad542b64d378495045b9ded9e9a3d1c5d55fd8f4103bc04f26b7de4,com.docker.compose.oneoff=False,com.docker.compose.project=gitea,com.docker.compose.project.config_files=/data/compose/44/docker-compose.yml,com.docker.compose.project.working_dir=/data/compose/44,com.docker.compose.replace=4c489cff626a566a393c6e00b71938697f8c8a2d99c6f35320136b10d93aa537,com.docker.compose.service=gitea,com.docker.compose.version=2.20.2,container_image=gitea/gitea,container_name=gitea,container_status=running,container_version=1.20,engine_host=myserver,host=myserver,maintainer=maintainers@gitea.io,org.opencontainers.image.created=2023-11-28T07:10:01.933Z,org.opencontainers.image.description=Git\ with\ a\ cup\ of\ tea!\ Painless\ self-hosted\ all-in-one\ software\ development\ service\,\ including\ Git\ hosting\,\ code\ review\,\ team\ collaboration\,\ package\ registry\ and\ CI/CD,org.opencontainers.image.licenses=MIT,org.opencontainers.image.revision=f831540ebdd79326b9761085f618f22fbd82acd3,org.opencontainers.image.source=https://github.com/go-gitea/gitea,org.opencontainers.image.title=gitea,org.opencontainers.image.url=https://github.com/go-gitea/gitea,org.opencontainers.image.version=1.20,server_version=25.0.3 container_id="dc437012fff7520e6ed7731c22e97bc2377067804a755b6e57f3f3b387ae01b7",exitcode=0i,finished_at=1709557566932699742i,oomkilled=false,pid=1677i,restart_count=0i,started_at=1709557589516223791i,uptime_ns=15008746660898i 1709572598000000000
> docker_container_mem,com.docker.compose.config-hash=1c107cc22d921c7933bd37fa95afa4bb31b3a9660dbbb23486d0b394a9f3a0ba,com.docker.compose.container-number=1,com.docker.compose.depends_on=gitea-db:service_started:false,com.docker.compose.image=sha256:c24cd4cd2ad542b64d378495045b9ded9e9a3d1c5d55fd8f4103bc04f26b7de4,com.docker.compose.oneoff=False,com.docker.compose.project=gitea,com.docker.compose.project.config_files=/data/compose/44/docker-compose.yml,com.docker.compose.project.working_dir=/data/compose/44,com.docker.compose.replace=4c489cff626a566a393c6e00b71938697f8c8a2d99c6f35320136b10d93aa537,com.docker.compose.service=gitea,com.docker.compose.version=2.20.2,container_image=gitea/gitea,container_name=gitea,container_status=running,container_version=1.20,engine_host=myserver,host=myserver,maintainer=maintainers@gitea.io,org.opencontainers.image.created=2023-11-28T07:10:01.933Z,org.opencontainers.image.description=Git\ with\ a\ cup\ of\ tea!\ Painless\ self-hosted\ all-in-one\ software\ development\ service\,\ including\ Git\ hosting\,\ code\ review\,\ team\ collaboration\,\ package\ registry\ and\ CI/CD,org.opencontainers.image.licenses=MIT,org.opencontainers.image.revision=f831540ebdd79326b9761085f618f22fbd82acd3,org.opencontainers.image.source=https://github.com/go-gitea/gitea,org.opencontainers.image.title=gitea,org.opencontainers.image.url=https://github.com/go-gitea/gitea,org.opencontainers.image.version=1.20,server_version=25.0.3 active_anon=0i,active_file=0i,container_id="dc437012fff7520e6ed7731c22e97bc2377067804a755b6e57f3f3b387ae01b7",inactive_anon=0i,inactive_file=0i,limit=0i,max_usage=0i,pgfault=0i,pgmajfault=0i,unevictable=0i,usage=0i,usage_percent=0 1709572598000000000
> docker_container_cpu,com.docker.compose.config-hash=1c107cc22d921c7933bd37fa95afa4bb31b3a9660dbbb23486d0b394a9f3a0ba,com.docker.compose.container-number=1,com.docker.compose.depends_on=gitea-db:service_started:false,com.docker.compose.image=sha256:c24cd4cd2ad542b64d378495045b9ded9e9a3d1c5d55fd8f4103bc04f26b7de4,com.docker.compose.oneoff=False,com.docker.compose.project=gitea,com.docker.compose.project.config_files=/data/compose/44/docker-compose.yml,com.docker.compose.project.working_dir=/data/compose/44,com.docker.compose.replace=4c489cff626a566a393c6e00b71938697f8c8a2d99c6f35320136b10d93aa537,com.docker.compose.service=gitea,com.docker.compose.version=2.20.2,container_image=gitea/gitea,container_name=gitea,container_status=running,container_version=1.20,cpu=cpu-total,engine_host=myserver,host=myserver,maintainer=maintainers@gitea.io,org.opencontainers.image.created=2023-11-28T07:10:01.933Z,org.opencontainers.image.description=Git\ with\ a\ cup\ of\ tea!\ Painless\ self-hosted\ all-in-one\ software\ development\ service\,\ including\ Git\ hosting\,\ code\ review\,\ team\ collaboration\,\ package\ registry\ and\ CI/CD,org.opencontainers.image.licenses=MIT,org.opencontainers.image.revision=f831540ebdd79326b9761085f618f22fbd82acd3,org.opencontainers.image.source=https://github.com/go-gitea/gitea,org.opencontainers.image.title=gitea,org.opencontainers.image.url=https://github.com/go-gitea/gitea,org.opencontainers.image.version=1.20,server_version=25.0.3 container_id="dc437012fff7520e6ed7731c22e97bc2377067804a755b6e57f3f3b387ae01b7",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=0i,usage_in_usermode=0i,usage_percent=0,usage_system=2095762880000000i,usage_total=0i 1709572598000000000
> docker_container_net,com.docker.compose.config-hash=1c107cc22d921c7933bd37fa95afa4bb31b3a9660dbbb23486d0b394a9f3a0ba,com.docker.compose.container-number=1,com.docker.compose.depends_on=gitea-db:service_started:false,com.docker.compose.image=sha256:c24cd4cd2ad542b64d378495045b9ded9e9a3d1c5d55fd8f4103bc04f26b7de4,com.docker.compose.oneoff=False,com.docker.compose.project=gitea,com.docker.compose.project.config_files=/data/compose/44/docker-compose.yml,com.docker.compose.project.working_dir=/data/compose/44,com.docker.compose.replace=4c489cff626a566a393c6e00b71938697f8c8a2d99c6f35320136b10d93aa537,com.docker.compose.service=gitea,com.docker.compose.version=2.20.2,container_image=gitea/gitea,container_name=gitea,container_status=running,container_version=1.20,engine_host=myserver,host=myserver,maintainer=maintainers@gitea.io,network=eth0,org.opencontainers.image.created=2023-11-28T07:10:01.933Z,org.opencontainers.image.description=Git\ with\ a\ cup\ of\ tea!\ Painless\ self-hosted\ all-in-one\ software\ development\ service\,\ including\ Git\ hosting\,\ code\ review\,\ team\ collaboration\,\ package\ registry\ and\ CI/CD,org.opencontainers.image.licenses=MIT,org.opencontainers.image.revision=f831540ebdd79326b9761085f618f22fbd82acd3,org.opencontainers.image.source=https://github.com/go-gitea/gitea,org.opencontainers.image.title=gitea,org.opencontainers.image.url=https://github.com/go-gitea/gitea,org.opencontainers.image.version=1.20,server_version=25.0.3 container_id="dc437012fff7520e6ed7731c22e97bc2377067804a755b6e57f3f3b387ae01b7",rx_bytes=1414058i,rx_dropped=0i,rx_errors=0i,rx_packets=7965i,tx_bytes=2253708i,tx_dropped=0i,tx_errors=0i,tx_packets=10293i 1709572598000000000
4

Those look good. Did test report that what you expected?

If so, I think I would check the database directly to see what is in there and if that looks good, move to your Grafana. Since you mentioned two deployments, see if your queries are hard-coded for only one of those deployments.

5

Unfortunately not. The plugin seams to have access to docker, but it still does not track mem values which are all 0 e.g. usage=0i, :pensive: and I don’t know why…

> docker_container_mem,
com.docker.compose.config-hash=1c107cc22d921c7933bd37fa95afa4bb31b3a9660dbbb23486d0b394a9f3a0ba,
com.docker.compose.container-number=1,
com.docker.compose.oneoff=False,
com.docker.compose.project=gitea,
com.docker.compose.service=gitea,
container_image=gitea/gitea,
container_name=gitea,
container_status=running,
container_version=1.20,
engine_host=myserver,
host=myserver,
server_version=25.0.3 active_anon=0i,
active_file=0i,
container_id="dc437012fff7520e6ed7731c22e97bc2377067804a755b6e57f3f3b387ae01b7",
inactive_anon=0i,
inactive_file=0i,
limit=0i,
max_usage=0i,
pgfault=0i,
pgmajfault=0i,
unevictable=0i,
usage=0i,
usage_percent=0 1709572598000000000
6

Ah now I see what you are saying. I saw the metrics existed, but didn’t verify the zero values.

Are you running this on a RaspberryPi? Or similar device? Becuase you are getting other data, it points to you not being in the memory cgroup.

7

The alpine system is running as a lxc container in proxmox 8.1.4.

What group or cgroup do you mean? The user telegraf is assinged to the groups docker and nogroup. The memory usage of the core system is tracked, but not the ones of the container…

8

cgroups are something different than Linux groups.

The memory usage of the core system is tracked, but not the ones of the container…

That points to there being an issue with your ability to monitor the container’s memory. Take a look at the docker stats docs, which reference that the metrics are pulled from the memory cgroup. It calls out that some distros may not enable this group.

We’ve seen a similar report where no memory data comes back on Raspberry PIs, which is why I asked what you were running on: Telegraf docker input reports 0 bytes memory usage on Raspberry Pi OS · Issue #8079 · influxdata/telegraf · GitHub

9

Thanks for the hint. I found a website from alpine talking about docker and cgroups. After I changed the settings I will drop a note.
https://wiki.alpinelinux.org/wiki/Docker

10

I am struggling with the cgroup configuration in Alpine/OpenRC. Up to now I am sure that this is the issue which does not provide the necessary values as they are also 0 in docker stats.
Currently cgroups V2 is activated and docker info shows that it is using cgroups V2. But it do not know how to delegate cpu, mem and io which should be necessary for the values. I open a ticket at Alpine.

11

Based on that wikipage you found it looks like the best way would be to edit the /etc/default/grub and modify the linux command line with:

GRUB_CMDLINE_LINUX_DEFAULT="... cgroup_enable=memory swapaccount=1"

Where the ... is the existing items on that line. Essentially adding cgroup_enable=memory swapaccount=1 to the line.

You may also need to run update-grub and then reboot for the changes to take effect.

Alternatively, use the extlinux option on that same wiki page, run update-extlinux, and reboot.

12

There is no folder /etc/default and no /boot where I can edit or drop a file. In /media are empty folders cdrom, floppy and usb. I cannot find the spot to define the delegation…

13

I’ve not run Alpine on baremetal before, but it looks like there are quite a few different bootloaders that you could have choosen from. Based on that list, you might need to look in /efi, does that exist?

14

I found the page, too. I checked all of them without success.

  • rEFInd not included in Alpine 3.19
  • Syslinux no directory /boot
  • EFI
    • no directory /boot and /mnt/EFI
    • efibootmgr not found
  • GRUB
    • no directory /etc/default/grub
    • grub-install not found

So far I am not able to add the cmdline parameter…

15

Proxmox itself recommends to create a VM instead of a LXC as docker is a container system like LXC.
So I will install docker in a VM and telegraf should show the expected results.

16

The stats including CPU and MEM are now visible in the VM.

1 Like