Telegraf agent not starting inputs.tail expect when starting from inside of docker container

Telegraf
, , ,
1

Hello, I tried to setup telegraf with docker with the tail plugin enabled.
My telegraf.conf looks like this:

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000

[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = "TOKEN"
  organization = "ORG"
  bucket = "BUCKET"

[[inputs.tail]]
  files = ["/var/log/syslog-ng/192.168.10.1.log"]
  from_beginning = true
  data_format = "grok"
  grok_patterns = ['%{TIMESTAMP_ISO8601:timestamp} %{IPV4:host} %{DATA:topics} %{GREEDYDATA:message}']

  [inputs.tail.tags]
    logsource = "logsource"

my docker-compose.yml

  telegraf:
    image: telegraf:1.29-alpine
    container_name: telegraf
    command: -debug
    volumes:
      - ./data/telegraf/config/telegraf.conf:/etc/telegraf/telegraf.conf
      - ./data/syslog-ng/log:/var/log/syslog-ng

docker logs -f telegraf output:

2024-07-23T09:51:14Z I! Loading config: /etc/telegraf/telegraf.conf
2024-07-23T09:51:14Z I! Starting Telegraf 1.29.5 brought to you by InfluxData the makers of InfluxDB
2024-07-23T09:51:14Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
2024-07-23T09:51:14Z I! Loaded inputs: tail
2024-07-23T09:51:14Z I! Loaded aggregators:
2024-07-23T09:51:14Z I! Loaded processors:
2024-07-23T09:51:14Z I! Loaded secretstores:
2024-07-23T09:51:14Z I! Loaded outputs: influxdb_v2
2024-07-23T09:51:14Z I! Tags enabled: host=d6eaa952a62d
2024-07-23T09:51:14Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"d6eaa952a62d", Flush Interval:10s
2024-07-23T09:51:14Z D! [agent] Initializing plugins
2024-07-23T09:51:14Z D! [agent] Connecting outputs
2024-07-23T09:51:14Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2024-07-23T09:51:14Z D! [agent] Successfully connected to outputs.influxdb_v2
2024-07-23T09:51:14Z D! [agent] Starting service inputs
2024-07-23T09:51:24Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-07-23T09:51:34Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-07-23T09:51:44Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-07-23T09:51:54Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-07-23T09:52:04Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

and no metrics are pushed in the bucket on influxdb.

When I enter the container docker exec -it telegraf sh and execute telegraf -debug it reads the conf complete and started the tail plugin:

2024-07-23T09:54:42Z I! Loading config: /etc/telegraf/telegraf.conf
2024-07-23T09:54:42Z I! Starting Telegraf 1.29.5 brought to you by InfluxData the makers of InfluxDB
2024-07-23T09:54:42Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
2024-07-23T09:54:42Z I! Loaded inputs: tail
2024-07-23T09:54:42Z I! Loaded aggregators:
2024-07-23T09:54:42Z I! Loaded processors:
2024-07-23T09:54:42Z I! Loaded secretstores:
2024-07-23T09:54:42Z I! Loaded outputs: influxdb_v2
2024-07-23T09:54:42Z I! Tags enabled: host=d6eaa952a62d
2024-07-23T09:54:42Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"d6eaa952a62d", Flush Interval:10s
2024-07-23T09:54:42Z D! [agent] Initializing plugins
2024-07-23T09:54:42Z D! [agent] Connecting outputs
2024-07-23T09:54:42Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2024-07-23T09:54:42Z D! [agent] Successfully connected to outputs.influxdb_v2
2024-07-23T09:54:42Z D! [agent] Starting service inputs
2024-07-23T09:54:42Z D! [inputs.tail] Tail added for "/var/log/syslog-ng/192.168.10.1.log"
2024-07-23T09:54:42Z D! [outputs.influxdb_v2] Wrote batch of 1000 metrics in 82.347552ms
2024-07-23T09:54:42Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-07-23T09:54:42Z D! [outputs.influxdb_v2] Wrote batch of 1000 metrics in 15.768439ms
2024-07-23T09:54:42Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2024-07-23T09:54:42Z D! [outputs.influxdb_v2] Wrote batch of 1000 metrics in 30.661502ms
2024-07-23T09:54:42Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

When started with docker compose up -d the following entry is missing:
[inputs.tail] Tail added for "/var/log/syslog-ng/192.168.10.1.log"

Anyone know my mistake?

2

Given it can work when you manually start, this sounds like permissions or the file is misisng when telegraf starts.

When you run by hand you are running as root, but as a service, telegraf runs as the telegraf user/group.

1 Like