Telegraf does not send metrics to influxdb v2

Telegraf
,
1

The Telegraf container is not sending logs.

After starting Telegraf, after a random amount of time passes, no logs are sent to InfluxDB, as shown in the logs below (with debug options set to true). This issue persists until Telegraf is restarted.

However, after restarting the Telegraf container, the logs that weren’t being sent are sent all at once, and after a short period of time, the issue reoccurs.

Is there a good solution to these issues? If not, I would appreciate guidance on where to check.

{"log":"2024-10-07T00:00:08Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:00:08.322711524Z"}
{"log":"2024-10-07T00:00:21Z D! [outputs.influxdb_v2]  Wrote batch of 1 metrics in 9.385599ms\n","stream":"stderr","time":"2024-10-07T00:00:21.489248638Z"}
{"log":"2024-10-07T00:00:21Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:00:21.489288619Z"}
{"log":"2024-10-07T00:00:34Z D! [outputs.influxdb_v2]  Wrote batch of 75 metrics in 14.067423ms\n","stream":"stderr","time":"2024-10-07T00:00:34.493847334Z"}
{"log":"2024-10-07T00:00:34Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:00:34.493888864Z"}
{"log":"2024-10-07T00:00:48Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:00:48.436896812Z"}
{"log":"2024-10-07T00:01:00Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:01:00.409561694Z"}
{"log":"2024-10-07T00:01:13Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:01:13.108160651Z"}
{"log":"2024-10-07T00:01:26Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:01:26.397890144Z"}
{"log":"2024-10-07T00:01:38Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:01:38.697116053Z"}
{"log":"2024-10-07T00:01:49Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:01:49.764733952Z"}
{"log":"2024-10-07T00:02:01Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:02:01.177849134Z"}
{"log":"2024-10-07T00:02:13Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:02:13.531836105Z"}
{"log":"2024-10-07T00:02:27Z D! [outputs.influxdb_v2]  Buffer fullness: 0 / 50000 metrics\n","stream":"stderr","time":"2024-10-07T00:02:27.221216468Z"}

Here are my versions and docker compose config.

  • Telegraf 1.31.3 (on Debian GNU/Linux 12)
  • Docker 27.2.0
  • Docker Compose v2.29.2
services:
  telegraf:
    container_name: telegraf
    image: telegraf:latest
    env_file:
      - ./telegraf/telegraf_env
    environment:
      HOST_PROC: /rootfs/proc
      HOST_SYS: /rootfs/sys
      HOST_ETC: /rootfs/etc
    hostname: ${APP_NAME}
    volumes:
      - ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf
      - /var/run/docker.sock:/var/run/docker.sock
      - /sys:/rootfs/sys:ro
      - /proc:/rootfs/proc:ro
      - /etc:/rootfs/etc:ro
    links:
      - application

  application:
    container_name: ${APP_NAME}
    image: ${APP_NAME}
    init: true
    ports:
      - "9999:9999"
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "15g"

The batch size and buffer limit were increased for buffer overflow issues.

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 5000
  metric_buffer_limit = 50000
  collection_jitter = "0s"
  flush_interval = "11s"
  flush_jitter = "3s"
  precision = ""
  debug = true
2

Please share you (redacted) Telegraf config!

3

Thank you for reply. Here is my config.

telegraf/telegraf.conf

[global_tags]

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 5000
  metric_buffer_limit = 50000
  collection_jitter = "0s"
  flush_interval = "11s"
  flush_jitter = "3s"
  precision = ""
  debug = true
  hostname = "${INFLUX_HOSTNAME}"
  omit_hostname = false

[[outputs.influxdb_v2]]
   urls = ["${INFLUX_URL}"]
   token = "${INFLUX_TOKEN}"
   organization = "${INFLUX_ORG}"
   bucket = "${INFLUX_BUCKET}"

[[inputs.docker_log]]
 endpoint = "unix:///var/run/docker.sock"
 from_beginning = false
 timeout = "5s"
 container_name_include = ["${CONTAINER_NAME}"]
 source_tag = false
 insecure_skip_verify = false

telegraf/telegraf_env

INFLUX_URL={influx-url}
INFLUX_TOKEN={influx-token}
INFLUX_ORG={influx-org}
INFLUX_BUCKET={influx-bucket}