Telegraf context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Telegraf
1

Hi . I have Problem telegraf can not write metrics to influxdb2 , Below is log message and enable debug = true . Please help.

This my telegraf.conf .

 cat  telegraf.conf |grep -v '#'

[agent]
  interval = "10s"
  round_interval = true

  metric_batch_size = 1000

  metric_buffer_limit = 10000

  collection_jitter = "0s"

  flush_interval = "10s"
  flush_jitter = "0s"

  precision = ""

  debug = true







  hostname = ""
  omit_hostname = false
[[outputs.influxdb_v2]]
  urls = ["http://influxdb2:8086"]

  token = "xbtHh1lLMPe-hdEuRNbnM-12S-UyirFZbQsjebRJBWhUGYxgQISoHbhJ9bwFoYd8kK44do72EtSQyAYnwxqzBg==******"

  organization = "monitor"

  bucket = "Test_manual"









  insecure_skip_verify = false
[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.diskio]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]


[root@host2telegraf]# telegraf
2022-07-07T08:54:19Z I! Using config file: /etc/telegraf/telegraf.conf
2022-07-07T08:54:19Z I! Starting Telegraf 1.22.4
2022-07-07T08:54:19Z I! Loaded inputs: cpu disk diskio mem net processes swap system
2022-07-07T08:54:19Z I! Loaded aggregators:
2022-07-07T08:54:19Z I! Loaded processors:
2022-07-07T08:54:19Z I! Loaded outputs: influxdb_v2
2022-07-07T08:54:19Z I! Tags enabled: host=host2telegraf
2022-07-07T08:54:19Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"host2telegraf", Flush Interval:10s
2022-07-07T08:54:19Z D! [agent] Initializing plugins
2022-07-07T08:54:19Z D! [agent] Connecting outputs
2022-07-07T08:54:19Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-07-07T08:54:19Z D! [agent] Successfully connected to outputs.influxdb_v2
2022-07-07T08:54:19Z D! [agent] Starting service inputs
2022-07-07T08:54:29Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-07-07T08:54:44Z E! [outputs.influxdb_v2] When writing to [http://10.209.6.197:8086]: Post "http://100.209.4.197:8086/api/v2/write?bucket=Test_manual&org=aiops": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-07-07T08:54:44Z D! [outputs.influxdb_v2] Buffer fullness: 125 / 10000 metrics
2022-07-07T08:54:44Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2022-07-07T08:54:54Z E! [outputs.influxdb_v2] When writing to [http://10.209.6.197:8086]: Post "http://100.209.4.197:8086/api/v2/write?bucket=Test_manual&org=aiops": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-07-07T08:54:54Z D! [outputs.influxdb_v2] Buffer fullness: 192 / 10000 metrics
2022-07-07T08:54:54Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2022-07-07T08:55:04Z E! [outputs.influxdb_v2] When writing to [http://10.209.6.197:8086]: Post "http://10.209.6.197:8086/api/v2/write?bucket=Test_manual&org=aiops": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-07-07T08:55:04Z D! [outputs.influxdb_v2] Buffer fullness: 259 / 10000 metrics
2022-07-07T08:55:04Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2022-07-07T08:55:14Z E! [outputs.influxdb_v2] When writing to [http://10.209.6.197:8086]: Post "http://10.209.6.197:8086/api/v2/write?bucket=Test_manual&org=aiops": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-07-07T08:55:14Z D! [outputs.influxdb_v2] Buffer fullness: 326 / 10000 metrics
2022-07-07T08:55:14Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2022-07-07T08:55:24Z E! [outputs.influxdb_v2] When writing to [http://10.209.6.197:8086]: Post "http://10.209.6.197:8086/api/v2/write?bucket=Test_manual&org=aiops": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-07-07T08:55:24Z D! [outputs.influxdb_v2] Buffer fullness: 393 / 10000 metrics
2022-07-07T08:55:24Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2022-07-07T08:55:34Z E! [outputs.influxdb_v2] When writing to [http://10.209.6.197:8086]: Post "http://10.209.6.197:8086/api/v2/write?bucket=Test_manual&org=aiops": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-07-07T08:55:34Z D! [outputs.influxdb_v2] Buffer fullness: 460 / 10000 metrics
2022-07-07T08:55:34Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2022-07-07T08:55:44Z E! [outputs.influxdb_v2] When writing to [http://10.209.6.197:8086]: Post "http://10.209.6.197:8086/api/v2/write?bucket=Test_manual&org=aiops": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-07-07T08:55:44Z D! [outputs.influxdb_v2] Buffer fullness: 527 / 10000 metrics
2022-07-07T08:55:44Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2022-07-07T08:55:54Z E! [outputs.influxdb_v2] When writing to [http://10.209.6.197:8086]: Post "http://10.209.6.197:8086/api/v2/write?bucket=Test_manual&org=aiops": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-07-07T08:55:54Z D! [outputs.influxdb_v2] Buffer fullness: 594 / 10000 metrics
2022-07-07T08:55:54Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)

Thank you.

2

Client.Timeout exceeded while awaiting headers is a common network error, it means that the client can’t reach the server for whatever reason (DNS, firewall, etc).

3

They’re both in the same vlan so there’s no firewall . And we don’t use DNS.

4

if you test the connectivity to that port using telnet (or whatever your OS has) I expect the very same error.

5

Hi @Atech

Did you find solution for this problem ?