Hi everyone,
I’m having some trouble identifying the cause of the errors below. Maybe someone can help…
2025-07-31T00:36:03-03:00 E! [outputs.influxdb_v2] Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T00:36:03-03:00 E! [outputs.influxdb_v2] When writing to [https://127.0.0.1:8086/api/v2/write]: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T00:36:03-03:00 E! [agent] Error writing to outputs.influxdb_v2: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T01:21:03-03:00 E! [outputs.influxdb_v2] Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T01:21:03-03:00 E! [outputs.influxdb_v2] When writing to [https://127.0.0.1:8086/api/v2/write]: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T01:21:03-03:00 E! [agent] Error writing to outputs.influxdb_v2: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T02:51:03-03:00 E! [outputs.influxdb_v2] Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T02:51:03-03:00 E! [outputs.influxdb_v2] When writing to [https://127.0.0.1:8086/api/v2/write]: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T02:51:03-03:00 E! [agent] Error writing to outputs.influxdb_v2: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T04:03:03-03:00 E! [outputs.influxdb_v2] Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T04:03:03-03:00 E! [outputs.influxdb_v2] When writing to [https://127.0.0.1:8086/api/v2/write]: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T04:03:03-03:00 E! [agent] Error writing to outputs.influxdb_v2: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2025-07-31T04:12:03-03:00 E! [outputs.influxdb_v2] Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": net/http: cannot rewind body after connection loss
2025-07-31T04:12:03-03:00 E! [outputs.influxdb_v2] When writing to [https://127.0.0.1:8086/api/v2/write]: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": net/http: cannot rewind body after connection loss
2025-07-31T04:12:03-03:00 E! [agent] Error writing to outputs.influxdb_v2: Post "https://127.0.0.1:8086/api/v2/write?bucket=telegraf-metrics&org=MyORG": net/http: cannot rewind body after connection loss
These log messages started appearing after I upgraded Telegraf from version 1.35.1 to 1.35.2. Unfortunately, enabling debug mode didn’t help, no additional information was provided. InfluxDB2 doesn’t show any error messages either.
Is anyone else experiencing this? How can I debug issues like this?
Here is my Telegraf configuration:
[agent]
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
hostname = ""
interval = "60s"
log_with_timezone = "America/Sao_Paulo"
logfile = "/var/log/telegraf/telegraf.log"
logfile_rotation_interval = "24h"
logfile_rotation_max_archives = 7
metric_batch_size = 1000
metric_buffer_limit = 10000
omit_hostname = false
precision = ""
round_interval = true
skip_processors_after_aggregators = true
[[inputs.cpu]]
collect_cpu_time = false
fieldinclude = [ "usage_nice", "usage_system", "usage_user" ]
interval = "180s"
percpu = false
report_active = false
totalcpu = true
[[inputs.disk]]
fieldinclude = [ "free", "total", "used", "used_percent" ]
ignore_fs = [ "aufs", "devfs", "devtmpfs", "iso9660", "overlay", "squashfs", "tmpfs" ]
interval = "180s"
mount_points = [ "/", "/var/lib/influxdb", "/var/lib/loki" ]
tagexclude = [ "mode", "fstype" ]
[[inputs.diskio]]
devices = [ "sda", "sdb", "sdc" ]
fieldinclude = [ "io_time" ]
interval = "180s"
[[inputs.mem]]
fieldinclude = [ "available", "available_percent", "free", "cached", "total", "used", "used_percent" ]
interval = "180s"
[[inputs.net]]
interfaces = [ "enp4s0f0" ]
fieldexclude = [ "packets_recv", "packets_sent", "speed" ]
ignore_protocol_stats = true
interval = "180s"
[inputs.net.tagpass]
interface = [ "enp4s0f0" ]
[[inputs.processes]]
fieldinclude = [ "blocked", "idle", "running", "sleeping", "total", "total_threads", "zombies" ]
interval = "180s"
[[inputs.swap]]
fieldexclude = [ "in", "out" ]
interval = "180s"
[[inputs.system]]
fieldinclude = [ "load1", "load5", "load15", "n_cpus", "n_users", "uptime" ]
interval = "180s"
[[inputs.snmp_trap]]
service_address = "udp://192.168.1.1:162"
version = "3"
sec_name = "SECNAME"
sec_level = "authPriv"
auth_protocol = "SHA"
auth_password = "AUTHPASSWORD"
priv_protocol = "DES"
priv_password = "PRIVPASSWORD"
[[inputs.tail]]
files = ["/var/log/icinga2/icinga2.log"]
from_beginning = false
name_override = "log_icinga"
data_format = "grok"
grok_patterns = ['%{ICINGA_MAIN}']
grok_custom_patterns = '''
ICINGA_TIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE}
ICINGA_MAIN \[%{ICINGA_TIMESTAMP:timestamp:ts-"2006-01-02 15:04:05 -0700"}\] %{WORD:level:tag}/%{GREEDYDATA:message}
'''
[inputs.tail.tags]
log_name = "Icinga"
[[inputs.tail]]
files = ["/var/log/telegraf/telegraf.log"]
from_beginning = false
name_override = "log_telegraf"
data_format = "grok"
grok_patterns = ['%{TIMESTAMP_ISO8601:timestamp:ts-rfc3339} %{TELEGRAF_LOG_LEVEL:level:tag}! %{GREEDYDATA:message}']
grok_custom_patterns = '''
TELEGRAF_LOG_LEVEL (?:[DIWE]+)
'''
[inputs.tail.tags]
log_name = "Telegraf"
[[processors.enum]]
order = 1
namepass = [ "log_telegraf" ]
[[processors.enum.mapping]]
tags = [ "level" ]
[processors.enum.mapping.value_mappings]
"E" = "error"
"W" = "warning"
"D" = "debug"
"I" = "info"
[[processors.rename]]
order = 2
namepass = [ "log_icinga", "log_telegraf" ]
[[processors.rename.replace]]
measurement = "log_icinga"
dest = "log"
[[processors.rename.replace]]
measurement = "log_telegraf"
dest = "log"
[[processors.starlark]]
order = 3
namepass = [ "net" ]
source = '''
timeNanoToSeconds = 1000000000
maxVariationTime = 185
bytesToBits = 8
fieldsToDerivate = [
"bytes_recv",
"bytes_sent",
"drop_in",
"drop_out",
"err_in",
"err_out"
]
fieldsBytesToBits = [
"bytes_recv",
"bytes_sent"
]
state = {
}
def apply(metric):
if metric.tags.get("interface") != None:
lastMetric = state.get(metric.tags["interface"])
state[metric.tags["interface"]] = deepcopy(metric)
if lastMetric == None:
for field in fieldsToDerivate:
metric.fields.pop(field)
else:
currentMetricTime = metric.time / timeNanoToSeconds
lastMetricTime = lastMetric.time / timeNanoToSeconds
metricsTimeVariation = currentMetricTime - lastMetricTime
if (currentMetricTime < lastMetricTime) or (metricsTimeVariation > maxVariationTime):
for field in fieldsToDerivate:
metric.fields.pop(field)
else:
for field in fieldsToDerivate:
actualValue = metric.fields[field]
lastValue = lastMetric.fields[field]
if (actualValue < lastValue):
metric.fields.pop(field)
else:
metric.fields[field] = ((actualValue - lastValue) / metricsTimeVariation)
if field in fieldsBytesToBits:
metric.fields[field] = metric.fields[field] * bytesToBits
return metric
'''
[[outputs.influxdb_v2]]
bucket = "telegraf-logs"
namepass = [ "log" ]
organization = "MyORG"
token = "$INFLUX_TOKEN"
urls = ["https://127.0.0.1:8086"]
[[outputs.influxdb_v2]]
bucket = "telegraf-snmptraps"
namepass = [ "snmp_trap" ]
organization = "MyORG"
token = "$INFLUX_TOKEN"
urls = ["https://127.0.0.1:8086"]
[[outputs.influxdb_v2]]
bucket = "telegraf-metrics"
namepass = [ "cpu", "disk", "diskio", "mem", "net", "processes", "system", "swap" ]
organization = "MyORG"
token = "$INFLUX_TOKEN"
urls = ["https://127.0.0.1:8086"]
