I have a Telegraf agent v1.34 deployed with the Influx helm chart v1.8.55. There is a syslog input plugin and several output plugins (pubsub, health and prometheus) as shown below:
[agent]
buffer_strategy = "memory"
collection_jitter = "0s"
debug = true
flush_interval = "20s"
flush_jitter = "0s"
hostname = "$HOSTNAME"
interval = "10s"
logfile = ""
metric_batch_size = 1000
metric_buffer_limit = 25000
omit_hostname = false
precision = ""
quiet = false
round_interval = true
[[processors.enum]]
[[processors.enum.mapping]]
dest = "status_code"
field = "status"
[processors.enum.mapping.value_mappings]
critical = 3
healthy = 1
problem = 2
[[outputs.cloud_pubsub]]
content_encoding = "identity"
credentials_file = "/config/gcp/credentials.json"
data_format = "json"
namedrop = [
"internal*"
]
project = "xxxx"
topic = "yyyyy"
[[outputs.prometheus_client]]
collectors_exclude = [
"gocollector"
]
listen = ":9273"
path = "/metrics"
[[outputs.health]]
service_address = "http://:8888"
namepass = ["internal_write"]
[[outputs.health.compares]]
field = "buffer_size"
lt = 20000.0
[[outputs.health.contains]]
field = "buffer_size"
[[inputs.syslog]]
server = "udp://:514"
syslog_standard = "RFC3164"
[[inputs.internal]]
collect_memstats = false
In normal conditions I can query Prometheus and retrieve input metrics (internal_gather_metrics_gathered) and output metrics (internal_write_metrics_written). However, I have a situation where the cloud_pubsub plugin errors:
2025-07-03T10:53:30Z E! [agent] Error writing to outputs.cloud_pubsub: rpc error: code = Unauthenticated desc = Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
When this issue happens I’m unable to retrieve error metrics from Prometheus, since all internal_write_errors,
internal_write_metrics_dropped and internal_write_metrics_rejected show zero results.
Is there any other metric that I should be using to retrieve output plugins errors?