Write error metrics

Telegraf
1

I have a Telegraf agent v1.34 deployed with the Influx helm chart v1.8.55. There is a syslog input plugin and several output plugins (pubsub, health and prometheus) as shown below:

[agent]
      buffer_strategy = "memory"
      collection_jitter = "0s"
      debug = true
      flush_interval = "20s"
      flush_jitter = "0s"
      hostname = "$HOSTNAME"
      interval = "10s"
      logfile = ""
      metric_batch_size = 1000
      metric_buffer_limit = 25000
      omit_hostname = false
      precision = ""
      quiet = false
      round_interval = true
    [[processors.enum]]
       [[processors.enum.mapping]]
        dest = "status_code"
        field = "status"
        [processors.enum.mapping.value_mappings]
            critical = 3
            healthy = 1
            problem = 2
    [[outputs.cloud_pubsub]]
      content_encoding = "identity"
      credentials_file = "/config/gcp/credentials.json"
      data_format = "json"
      namedrop = [
        "internal*"
      ]
      project = "xxxx"
      topic = "yyyyy"
    [[outputs.prometheus_client]]
      collectors_exclude = [
        "gocollector"
      ]
      listen = ":9273"
      path = "/metrics"

    [[outputs.health]]
      service_address = "http://:8888"
      namepass = ["internal_write"]
    [[outputs.health.compares]]
        field = "buffer_size"
        lt = 20000.0
    [[outputs.health.contains]]
        field = "buffer_size"
    [[inputs.syslog]]
      server = "udp://:514"
      syslog_standard = "RFC3164"
    [[inputs.internal]]
      collect_memstats = false

In normal conditions I can query Prometheus and retrieve input metrics (internal_gather_metrics_gathered) and output metrics (internal_write_metrics_written). However, I have a situation where the cloud_pubsub plugin errors:

2025-07-03T10:53:30Z E! [agent] Error writing to outputs.cloud_pubsub: rpc error: code = Unauthenticated desc = Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

When this issue happens I’m unable to retrieve error metrics from Prometheus, since all internal_write_errors,
internal_write_metrics_dropped and internal_write_metrics_rejected show zero results.

Is there any other metric that I should be using to retrieve output plugins errors?

2

@Santiago_Cardin Welcome to Influxdata Community!

The authentication error you’re seeing suggests the GCP credentials are invalid or expired. This type of error might not increment the typical “dropped” or “rejected” metrics because Telegraf considers it a transport/connection error rather than a data processing error.

For immediate troubleshooting of your authentication issue, verify that:

  1. The credentials file exists and is readable
  2. The service account has the necessary Pub/Sub permissions
  3. The credentials haven’t expired