Telegraf problems with secretstores.systemd: invalid header field value for "Authorization"

[reed-foster]

Hi,

I’ve set up an influxdb database, and would like to add telegraf on the same machine to push metrics to influxdb. The system is running nixos-unstable, and I’m using agenix for secret management.

However, when I try to start the telegraf service, it is able to load the configuration, but no traffic actually goes to the influxdb database, and the authentication fails. The telegraf log has the following entries:

Nov 22 09:33:55 nixos systemd[1]: Starting Telegraf Agent...
Nov 22 09:33:55 nixos systemd[1]: Started Telegraf Agent.
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! Loading config: /nix/store/5mx43rnag6h47kb8rghk6lydhggf467b-config.toml
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! Starting Telegraf 1.36.3 brought to you by InfluxData the makers of InfluxDB
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! Available plugins: 239 inputs, 9 aggregators, 35 processors, 26 parsers, 65 outputs, 6 secret-stores
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! Loaded inputs: cpu mem
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! Loaded aggregators:
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! Loaded processors:
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! Loaded secretstores: systemd
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! Loaded outputs: influxdb_v2
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! Tags enabled: host=nixos
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"nixos", Flush Interval:10s
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z W! [agent] The default value of 'skip_processors_after_aggregators' will change to 'true' with Telegraf v1.40.0! If you need the current default behavior, please explicitly set the option to 'false'!
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z D! [agent] Initializing plugins
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z D! [agent] Connecting outputs
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z D! [agent] Attempting connection to [outputs.influxdb_v2]
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z D! [agent] Successfully connected to outputs.influxdb_v2
Nov 22 09:33:55 nixos telegraf[50317]: 2025-11-22T14:33:55Z D! [agent] Starting service inputs
Nov 22 09:34:05 nixos telegraf[50317]: 2025-11-22T14:34:05Z E! [outputs.influxdb_v2] Post "http://localhost:8086/api/v2/write?bucket=default&org=org": net/http: invalid header field value for "Authorization"
Nov 22 09:34:05 nixos telegraf[50317]: 2025-11-22T14:34:05Z E! [outputs.influxdb_v2] When writing to [http://localhost:8086/api/v2/write]: Post "http://localhost:8086/api/v2/write?bucket=default&org=org": net/http: invalid header field value for "Authorization"
Nov 22 09:34:05 nixos telegraf[50317]: 2025-11-22T14:34:05Z D! [outputs.influxdb_v2] Buffer fullness: 8 / 10000 metrics
Nov 22 09:34:05 nixos telegraf[50317]: 2025-11-22T14:34:05Z E! [agent] Error writing to outputs.influxdb_v2: Post "http://localhost:8086/api/v2/write?bucket=default&org=org": net/http: invalid header field value for "Authorization"
Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z D! [agent] Stopping service inputs
Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z D! [agent] Input channel closed
Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z I! [agent] Hang on, flushing any cached metrics before shutdown
Nov 22 09:34:07 nixos systemd[1]: Stopping Telegraf Agent...
Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z E! [outputs.influxdb_v2] Post "http://localhost:8086/api/v2/write?bucket=default&org=org": net/http: invalid header field value for "Authorization"
Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z E! [outputs.influxdb_v2] When writing to [http://localhost:8086/api/v2/write]: Post "http://localhost:8086/api/v2/write?bucket=default&org=org": net/http: invalid header field value for "Authorization"
Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z D! [outputs.influxdb_v2] Buffer fullness: 8 / 10000 metrics
Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z E! [agent] Error writing to outputs.influxdb_v2: Post "http://localhost:8086/api/v2/write?bucket=default&org=org": net/http: invalid header field value for "Authorization"
Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z I! [agent] Stopping running outputs
Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z D! [agent] Stopped Successfully
Nov 22 09:34:07 nixos systemd[1]: telegraf.service: Deactivated successfully.
Nov 22 09:34:07 nixos systemd[1]: Stopped Telegraf Agent.

The relevant line seems to be

Nov 22 09:34:07 nixos telegraf[50317]: 2025-11-22T14:34:07Z E! [agent] Error writing to outputs.influxdb_v2: Post "http://localhost:8086/api/v2/write?bucket=default&org=org": net/http: invalid header field value for "Authorization"

When running wireshark, I don’t actually see any HTTP traffic on the loopback interface, which seems odd.

I checked to make sure that systemd can read the token with the same UID as the service is being run with:

nixos$ sudo systemd-run -P --wait -p LoadCredential=telegraf.influxdb_token:/run/agenix/influxdb-write-token systemd-creds --uid=256 cat telegraf.influxdb_token
Running as unit: run-p50553-i50554.service; invocation ID: 41b962071f33424f9bae46db2c680099
<contents of base64 token>
          Finished with result: success
Main processes terminated with: code=exited, status=0/SUCCESS
               Service runtime: 25ms
             CPU time consumed: 10ms
                   Memory peak: 2.2M (swap: 0B)

I have also confirmed that the token works if I pass it in plain-text to the token field in [[outputs.influxdb_v2]]. When I do that, telegraf is able to write to the influxdb database. This makes it seem like it’s maybe be an issue with how telegraf is getting the secret from the systemd secretstore?

Systemd unit generated from nixos configuration

$ systemd-analyze cat-config /etc/systemd/system/telegraf.service
/etc/systemd/system/telegraf.service → /nix/store/ly0di2ng0cfhk2nncby6sr0q2lj4w6aq-unit-telegraf.service/telegraf.service

[Unit]
After=network-online.target
Description=Telegraf Agent
Wants=network-online.target

[Service]
Environment=“LOCALE_ARCHIVE=/nix/store/7l6arlis4halhgnkw2mp5v4rsf2z3f1g-glibc-locales-2.40-66/lib/locale/locale-archive”
Environment=“PATH=/nix/store/xs8scz9w9jp4hpqycx3n3bah5y07ymgj-coreutils-9.8/bin:/nix/store/qqvfnxa9jg71wp4hfg1l63r4m78iwvl9-findutils-4.10.0/bin:/nix/store/22r4s6lqhl43jkazn51f3c18qwk894g4-gnugrep-3.12/bin:/nix/store/zppkx0lkizglyqa9h26wf495qkllrjgy-gnused-4.9/bin:/nix/store/f8plklbbq3gwkq1wfq89i3f4wy8rabzn-systemd-258/bin:/nix/store/xs8scz9w9jp4hpqycx3n3bah5y07ymgj-coreutils-9.8/sbin:/nix/store/qqvfnxa9jg71wp4hfg1l63r4m78iwvl9-findutils-4.10.0/sbin:/nix/store/22r4s6lqhl43jkazn51f3c18qwk894g4-gnugrep-3.12/sbin:/nix/store/zppkx0lkizglyqa9h26wf495qkllrjgy-gnused-4.9/sbin:/nix/store/f8plklbbq3gwkq1wfq89i3f4wy8rabzn-systemd-258/sbin”
Environment=“TZDIR=/nix/store/c6lkjqkkc4cl4pffj4i3l22rv4ihhpb9-tzdata-2025b/share/zoneinfo”
AmbientCapabilities=CAP_NET_RAW
ExecReload=/nix/store/xs8scz9w9jp4hpqycx3n3bah5y07ymgj-coreutils-9.8/bin/kill -HUP $MAINPID
ExecStart=/nix/store/zjadjpzyz9h8kdc0sk488zlh55dd2h9l-telegraf-1.36.3/bin/telegraf -config /nix/store/5mx43rnag6h47kb8rghk6lydhggf467b-config.toml
LoadCredential=telegraf.influxdb_token:/run/agenix/influxdb-write-token
Restart=on-failure
RuntimeDirectory=telegraf
User=telegraf
Group=telegraf

[Install]
WantedBy=multi-user.target

.toml configuration

[agent]
debug = true
interval = “10s”
quiet = false

[[inputs.cpu]]
collect_cpu_time = true
percpu = true
totalcpu = true

[[inputs.mem]]

[[outputs.influxdb_v2]]
bucket = “default”
organization = “org”
token = “@{systemd:influxdb_token}”
urls = [“http://localhost:8086”]

[[secretstores.systemd]]
id = “systemd”

Relevant nixos configuration:

{ config, lib, ... }:
{
  services.telegraf = {
    enable = true;
    extraConfig = {
      secretstores = {
        systemd = [{
          id = "systemd";
        }];
      };
      outputs = {
        influxdb_v2 = [{
          token = "@{systemd:influxdb_token}";
          organization = "org";
          bucket = "default";
          urls = [ "http://localhost:8086" ];
        }];
      };
      inputs = {
        cpu = [{
          percpu = true;
          totalcpu = true;
          collect_cpu_time = true;
        }];
        mem = [{ }];
      };
      agent = {
        interval = "10s";
        debug = true;
        quiet = false;
      };
    };
  };
  # https://fossies.org/linux/telegraf/plugins/secretstores/systemd/README.md
  systemd.services.telegraf = {
    serviceConfig = {
      LoadCredential = [ "telegraf.influxdb_token:${config.age.secrets.influxdb-write-token.path}" ];
    };
  };
  age.secrets.influxdb-write-token = {
    file = ../../secrets/influxdb-token.age;
    owner = "telegraf";
    group = "telegraf";
    mode = "600";
  };
}

Any suggestions? Is this the right way to use the systemd secretstore?

[reed-foster]

For anyone wondering, turns out the issue is that when saving the key with vim (my default editor), agenix stores the key with a newline at the end.

InfluxDBv2 authorization fails if docker secrets are used

I found the problem, it’s the token itself. Provide token via environment: docker compose file: secrets: telegraf_influxdb_token: environment: "TELEGRAF_TOKEN" results in: alexander@ubuntu:~/home-automation$ docker exec -it telegraf cat /run/secrets/telegraf_influxdb_token jCSAWc***Ong==alexander@ubuntu:~/home-automation$ Provide token via file: docker compose file: secrets: telegraf_influxdb_token: file: ./.secrets/telegraf_influxdb_token results in: alexander@ubuntu:~/ho…

I was able to resolve the issue by running executing the following in vim (found here)

:set binary
:set noeol
:set nofixeol
:set nofixendofline
:wq

If you’re using a different editor there are probably other options.