Telegraf on kubernetes: environment variable for influxdb2 output token not working

Telegraf
1

I have installed Telegraf on a kubernetes cluster and have been testing the use of environment variables in the telegraf configuration file, via the “envFromSecret” setting in the helm chart. In the kubernetes secret referenced by envFromSecret I have configured 2 keys MQTT_SERVER_HOST and INFLUXDB2_TOKEN_DEV. The Telegraf configuration file contains the following settings:

...
[[outputs.influxdb_v2]]
  urls = ["http://influxdb2"]
  token = "${INFLUXDB2_TOKEN_DEV}"
  organization = "dev-customer-1"
  tagexclude = ["org"]
  bucket_tag = "bucket"
  exclude_bucket_tag = true
  [outputs.influxdb_v2.tagpass]
    org = ["dev-customer-1"]

[[inputs.mqtt_consumer]]
  servers = ["${MQTT_SERVER_HOST}"]
  ...

With this configuration the mqtt consumer connects correctly to the configured server, but the output fails with a 401 unauthorized error. If I remove the environment variable for the output token and use the raw token value it works ok. I have triple checked the token, even extracted the token from the secret and copied it to the config file (which works). Any ideas why the token environment variable wouldn’t work in this configuration?

2

Hi,

It sounds like the environment is not getting the variables set for some reason.

I have not used this feature but was looking at the original PR, which shows these values getting defined in an env section as well. Do you have something similar?

Thanks!

3

Thanks @jpowers I believe you can use both the env and envFromSecret sections for different variables. I’m using both, I set the HOST variable in env and the 2 variables mentioned above in the secret referenced in envFromSecret. What’s unusual is that it is picking up one of the variables from the secret (MQTT_SERVER_HOST) but not the token value for influxdb2. I just realized while writing this that I might be able to echo the token variable directly in the pod via kubectl exec so I will try that to see whether the variable is set. If the variable is set it must be something with the telegraf.conf that doesn’t correctly pull that value for some reason. Will let you know what I find out.

4

Update: running the command:

printenv INFLUXDB2_TOKEN_DEV

directly on the telegraf pod correctly returns the token. So the envFromSecret part is working, but including the token variable in the telegraf.conf as shown in my first post does not appear to send the correct value. Any help appreciated.

5

Does anyone have any ideas on why environment variables don’t populate the token value for influxdb2 output plugin (but do populate other properties)?

6

I’m not 100% sure how I fixed this, but I think it has something to do with the way I was retrieving and parsing the token returned from influxdb2 cli. The cli output is json and I’m using jq to parse it. After adding the -r (return raw output) flag to jq I noticed the problem stopped happening. Hope this is useful for someone else. From influxdb2 running on a k8s pod, here’s the bash script line to generate and parse the token and set the env variable before adding it to a generic secret (for use in envFromSecret):

PW_INFLUXDB2_TELEGRAF_TOKEN=$(kubectl exec -n packwise influxdb2-0 -- influx auth create --write-buckets -org $PW_INITIAL_ORG_NAME --json | jq -r '.token')
1 Like