InfluxDBv2 authorization fails if docker secrets are used

alexander-zimmermann · January 15, 2024, 8:44am

Hi folks,

as soon as I switch to docker secrets (file), telegraf authorization to influx fails. If I hardcode the token into the config file, it works.

Bind-mount of docker secret is (IMO) correct:

alexander@ubuntu:~/home-automation$ docker exec -it telegraf sh
# telegraf --version
Telegraf 1.29.2 (git: HEAD@d92d7073)
# cat /run/secrets/telegraf_influxdb_token
jCSA****Ong==
#

Relevant parts of telegraf config:

# Secret-store to access Docker Secrets
[[secretstores.docker]]
  ## Unique identifier for the secretstore.
  ## This id can later be used in plugins to reference the secrets
  ## in this secret-store via @{<id>:<secret_key>} (mandatory)
  id = "docker_secretstore"

and

# Configuration for sending metrics to InfluxDB 2.0
[[outputs.influxdb_v2]]
  urls = ["http://influx.zimmermann.XXX:8086"]
  token = "@{docker_secretstore:telegraf_influxdb_token}"
  #token = "jCSA****Ong=="
  organization = "zimmermann.XXX"
  bucket = "telegraf/autogen"

Any idea?

jpowers · January 16, 2024, 3:29pm

Hi,

as soon as I switch to docker secrets (file), telegraf authorization to influx fails. If I hardcode the token into the config file, it works.

This generally means one of few things:

Something is misconfigured
The secret store is not actually readable by the user running telegraf
The password is wrong/has special characters in that are not escaped

Based on what you copied and pasted the config looks fine, so no typo.

What user is running telegraf? Is this as a service?

alexander-zimmermann · January 16, 2024, 9:33pm

alexander@ubuntu:~$ docker exec -it telegraf bash
root@telegraf:/# ps auxx
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
telegraf       1  0.2  0.7 5647052 133932 ?      SLsl Jan15   4:06 telegraf --config=/etc/telegraf/telegraf.conf --config-directory=/etc/telegraf/telegraf.d
root          25  0.1  0.0   4608  3564 pts/0    Ss   22:22   0:00 bash
root          31  0.0  0.0   8480  4220 pts/0    R+   22:22   0:00 ps auxx
root@telegraf:/# ls -la /var/run/secrets/telegraf_influxdb_token 
-rw-rw-r-- 1 1000 1000 89 Jan 14 12:22 /var/run/secrets/telegraf_influxdb_token
root@telegraf:/#

Hmm, telegraf user should have access, but …

root@telegraf:~# su telegraf -c cat /var/run/secrets/telegraf_influxdb_token
root@telegraf:~#

I don’t get it

jpowers · January 16, 2024, 10:27pm

What are the permissions on the folder above?

alexander-zimmermann · January 17, 2024, 7:43am

alexander@ubuntu:~/home-automation**$ docker exec -it telegraf bash

root@telegraf:/# ls -la /var/run/secrets
total 12
drwxr-xr-x 2 root root 4096 Jan 15 16:52 .
drwxr-xr-x 1 root root 4096 Jan 15 16:52 ..
-rw-rw-r-- 1 1000 1000 89 Jan 14 12:22 telegraf_influxdb_token

root@telegraf:/# ls -la /var/run/
total 16
drwxr-xr-x 1 root root 4096 Jan 15 16:52 .
drwxr-xr-x 1 root root 4096 Jan 15 16:52 ..
drwxrwxrwt 2 root root 4096 Jan 10 01:00 lock
drwxr-xr-x 2 root root 4096 Jan 15 16:52 secrets

root@telegraf:/# ls -la /var/
total 44
drwxr-xr-x 1 root root 4096 Jan 10 01:00 .
drwxr-xr-x 1 root root 4096 Jan 15 16:52 ..
drwxr-xr-x 2 root root 4096 Dec 9 22:08 backups
drwxr-xr-x 1 root root 4096 Jan 10 01:00 cache
drwxr-xr-x 1 root root 4096 Jan 11 16:26 lib
drwxrwsr-x 2 root staff 4096 Dec 9 22:08 local
lrwxrwxrwx 1 root root 9 Jan 10 01:00 lock -> /run/lock
drwxr-xr-x 1 root root 4096 Jan 11 16:26 log
drwxrwsr-x 2 root mail 4096 Jan 10 01:00 mail
drwxr-xr-x 2 root root 4096 Jan 10 01:00 opt
lrwxrwxrwx 1 root root 4 Jan 10 01:00 run -> /run
drwxr-xr-x 2 root root 4096 Jan 10 01:00 spool
drwxrwxrwt 2 root root 4096 Dec 9 22:08 tmp

alexander-zimmermann · January 17, 2024, 8:05am

I changed /etc/passwd for telegraf user from /bin/false to /bin/bash/ as login shell and run:

root@telegraf:/run/secrets# su telegraf
telegraf@telegraf:/run/secrets$ cat telegraf_influxdb_token
jCSAWc***

Access should not be a problem, IMO. Let me double check the token a third time…

alexander-zimmermann · January 19, 2024, 9:10am

I found the problem, it’s the token itself.

Provide token via environment:

docker compose file:

secrets:
  telegraf_influxdb_token:
    environment: "TELEGRAF_TOKEN"

results in:

alexander@ubuntu:~/home-automation$ docker exec -it telegraf cat /run/secrets/telegraf_influxdb_token
jCSAWc***Ong==alexander@ubuntu:~/home-automation$

Provide token via file:

docker compose file:

secrets:
  telegraf_influxdb_token:
    file: ./.secrets/telegraf_influxdb_token

results in:

alexander@ubuntu:~/home-automation$ docker exec -it telegraf cat /run/secrets/telegraf_influxdb_token
jCSAWc***Ong==
alexander@ubuntu:~/home-automation$

The new line at the end is the problem. However my PWD file doesn’t contain any new line:

alexander@ubuntu:~/home-automation$ cat -A .secrets/telegraf_influxdb_token 
jCSAW***Ong==$
alexander@ubuntu:~/home-automation$

I believe that is a bug in how telegraf reads the file. Google a bit, I found “Trim contents of slack api urls from files by srhb · Pull Request #2929 · prometheus/alertmanager · GitHub”

Different SW, different use case, but same problem.

Any thoughts?

jpowers · January 19, 2024, 1:53pm

However my PWD file doesn’t contain any new line:

I’m not following. You said the newline is the problem, but that the file does not have a newline

Instead of cat -A $file, can you show the output of od -x $file

How is your file different than our test cases:

srebhan · January 19, 2024, 3:39pm

@alexander-zimmermann you show above that with

secrets:
  telegraf_influxdb_token:
    file: ./.secrets/telegraf_influxdb_token

there is a newline in the container’s secret, while there is none with the environment-variable setup.

Telegraf can only read the secret-file and use as-is! We should not mess with the content of the secret and we don’t. So if there is a newline at the end of the secret-file you will get a Telegraf secret with a newline at the end!

If your ./.secrets/telegraf_influxdb_token file does not contain a trailing newline, this clearly is an issue with docker or your configuration and should be fixed there.

Did you, by chance, echo the token into the file on your host? If so, did you use -n?

alexander-zimmermann · January 19, 2024, 4:32pm

alexander@ubuntu:~$ od -x home-automation/.secrets/telegraf_influxdb_token 
0000000 436a 4153 6357 3455 6b62 4878 705f 6954
***
0000120 756c 4f74 676e 3d3d 000a
0000131

alexander@ubuntu:~$ docker exec -it telegraf od -x /var/run/secrets/telegraf_influxdb_token
0000000 436a 4153 6357 3455 6b62 4878 705f 6954
***
0000120 756c 4f74 676e 3d3d
0000130

alexander@ubuntu:~/home-automation$ docker exec -it telegraf od -x /var/run/secrets/telegraf_influxdb_token
0000000 436a 4153 6357 3455 6b62 4878 705f 6954
***
0000120 756c 4f74 676e 3d3d 000a
0000131

First run is the host, second run is the secret mapping via environment, third one via file.

@srebhan Did you agree that docker has no bug since it maps exactly the file from the host?

alexander-zimmermann · January 19, 2024, 4:39pm

@srebhan not sure if I understand you correctly. Do you mean the token generation in general?

I generated the token at influx, copy it to clipboard, open vim and paste it, for both the token file ~/home-automation/.secrets/telegraf_influxdb_token and the environment variable in ~/home-automation/.env

srebhan · January 19, 2024, 4:44pm

Did you notice the 0a at the end of the data when not mapped via environment? Guess what that is!

srebhan · January 19, 2024, 4:50pm

I’m pretty sure that vim will add a newline to a text-file… linux - Why is Vim adding a newline? Is this a convention? - Super User

alexander-zimmermann · January 19, 2024, 5:10pm

That’s correct. I also found it out in the meantime.

However, see here: Why Should Text Files End With a Newline in Linux, Section 3.

" POSIX defines a line as a possibly empty sequence of non-newline characters, terminating in a newline, also called EOL (End-Of-Line), ASCII code 0x0A. Meanwhile, a text file is defined as consisting of lines.

Thus, the last line of a file, by definition and standard, should conclude that file with EOL."

I also checked the POSIX standard by myself now. It’s correct, a POSIX conform file must be terminated with a newline.

Sure, I can now tweak my file to be none POSIX conform, but I would be much better to change the telegraf code.

Agree?

srebhan · January 19, 2024, 5:41pm

Then how do we know it is a text file? What if someone passes a binary AES key in there? Should we just remove random parts of that file?

srebhan · January 19, 2024, 5:44pm

Please check Manage sensitive data with Docker secrets | Docker Docs. To cite:

[…]

Generic strings or binary content (up to 500 kb in size)

alexander-zimmermann · January 20, 2024, 10:34am

@srebhan All good questions and I don’t have a finale answer on that. Let’s do the following (suggestion): we enhance the documentation on GitHub and on the webpage and clearly state that telegraf handle/interprate all docker secrets as a binary file even if the file on the host was txt file. So the user should take care that file should not contain a \n at the end.

BTW: why we don’t have an issue w/ influx itself? All tokens work fine. All my tokens have a \n. What is the difference?

Jason_Stirnaman · January 24, 2024, 5:56pm

@alexander-zimmermann

Regarding docs, I agree. This looks like a common and easy mistake. I see many similar reports across projects.

Please submit an issue or PR to GitHub - influxdata/docs-v2: InfluxData Documentation that covers InfluxDB Cloud, InfluxDB OSS 2.x, InfluxDB OSS 1.x, InfluxDB Enterprise, Telegraf, Chronograf, Kapacitor, and Flux. and I’ll be happy to review or work on it for docs.influxdata.com and Telegraf repo docs.

All my tokens have a \n.

Where? For example, in a .env file, variables are key-value pairs; = and the line terminating character are used as delimiters for the value. The token doesn’t contain a new-line character.

srebhan · January 24, 2024, 8:37pm

@alexander-zimmermann and @Jason_Stirnaman even though, this is not a Telegraf specific issue I would appreciate a PR enhancing the documentation to make people aware of this behavior!

Topic		Replies	Views
Telegraf Docker Container and env file (/etc/default/telegraf) Telegraf influxdb , telegraf , docker	5	1847	June 1, 2021
Telegraft unauthorized to connect - even with a token Telegraf influxdb , telegraf	3	747	October 7, 2021
InfluxDB 2 / Telegraf "authorization not found" [Docker/Windows] InfluxDB 2 telegraf , windows , docker	1	3649	April 20, 2021
Influx v2 - telegraf configuration on docker Welcome & Getting Started telegraf	8	6968	February 5, 2021
InfluxDB 2, Telegraf on Docker with HTTPS InfluxDB 2	0	2971	January 3, 2020

InfluxDBv2 authorization fails if docker secrets are used

Related Topics