# InfluxDBv2 authorization fails if docker secrets are used

**URL:** <https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809>\
**Category:** Telegraf\
**Tags:** influxdb, docker\
**Created:** [January 15, 2024, 8:44am UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809 "2024-01-15T08:44:30Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![alexander-zimmermann](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/alexander-zimmermann/32/14909_2.png) [@alexander-zimmermann](https://community.influxdata.com/u/alexander-zimmermann)\
**Post date:** [January 15, 2024, 8:44am UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/1 "2024-01-15T08:44:30Z")

</div>

Hi folks,

as soon as I switch to docker secrets (file), telegraf authorization to influx fails. If I hardcode the token into the config file, it works.

Bind-mount of docker secret is (IMO) correct:

```auto
alexander@ubuntu:~/home-automation$ docker exec -it telegraf sh
# telegraf --version
Telegraf 1.29.2 (git: HEAD@d92d7073)
# cat /run/secrets/telegraf_influxdb_token
jCSA **** Ong==
#

```

Relevant parts of telegraf config:

```auto
# Secret-store to access Docker Secrets
[[secretstores.docker]]
  ## Unique identifier for the secretstore.
  ## This id can later be used in plugins to reference the secrets
  ## in this secret-store via @{<id>:<secret_key>} (mandatory)
  id = "docker_secretstore"

```

and

```auto
# Configuration for sending metrics to InfluxDB 2.0
[[outputs.influxdb_v2]]
  urls = ["http://influx.zimmermann.XXX:8086"]
  token = "@{docker_secretstore:telegraf_influxdb_token}"
  #token = "jCSA **** Ong=="
  organization = "zimmermann.XXX"
  bucket = "telegraf/autogen"

```

Any idea?

---

<div class="post-metadata">

**Author:** ![jpowers](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/jpowers/32/8569_2.png) [@jpowers](https://community.influxdata.com/u/jpowers)\
**Post date:** [January 16, 2024, 3:29pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/2 "2024-01-16T15:29:27Z")

</div>

Hi,

> as soon as I switch to docker secrets (file), telegraf authorization to influx fails. If I hardcode the token into the config file, it works.

This generally means one of few things:

1. Something is misconfigured
2. The secret store is not actually readable by the user running telegraf
3. The password is wrong/has special characters in that are not escaped

Based on what you copied and pasted the config looks fine, so no typo.

What user is running telegraf? Is this as a service?

---

<div class="post-metadata">

**Author:** ![alexander-zimmermann](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/alexander-zimmermann/32/14909_2.png) [@alexander-zimmermann](https://community.influxdata.com/u/alexander-zimmermann)\
**Post date:** [January 16, 2024, 9:33pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/3 "2024-01-16T21:33:28Z")

</div>

```auto
alexander@ubuntu:~$ docker exec -it telegraf bash
root@telegraf:/# ps auxx
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
telegraf 1 0.2 0.7 5647052 133932 ? SLsl Jan15 4:06 telegraf --config=/etc/telegraf/telegraf.conf --config-directory=/etc/telegraf/telegraf.d
root 25 0.1 0.0 4608 3564 pts/0 Ss 22:22 0:00 bash
root 31 0.0 0.0 8480 4220 pts/0 R+ 22:22 0:00 ps auxx
root@telegraf:/# ls -la /var/run/secrets/telegraf_influxdb_token 
-rw-rw-r-- 1 1000 1000 89 Jan 14 12:22 /var/run/secrets/telegraf_influxdb_token
root@telegraf:/#

```

Hmm, telegraf user should have access, but …

```auto
root@telegraf:~# su telegraf -c cat /var/run/secrets/telegraf_influxdb_token
root@telegraf:~#

```

I don’t get it ☹

---

<div class="post-metadata">

**Author:** ![jpowers](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/jpowers/32/8569_2.png) [@jpowers](https://community.influxdata.com/u/jpowers)\
**Post date:** [January 16, 2024, 10:27pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/4 "2024-01-16T22:27:34Z")

</div>

> [@alexander-zimmermann](#):
>
> /var/run/secrets

What are the permissions on the folder above?

---

<div class="post-metadata">

**Author:** ![alexander-zimmermann](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/alexander-zimmermann/32/14909_2.png) [@alexander-zimmermann](https://community.influxdata.com/u/alexander-zimmermann)\
**Post date:** [January 17, 2024, 7:43am UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/5 "2024-01-17T07:43:20Z")

</div>

```auto
alexander@ubuntu:~/home-automation**$ docker exec -it telegraf bash

root@telegraf:/# ls -la /var/run/secrets
total 12
drwxr-xr-x 2 root root 4096 Jan 15 16:52 .
drwxr-xr-x 1 root root 4096 Jan 15 16:52 ..
-rw-rw-r-- 1 1000 1000 89 Jan 14 12:22 telegraf_influxdb_token

root@telegraf:/# ls -la /var/run/
total 16
drwxr-xr-x 1 root root 4096 Jan 15 16:52 .
drwxr-xr-x 1 root root 4096 Jan 15 16:52 ..
drwxrwxrwt 2 root root 4096 Jan 10 01:00 lock
drwxr-xr-x 2 root root 4096 Jan 15 16:52 secrets

root@telegraf:/# ls -la /var/
total 44
drwxr-xr-x 1 root root 4096 Jan 10 01:00 .
drwxr-xr-x 1 root root 4096 Jan 15 16:52 ..
drwxr-xr-x 2 root root 4096 Dec 9 22:08 backups
drwxr-xr-x 1 root root 4096 Jan 10 01:00 cache
drwxr-xr-x 1 root root 4096 Jan 11 16:26 lib
drwxrwsr-x 2 root staff 4096 Dec 9 22:08 local
lrwxrwxrwx 1 root root 9 Jan 10 01:00 lock -> /run/lock
drwxr-xr-x 1 root root 4096 Jan 11 16:26 log
drwxrwsr-x 2 root mail 4096 Jan 10 01:00 mail
drwxr-xr-x 2 root root 4096 Jan 10 01:00 opt
lrwxrwxrwx 1 root root 4 Jan 10 01:00 run -> /run
drwxr-xr-x 2 root root 4096 Jan 10 01:00 spool
drwxrwxrwt 2 root root 4096 Dec 9 22:08 tmp

```

---

<div class="post-metadata">

**Author:** ![alexander-zimmermann](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/alexander-zimmermann/32/14909_2.png) [@alexander-zimmermann](https://community.influxdata.com/u/alexander-zimmermann)\
**Post date:** [January 17, 2024, 8:05am UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/6 "2024-01-17T08:05:49Z")

</div>

I changed `/etc/passwd` for `telegraf` user from `/bin/false` to `/bin/bash/` as login shell and run:

```auto
root@telegraf:/run/secrets# su telegraf
telegraf@telegraf:/run/secrets$ cat telegraf_influxdb_token
jCSAWc***

```

Access should not be a problem, IMO. Let me double check the token a third time…

---

<div class="post-metadata">

**Author:** ![alexander-zimmermann](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/alexander-zimmermann/32/14909_2.png) [@alexander-zimmermann](https://community.influxdata.com/u/alexander-zimmermann)\
**Post date:** [January 19, 2024, 9:10am UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/7 "2024-01-19T09:10:27Z")

</div>

I found the problem, it’s the token itself.

**Provide token via environment:**

docker compose file:

```auto
secrets:
  telegraf_influxdb_token:
    environment: "TELEGRAF_TOKEN"

```

results in:

```auto
alexander@ubuntu:~/home-automation$ docker exec -it telegraf cat /run/secrets/telegraf_influxdb_token
jCSAWc***Ong==alexander@ubuntu:~/home-automation$ 

```

**Provide token via file:**

docker compose file:

```auto
secrets:
  telegraf_influxdb_token:
    file: ./.secrets/telegraf_influxdb_token

```

results in:

```auto
alexander@ubuntu:~/home-automation$ docker exec -it telegraf cat /run/secrets/telegraf_influxdb_token
jCSAWc***Ong==
alexander@ubuntu:~/home-automation$ 

```

The new line at the end is the problem. However my PWD file doesn’t contain any new line:

```auto
alexander@ubuntu:~/home-automation$ cat -A .secrets/telegraf_influxdb_token 
jCSAW***Ong==$
alexander@ubuntu:~/home-automation$

```

I believe that is a bug in how telegraf reads the file. Google a bit, I found “[Trim contents of slack api urls from files by srhb · Pull Request #2929 · prometheus/alertmanager · GitHub](https://github.com/prometheus/alertmanager/pull/2929)”

Different SW, different use case, but same problem.

Any thoughts?

---

<div class="post-metadata">

**Author:** ![jpowers](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/jpowers/32/8569_2.png) [@jpowers](https://community.influxdata.com/u/jpowers)\
**Post date:** [January 19, 2024, 1:53pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/8 "2024-01-19T13:53:16Z")

</div>

> However my PWD file doesn’t contain any new line:

I’m not following. You said the newline is the problem, but that the file does not have a newline 😃

Instead of `cat -A $file`, can you show the output of `od -x $file`

How is your file different than our test cases:

> <https://github.com/influxdata/telegraf/tree/master/plugins/secretstores/docker/testdata>
>
> //github.com/influxdata/telegraf/tree/master/plugins/secretstores/docker/testdata

---

<div class="post-metadata">

**Author:** ![srebhan](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/srebhan/32/8572_2.png) [@srebhan](https://community.influxdata.com/u/srebhan)\
**Post date:** [January 19, 2024, 3:39pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/9 "2024-01-19T15:39:20Z")

</div>

@alexander-zimmermann you show above that with

```auto
secrets:
  telegraf_influxdb_token:
    file: ./.secrets/telegraf_influxdb_token

```

there is a newline in the container’s secret, while there is none with the environment-variable setup.

Telegraf can only read the secret-file and use **as-is**! We should not mess with the content of the secret and we don’t. So if there is a newline at the end of the secret-file you will get a Telegraf secret with a newline at the end!

If your `./.secrets/telegraf_influxdb_token` file does not contain a trailing newline, this clearly is an issue with docker or your configuration and should be fixed there.

Did you, by chance, `echo` the token into the file on your host? If so, did you use `-n`?

---

<div class="post-metadata">

**Author:** ![alexander-zimmermann](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/alexander-zimmermann/32/14909_2.png) [@alexander-zimmermann](https://community.influxdata.com/u/alexander-zimmermann)\
**Post date:** [January 19, 2024, 4:32pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/10 "2024-01-19T16:32:50Z")

</div>

```auto
alexander@ubuntu:~$ od -x home-automation/.secrets/telegraf_influxdb_token 
0000000 436a 4153 6357 3455 6b62 4878 705f 6954
***
0000120 756c 4f74 676e 3d3d 000a
0000131

alexander@ubuntu:~$ docker exec -it telegraf od -x /var/run/secrets/telegraf_influxdb_token
0000000 436a 4153 6357 3455 6b62 4878 705f 6954
***
0000120 756c 4f74 676e 3d3d
0000130

alexander@ubuntu:~/home-automation$ docker exec -it telegraf od -x /var/run/secrets/telegraf_influxdb_token
0000000 436a 4153 6357 3455 6b62 4878 705f 6954
***
0000120 756c 4f74 676e 3d3d 000a
0000131

```

First run is the host, second run is the secret mapping via environment, third one via file.

@srebhan Did you agree that docker has no bug since it maps exactly the file from the host?

---

<div class="post-metadata">

**Author:** ![alexander-zimmermann](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/alexander-zimmermann/32/14909_2.png) [@alexander-zimmermann](https://community.influxdata.com/u/alexander-zimmermann)\
**Post date:** [January 19, 2024, 4:39pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/11 "2024-01-19T16:39:58Z")

</div>

> [@srebhan](#):
>
> Did you, by chance, `echo` the token into the file on your host? If so, did you use `-n`?

@srebhan not sure if I understand you correctly. Do you mean the token generation in general?

I generated the token at influx, copy it to clipboard, open `vim` and paste it, for both the token file `~/home-automation/.secrets/telegraf_influxdb_token` and the environment variable in `~/home-automation/.env`

---

<div class="post-metadata">

**Author:** ![srebhan](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/srebhan/32/8572_2.png) [@srebhan](https://community.influxdata.com/u/srebhan)\
**Post date:** [January 19, 2024, 4:44pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/12 "2024-01-19T16:44:41Z")

</div>

Did you notice the `0a` at the end of the data when **not** mapped via environment? Guess what that is! 😜

---

<div class="post-metadata">

**Author:** ![srebhan](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/srebhan/32/8572_2.png) [@srebhan](https://community.influxdata.com/u/srebhan)\
**Post date:** [January 19, 2024, 4:50pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/13 "2024-01-19T16:50:21Z")

</div>

I’m pretty sure that vim will add a newline to a text-file… [linux - Why is Vim adding a newline? Is this a convention? - Super User](https://superuser.com/questions/745111/why-is-vim-adding-a-newline-is-this-a-convention)

---

<div class="post-metadata">

**Author:** ![alexander-zimmermann](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/alexander-zimmermann/32/14909_2.png) [@alexander-zimmermann](https://community.influxdata.com/u/alexander-zimmermann)\
**Post date:** [January 19, 2024, 5:10pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/14 "2024-01-19T17:10:32Z")

</div>

That’s correct. I also found it out in the meantime.

However, see here: [Why Should Text Files End With a Newline in Linux](https://www.baeldung.com/linux/files-end-with-newlines#:~:text=Newlines%20at%20the%20End%20of,defined%20as%20consisting%20of%20lines.), Section 3.

" [POSIX](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html) defines a line as a possibly empty sequence of non-newline characters, terminating in a newline, also called EOL (End-Of-Line), [ASCII](https://www.baeldung.com/cs/ascii-code) code _0x0A_. Meanwhile, a _text_ file is defined as consisting of lines.

Thus, **the last line of a file, by definition and standard, should conclude that file with EOL**."

I also checked the [POSIX standard](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html) by myself now. It’s correct, a POSIX conform file must be terminated with a newline.

Sure, I can now tweak my file to be none POSIX conform, but I would be much better to change the telegraf code.

Agree?

---

<div class="post-metadata">

**Author:** ![srebhan](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/srebhan/32/8572_2.png) [@srebhan](https://community.influxdata.com/u/srebhan)\
**Post date:** [January 19, 2024, 5:41pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/15 "2024-01-19T17:41:08Z")

</div>

Then how do we know it is a **text** file? What if someone passes a binary AES key in there? Should we just remove random parts of that file?

---

<div class="post-metadata">

**Author:** ![srebhan](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/srebhan/32/8572_2.png) [@srebhan](https://community.influxdata.com/u/srebhan)\
**Post date:** [January 19, 2024, 5:44pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/16 "2024-01-19T17:44:32Z")

</div>

Please check [Manage sensitive data with Docker secrets | Docker Docs](https://docs.docker.com/engine/swarm/secrets/). To cite:

> […]
> 
> - Generic strings or binary content (up to 500 kb in size)

---

<div class="post-metadata">

**Author:** ![alexander-zimmermann](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/alexander-zimmermann/32/14909_2.png) [@alexander-zimmermann](https://community.influxdata.com/u/alexander-zimmermann)\
**Post date:** [January 20, 2024, 10:34am UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/17 "2024-01-20T10:34:59Z")

</div>

@srebhan All good questions and I don’t have a finale answer on that. Let’s do the following (suggestion): we enhance the documentation on GitHub and on the webpage and clearly state that telegraf handle/interprate all docker secrets as a binary file even if the file on the host was txt file. So the user should take care that file should not contain a \n at the end.

BTW: why we don’t have an issue w/ influx itself? All tokens work fine. All my tokens have a \n. What is the difference?

---

<div class="post-metadata">

**Author:** ![Jason\_Stirnaman](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/jason_stirnaman/32/14433_2.png) [@Jason\_Stirnaman](https://community.influxdata.com/u/Jason_Stirnaman)\
**Post date:** [January 24, 2024, 5:56pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/18 "2024-01-24T17:56:51Z")

</div>

@alexander-zimmermann

Regarding docs, I agree. This looks like a common and easy mistake. I see many similar reports across projects.

Please submit an issue or PR to [GitHub - influxdata/docs-v2: InfluxData Documentation that covers InfluxDB Cloud, InfluxDB OSS 2.x, InfluxDB OSS 1.x, InfluxDB Enterprise, Telegraf, Chronograf, Kapacitor, and Flux.](https://github.com/influxdata/docs-v2) and I’ll be happy to review or work on it for [docs.influxdata.com](http://docs.influxdata.com) and Telegraf repo docs.

> All my tokens have a \n.

Where? For example, in a .env file, variables are key-value pairs; `=` and the line terminating character are used as delimiters for the value. The token doesn’t contain a new-line character.

---

<div class="post-metadata">

**Author:** ![srebhan](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/srebhan/32/8572_2.png) [@srebhan](https://community.influxdata.com/u/srebhan)\
**Post date:** [January 24, 2024, 8:37pm UTC](https://community.influxdata.com/t/influxdbv2-authorization-fails-if-docker-secrets-are-used/32809/19 "2024-01-24T20:37:32Z")

</div>

@alexander-zimmermann and @Jason_Stirnaman even though, this is **not** a Telegraf specific issue I would appreciate a PR enhancing the documentation to make people aware of this behavior!