Creating and using a secret store (OS) in a single (re)start of telegraf

1

I have a configuration that includes something like this:

[[secretstores.os]]
  id = "some_store"
  keyring = "telegraf"

[[inputs.mysql]]
  servers = ["mysql_user:@{some_store:mysql_password}@tcp(mysqlserver.example.com:3306)/"]

Starting telegraf with this configuration fails because the secret store doesn’t exist. I don’t see a way to create the secret store from the command line or any other way to ensure that the secret store exists when telegraf starts with this configuration.

I have a solution, but it doens’t seem like it could be the best or most straightforward way to do this. For context, I’m doing this in Ansible.

In my initial configuration, I replace the reference to the secret store with a placeholder:

[[secretstores.os]]
  id = "some_store"
  keyring = "telegraf"

[[inputs.mysql]]
  servers = ["mysql_user:mysql_password_placeholder@tcp(mysqlserver.example.com:3306)/"]

I then (re)start telegraf, and that creates the secret store.

Next, I set the password in the secret store so that I can now use it.

Lastly, I replace the password placeholder with a reference to the password in the secret store (see ansible.builtin.lineinfile):

ansible.builtin.lineinfile:
  path: "{{ telegraf_config_file }}"
  regexp: "^(.*)mysql_password_placeholder(.*)$"
  line: '\1@{some_store:mysql_password}\2'
  backrefs: yes
notify: restart telegraf

This all works, but it just feels a bit weird:

  • create a dummy configuration, kinda like a template
  • (re)start telegraf so it will create the secret store
  • correct the dummy configuration to be a correct configuration
  • restart telegraf

Is there a better way to do this?

Many thanks!

2

@Jim_Ivey,
Hmmmm I’m not sure…Your current solution is quite creative, but I understand why it feels less than ideal since you’re effectively starting Telegraf twice to handle the secret store. @srebhan do you know?
Can you maybe use execd plugins? To first start the mysql server? That doesn’t feel much better though.
Use something like ansible to manage the secrets?

3

Thanks! We use AWS Secrets Manager to manage the secrets. Before migrating from inputs.exec to inputs.mysql, we tried using mysql_config_editor without luck. The telegraf secret store works well for locally caching the password from AWS Secrets Manager.

Current thinking is to move this to a separate conf file to avoid having to use the placeholder:

[[secretstores.os]]
  id = "some_store"
  keyring = "telegraf"

We still have to (re)start twice, but it’s easier to read and follow.

Many thanks!

4

I may have found my answer.

I didn’t understand what --config did in telegraf secrets set. I now store the secret with

telegraf \
    --config .../secret_store.conf \
    secrets \
    store \
    some_store \
    mysql_password \
    "This is my super secret password."

Once I do that, the secrets store exists without having to (re)start telegraf the first time.

Thanks!

1 Like
5

It would be interesting to provide a secret-store plugin for AWS secret manager, wouldn’t it? If you are willing to assist in testing, please open a feature request and drop me a link or mention me there…

1 Like