Hi guys, I’d really like to ask your input on something we’re working on in Telegraf - InfluxDB.
We are working on a project that will use Telegraf in a slightly non-standard way. An application will determine which plugins will be used and what their configuration will be - in terms of connection details and query - and will write this to InfluxDB.
This will be dynamic - plugins can be added and plugin configuration can be changed at any time. As it stands, Telegraf’s configuration (driven by a configuration file) assumes that configuration is static so it’s not clear what the best way to drive Telegraf’s configuration dynamically is. We’ve got a few possible solutions (below) - I’d be really interested in your feedback on these approaches or on any others that we might have missed.
generate Telegraf configuration file based on the dynamic data
introduce a new implementation of the Telegraf Configuration struct which can read from the database storing desired configuration
create a single plugin which is capable of reading the configuration from our database; this plugin would collect data from multiple sources as determined by the configuration in the database
The best way to manage configuration in a production environment is to use a configuration management tool like Puppet, Chef, or Ansible. Each of these gives you the ability to create configuration files, based on templates, using external data that you can provide through various mechanisms. This lets you completely automate the process of generating a config; any changes that need to be made to the process are made in code and committed to version control.
Using one of these tools, you’d be able to easily generate a new config based on your data. You can then tell Telegraf to re-load the configuration by sending the process a SIGHUP signal.
The other approaches sound like a fair bit of work, and you’d have to maintain your own fork of Telegraf. I’m not sure what additional benefits they would bring.
We do have some prototype code that would add a system for centralized storage of configuration. It is on the back burner right now and there is no planned date, but it is coming. If you are curious, right now the prototype code is available in this repo, but it isn’t suitable for use. It is most similar to your second bullet point.
In the meantime, I do suggest using @noahcrowley’s suggestion to generate the config and reload Telegraf. However, if the config file were need updated every few minutes then you may need to integrate it into the plugin or maintain a fork of Telegraf since the metric buffer is flushed on restart.
