I’m trying to use the SQL output plugin to store my metrics but I’m running into the issue that pretty much every input plugin doesn’t output all fields all the time, so after the tables have been created they are usually missing columns.
I’m not really using too many or too exotic input plugins, here’s a typical config:
[[inputs.internal]]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.net]]
interfaces = ["eth*","lo", "wg*", "zt*", "wlan*", "en*"]
I increased the flush interval in the hope that would lead to a more complete set of columns but I’m still getting missing column errors.
How are others dealing with this? Do you manually create the missing columns? How do you know the type? Or do you use fieldpass and simply discard anything you didn’t expect in the first place?
@srebhan Pinging you as it’ll likely end up on your desk. Do you think I should work on a feature to allow the SQL output plugin to issue ALTER TABLEs when columns are missing? This isn’t super trivial, a couple decisions would have to be made, like how to detect across database engines that a column is missing (for example by inspecting the schema or by matching the log message) and when exactly to trigger the check (which would also solve this).
As a simpler workaround, if we could add some debug output listing the fields and their types whenever an insert error happens, that would already help creating the missing columns manually.
