The first step would be to increase your metric buffer. The buffer defaults to 10,000. Given your previous statement of 28,500 metrics, I would increase it to 100,000 at first and see how it responds.
Can do. It says that limit is a per input item. How does that work if you have the same input used in multiple different config files in the telegraf.d folder? Does one override them all or does each config file act on its own?
It is not per input, but per output:
- metric_buffer_limit: Maximum number of unwritten metrics per output. Increasing this value allows for longer periods of output downtime without dropping metrics at the cost of higher maximum memory usage.
Each output has it’s own buffer in case an output goes down for a period of time.
You can set that above option in the agent settings, or per output.
That was definitely the problem. I upped the metric_buffer_limit and now it is all working perfectly. I also created a dashboard with internal_write_buffer_limit and internal_write_buffer_size on it so I can keep an eye on it from now on. Thank you!
