Hi Team,
I’d like to understand how the retry mechanism works in the output.http plugin. Suppose that the Telegraf agent sends metrics and the server is unresponsive or down, will those metrics
get lost or will they be stored in a buffer? If so, for some time the Telegraf agent will attempt to resend the metrics to the server, but if it still doesn’t respond, then what
happens to the buffered metrics?
Do they get overwritten after a certain amount of time, or is there a mechanism to prevent this from happening? I’m also curious about the timeout period - how long does the Telegraf
agent wait before retrying the send, and are there any configurable options for this timeout?
Additionally, if the server remains unresponsive after multiple retries, will the buffered metrics be persisted in some way (e.g., logged to disk), or will they indeed get lost?
Also, are there any best practices or recommendations for configuring the retry mechanism in the output.http plugin to minimize data loss in case of server outages?
Hi!
Telegraf agent has in its telegraf.conf file the parameters “interval”
and “metric_buffer_limit”. This buffer is the amount of metrics it will store on memory for sending to the outputs plugins. If one of them could not be reached, this buffer will increase. You can se this buffer in action activating debug = true or running the process using -debug.
This buffer could be changed, but you have to check the amount of metrics you are generating on each data collection and for how long you want to store them. This will consume ram memory for the process. If you need a special case for saving data when another plugins do not work, you can work with the output.file plugin and save it locally, but telegraf will not use it to resend to any plugin that failed.
The outputs.http has the timeout parameter that you can use based on what is the best for your http request. Based on that time, you can define the interval for the agent or override this property inside the desired plugin.
Hope it helps
2 Likes
Got it. Thanks for the detailed explaination
1 Like
