@telegraf_vmware thanks for the questions, let me try to clarify both points.
1. Where is the data stored?
When you use the Prometheus output plugin, metrics are only kept in memory. Input plugins collect them on their schedule, Telegraf passes them through the pipeline, and the Prometheus plugin holds the latest values in RAM. When Prometheus scrapes /metrics, it just serves whatever is currently in memory. Nothing is written to disk, so if Telegraf restarts, those metrics are lost.
2. About the timing mismatch
You’re right, this is expected behaviour. Inputs collect on their own intervals, and Prometheus scrapes on its own schedule. That means you can end up with a little lag, like in your example where a metric collected at 10:05 might only get scraped at 10:07. This isn’t a bug, it’s just how the pull model works — Telegraf keeps the latest values ready, and Prometheus decides when to pull them.
On-demand collection
Your idea of triggering inputs only when Prometheus scrapes would be a big shift. It would mean scrapes might block while waiting for inputs, it would bypass Telegraf’s normal processing pipeline, and it would get tricky if multiple inputs are involved. So the current design keeps things simpler and more reliable, even if it means the timing isn’t always exact.
What you can do
- Make sure your input collection interval is shorter or equal to the scrape interval (e.g. collect every 30s if Prometheus scrapes every 60s).
- Use
metric_buffer_limitto avoid dropping metrics between scrapes. - Look at timestamps if you need more precision — Telegraf can pass those along.
- If you really need real-time data, a push-based output like InfluxDB might fit better than Prometheus.
In short, Telegraf prioritizes reliability and simplicity, which lines up with how Prometheus normally operates.
