Utilize telegraf to send data at different rates (downsampling)

Telegraf
,
1

I am trying to balance pipeline simplicity and data robustness. As an example, I am interested in collecting network metrics (counters et al.) at a very high right (<30seconds) and retaining that data for a short period of time (1month). However, I also want to store the same data in a less aggressive window (~5min).

The first option I cam up with was just run the collector twice, one running every 30s and one running every 5mins. Each output would either go into different buckets or with different tags.

The second option I found was adding quix streams to our pipeline to downsample the data.

I’m wondering if there is a simpler third option to have telegraf do something like this natively. Any advice is appreciated

2

You could definitely run two different collection services but instead I would look at the TSDB you are using. Is there a built in way to compress data base on time? That’s how we have ours setup, we have intervals going back into history where it will start to avg and compress the datapoints. Otherwise you are going to still be sending duplicate requests to devices and duplicate data one or multiple databases only for it to be discarded in a month anyways.

3

For downsampling metrics, you could have a look at the final plugin.