I’m currently in a quandary where I have to rollup metrics and store them for years at a time, but I only have Elasticsearch as a backend, not InfluxDB. It was suggested by @daniel that Kapacitor could potentially serve the purpose of taking the stream data from Telegraf, roll it up, and send it back to Telegraf, however I’m a bit perplexed for the configuration of this.
What I’m trying to do is thus…
I have roughly 1600 devices that I’ll be querying many interfaces on, pulling inbound octets, outbound octets. I’ll be polling them with Telegraf via SNMP every 2.5 minutes, storing this resolution of data for 30 days rolling in daily indices in Elasticsearch.
My hope is that Kapacitor can then rollup the data into daily resolution points that I can store long term. To achieve this, I’d need to group it per-interface per-agent-host, separately doing derivatives across all the fine-point data (to resist counter rolls/clears) and then calculating the mean of the derivatives (byte counters are cumulative, as you’re no doubt well aware) for a single daily mean of the bandwidth utilization for the day for each interface on each device.
I’m flailing around in the lab with it, but if anyone has any pointers I’d be grateful.