I’m currently in a quandary where I have to rollup metrics and store them for years at a time, but I only have Elasticsearch as a backend, not InfluxDB. It was suggested by @daniel that Kapacitor could potentially serve the purpose of taking the stream data from Telegraf, roll it up, and send it back to Telegraf, however I’m a bit perplexed for the configuration of this.
What I’m trying to do is thus…
I have roughly 1600 devices that I’ll be querying many interfaces on, pulling inbound octets, outbound octets. I’ll be polling them with Telegraf via SNMP every 2.5 minutes, storing this resolution of data for 30 days rolling in daily indices in Elasticsearch.
My hope is that Kapacitor can then rollup the data into daily resolution points that I can store long term. To achieve this, I’d need to group it per-interface per-agent-host, separately doing derivatives across all the fine-point data (to resist counter rolls/clears) and then calculating the mean of the derivatives (byte counters are cumulative, as you’re no doubt well aware) for a single daily mean of the bandwidth utilization for the day for each interface on each device.
I’m flailing around in the lab with it, but if anyone has any pointers I’d be grateful.
) is actually how the derivative aggregations are going to be done in Kapacitor and whether it will tolerate a counter clear/roll. If it’s doing a derivative between each row and then doing an average across that, then it shouldn’t miss a beat. However, if it attempts to just take the min/max during that interval and then do a derivative across the whole time interval, a counter roll/clear would end up ruining that whole interval, something I’m trying to avoid.
