Continuous query doesn't process all the available data

1

Hello,
I’ve about 250 hosts reporting into a measurement called xtables every minute. Those are raw counters.
I’m converting these raw counters to 5 minute averages into another metric with a continuous query to avoid recalculating every time I select the same data.

So I have this CQ:

CREATE CONTINUOUS QUERY cq_rate_recalc ON my_stats BEGIN SELECT (8 * non_negative_derivative(mean("inet"), 1s)) AS internet, (8 * non_negative_derivative(mean("http"), 1s)) AS http, (8 * non_negative_derivative(mean("https"), 1s)) AS https, (8 * non_negative_derivative(mean("http_alt"), 1s)) AS http_alt INTO my_stats.autogen.recalculated FROM my_stats.autogen.iptables GROUP BY time(5m), host fill(0) END

And it works for 197 hosts. Meanwhile, all configured hosts are still submitting their data to the iptables measurement and if I run the query by hand I get proper data. However, running a query in a cronjob seems very counter productive, especially since CQs exist to do this particular thing.

Does anyone know why is this happening?
Any tips are greatly appreciatted!

Thanks!

2

TIL: fill(0) has no impact on CQs.

Also, most hosts are under heavy load but the ones that didn’t get calculated were under heavy network load as well and were lagging up to 8 minutes on their reports. So I enabled the queries to RESAMPLE EVERY 5m FOR 10m and it started working, albeit incorrectly. The final solution was to RESAMPLE EVERY 5m FOR 1h which is a terrible way to approach this but it gives enough slack to all servers to lag behind witht their reports while producing proper data in the long run. Not very elegant, nor efficient but so far it seems to be working so there’s that. Hope this saves someone all the headbanging and cursing.