We are looking for a way to create a macro alerts in kapacitor.
So we have a cluster containing multiple hosts and we have created micro alert where if any host shows error, it will fire a warning alert per host. Number of errors per hosts are stored in a “errors” field.
So if “errors” field has value > 0, it should trigger a warning alert.
We now want to create Macro alerts where if 50% of hosts shows errors we need to create a critical alert.
Creating micro alert was simple because all we have to do was to check the field and if errors > 0, trigger a warning alert. We are facing issue while creating macro alerts because this now involves calculating how many hosts are having errors > 0 and since field cannot be used in where filter, we are not able to get the number of hosts which are having errors.
Any thought on how we can create macro alerts?
Also, do we have anything in kapacitor where we can create nested alerts and if child alerts fire for 5 different hosts, it will fire parent alert as well?