We are working on productionalizing how we create kapacitor tasks, templates etc. We are planning on using https://docs.influxdata.com/kapacitor/v1.4/guides/load_directory/ plus ansible. One of the issues we are thinking we will run into is how to prevent the tasks from ballooning into a lot of files that are nearly identical.
For example we have dozens of queues we want to monitor. We have a tickscript that pulls out the depth thats generic, it primarily takes the queue names as its input. But, we have different thresholds for each queue. So our initial solution is to render a task file per queue that contains the queue name and threshold. Problem is this results in a lot of files especially when you multiply across all the different kinds of alerts we want to generate.
Does anyone have any experience with a better way to manage kapacitor alerts?