Kapacitor has a very powerful engine for analyzing and alerting based on time-series data. I was impressed with ease of setup and after initial struggle could create some basic alerts.
Struggle stemmed from complicated documentation, which was listing available nodes alphabetically, not providing enough description and examples. Chaining methods showing up on every page complicate documentation even further.
Chronograf helped to create some basic alerts, but hid the power of Kapacitor under the hood. TICK script editor was not very helpful as well. It was lacking intellisense and syntax highlighting.
I think, the most power comes from alerting based on stream of incoming data. However, these tasks are hard to setup:
-
There is no compatibility with batch based alerting. We want to alert in real time, but want to compare against baseline from the database. Right now, we would have to implement side process to query and cache historic data. (I am not even talking that the functionality of UDFs is dependent on unix-specific sockets.)
-
There is no way to test the stream-based task, except for waiting for alert to get triggered. It would be very, very, very helpful if there was an option to replay historic data as if it was coming in real-time and generate the log of alerts that would be generated. My understanding is that the filtering from TICK script need to be translated into InfluxQL to fetch data ordered by time alone, or is there anything else I am missing?
After setting up Kapacitor and creating subscription, I was shocked that Kapacitor started receiving ALL data from InfluxQL. My expectation was that it would subscribe to only the data it needs. Kapacitor has implementation for filtering incoming data, so why not to reuse this code at InfluxDB?
While Kapacitor is lacking scalability, parsing all data incoming from the InfluxDB cluster on a single machine may be problematic. I heard, that the recommendation was to switch to batch-processing, which is not the option, as we have users who are used to getting alerts in real-time.
I was also thinking that the reason, Kapacitor gets data through InfluxDB, is that InfluxDB would serve as a message queue, accumulating incoming data in WAL until it gets processed by Kapacitor, but found it not to be the case. Also, it appears, that undelivered data doesn’t get resent. Just because we already have Kafka in front of InfluxDB clusters (to queue, handle multiple senders, and multiplex) we think about sending from it directly to Kapacitor. Although Kapacitor has to have a subscription, it may be set up on a dummy instance of InfluxDB.
Are there plans to implement the above functionality? Otherwise I may have to implement it myself, but would prefer to contribute in that case. Any guidance would be appreciated.
I would really like to see Kapacitor suit our use case. InfluxData has the great future. It’s just that we always want that future to be available right now. 
Thank you!
