With the new alert system is it now possible to have delayed alert ? What I mean is that I do not want to send an email if an alert duration is less than 5 minutes, I still want it to be logged somewhere else (probably influxdb).
For my first prototype I just gave up and added my own daemon querying influxdb and sending the alert but I would prefer if kapacitor could handle everything alert related.
@schmurfy That is definitely supported in kapacitor! You just need to set the alert level as a matcher (match: level() == INFO) when when you are defining alert handlers.
I am not sure I understand how this solve my problem, topic handlers seems to be pretty basic looking at the doc, they have no logic in them so how this would achieve my goal ?
To be clearer here is what I meant:
00:00 alert A1 is raised
00:01 alert A2 is raised
00:04 alert A1 is cleared
00:06 an email is sent to notify A2
In this example A1 was cleared under the 5min threshold and would not be notified but since A2 is still active after that delay it would.
@schmurfy To help me understand, you want to record that alert A1 was raised but not take any action until after 5m of the alert still being raised?
Because you could simply change the way you define the alert in TICKscript to only raise the first time after its has already met the condition for 5m. But if you want to have a record of the alert thats a bit different.
Could you expound on your use case a bit more? What do you use the record for? I have a few ideas of how this could work but having a better idea of what you want to accomplish would be helpful.
Ideally all the alerts UP and DOWN events are stored in influxdb so I can later display them but for the emails I just want one to be sent if there an action required on our side, we are monitoring connection status for our clients, if their connection drop and come back within 5 minutes there is nothing to do and I don’t want any email to be sent, on the other end if an alert persist after that this is a problem we want to be notified about.
Currently I have an external process querying the alert database in influxdb and doing that but I wish kapacitor could all of it by itself without any external requirement.
@schmurfy So I think doing what Jack suggested would be best. Use different alert levels to control where the alerts are sent.
For example define the warn level as being the UP and DOWN state of the alert and reserve the critical level for DOWN > 5m. Then as Jack suggests you can setup an alert handler to send emails in only the critical state. And all the state transitions can be recorded in InfluxDB.
@nathaniel is it still the only/best way to do this ?
I really don’t like this idea, the levels I can use with kapacitor are really limited (info, warning, critical) and if I want to have one taken by the emails I am only left with 2 if I am correct…
I already had the same issue a while ago when I first tried kapacitor and was told that the new alert system would solve that so I am a little disappointed.
A solution would be the ability to add custom level (ex: crit_email) so that the transition is clearly visible later (warn => crit => crit_email => OK), I don’t really want to have a crit alert transition to info to means it was notified and don’t want to loose crit either.
What I did before and really don’t want to do again is to have my own daemon beside kapacitor handling the sending of emails but I hate that, I wish kapacitor could handle all by itself, another solution would be to not trigger an alert unless the defect has been seen for X minutes with a batch but it also means this defect will not be visible on the dashboard before the email is sent 
