Keeping alerts open

InfluxDB 1 Kapacitor
1

Hello dear community,

I have a question about Kapacitor.
Is there a way to prevent kapacitor to automatcally close alerts when the criteria is not matched anymore.

For example, i trigger some alert based on a combinaison of criteria and I would lile to keep my alert open in Opsgenie so i can go and manually check the problem.
If the criterias are not matched anymore then kapacitor automatically closes my alert on opsgenie and then i don’t see.

Thank you in advance for helping me with this case

2

Hello @Adam_BEN_LTAIFA,
Welcome!
What exactly do you mean by keeping your alert open? Like once your conditions are met keep sending alerts even after the conditions have returned to normal? How will you determine when you have met your alert conditions again if you ignore the change back to normal conditions?

1 Like
3

Okay so here is my setup
I have some influxDB instance
I have telegraf that forwards data every 3h based on some measurements made on several vms.
I have a tick script that checks when some vms lose connectivity for a period of time. For that i made an alert that tells me when we I had an interruption. Even if the connectivity comes back, i still want to see my alert about that connectivity interruption so i can manually check for root cause analysis.
FYI, kapacitor sends alerts to Opsgenie.

4

But if this happen during the night, since it’s not CRITICAL, it will close it in opsgenie.

Here is an example of data:
At instant t:
Vm=A,status=1
At instant t-3h:
Vm=A,status=0

Alert is raised if status(t-3h) - status(t) is < 0
But let say that it stays with status = 1 forever.
Then criteria is not matched and i won’t be alerted for new alerts but i would like to keep the first one open. ( hope this helps )

other option, is there a way to close an alert when matching condition X and only close it when matching condition Y ?

5

Or another option :smiley:
Lets assume that Kapacitor closes the alert on its side but is it possible that it doesn’t send an event to opsgenie so at least it stays open on Opsgenie ? ( idk if NoRecoveries can make the trick )
@Anaisdg

6

@Adam_BEN_LTAIFA,
This is outside my scope. I made an issue so our Kapacitor engineer can help.

7

Yes this thing is possible. You can change the alert status using crit and critReset

    .crit(lambda: "value" > 80)
    .critReset(lambda: "value" < 70)
1 Like
8

Hello @Emrys_Landivar

Thanks for the answer ! Thanks @Anaisdg for the support !
Just for my knowledge, could noRecovies() help me to not send closing alert events to opsgenie ?
I saw in the opsgenie v2 doc that i could also overwrite the closing behavior with .recoveryAction(). Could this work ?

1 Like