A "Back to Normal" message in a .tick script?

kapacitor
#1

Is it possible to define a ‘back to normal’ version of the message string that you define in an alert?

For example lets say I’m counting the number of api requests made every couple minutes, and then via a udf detecting spikes above/below some fixed multiplier…

|alert()
        .id('request_{{ index .Tags "request_type"}}')
        .warn(lambda: "multiplier" > max_multiplier)
        .warnReset(lambda: "multiplier" <= max_multiplier)
        .stateChangesOnly()
        .message('{{ index .Fields "count" }} calls to {{ index .Tags "request_type" }} in the last 2 minutes is x{{ index .Fields "multiplier" }} more than normal')

My first warning (in slack) might say…

(orange-icon) 400 calls to login in the last 2 minutes is x10.87 more than normal

When reset kicks-in the new message would be something like…

(green-icon) 36 calls to login in the last 2 minutes is x1.45 more than normal

I guess there’s no way to change the reset message to be something different, i.e. to include the phrase “is back to normal”?

One thought would be to generate the message, or some element of it, in the UDF and pass it back to kapacitor.

How would you guys normally manage the message strings?

#2

The templating language for the message can have conditional statements. So you can wrap the message in an if statement for the current level of the alert.

Something like this:

{{ if eq .Level "OK" }}is normal message here{{ else }}not normal message here{{ end }}
2 Likes