Hi,
Looks like Deadman alert works well for many users, but I cannon figure several moments.
I’ve created 1 Alert Rule to monitor with net_response Telegraf plugin several web ports - 80 and 81. Alert rules created with a help of Chronograf WebGUI.
When the is no data (here I mimic this with disabling 81 port with firewall), it sends one alert (in my case it writes some info to file /tmp/alert_test.log).
After this I enable monitored and data comes in. Then, after several minutes, when I disable port 81 - nothing happens. Should I clean my alert somehow?
When I am created custom rule based on auto generated one it works the same - alerting me only once:
Custom alert based on auto generated code
var whereFilter = lambda: (“port” == ‘80’ OR “port” == ‘81’) AND (“server” == ‘1.1.2.1’)
var name = 'Rule-4’
var message = '{{.Level}} - {{.Time}} no data received from {{ index .Fields “value” }} during 1 minute. did the it crash?'
var idVar = name + ':{{.Group}}‘
var idTag = ‘alertID’
var data = stream
|from()
.database(‘telegraf’)
.retentionPolicy(‘autogen’)
.measurement(‘net_response’)
.groupBy([‘port’])
.where(whereFilter)
|deadman(1.0, 30s)
.stateChangesOnly()
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(‘level’)
.messageField(‘message’)
.durationField(‘duration’)
.log(’/tmp/alert_test.log’)
Thanks in advance.
P.S. Is there way to alert when data begin to come?
