In my test setup I have a measurement (a sin wave with small noise) and the pattern repeats each 60s, using a simple anomaly detection tick script that checks if recent values are close to the ones on previous cycles/seasons. (with a error margin based on stddev)
When the anomaly occurs an alarm is triggered as expected, but in the next ‘season/cycle’ the ‘previous values’ does not reflect the ‘normal behavior’ of the measurement, so the comparison will trigger a false positive.
Is there any advice/suggestion to avoid trigger the false positive in the next cycle after a anomaly is detected?
Thanks in advance
The script used is the following:
var query_now = '''SELECT mean("value") FROM "TEST_KPI"."autogen"."foobar"'''
var query_previous = '''SELECT mean("value"), stddev("value") FROM "TEST_KPI"."autogen"."foobar"'''
var offset = 60s
var shift = 64s
var period = 4s
var every = 4s
var fill = 0
var tolerance = 1s
var post = 'http://logstash:8080/kapacitor'
var threshold_multiplier = 2.0
var outputMeasurement = 'foobar_historical'
var now = batch
|query(query_now)
.period(period)
.every(every)
.fill(fill)
var previous = batch
|query(query_previous)
.offset(offset)
.period(period)
.every(every)
.fill(fill)
|shift(shift)
now
|join(previous)
.as('now', 'previous')
.tolerance(tolerance)
.fill(0.0)
|log()
|alert()
.crit(lambda: abs("now.mean" - "previous.mean") > (threshold_multiplier * "previous.stddev"))
.post(post)
And the chart showing the issue is this one, where the green line is the measurement to monitor and the yelow band is the threshold where is considered a ‘normal behaviour’
