In my test setup I have a measurement (a sin wave with small noise) and the pattern repeats each 60s, using a simple anomaly detection tick script that checks if recent values are close to the ones on previous cycles/seasons. (with a error margin based on stddev)
When the anomaly occurs an alarm is triggered as expected, but in the next ‘season/cycle’ the ‘previous values’ does not reflect the ‘normal behavior’ of the measurement, so the comparison will trigger a false positive.
Is there any advice/suggestion to avoid trigger the false positive in the next cycle after a anomaly is detected?
Thanks in advance
The script used is the following:
var query_now = '''SELECT mean("value") FROM "TEST_KPI"."autogen"."foobar"''' var query_previous = '''SELECT mean("value"), stddev("value") FROM "TEST_KPI"."autogen"."foobar"''' var offset = 60s var shift = 64s var period = 4s var every = 4s var fill = 0 var tolerance = 1s var post = 'http://logstash:8080/kapacitor' var threshold_multiplier = 2.0 var outputMeasurement = 'foobar_historical' var now = batch |query(query_now) .period(period) .every(every) .fill(fill) var previous = batch |query(query_previous) .offset(offset) .period(period) .every(every) .fill(fill) |shift(shift) now |join(previous) .as('now', 'previous') .tolerance(tolerance) .fill(0.0) |log() |alert() .crit(lambda: abs("now.mean" - "previous.mean") > (threshold_multiplier * "previous.stddev")) .post(post)
And the chart showing the issue is this one, where the green line is the measurement to monitor and the yelow band is the threshold where is considered a ‘normal behaviour’