Notification rule misses latest status changes

InfluxDB 2 Checks & Notifications
, ,
1

Hi there,

I have a monitoring task that outputs OK/CRIT level and I need to send out notification only when there’s a change in the level.

Here’s the current execution timing and monitoring data:

monitoring task:               5min |...OK          CRIT          OK          OK           OK           OK...


notification rule: 10min, 2m offset |...     ^                         ^                         ^        ...

My question is that, I only receive the notification for the OK → CRIT and lost the CRIT → OK message. Essentially on Slack, I see multiple “xxxx level is now CRIT”, and never see it comes back OK.

Here’s my monitoring task:

import "influxdata/influxdb/monitor"
import "experimental"
import "slack"
import "influxdata/influxdb/schema"

option task = {name: "network down", every: 5m}

slack_channel =
    slack["endpoint"](url: "https://hooks.slack.com/services/x/y")

from(bucket: "my-bucket")
    |> range(start: -1d)
    |> filter(fn: (r) => r["_measurement"] == "networking" and r["_field"] == "ping_rtt")
    |> schema.fieldsAsCols()
    |> monitor.deadman(t: experimental.subDuration(d: 5m, from: now()))
    |> monitor.check(
        crit: (r) => r.dead == true,
        ok: (r) => r.dead == false,
        messageFn: (r) => "anything",
        data: {
            _check_name: "Network Status",
            _check_id: "network_status",
            _type: "deadman",
            tags: {deadman: "deadman"},
        },
    )

And here’s my notification task:

import "influxdata/influxdb/monitor"
import "experimental"
import "slack"
import "influxdata/influxdb/schema"

option task = {name: "network down slack alert", every: 10m, offset: 2m}

slack_channel =
    slack["endpoint"](url: "https://hooks.slack.com/services/x/y")

monitor["from"](start: -15m, fn: (r) => r["_check_name"] == "Network Status")
    |> monitor.stateChangesOnly()
    |> map(fn: (r) => ({r with _value: if r["_level"] == "crit" then "down" else "ok"}))
    |> group(columns: ["host"])
    |> first()
    |> slack_channel(
        mapFn: (r) =>
            ({
                channel: "",
                text: "host:${r.host} network status: ${r._value}",
                color: if r["_level"] == "crit" then "danger" else "good",
            }),
    )()

I might misuse the first() function, but how do I notify only the latest level ? What I want to achieve is on Slack I want to see some things go down and later on see them coming up. If I leave out the first(), it’d just spitting up/down up/down multiple times in the channel that doesn’t make sense.

2

Hello @Nam_Giang,
It looks like you want to use this function:

3

Hi Anaisdg,

If you look at my code, I did use such function.

4

Anaisdg, any way to achieve what I wanted, i.e. only notify the most recent status change?

Thanks

5

Anaisdg, any help is very appreciated.

Thanks