Cannot get monitor.check function to write to _monitoring Bucket

InfluxDB 1 Kapacitor
1

Hello guys I’m new to InfluxDB and chances are I might be overlooking something very simple.

Basically I’m trying to do an alert if a data point gets received, like a reverse deadman.

I have created a deadman alert and have gathered the flux code using the check/{checkID}/query API endpoint and edited it to my specification. I went through mutliple reformating sessions and ended up rewriting the whole thing myself based on the examples in the knowlegde base. The problem is that even though the task succesfully completed without errors with various versions of my flux code but the monitor.check function has not once written a single thing into the _monitoring Bucket. There is no entry for my _checkID.

This is my current code, can someone help me figure out what I’m missing?

import “influxdata/influxdb/monitor”

import “experimental”

import “influxdata/influxdb/v1”

option task = {name: “testalarm deadman2”, every: 5m}

from(bucket: “Smart Maintenance”)

|> range(start: -10m)

|> filter(fn: (r) =>

    (r["labels_description"] == "Testlabel"))

|> filter(fn: (r) =>

    (r["_field"] == "event_eventId"))

|> v1.fieldsAsCols()

|> monitor.deadman(t: experimental.subDuration(from: now(), d: 90s))

|> monitor.check(

    ok: (r) =>

        (r["dead"]),

    crit: (r) =>

        (r["dead"] == false),

    messageFn: (r) =>

        ("Check: ${r._check_name} is: ${r._level}"),

    data: {

        _check_id: "0000000000000001",

        _check_name: "testalarm deadman2",

        _type: "deadman",

        tags: {},

    },

)
2

bumping this thread because I have more than one use case where it is necessary to alert based any record being received… the deadman check i created in the UI does work (if no events received set status to OK) but as soon as I do receive an event the check doesn’t work any more either. is it possible that I’m running into a logic problem? similarly i tried to do a threshold check based on the “count” aggregate function but this always runs into “InfluxDB issues” and no data can be displayed.

3

Hi tcasanova,

I was playing around with monitor.deadman too.
The below worked for me - I’ve swapped the logic around because I want to alert on no data. The problem I had is this only works when there has been some data in the range. For your use case, if the last time you see a datapoint is outside of the your “start: -10m” range, then your query might return “no results” and the monitor.check doesn’t do anything.
I was running the entire task code in the query window - not sure whether you have done this, I don’t think you can rely on seeing “success” in the task logs.

    |> monitor.deadman(t: experimental.subDuration(d: 5m, from: now() ))
    |> group(columns: ["_measurement"])
    |> yield()
    |> monitor.check(
      ok: (r) => (r["dead"] == false),
      crit: (r) => (r["dead"] == true),
      messageFn: (r) => ("Check: ${r._check_name} is: ${r._level}"),
      data: data,
    )

The “group” statement avoids this error:

runtime error @28:8-33:6: check: found column "_field" in the group key; experimental.to() expects pivoted data

and the yield() statement just provides some useful output - you can check you have the “dead” column as expected before you see output from the monitor.check.
Suggest you run the whole lot in the query window - it will still write to the _monitor measurement and you can see exactly what is - or isn’t - happening.

Hope this helps,
Tom

By the way, I have defined my “data” after the import statements, like:

data = {
  _check_id: "0000000000000001",
  _check_name: "testalarm deadman2",
  _type: "deadman",
  tags: {},
 }
1 Like
4

Hi thopewell,

Thank you for your in-depth reply, your idea of using the group function is very interesting.
I have since tried to avoid the Null/No Data situation by using the fill() function, that’s when I discovered this issue that has been known to InfluxDB since 2016 but noone has feeled compelled to solve yet:

Seems like influxdb has a general problem when working with queries that return no data.
I have decided to give Grafana a try and as it seems it was just what I needed for my project.

Not only can you define how it should handle queries that return no data, you can also share dashboards and enable anonymous access, which was necessary for my use case.

Basically I’m now running tasks that convert the string values of my events that I want to alert on into numerical values and I am alerting based on that value within grafana. In case the query returns no data I was able to define to “set state to OK” within grafana.

I think InfluxDB Cloud has quite a way to go before it can be used as an “All-In-One” solution but I’m very happy how it works together with Grafana now.

5

Just wanted to say thanks for the conversation. It helped me figure out why it wasn’t reporting anything. I have some values that don’t get reported often but I figured it would write something somewhere. Using the yield function helped breakdown the steps.

As for InfluxDB UI vs Grafana, Grafana is the only way to go right now for end users. I was using chronograf for exploring data and running some internal graphs but the UI for 2.0 needs some time to mature more. I will use it for myself but I don’t want any end users playing with it.