stateChangesOnly() does not work as expected

delapuentem · December 24, 2020, 7:26am

I have a task that checks if a value exceeds a threshold and reports by slack.

This task has an interval of 10 minutes, the problem is that it notifies every 10 minutes even with the same status.

I am storing in my bucket a _level (tag) column with critical and ok values.

But when I use monitor.stateChangesOnly () it never notify by slack when _level changes from critical to ok or ok to critical. Does monitor.stateChangesOnly () work the same as statechangeonly in kapacitor?

I do not know what I’m doing wrong.

The Slack notification task:

import "influxdata/influxdb/monitor"
import "slack"

option task = {name: "notify slack", every: 10m}

toSlack = slack.endpoint(url: "mySecretWebhook", token: "")

notify = from(bucket: "alerts")
    |> range(start: -task.every)
    |> filter(fn: (r) => (r._measurement == "alerts" and r._field == "temp"))     
    |> keep(columns: ["_time", "_value", "_field", "_level", "location", "alertid"])
    |> group(columns: ["location", "_level", "alertid"])
    |> last()

notify
    |> monitor.stateChangesOnly()
    |> toSlack(mapFn: (r) =>
        ({r with channel: "#alertas-influxdb", text: "${r.alertid} ${r.location} ${r._level} Value: " + string(v: r._value) + " Threshold: 18", color: "info"}))()

Anaisdg · December 28, 2020, 6:33pm

Hello @delapuentem,
Just to confirm, stateChangesOnly is notifying when the level has not changed? That’s not right. It should only notify when the state changes as per documentation:

Are there values after

notify
    |> monitor.stateChangesOnly()

If your state hasn’t changed I don’t expect to see any values. I tested it for myself on InfluxDB cloud with system plugin. I set all “_level” values to “critical”:

import "influxdata/influxdb/monitor"
from(bucket: "my-bucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "cpu")
  |> filter(fn: (r) => r["_field"] == "usage_system")
  |> filter(fn: (r) => r["cpu"] == "cpu-total")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> map(fn: (r) => ({  r with
    _level:
      if r._value >= 0.0 then "critical"  else "normal" }))
  |> monitor.stateChangesOnly()

No values are returned.

What message are you getting to slack?

delapuentem · January 1, 2021, 9:29am

Hello @Anaisdg , thank you very much for your answer !! Sorry, I’ll try to explain myself better.

I have this table with this data.

I understand that if I apply the stateChangesOnly () function it would return the “crit” with value 14 and the “ok” with the value 2, according to the documentation.

But it doesn’t work as I expect.

import "influxdata/influxdb/monitor"

from(bucket: "alertas")
  |> range(start: -1h)
  |> filter(fn: (r) => r["_measurement"] == "alertas")
  |> filter(fn: (r) => r["_field"] == "value")
  |> last()
  |> monitor.stateChangesOnly()

I have also tried this, as in the documentation, but it has not given me good results.

import "influxdata/influxdb/monitor"

from(bucket: "alertas")
  |> range(start: -1h)
  |> monitor.stateChangesOnly()

Also if I try stateChanges(toLevel: “ok”) it returns results. But stateChanges(toLevel: “crit”) doesn’t.

stateChanges(toLevel: “ok”)

stateChanges(toLevel: “crit”)

I am using Version 2.0.3
Thanks and happy new year !

Anaisdg · January 4, 2021, 4:37pm

Hello @delapuentem,
That’s because you’re using last() so you’re limiting your data to the one line where the status is “ok”. Please try

import "influxdata/influxdb/monitor"

from(bucket: "alertas")
  |> range(start: -1h)
  |> filter(fn: (r) => r["_measurement"] == "alertas")
  |> filter(fn: (r) => r["_field"] == "value")
  |> monitor.stateChangesOnly()

delapuentem · January 4, 2021, 4:54pm

@Anaisdg My data:

With stateChangeOnly() :

the most recent “ok” status is at 21:27, but it shows me another time. Also, the “crit” status does not show it. I am doing something wrong?

Thanks!

Anaisdg · January 4, 2021, 5:46pm

Can you please try using the exact same query as above with the pivot() and the group() and then apply the stateChangesOnly function.

delapuentem · January 4, 2021, 7:12pm

hi @Anaisdg, same result, i tried with yield(“last”) and without yield(“last”) :

Could it be a bug in the monitor.stateChangesOnly() function ?

Thanks !

Anaisdg · January 4, 2021, 7:25pm

Hello @delapuentem,
I’m not convinced it is yet. Can you please export this data to csv (please note the export CSV button in the bottom right) and share those 5 points with me? Thank you. From this:

delapuentem · January 4, 2021, 7:52pm

@Anaisdg sure!

I have added the .txt extension to the file, as .csv was telling me it was not authorized.

2021-01-04-20-49_chronograf_data.csv.txt (840 Bytes)

Thank you very much for your time and your help

Anaisdg · January 4, 2021, 8:15pm

Hello @delapuentem,
With the data you sent me, the output makes sense and stateChangesOnly() works.

import "csv"
import "influxdata/influxdb/monitor"
csvData = "#group,false,false,false,false,false,false,false,false,false,false
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,string,string,string,double,double
#default,_result,,,,,,,,,
,result,table,_time,_start,_stop,_level,_measurement,host,threshold,value
,,0,2021-01-03T20:26:34Z,2021-01-03T19:48:56.599321105Z,2021-01-04T19:48:56.599321105Z,crit,alertas,host1,10,14
,,0,2021-01-03T20:26:49Z,2021-01-03T19:48:56.599321105Z,2021-01-04T19:48:56.599321105Z,crit,alertas,host1,10,12
,,0,2021-01-03T20:26:55Z,2021-01-03T19:48:56.599321105Z,2021-01-04T19:48:56.599321105Z,crit,alertas,host1,10,16
,,0,2021-01-03T20:26:44Z,2021-01-03T19:48:56.599321105Z,2021-01-04T19:48:56.599321105Z,ok,alertas,host1,10,2
,,0,2021-01-03T20:27:02Z,2021-01-03T19:48:56.599321105Z,2021-01-04T19:48:56.599321105Z,ok,alertas,host1,10,4
"
csv.from(csv: csvData)
|> monitor.stateChangesOnly()

It only returns the value where the state changes at 2021-01-03T20:26:44.000Z because that’s when the state changes from “ok” to “crit”.

delapuentem · January 4, 2021, 8:46pm

But if so, it should show “ok” at 21:27:02 GMT +1

Also, I have inserted a new value “crit”, and it keeps showing me “ok” from 21:26:44 GMT +1, and it should show “crit” right?

And stateChanges(toLevel: “ok”) show me the “ok” row with the time 21:27:02 GMT +1
but stateChanges(toLevel: “crit”) show me “No results”

fluxator · February 5, 2021, 9:19am

Hi @Anaisdg
I also think there is some strange behaviours with the function “stateChangesOnly()” and “stateChanges()”.

My UseCase: I monitor VMs and I want to get the timestamp each time the VM was moved to another host (each the field “host” changed).
So, if I understand the documentation right, I can use the this function.

So with my data I want each row thats marked:

But I get “No Results” with following query:
import “influxdata/influxdb/monitor”

from(bucket: "telegraf") 
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop) 
  |> filter(fn: (r) => r._measurement == "hyperv_vid" and 
    r._field == "Physical_Pages_Allocated" and 
    r.instance == "TestRHE01" 
  ) 
  |> group() 
  |> duplicate(as: "_level", column: "host") 
  |> monitor.stateChangesOnly()

Can you help me, or is this a bug?

Anaisdg · February 5, 2021, 4:53pm

Hello @fluxator,
Hmm that is quite odd. Can you please export the data that you shared in a screen shot to annotated csv with the button in the UI near the submit button in the Data Explorer?

fluxator · February 8, 2021, 6:57am

Thanks for your reply, unfortunaltey I can’t upload my csv. That’s why I paste the content here:

#group,false,false,true,true,false,false,false,false,false,false,false,false 
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string,string,string 
#default,_result,,,,,,,,,,, 
,result,table,_start,_stop,_time,_value,_field,_measurement,host,instance,objectname,_level 
,,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:43:00Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9406,TestRHE01,Hyper-V VM Vid Partition,FPPW9406 
,,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:43:31Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9406,TestRHE01,Hyper-V VM Vid Partition,FPPW9406 
,,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:45:01Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9406,TestRHE01,Hyper-V VM Vid Partition,FPPW9406 
,,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:46:31Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9406,TestRHE01,Hyper-V VM Vid Partition,FPPW9406 
,,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:44:01Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9407,TestRHE01,Hyper-V VM Vid Partition,FPPW9407 
,,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:44:30Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9407,TestRHE01,Hyper-V VM Vid Partition,FPPW9407 
,,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:45:31Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9407,TestRHE01,Hyper-V VM Vid Partition,FPPW9407 
,,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:46:00Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9407,TestRHE01,Hyper-V VM Vid Partition,FPPW9407

fluxator · February 15, 2021, 9:47am

@Anaisdg are you able to reproduce the issue?
Maybe my CSV is not good to reproduce the issue because of the field “_level”.

Here is a new CSV with the same data but without the “_level”:

#group,false,false,true,true,false,false,false,false,false,false,false
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string,string
#default,_result,
,result,table,_start,_stop,_time,_value,_field,_measurement,host,instance,objectname
,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:43:00Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9406,TestRHE01,Hyper-V VM Vid Partition
,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:43:31Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9406,TestRHE01,Hyper-V VM Vid Partition
,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:45:01Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9406,TestRHE01,Hyper-V VM Vid Partition
,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:46:31Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9406,TestRHE01,Hyper-V VM Vid Partition
,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:44:01Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9407,TestRHE01,Hyper-V VM Vid Partition
,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:44:30Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9407,TestRHE01,Hyper-V VM Vid Partition
,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:45:31Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9407,TestRHE01,Hyper-V VM Vid Partition
,0,2021-02-05T07:43:00Z,2021-02-05T07:47:00Z,2021-02-05T07:46:00Z,1050624,Physical_Pages_Allocated,hyperv_vid,FPPW9407,TestRHE01,Hyper-V VM Vid Partition

fluxator · February 25, 2021, 7:23am

@Anaisdg any news on this?

Anaisdg · February 25, 2021, 4:40pm

Hello @fluxator,
I’m sorry for the delay. I am based in Texas, and didn’t have power for 10 days.
It looks like a bug to me, sorry to say. Can you please submit an issue

Thank you.

delapuentem · October 26, 2021, 3:05pm

For those who want to follow this topic, I have recently returned to this without results and I have opened a thread where @Anaisdg indicates.

stateChangesOnly() does not work as expected · Issue #4179 · influxdata/flux (github.com)

Thanks.

delapuentem · October 29, 2021, 6:53am

After opening an issue on Github, I found the solution thanks to @wolffcm in this post. And it is adding |> sort (columns: ["_time"]) to the query.

It would look like this:

import "influxdata/influxdb/monitor"

from(bucket: "monitoring")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
    |> filter(fn: (r) =>
        (r._measurement == "http_response" and r._field == "result_code"))
    |> filter(fn: (r) =>
        (r["server"] == "https://www.mywebpage.org"))
    |> keep(columns: ["_time", "_value", "_field", "server"])
    |> group(columns: ["server"])
    |> map(fn: (r) => ({  r with
    _level:
      if r._value > 0 then "crit"  else "ok" }))
    |> sort(columns: ["_time"])      
    |> monitor.stateChangesOnly()

And so it returns only the last change change of each state like the documentation.

Many thanks also to @Anaisdg for the help!

fluxator · November 24, 2021, 1:13pm

Unfortunately this does not work for my usecase But thanks anyway

Topic		Replies	Views
Monitor State Change Not Working Correctly on Notification Rule Fluxlang	1	489	February 24, 2022
How to send alerts only once InfluxDB 2 influxdb , tasks	2	681	February 4, 2022
How to use stateChangesOnly() Fluxlang	7	1185	March 16, 2022
Notification rule misses latest status changes Checks & Notifications influxdb , checks , tasks	4	1047	January 25, 2022
Simple task alert slack example InfluxDB 2 flux	3	1381	December 1, 2020

stateChangesOnly() does not work as expected

Related Topics