How to compare the current series of data with previous series using kapacitor (tick script)

kapacitor

#21

from your example it’s hard to tell what are fields, tags, etc. Are Device, Interfacename, and status tags or fields?


#22

Device and Interface name – tags
Interface Status – Field


#23

Using the following tickscript

dbrp "test"."autogen"

var data = stream
    |from()
        .measurement('test')
        .groupBy(*)
    |changeDetect('status')
    |alert()
        .info(lambda: TRUE)
P0: test,Device=Device1,Interfacename=Interface1 status="UP",MTU=1500 T0
P1: test,Device=Device1,Interfacename=Interface2 status="DOWN",MTU=1500 T1
P2: test,Device=Device2,Interfacename=Interface1 status="DOWN",MTU=1500 T2
P3: test,Device=Device2,Interfacename=Interface2 status="UP",MTU=1500 T3
P4: test,Device=Device1,Interfacename=Interface1 status="DOWN",MTU=1500 T4
P5: test,Device=Device1,Interfacename=Interface3 status="UP",MTU=1500 T5

I get alerts for every point because each point has changed from it’s previous value.

For points P0, P1, P2, P3, and P5 I get an alert because there the state was changed from null to a value; this should only happen at first. After things have been primed it should not occur.

For point P4, I get an alert because the series test,Device=Device1,Interfacename=Interface1 changed from status="UP" in P0 to status="DOWN" in P4.

If I further wrote the data

P6: test,Device=Device1,Interfacename=Interface1 status="DOWN",MTU=1500 T6
P7: test,Device=Device1,Interfacename=Interface1 status="DOWN",MTU=1500 T7
P8: test,Device=Device1,Interfacename=Interface1 status="UP",MTU=1500 T8
P9: test,Device=Device1,Interfacename=Interface3 status="UP",MTU=1500 T9
P10: test,Device=Device1,Interfacename=Interface3 status="DOWN",MTU=1500 T10

I would receive alerts for P8 since and P10, but not for P6, P7, and P9.

Is this the behavior that you’re looking for? If not, what is different?


#24

Thanks @michael

Our requirement is to get whenever there is some change in the state ‘n’ no.of times (which should not include the first one as you mentioned). Though, it is changed from null to some value.

For example, if the interface1 state has changed from UP --> DOWN --> UP --> DOWN–> UP–>DOWN

Here, 3 events happened. Each event is state change from UP–>DOWN

Another, usecase is to identify all the state changes UP --> DOWN --> UP --> DOWN–> UP–>DOWN
here, 5 state changes happened in last 1minute then we should get alert.

Let me know, if you need any other info on this.


#25

For example, if the interface1 state has changed from UP --> DOWN --> UP --> DOWN–> UP–>DOWN

This can be done with

dbrp "test"."autogen"

var data = stream
    |from()
        .measurement('test')
        .groupBy(*)
    |changeDetect('status')
    |alert()
        .info(lambda: "status" == 'DOWN')

and

Another, usecase is to identify all the state changes UP --> DOWN --> UP --> DOWN–> UP–>DOWN
here, 5 state changes happened in last 1minute then we should get alert.

This is possible with the tickscript we had previously

dbrp "test"."autogen"

var data = stream
    |from()
        .measurement('test')
        .groupBy(*)
    |changeDetect('status')
    |alert()
        .info(lambda: TRUE)

The hard part is to make things not trigger for their first value.

I think we can add |where(lambda: count() > 1) so that we skip the first time it happens.

dbrp "test"."autogen"

var data = stream
    |from()
        .measurement('test')
        .groupBy(*)
    |changeDetect('status')
    |where(lambda: count() > 1)
    |alert()
        .info(lambda: TRUE)

I think having the count there for both scripts would be helpful.

Does this achieve what you’re trying to do?


#26

Logic looks fine to me. I will run this with actual data.

Meanwhile, can you answer another question, which is at the top of our discussion in the same thread.

var current = stream
|from()
.measurement(‘test’)
.groupBy()
|window()
.period(1m)
.every(1m)

var old = current
|shift(-1m)

|alert()
.id(’/jobs:jobs/telemetry-measurement:measurement/measurement=test/query=test’)
.crit(lambda: ((‘current.tag1’== ‘old.tag1’)AND(‘current.tag2’==‘old.tag2’)AND(‘current.field1’ != ‘old.field1’)))
.log(’/tmp/out.tmp’)

Especially, i had a doubt at lambda condition: there tag1 and tag2 are tags and field1 is the field. And where the single or double quotes can be added…

It required for us to store in one variable and compare with some other values later.

q1: how to use the tags/fields from the variable?
q2: How to compare with plain string with varible’s field?
ex: temp1.field1==“hello”

more information can be find out at How to compare 2 consecutive series of data and the ways to access/compare variables in Tick?

Due to duplicate in the content it is closed.

It is useful for number of our usecases.


#27

q1: how to use the tags/fields from the variable?
q2: How to compare with plain string with varible’s field?
ex: temp1.field1==“hello”

I’m not sure I follow, can you give me an example with the data in line protocol and an example of what output you want.


#28

We have 2 Devices: Device1, Device2. device-id and interface-name are tags and all others fields.

We have 3 measurements: one for device1 neighbor details, one for device2 neighbor details, another one for MTU details of device1, device2

First, we have to identify the connected interfaces between them. the condition will be local interface-name or port-id is equal to remote sides receiving-interface-name.

Then we have to check the other value (MTU) on those interfaces if the value is different then raise a critical alert.

Measurement1:
m1, device-id=Device1,remote-device=Device2,port-id=Interface1,receiving-interface-name=Interface1,timestamp= T1 --> S2**
m1, device-id=Device1,remote-device=Device2,port-id=Interface2,receiving-interface-name=Interface4,timestamp= T1 --> S3
m1, device-id=Device1,remote-device=Device2,port-id=Interface3,receiving-interface-name=Interface5,timestamp= T1 --> S4
m1, device-id=Device1,remote-device=Device2,port-id=Interface6,receiving-interface-name=Interface8,timestamp= T1 --> S4**

Measurement2:
m2, device-id=Device2,remote-device=Device1,port-id=Interface1,receiving-interface-name=Interface1,timestamp= T1 --> S1 **
m2, device-id=Device2,remote-device=Device1,port-id=Interface2,receiving-interface-name=Interface19,timestamp= T1 --> S2
m2, device-id=Device2,remote-device=Device3,port-id=Interface4,receiving-interface-name=Interface2,timestamp= T2–> S3
m2, device-id=Device2,remote-device=Device1,port-id=Interface5,receiving-interface-name=Interface11,timestamp= T1 --> S4
m2, device-id=Device2,remote-device=Device1,port-id=Interface8,receiving-interface-name=Interface6,timestamp=T1–> S4**

Now, we got the neighboring devices based on the above 2 measurements. We have to use those device-id’s and port-id’s with below measurement to get MTU mismatched interfaces. Now, we have to check the MTU value of interface1 of device1 and interface1 of devices2 && interface6 of Device1 and interface8 of device2. If there is a mismatch we have to report it.

Here, even we can use a single measurement(m1) alone and compare the MTU values with m3.

Measurement3:
m3, device-id=Device1,interface-name=Interface1,MTU=1650, timestamp=T1 --> S1**
m3, device-id=Device1,interface-name=Interface2,MTU=1550, timestamp=T1 --> S2
m3, device-id=Device1,interface-name=Interface3,MTU=1600, timestamp=T1 --> S3
m3, device-id=Device1,interface-name=Interface6,MTU=1550, timestamp=T1 --> S4
m3, device-id=Device2,interface-name=Interface1,MTU=1600, timestamp=T1 --> S5**
m3, device-id=Device2,interface-name=Interface2,MTU=1550, timestamp=T1 --> S6
m3, device-id=Device2,interface-name=Interface4,MTU=1600, timestamp=T1 --> S7
m3, device-id=Device2,interface-name=Interface8,MTU=1550, timestamp=T1 --> S8

**

Result will be Device1, interface1,MTU=1650 and Device2,Interface1,MTU=1600.

**

We have to use a stream instead of batch. If it is not possible with streaming node then we can go for a batch.

dbrp “ncx”.“autogen”
var output = batch
|query(‘select “port-id”,“receiving-interface-name”,“neigh-device-id”,“device-id” from “m1”’)
.period(1m)
.every(1m)
.groupBy(*)

var output_neighbor = batch
|query(‘select “port-id”,“receiving-interface-name”,“neigh-device-id”,“device-id” from “m2”’)
.period(1m)
.every(1m)
.groupBy(*)

var mtu = ‘select “mtu” from “m3” where interface-name=‘output’.“receiving-interface-name”’

var intf_name = ‘select “interface-name” from “m2” where “output.device-id == output_neighbor.device-id” AND “output.port-id=output_neighbor.receiving-interface-name” AND “output.receiving-interface-name=output_neighbor.port-id”’

var mtu_neighbor = ‘select “mtu” from “m3” where interface-name=’+intf_name
|alert()
.crit(lambda: (mtu == mtu_neighbor))
.log(’/tmp/out.tmp’)

Here, we want to know how to access a particular field/tag from variable.

> var mtu = 'select “mtu” from “m3” where interface-name=‘output’.“receiving-interface-name”'

here we are accessing the receiving-interface-a name from variable output.

Is there anyway to write nested select queries in Influx?


#29

@michael I have tried with script which you have provided… But no luck.

It is giving alert only for the first interface which matches that condition. Only one time the alert has been triggered. There are continuous changes on the interfaces state.

Let’s say Interface1 status got changed at T2 and then there are several other interfaces where their status got changed later like Inteface2, Interface3…etc… But the alert has been triggered only for the Interface1.

can you check on this.

dbrp “atom”.“autogen”

var data = stream
|from()
.measurement(‘InterfaceFlapping’)
.groupBy(*)
|changeDetect(‘openconfig-interfaces:interfaces/interface/subinterfaces/subinterface/state/admin-status’)
|where(lambda: count() > 1)
|alert()
.info(lambda: TRUE)
.log(’/tmp/result.tmp’)


#30

Can you provide data in line protocol to reproduce the issue.


#31

@michael – its working fine. connection was flaky between kapacitor and influxdb. Sorry for that:joy:

can you answer How to compare the current series of data with previous series using kapacitor (tick script)


#32

@nrajesh I’ve read over this a few times and I cant quite figure out what the issue is. Could you try restating the issue?


#33

@michael – The question is about when you store some result in a variable. How to use that in next steps to process or compare?

For example, I have one variable which is the result of certain condition. It has several fields, tags.

var result = select x,y from test where z>20 (It’s not the exact syntax)

Now, how to use this result to compare with the next set of statements

check==“result”.‘x’ ?? (How to use the interim resulted variable tag and fields, single quote or double quote or no quotes?)

if it is a single value, how to compare?

It is Just about the syntax


#34

In general, this is not possible with kapacitor. This can be achieved using certain types of nodes for specific use cases. But from the sounds of it, I think you might need to use a UDF.


#35

@michael Hi Michael, does the tick script could work properly? I tried use it to run on the data you provided but do not get any result at all. I just changed the names of database and measurement. So, how to test the node changeDetect? Does the version of 1.4 has the corresponding node of changeDetect? Thanks in advance for your help. :slight_smile:

dbrp "test"."autogen"

var data = stream
    |from()
        .measurement('test')
        .groupBy(*)
    |changeDetect('status')
    |alert()
        .info(lambda: TRUE)
P0: test,Device=Device1,Interfacename=Interface1 status="UP",MTU=1500 T0
P1: test,Device=Device1,Interfacename=Interface2 status="DOWN",MTU=1500 T1
P2: test,Device=Device2,Interfacename=Interface1 status="DOWN",MTU=1500 T2
P3: test,Device=Device2,Interfacename=Interface2 status="UP",MTU=1500 T3
P4: test,Device=Device1,Interfacename=Interface1 status="DOWN",MTU=1500 T4
P5: test,Device=Device1,Interfacename=Interface3 status="UP",MTU=1500 T5
P6: test,Device=Device1,Interfacename=Interface1 status="DOWN",MTU=1500 T6
P7: test,Device=Device1,Interfacename=Interface1 status="DOWN",MTU=1500 T7
P8: test,Device=Device1,Interfacename=Interface1 status="UP",MTU=1500 T8
P9: test,Device=Device1,Interfacename=Interface3 status="UP",MTU=1500 T9
P10: test,Device=Device1,Interfacename=Interface3 status="DOWN",MTU=1500 T10

#36

I think it should work properly. Can you explain in a bit more detail what you tried and what you mean by it didn’t work properly.


#37

@michael Thanks for your help. I want to know how to get the same result in the version 1.4 with the corresponding node of changeDetect because I did not find any node related to changeDetect in the version 1.4? For the incorrect running, I made a mistake yesterday. I’m sorry.


#38

Is there a reason you cannot upgrade to 1.5?


#39

@michael Thanks for your kindly help. Because our systems are using v1.4 and I also want to detect changes between several fields which supported by a higher version over v1.5.1, I found another way to implement. But I cannot get the last point time field of a group. something like following.
I want to get the last timestamp of each group then have a comparison with the current one point, but the following script does not have any output. So, how can I get last point time of each group? Thank you.

dbrp “test”.“autogen”

var data = stream
|from()
.measurement(‘test’)
.groupBy(*)
|last(‘time’)
|alert()
.info(lambda: TRUE)
.log(’/tmp/time_test.log’)


#40

I don’t think this functionality is possible without using changeDetect in a newer version of kapacitor.