Error seen in logs for Kapacitor scripts

1

I have where condition in Tick script where it is checking the process_name and alerting based on that. I get the correct alert, However i see continuous errors in the logs
E! error while evaluating WHERE expression: no field or tag exists for process_name

Please guide.

2

Post your tick script, might be easier for us to debug :wink:

3

Thank you. I see the error with other tags also I have included under global tags, e.g dc and user. Below is the script for process_name that I have reported in this thread.

stream
|from().database(‘telegraf’).measurement(‘procstat’).groupBy(‘host’,‘process_name’).where(lambda: “exe” == ‘audit’ AND “process_name” == ‘auditd’)
|deadman(0.0, 5m)
|alert()
.id(‘deadman/{{ index .Tags “host” }}’)
.details(’ Exe {{ index .Tags “exe” }} ,processname: {{ index .Tags “process_name” }} on Node {{ index .Tags “host” }} {{ if ne .Level “OK” }}is down. Seve
rity:1{{ else }}is back online. Severity:0{{ end }}’)

4

Looks like you are just trying to determine if you get no datapoints in a window. You should be able to get this just using the deadman, and do not require the alert node.

var data = stream
|from()
	.database('telegraf')
	.measurement('procstat')
	.groupBy('host','process_name')
    |log()
    |eval(lambda: "process_name")
       .as('process_name')
       .keep()
       .quiet()
   |log()
	.where(lambda: "exe" == 'audit' AND "process_name" == 'auditd')
data
	|deadman(0.9, 5m)
		.message('Exe {{ index .Tags "exe" }} ,processname: {{ index .Tags "process_name" }} on Node {{ index .Tags "host" }} {{ if ne .Level "OK" }}is down. Severity:1{{ else }}is back online. Severity:0{{ end }}')
     |log()

  1. The cause of errors would likely be exactly that there is no tag or field called “process_name”. I would guess that there are measurements that called “procstat” that do not have a “process_name” tag set. you can filter that as above using the eval() node. You can verify that with log() or doing a query in the db directly

  2. deadman will fire if it receives less than N data points in the window. I have no idea what will happen if you ask for 0 I, looked at the code a while back. I have never seen a deadman feed into an alert node, I guess that works.

  3. alert nodes normally have a lambda clause in them as below as an example
    |alert()
    .stateChangesOnly()
    .all() //Check that all points in the window against the lambda
    .warn(lambda: “jvm_memory_pressure_maximum” > elasticsearch_memory_pressure_warn_level)
    .crit(lambda: “jvm_memory_pressure_maximum” > elasticsearch_memory_pressure_critical_level)
    again you only need the deadman in your case

  4. the log() node is extremely helpful for debugging tick scripts. I would add them after each node in your script and that will really help you.

1 Like
5

Thank you. Couple of questions

  • I need alert to send the data to next level. I am using .tcp for that. What do you suggestion for this.
  • What does .log() provide.
  • I know deadman(0,5m) means that if there is no result, then alert. But not sure what is 0.9 in deadman represent.

Please guide.

Thanks,

6
  • You can reference the TCP directly in deadman, as it uses alert() under the covers. See below
  • Log() just logs to the kapacitor logs, but will show you all the measurement points, including their tags, that will help you to debug things.
    -You can ignore the deadman 0 vs 0.9. I use 0.9 in most of my stuff as I expect it not to fire when it gets 1 point in a time window, see Deadman fires on "<=", intentional? · Issue #1179 · influxdata/kapacitor · GitHub

Here is what I would do (untested)

var data = stream
|from()
	.database('telegraf')
	.measurement('procstat')
	.groupBy('host','process_name')
|log()
|eval(lambda: "process_name")
   .as('process_name')
   .keep()
   .quiet()
|log()
|where(lambda: "exe" == 'audit' AND "process_name" == 'auditd')


#Data --> Deadman
data
|deadman(0.0, 5m)
		.message('Exe {{ index .Tags "exe" }} ,processname: {{ index .Tags "process_name" }} on Node {{ index .Tags "host" }} {{ if ne .Level "OK" }}is down. Severity:1{{ else }}is back online. Severity:0{{ end }}')
		.tcp(<TCP STRING>)

You can of course remove the log() nodes after debugging.

1 Like
7

I am getting error.

invalid TICKscript: line 9 char 2: no property method “where” on *pipeline.LogNode, but chaining method does exist. Use ‘|’ operator instead: ‘node|where(…)’.

8

Tried some changes , and it worked . The working script as below. Thank you so much. Question : Lets say I have multiple tags that I want to put where condition on, so do I need to include multiple evals. Could you please provide me example how would it look like.

stream
|from().database(‘telegraf’).measurement(‘procstat’).groupBy(‘host’,‘process_name’)
|eval(lambda: “process_name”)
.as(‘process_name1’)
.keep()
.quiet()
|where(lambda: “exe” == ‘audit’ AND “process_name1” == ‘auditd’)
|deadman(0.0, 5m)
|alert()
.id(‘deadman/{{ index .Tags “host” }}’)
.details(’ Exe {{ index .Tags “exe” }} ,processname: {{ index .Tags “process_name” }} on Node {{ index .Tags “host” }} {{ if ne .Level “OK” }}is down. Severity:1{{ else }}is back online. Sev
erity:0{{ end }}’)
.stateChangesOnly()
.info(lambda: TRUE)

9

If I understand your question properly, if you want to add more conditions and make certain that the tags are present, you can simply extend the eval() and where() nodes.

Something like the following.

|eval(lambda: "process_name", lambda: "tag2", lambda: "tag3")
.as('process_name1', 'tag2', 'tag3')
.keep()
.quiet()
|where(lambda: "exe" == 'audit' AND "process_name1" == 'auditd' AND "tag2" == 'foobar')
2 Likes
10

Yes, This was what I was looking for. Thank you. :slight_smile: