Kapacitor instance got terminated when we have divide by zero in tick script

InfluxDB 1 Kapacitor
1

When we have divide by zero in tick script, Instead of throwing some exception or terminating that particular task, Kapacitor instance got terminated. (Is is happened in both 1.4.0 and 1.5.0 )

Below is the example snippet to reproduce:

dbrp "testing"."autogen"

var current = stream
|from()
.measurement('test')
.groupBy(*)
|window()
.period(1m)
.every(1m)

|alert()
.id('/jobs:jobs/telemetry-measurement:measurement/measurement=test/query=test')
.crit(lambda: (1/0))
.log('/tmp/affectedintf.tmp')

Below are the logs after terminating the particular instance:(docker logs)

2018/06/06 11:51:16 Using configuration at: /etc/kapacitor/kapacitor.conf
panic: runtime error: integer divide by zero

goroutine 5697 [running]:
github.com/influxdata/kapacitor/tick/stateful.glob..func51(0xc4205cf080, 0xc42097e9f0, 0x34b2d80, 0xc420617318, 0x34b2d80, 0xc420617320, 0x0, 0x0, 0x0, 0x0, ...)
	/root/go/src/github.com/influxdata/kapacitor/tick/stateful/evaluation_funcs.go:1037 +0x2e9
github.com/influxdata/kapacitor/tick/stateful.(*EvalBinaryNode).eval(0xc420c68200, 0xc4205cf080, 0xc42097e9f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/root/go/src/github.com/influxdata/kapacitor/tick/stateful/eval_binary_node.go:254 +0xea
github.com/influxdata/kapacitor/tick/stateful.(*EvalBinaryNode).EvalBool(0xc420c68200, 0xc4205cf080, 0xc42097e9f0, 0xc4206b7402, 0x0, 0x0)
	/root/go/src/github.com/influxdata/kapacitor/tick/stateful/eval_binary_node.go:199 +0x3c6
github.com/influxdata/kapacitor/tick/stateful.(*expression).EvalBool(0xc420345fa0, 0xc4205cf080, 0xc4205cf080, 0x2, 0x0)
	/root/go/src/github.com/influxdata/kapacitor/tick/stateful/expr.go:68 +0x46
github.com/influxdata/kapacitor.EvalPredicate(0x34b1e00, 0xc420345fa0, 0x349d720, 0xc42097ea20, 0x7f0c9ea89f20, 0xc4204d9710, 0x1ff3600, 0x0, 0x0)
	/root/go/src/github.com/influxdata/kapacitor/expr.go:25 +0x15d
github.com/influxdata/kapacitor.(*AlertNode).findFirstMatchLevel(0xc420b1aa00, 0x3, 0x0, 0x7f0c9ea89f20, 0xc4204d9710, 0x4, 0xc4201a1180)
	/root/go/src/github.com/influxdata/kapacitor/alert.go:679 +0xe8
github.com/influxdata/kapacitor.(*AlertNode).determineLevel(0xc420b1aa00, 0x7f0c9ea89f20, 0xc4204d9710, 0x0, 0xc4204d9710)
	/root/go/src/github.com/influxdata/kapacitor/alert.go:654 +0x69
github.com/influxdata/kapacitor.(*alertState).BufferedBatch(0xc4200f9570, 0x34b4620, 0xc420b4cd00, 0xcc000000034a7de0, 0xcc741ab5e33628e4, 0xc4205d88c0, 0x7f0c9ea89e18)
	/root/go/src/github.com/influxdata/kapacitor/alert.go:773 +0x2c7
github.com/influxdata/kapacitor/edge.(*timedForwardBufferedReceiver).BufferedBatch(0xc42035d9e0, 0x34b4620, 0xc420b4cd00, 0x1f61640, 0x1ff36e0, 0x1, 0x7f0c9ea89ec8)
	/root/go/src/github.com/influxdata/kapacitor/edge/timed.go:55 +0x62
github.com/influxdata/kapacitor/edge.(*forwardingBufferedReceiver).BufferedBatch(0xc420b4cdc0, 0x34b4620, 0xc420b4cd00, 0x7f0c9ea89ec8, 0xc420b4cdc0)
	/root/go/src/github.com/influxdata/kapacitor/edge/forwarding.go:69 +0x4c
github.com/influxdata/kapacitor/edge.receiveBufferedBatch(0x34aeb00, 0xc420b4cdc0, 0x34b4620, 0xc420b4cd00, 0x0, 0xc420b4ccc0)
	/root/go/src/github.com/influxdata/kapacitor/edge/consumer.go:76 +0x1ad
github.com/influxdata/kapacitor/edge.(*groupedConsumer).BufferedBatch(0xc420bfe000, 0x34b4620, 0xc420b4cd00, 0x7f0c9ea89d70, 0xc420bfe000)
	/root/go/src/github.com/influxdata/kapacitor/edge/grouped.go:99 +0x176
github.com/influxdata/kapacitor/edge.receiveBufferedBatch(0x34aebc0, 0xc420bfe000, 0x34b4620, 0xc420b4cd00, 0xc420b4cd00, 0xc4203d2001)
	/root/go/src/github.com/influxdata/kapacitor/edge/consumer.go:76 +0x1ad
github.com/influxdata/kapacitor/edge.(*consumer).Consume(0xc420345bc0, 0x13, 0xc42097b3c8)
	/root/go/src/github.com/influxdata/kapacitor/edge/consumer.go:53 +0x257
github.com/influxdata/kapacitor/edge.(*groupedConsumer).Consume(0xc420bfe000, 0x2201c37, 0x13)
	/root/go/src/github.com/influxdata/kapacitor/edge/grouped.go:51 +0x33
github.com/influxdata/kapacitor.(*AlertNode).runAlert(0xc420b1aa00, 0x0, 0x0, 0x0, 0xc4208e4f80, 0x0)
	/root/go/src/github.com/influxdata/kapacitor/alert.go:511 +0x5f2
github.com/influxdata/kapacitor.(*AlertNode).(github.com/influxdata/kapacitor.runAlert)-fm(0x0, 0x0, 0x0, 0xc4208e4fa0, 0x0)
	/root/go/src/github.com/influxdata/kapacitor/alert.go:87 +0x48
github.com/influxdata/kapacitor.(*node).start.func1(0xc420b1aa00, 0x0, 0x0, 0x0)
	/root/go/src/github.com/influxdata/kapacitor/node.go:181 +0x8e
created by github.com/influxdata/kapacitor.(*node).start
	/root/go/src/github.com/influxdata/kapacitor/node.go:159 +0x5d

Regards,

Rajesh

2

Hi,

I suffer the same…Should implement a better way to solve this…Return 0? Disable the tick?

3

Hi @nrajesh, @arturo-mondelo,

Can you provide some color around what the scenario is that is causing a divide-by-zero scenario? I’m assuming you’re dividing one value by another value (not hard-coding a divide-by-zero expression like in the example). Given that, are you able to fill null points in your data set? That is the work-around I typically see.

@katy

Sam

4

Hi Sam,

Actually we have 2 fields which are integers. When we have an expression like a/b. It is causing this issue. To avoid this, we have written one more condition on top of this, so that expression will be evaluated only if denominator is not Zero. But, I guess it won’t be feasible when you go for any other use case. So instead of terminating the instance, can we just throw an exception.

Rajesh

1 Like
5

Hi,
Well we are evaluating 5XX errors on the http requests and calculating the error percent of those versus total requests.
Actually, we evaluate only when requests > 0 (WHERE condition) otherwise we suffer instance breaks on kapacitor.
What i mean is that this could be managed in other way…
@Sam you mean set a fill with value 1 if a point arrive with nothing? Would be much performant than restricting the query/Stream?