I have tried multiple different versions of this. doing a stream, batch, putting the where in/out of the query, putting the groupby in/out of the query, but i still see to only get the deadman to run.
The data is coming from Heapster.
I know the data exists, graphs for grafana look fine, and queries directly to influx return results.
This alert gives me a deadman for each host that exists.
nodeCPUAlert.k8s.default.batch.tick: |-
var windowDuration=5m
var data = batch
|query ('''
SELECT mean("value") as stat
FROM "k8s"."default"."cpu/node_utilization"
''')
.period(windowDuration)
.every(windowDuration)
.groupBy(time(1m),'hostname')
var alert = data
|eval(lambda: "stat" * 100)
.as('mean_percent')
alert
|alert()
.id('**High CPU Usage on {{ index .Tags "hostname" }}**')
.message('{{ .ID }} detected. {{ index .Fields "mean_percent" }} is above 80 for 5 minutes')
.warn(lambda: "mean_percent" >= 80)
.warnReset(lambda: "mean_percent" < 75)
.stateChangesOnly(1d)
.slack()
.log('/tmp/mem_alert_log.txt')
alert
|deadman(1.0, 10m)
.id('**High CPU Usage Alarm for {{ index .Tags "hostname" }}**')
.message('''{{ .ID }} is not reporting CPU Metrics''')
.critReset(lambda: "emitted" > 0.0)
.stateChangesOnly(1d)
.slack()
.log('/tmp/mem_alert_log.txt')