I am trying to setup an alert with template that will use both the load service and the CLI commands.
We need the load service so it will start automatically when we deploy the box and the CLI so users with only the permissions to run them (and not restarting the whole service) would be able to change the json parameters and re-define the alert.
The files I am using are:
├── kapacitor.conf
├── kapacitor_default.conf
└── load
├── handlers
│ └── childprocs.yaml
├── tasks
│ └── kp_childprocs.json
└── templates
└── kp_abnormal_state_percent_template.tick
kp_abnormal_state_percent_template.tick
// Which measurement to consume
var measurement string// Optional where filter
var where_filter = lambda: TRUE// Optional list of group by dimensions
var groups = [ * ]// Expression to calculate the mean and use in the state comparison
var lambda_calc lambda// Which field to process to trigger the alert
var stateLambda lambda// How long this should happen before alerting
var stateDuration = 1m// Critical criteria, the change of state
var crit = lambda: “state_duration” > 1// How much data to window
var window = 5m// Application name producing the alert
var application = ‘Unknown App’// The topic to send the alerts
// var alert_topic topic// The slack channel for alerts
var slack_channel = ‘#alerts_c’// Emoji to use in slack message
var slack_emoji = ‘’// Local file to write the alerts
var log_alerts_file = ‘/tmp/alerts2.dat’// Message the modified part
var message_part = 'Mean Lag: ’// Topic to post the alerts
var topic = ‘state_percent_alert’dbrp “stats”.“raw_data”
stream
|from()
.measurement(measurement)
.where(where_filter)
|groupBy(groups)
|window()
.period(window)
.every(window)
|eval(lambda_calc)
.as(‘field_calc’)
|mean(‘field_calc’)
.as(‘mean_field’)
|stateDuration(stateLambda)
.unit(stateDuration)
|alert()
.idTag(application)
.message(‘{{ .Time }} - {{ .Level }} "’ + message_part + ‘" {{ index .Fields “mean_field” | printf “%0.2f” }} in "’ + application + ‘"’)
.details(‘’)
.crit(crit)
.log(log_alerts_file)
.topic(topic)
childprocs.yaml
topic: childprocs
id: childprocs
kind: slack
match: changed() == TRUE
options:
channel: ‘#alerts_d’
kp_childprocs.json
{
“template-id”: “kp_abnormal_state_percent_template”,
“vars”: {
“measurement”: {“type” : “string”, “value” : “appserv.childProcStats” },
“where_filter”: {“type”: “lambda”, “value”: “"app" =~ /x_server/”},
“groups”: {“type”: “list”, “value”: [{“type”:“string”, “value”:“app”}]},
“lambda_calc”: {“type”: “lambda”, “value” : “"procs_used" / "procs_avail" * 100” },
“stateLambda”: {“type” : “lambda”, “value” : “"mean_field" > 80” },
“stateDuration”: {“type” : “duration”, “value” : “1s”},
“crit”: {“type” : “lambda”, “value” : “"state_duration" > 1” },
“window”: {“type” : “duration”, “value”: “10s” },
“application”: {“type” : “string”, “value” : “x_server” },
“message_part”: {“type” : “string”, “value” : "Child proc usage above 80% for more than 1 s " },
“topic”: {“type”:“string”, “value”: “childprocs”}
}
}
The above format seems to work fine with the load service (I can see the alerting stream enabled and getting alerts).
However, if I try to do this:
[root@x-server01 kapacitor]# kapacitor define-template kp_tmp_template -tick load/templates/kp_abnormal_state_percent_template.tick
[root@x-server01 kapacitor]# kapacitor define kp_tmp_alert -template kp_tmp_template -vars load/tasks/kp_childprocs.json
invalid JSON in file load/tasks/kp_childprocs.json: json: cannot unmarshal string into Go value of type client.Var
Removing the:
{
“template-id”: “kp_abnormal_state_percent_template”,
“vars”:
…}
part from kp_childprocs.json allows the define commands to run fine, but then the load service is not working for this alert.
Any ideas on what am I doing wrong here?
Thank you.
