Message is missing in Kapacitor alert

Hello, I have a Kapacitor template i use to alert on Nutanix over all storage. The alert triggers when the total capacity drops to signify a potentional disk issue. That part works fine, however the alert message is not written into the alerts database and i can’t work out why. I have another template similar (monitoring the current usage to trigger at < 20% and < 30%) - This one works perfectly, the message is inserted with the alert data.

The message in each template is very similar (apart from the name of the field in the message) - I’m at a loss as to what is happening :frowning:

If i tail the kapacitor logs and run kapacitor show taskname i can see there are no errors. If I tail the influxdb logs i don’t see anything about the point being rejected. Also if i add a .log() node to my template i can see the message as it should be.

As a point, all of my templates use the same message format. The only difference is the fields in the message and the wording describing what the alert is.

// Database settings
var db = 'input'
var rp = 'autogen'
var measurement = 'nutanix_cluster_storage_stats'
// Alerts database
var outputDB = 'output'
var outputRP = 'autogen'
var outputMeasurement = 'NutanixAlerts'

var groupBy = ['cluster_uuid','host','CustomerName','Location']
// Customer name. This must be entered in upper exactly as it appears in the database. 
var customer string
var host string
var whereFilter = lambda: ("cluster_uuid" == host AND "CustomerName" == customer)
var priority string
var period = 4m

var every = 2m

var name = 'NutanixClusterStorageCapacity'

var idVar = name + '-{{index .Tags "CustomerName"}}-' + priority + '-' + host
var message = 'Priority: ' + priority + ', host: '+ host +', Capacity Usage: {{index .Fields "capacity_tb" | printf "%0.2f"}}TB at {{.Time}}'

var idTag = 'alertID'

var levelTag = 'level'

var messageField = 'message'

var durationField = 'duration'

var triggerType = 'threshold'

var source = 'Source'
var metric = 'Total Storage Capacity'
var unit = 'TB'
var module = 'Nutanix'

var mb = 1024.0 * 1024.0 * 1024.0 * 1024.0
//  This is the total amount of disk space in your cluster. If the capacity is not equal to or is less than this value an alert is triggered.
// Value should be entered in TB. Example: A 10 TB threshold should be entered as 10.0
var crit lambda

var data = stream

    // |eval(lambda: float("capacity_bytes"))
    //     .as('capacity_bytes')

    |eval(lambda: float("capacity_bytes") / mb,
          lambda: float("capacity_bytes") / mb,
          lambda: string(float("capacity_bytes") / mb), // Var1
          lambda: string(host), // Var2
          lambda: string(source), // Alert Source
          lambda: string(metric), // Metric that triggered the alert
          lambda: string(unit), // unit type %, MB, GB
          lambda: string(module), // Module - Nutanix
          lambda: string(priority) // Alert priority. not to be confused with alert level

var trigger = data
        .crit(lambda: float("capacity_tb") != crit)
        .log('/tmp/' + name + '.log')


    |eval(lambda: float("capacity_bytes"),
          lambda: float("capacity_bytes") / mb)
        .tag('alertName', name)
        .tag('Priority', priority)
        .tag('triggerType', triggerType)


InfluxDB 1.5.2
Kapacitor 1.5.1

Updating Influx is not an option currently.

just seems odd that this one alone refuses to work as it should. I have a second template that alerts on percentage of the storage remaining. They are basically the same template but monitoring different fields. That one works!?!!?

Any ideas?



Hello @philb,
can you please share the output of kapacitor show task?
Thank you

Hi @Anaisdg thanks for responding.

The output for show task is here:

This is the same script I’m having the |window() issue with. You can ignore the errors for the http endpoint, it has been disabled for now.



Can you share the data that is being written to Influxdb please?
Thank you.

Morning @Anaisdg, heres the output from the database.

The message is…
var message = 'Priority: ' + priority + ', host: '+ host +', Capacity Usage: {{index .Fields "capacity_tb" | printf "%0.2f"}}TB at {{.Time}}'

We use the same format in our templates, basically copy & pasta it and update the relevant fields. A working example for a similar alert but on a different metric: