Message is missing in Kapacitor alert

Hello, I have a Kapacitor template i use to alert on Nutanix over all storage. The alert triggers when the total capacity drops to signify a potentional disk issue. That part works fine, however the alert message is not written into the alerts database and i can’t work out why. I have another template similar (monitoring the current usage to trigger at < 20% and < 30%) - This one works perfectly, the message is inserted with the alert data.

The message in each template is very similar (apart from the name of the field in the message) - I’m at a loss as to what is happening :frowning:

If i tail the kapacitor logs and run kapacitor show taskname i can see there are no errors. If I tail the influxdb logs i don’t see anything about the point being rejected. Also if i add a .log() node to my template i can see the message as it should be.

As a point, all of my templates use the same message format. The only difference is the fields in the message and the wording describing what the alert is.

// Database settings
var db = 'input'
var rp = 'autogen'
var measurement = 'nutanix_cluster_storage_stats'
// Alerts database
var outputDB = 'output'
var outputRP = 'autogen'
var outputMeasurement = 'NutanixAlerts'

var groupBy = ['cluster_uuid','host','CustomerName','Location']
// Customer name. This must be entered in upper exactly as it appears in the database. 
var customer string
var host string
var whereFilter = lambda: ("cluster_uuid" == host AND "CustomerName" == customer)
var priority string
var period = 4m

var every = 2m

var name = 'NutanixClusterStorageCapacity'

var idVar = name + '-{{index .Tags "CustomerName"}}-' + priority + '-' + host
var message = 'Priority: ' + priority + ', host: '+ host +', Capacity Usage: {{index .Fields "capacity_tb" | printf "%0.2f"}}TB at {{.Time}}'

var idTag = 'alertID'

var levelTag = 'level'

var messageField = 'message'

var durationField = 'duration'

var triggerType = 'threshold'


var source = 'Source'
var metric = 'Total Storage Capacity'
var unit = 'TB'
var module = 'Nutanix'


var mb = 1024.0 * 1024.0 * 1024.0 * 1024.0
//  This is the total amount of disk space in your cluster. If the capacity is not equal to or is less than this value an alert is triggered.
// Value should be entered in TB. Example: A 10 TB threshold should be entered as 10.0
var crit lambda


var data = stream
    |from()
        .database(db)
        .retentionPolicy(rp)
        .measurement(measurement)
        .groupBy(groupBy)
        .where(whereFilter)
    |window()
        .period(period)
        .every(every)
        .align()

    // |eval(lambda: float("capacity_bytes"))
    //     .as('capacity_bytes')

    |eval(lambda: float("capacity_bytes") / mb,
          lambda: float("capacity_bytes") / mb,
          lambda: string(float("capacity_bytes") / mb), // Var1
          lambda: string(host), // Var2
          lambda: string(source), // Alert Source
          lambda: string(metric), // Metric that triggered the alert
          lambda: string(unit), // unit type %, MB, GB
          lambda: string(module), // Module - Nutanix
          lambda: string(priority) // Alert priority. not to be confused with alert level
          )
        .as('capacity_bytes','capacity_tb','Var1','Var2','Source','Metric','Unit','Module','Priority')
        .tags('Var1','Var2','Source','Metric','Unit','Module','Priority')
        .keep('capacity_bytes','capacity_tb')

        
    
var trigger = data
    |alert()
        .crit(lambda: float("capacity_tb") != crit)
        //.stateChangesOnly()
        .message(message)
        .id(idVar)
        .idTag(idTag)
        .levelTag(levelTag)
        .messageField(messageField)
        .durationField(durationField)
        .log('/tmp/' + name + '.log')
        |httpPost()
            .endpoint('myendpoint')
        

trigger
    |delete()
        .tag('Var1')
        .tag('Var2')
        .tag('Source')
        .tag('Metric')
        .tag('Unit')
        .tag('Module')
        .tag('Priority')


    |eval(lambda: float("capacity_bytes"),
          lambda: float("capacity_bytes") / mb)
        .as('capacity_bytes','capacity_tb')
        .keep('capacity_bytes','capacity_tb')
    |influxDBOut()
        .create()
        .database(outputDB)
        .retentionPolicy(outputRP)
        .measurement(outputMeasurement)
        .tag('alertName', name)
        .tag('Priority', priority)
        .tag('triggerType', triggerType)

trigger
    |httpOut('output')

InfluxDB 1.5.2
Kapacitor 1.5.1

Updating Influx is not an option currently.

just seems odd that this one alone refuses to work as it should. I have a second template that alerts on percentage of the storage remaining. They are basically the same template but monitoring different fields. That one works!?!!?

Any ideas?

Regards,

PhilB

Hello @philb,
can you please share the output of kapacitor show task?
Thank you

Hi @Anaisdg thanks for responding.

The output for show task is here:

This is the same script I’m having the |window() issue with. You can ignore the errors for the http endpoint, it has been disabled for now.

Thanks

PhilB

@philb,
Can you share the data that is being written to Influxdb please?
Thank you.

Morning @Anaisdg, heres the output from the database.

The message is…
var message = 'Priority: ' + priority + ', host: '+ host +', Capacity Usage: {{index .Fields "capacity_tb" | printf "%0.2f"}}TB at {{.Time}}'

We use the same format in our templates, basically copy & pasta it and update the relevant fields. A working example for a similar alert but on a different metric: