Error with Kapacitor script that replaces slashes in disk path with underscores in alert .id

1

I’m having trouble getting a Kapacitor script working that monitors disk space. I need the disk path in the .id field of the alert node. However, this is going to Sensu, and Sensu doesn’t like slashes in the alert names. So I need to replace slashes in the disk path with underscores as a workaround.

I nearly have this working. I managed to use strReplace to do the character replacement. I also verified in the Kapacitor log that I could see all expected tags and fields, including the new “pathsub” field that contains the file path with the substituted underscores for the slashes.

However, I’m now getting hung up where the .id property is getting set for the alert node. Here’s the error I’m seeing.

[task_master:main] 2017/10/06 14:17:30 E! Stopped task: testdisk-alert alert4: template: id:1:32: executing "id" at <.Fields>: can't evaluate field Fields in type kapacitor.idInfo

I’m guessing that it’s complaining about the way I’m attempting to use the new “pathsub” field, though the message doesn’t explicitly say this. Hoping someone has some ideas on where to go from here. Below is the complete TICK script as it currently stands. (The warn and crit values are set artificially low just so I can actually generate some alerts once I finally get this working.) Thanks.

testdisk.tick:
//parameters
var warn = 1
var crit = 2
var unit = 30m

//Dataframe
var data = stream
    |from()
        .measurement('disk')
        .groupBy('host', 'path')
    |eval(lambda: strReplace("path", '/', '_', -1))
        .as('pathsub')
        .keep()
    |log()

var alert = data
    |alert()
        .id('disk-space.TEST_IGNORE.{{ index .Fields "pathsub" }}.{{ index .Tags "host" }}')
        .message('{{ .Level }}: the disk usage for path {{ index .Tags "path" }} of {{ index .Tags "cloud_application_name" }}-{{ index .Tags "instance_id" }} with ip address of {{ index .Tags "host"}} is currently at {{ index .Fields "used_percent" }}%')
        .info(lambda: "used_percent" < warn)
        .warn(lambda: "used_percent" > warn)
        .crit(lambda: "used_percent" > crit)
        .stateChangesOnly(unit)

//alert
alert
    .log('/tmp/disk_alert_log.txt')
    .sensu()
    .source('{{ index .Tags "cloud_application_name" }}-{{ index .Tags "instance_id" }}')
2

The alert ID needs to be consistent from alert to alert, as such the .id template does not allow you to use the fields on the data since those can vary. The solution is to use a tag, have the eval node create a tag.

//Dataframe
var data = stream
    |from()
        .measurement('disk')
        .groupBy('host', 'path')
    |eval(lambda: strReplace("path", '/', '_', -1))
        .as('pathsub')
        //Indicate `pathsub` should be a tag
        .tags('pathsub')
        .keep()
    |log()

var alert = data
    |alert()
        .id('disk-space.TEST_IGNORE.{{ index .Tags "pathsub" }}.{{ index .Tags "host" }}')
        .message('{{ .Level }}: the disk usage for path {{ index .Tags "path" }} of {{ index .Tags "cloud_application_name" }}-{{ index .Tags "instance_id" }} with ip address of {{ index .Tags "host"}} is currently at {{ index .Fields "used_percent" }}%')
        .info(lambda: "used_percent" < warn)
        .warn(lambda: "used_percent" > warn)
        .crit(lambda: "used_percent" > crit)
        .stateChangesOnly(unit)

//alert
alert
    .log('/tmp/disk_alert_log.txt')
    .sensu()
    .source('{{ index .Tags "cloud_application_name" }}-{{ index .Tags "instance_id" }}')
1 Like
3

Thanks much, Nathaniel. I just figured that out at the same time you were apparently replying with that info. Thanks for confirming that I have the correct solution. I’ve been pulling my hair out over this. The “.keep()” property threw me for a loop too – I couldn’t figure out where the rest of my field values were going until I added this.

It’s all working now, and it’s creating disk space alerts qualified by host and path in Sensu. Now I just need to figure out why I’m only getting emails for the “root” path (/) alerts. I see the alerts for other paths in Sensu/Uchiwa – just no emails, doggone it anyway! Clearly not Kapacitor issue at this point – just sharing a little of my pain. :face_with_symbols_over_mouth: