Can't monitor services properly using [[inputs.systemd_units]]

Telegraf
, ,
1

Hello, i have experienced some strange behavior while trying to monitor services on another server. I have created tick script named systemd_units_task.tick to monitor status of systemd services.

I am using Influxdb 1.8.6 with Kapacitor OSS 1.5.9 on same Debian 10 machine. On host1 is installed Telegraf 1.20.2 .

When i start kapacitor task systemd_units_task i get one series with current state of ntp.service unit:

{"series":[{"name":"systemd_units","tags":{"active":"active","dc":"ca","host":"host1","load":"loaded","name":"ntp.service","sub":"running"},"columns":["time","active_code","load_code","sub_code"],"values":[["2021-10-14T08:00:30Z",0,0,0]]}]}

and i get slack message. But when i stop ntp.service (with command “systemctl stop ntp”) second series appear:

{"series":[{"name":"systemd_units","tags":{"active":"active","dc":"ca","host":"host1","load":"loaded","name":"ntp.service","sub":"running"},"columns":["time","active_code","load_code","sub_code"],"values":[["2021-10-14T08:02:10Z",0,0,0]]},

{"name":"systemd_units","tags":{"active":"inactive","dc":"ca","host":"host1","load":"loaded","name":"ntp.service","sub":"dead"},"columns":["time","active_code","load_code","sub_code"],"values":[["2021-10-14T08:02:20Z",2,0,1]]}]}

As i understood how streams on kapacitor works, second series should not appear. It should change first series, not add another for different status.
Kapacitor is able to catch that new line and send me a slack notification, but when i start service again it won’t alert me that service is started.
Only thing that is changed is value of “time” field in series like this:

{"series":[{"name":"systemd_units","tags":{"active":"active","dc":"ca","host":"host1","load":"loaded","name":"ntp.service","sub":"running"},"columns":["time","active_code","load_code","sub_code"],"values":[["2021-10-14T08:05:40Z",0,0,0]]},

{"name":"systemd_units","tags":{"active":"inactive","dc":"ca","host":"host1","load":"loaded","name":"ntp.service","sub":"dead"},"columns":["time","active_code","load_code","sub_code"],"values":[["2021-10-14T08:05:20Z",2,0,1]]}]}

And every time i start or stop service only “time” field is updated and both series remain even if there is no value for one of them.

I am not sure if this is issue for Telegraf but i have tested another tick script which measures cpu load on host1 and it worked perfectly fine.
Also i have tested batch and saw same behavior.

Tickscript:

dbrp "telegraf"."autogen"

stream
        |from()
                .measurement('systemd_units')
                .groupBy('host','name','active')
                .where(lambda: "load" != 'not-found')
                .where(lambda: "host" == 'host1')
                .where(lambda: "name" == 'ntp.service')
        |httpOut('out')
        |alert()
                .id('{{ index .Tags "host" }}-{{ index .Tags "name" }}')
                .message('{{ .ID }} - has status: {{ index .Tags "active" }}.')
                .warn(lambda: "active_code" == 2)
                .crit(lambda: "active_code" == 3)
                .stateChangesOnly()
                .slack()
                .workspace('default')

Telegraf configuration on host “host1”:

[global_tags]
dc="ca"

[agent]
flush_interval = "10s"
flush_jitter = "10s"
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
precision = ""
debug = false
quiet = false
logfile = "/var/log/telegraf/telegraf.log"
hostname = ""
omit_hostname = false

[[outputs.influxdb]]
urls = ["https://influx"]
database = "telegraf"
retention_policy = ""
write_consistency = "any"
timeout = "5s"
username = "user"
password = "password"
user_agent = "telegraf"
ssl_ca = "/etc/telegraf/ca.pem"
ssl_cert = "/etc/telegraf/cert.pem"
ssl_key = "/etc/telegraf/key.pem"
insecure_skip_verify = false
content_encoding = "gzip"

[[inputs.cpu]]
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "squashfs"]
[[inputs.diskio]]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.net]]
[[inputs.netstat]]
[[inputs.nstat]]
[[inputs.temp]]
[[inputs.systemd_units]]
timeout = "5s"
unittype = "service"
pattern = "nginx* ntp* ssh* logrotate* influx* syncflux* kapacitor* snap* mysql*"