Monitoring Kafka - distinguishing empty topic from data collection problem?

kapacitor

#1

I’m currently monitoring lag between a data store and a kafka topic having the same data. If all of the messages on the kafka topic expire (it’s the default 1 week retention period), my alert goes to a deadman state.

Does anyone have any ideas on how to distinguish between happily finding an empty topic, and failing to collect data?

Using kapacitor 1.5. Collecting kafka message age data with the exec plugin and a custom script (based on kafkacat).