Telegraf Kafka Aggregation Issue: Not Processing Historical Data On Kafka Topic

Telegraf
,
1

Telegraf’s basicstats aggregator fails to process one-month-old Kafka data despite:

  • :white_check_mark: Setting offset = "oldest"
  • :white_check_mark: Resetting consumer group offsets
  • :white_check_mark: Kafka containing full historical data
# Kafka Input
[[inputs.kafka_consumer]]
  brokers = [xxxxxxxxxxxxx]
  topics = ["sensors_raw"]
  offset = "oldest"
  consumer_group = "telegraf_consumer_group_1h"
  data_format = "json"
  json_time_key = "timestamp"
  json_time_format = "unix_ns"

# Aggregator
[[aggregators.basicstats]]
  period = "1h"
  drop_original = true
  stats = ["count", "min", "max", "mean"]

Observed Behavior:

  • Telegraf only aggregates new data (from start time)
  • Ignores historical Kafka messages (1 month back)
  • No errors in logs, but old data never appears in InfluxDB

InfluxDB v2.7.8