Hey folks, my goal is to pull metric events from a Kafka cluster and ingest them into an InfluxDB bucket.
With my current configuration the timestamp in InfluxDB is always set to the current day/time and not to the timestamp set in the Kafka event.
Below you can find my stripped down config,
[[inputs.kafka_consumer]]
brokers = ["kafka.localhost:9092"]
sasl_username = "user"
sasl_password = "password"
data_format = "value"
data_type = "float"
offset = "newest"
[[outputs.influxdb_v2]]
urls = ["https://influx.localhost"]
token = "asdf1234"
organization = "company"
bucket = "default"
@brennerm interesting. Could you please open a feature request for this!?!
Hey @srebhan , thanks for responding. Sure thing, here you go:
opened 02:07PM - 27 Aug 24 UTC
feature request
### Use Case
Each Kafka event contains a timestamp. This can either be set manu… ally by the producer or afaik will default to the time the event has been submitted.
For my use case, reading events from Kafka and sending them to InfluxDB, I'd like to use the timestamp of the Kafka event as the timestamp of my metric. Alternatively I could add the timestamp to the event payload but:
1. This would require me to change to a different data format, e.g like JSON. Currently I'm simply sending a plain value.
2. Would lead to duplicated data being transmitted, as the timestamp would be part of the event metadata as well as the payload.
### Expected behavior
There should be an option to extract the metric timestamp from the Kafka event.
### Actual behavior
Currently the timestamp defaults to the current time.
### Additional info
My current Telegraf config:
```toml
[[inputs.kafka_consumer]]
brokers = ["kafka.localhost:9092"]
sasl_username = "user"
sasl_password = "password"
data_format = "value"
data_type = "float"
offset = "newest"
[[outputs.influxdb_v2]]
urls = ["https://influx.localhost"]
token = "asdf1234"
organization = "company"
bucket = "default"
```
1 Like
