Hi,
I wonder difference between kafka_consumer_legacy and kafka_consumer input plugins?
Are they depend on each other in order to consume data from kafka? Or, each of them should work independently?
I installed influxdb and kafka in my environment and set some telegraf agents to consume from kafka using kafka_consumer plugin. After a while, I realized that there is missing data in influxdb database.
For example,
I have producer telegraf agents running on host1, host2 and host3 (writing data to kafka)
And I have consumer telegraf agents running on host5 and host6 (reading data from kafka, writing data to influxdb)
Suppose they all are running without any problem.
I added one more producer running on host4. But it didn’t appear in influxdb. I changed debug = true in telegraf.conf on host4. From logs, I confirmed that it was writing data to kafka.
I looked into kafka to check if data exist in for host4 using kafka-console-consumer.sh command. Yes, it was there.
I also tried adding more producers. Some of them appeared in influxdb but some of them did not.
The issue is; host5 and host6 is not consuming data for some hosts. But I couldn’t find the reason.
Then, I used kafka_consumer_legacy plugin. With this one, I have now data in influxdb from all hosts. This is working fine.
So, this is the reason I wonder if kafka_consumer_legacy and kafka_consumer depend on each other.
What is your experience? Do you think that I did something wrong or miss?
Regards,
ismail