Difference between kafka_consumer_legacy & kafka_consumer




I wonder difference between kafka_consumer_legacy and kafka_consumer input plugins?
Are they depend on each other in order to consume data from kafka? Or, each of them should work independently?

I installed influxdb and kafka in my environment and set some telegraf agents to consume from kafka using kafka_consumer plugin. After a while, I realized that there is missing data in influxdb database.

For example,
I have producer telegraf agents running on host1, host2 and host3 (writing data to kafka)
And I have consumer telegraf agents running on host5 and host6 (reading data from kafka, writing data to influxdb)
Suppose they all are running without any problem.
I added one more producer running on host4. But it didn’t appear in influxdb. I changed debug = true in telegraf.conf on host4. From logs, I confirmed that it was writing data to kafka.
I looked into kafka to check if data exist in for host4 using kafka-console-consumer.sh command. Yes, it was there.
I also tried adding more producers. Some of them appeared in influxdb but some of them did not.

The issue is; host5 and host6 is not consuming data for some hosts. But I couldn’t find the reason.

Then, I used kafka_consumer_legacy plugin. With this one, I have now data in influxdb from all hosts. This is working fine.

So, this is the reason I wonder if kafka_consumer_legacy and kafka_consumer depend on each other.
What is your experience? Do you think that I did something wrong or miss?



The legacy plugin is no longer maintained, and will be removed when we release 2.0, so we should try to get kafka_consumer working for you. This is a pretty strange that it would skip the results sent by a particular client, can you try to come up with a smaller example that reproduces it, perhaps needing fewer hosts?