Inputs.kafka_consumer not working

Telegraf
1

I wanted to use apache kafka as input for telegraf. I am running docker containers. Docker image is from - GitHub - wurstmeister/kafka-docker: Dockerfile for Apache Kafka
This is my docker-compose file:

version: ‘3’
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- “2181:2181”
kafka:
image: wurstmeister/kafka
ports:
- “9092:9092”
environment:
- KAFKA_ADVERTISED_HOST_NAME=localhost
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_CREATE_TOPICS=“test:1:1”
depends_on:
- zookeeper
volumes:
- /var/run/docker.sock:/var/run/docker.sock

I have pushed data to topic and it’s working. I am getting the output. But for telegraf config i am getting - kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

[[inputs.kafka_consumer]]
brokers = [“localhost:9092”]
topics = [“test”]
version = “2.7.0”

Then i referred here - Possibility to access kafka contaner from other container · Issue #575 · wurstmeister/kafka-docker · GitHub

version: ‘2’
services:
zookeeper:
image: wurstmeister/zookeeper:3.4.6
ports:
- “2181:2181”
kafka:
image: wurstmeister/kafka:2.11-2.0.0
depends_on:
- zookeeper
ports:
- “9092:9092”
expose:
- “9093”
environment:
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_CREATE_TOPICS: ‘Sometopic:1:1’
KAFKA_AUTO_CREATE_TOPICS_ENABLE: ‘true’
volumes:
- /var/run/docker.sock:/var/run/docker.sock

Again producer and consumer is working fine but with telegraf same error is showing. For broker i tried all configs - kafka:9093,localhost:9092,kafka:9092,localhost:9093 still every time error is showing. What to do now?

2

Hi @Pratik_Das_Baghel,

I used the below docker-compose file:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    environment:
      - JAVA_OPTS="-Xms256m -Xmx256m"
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=localhost
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - JAVA_OPTS="-Xms256m -Xmx256m"
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper

and this config:

[[inputs.kafka_consumer]]
brokers = ["localhost:9092"]
topics = ["test"]
version = "2.7.0"

And it connected to the broker, initialized a new client and added subscription to test/0.

Could you try using this docker-compose and see if you get the same error message?

What version of telegraf are you using?

Do you have debug set to true in the agent portion of your config?

This closed issue might have some pointers also as it was a similar error message.

I also found a stack overflow issue about setting up docker-compose for kafka which may be useful also.

Let me know if that helps.
Thanks!