I have an issue when trying to write data to Kafka broker with outputs.kafka plugin. The Telegraf logs are the following (Kafka server: 192.168.55.10, Telegraf kafka producer: 192.168.200.45):
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z I! Loaded aggregators:
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z I! Loaded processors:
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z I! Loaded outputs: kafka
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z I! Tags enabled: host=rancid type=linux
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"rancid", Flush Interval:10s
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z D! [agent] Initializing plugins
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z W! [kafka] enable_tls is deprecated, and the setting does nothing, you can safely remove it from the config
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z D! [sarama] Initializing new client
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z D! [sarama] client/metadata fetching metadata for all topics from broker 192.168.55.10:9093
Apr 23 15:17:47 rancid telegraf[24342]: 2021-04-23T12:17:47Z D! [sarama] Connected to broker at 192.168.55.10:9093 (unregistered)
Apr 23 15:20:17 rancid telegraf[24342]: 2021-04-23T12:20:17Z D! [sarama] client/metadata got error from broker -1 while fetching metadata: read tcp 192.168.200.45:39128->192.168.55.10:9093: read: connection timed out
Apr 23 15:20:17 rancid telegraf[24342]: 2021-04-23T12:20:17Z D! [sarama] Closed connection to broker 192.168.55.10:9093
Apr 23 15:20:17 rancid telegraf[24342]: 2021-04-23T12:20:17Z D! [sarama] client/metadata no available broker to send metadata request to
Apr 23 15:20:17 rancid telegraf[24342]: 2021-04-23T12:20:17Z D! [sarama] client/brokers resurrecting 1 dead seed brokers
Apr 23 15:20:17 rancid telegraf[24342]: 2021-04-23T12:20:17Z D! [sarama] client/metadata retrying after 250ms... (3 attempts remaining)
Apr 23 15:20:17 rancid telegraf[24342]: 2021-04-23T12:20:17Z D! [sarama] client/metadata fetching metadata for all topics from broker 192.168.55.10:9093
Apr 23 15:20:17 rancid telegraf[24342]: 2021-04-23T12:20:17Z D! [sarama] Connected to broker at 192.168.55.10:9093 (unregistered)
As you can see, first the kafka producer connects to the broker. Then after bout 1 minute it gets a timeout error and retries to connect again.
An interesting thing I observed is that while inspecting the network packets, it seems that the TLS Client/Server Hello is successful but then, the Change Cipher Spec packet sent from Kafka Server never reaches its destination. It is continuously retransmitted until the tieout is reached and reperform the Client/Server Hello procedure.
This repeats for 3 times, which is the maximum retries set by default on telegraf kafka output configuration. Another telegraf kafka output runs on the same host as the Kafka server and writes data successfully to it without any problems.
On kafka server.properties config file, I have enabled the following protocols:
ssl.enabled.protocols=TLSv1.3,TLSv1.2
SSL is used on port 9093. Can you provide assistance on troubleshooting that issue?
Thanks a lot.
