Kafka Consumer Input Plugin error

Telegraf
1

I use Kafka Consumer Input Plugin and I have the following telegraf.conf:

# INPUTS
[[inputs.kafka_consumer]]
  topics = ["telegraf"]
  brokers = ["bootstrap-server:9092"]
  consumer_group = "telegraf_consumer"
  offset = "oldest"
  data_format = "influx"
  max_message_len = 65536

# OUTPUTS
[[outputs.influxdb]]
  url = "host_name:8086"
  database = "telegraf"
  precision = "s"

After starting Telegraf I get error message:

2017/06/09 17:48:54 E! Error parsing /etc/telegraf.conf, line 4: field corresponding to `brokers' is not defined in `*kafka_consumer.Kafka'

Is anything wrong with config file?

2

@Steven_1 What version of telegraf are you running?

3

I am running Telegraf v1.3

docker run -d --name telegraf --link influxdb:influxdb -v /etc/telegraf.conf:/etc/telegraf/telegraf.conf telegraf:1.3
4

To generate configuration proper for the telgraf version you are running use the config generation:

docker run -it telegraf:1.3 telegraf -sample-config -input-filter kafka_consumer -output-filter influxdb

We have changed to a new version of kafka consumer on master. The documentation for the older version is here. I think if you update your config accordingly then you will not run into any problems.

I always suggest that folks generate their config from the binary rather than looking online. Its faster and far less error prone.

5

Thanks. It looks like it doesn’t support latest Kafka without Zookeeper, correct?

#                            SERVICE INPUT PLUGINS                            #
###############################################################################

# Read metrics from Kafka topic(s)
[[inputs.kafka_consumer]]
  ## topic(s) to consume
  topics = ["telegraf"]
  ## an array of Zookeeper connection strings
  zookeeper_peers = ["localhost:2181"]
  ## Zookeeper Chroot
  zookeeper_chroot = ""
  ## the name of the consumer group
  consumer_group = "telegraf_metrics_consumers"
  ## Offset (must be either "oldest" or "newest")
  offset = "oldest"

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "influx"

  ## Maximum length of a message to consume, in bytes (default 0/unlimited);
  ## larger messages are dropped
  max_message_len = 65536
6

@Steven_1 The new plugin that we merged does support that and will be merged into 1.4. If you need to use that code you will need to build from master.

7

@jackzampolin Ok, thanks. Do you have any info when it will be officially released?

8

Currently planned for mid July, I’m tracking the date using the github milestones.

9

I am curious if new Telegraf version will be release this week.

10

@Steven_1 We are going to hold off a bit longer on 1.4 since many of the features we wanted are not ready yet. I’ll keep the milestone updated with the tentative date but of course it is always subject to change.

1 Like