Kafka monitoring on influxdb

[rijaluc]

Hi all,

How to config and deploy influxdb (telegraf) to monitor Kafka

Thanx

[Anaisdg]

Hello @rijaluc,
Welcome!
You’ll need to install telegraf and InfluxDB:
Or sign up for InfluxDB Cloud:

Then you can either configure Telegraf through the UI:

Automatically configure Telegraf for InfluxDB v2.0 | InfluxDB Cloud...

Or configure it mannually:

You’ll want to follow:
Getting started with telegraf

Get started with Telegraf | Telegraf 1.21 Documentation

And then use the kafka input plugin

influxdata/telegraf/blob/master/plugins/inputs/kafka_consumer/README.md

And the InfluxDB v2 output plugin:

You’ll have to configure the Influx v2 output plugin portion of your telegraf config with the following information:

• a bucket: Create a bucket in InfluxDB | InfluxDB Cloud Documentation
• a token: Create an API token in InfluxDB | InfluxDB Cloud Documentation
• url (OSS InfluxDB OSS URLs | InfluxDB OSS 2.3 Documentation) or (Free tier cloud InfluxDB Cloud regions | InfluxDB Cloud Documentation)

Watch this on how to manually configure it:

Manually configure Telegraf for InfluxDB v2.0 | InfluxDB Cloud Documentation

Let me know if you have questions after taking a look at that!

[rijaluc]

Hello @Anaisdg
thanks for your support but I has on problem on Jolokia

Below the step we has done :

1. Download the latest version of the Jolokia JVM-Agent from
2. Save jolokia-jvm-agent.jar on the Kafka server in /home/rijaluc/confluent-7.2.1/bin
3. Configure Kafka to use Jolokia: Add the following server.propreties
   
   
4. Restart the Kafka service.
5. Verify that you can access Jolokia on port 8778 by running:

After vérification on localhost:8778/jolokia/list the page is unreachable.
Can you help us please ?
Note: I’m using Confluent Kafka

[Anaisdg]

Hello @rijaluc,
Can you include:

debug = true

In the agent portion of your config and share the logs?

Thank you.

[rijaluc]

Hello @Anaisdg , thanks for your support, the problem of jolokia is resloved.

I have an issue question about telegraf to Influxdb.
1- We install telegraf and Kafka on my windows OS
2- We configure the telegraf.conf. Below

Configuration for telegraf agent

[agent]
interval = “20s”
round_interval = true
metric_batch_size = 4000
metric_buffer_limit = 40000
collection_jitter = “0s”
flush_interval = “10s”
flush_jitter = “0s”
precision = “”
hostname = “”
omit_hostname = false
###############################################################################

OUTPUT PLUGINS

###############################################################################

Configuration for sending metrics to InfluxDB

[[outputs.influxdb]]

urls = [“http://localhost:8086”]

The target database for metrics; will be created as needed.

For UDP url endpoint database needs to be configured on server side.

database = “telegraf”

skip_database_creation = true

###############################################################################

INPUT PLUGINS

###############################################################################

Read JMX metrics through Jolokia

[[inputs.jolokia2_agent]]

An array of Kafka servers URI to gather stats.

urls = [“http://localhost:8778/jolokia”]
name_prefix = “kafka.”

[[inputs.jolokia2_agent.metric]]
name = “heap_memory_usage”
mbean = “java.lang:type=Memory”
paths = [“HeapMemoryUsage”]

This collects thread counts metrics.

[[inputs.jolokia2_agent.metric]]
name = “thread_count”
mbean = “java.lang:type=Threading”
paths = [“TotalStartedThreadCount”,“ThreadCount”,“DaemonThreadCount”,“PeakThreadCount”]

This collects garbage collection metrics.

[[inputs.jolokia2_agent.metric]]
name = “garbage_collector”
mbean = “java.lang:type=GarbageCollector,name=*”
paths = [“CollectionCount”,“CollectionTime”]
tag_keys = [“name”]

Kafka Server Broker Topic Metrics

[[inputs.jolokia2_agent.metric]]
name = “server_brokertopics_messagesinpersec”
mbean = “kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec”
[[inputs.jolokia2_agent.metric]]
name = “server_brokertopics_bytesinpersec”
mbean = “kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec”
[[inputs.jolokia2_agent.metric]]
name = “server_brokertopics_bytesoutpersec”
mbean = “kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec”

Kafka Server Request Handler Metrics

[[inputs.jolokia2_agent.metric]]
name = “server_requesthandler_avgidlepct”
mbean = “kafka.server:name=RequestHandlerAvgIdlePercent,type=KafkaRequestHandlerPool”

Kafka Server Delayed Operation Purgatory Metrics

[[inputs.jolokia2_agent.metric]]
name = “server_delayedoperationpugatory_fetch”
mbean = “kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=Fetch”
[[inputs.jolokia2_agent.metric]]
name = “server_delayedoperationpugatory_produce”
mbean = “kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=Produce”

Kafka Server Replica Fetcher Manager Metrics

[[inputs.jolokia2_agent.metric]]
name = “server_replicafetchmanager.maxlag”
mbean = “kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica”

Kafka Server Replica Manager Metrics

[[inputs.jolokia2_agent.metric]]
name = “server_replicamanager_underreplicated”
mbean = “kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions”
[[inputs.jolokia2_agent.metric]]
name = “server_replicamanager_partitioncount”
mbean = “kafka.server:type=ReplicaManager,name=PartitionCount”
[[inputs.jolokia2_agent.metric]]
name = “server_replicamanager_leadercount”
mbean = “kafka.server:type=ReplicaManager,name=LeaderCount”
[[inputs.jolokia2_agent.metric]]
name = “server_replicamanager_isrshrinkspersec”
mbean = “kafka.server:type=ReplicaManager,name=IsrShrinksPerSec”
[[inputs.jolokia2_agent.metric]]
name = “server_replicamanager_isrexpandspersec”
mbean = “kafka.server:type=ReplicaManager,name=IsrExpandsPerSec”

Kafka Network Request Metrics

[[inputs.jolokia2_agent.metric]]
name = “network_requestmetrics_requests_fetch_consumer”
mbean = “kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer”
[[inputs.jolokia2_agent.metric]]
name = “network_requestmetrics_requests_fetch_follower”
mbean = “kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower”
[[inputs.jolokia2_agent.metric]]
name = “network_requestmetrics_requests_produce”
mbean = “kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce”
[[inputs.jolokia2_agent.metric]]
name = “network_requestmetrics_totaltime_fetch_consumer”
mbean = “kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer”
[[inputs.jolokia2_agent.metric]]
name = “network_requestmetrics_totaltime_fetch_follower”
mbean = “kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower”
[[inputs.jolokia2_agent.metric]]
name = “network_requestmetrics_totaltime_produce”
mbean = “kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce”

Kafka Network Processor Metrics

[[inputs.jolokia2_agent.metric]]
name = “network_processor_avgidlepct”
mbean = “kafka.network:name=NetworkProcessorAvgIdlePercent,type=SocketServer”

Kafka Controller Metrics

[[inputs.jolokia2_agent.metric]]
name = “controller_activecontrollers”
mbean = “kafka.controller:type=KafkaController,name=ActiveControllerCount”
[[inputs.jolokia2_agent.metric]]
name = “controller_offlinepartitions”
mbean = “kafka.controller:type=KafkaController,name=OfflinePartitionsCount”
[[inputs.jolokia2_agent.metric]]
name = “controller_stats_leaderelectionrateandtime”
mbean = “kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs”
[[inputs.jolokia2_agent.metric]]
name = “controller_stats_uncleanleaderelections”
mbean = “kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec”

Zookeeper Metrics

[[inputs.jolokia2_agent.metric]]
name = “zookeeper_disconnects”
mbean = “kafka.server:type=SessionExpireListener,name=ZooKeeperDisconnectsPerSec”
paths = [“Count”,“OneMinuteRate”,“FiveMinuteRate”,“FifteenMinuteRate”,“MeanRate”]
[[inputs.jolokia2_agent.metric]]
name = “zookeeper_sync_connects”
mbean = “kafka.server:type=SessionExpireListener,name=ZooKeeperSyncConnectsPerSec”
paths = [“Count”,“OneMinuteRate”,“FiveMinuteRate”,“FifteenMinuteRate”,“MeanRate”]
[[inputs.jolokia2_agent.metric]]
name = “zookeeper_auth_failures”
mbean = “kafka.server:type=SessionExpireListener,name=ZooKeeperAuthFailuresPerSec”
paths = [“Count”,“OneMinuteRate”,“FiveMinuteRate”,“FifteenMinuteRate”,“MeanRate”]
[[inputs.jolokia2_agent.metric]]
name = “zookeeper_readonly_connects”
mbean = “kafka.server:type=SessionExpireListener,name=ZooKeeperReadOnlyConnectsPerSec”
paths = [“Count”,“OneMinuteRate”,“FiveMinuteRate”,“FifteenMinuteRate”,“MeanRate”]
[[inputs.jolokia2_agent.metric]]
name = “zookeeper_authentications”
mbean = “kafka.server:type=SessionExpireListener,name=ZooKeeperSaslAuthenticationsPerSec”
paths = [“Count”,“OneMinuteRate”,“FiveMinuteRate”,“FifteenMinuteRate”,“MeanRate”]
[[inputs.jolokia2_agent.metric]]
name = “zookeeper_expires”
mbean = “kafka.server:type=SessionExpireListener,name=ZooKeeperExpiresPerSec”
paths = [“Count”,“OneMinuteRate”,“FiveMinuteRate”,“FifteenMinuteRate”,“MeanRate”]

3- We start the telegraf and is UP

So after checking on Influxb (localhost:8086) we don’t see anything .

[Anaisdg]

Hello @rijaluc,
Can you please include:

debug = true

In the agent portion of your config and share the logs?

Like:

[agent]
interval = “20s”
round_interval = true
metric_batch_size = 4000
metric_buffer_limit = 40000
collection_jitter = “0s”
flush_interval = “10s”
flush_jitter = “0s”
precision = “”
hostname = “”
omit_hostname = false
debug = true

Without doing this and sharing the logs I don’t have any information I need to help you. Thank you