Apache Kafka Management and Monitoring

:heavy_exclamation_mark: This post is older than a year. Consider some information might not be accurate anymore. :heavy_exclamation_mark:

Used:   apache kafka v1.0.0  kafka-manager v1.3.3.16 

Monitoring for Apache Kafka is crucial to know the moment when to act or scale out your Kafka clusters. Besides the CLI commands, metrics are also accessible over JMX and jconsole. A more convenient way is to have a GUI that displays it. This post focus on Kafka Manager, a administration GUI for Kafka by Yahoo.

Other solutions like elasticsearch and metricbeat kafka module by elastic also exists, but the module itself is beta and subject to change. Therefore I discard the idea using Kibana and Elasticsearch.

Enable JMX

JMX must be enabled in order to get them displayed in kafka-manager.

In my ansible playbook.yml for the Apache Kafka Docker container, add these environment variables.

KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Djava.rmi.server.hostname={{ansible_hostname}} \
JMX_PORT: 9099

Pay attention to the exposed JMX port. You can choose any port. I think port 9099 is fitting, as the default exposed port is 9092.

Kafka Manager

If Apache ZooKeeper and Apache Kafka is operational I start a separate docker container for kafka-manager. The most recent docker image for kafka-manager is by sheepkiller.

Run interactive, adjust the ZK_HOSTS to your needs.

docker run -it --rm -p 9000:9000 \
-e ZK_HOSTS="alpha:2181,beta:2181,gamma:2181" \
sheepkiller/kafka-manager -Djava.net.preferIPv4Stack=true

Or run detached for production mode

docker run -d --name kafka-manager -p 9000:9000 \
-e ZK_HOSTS="alpha:2181,beta:2181,gamma:2181" \
--restart always \
--log-driver json-file --log-opt max-size=10m \
sheepkiller/kafka-manager -Djava.net.preferIPv4Stack=true

You can access the GUI on the docker host over port 9000. See below exposed metrics.

Kafka Manager Admin Interface
Please remember the terms for blog comments.