Loading...

Provide health check port for Kibana

:heavy_exclamation_mark: This post is older than a year. Consider some information might not be accurate anymore. :heavy_exclamation_mark:

Used:   kibana v5.1.2 

The new Kibana 5.1.2 x-pack monitoring plugin is a job well done! With the major version, you can also monitor your Kibana instances - yes Kibana can be run clustered :sunglasses:. Having Kibana running in Docker, it allows you deploy it even faster. Back to the monitoring part. It was a real lifesaver and gave me some insights on the Kibana application life-cycle.

I notice the amount of http connections is astonishing. :dizzy_face:

Connection Problems

Even I know that our Kibana is popular, but it is very unlikely that this many users and sessions are alive. I just made a control check by restarting one instance. And my suspicion was right. After the docker restart the connection started at zero.

Connections after restart

If I compare the instances, I can easily figure out which instance was rebooted and which one runs for days. :wink:

Memory Consumption

My personal suspicion: The DNS network load-balancer is doing a probe on the application port 5601 for the health-check. If you have a probe from 30 seconds to 2 minutes, it would explain the numbers. If the load-balancer is responsible for that, I should give him another port for the health-check or probe. For this task I choose netcat.

Netcat is a computer networking utility for reading from and writing to network connections using TCP or UDP.

Netcat Homepage

In the Dockerfile we install netcat and expose port 5602 for the health check.

FROM kibana:5.1.2

# .. do some setup

COPY docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh

# install netcat
RUN apt-get update -y && apt-get install netcat -y

EXPOSE 5601 5602
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["kibana"]

In docker-entrypoint.sh just start netcat to listen to port 5602 and reply with hello. The other stuff is for stopping the loop if CTRL + C is pressed, which is in a docker terminal not the usual case. netcat will terminate the connection for sure after echoing hello! ( Adele greets ya :wink: )

while [ 1 ]; do nc -l -p 5602 -c "echo hello"; test $? -gt 128 && break; done &

Build and deployed the new docker container for Kibana with integrated health-check port. I talk to the network guys and they change the health-check probe to 5602. Of course it took some time for them to switch it. As you can see the http connections stopped to increase exponentially from the point on they performed the switch.

Increase Stop

After a rolling restart I have the regular numbers what I would have expected from the beginning.

Normal Connections

From my personal view, I was afraid that Kibana was only an UI improvement, but looking at the commercial plug-ins, it has improved a lot. Sadly they don’t come for free, but that doesn’t mean they aren’t worth their license.

This is only a temporary solution. Use Nginx for instance with Kibana to provide a healthcheck, but ensure to close the connection by the load balancer probe.

Please remember the terms for blog comments.