I got an exceptional case in the office. Some application logs, which belongs to a dev and testing environment, were stored or reported in the elasticsearch production cluster. Therefore a cleanup or migration was necessary.
I got an exceptional case in the office. Some application logs, which belongs to a dev and testing environment, were stored or reported in the elasticsearch production cluster. Therefore a cleanup or migration was necessary.
Writing software also results in writing application logs. Therefore log rotating or house keeping is essential to free the space of old and unused log files. While Linux provides logrotate, you may run into the situations that you aren’t root or an user with root permissions and are not eligible to use logrotate. A simple shell script will also provide the essential cleanup.
Bash 4 supports dictionaries, hash tables or associative arrays. I was in need of that feature writing an logstash script, working with environment variables in logstash itself. A simple demonstration.
My previous post, demonstrated how to use Elasticsearch Watcher for log file alerting. Elasticsearch Watcher itself keeps data, about its watches and actions.
Watcher is a commercial plugin for alerting based on elasticsearch documents. The required knowledge could be overwhelming, but is rather straightforward and pretty simple after understanding the fundamental concepts. This post will give you a simple watch definition to grasp the concept. If you have application logs and store them into elasticsearch, you want to be alerted if a log entry with log level ERROR is reported. Let’s do this.
Shield is the security plugin for Elasticsearch. Security in Elasticsearch is based on users with associated roles. A quick demonstration how to use it.
I love reveal.js - The HTML Presentation Framework. Attending at the Javaland 2016 Conference I saw a awesome usage of reveal.js within a docker container in the Docker Patterns Talk by Roland Huß. Curious and eager to know I explored his github account. Mr. Huß offers the basics in the docker-reveal repository. Using github for docker builds is a great idea. Then I started to play around with docker myself, mostly to maintain and ease administering multiple elasticsearch nodes in a cluster. I felt using github offers me the opportunity to use Travis CI to build the docker image and deploy it to dockerhub - the docker image storage. Is was easier than I thought and is much better than building it manually everytime. This post covers the progress and results.
From time to time, you need to perform a cluster upgrade in elasticsearch. During an upgrade, usually the cluster health turn from green to yellow. If it turns red, it is a critical state. One reason might be, that elasticsearch can’t replicate data shards, though the replicas are gone or lost. Using the ES Health REST API, allows you to identify the corrupt indices and delete them.
A quick example how to run Elasticsearch with Docker. For the demo I use the the official image from Elasticsearch at dockerhub.