Using Sidecar Container for Elasticsearch Configuration

This post is older than a year. Consider some information might not be accurate anymore.

Used:   elasticsearch v5.6.8 

Applications shipped in Docker containers are a major game changer, especially having a Elasticsearch cluster. My production cluster consists of 11 nodes. In the core, Elasticsearch is the same. Each node though has its specific configuration, settings and purpose. On top of that, Elasticsearch X-Pack Security in Version 6 requires that the communication within the cluster must run encrypted. This is accomplished by SSL certificates. Each node has its own private key and certificate. So I was facing with the problem, how to ship the node specific parts along with the core elasticsearch container. Use the core container as baseline and copy the configuration and certificate into the container? This would resolve in 11 specific images. Not in the spirit of reusability though. The better approach or answer came by remembering the tech talk Docker Patterns by Roland Huss, given at the Java Conference (Javaland 2016). Use a configuration container as a sidecar!

Concept

The basic question to the pattern is: How to configure containerized applications for different environments ?

Solution Patterns:

• Env-Var Configuration
• Configuration Container
• Configuration Service

ENV-VAR Configuration

The bullet points from Docker Patterns:

• Standard configuration method for Docker container
• Specified during build or run time
• Universal

We can define environment variables and configure the docker application to use them. Environment variables can be overwritten by passing them.

Elasticsearch already does that, for example:

We have multiple environment bypasses in above example.

• For instance we set docker environment variables like timezone (TZ).
• The -E node.name=alpha is an Elasticsearch argument.

The elasticsearch node configuration was identical. Only the node specific information were provided.

This was done by me in the past. It worked until the requirement for node certificates. This doesn’t work for certificate files. Let’s take a look on the next approach.

Configuration container

The bullet points from Docker Patterns:

• Configuration in extra Docker container
• Volume linked during runtime

Pros

• Flexible
• Explicit

Cons

• Static
• Maintenance overhead
• Custom file layout

The basic idea is to run 11 elasticsearch containers and just attaching or linking them to the configuration container (symbolically as sidecar).

Summary

The third approach is using some kind of central configuration registry or service. Consul, etcd or Apache Zookeeper are viable solutions, but in my scenario with Elasticsearch not applicable.

So Sidecar it is!

Sidecar

This pattern is named Sidecar because it resembles a sidecar attached to a motorcycle. In the pattern, the sidecar is attached to a parent application and provides supporting features for the application. The sidecar also shares the same lifecycle as the parent application, being created and retired alongside the parent. The sidecar pattern is sometimes referred to as the sidekick pattern and is a decomposition pattern.

Creating the Sidecar Container

I will demonstrate how I applied the sidecar pattern for my elasticsearch test-cluster of 3 nodes.

First my project layout:

Each node has following node specific configuration. Mandatory for Elasticsearch is elasticsearch.yml and certs folder for X-Pack security.

Dockerfile

The Dockerfile to build the sidecar container.

The COPY command copies the configuration to the container /config/ directory. Pay attention to ignore unwanted files in the .dockerignore file.

Check contents

You can inspect the sidecar container by executing the list directory command ls. After the execution, the docker container is automatically removed.

Deployment

On the docker host, we don’t need to run the sidecar container. If we just create the container the docker volume is available.

Usage

Using Elasticsearch with configuration: To use the configuration volume from our sidecar container, omit this option --volumes-from es_config, where es_config is the name of the docker container.

Container Lifecycle

The sidecar container wil be removed, if you have some cleanup, but the volume still exist, since it used in by the elasticsearch application container. A docker inspect of the application container will give you the name of the used volume.

The volume name f89912de22e2b34170e0b331c8a5e25b00f921f4e2417c6b140382389fadee7e is used.

To check:

As you can see the volume name is hard for humans to remember. Docker offers to create named volumes as cherry on top.

Summary

• The sidecar pattern is very useful.
• Sidecar containers have multiple purposes:
  • Configuration
  • Proxy (using Nginx as Load Balancer or Proxy)
  • Logging (running log shipper for centralized logging)
  • every other abstraction
• A sidecar container lifecycle is tight to its application container(s).