Elasticsearch Hot Warm Architecture

This post is older than a year. Consider some information might not be accurate anymore.

Used:   elasticsearch v5.4.1 

Running a Elasticsearch cluster can be an easy task. If you have the need to store data for a long time, but you know the data is infrequently requested, you may think of a hot-warm architecture. This post is a brief summary of my setup for my company at work. At that time Elasticsearch v5.4.1 was running.

Terminology

In terms of speed in an ideal world, a hot node is usually a data node with high IO operations, basically an SSD raid0. Warm nodes are typically magnetic HDD raid0 with significant greater capacity. Elasticsearch is already distributed and redundant. Utilize the striping speed of raid0.

Above description must not comply with your needs. Just to be clear, a hot node is a data node with a fast storage solution and contains the newest data like the last 2 weeks or 1 month.

A warm node is just a data node with less fast storage, because it contains old data, which aren’t access so often. Nothing is written anymore, so it comes down basically to read only operations.

Setup

For this concept to work, you need to have discriminator for each data node. In the elasticsearch.yml add to attributes section the box_type attribute. You are free to choose the attribute name as you like it to have.

node:
  master: false
  data: true
  ingest: true
  attr:
    rack: "with-nas"
    box_type: "hot"

You could also pass the attribute as environment variable, overriding the value in the elasticsearch.yml. For instance

bin/elasticsearch -d -E node.attr.box_type="warm"

Delay Allocation

One important concept of indices, is to relocate its missing shards to other data nodes. If you have a cluster upgrade, rolling restarts are mandatory. Either you can disable the shard allocation or just put an timeout in the index setting. Below recipe will set a timeout to all indices.

curl -XPUT "http://fo-itu:9200/_all/_settings" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.unassigned.node_left.delayed_timeout": "10m"
  }
}' -u tknga

The allocation will delay for 10 minutes, while the missing data node is recovered and the shards are available again. You safe some network and disk IO. Read more on Delay Allocation.

Restart Node

For the hot warm architecture to work, the data node must be restarted. Otherwise you won’t be able to allocate anything.

Check with the Nodes API if the attribute was applied to the respective data node. Use the Console to list only data nodes

GET _nodes/data:true

The response body should contain something like this:

"attributes": {
    "box_type": "warm",
    "ml.enabled": "false"
}

Allocation

You can set routing for all indices with this template. Below recipe will force an allocation to hot nodes only.

PUT _all/_settings
{
  "settings": {
    "index.routing.allocation.require.box_type": "hot"
  }
}

Adjust Index Template

If new indices are created, add the attributes to your index templates.

"settings": {
    "index": {
        "number_of_shards": "1",
        "refresh_interval": "5s",
        "routing.allocation.require.box_type": "hot",
        "unassigned.node_left.delayed_timeout": "10m"
    }
}

Curator

• With Elasticsearch Curator we will allocate every index, that is older than 14 days to the warm node → action allocation
• We will reduce the shards to 1 → action forcemerge
• We will set the replica to 0 → action replicas

This action file template contains in logical order the allocation of indices from hot nodes to warm nodes.

• First we set the replicas to 0. This might be a danger if we lose a data node permanently, but speeds up the allocation.
• Based on a file name pattern all eligible indices are moved to the warm node.
• After the allocation we reduce segments, to free some resources.

actions:
  1:
    action: replicas
    description: >-
      Reduce the replica count to 0 for fo- prefixed indices older than
      4 days (based on index creation_date)
    options:
      count: 0
      wait_for_completion: True
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: fo-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 14
  2:
    action: allocation
    description: "Apply shard allocation filtering rules to the specified indices"
    options:
      key: box_type
      value: warm
      allocation_type: require
      wait_for_completion: true
      timeout_override:
      continue_if_exception: false
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: fo-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 14
  3:
    action: forcemerge
    description: "Perform a forceMerge on selected indices to 'max_num_segments' per shard"
    options:
      max_num_segments: 1
      delay:
      timeout_override: 21600
      continue_if_exception: false
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: fo-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 14