Resolve critical elasticsearch cluster health

This post is older than a year. Consider some information might not be accurate anymore.

From time to time, you need to perform a cluster upgrade in elasticsearch. During an upgrade, usually the cluster health turn from green to yellow. If it turns red, it is a critical state. One reason might be, that elasticsearch can’t replicate data shards, though the replicas are gone or lost. Using the ES Health REST API, allows you to identify the corrupt indices and delete them.

First query the cluster health, below example has status red. BTW is was done with Sense. You can guess the respective curl command for the http request/rest call.

GET _cluster/health
{
  "cluster_name": "prod",
  "status": "red",
  "timed_out": false,
  "number_of_nodes": 5,
  "number_of_data_nodes": 4,
  "active_primary_shards": 336,
  "active_shards": 675,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 4,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 99.41089837997055
}

Dig deeper on index level with GET _cluster/health?level=indices. This will give you a large result set. Filter it for unassigned_shards > 0 or simply "status": "red".

    ".marvel-es-1-2016.07.18": {
      "status": "red",
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "active_primary_shards": 0,
      "active_shards": 0,
      "relocating_shards": 0,
      "initializing_shards": 0,
      "unassigned_shards": 2
    },
    ".marvel-es-1-2016.07.21": {
      "status": "red",
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "active_primary_shards": 0,
      "active_shards": 0,
      "relocating_shards": 0,
      "initializing_shards": 0,
      "unassigned_shards": 2
    },

Delete the indices and the cluster will turn green again. In this example:

DELETE .marvel-es-1-2016.07.18
DELETE .marvel-es-1-2016.07.21