Reindex Watcher Indices with Curator

This post is older than a year. Consider some information might not be accurate anymore.

Used:   elasticsearch v5.4.3  curator v5.1.1 

Elasticsearch Alerting with X-Pack (formerly known as Watcher), writes it watch executions in a daily indices. If you don’t keep an eye on that, you use a lot of shards on small indices. Curator offers the capability of the reindex action, i.e. write data from a daily index into a month or year index. This post contains an example for Elasticsearch v5.4.3 and Elasticsearch Curator v5.1.1.

The actionfile in yaml

actions:
  1:
    description: "Create target index as named"
    action: create_index
    options:
      name: '.watcher-history-3-2017'
  2:
    description: "Reindex daily watcher index into monthly index"
    action: reindex
    options:
      disable_action: False
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: REINDEX_SELECTION
        dest:
          index: .watcher-history-3-2017
    filters:
    - filtertype: pattern
      kind: prefix
      value: .watcher-history-3-2017.
  3:
    description: >-
      WATCHER: Delete indices older than 1 day
    action: delete_indices
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: .watcher-history-3-2017.
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1
      exclude:

The actions explained

1. If the target index does not exists, it will be created. If it exists, nothing will happen .
2. The reindex action will take all daily indices and reindex it to the target index.
3. After the reindex the daily indices are deleted, since the data is then redundant.

Curator is a great tool to tend to Elasticsearch indices, but on the reindex action I miss a little bit of flexibility. So far no date pattern can be used for replacing the year or current month. If you reindex the data into a year index, you don’t have to touch the actionfile so often.