Backup your Elasticsearch data with Amazon S3

This post is older than a year. Consider some information might not be accurate anymore.

Used:   elasticsearch v5.1.2 

As I mentioned before, how easy it is to backup your Elasticsearch data with the snapshot and restore API, today’s post demonstrates how to backup the data to Amazon S3 file storage.

Install the plugin

First you need to install the elasticsearch plugin for that:

sudo bin/elasticsearch-plugin install repository-s3

Create a user for S3

There are various ways how to access the S3 storage. I demonstrate the simplest one.

Log in into your AWS console and go to Security. Create a user with an access key. I named the user elasticsearch.

And assign the user to the group backup with permission for S3.

After that AWS will generate the access and secret key for the Elasticsearch user.

Configure your S3 access

You need to alter your elasticsearch.yml for that. Following settings are exemplary and doesn’t represent real values.

cloud:
    aws:
        access_key: AAAABBBB1234CCCC5678
        secret_key: Ahfk380HqZR7sUYdeH2Xw*ZxyY8fwlF5QVQoxiJ$
        s3.region: eu-central

Do the backup

Following steps demonstrates the backup process.

Check plugin

In the Kibana console you can check if the repository-s3 is installed.

GET _cat/plugins

23Y9vRH repository-s3 5.1.2
23Y9vRH x-pack        5.1.2

Now we can define the S3 bucket. Replace my-s3-bucket with your bucket name and maybe replace the region, as you need it. The repository will also be verified.

PUT _snapshot/s3
{
  "type": "s3",
  "settings": {
    "bucket": "my-s3-bucket",
    "compress": true,
    "region": "eu-central-1"
  }
}

To verify it manually:

POST /_snapshot/s3/_verify

Backup Kibana

We do a backup and name the snapshot upgrade_512.

PUT _snapshot/s3/upgrade_512
{
  "indices": ".kibana",
  "include_global_state": false
}

Check snapshot status

GET _snapshot/s3/_all
{
  "snapshots": [
    {
      "snapshot": "upgrade_512",
      "uuid": "RPUl1FXuRzyt6_pRxEcgWw",
      "version_id": 5010299,
      "version": "5.1.2",
      "indices": [
        ".kibana"
      ],
      "state": "SUCCESS",
      "start_time": "2017-01-15T20:37:47.333Z",
      "start_time_in_millis": 1484512667333,
      "end_time": "2017-01-15T20:37:50.838Z",
      "end_time_in_millis": 1484512670838,
      "duration_in_millis": 3505,
      "failures": [],
      "shards": {
        "total": 1,
        "failed": 0,
        "successful": 1
      }
    }
  ]
}

Looking in S3 I notice that elasticsearch did the upgrade in the root of the bucket.

This was never intended. We can easily delete it with:

DELETE _snapshot/s3/upgrade_512

The right definition is to put a base_path to it. This will place the data into the folder elasticsearch.

PUT _snapshot/s3
{
  "type": "s3",
  "settings": {
    "bucket": "my-s3-bucket",
    "compress": true,
    "base_path": "elasticsearch",
    "region": "eu-central-1"
  }
}