Delete Elasticsearch documents by query in Version 5

This post is older than a year. Consider some information might not be accurate anymore.

Used:   elasticsearch v5.1.2 

To delete documents from an index has changed in Version 5. A little example how to delete documents in Elasticsearch v5.1.x, how to monitor the status and free up the disk space.

Warning: There are significant differences between version 2 and 5.

• See Delete by Query docs

Search Query

Check for log messages of application ep2-batch

GET logs-2017.02.07/logs/_search
{
  "query": {
    "term": {
      "application": {
        "value": "ep2-batch"
      }
    }
  },
  "size": 0,
  "aggs": {
    "levels": {
      "terms": {
        "field": "level"
      }
    }
  }
}

Too many log messages with DEBUG

{
  "took": 2137,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "hits": {
    "total": 44582853,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "levels": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "DEBUG",
          "doc_count": 24501347
        },
        {
          "key": "INFO",
          "doc_count": 20075370
        },
        {
          "key": "ERROR",
          "doc_count": 5225
        },
        {
          "key": "WARN",
          "doc_count": 911
        }
      ]
    }
  }
}

Delete Query

POST logs-2017.02.07/logs/_delete_by_query?conflicts=proceed
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "application": {
              "value": "ep2-batch"
            }
          }
        }
      ],
      "filter": {
        "term": {
          "level": "DEBUG"
        }
      }
    }
  }
}

Tip: run this in a console!

curl -XPOST "http://elasticsearch:9200/logs-2017.02.07/logs/_delete_by_query?conflicts=proceed" -d'
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "application": {
              "value": "ep2-batch"
            }
          }
        }
      ],
      "filter": {
        "term": {
          "level": "DEBUG"
        }
      }
    }
  }
}' -u tan

Check task status

Since the task itself may run a long time, you can check the status with the task API.

GET _tasks?actions=indices:data/write/delete/byquery

{
  "nodes": {
    "UIETB7IDTUa7-vZMb3F11g": {
      "name": "kibana-lb",
      "transport_address": "10.22.62.141:9300",
      "host": "elasticsearch",
      "ip": "10.22.62.141:9300",
      "roles": [],
      "tasks": {
        "UIETB7IDTUa7-vZMb3F11g:2866377": {
          "node": "UIETB7IDTUa7-vZMb3F11g",
          "id": 2866377,
          "type": "transport",
          "action": "indices:data/write/delete/byquery",
          "start_time_in_millis": 1486545212270,
          "running_time_in_nanos": 574241493292,
          "cancellable": true
        }
      }
    }
  }
}

Free disk space

The index itself won’t be truncated or optimized. The force merge API allows to force merging of one or more indices through an API. The merge relates to the number of segments a Lucene index holds within each shard. The force merge operation allows to reduce the number of segments by merging them.

• Force Merge Docs v5.1

POST logs-2017.02.07/_forcemerge?only_expunge_deletes=true

You can also check the forcemege task

GET _tasks?actions=indices:admin/forcemerge*