HTTP Input for Elasticsearch Watcher

This post is older than a year. Consider some information might not be accurate anymore.

Used:   elasticsearch v6.2.2 

Elasticsearch X-Pack Alerting or aka Watcher offers the capability to alert on specific events/constellation in the Elasticsearch data. Watcher can retrieve data from the cluster where it runs (on the master node), or fetch data from Restful Web-Services via the http input. Preferably having a production cluster, you should report the monitoring data to a dedicated Elasticsearch monitoring cluster. This monitoring cluster can also run watches. The watch I’am going to introduce is the cluster health watch.

Purpose

The health of your production cluster is of utmost importance. Elasticsearch provides the _cluster/health endpoint and returns three states. The cluster health status is: green, yellow or `red. By severity you might consider everything which is not green as alert. Having a dedicated monitoring cluster allows you to run watches, if you have a license subscription. Knowing when your cluster is in trouble, might give you the necessary time to act accordingly or react at least faster. Watcher can give you that time.

The cluster health watch must not be performed by Elasticsearch Watcher. A Jenkins job or a cron job is also a viable option. Watcher is a integrated all in one solution in the elastic stack. It documents every watch and can you compare or scroll through the history. Regardless how you do it, it must be done.

Watch Definition

Find below the watch definition with example data. Replace it, to your needs.

Following watch definition:

• input
  • request the input from the production cluster over the http endpoint
  • uses basic auth, optional if you don’t have X-Pack Security or Nginx protected endpoint
• condition
  • everything that is not green, causes an action
  • you might invert the condition by asking for health state equal to red
• action
  • sends a email to the Elasticsearch Administrator or SMS through a mail gateway

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "http": {
      "request": {
        "scheme": "http",
        "host": "elasticsearch",
        "port": 9200,
        "method": "get",
        "path": "/_cluster/health",
        "params": {},
        "headers": {},
        "auth": {
          "basic": {
            "username": "healthcheck",
            "password": "check_it_out"
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.status": {
        "not_eq": "green"
      }
    }
  },
  "actions": {
    "notify_admins": {
      "email": {
        "profile": "standard",
        "from": "watcher@cinhtau.net",
        "reply_to": [
          "le_mapper@cinhtau.net"
        ],
        "to": [
          "le_mapper@cinhtau.net"       
        ],
        "subject": "Status  detected for Production Cluster.",
        "body": {
          "html": "Please check cluster! This watch is deactivated during maintenance!"
        }
      }
    }
  }
}

Summary

• Provided example could also be applied to any other Web-Service.
• You might also consider a Statuspage if you want to provide the information for your customers/users.
• The http input allows to query data from other endpoints.
• Response must be in JSON or YAML for watcher to process it.