This post is older than a year. Consider some information might not be accurate anymore.
Used: elasticsearch v6.2.4
X-Pack Alerting aka Elasticsearch Watcher can run distributed in a Elasticsearch cluster.
The basic concept: The watcher index .watches or .watches-6 contains all watches. If you increase the replicas, multiple elasticsearch nodes, that contains the index shard (replica), are used by the master node for alerting.
On each node, that contains one of the .watches shards, a scheduler, .. runs. .. The more replica shards you add, the more distributed the watches can be executed.
(Source: https://www.elastic.co/guide/en/x-pack/6.2/how-watcher-works.html)
To increase the replica size:
PUT .watches-6/_settings
{
"number_of_replicas": 2
}
With the primary shard and 2 replicas, you should have three elasticsearch nodes that run your watchers.
You can check it by querying all thread pools:
GET _cat/thread_pool?v&s=name:desc&h=host,name,active,rejected,completed
The watcher thread pool will appear on top of the results.
host name active rejected completed
alpha watcher 0 0 0
master1 watcher 0 0 0
gamma watcher 0 0 12668
delta watcher 0 0 4275
beta watcher 0 0 0
master2 watcher 0 0 0
master3 watcher 0 0 0
omega watcher 0 0 15693
