Loading...

Using Aggregations with the Java Rest Client for Elasticsearch

:heavy_exclamation_mark: This post is older than a year. Consider some information might not be accurate anymore. :heavy_exclamation_mark:

Used:   elasticsearch v6.2.4  java 8 

The aggregations framework helps provide aggregated data based on a search query. Using aggregations on my audit data gives me insight who used my Elasticsearch cluster at what time. This post demonstrates how to translate the Elasticsearch Query DSL into the respective Java Objects of Elastic Java Rest Client.

Following query for the Kibana Console (previously Sense) creates one hour buckets with all unique users, who have accessed Elasticsearch:

POST audit-2018.06.02/_search?size=0
{
  "query": {
    "match": {
      "request": "MultiSearchRequest"
    }
  },
  "aggs": {
    "users_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "1h"
      },
      "aggs": {
        "users": {
          "terms": {
            "field": "principal"
          }
        }
      }
    }
  }
}

Truncated output with one bucket as example.

{
  "took": 6,  
  "aggregations": {
    "users_over_time": {
      "buckets": [
        {
          "key_as_string": "2018-06-02T10:00:00.000Z",
          "key": 1527933600000,
          "doc_count": 839,
          "users": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "john",
                "doc_count": 470
              },
              {
                "key": "jane",
                "doc_count": 305
              },
              {
                "key": "james",
                "doc_count": 41
              },
              {
                "key": "jim",
                "doc_count": 22
              },
              {
                "key": "jaime",
                "doc_count": 1
              }
            ]
          }
        }
      ]
    }
  }
}

To utilize this data and write it into long term reporting index, we could utilize Elasticsearch Alerting (previously Watcher) to store the aggregated data.

Another possibility is to use the Java Rest Client provided by Elastic. It is more fun. Java Programming and runs independent in a Spring Boot Docker container.

Search API

To start a search in the Java High Level Rest Client, we need a SearchRequest.

// we limit it to one index, wildcard patterns are working
SearchRequest searchRequest = new SearchRequest("six-audit-2018.06.02");
// set document type, since v6 optional
searchRequest.types("doc");
// Use a builder to construct the search query
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

Match Query

First we need to translate the query part.

{
  "query": { "match": { "request": "MultiSearchRequest" } }
}

The QueryBuilders class provides all query builder classes.

QueryBuilder queryBuilder = QueryBuilders.matchQuery("request", "MultiSearchRequest");

Aggregations

The aggregations part has a date histogram aggregation with a terms sub-aggregation.

{
  "aggs": {
    "users_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "1h"
      },
      "aggs": {
        "users": {
          "terms": {
            "field": "principal"
          }
        }
      }
    }
  }
}  

The AggregationsBuilders class provides all aggregation builder classes.

// parent aggregation
DateHistogramAggregationBuilder aggregation = AggregationBuilders.
    dateHistogram("users_over_time").
    field("@timestamp").
    dateHistogramInterval(DateHistogramInterval.hours(1));

3 = set the aggregation name to users_over_time

The terms aggregation is a sub-aggregation.

// sub-aggregation
aggregation.subAggregation(AggregationBuilders.terms("users")
    .field("principal"));

More on aggregations.

Complete Search Request

Putting it all together:

searchSourceBuilder.query(queryBuilder);      //set query part
searchSourceBuilder.aggregation(aggregation); //set aggs part
// assign search query to search request
searchRequest.source(searchSourceBuilder);

To execute the search, pass the results to the SearchResponse object.

// construct client
RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(..));
SearchResponse searchResponse = client.search(searchRequest);

Retrieving aggregations

Aggregations can be retrieved from the SearchResponse by first getting the root of the aggregation tree, the Aggregations object, and then getting the aggregation by name.

RestStatus status = searchResponse.status();

if (status == RestStatus.OK) {
    Aggregations aggregations = searchResponse.getAggregations();

    Histogram dateHistogram = aggregations.get("users_over_time");

    for (Histogram.Bucket bucket : dateHistogram.getBuckets()) {

        LOGGER.info("Key: {}", bucket.getKeyAsString());

        Map<String, Aggregation> aggregationMap = bucket.getAggregations().asMap();

        Aggregation userAggregration = aggregationMap.get("users");

        List<? extends Terms.Bucket> buckets = ((Terms) userAggregration).getBuckets();
        LOGGER.info("-- {} active users", buckets.size());

        for (Terms.Bucket user : buckets) {
            LOGGER.info("   User {}", user.getKey());
        }
    }
}

This will result in following log output.

Key: 2018-06-07T00:00:00.000Z
-- 2 active users
   User operations
   User support
Key: 2018-06-07T01:00:00.000Z
-- 3 active users
   User operations
   User reporting_agent
   User support
Key: 2018-06-07T02:00:00.000Z
-- 2 active users
   User operations
   User support
Key: 2018-06-07T03:00:00.000Z
-- 3 active users
   User john
   User operations
   User support
Key: 2018-06-07T04:00:00.000Z
-- 4 active users
   User john
   User operations
   User support
   User jane
Key: 2018-06-07T05:00:00.000Z
-- 5 active users
   User john
   User urs
   User operations
   User support
   User beat
Key: 2018-06-07T06:00:00.000Z
-- 7 active users
   User urs
   User john
   User operations
   User beat
   User support
   User peter
   User luke
Key: 2018-06-07T07:00:00.000Z
-- 8 active users
   User john
   User alain
   User urs
   User daniel
   User operations
   User support
   User peter
   User saltzmann
Key: 2018-06-07T08:00:00.000Z
-- 9 active users
   User john
   User alain
   User urs
   User operations
   User support
   User peter
   User saltzmann
   User jane
   User lemapper
Key: 2018-06-07T09:00:00.000Z
-- 9 active users
   User john
   User urs
   User peter
   User support
   User beat
   User operations
   User nicolas
   User thomas
   User luke
Key: 2018-06-07T10:00:00.000Z
-- 10 active users
   User john
   User urs
   User peter
   User alain
   User operations
   User thomas
   User joe
   User nicolas
   User richard
   User support
Key: 2018-06-07T11:00:00.000Z
-- 10 active users
   User john
   User alain
   User urs
   User saltzmann
   User daniel
   User peter
   User richard
   User operations
   User francesco
   User micha
Key: 2018-06-07T12:00:00.000Z
-- 10 active users
   User urs
   User john
   User alain
   User peter
   User lukas
   User operations
   User andreas
   User daniel
   User support
   User michael
Key: 2018-06-07T13:00:00.000Z
-- 10 active users
   User urs
   User peter
   User alain
   User operations
   User richard
   User daniel
   User andreas
   User john
   User jane
   User saltzmann
Key: 2018-06-07T14:00:00.000Z
-- 10 active users
   User urs
   User alain
   User john
   User daniel
   User peter
   User nicolas
   User operations
   User patrick
   User andreas
   User christian
Key: 2018-06-07T15:00:00.000Z
-- 8 active users
   User urs
   User alain
   User john
   User operations
   User peter
   User patrick
   User christian
   User support
Key: 2018-06-07T16:00:00.000Z
-- 2 active users
   User operations
   User patrick
Key: 2018-06-07T17:00:00.000Z
-- 2 active users
   User operations
   User patrick
Key: 2018-06-07T18:00:00.000Z
-- 2 active users
   User operations
   User patrick
Key: 2018-06-07T19:00:00.000Z
-- 3 active users
   User operations
   User patrick
   User francesco
Key: 2018-06-07T20:00:00.000Z
-- 2 active users
   User operations
   User patrick
Key: 2018-06-07T21:00:00.000Z
-- 2 active users
   User urs
   User operations
Key: 2018-06-07T22:00:00.000Z
-- 2 active users
   User urs
   User operations
Key: 2018-06-07T23:00:00.000Z
-- 1 active users
   User operations

Summary

  • Translating Search Requests into Java Objects is pretty straightforward.
  • QueryBuilders help you construct queries in Java.
  • AggregationBuilders help you construct aggregations in Java.
  • Having the results in Java offers a lot of opportunities.
Please remember the terms for blog comments.