This post is older than a year. Consider some information might not be accurate anymore.
Used: elasticsearch v6.2.4 java 8
The aggregations framework helps provide aggregated data based on a search query. Using aggregations on my audit data gives me insight who used my Elasticsearch cluster at what time. This post demonstrates how to translate the Elasticsearch Query DSL into the respective Java Objects of Elastic Java Rest Client.
Following query for the Kibana Console (previously Sense) creates one hour buckets with all unique users, who have accessed Elasticsearch:
POST audit-2018.06.02/_search?size=0
{
"query": {
"match": {
"request": "MultiSearchRequest"
}
},
"aggs": {
"users_over_time": {
"date_histogram": {
"field": "@timestamp",
"interval": "1h"
},
"aggs": {
"users": {
"terms": {
"field": "principal"
}
}
}
}
}
}
Truncated output with one bucket as example.
{
"took": 6,
"aggregations": {
"users_over_time": {
"buckets": [
{
"key_as_string": "2018-06-02T10:00:00.000Z",
"key": 1527933600000,
"doc_count": 839,
"users": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "john",
"doc_count": 470
},
{
"key": "jane",
"doc_count": 305
},
{
"key": "james",
"doc_count": 41
},
{
"key": "jim",
"doc_count": 22
},
{
"key": "jaime",
"doc_count": 1
}
]
}
}
]
}
}
}
To utilize this data and write it into long term reporting index, we could utilize Elasticsearch Alerting (previously Watcher) to store the aggregated data.
Another possibility is to use the Java Rest Client provided by Elastic. It is more fun. Java Programming and runs independent in a Spring Boot Docker container.
Search API
To start a search in the Java High Level Rest Client, we need a SearchRequest.
// we limit it to one index, wildcard patterns are working
SearchRequest searchRequest = new SearchRequest("six-audit-2018.06.02");
// set document type, since v6 optional
searchRequest.types("doc");
// Use a builder to construct the search query
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
Match Query
First we need to translate the query part.
{
"query": { "match": { "request": "MultiSearchRequest" } }
}
The QueryBuilders class provides all query builder classes.
QueryBuilder queryBuilder = QueryBuilders.matchQuery("request", "MultiSearchRequest");
Aggregations
The aggregations part has a date histogram aggregation with a terms sub-aggregation.
{
"aggs": {
"users_over_time": {
"date_histogram": {
"field": "@timestamp",
"interval": "1h"
},
"aggs": {
"users": {
"terms": {
"field": "principal"
}
}
}
}
}
}
The AggregationsBuilders class provides all aggregation builder classes.
// parent aggregation
DateHistogramAggregationBuilder aggregation = AggregationBuilders.
dateHistogram("users_over_time").
field("@timestamp").
dateHistogramInterval(DateHistogramInterval.hours(1));3 = set the aggregation name to users_over_time
The terms aggregation is a sub-aggregation.
// sub-aggregation
aggregation.subAggregation(AggregationBuilders.terms("users")
.field("principal"));
More on aggregations.
Complete Search Request
Putting it all together:
searchSourceBuilder.query(queryBuilder); //set query part
searchSourceBuilder.aggregation(aggregation); //set aggs part
// assign search query to search request
searchRequest.source(searchSourceBuilder);
To execute the search, pass the results to the SearchResponse object.
// construct client
RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(..));
SearchResponse searchResponse = client.search(searchRequest);
Retrieving aggregations
Aggregations can be retrieved from the SearchResponse by first getting the root of the aggregation tree, the Aggregations object, and then getting the aggregation by name.
RestStatus status = searchResponse.status();
if (status == RestStatus.OK) {
Aggregations aggregations = searchResponse.getAggregations();
Histogram dateHistogram = aggregations.get("users_over_time");
for (Histogram.Bucket bucket : dateHistogram.getBuckets()) {
LOGGER.info("Key: {}", bucket.getKeyAsString());
Map<String, Aggregation> aggregationMap = bucket.getAggregations().asMap();
Aggregation userAggregration = aggregationMap.get("users");
List<? extends Terms.Bucket> buckets = ((Terms) userAggregration).getBuckets();
LOGGER.info("-- {} active users", buckets.size());
for (Terms.Bucket user : buckets) {
LOGGER.info(" User {}", user.getKey());
}
}
}
This will result in following log output.
Key: 2018-06-07T00:00:00.000Z
-- 2 active users
User operations
User support
Key: 2018-06-07T01:00:00.000Z
-- 3 active users
User operations
User reporting_agent
User support
Key: 2018-06-07T02:00:00.000Z
-- 2 active users
User operations
User support
Key: 2018-06-07T03:00:00.000Z
-- 3 active users
User john
User operations
User support
Key: 2018-06-07T04:00:00.000Z
-- 4 active users
User john
User operations
User support
User jane
Key: 2018-06-07T05:00:00.000Z
-- 5 active users
User john
User urs
User operations
User support
User beat
Key: 2018-06-07T06:00:00.000Z
-- 7 active users
User urs
User john
User operations
User beat
User support
User peter
User luke
Key: 2018-06-07T07:00:00.000Z
-- 8 active users
User john
User alain
User urs
User daniel
User operations
User support
User peter
User saltzmann
Key: 2018-06-07T08:00:00.000Z
-- 9 active users
User john
User alain
User urs
User operations
User support
User peter
User saltzmann
User jane
User lemapper
Key: 2018-06-07T09:00:00.000Z
-- 9 active users
User john
User urs
User peter
User support
User beat
User operations
User nicolas
User thomas
User luke
Key: 2018-06-07T10:00:00.000Z
-- 10 active users
User john
User urs
User peter
User alain
User operations
User thomas
User joe
User nicolas
User richard
User support
Key: 2018-06-07T11:00:00.000Z
-- 10 active users
User john
User alain
User urs
User saltzmann
User daniel
User peter
User richard
User operations
User francesco
User micha
Key: 2018-06-07T12:00:00.000Z
-- 10 active users
User urs
User john
User alain
User peter
User lukas
User operations
User andreas
User daniel
User support
User michael
Key: 2018-06-07T13:00:00.000Z
-- 10 active users
User urs
User peter
User alain
User operations
User richard
User daniel
User andreas
User john
User jane
User saltzmann
Key: 2018-06-07T14:00:00.000Z
-- 10 active users
User urs
User alain
User john
User daniel
User peter
User nicolas
User operations
User patrick
User andreas
User christian
Key: 2018-06-07T15:00:00.000Z
-- 8 active users
User urs
User alain
User john
User operations
User peter
User patrick
User christian
User support
Key: 2018-06-07T16:00:00.000Z
-- 2 active users
User operations
User patrick
Key: 2018-06-07T17:00:00.000Z
-- 2 active users
User operations
User patrick
Key: 2018-06-07T18:00:00.000Z
-- 2 active users
User operations
User patrick
Key: 2018-06-07T19:00:00.000Z
-- 3 active users
User operations
User patrick
User francesco
Key: 2018-06-07T20:00:00.000Z
-- 2 active users
User operations
User patrick
Key: 2018-06-07T21:00:00.000Z
-- 2 active users
User urs
User operations
Key: 2018-06-07T22:00:00.000Z
-- 2 active users
User urs
User operations
Key: 2018-06-07T23:00:00.000Z
-- 1 active users
User operations
Summary
- Translating Search Requests into Java Objects is pretty straightforward.
-
QueryBuildershelp you construct queries in Java. -
AggregationBuildershelp you construct aggregations in Java. - Having the results in Java offers a lot of opportunities.
