Custom grok patterns for logstash

This post is older than a year. Consider some information might not be accurate anymore.

grok is a powerful filter plugin. It allows you to use regular expressions (regex). This post gives some advices how to test and develop custom grok patterns for logstash.

First, you can use the Oniguruma syntax for named capture which will let you match a piece of text and save it as a field:

(?<field_name>the pattern here)

Oniguruma is the regular expression engine that is built-in in Ruby since v1.9.

Having a log entry

09:33:45,416 (metrics-logger-reporter-1-thread-1) type=GAUGE, name=notifications.received, value=2

You can use the grok pattern

(?<logtime>%{HOUR}:%{MINUTE}:%{SECOND}) (?<logthread>(?:[()a-zA-Zd-]+)) type=(?<type>([A-Z])w+), name=(?<name>(?:[A-Za-z.]*)w+), value=(?<value>([0-9]|NaN)+)

Pay attention that the value might be number or NaN. It is easier to use the kv (key-value) plugin, but this is solely for demonstration. You may also define custom patterns from the retrieved regular expressions, but this won’t be handled in this article.

The logstash configuration

filter {
  grok {
    match => { "message" => "(?<logtime>%{HOUR}:%{MINUTE}:%{SECOND}) (?<logthread>(?:[()a-zA-Zd-]+)) type=(?<type>([A-Z])w+), name=(?<name>(?:[A-Za-z.]*)w+), value=(?<value>([0-9]|NaN)+)" }
  }
}

Regular expressions itself may be a complicated issue, therefore there is a fantastic online ruby regular expression editor: Rubular to assist you finding the right regex.

You can also use the grok debugger for testing the regex.