Resolve multiline grokparsefailure with regex

This post is older than a year. Consider some information might not be accurate anymore.

On a daily logrotate I have noticed that one of my monitored applications is doing a multiline input, but due to a misconfiguration in logstash, the grokparsefailure appears. I just want to leave the message as it is. Therefore logstash provides event dependent configuration capabilities with regular expression comparisons.

Following Kibana picture shows the message and its grokparsefailure in the field tags.

Every daily logrotate starts with three dots (...). A small extension in the filter configuration does the work:

filter {
    # if not daily log rotate
    if [message] !~ "^(\.{3})" {
        grok {
          # do some grokking
        }
    }
}

!~ is a regex expression that checks, that the message does not start with three dots. Since a dot is a special sign in regex, it needs an escape and we use the quantifier 3 to omit the occurence of three dots.

Regular expressions can be hard, therefore I recommend using for logstash (ruby) a regex tester like Rubular (see below) for testing.

For more information about Logstash Event Dependent Configuration.