Ship Monit logs with Filebeat

This post is older than a year. Consider some information might not be accurate anymore.

A quick recipe how to ship Monit logs to Elasticsearch. Some initial configuration was in place but I ran into some troubles.

Problems

1. Monit logs disappeared on the 1st of October
2. Multiline messages haven’t been properly processed

Example logs

[CEST Oct  5 16:54:19] error    : 'batch_healthcheck' failed protocol test [HTTP] at [0.0.0.0]:80/batch/systemStatus [TCP/IP] -- Connection refused
[CEST Oct  5 16:54:19] info     : 'batch_healthcheck' exec: /bin/bash
[CEST Oct  5 16:54:19] error    : 'imp4' process is not running
[CEST Oct  5 16:54:19] info     : 'imp4' trying to restart
[CEST Oct  5 16:54:19] info     : 'imp4' start: /opt/six/fo/jboss/bin/jboss-opr.sh
[CEST Oct  5 16:54:52] error    : 'imp4' failed to start (exit status 1) -- /opt/six/fo/jboss/bin/jboss-opr.sh: tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
2017-10-05 16:54:19 root jboss.sh: imp4 not known
2017-10-05 16:54:19 root jboss.sh: imp4 could not be started

1. Monit logs disappeared

The reason is a simple one. Monit just don’t log day of month with leading zero.

[CEST Oct  5 16:54:19] error    : 'imp4' process is not running

Just add the date pattern for a single day in the date processor formats.

{
  "date": {
    "field": "jesus",
    "target_field": "datetime",
    "formats": [
      "MMM dd HH:mm:ss",
      "MMM  d HH:mm:ss"
    ],
    "timezone": "Europe/Zurich"
  }
}

2. Multiline messages

Filebeat sends multiline messages by this configuration:

multiline.pattern: '^\['
multiline.negate: false
multiline.match: before

In the ingest pipeline the message is truncated since the newline character collides with the grok processor. With the gsub processor you can remove the newline in order to be properly kept by the grok processor.

{
  "gsub": {
    "field": "message",
    "pattern": "\n",
    "replacement": " "
  }
}

Solution

Extend pipeline

PUT _ingest/pipeline/monit_logs

{
  "description": "grok pipeline for monit logs",
  "processors": [
    {
      "gsub": {
        "field": "message",
        "pattern": "\n",
        "replacement": " "
      }
    },
    {
      "grok": {
        "field": "message",
        "patterns": [
          ""
          "\[%{WORD} %{GREEDYDATA:jesus}\] %{WORD:level} %{SPACE} : \'%{GREEDYDATA:service}\' %{GREEDYDATA:logmessage}"
          ""
        ]
      }
    },
    {
      "date": {
        "field": "jesus",
        "target_field": "datetime",
        "formats": [
          "MMM dd HH:mm:ss",
          "MMM  d HH:mm:ss"
        ],
        "timezone": "Europe/Zurich"
      }
    },
    {
      "remove": {
        "field": [
          "message",
          "jesus"
        ]
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "error",
        "value": " on operation: "
      }
    }
  ]
}

Configure filebeat

# monit logs
- input_type: log
  paths:
     - /var/log/monit.log
  exclude_files: [".gz$"]
  fields:
    type: "logs"
    host: "${beat.hostname:dhost}"
    application: "monit"
    environment: "${FO_ENV:default}"
  fields_under_root: true
  multiline.pattern: '^\['
  multiline.negate: false
  multiline.match: before
  pipeline: "monit_logs"