This post is older than a year. Consider some information might not be accurate anymore.
Used: logstash 6.1.2
A customer of mine, requires xml data as separate field data for further investigation. The data itself is part of a log message that is processed by Logstash. Logstash provides the powerful XML filter plugin for further parsing.
A test configuration `xml-filter.conf
input {
stdin {
codec => plain
}
}
filter {
grok {
match => {
"message" => "body=%{GREEDYDATA:xml}"
}
}
xml {
source => "xml"
target => "body"
remove_field => ["message","xml"]
}
}
output {
stdout { codec => "rubydebug" }
}Start logstash with above config:
bin/logstash -f xml-filter.conf
[2018-02-01T11:32:06,963][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250, :thread=>"#<Thread:0x517d9c29 run>"}
[2018-02-01T11:32:07,045][INFO ][logstash.inputs.stdin ] Automatically switching from plain to line codec {:plugin=>"stdin"}
[2018-02-01T11:32:07,148][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2018-02-01T11:32:07,328][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}Paste this example input into the console:
response: header={ "headerVersion" : "3.0", "specVersion" : "6.3.0", "securityLevel" : "MAC", "mac" : "xxx" }, body=<?xml version="1.0" encoding="UTF-8"?><ep2:message xmlns:ep2="http://www.eftpos2000.ch" specversion="0630"><ep2:authrsp msgnum="1141"><ep2:AcqID>4711</ep2:AcqID></ep2:authrsp></ep2:message>Logstash process it and outputs it:
{
"@timestamp" => 2018-02-01T10:32:07.397Z,
"body" => {
"specversion" => "0630",
"authrsp" => [
[0] {
"AcqID" => [
[0] "4711"
],
"msgnum" => "1141"
}
],
"xmlns:ep2" => "http://www.eftpos2000.ch"
},
"host" => "omega",
"@version" => "1"
}