Convert from CSV to JSON with logstash

This post is older than a year. Consider some information might not be accurate anymore.

logstash allows with various plugins a quick solution, to convert CSV input to JSON output.

This test configuration demonstrates a simple example with nested properties.

# convert from csv to json
input {
    generator {
        type => 'csv'
        message => '2015-10-08,CH,4711'
        count => 1
    }
}
filter {
    csv {
        columns => ['date', 'countryCode', 'value']
    }
    mutate {
        rename => {
            "date" => "[payment][date]"
            "countryCode" => "[payment][origin]"
            "value" => "[payment][value]"
        }
    }
}
output {
    stdout {
        codec => rubydebug{}
    }
}
output {
    stdout {
        codec => json{}
    }
}

• With the generator input plugin, we generate a CSV line.
• The csv filter plugin, allows the parsing of the CSV data.
• The mutate filter plugin, allows with rename to nest the properties under payment.
• The output contains two definitions, the rubydebug and the JSON output (pretty printed by me).

The output

Logstash startup completed
{
       "message" => [
        [0] "2015-10-08,CH,4711"
    ],
      "@version" => "1",
    "@timestamp" => "2015-10-08T14:31:03.344Z",
          "type" => "csv",
          "host" => "cinhtau",
      "sequence" => 0,
       "payment" => {
          "date" => "2015-10-08",
        "origin" => "CH",
         "value" => "4711"
    }
}
{
    "message":["2015-10-08,CH,4711"],
    "@version":"1",
    "@timestamp":"2015-10-08T14:31:03.344Z",
    "type":"csv",
    "host":"cinhtau",
    "sequence":0,
    "payment":{
        "date":"2015-10-08",
        "origin":"CH",
        "value":"4711"
    }
}
Logstash shutdown completed