Migrate elasticsearch indices from different clusters with logstash

This post is older than a year. Consider some information might not be accurate anymore.

I got an exceptional case in the office. Some application logs, which belongs to a dev and testing environment, were stored or reported in the elasticsearch production cluster. Therefore a cleanup or migration was necessary.

logstash is an easy solution for migrating data from cluster a to cluster b. In my case cluster production to cluster test. logstash provides elasticsearch as input and output plugin. The input queries elasticsearch and retieves the documents as json. The output writes the json to the target elasticsearch cluster. An example configuration.

vinh@omega:~/logstash-2.4.0> cat copy-data.conf
input{
    elasticsearch {
        hosts => [ "prod-dc1", "prod-dc2", "alpha-dc2", "beta-dc2" ]
        index => "trx-*"
        user => "admin"
        password => "SiriSingsRiri"
    }
}
output {
    stdout { codec => "rubydebug" }
    elasticsearch {
        hosts => [ "dev", "delta", "gamma" ]
        index => "trx-%{+YYYY.MM.dd}"
    }
}