Elasticsearch Date Processor Pipeline

This post is older than a year. Consider some information might not be accurate anymore.

I write some configuration documents with the Elasticsearch low level Java Rest Client. The documents are missing a timestamp, therefore I define a simple pipeline, which adds the @timestamp field to my documents.

• Definition
• Test
• REST Endpoint

Definition

Create pipeline

PUT _ingest/pipeline/timestamp
{
  "description" : "add timestamp field to the document",
   "processors" : [
    {
      "date" : {
        "field" : "timestamp",
        "formats" : ["yyyyMMddHHmm"],
        "timezone" : "Europe/Zurich"
      }
    }
  ]
}

Test

Test the pipeline, we take test data from an existing document

GET blackops/logstream/dev-F

The output

{
  "_index": "blackops",
  "_type": "logstream",
  "_id": "dev-F",
  "_version": 93,
  "found": true,
  "_source": {
    "logfile": "$POSDAT.DVTKSMDL.LF000007",
    "logfilePosition": 1546188226561,
    "timestamp": "201707191542",
    "logstrom": "F"
  }
}

Simulate with the test data

POST _ingest/pipeline/timestamp/_simulate
{
  "docs": [
    {
      "_source": {
        "logfile": "$POSDAT.DVTKSMDL.LF000007",
        "logfilePosition": 1546188226561,
        "timestamp": "201707191542",
        "logstrom": "F"
      }
    }
  ]
}

Output with the new timestamp field

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_type": "_type",
        "_source": {
          "@timestamp": "2017-07-19T15:42:00.000+02:00",
          "logfile": "$POSDAT.DVTKSMDL.LF000007",
          "logfilePosition": 1546188226561,
          "logstrom": "F",
          "timestamp": "201707191542"
        },
        "_ingest": {
          "timestamp": "2017-07-19T13:49:15.480Z"
        }
      }
    }
  ]
}

REST Endpoint

Use the pipeline by passing the param

PUT blackops/logstream/dev-F?pipeline=timestamp