Add automatic timestamp to new documents

This post is older than a year. Consider some information might not be accurate anymore.

In the past Elasticsearch could add automatically a timestamp field. Since Elasticsearch 5.x I have to use a pipeline to ingest that timestamp field to the document. As a major change the internal `` value has also changed.

• Release Notes 5.5.0
• Breaking Mapping Changes

In short

• Old Timestamp: 2017-09-04T15:48:52.560+0000
• New Timestamp: Mon Sep 04 15:48:52 CEST 2017 or Mon Sep 04 15:48:52 UTC 2017

For the new timestamp it results in a new date format, that contains zone names.

Zone names: Time zone names (‘z’) cannot be parsed → (Joda Time)[http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html].

The workaround for that is just to take the literal and match the date with the respective timezone. This results in following pipeline definition:

PUT _ingest/pipeline/timestamp
{
  "description": "add timestamp field to the document, requires a datetime field date mapping",
  "processors": [
    {
      "set": {
        "field": "datetime",
        "value": ""
      },
      "date" : {
        "field" : "datetime",
        "formats" : ["EEE MMM dd HH:mm:ss 'UTC' yyyy", "EEE MMM dd HH:mm:ss 'CEST' yyyy"],
        "timezone" : "Europe/Zurich",
        "target_field": "datetime"
      }
    }
  ]
}

To use this pipeline

PUT test/logs/vinh4711?pipeline=timestamp
{
  "message": "Hi cinhtau!"
}

Query the result

GET test/logs/vinh4711

The new ingested date

{
  "_index": "test",
  "_type": "logs",
  "_id": "vinh4711",
  "_version": 1,
  "found": true,
  "_source": {
    "datetime": "2017-09-04T15:54:29.000+02:00",
    "message": "Hi cinhtau!"
  }
}