How to create a new field from existing fields during indexing in ElasticSearch?

Question

I have a Lambda that receives events from Kinesis and writes the event to ElasticSearch cluster.

doc id	FirstTimestamp
d1	15974343498

question from:https://stackoverflow.com/questions/65898754/how-to-create-a-new-field-from-existing-fields-during-indexing-in-elasticsearch

Answer 1

The GET operation won't be necessary.

Depending on how easily you can configure how your writes happen, you could do the following:

1. Store a script which expects the doc-to-be-updated content as params:

POST _scripts/manage_time_tags
{
  "script": {
    "lang": "painless", 
    "source": """
      if (ctx._source.FirstTimestamp != null && params.FirstTimestamp != null) {
        ctx._source.SecondTimestamp = params.FirstTimestamp;
        ctx._source.TimeTag = ctx._source.SecondTimestamp - ctx._source.FirstTimestamp;
      }
    """
  }
}

2. Instead of directly writing to ES as you were up until now, use the upsert method of the Update API:

POST myindex/_update/1
{
  "upsert": {
    "id": 1,
    "FirstTimestamp": 15974343498
  },
  "script": {
    "id": "manage_time_tags",
    "params": {
      "id": 1,
      "FirstTimestamp": 15974343498
    }
  }
}

This will ensure that if the document does not exist yet, the contents of upsert are synced and the script doesn't even run.

3. As new events come in, simply call /_update/your_id again but with the most recent contents of id and FirstTimestamp.

POST myindex/_update/1
{
  "upsert": {
    "id": 1,
    "FirstTimestamp": 15974344498         
  },
  "script": {
    "id": "manage_time_tags",
    "params": {
      "id": 1,
      "FirstTimestamp": 15974344498
    }
  }
}

---

Note: this should not be confused with a rather poorly named scripted upsert which'll run the script irregardless of whether the doc already exists or not. This option should be omitted (or set to false).