python - How to eliminate same docs to es index

Question

Imagine that I have some docs and I am putting them on an elasticsearch index with the below python code;

es.index(index=index_name, doc_type='_doc', body=doc)

If i run the code twice, same 2 docs will be on this elasticsearch index. How can i eliminate this duplication docs on same index ? Thanks for answering

question from:https://stackoverflow.com/questions/66049082/how-to-eliminate-same-docs-to-es-index

Answer 1

You can set the ?op_type=create parameter to index a document only if no document with that ID exists. For example:

Request 1

PUT /customers/_doc/1?op_type=create
{
  "name": "Jane Doe"
}

Response 1

{
  "_index" : "customers",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Request 2

PUT /customers/_doc/1?op_type=create
{
  "name": "Jane Doe"
}

Response 2

{
  "error" : {
    "root_cause" : [
      {
        "type" : "version_conflict_engine_exception",
        "reason" : "[1]: version conflict, document already exists (current version [1])",
        "index_uuid" : "5Oar4GFhQjq_EM88qqd6PA",
        "shard" : "0",
        "index" : "customers"
      }
    ],
    "type" : "version_conflict_engine_exception",
    "reason" : "[1]: version conflict, document already exists (current version [1])",
    "index_uuid" : "5Oar4GFhQjq_EM88qqd6PA",
    "shard" : "0",
    "index" : "customers"
  },
  "status" : 409
}