elasticsearch - Kibana Error Alerting - Filter Similiar Error Messages | Similarity Query [Elastalert]

Question

My goal is to be alerted for unique error types only (Elasticsearch 6.8 + Elastalert).

Example error logs (5 documents, same index, same field):

• Error 9837 in Component X trying to reconnect in 9.3 seconds
• Error 9837 in Component X trying to reconnect in 8.7 seconds
• Attempt to connect to 19.324.21.234 failed
• 2021/01/01 08:51:54.203 Errno 111 tensorflow/stream_executor...
• 2021/01/01 08:52:76.009 Errno 111 tensorflow/stream_executor...

Using query_key from Elastalert I can make sure to be alerted for unique error messages only. The error messages above are all unique and therefore I will receive 5 alerts. However I only want to receive 3 alerts, one per error type.

Elastalert allows you to write regular queries with Elasticsearch's Query DSL.

Is there a way to write a query which filters out documents if a specific field is 100% - 90% similiar to other documents?

I suspect there might be a totally different solution to this. If there are better ways to handle alerting or some best practices, feel free to share them.

question from:https://stackoverflow.com/questions/65937921/kibana-error-alerting-filter-similiar-error-messages-similarity-query-elast

Answer 1

One can add a new field with the Levenshtein distance