Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
477 views
in Technique[技术] by (71.8m points)

elasticsearch - Is there any way to aggregate an average without outliers on elastic?

I need a way to create a transform that will aggregate the average of a field but without the outliers (let's say only the values that falls between 10%-90% percentiles). for example if I have the following documents:

[
{someField:1},
{someField:2},
{someField:3},
{someField:4},
{someField:5},
{someField:6},
{someField:7},
{someField:8},
{someField:9},
{someField:10}
]

It would calculate the average of 2-9

Edited: renamed "value" to "someField"


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

You can do this in one go with a scripted_metric aggregation but you'd have to write the percentiles function and then the avg function -- I wrote one here. But the script isn't going to be performant so I don't think it'd be worth the effort…

I'd instead suggest to first retrieve the percentile bounds:

POST myindex/_search
{
  "size": 0,
  "aggs": {
    "boundaries": {
      "percentiles": {
        "field": "value",
        "percents": [
          10,
          90
        ]
      }
    }
  }
}

yielding [1.5, 9.5] and then plug these numbers into a weighted average aggregation:

POST myindex/_search
{
  "size": 0,
  "aggs": {
    "avg_without_outliers": {
      "weighted_avg": {
        "value": {
          "field": "value"
        },
        "weight": {
          "script": {
            "source": "def v = doc.value.value; return v <= params.min || v >= params.max ? 0 : 1",
            "params": {
              "min": 1.5,
              "max": 9.5
            }
          }
        }
      }
    }
  }
}

The weight is either 0 or 1, depending on whether the particular doc that's being traversed is an outlier or not.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...