Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
827 views
in Technique[技术] by (71.8m points)

mongodb - How to efficiently perform "distinct" with multiple keys?

For example, there is a collection like this:

{market: 'SH', code: '000001', date: '2012-01-01', price: 1000}
{market: 'SZ', code: '000001', date: '2012-01-01', price: 1000}
{market: 'SH', code: '000001', date: '2012-01-02', price: 1000}
{market: 'SZ', code: '000001', date: '2012-01-02', price: 1000}
{market: 'SH', code: '000002', date: '2012-01-03',price: 1000}
...

This collection contains tens of millions documents.

I want to call distinct with two keys:

collection.distinct('market', 'code');

and get result :

[{market: 'SH', code:'000001'}, {market: 'SZ', code:'000001'}, {market: 'SH', code:'000002'}]

As native distinct command accept only one key, I try to implement it by using map-reduce. But map-reduce is far too slow to native distinct. In my one-key distinct test, map-reduce spend about ten times longer than native distinct.
Is there a efficient way to implement multikey distinct?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

If you are willing to wait for the upcoming 2.2 release of MongoDB, you can run this query efficiently using the aggregation framework:

collection = db.tb;
result = collection.aggregate( 
            [
                {"$group": { "_id": { market: "$market", code: "$code" } } }
            ]
        );
printjson(result);

On a million-record collection on my test machine, this ran in 4 seconds, while the map/reduce version took over a minute.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...