You may organize your points in an Octree. Then you only need to search a small subset.
A Octree is a fairly simple data structure you can implement yourself (which would be a valuable learning experience), or you may find some helpful libraries to get you going.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…