opengl - Rotating a Group of Vectors

Question

I am trying to rotate a group of vectors I sampled to the normal of a triangle

If this was correct, the randomly sampled hemisphere would line up with the triangle.

Currently I generate it on the Z-axis and am attempting to rotate all the samples to the normal of the triangle.

but it seems to be "just off"

glm::quat getQuat(glm::vec3 v1, glm::vec3 v2)
{

    glm::quat myQuat;
    float dot = glm::dot(v1, v2);
    if (dot != 1)
    {
        glm::vec3 aa = glm::normalize(glm::cross(v1, v2));
        float w = sqrt(glm::length(v1)*glm::length(v1) * glm::length(v2)*glm::length(v2)) + dot;
        myQuat.x = aa.x;
        myQuat.y = aa.y;
        myQuat.z = aa.z;
        myQuat.w = w;
    }
    return myQuat;
}

Which I pulled from the bottom of this page : http://lolengine.net/blog/2013/09/18/beautiful-maths-quaternion-from-vectors

Then I :

glm::vec3 zaxis = glm::normalize( glm::vec3(0, 0, 1) );  // hardcoded but test orginal axis
glm::vec3 n1 = glm::normalize( glm::cross((p2 - p1), (p3 - p1)) ); //normal
glm::quat myQuat = glm::normalize(getQuat(zaxis, n1));

glm::mat4 rotmat = glm::toMat4(myQuat); //make a rotation matrix
glm::vec4 n3 = rotmat * glm::vec4(n2,1); // current vector I am trying to rotate

Answer 1

Construct 4x4 transform matrix instead of Quaternions.

1. Do not forget that OpenGL has column wise matrix

   so for double m[16];
   is X axis vector in m[ 0],m[ 1],m[ 2]
   is Y axis vector in m[ 4],m[ 5],m[ 6]
   is Z axis vector in m[ 8],m[ 9],m[10]
   and position is in m[12],m[13],m[14]

   The LCS mean local coordinate system (your triangle or object or whatever)
   and GCS mean global coordinate system (world or whatever).

   All the X,Y,Z vectors should be normalized to unit vectors otherwise scaling will occur.

2. construction

   1. set Z-axis vector to your triangle normal
   2. set position (LCS origin) to mid point of your triangle (or average point form its vertexes)

   3. now you just need X and Y axises which is easy

      let X = any triangle vertex - triangle midpoint
      or X = substraction of any 2 vertexes of triangle

      The only condition that must be met for X is that it must lie on triangle plane.
      Now let Y = X x Z the cross product will create vector perpendicular to X and Z (which also lies in triangle plane).

   4. now put all this inside matrix and load it to OpenGL as ModelView matrix or what ever.