使用 GL_TEXTURE_2D 的 iOS YUV 420v 在 OpenGL 着色器中显示错误的颜色
<p><p><strong>目标</strong>:使用 GL_TEXTURE_2D 而不是 CVOpenGLESTextureRef 将 YUV 数据(格式为 '<strong>420v</strong>' <strong>kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange</strong>)推送到着色器(为什么? 因为我需要使用glTexSubImage2d来操作像素，而目标是<code>CVOpenGLESTextureGetTarget(<name>)</code>的情况下不能使用，所以没有效果，必须使用GL_TEXTURE_2D)</p>
<p><strong>问题:</strong>
我正在使用自定义视频合成器来操作 AVPlayer 视频。当我在 Apple 的 <strong>AVCustomEdit</strong> 示例代码中使用 CVOpenGLESTextureRef 时，它使用 2 个单独的着色器，一个用于 Y(亮度)，一个用于 UV(色度)，视频看起来像这样正常:</p>
<p> <a href="/image/soMF4.jpg" rel="noreferrer noopener nofollow"><img src="/image/soMF4.jpg" alt="enter image description here"/></a> </p>
<p>但如果尝试使用 GL_TEXTURE_2D 来代替，视频只会显示绿色和粉红色，如下所示:</p>
<p> <a href="/image/ydiVt.jpg" rel="noreferrer noopener nofollow"><img src="/image/ydiVt.jpg" alt="enter image description here"/></a> </p>
<p>如果我将 GL_TEXTURE_2D 与结合了 Y 和 UV 纹理的片段着色器一起使用，它看起来会更糟:<a href="/image/1DpnT.jpg" rel="noreferrer noopener nofollow"><img src="/image/1DpnT.jpg" alt="enter image description here"/></a> </p>
<p>我的代码:</p>
<p>首先创建轨道缓冲区和目标缓冲区:</p>
<pre><code>CVPixelBufferRef foregroundSourceBuffer = ;
CVPixelBufferRef dstBuffer = ;
</code></pre>
<p>然后它们被传递给包含以下相关代码的渲染函数:</p>
<pre><code>CVOpenGLESTextureRef foregroundLumaTexture= ;
CVOpenGLESTextureRef foregroundChromaTexture = ;
CVOpenGLESTextureRef destLumaTexture = ;
CVOpenGLESTextureRef destChromaTexture = ;
</code></pre>
<p>亮度纹理函数返回:</p>
<pre><code>CVOpenGLESTextureRef luma = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
_videoTextureCache,
pixelBuffer,
NULL,
GL_TEXTURE_2D,
GL_RED_EXT,
(int)CVPixelBufferGetWidth(pixelBuffer),
(int)CVPixelBufferGetHeight(pixelBuffer),
GL_RED_EXT,
GL_UNSIGNED_BYTE,
0,
&lumaTexture);
</code></pre>
<p>色度纹理函数返回:</p>
<pre><code>CVOpenGLESTextureRef chroma = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
_videoTextureCache,
pixelBuffer,
NULL,
GL_TEXTURE_2D,
GL_RG_EXT,
(int)CVPixelBufferGetWidthOfPlane(pixelBuffer, 1),
(int)CVPixelBufferGetHeightOfPlane(pixelBuffer, 1),
GL_RG_EXT,
GL_UNSIGNED_BYTE,
1,
&chromaTexture);
</code></pre>
<p>现在渲染函数的相关主体:</p>
<pre><code> glBindFramebuffer(GL_FRAMEBUFFER, self.offscreenBufferHandle);
glViewport(0, 0, (int)CVPixelBufferGetWidthOfPlane(destinationPixelBuffer, 0), (int)CVPixelBufferGetHeightOfPlane(destinationPixelBuffer, 0));
#ifdef USE_GL_TEXTURE_2D
int bufferWidth = CVPixelBufferGetWidth(foregroundPixelBuffer);
int bufferHeight = CVPixelBufferGetHeight(foregroundPixelBuffer);
GLuint frameTextureY;
GLuint frameTextureUV;
glGenTextures(1, &frameTextureY);
glGenTextures(1, &frameTextureUV);
if(CVPixelBufferLockBaseAddress(foregroundPixelBuffer, 0) == kCVReturnSuccess){
glBindTexture(GL_TEXTURE_2D, frameTextureY);
glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE, bufferWidth, bufferHeight, 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddressOfPlane(foregroundPixelBuffer, 0));
glBindTexture(GL_TEXTURE_2D, frameTextureUV);
glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE_ALPHA, bufferWidth/2, bufferHeight/2, 0, GL_LUMINANCE_ALPHA, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddressOfPlane(foregroundPixelBuffer, 1));
CVPixelBufferUnlockBaseAddress(foregroundPixelBuffer, 0);
}
#endif
glActiveTexture(GL_TEXTURE0);
#ifdef USE_GL_TEXTURE_2D
glUseProgram(self.programYUV_2);
glBindTexture(GL_TEXTURE_2D, frameTextureY);
glUniformMatrix4fv(uniforms, 1, GL_FALSE, preferredRenderTransform);
#else
glUseProgram(self.programY);
glBindTexture(CVOpenGLESTextureGetTarget(foregroundLumaTexture), CVOpenGLESTextureGetName(foregroundLumaTexture));
glUniformMatrix4fv(uniforms, 1, GL_FALSE, preferredRenderTransform);
#endif
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
// Attach the destination texture as a color attachment to the off screen frame buffer
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, CVOpenGLESTextureGetTarget(destLumaTexture), CVOpenGLESTextureGetName(destLumaTexture), 0);
if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
NSLog(@"Failed to make complete framebuffer object %x", glCheckFramebufferStatus(GL_FRAMEBUFFER));
goto bail;
}
glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
glClear(GL_COLOR_BUFFER_BIT);
#ifdef USE_GL_TEXTURE_2D
glUniform1i(uniforms, 0);
glVertexAttribPointer(ATTRIB_VERTEX_Y_UV_INONESHADER, 2, GL_FLOAT, 0, 0, quadVertexData1);
glEnableVertexAttribArray(ATTRIB_VERTEX_Y_UV_INONESHADER);
glVertexAttribPointer(ATTRIB_TEXCOORD_Y_UV_INONESHADER, 2, GL_FLOAT, 0, 0, quadTextureData1);
glEnableVertexAttribArray(ATTRIB_TEXCOORD_Y_UV_INONESHADER);
#else
glUniform1i(uniforms, 0);
glVertexAttribPointer(ATTRIB_VERTEX_Y, 2, GL_FLOAT, 0, 0, quadVertexData1);
glEnableVertexAttribArray(ATTRIB_VERTEX_Y);
glVertexAttribPointer(ATTRIB_TEXCOORD_Y, 2, GL_FLOAT, 0, 0, quadTextureData1);
glEnableVertexAttribArray(ATTRIB_TEXCOORD_Y);
#endif
glDrawArrays(GL_TRIANGLE_STRIP, 0, 5);
glActiveTexture(GL_TEXTURE1);
#ifdef USE_GL_TEXTURE_2D
//no need to use different program
glBindTexture(GL_TEXTURE_2D, frameTextureUV);
glUniformMatrix4fv(uniforms, 1, GL_FALSE, preferredRenderTransform);
#else
glUseProgram(self.programUV);
glBindTexture(CVOpenGLESTextureGetTarget(foregroundChromaTexture), CVOpenGLESTextureGetName(foregroundChromaTexture));
glUniformMatrix4fv(uniforms, 1, GL_FALSE, preferredRenderTransform);
#endif
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glViewport(0, 0, (int)CVPixelBufferGetWidthOfPlane(destinationPixelBuffer, 1), (int)CVPixelBufferGetHeightOfPlane(destinationPixelBuffer, 1));
// Attach the destination texture as a color attachment to the off screen frame buffer
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, CVOpenGLESTextureGetTarget(destChromaTexture), CVOpenGLESTextureGetName(destChromaTexture), 0);
if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
NSLog(@"Failed to make complete framebuffer object %x", glCheckFramebufferStatus(GL_FRAMEBUFFER));
goto bail;
}
glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
glClear(GL_COLOR_BUFFER_BIT);
#ifdef USE_GL_TEXTURE_2D
glUniform1i(uniforms, 1);
glVertexAttribPointer(ATTRIB_VERTEX_Y_UV_INONESHADER, 2, GL_FLOAT, 0, 0, quadVertexData1);
glEnableVertexAttribArray(ATTRIB_VERTEX_Y_UV_INONESHADER);
glVertexAttribPointer(ATTRIB_TEXCOORD_Y_UV_INONESHADER, 2, GL_FLOAT, 0, 0, quadTextureData1);
glEnableVertexAttribArray(ATTRIB_TEXCOORD_Y_UV_INONESHADER);#else
glUniform1i(uniforms, 1);
glVertexAttribPointer(ATTRIB_VERTEX_UV, 2, GL_FLOAT, 0, 0, quadVertexData1);
glEnableVertexAttribArray(ATTRIB_VERTEX_UV);
glVertexAttribPointer(ATTRIB_TEXCOORD_UV, 2, GL_FLOAT, 0, 0, quadTextureData1);
glEnableVertexAttribArray(ATTRIB_TEXCOORD_UV);
#endif
glDrawArrays(GL_TRIANGLE_STRIP, 0, 5);
glFlush();
bail:
#ifdef USE_GL_TEXTURE_2D
glDeleteTextures(1, &frameTextureY);
glDeleteTextures(1, &frameTextureUV);
#endif
CFRelease(foregroundLumaTexture);
CFRelease(foregroundChromaTexture);
CFRelease(destLumaTexture);
CFRelease(destChromaTexture);
// Periodic texture cache flush every frame
CVOpenGLESTextureCacheFlush(self.videoTextureCache, 0);
</code></pre>
<p>这是我的片段着色器，我根据不同的测试用例使用(无论是单独绘制 Y 和 UV 还是一起绘制):</p>
<pre><code>static const char kFragmentShaderY[] = {
"varying highp vec2 texCoordVarying; \n \
uniform sampler2D s_texture_y; \n \
void main() \n \
{ \n \
gl_FragColor.r = texture2D(s_texture_y, texCoordVarying).r; \n \
}"
</code></pre>
<p>};</p>
<pre><code>static const char kFragmentShaderUV[] = {
"varying highp vec2 texCoordVarying; \n \
uniform sampler2D s_texture_uv; \n \
void main() \n \
{ \n \
gl_FragColor.rg = texture2D(s_texture_uv, texCoordVarying).rg; \n \
}"
</code></pre>
<p>};</p>
<pre><code>static const char kFragmentShaderYUV_2Textures[] = {
"varying highp vec2 texCoordVarying; \n \
uniform sampler2D s_texture_y; \n \
uniform sampler2D s_texture_uv; \n \
\n \
void main() \n \
{ \n \
mediump vec3 yuv;// = vec3(1.1643 * (texture2D(s_texture_y, texCoordVarying).r - 0.0625), \n \
lowp vec3 rgb; \n \
yuv.x = texture2D(s_texture_y, texCoordVarying).r; \n \
yuv.yz = texture2D(s_texture_uv, texCoordVarying).rg - vec2(0.5, 0.5); \n \
\n \
rgb = mat3( 1, 1, 1, \n \
0, -.21482, 2.12798, \n \
1.28033, -.38059, 0) * yuv; \n \
gl_FragColor = vec4(rgb, 1.0); \n \
}"
</code></pre>
<p>};</p>
<p>使用 GL_TEXTURE_2D，如果我使用包含 Y 和 UV 纹理的片段着色器，视频看起来像上面的 #3。如果我使用两个单独的片段着色器(一个用于 Y，一个用于 UV)，图片是上面的 #2(几乎正确，但色度颜色只是绿色和粉红色)*(请注意，我确实注释掉了上面的一些代码能够使用 2 个单独的片段着色器，当然我 glBind 到 GL_TEXTURE_2D 而不是 CV，依此类推，依此类推)。</p>
<p>再次，我的问题是我需要使用 GL_TEXTURE_2D 而不是 CVOpenGLESTextureGetTarget，但如果我这样做，它不会显示正确的色度颜色。我想知道我做错了什么。可能与 YUV 格式 kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange 而不是 kCVPixelFormatType_420YpCbCr8BiPlanarFullRange 有关吗？我还尝试了制作 3 个 GL_LUMINANCE 纹理的方法，以及许多其他没有运气的排列。</p></p>
<br><hr><h1><strong>Best Answer-推荐答案</ strong></h1><br>
<p><p>事实证明，问题在于使用 GL_LUMINANCE 和 GL_LUMINANCE_ALPHA，这显然是不推荐使用的格式。当我将它们切换到 GL_RED_EXT 和 GL_RG_EXT 时，它起作用了，色度颜色终于正确了。我希望这个问题和答案可以节省其他人的时间。</p></p>
<p style="font-size: 20px;">关于使用 GL_TEXTURE_2D 的 iOS YUV 420v 在 OpenGL 着色器中显示错误的颜色，我们在Stack Overflow上找到一个类似的问题：
<a href="https://stackoverflow.com/questions/37091132/" rel="noreferrer noopener nofollow" style="color: red;">
https://stackoverflow.com/questions/37091132/
</a>
</p>
页:
[1]