python - How to process VLC UDP stream over OpenCV

Question

I was able to receive/view UDP h264 packets through VLC command line (i.e. VLC --network-caching 0 --demux h264 udp://...)

I am planning on processing those frames through OpenCV algorithms. However, I can't seem to find a way to send the VLC frames over to my Python OpenCV script.

Is it possible to pipe VLC stream output to be processed through Numpy in a separate script?

I have previously tried directly streaming to OpenCV by using its VideoCapture function, but for some reason, the video stalls and stops at a distorted black image. For now, it seems like incorporating VLC is the only solution (even though I am not totally sure why VLC works over other methods).

Thank you.

EDIT:

The following is a snippet of the error message on the terminal. It seems like there are problems with the first few frames, but I don't know why the stream works on VLC. From the client, I first sent a default key frame data, and then sent video feed h264 data.

[h264 @ 0x7f9c50020200] top block unavailable for requested intra mode -1
[h264 @ 0x7f9c50020200] error while decoding MB 7 0, bytestream 7208
[h264 @ 0x7f9c50020200] top block unavailable for requested intra mode -1
[h264 @ 0x7f9c50020200] error while decoding MB 8 9, bytestream 7381

Answer 1

You can use ffmpeg for streaming.

First test ffmpeg streaming in terminal. In linux, we use v4l2 to grab frames from camera.

Server:

ffmpeg -f v4l2 -i /dev/video0 -preset ultrafast -vcodec libx264 -tune zerolatency -b 900k -f h264 udp://127.0.0.1:5000

Client:

ffplay udp://127.0.0.1:5000

If you're able to view the stream on the client side, then we can use OpenCV for image processing. OpenCV must have ffmepg support. See this link for ffmpeg support check.

    cap = cv2.VideoCapture('udp://127.0.0.1:5000',cv2.CAP_FFMPEG)
    if not cap.isOpened():
        print('VideoCapture not opened')
        exit(-1)

    while True:
        ret, frame = cap.read()

        if not ret:
            print('frame empty')
            break

        cv2.imshow('image', frame)

        if cv2.waitKey(1)&0XFF == ord('q'):
            break

    cap.release()
    cv2.destroyAllWindows()