Error Releasing VideoCapture in OpenCV Using GStreamer Pipeline

[gasparramoa]

Dear all,

Thank you for developing such a helpful library.

Problem:

I am trying to get frames using opencv.
I know that libcamera does not entirely support cv2 (I already created another issue for that kbingham/libcamera#96).
I am actually able to get frames via cv2.VideoCapture through GStreamer using the following:

import cv2
camera = "/base/soc/i2c0mux/i2c@1/imx219@10"
pipeline = f"libcamerasrc camera-name={camera} ! video/x-raw,width=1640,height=1232,format=BGR,framerate=40/1 ! videoconvert ! appsink"
cap = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)

However, when I try to release the capture (for ending the program or maybe adjusting raw image size): cap.release()
I get the following error:

[1:39:52.854669803] [1885] ERROR DeviceEnumerator device_enumerator.cpp:172 Removing media device /dev/media4 while still in use
[1:39:52.854709759] [1885] ERROR DeviceEnumerator device_enumerator.cpp:172 Removing media device /dev/media3 while still in use

System:

• OS Bullseye LITE
• Arducam IMX219
• Rbpi 4
• Libcamera: v0.0.5+83-bde9b04f
• Python 3.9.2

What I already tried:

I updated my libcamera from v0.0.5 to v0.3.0+65-6ddd79b5 (clone this repo and use the build instructions)
I updated the libcamera-apps build to rpicam-apps build: 511941fdb1fe-intree
I build the custom cv2 version of advait-0, used cv2.VideoCapture(0, cv2.CAP_LIBCAMERA), but unfortunately I got the same problem. I cannot release the camera without an error in DeviceEnumerator.

More Details:

This has definitely to do with gstlibcamerasrc.cpp. Apparently this error happened before and this was the solution: fa63d42
I am confident it has to do with that... I also activated the logs of Gstreamer and Libcamera and present them here as soon as I start the release():

(...)
[2:28:30.630557394] [2011] DEBUG RPiDelayedControls delayed_controls.cpp:240 frame 147 started
[2:28:30.630634539] [2011] DEBUG RPiDelayedControls delayed_controls.cpp:283 Queue is empty, auto queue no-op.
[2:28:30.655460147] [2011] DEBUG RPI pipeline_base.cpp:1356 Frame start 148
[2:28:30.655547513] [2011] DEBUG RPiDelayedControls delayed_controls.cpp:240 frame 148 started
[2:28:30.655623807] [2011] DEBUG RPiDelayedControls delayed_controls.cpp:283 Queue is empty, auto queue no-op.

Trying to Releasing camera...True
0:00:03.891307777  2007  0x16aa820 DEBUG           libcamerasrc gstlibcamerasrc.cpp:691:gst_libcamera_src_task_leave:<libcamerasrc0> Streaming thread is about to stop
[2:28:30.669317521] [2013] DEBUG Camera camera.cpp:1385 Stopping capture
0:00:03.909948584  2007  0x169b900 DEBUG           libcamerasrc gstlibcamerasrc.cpp:713:gst_libcamera_src_close:<libcamerasrc0> Releasing resources
[2:28:30.688185411] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1539 /dev/video0[13:cap]: Releasing buffers
[2:28:30.688691723] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1254 /dev/video0[13:cap]: 0 buffers requested.
[2:28:30.688885030] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1539 /dev/video20[14:out]: Releasing buffers
[2:28:30.689265033] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1254 /dev/video20[14:out]: 0 buffers requested.
[2:28:30.689374269] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1539 /dev/video21[15:cap]: Releasing buffers
[2:28:30.689745217] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1254 /dev/video21[15:cap]: 0 buffers requested.
[2:28:30.689864860] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1539 /dev/video22[16:cap]: Releasing buffers
[2:28:30.690043667] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1254 /dev/video22[16:cap]: 0 buffers requested.
[2:28:30.690706177] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1539 /dev/video23[17:cap]: Releasing buffers
[2:28:30.690889540] [2013] DEBUG V4L2 v4l2_videodevice.cpp:1254 /dev/video23[17:cap]: 0 buffers requested.
[2:28:30.691083699] [2011] ERROR DeviceEnumerator device_enumerator.cpp:172 Removing media device /dev/media4 while still in use
[2:28:30.691222489] [2011] ERROR DeviceEnumerator device_enumerator.cpp:172 Removing media device /dev/media3 while still in use
Segmentation fault

[ShevaDavid]

faced the same problem.
@gasparramoa did you find any workaround for it ?

[nvvnst]

I have the same error.

• RPi 4 with OS Bullseye 64-bit
• imx477 and imx296 sensors

I tried both libcamera v0.0.5 and v0.3.0+65-6ddd79b5 versions.

Test script for rerpoducing error:

#include <opencv2/opencv.hpp>
#include <string>
#include <iostream>

int main() {
    std::string gst_pipeline = "libcamerasrc ! video/x-raw,framerate=30/1,width=1440,height=1080 ! appsink";
    cv::VideoCapture cam_(gst_pipeline, cv::CAP_GSTREAMER);
    int counter = 20;
    cv::Mat frame;
    while (counter > 0) {
        if (!cam_.read(frame)) {
            std::cout << "Could not read frame from the video stream." << std::endl;
            break;
        }
        counter--;
        std::cout << counter << std::endl;
    }
    cam_.release();
}

And gdb backtrace:

[Thread 0x7fe77fd640 (LWP 2536) exited]
[0:09:19.619052205] [2532] ERROR DeviceEnumerator device_enumerator.cpp:166 Removing media device /dev/media0 while still in use
[0:09:19.619125072] [2532] ERROR DeviceEnumerator device_enumerator.cpp:166 Removing media device /dev/media2 while still in use
[Thread 0x7fede1f640 (LWP 2532) exited]

Thread 1 "test" received signal SIGBUS, Bus error.
0x0000000000000006 in ?? ()
(gdb) backtrace
#0  0x0000000000000006 in  ()
#1  0x0000007fedee0b38 in libcamera::thread::postMessage(std::unique_ptr<libcamera::Message, std::default_delete<libcamera::Message> >, libcamera::Object*) ()
    at /lib/aarch64-linux-gnu/libcamera-base.so.0.0
#2  0x0000007fedede330 in libcamera::Object::postMessage(std::unique_ptr<libcamera::Message, std::default_delete<libcamera::Message> >) () at /lib/aarch64-linux-gnu/libcamera-base.so.0.0
#3  0x0000007fedede3e0 in libcamera::Object::deleteLater() ()
    at /lib/aarch64-linux-gnu/libcamera-base.so.0.0
#4  0x0000007fedf8b7e8 in  () at /lib/aarch64-linux-gnu/libcamera.so.0.0
#5  0x0000007fedfaad88 in libcamera::FrameBufferAllocator::~FrameBufferAllocator() ()
    at /lib/aarch64-linux-gnu/libcamera.so.0.0
#6  0x0000007fee097264 in  () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibcamera.so
#7  0x0000007ff700a960 in g_object_unref () at /lib/aarch64-linux-gnu/libgobject-2.0.so.0
#8  0x0000007fee098508 in  () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibcamera.so
#9  0x0000007ff700a960 in g_object_unref () at /lib/aarch64-linux-gnu/libgobject-2.0.so.0
#10 0x0000007ff70ab7b4 in  () at /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#11 0x0000007ff70e609c in gst_mini_object_unref () at /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#12 0x0000007ff7112ab0 in  () at /lib/aarch64-linux-gnu/libgstreamer-1.0.so.0
#13 0x0000007ff6e93928 in  () at /lib/aarch64-linux-gnu/libgstapp-1.0.so.0
#14 0x0000007ff700a8d0 in g_object_unref () at /lib/aarch64-linux-gnu/libgobject-2.0.so.0
#15 0x0000007ff7f699a4 in cv::GStreamerCapture::~GStreamerCapture() ()
    at /usr/local/lib/aarch64-linux-gnu/libopencv_videoio.so.407
#16 0x0000007ff7f39b44 in cv::VideoCapture::release() ()
    at /usr/local/lib/aarch64-linux-gnu/libopencv_videoio.so.407
#17 0x00000055555512a4 in main ()
(gdb)

[gasparramoa]

So far, I am still looking for a workaround for this. But I am confident it has to do with what Laurent Pinchart(@pinchartl) worked on in this issue: fa63d42.

[nvvnst]

I tested this on RPi OS Bookworm 64 bit and faced the same error.

[naushir]

I presume this is still an issue? To be honest, I don't know anything about gstreamer so I cannot really advise on this issue. Perhaps the libcamera mailing list might be a better place for this query?

[gasparramoa]

Hi @naushir! Thanks for reaching out. This is really important for the library. I was hoping that @pinchartl would come up with a solution since he/she has worked on this specific problem before. I will create a query for the libcamera mailing list, but I am not sure if that will reach more people or not.

[gasparramoa]

Hello, everyone. I sent an email to the libcamera development mailing list and immediately got a reply from @pinchartl.

I removed my libcamera and manually installed the latest release v0.3.2+98-75fe515a

[libcamera] gaspar@Ramoa:~ $ rpicam-still --version
rpicam-apps build: v1.5.3 50958df98d3c 13-11-2024 (16:55:55)
rpicam-apps capabilites: egl:1 qt:1 drm:1 libav:1
libcamera build: v0.3.2+98-75fe515a

One thing was that, on this version libcamerasrc is not recognized... (I am not sure if it is an option you set when building the library, but it would be cool to know why)

[ WARN:0] global ../modules/videoio/src/cap_gstreamer.cpp (734) open OpenCV | GStreamer warning: Error opening bin: no element "libcamerasrc"
[ WARN:0] global ../modules/videoio/src/cap_gstreamer.cpp (501) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Trying to Releasing camera...False

I fixed this by activating the following environment in my libcamera build folder:

gaspar@Ramoa:~/libcamera $ meson devenv -C build
[libcamera] gaspar@Ramoa:~/libcamera/build $

In this version, I can release the frame using gstreamer and opencv! 🎉🎉🎉 This is the pipeline I used:

[libcamerasrc camera-name=/base/soc/i2c0mux/i2c@1/imx219@10 ! video/x-raw, width=1600, height=1200, format=BGR, framerate=41/1 ! videoconvert ! appsink]

---

One crucial detail and another problem I had was choosing the right frame-size and frame-rate.
If I print my camera supported frame-sizes, it says it supports 1640x1232 at 41fps:

Available cameras
-----------------

0 : imx219 [3280x2464 10-bit RGGB] (/base/soc/i2c0mux/i2c@1/imx219@10)
    Modes: 'SRGGB10_CSI2P' : 640x480 [103.33 fps - (1000, 752)/1280x960 crop]
                             1640x1232 [41.85 fps - (0, 0)/3280x2464 crop]
                             1920x1080 [47.57 fps - (680, 692)/1920x1080 crop]
                             3280x2464 [21.19 fps - (0, 0)/3280x2464 crop]
           'SRGGB8' : 640x480 [103.33 fps - (1000, 752)/1280x960 crop]
                      1640x1232 [41.85 fps - (0, 0)/3280x2464 crop]
                      1920x1080 [47.57 fps - (680, 692)/1920x1080 crop]
                      3280x2464 [21.19 fps - (0, 0)/3280x2464 crop]

But for some reason gstreamer does not support this (I also printed the supported formats). The image has a stripping effect that suggests the frames are not being captured or processed.

[libcamera] tactonom@TactonomPro:~ $ gst-device-monitor-1.0 Video
Probing devices...


[0:35:26.013648824] [1350]  INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/home/tactonom/libcamera/build/src/ipa' to the IPA search path
[0:35:26.015230581] [1350]  INFO Camera camera_manager.cpp:325 libcamera v0.3.2+98-75fe515a
[0:35:26.029373562] [1351]  INFO IPAProxy ipa_proxy.cpp:134 libcamera is not installed. Loading IPA configuration from '/home/tactonom/libcamera/src/ipa/rpi/vc4/data'
[0:35:26.034870999] [1351]  WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[0:35:26.037532995] [1351]  WARN RPI vc4.cpp:393 Mismatch between Unicam and CamHelper for embedded data usage!
[0:35:26.038540068] [1351]  INFO RPI vc4.cpp:447 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media1 and ISP device /dev/media2

Device found:

        name  : /base/soc/i2c0mux/i2c@1/imx219@10
        class : Source/Video
        caps  : video/x-raw, format=NV21, width=160, height=120
                (...) ETC
                video/x-raw, format=NV21, width=1280, height=1024
                video/x-raw, format=NV21, width=1536, height=864
                video/x-raw, format=NV21, width=1280, height=1080
                video/x-raw, format=NV21, width=1600, height=900
                video/x-raw, format=NV21, width=1400, height=1050
                video/x-raw, format=NV21, width=1680, height=1050
                video/x-raw, format=NV21, width=1600, height=1200
                video/x-raw, format=NV21, width=1920, height=1080
                video/x-raw, format=NV21, width=2048, height=1080
                (...) ETC
                video/x-raw, format=NV21, width=[ 64, 3280, 2 ], height=[ 64, 2464, 2 ]
                video/x-raw, format=I420, width=160, height=120
(...) it continues.. but you get the point

The format closest to the original 1640x1232 is 1600x1200 and I get the correct frame by using this size format: