Road to GTK4

[hongquan]

GTK4 has been released for a while and GTK3 will be deprecated. CoBang should migrate to GTK4 before we add more features.

The roadblock is how to rendering video (from webcam) to GTK4 widget. There is no Python binding for gst-plugins-rs which provides Gstreamer Sink for GTK 4. The only way to use this plugin is to rewrite CoBang in Rust language. In Rust, we can also use aperture which simplify usage of GTK4 Sink.

[bkmgit]

The rust code can be compiled to be interoperable with C and Python, see the example

A Rust and C version would be easier to make cross-platform than a Rust, C and Python version.

OpenCV maybe another choice:

• https://learnopencv.com/opencv-qr-code-scanner-c-and-python/
• https://learnopencv.com/barcode-and-qr-code-scanner-using-zbar-and-opencv/
• https://www.geeksforgeeks.org/webcam-qr-code-scanner-using-opencv/
• https://discourse.gnome.org/t/display-opencv-webcam-in-gtk-4-using-pygobject/17137/9

[hongquan]

@bkmgit
I know OpenCV, but OpenCV adds up too many unnecessary stuff (scanning QR is only a very small part of it). It also doesn't solve the issue of "porting to GTK4" (it only replaces ZBar). The last link you gave is just to paint "image" to GTK widget, not to play video (painting frame by frame as pictures is very laggy).

In order to follow the gtksink.py example, we have to find a way to build gst-plugins-rs to *.so files and package them to *.deb files. Its README has a guide to build, but to make *.deb packages is unknown. The process involves cargo cbuild command and I haven't found a way to integrate to debuilder process.

When thinking about the Rust version, I can see the ability to build both CoBang and gst-plugins-rs in a single ELF file, and hence, a single .deb package. It is just what I'm thinking, I haven't verified if it is doable or not.

[bkmgit]

Many Fedora packages which use rust bundle dependencies as they are usually pinned to specific versions that are often incompatible. This is not great.

https://apps.kde.org/qrca/ is built for KDE using Qt. MauiKit and WxWidgets are other options, there are other choices as well depending on how much change could be introduced to the code, and if using at most C/C++/Python/Rust is a constraint.

There has been a port of gtksink for C, but it has not been merged in main tree.

[bkmgit]

SDL3 has Python bindings https://github.com/Aermoss/PySDL3

Is there still interest in builds for mobile devices, in particular Android?

[hongquan]

• wxWidgets is linked to GTK3, not GTK4, so using wxWidget will make CoBang looks the same as current (already be GTK3).
• Qt-based toolkits like MauiKit may be an option. But because I use GTK-based DE, like GNOME, I'm not very fluent in Qt (doesn't know how to both show webcam video on Qt UI and extract its frame, good practice to make my Qt app in accordance with DE looks and feel, etc).

This is a typical GTK4 app, and how it looks in GNOME. I want CoBang to be like it.

Being GTK3, this is how CoBang looks currently:

CoBang can be valuable for Linux phones, like Librem, but not Android, because there are already dozen of QR scanning apps on Android. I tried to make CoBang support Librem once, but failed because the phone uses a type of camera which was not supported by Video4Linux subsystem. There has been a new lib, libcamera, to support wide variants of camera. But its integration with GStreamer is still limited. On Ubuntu 24.04, if Pipewire and libcamera are installed, CoBang will fails to access webcam.

[bkmgit]

Mostly use Fedora. Libcamera integrates with a wide variety of libraries. https://libcamera.org/getting-started.html
Can one use glsink instead of gtksink?

[bkmgit]

May need pipewire-libcamera?
https://launchpad.net/ubuntu/noble/+package/pipewire-libcamera

Pipewire integration seems useful:
https://blogs.gnome.org/uraeus/2024/03/15/pipewire-camera-handling-is-now-happening/

[bkmgit]

glimagesink not glsink

https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/issues/338

[bkmgit]

Libcamera -> pipewire -> gtk4 seems feasible
https://venam.net/blog/unix/2021/06/23/pipewire-under-the-hood.html

[bkmgit]

https://pypi.org/project/pipewire_python/ could be used for integration

ModernGl or PyOpenGl are other options https://discourse.gnome.org/t/moderngl-python-with-gtk4/18826

[bkmgit]

An example with GTK4 https://github.com/yucefsourani/python-gtk4-examples/tree/main/small_programs/pyscanbc

Maybe Cambalache could replace Glade?

[hongquan]

The glimagesink receives window ID, it means that it draw to the whole window, not a part of window as CoBang wants (we need to add some buttons, dropdown, tab, panes, not just video). Another problem is that, window ID is a notion in X Window System. It doesn't exist in Wayland.

I know that libcamera + PipeWire + GStreamer have been integrated together. It is just that the integration is not mature enough. CoBang source code has a block to handle this case, where it has to ignore the libcamera-based device that GStreamer reports. The data is private, no methods, attributes are exported for Python to handle.

The pyscanbc example doesn't solve the problem. My problem is "packaging gst-plugins-rs as *.deb", not "how to use Python + GTK4 + GStreamer".
On the other hand, the pyscanbc example just assume that you already have GStreamer-GTK4 plugin installed. It is doable by installing from source. But to packaging, to distribute your app, is a different story.

The ModernGl is also not an appropriate approach. If you use OpenGL directly, its mean that you have to draw the window, the buttons, the dropdown, the tabs, the menu yourselves. Using OpenGL directly is appropriate when you are making video game. All the buttons in video games are specific to the game and don't need to follow style of your DE.

Yes, dropping Glade is in my plan, when migrating to GTK4.

[bkmgit]

Dino uses Vala and has video integration using gdk paintable https://github.com/dino/dino/blob/master/plugins/rtp/src/video_widget.vala

Vala does not have packaging problems Rust has, and it is possible call from Python, see for example https://stackoverflow.com/questions/59987525/how-to-use-a-custom-gtk-widget-written-in-vala-from-python

Vala ecosystem has a few problems, https://blogs.gnome.org/mcatanzaro/2017/02/19/on-problems-with-vala/ but seems a better choice than Rust if the aim is to use GTK4.

[bkmgit]

gdkpaintable had Python bindings https://amolenaar.pages.gitlab.gnome.org/pygobject-docs/Gdk-4.0/interface-Paintable.html

[hongquan]

Good finding. Dino team is so strong that they implement a custom GStreamer sink themselves. This video_widget.vala is equivalent to the official Rust-based gst-plugin-gtk4.

But, unfortunately, Dino doesn't distribute their video sink as an independent lib for other apps to use. This video sink is linked with other Dino specific code to build the /usr/lib/x86_64-linux-gnu/dino-im/plugins/rtp.so file, which is just specific to Dino, CoBang cannot use it.
So, if CoBang wants to avoid building the Rust gst-plugins-rs, it has to implement a custom GStreamer video sink like Dino does. But I am not sure if it is doable in Python. I found this example https://github.com/GStreamer/gst-python/blob/master/examples/plugins/python/sinkelement.py but it is too minimal.

[bkmgit]

Will see if can implement a minimal Python example later this week.

[hongquan]

@bkmgit Great news, I found that the Gstreamer GTK4 plugin has been prebuilt for Ubuntu 24.10+.
It means that we can proceed without worrying about building + packaging gst-plugins-rs, and we can continue with Python, instead of migrating to Rust.

[bkmgit]

Still prefer to create custom plugin without rust dependencies.

[hongquan]

I don't know how it is for other distros, but for Debian / Ubuntu, the above prebuilt package is *.so file, we can use it without depending on Rust, just need to declare gstreamer1.0-gtk4 as one of dependencies.

[bkmgit]

Rust dependencies are usually vendored and pinned at particular versions. They need not be updated if there are any problems, and one would need to keep checking the package dependencies independently, rather than checking for library upgrades.

[bkmgit]

It is also in Fedora https://packages.fedoraproject.org/pkgs/rust-gst-plugin-gtk4/gstreamer1-plugin-gtk4/ but rust dependencies are all vendored. Would prefer not to use it and add necessary functionality if you are ok checking this.

[hongquan]

@bkmgit I believe you misunderstood the gstreamer1-plugin-gtk4 package in Fedora and the gstreamer1.0-gtk4 package in Ubuntu.
These two packages provide *.so file, which is prebuilt and no longer depend on Rust. You can check the dependency list of the Fedora here.

Because Rust does not have stable API, there are two forms of packaging Rust libs (in Debian and Fedora, at least to my knowledge):

• Include the Rust souce code and the package will depend on Rust. In this case, the package name starts with librust and when user install the package, the build process will run on user machine (that is why it needs to depend on Rust compiler).
• Prebuild to *.so file, using C ABI. The package no longer depends on Rust. When user install this package, he gets the .*so file which is ready to be used by applications. No build process happens on user machine, hence, not require Rust. This form is not applied to all Rust libs because it requires author preparation. This is the case for GStreamer because the GStreamer project has been distributed C ABI *.so file for applications to use, long before they adopt Rust.

The 2nd option may not apply to Arch Linux, Gentoo because they seem to always build package from source when user want to install a package.

[bkmgit]

The rust dependencies used to build the shared library are vendored. They only get used for building that library and are only updated when the library is updated. Typically, other dependencies are shared across the distribution, and consumed by any packages that use them. This leads to wider testing of the dependencies, and more reliable software but slows down development speeds. Something like xz library security bug would be harder to catch with rust dependencies.

[hongquan]

The rust dependencies used to build the shared library are vendored

Agree with you, but "vendored" in Debian context are the rustc, cargo and librust-xxx packages which are distributed by Debian repo. Hence, the GStreamer libs and the Rust vendored packages can be updated independently, as long as they are still compatible.

[hongquan]

The work of porting CoBang to GTK4 is in progress, at https://codeberg.org/quan/CoBangB
It is placed in other repository, because the source code is created by GNOME Builder with different structure. Later on I will copy to this repository.

[hongquan]

Code for GTK4 has been copied to feature/gtk4 branch.

[bkmgit]

Ok, will check. Can follow on Codeberg as well.

[hongquan]

v1.0 has been released to mark the migration to GTK4.
I wonder if it can be packaged for Fedora. I recently heard that Fedora faces the same issue as Debian / Ubuntu in terms of packaging Rust app / libraries, given that CoBang is depending on Rust-based GStreamer GTK4 plugin.