Backport HEVC decoder as submitted to linux-media as v2 #6666

6by9 · 2025-02-12T17:34:24Z

This PR:

reverts the downstream HEVC driver and supporting patches
backports the driver as submitted as v2 (https://lore.kernel.org/r/[email protected]) with DT binding fixup requested by Rob H, and dependencies.
adds in the new SAND handling for DRM
reinstates the downstream single planar pixel formats for decode
reinstates being able to set the /dev/videoN node

This doesn't work against the existing FFmpeg as it is looking for extra flags to identify a device that it thinks can decode HEVC. @jc-kynesim was going to release his updated FFmpeg that handles this, otherwise I can look at whether we can expose a second /dev/videoN node that adds in those flags as an interim.

This wants more thorough testing, hence looking to push it to rpi-6.12.y.

popcornmix · 2025-02-12T19:42:32Z

I think we want ffmpeg update public (or the second /dev/videoN node) before merging this.

6by9 · 2025-02-12T19:54:41Z

I think we want ffmpeg update public (or the second /dev/videoN node) before merging this.

I'd asked @jc-kynesim whether the FFmpeg side had been released on Friday, and he'd replied

Not yet - but that is only the work of an hour or two. I'll do that today

I don't know if that happened.

6by9 · 2025-02-27T18:22:52Z

Rebased and reworked so that the client is required to pass the column height in the pitch field if the modifier is DRM_FORMAT_MOD_BROADCOM_SAND128.

It does require removing one validation clause in the core DRM code if a modifier is set, and that may be an argument with upstream (I'll look forward to it!)

We can now have a single new path that works with both DRM and EGL (as of Mesa 24.2.5) rendering.

This still has the revert of the new pixel format so that it shouldn't change anything on systems without Mesa 24.

This reverts commit d7c70b0.

This reverts commit 9629d77.

This reverts commit 64ff380.

By default when the last request object is completed, the whole request completes as well. But sometimes you want to manually complete a request in a driver, so add a manual complete mode for this. In req_queue the driver marks the request for manual completion by calling media_request_mark_manual_completion, and when the driver wants to manually complete the request it calls media_request_manual_complete(). Signed-off-by: Hans Verkuil <[email protected]>

Manually complete the requests: this tests the manual completion code. Signed-off-by: Hans Verkuil <[email protected]>

Keep track of the number of requests and request objects of a media device. Helps to verify that all request-related memory is freed. Signed-off-by: Hans Verkuil <[email protected]>

The Raspberry Pi HEVC decoder uses a tiled format based on columns for 8 and 10 bit YUV images, so document them as NV12MT_COL128 and NV12MT_10_COL128. Signed-off-by: Dave Stevenson <[email protected]>

Add V4L2_PIXFMT_NV12MT_COL128 and V4L2_PIXFMT_NV12MT_10_COL128 to describe the Raspberry Pi HEVC decoder NV12 multiplanar formats. Signed-off-by: Dave Stevenson <[email protected]>

Adds a binding for the HEVC decoder found on the BCM2711 / Raspberry Pi 4, and BCM2712 / Raspberry Pi 5. Signed-off-by: Dave Stevenson <[email protected]>

The BCM2711 and BCM2712 SoCs used on Rapsberry Pi 4 and Raspberry Pi 5 boards include an HEVC decoder block. Add a driver for it. Signed-off-by: John Cox <[email protected]> Signed-off-by: Dave Stevenson <[email protected]>

Add the configuration information for the HEVC decoder. Signed-off-by: Dave Stevenson <[email protected]>

6by9 · 2025-03-05T16:40:35Z

Rebased.

AFAIK this is ready to go.
The FFmpeg changes should already be in Raspberry Pi OS.
(The latest changes for the newer image formats are not needed at present, nor is Mesa 24)

The only question is whether Libreelec folks are aware of the change, and have picked up the base FFmpeg changes before taking this kernel.

popcornmix · 2025-03-05T16:45:42Z

The only question is whether Libreelec folks are aware of the change, and have picked up the base FFmpeg changes before taking this kernel.

I've let them know (but they'd already been following this PR).

popcornmix · 2025-03-05T18:07:40Z

Ah - this breaks hevc decode with kodi21 in RPiOS, as it uses embedded ffmpeg (kodi21 wants a newer ffmpeg).
I'll need to rebuild the package with updated ffmpeg.

It looks like the older kodi package still works as that uses system ffmpeg.

I'm happy with this being merged for now (we can ensure kodi21 is updated before this kernel reaches apt).

popcornmix · 2025-03-05T19:54:37Z

While kodi 20 worked on a Pi5. I can't play hevc on a Pi4.
ffmpeg is updated. hevc works without this PR but stops with it.

2025-03-05 19:52:14.984 T:2002    error <general>: ffmpeg[0x0]: [hevc] mediabufs_poll_cb: Timeout
2025-03-05 19:52:14.984 T:2002     info <general>: ffmpeg[0x0]: Unable to reinit media request: Device or resource busy
2025-03-05 19:52:14.984 T:1996    error <general>: ffmpeg[0x0]: [hevc] Unable to set controls: Device or resource busy
2025-03-05 19:52:14.984 T:1996    error <general>: ffmpeg[0x0]: [hevc] send_slice: Failed to set req ctls
2025-03-05 19:52:14.984 T:1996     info <general>: ffmpeg[0x0]: Unable to reinit media request: Device or resource busy
2025-03-05 19:52:14.984 T:1996    error <general>: ffmpeg[0x0]: [hevc] hardware accelerator failed to decode picture
2025-03-05 19:52:14.984 T:1996    debug <general>: ffmpeg[0x0]: v4l2_req_frame_free: avctx=0x7ef400f5f0 data=0x7ee8081640
2025-03-05 19:52:14.999 T:2002     info <general>: ffmpeg[0x0]: Unable to reinit media request: Device or resource busy

@HiassofT also saw similar behaviour is libreelec.

Suprisingly Pi 5 is okay with the same hevc file (sony football one).

HiassofT · 2025-03-05T20:17:43Z

arch/arm/boot/dts/broadcom/bcm2711-rpi-ds.dtsi

-	};
+&hevc_dec {
+	reg = <0x0 0x7eb10000  0x0 0x1000>,  /* INTC */
+	      <0x0 0x7eb00000  0x0 0x10000>; /* HEVC */


INTC and HEVC are swapped compared to bcm2711.dtsi - this breaks HEVC on RPi4.

~~likely the hevc_dec override should just be dropped as reg is set in bcm2711.dtsi anyways~~ EDIT: scratch that

Thanks.
Yes it is needed as size-cells differs between upstream and downstream, but the swapping is wrong.

Signed-off-by: Dave Stevenson <[email protected]>

For downstream only, add back in the legacy single planar SAND formats. Signed-off-by: Dave Stevenson <[email protected]>

Upstream will take the multi-planar SAND format, but add back in the downstream single planar variant for backwards compatibility Signed-off-by: Dave Stevenson <[email protected]>

Signed-off-by: Dave Stevenson <[email protected]>

vc4's debugfs support was updated with drm_debugfs_entry whilst BCM2712 support was in progress, and missed that the lookup in vc6_hvs_debugfs_dlist still followed the old pattern. Correct that lookup to avoid an invalid dereference. Fixes: f7af8ae ("drm/vc4: hvs: Add support for BCM2712 HVS") Signed-off-by: Dave Stevenson <[email protected]>

The SAND handling had been using what was believed to be a runtime parameter in the modifier, however that has been clarified that all permitted variants of the modifier must be advertised, so making it variable wasn't practical. With a rationalisation of how the producers of this format are configured, we can switch to a variant that doesn't have as much variation, and can be configured such that only 2 options are required. Add a modifier with value 0 to denote that the height of the luma column matches the buffer height, and chroma column will be half that due to YUV420. A modifier of 1 denotes that the height of the luma column still matches the buffer height, but the chroma column height is the same. This can be used to replicate the previous behaviour. Signed-off-by: Dave Stevenson <[email protected]>

The SAND modifier with height 0 is now using the provided pitch as the column stride, but the UBM allocation needs to be done based on the plane width. Recompute the width in these conditions. Signed-off-by: Dave Stevenson <[email protected]>

Pitch has no meaning if the modifier isn't DRM_FORMAT_MOD_LINEAR as there is no guarantee that the value passed follows the pattern that pitch * height = size. Remove that check from framebuffer_check. Signed-off-by: Dave Stevenson <[email protected]>

To avoid user complaints that /dev/video0 isn't their USB webcam, add downstream patch that allows setting the preferred video device number. Signed-off-by: Dave Stevenson <[email protected]>

Supporting GL rendering of the new HEVC decoder pixel formats requires Mesa 24.2.5 or later. There are a couple of minor issues holding up switching to Mesa 24. Drop the new pixel formats from enum_fmt so that FFMpeg will use the older ones that earlier versions of Mesa do support. Signed-off-by: Dave Stevenson <[email protected]>

6by9 · 2025-03-06T12:16:30Z

Updated to correct bcm2711-rpi-ds.dtsi

pelwell · 2025-03-06T13:59:47Z

I'm happy for this to be merged if somebody can confirm that Pi 4 is working now.

popcornmix · 2025-03-06T14:37:38Z

Yes - hevc is working on Pi4 for me with this update.

kernel: Backport HEVC decoder as submitted to linux-media as v2 See: raspberrypi/linux#6666 kernel: Add the wifimac overlay See: raspberrypi/linux#6707 kernel: configs: Add GPIO_FXL6408=m See: raspberrypi/linux#6711 kernel:dtoverlays: Add adxl345 to i2c-sensor See: raspberrypi/linux#6712 kernel: overlays: Use current I2C Sysfs in README examples See: raspberrypi/linux#6713 kernel: dts: Remove the power key debounce on Pi 500

popcornmix force-pushed the rpi-6.12.y branch 2 times, most recently from fd2db7e to 031a382 Compare February 19, 2025 14:09

popcornmix force-pushed the rpi-6.12.y branch from 60e5cbd to d024995 Compare February 24, 2025 12:49

6by9 force-pushed the rpi-6.12.y-hevc-dec-v2 branch 2 times, most recently from a4a537b to 0b989c6 Compare February 27, 2025 18:08

6by9 force-pushed the rpi-6.12.y-hevc-dec-v2 branch from 0b989c6 to 931822d Compare February 27, 2025 19:44

popcornmix force-pushed the rpi-6.12.y branch from e984a1f to 0f48fde Compare February 28, 2025 17:05

6by9 and others added 11 commits March 5, 2025 16:24

Revert "staging: media: rpivid: Add Raspberry Pi V4L2 H265 decoder"

c09c474

This reverts commit d7c70b0.

Revert "media: videodev2.h: Add a format for column YUV4:2:0 modes"

f250d1c

This reverts commit 9629d77.

Revert "RFC: media: Add media_request_{pin,unpin} API"

31fd239

This reverts commit 64ff380.

media: vicodec: add support for manual completion

4f33272

Manually complete the requests: this tests the manual completion code. Signed-off-by: Hans Verkuil <[email protected]>

media: mc: add debugfs node to keep track of requests

d54098b

Keep track of the number of requests and request objects of a media device. Helps to verify that all request-related memory is freed. Signed-off-by: Hans Verkuil <[email protected]>

docs: uapi: media: Document Raspberry Pi NV12 column format

ad03124

The Raspberry Pi HEVC decoder uses a tiled format based on columns for 8 and 10 bit YUV images, so document them as NV12MT_COL128 and NV12MT_10_COL128. Signed-off-by: Dave Stevenson <[email protected]>

media: ioctl: Add pixel formats NV12MT_COL128 and NV12MT_10_COL128

4f48cf8

Add V4L2_PIXFMT_NV12MT_COL128 and V4L2_PIXFMT_NV12MT_10_COL128 to describe the Raspberry Pi HEVC decoder NV12 multiplanar formats. Signed-off-by: Dave Stevenson <[email protected]>

media: dt-bindings: media: Add binding for the Raspberry Pi HEVC decoder

1444c94

Adds a binding for the HEVC decoder found on the BCM2711 / Raspberry Pi 4, and BCM2712 / Raspberry Pi 5. Signed-off-by: Dave Stevenson <[email protected]>

media: platform: Add Raspberry Pi HEVC decoder driver

5a86d17

The BCM2711 and BCM2712 SoCs used on Rapsberry Pi 4 and Raspberry Pi 5 boards include an HEVC decoder block. Add a driver for it. Signed-off-by: John Cox <[email protected]> Signed-off-by: Dave Stevenson <[email protected]>

arm: dts: bcm2711-rpi: Add HEVC decoder node

6a075cf

Add the configuration information for the HEVC decoder. Signed-off-by: Dave Stevenson <[email protected]>

6by9 force-pushed the rpi-6.12.y-hevc-dec-v2 branch from 931822d to 892b669 Compare March 5, 2025 16:37

HiassofT reviewed Mar 5, 2025

View reviewed changes

6by9 added 3 commits March 6, 2025 12:14

arm: dt: broadcom: fixup downstream dts for HEVC decoder

fcb6850

dts: bcm2712: Fixup HEVC decoder nodes for new driver.

7637798

Signed-off-by: Dave Stevenson <[email protected]>

media: v4l2: Add single planar NV12 column formats

978534b

For downstream only, add back in the legacy single planar SAND formats. Signed-off-by: Dave Stevenson <[email protected]>

6by9 added 9 commits March 6, 2025 12:14

media: hevc_dec: Add in downstream single planar SAND variant

b1d340d

Upstream will take the multi-planar SAND format, but add back in the downstream single planar variant for backwards compatibility Signed-off-by: Dave Stevenson <[email protected]>

arm: defconfig: Replace VIDEO_RPIVID with VIDEO_RPI_HEVC_DEC

5145a5a

Signed-off-by: Dave Stevenson <[email protected]>

arm64: defconfig: Replace VIDEO_RPIVID with VIDEO_RPI_HEVC_DEC

f100557

Signed-off-by: Dave Stevenson <[email protected]>

media: hevc_dec: Add module parameter for video_nr

aea2a89

To avoid user complaints that /dev/video0 isn't their USB webcam, add downstream patch that allows setting the preferred video device number. Signed-off-by: Dave Stevenson <[email protected]>

6by9 force-pushed the rpi-6.12.y-hevc-dec-v2 branch from 892b669 to 69a3783 Compare March 6, 2025 12:15

pelwell merged commit bb4bab0 into raspberrypi:rpi-6.12.y Mar 6, 2025
11 of 12 checks passed

popcornmix mentioned this pull request Mar 12, 2025

Prep HEVC decoder for upstreaming #6519

Closed

6by9 mentioned this pull request Mar 13, 2025

Latest kernel trees: hevc: Failed to find a V4L2 device for H265 #6718

Closed

popcornmix mentioned this pull request Mar 18, 2025

4k Gets Broken After linux-rpi 6.12.18-1 update on PI4 #6723

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Backport HEVC decoder as submitted to linux-media as v2 #6666

Backport HEVC decoder as submitted to linux-media as v2 #6666

Uh oh!

6by9 commented Feb 12, 2025

Uh oh!

popcornmix commented Feb 12, 2025

Uh oh!

6by9 commented Feb 12, 2025

Uh oh!

6by9 commented Feb 27, 2025

Uh oh!

6by9 commented Mar 5, 2025

Uh oh!

popcornmix commented Mar 5, 2025

Uh oh!

popcornmix commented Mar 5, 2025 •

edited

Loading

Uh oh!

popcornmix commented Mar 5, 2025

Uh oh!

HiassofT Mar 5, 2025 •

edited

Loading

Uh oh!

6by9 Mar 6, 2025

Uh oh!

6by9 commented Mar 6, 2025

Uh oh!

pelwell commented Mar 6, 2025

Uh oh!

popcornmix commented Mar 6, 2025

Uh oh!

Uh oh!

Uh oh!

Backport HEVC decoder as submitted to linux-media as v2 #6666

Backport HEVC decoder as submitted to linux-media as v2 #6666

Uh oh!

Conversation

6by9 commented Feb 12, 2025

Uh oh!

popcornmix commented Feb 12, 2025

Uh oh!

6by9 commented Feb 12, 2025

Uh oh!

6by9 commented Feb 27, 2025

Uh oh!

6by9 commented Mar 5, 2025

Uh oh!

popcornmix commented Mar 5, 2025

Uh oh!

popcornmix commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

popcornmix commented Mar 5, 2025

Uh oh!

HiassofT Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

6by9 Mar 6, 2025

Choose a reason for hiding this comment

Uh oh!

6by9 commented Mar 6, 2025

Uh oh!

pelwell commented Mar 6, 2025

Uh oh!

popcornmix commented Mar 6, 2025

Uh oh!

Uh oh!

Uh oh!

popcornmix commented Mar 5, 2025 •

edited

Loading

HiassofT Mar 5, 2025 •

edited

Loading