[DRAFT] Implement motion vectors in compatibility renderer

[devloglogan]

This PR implements motion vectors in the compatibility renderer, primarily for use with the XR_FB_space_warp OpenXR extension. For testing purposes, this PR currently implements this extension as well. Eventually this logic will be handled via GDExtension in godot_openxr_vendors. This PR will remain as a draft until that's the case.

Edit: Space warp functionality has been relocated to GodotVR/godot_openxr_vendors#222.

Technical details

I've tried to emulate how motion vectors are implemented in the Forward+ renderer where possible, but it's worth noting that space warp requires 3D motion vectors. Whereas the Forward+ renderer is rendering 2D motion vectors for TAA.

Motion vectors are rendered in a new, separate render pass. Why this was done:

• Since motion vectors can be rendered at a much smaller resolution this extra pass should be quite fast (e.g. the Quest 3 recommends a resolution of around 420x420).
• This will make it easier to allow developers to include or exclude specific objects from motion vectors, which can be helpful in minimizing the artifacts from space warp. (Note: this isn’t part of the current PR, but something planned for the future.)
• Combining motion vectors with the color pass would hinder future compatibility between Application Space Warp and the XR_META_recommended_layer_resolution OpenXR extension, since all attachments would then have to be of the same resolution.

The vertex shader code in scene.glsl has been placed in a new vertex_shader() function. When motion vectors are used, it will run the vertex shader on both previous and current vertex data. The scene state data and multiview data have been duplicated to hold the previous frame’s data. The vertex shader outputs both the previous and current clip positions, which are used to calculate motion vectors in the fragment shader.

For skeletons, GLES::MeshInstance::Surface now stores two vertex buffers, generating the second one on demand when motion vectors are enabled. The VBOs will alternate between current/previous, and the previous VBO is used to populate new scene.glsl vertex attribs at locations 16/17.

Similarly, for multimeshes/particles, GLES::MultiMesh now also stores two vertex buffers with the secondary one generated on demand. New scene.glsl vertex attribs (location 18 to 21) store the previous data, and the setup for both the current/previous vertex attribs is now done in a new GLES3::MeshStorage::multimesh_vertex_attrib_setup() function.

Issues that need to be addressed

There are a couple things that need to be ironed out that I would really appreciate some input on from those who have a bit more rendering know how.

• This implementation of motion vectors requires the GPU to have a max vertex attribute value of at least 22. This requirement is met on Quest hardware (or any other headset which uses the Qualcomm XR-series chips), and we check it before running the motion vector pass. But when opening the editor on desktop some errors might be thrown from ShaderGLES3::_initialize_version(), which attempts to compile all shader variants at startup, including the motion vector variant of scene.glsl. What is the best way to prevent it from compiling this variant if there aren’t enough vertex attributes?

• In non-one-shot particle systems, it's possible for a particle instance to go from the end of its lifetime one frame to the beginning of its lifetime the next frame. At the moment this causes an undesired result on the motion vector texture (we should draw nothing for the particle on these frames). Is there a way we can determine if this is the case in scene.glsl?

• As mentioned above the vertex shader in scene.glsl has moved into a new vertex_shader() function. For multiview data, I'm currently passing in a boolean value indicating whether to use multiview_data_block.prev_data or multiview_data_block.data. I'd rather just pass in the MultiviewData structs themselves rather than the boolean, similar to how the SceneData structs are passed in. However, doing this seems to cause anything using multiview data to just not be rendered. What I have in the PR works, so maybe it's not a big deal, but if anybody has ideas on how to fix this I'd appreciate it!

UPDATED (2024-10-28): I believe all of the areas of concern have been addressed. While technically the second point regarding particle systems still exists, I don't think this is something that needs to be addressed in this PR. Beyond just particles, there should be a way to exclude objects from being drawn to motion vectors (Ex: when an object is teleporting rather than moving smoothly). This is something the current Vulkan motion vectors implementation could benefit from. I think a fix for this warrants its own PR.

Contributed with ❤️ by W4Games

[clayjohn]

This implementation of motion vectors requires the GPU to have a max vertex attribute value of at least 22. This requirement is met on Quest hardware (or any other headset which uses the Qualcomm XR-series chips), and we check it before running the motion vector pass. But when opening the editor on desktop some errors might be thrown from ShaderGLES3::_initialize_version(), which attempts to compile all shader variants at startup, including the motion vector variant of scene.glsl. What is the best way to prevent it from compiling this variant if there aren’t enough vertex attributes?

Yes, motion vectors should be a specialization for now. Then the version will only be compiled when it is actually used.

In non-one-shot particle systems, it's possible for a particle instance to go from the end of its lifetime one frame to the beginning of its lifetime the next frame. At the moment this causes an undesired result on the motion vector texture (we should draw nothing for the particle on these frames). Is there a way we can determine if this is the case in scene.glsl?

Not from scene.glsl, no. We will have to do something in the actual particle system. I think that we handled this case already in the RD renderers. Most likely we will need to duplicate the particle buffer to effectively clear the history

As mentioned above the vertex shader in scene.glsl has moved into a new vertex_shader() function. For multiview data, I'm currently passing in a boolean value indicating whether to use multiview_data_block.prev_data or multiview_data_block.data. I'd rather just pass in the MultiviewData structs themselves rather than the boolean, similar to how the SceneData structs are passed in. However, doing this seems to cause anything using multiview data to just not be rendered. What I have in the PR works, so maybe it's not a big deal, but if anybody has ideas on how to fix this I'd appreciate it!

I can't think of any reason why one struct would work but not another. We should try to get it working though as the bool will likely create a dynamic branch which we really don't want

[dsnopek]

Thanks! This is looking really great to me :-)

I'm going to test the latest version soon, but in the meantime, I have some notes on the public APIs this adds. Some of this will need discussion with the rest of the XR team.

[dsnopek]

Doing some testing with AppSW via GodotVR/godot_openxr_vendors#222, I'm getting this message spammed to logs:

I don't see why we should be seeing this unhandled type, I think this is meant to be something that the application logs via OpenGL? And I don't think Godot is doing it. Maybe it's something from the OpenXR runtime?

But just this getting sent to the log so often is going to have a negative affect on performance. One option could be updating _gl_debug_print() to handle this type? Or, I guess figure out what is doing this logging, and somehow stop it from doing that?

These appear to be the types we're not handling:

#define _EXT_DEBUG_TYPE_MARKER_ARB              0x8268
#define _EXT_DEBUG_TYPE_PUSH_GROUP_ARB          0x8269
#define _EXT_DEBUG_TYPE_POP_GROUP_ARB           0x826A

[devloglogan]

@dsnopek it's of type "Marker" and severity "Notification", I noticed this recently and added the code to make it actually share the log and this is what it shared:

ERROR: GL ERROR: Source: Third Party Type: Marker    ID: 1   Severity: Notification  Message: vr-marker,frame_end,type,application,frame_num,175
at: _gl_debug_print (drivers\gles3\rasterizer_gles3.cpp:188)

Edit: It logged this as well

ERROR: GL ERROR: Source: Third Party Type: Marker    ID: 1   Severity: Notification  Message: VR Layer 0, SubImage Rect L0.0/0.0/1680.0/1760.0 R0.0/0.0/1680.0/1760.0, ColorScale 1.0/1.0/1.0/1.0
at: _gl_debug_print (drivers\gles3\rasterizer_gles3.cpp:188)

I can add the code to this PR so that its actually logged like above, but yes I'm thinking I'll have to figure out whats causing this and try to stop it. Since it's Marker/Notification I'm assuming what causes this is not actually a problem, but the log itself would be an issue.

[BastiaanOlij]

@devloglogan @dsnopek I've seen those errors for some time now, even before we added support for OpenXR debug logging (which is turned off by default anyway).
We don't seem to have a way to control the level of logging coming from the OpenGL message callback

[devloglogan]

I'm opening this PR since the XR_FB_space_warp code has been removed and it's in a now arguably merge-able state. :)

[dsnopek]

@dsnopek I would just add a parameter to the existing function for the velocity depth texture.

Ah, ok, that makes sense! Then we can just have a new method for setting the motion vector target size.

[devloglogan]

@dsnopek @BastiaanOlij I've added _EXT_DEBUG_TYPE_MARKER_ARB to the ignored opengl types to suppress that error log.

@clayjohn I've made changes to address your suggestions, thanks for the review!

[dsnopek]

I have no more notes. :-) This is looking great to me!

[devloglogan]

@AThousandShips thanks so much for the review! I've applied all your suggestions :)

[dsnopek]

This'll need to be rebased now that PR #98831 was merged (sorry :-))

[devloglogan]

@dsnopek No worries, rebased. :)

[devloglogan]

I'm marking this PR with DRAFT again as I'm breaking it up into two separate PRs. The renderer-agnostic part of this PR can now be found in #100282, this PR will be based on top of that and soon contain just the stuff relevant to the compatibility renderer.