Native media Equirect layers creation #31033
Conversation
…t loss issue doing it outside the manager. mrdoob#31029
📦 Bundle sizeFull ESM build, minified and gzipped.
🌳 Bundle size after tree-shakingMinimal build including a renderer, camera, empty scene, and dependencies.
|
|
It took a bit to design how to go about it. The only issue is you have to toggle the video mesh layers when going into XR and ending the session. Maybe there is other ways to do it. https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L130 live example https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html |
|
I have tried to improve the JSDoc a bit. For a functional review I hope @cabanier can have a look. BTW: Please create a E2E screenshot for the new example |
|
The requirement to toggle off the video texture with the meshes. I'm not sure how to make better. It causes a crash if the texture is also rendered. |
Move equirect layer creation into common layer creation method Fixup example with new media layer usage
|
I've refactored it quite a bit. And followed the structure of modifying the geometry externally to keep that external. I could add that in. It returns a group of meshes which works in 2D, Rift, Quest. There is still the problem of needing to turn off meshes for native layers. Can be done internally or keep it external. I'm not across everything to figure out a better way to render from 2D to native. https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L114 https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html mono example https://electroteque.org/dev/threejs/examples/webgpu_xr_media_mono_layer.html |
|
Is this looking good setting the geometry externally. I have it working implementing and tested. Unfortunately media layer needs to be added in the manager as doing it external caused a context loss fault. It may be a problem adding other layers externally as the media layers list needs to be prepended in the list as backgrounds as I have done. The only issue still I can't figure out when going from the 2D mesh to native layer in XR. It will be rendered including the video texture in the projection layer as its added to the scene so those layers need to be turned off. Or maybe simply removing those mesh groups from the scene ?. I can look at doing it in the manager perhaps. Turn off mesh layers for all the media layers in the list on session start. Unless there is a better way to not render those meshes in the scene into the projection layer. |
Do you have an example for me to look at? You shouldn't be getting context losses.
Yes, can't you do the same as what I did for quad/cylinder layer? |
|
If the external geometry setter is ok. Or I make them default SphereGeometry requiring to update it's settings. I just set them to PlaneGeometry. https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L105 I add the mesh group to the layers params. https://github.com/danrossi/three.js/blob/equirect-layers/src/renderers/common/XRManager.js#L667 So this requirement to turn off texture rendering in the projection layer can be done internally perhaps and make it transparent. https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L114 There is an issue I can't figure out if it's the browser or three. Native media layer rendering is blank if it's an immersive-vr session. Only works in immersive-ar session. The controllers don't connect all the time either and can't figure out where to report. There is still the makeXRCompatible error in the console I can't find where to report. |
|
It might need a blank render target like the other layer examples. When using immersive-vr mode instead of immersive-ar mode like the XRButton uses. The projection layer render target blocks out the background native media layer. So it seems it's a blank render. https://github.com/mrdoob/three.js/blob/dev/src/renderers/common/XRManager.js#L1480 https://electroteque.org/dev/threejs/examples/webgpu_vr_media_layer.html Doing something like this in the renderLayers method seems to work but not sure if its worth it and just keep it an immersive-ar session. If it peforms the same. Is just to be able to not have the projection layer render target cover it up. I noticed all the demos show up weird because they are AR sessions though. So stuff around the camera. for (const layer2 of this._mediaLayers) {
layer2.renderTarget.isXRRenderTarget = this._session !== null;
layer2.renderTarget.hasExternalTextures =
layer2.renderTarget.isXRRenderTarget;
layer2.renderTarget.autoAllocateDepthBuffer = !
layer2.renderTarget.isXRRenderTarget;
if ( layer2.renderTarget.isXRRenderTarget && this._supportsLayers ) {
const glSubImage = this._glBinding.getSubImage( layer2.xrlayer, this._xrFrame );
this._renderer.backend.setXRRenderTargetTextures(
layer2.renderTarget,
glSubImage.colorTexture,
glSubImage.depthStencilTexture );
}
} |
Tidy up media layer exanmple and add XR Controllers displayed on the projection layer
|
I was able to figure out fixing the conflict in the editor but have been testing on a merged branch. I'm still unable to figure out how to work around three projection layer render target hiding the native layer in immersive-vr mode. Only works in immersive-ar mode. Which is what all the XR examples use. Hopefully disabling the 2D mesh layer internally to not render in the projection layer is the right solution. |
Add uv factor mappings for both stereo-left-right and stereo-top-bottom layouts for each eye index. For the geometry uv transformations. Update media layer example with some extra comments on usage.
…reo-top-bottom' layout. update code docs to suit.
…w the camera which displays wrong. Fixup media layer feature to render blend spheres while in XR to poke through the projection layer to properly show the native media layers while in immersive-vr mode.
…ent layout or 180 content while in a xr session. Prevent prepenmding new layers and replace layers at specific indexes.
Related issue: #31029
Description
Adding native media layer creation for WebGPU XR. This fixes a context loss issue doing it outside the manager. There is a potential Oculus browser bug complaining makeXRCompatible isn't set when setting up a media layer still.
Media layers are prepended into the background before the other layers.
Example https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html