`DataTexture`: Proposal to support partial update

[agargaro]

Description

Hi!

I am using DataTexture quite a lot to handle data with BatchedMesh and InstancedMesh2 (InstancedMesh + indirection).

In my case, I would like to update the color of only one instance (on the mouse over), but send the gpu the whole texture is expensive because it's very large.

I tried using the WebGLRenderer.copyTextureToTexture method but it doesn't work when src and dest are the same texture (I might open a separate bug for this).

Anyway, this method is useless if BatchedMesh automatically updates .needsUpdate flag which will make the whole texture update anyway.

It would be fantastic to have a partial update system like BufferAttribute.

I know that it's an important change, but if you want I can help.

Thank you for all the work you do. 😄

Solution

Implementing an addUpdateRange method similar to that of BufferAttribute.

.addUpdateRange ( region : Box2 ) : this

Alternatives

Fix and use WebGLRenderer.copyTextureToTexture, but we should remove .needsUpdate = true from BatchedMesh?

Additional context

No response

[RenaudRohlinger]

Nice to keep this idea on the table. So essentially, adding x, y, width, height parameters to [compressed]texSubImage(2D/3D).

By the way for your specific case, you could take this a step further by avoiding CPU-to-GPU stalls entirely through the use of Pixel Buffer Objects (PBOs). Adding a new API in Three.js that would be designed to enable asynchronous data transfers by leveraging gl.PIXEL_UNPACK_BUFFER. Instead of directly interacting with the GPU as we currently do, the CPU would write data to the PBO, which acts as a staging area, and the GPU processes the buffer asynchronously. This would eliminate blocking and probably improves by a lot your advanced BatchedMesh performances.

For example:

// Bind the PBO for writing (created on the first upload with `texImage2D` for example)
gl.bindBuffer(gl.PIXEL_UNPACK_BUFFER, pbo);

// Write pixel data to the PBO
const pixelData = new Uint8Array(bufferSize); // Your pixel data
gl.bufferSubData(gl.PIXEL_UNPACK_BUFFER, 0, pixelData);

// Upload the data from the PBO to the texture
gl.bindTexture(gl.TEXTURE_2D, texture);
gl.texSubImage2D(
    gl.TEXTURE_2D,
    0,               // Mipmap level
    0, 0,            // x and y offsets
    textureWidth,     // Width of the sub-image
    textureHeight,    // Height of the sub-image
    gl.RGBA,          // Format
    gl.UNSIGNED_BYTE, // Type
    0                 // Offset in the PBO
);

// Unbind the PBO
gl.bindBuffer(gl.PIXEL_UNPACK_BUFFER, null);

Basically:
No PBO:

CPU --> GPU (Direct transfer)
         [GPU busy → CPU stalls]

PBO:

CPU --> PBO (Asynchronous transfer)
PBO --> GPU (GPU reads later when ready)
         [CPU free → No stall]

Basically, the PBO serves as a staging area in GPU-accessible memory. Pixel data is first copied into the PBO, and the GPU reads from it at a later time. Spamming DataTexture update wouldn't stall the CPU anymore, adding to the command queue of the Pixel Buffer Object instead, and get processed asynchronously.

[gkjohnson]

I think this would be a great addition - even for any texture. Uploading the whole image for just a single pixel is a waist. A couple comments - I think the signature can be just a single box, right? As in .addUpdateRange ( box : Box2 )

I tried using the WebGLRenderer.copyTextureToTexture method but it doesn't work when src and dest are the same texture (I might open a separate bug for this).

I think the copyTextureToTexture mental model is different to what you're interested in. Due to copying operations the contents of a texture can become out-of-sync with the gpu-memory representation and copyTextureToTexture is designed to copy between gpu-memory instances. Adding addUpdateRange seems like a good, API-consistent addition.

I'm very supportive of this addition.

@RenaudRohlinger

Adding a new API in Three.js that would be designed to enable asynchronous data transfers by leveraging gl.PIXEL_UNPACK_BUFFER. Instead of directly interacting with the GPU as we currently do, the CPU would write data to the PBO, which acts as a staging area, and the GPU processes the buffer asynchronously.

This is interesting. Is there a reason to not always do this when uploading DataTextures? It seems like a strict improvement without drawbacks? Does it need a new three.js API?

[RenaudRohlinger]

This implies an additional array buffer (or 2, one write and another one read to prevent block in lockstep) per DataTexture at initialization:

// Create a PBO
const pbo = gl.createBuffer();

// Bind it to PIXEL_UNPACK_BUFFER
gl.bindBuffer(gl.PIXEL_UNPACK_BUFFER, pbo);

// Allocate storage (size in bytes)
const bufferSize = textureWidth * textureHeight * 4; // Assuming RGBA, 4 bytes per pixel
gl.bufferData(gl.PIXEL_UNPACK_BUFFER, bufferSize, gl.STREAM_DRAW);

// Unbind the PBO
gl.bindBuffer(gl.PIXEL_UNPACK_BUFFER, null);

So I guess we would need to consider the performance benefit/cost before thinking about adding this to the core. But I agree that it sounds like a direct improvement.

[gkjohnson]

It's true that this would cause unnecessary resource creation. I think PBO support for faster uploads can be considered in a separate issue - perhaps a flag can be added to enabled / disable this use.

Before any work is done on this I think it would be good to get opinions from @Mugen87 or @mrdoob. But I think this is a good change.

[agargaro]

I think the signature can be just a single box, right? As in .addUpdateRange ( box : Box2 )

Yes, sorry. I edited the post.

@RenaudRohlinger thanks for the information!

[Nmzik]

Just to let you know, texSubImage2Dwith the D3D11 backend on Windows can trigger CPU readback, meaning it copies buffer memory from the GPU back to the CPU before copying it to the texture again (if supportsFastCopyBufferToTexture returns false). I remember trying to partially update a large (4096x4096) texture via PBO but didn't see any performance improvements—there were still framerate hitches. Again, I could be completely wrong, but that's what I see in the source code of the ANGLE project. Also, for future reference, compressedTexSubImage2D will ALWAYS trigger CPU readback from PBO Link

Remark: I remember that WebGL calls don't directly invoke the OS graphics API. Instead, they are added to a ring buffer that is processed by a separate thread in the browser's renderer thread. So, the texSubImage2D command may return immediately on the JavaScript side but still perform a GPU-CPU copy in a separate thread. Then again, I might be overthinking it...

bool Renderer11::supportsFastCopyBufferToTexture(GLenum internalFormat) const
{
    const gl::InternalFormat &internalFormatInfo = gl::GetSizedInternalFormatInfo(internalFormat);
    const d3d11::Format &d3d11FormatInfo =
        d3d11::Format::Get(internalFormat, mRenderer11DeviceCaps);

    // sRGB formats do not work with D3D11 buffer SRVs
    if (internalFormatInfo.colorEncoding == GL_SRGB)
    {
        return false;
    }

    // We cannot support direct copies to non-color-renderable formats
    if (d3d11FormatInfo.rtvFormat == DXGI_FORMAT_UNKNOWN)
    {
        return false;
    }

    // We skip all 3-channel formats since sometimes format support is missing
    if (internalFormatInfo.componentCount == 3)
    {
        return false;
    }

    // We don't support formats which we can't represent without conversion
    if (d3d11FormatInfo.format().glInternalFormat != internalFormat)
    {
        return false;
    }

    // Buffer SRV creation for this format was not working on Windows 10.
    if (d3d11FormatInfo.texFormat == DXGI_FORMAT_B5G5R5A1_UNORM)
    {
        return false;
    }

    // This format is not supported as a buffer SRV.
    if (d3d11FormatInfo.texFormat == DXGI_FORMAT_A8_UNORM)
    {
        return false;
    }

    return true;
}

[Spiri0]

@agargaro I can't assess to what extent that's an option for you. With WebGPU, you can use SBOs instead of DataTextures, and they don't have the limitation you mentioned. I use them extensively. If it has to be WebGL, I can't help, but if you want to use SBOs with three.webgpu.js, I can. I recently converted my ocean repo to SBOs. They make the code significantly leaner and much more efficient. With SBOs, you can read and write without any problems.
To what extent it makes sense to invest more effort in enabling WebGL to do something that is significantly easier to achieve with the new WebGPU technology is more a matter of personal preference.

[agargaro]

Hi @Spiri0, thank you 😄

Currently I am still using WebGLRenderer but soon I will start to study WebGPURenderer as well, in order to migrate my InstancedMesh2 library.

But for those who are still using WebGLRenderer, I think the partial texture update can be very important, even if it is possible to implement it manually as I did here (I'm not an expert but maybe it can help someone).

If someone would like to provide some technical details, I could try to do a PR for only the WebGLRenderer if it makes sense.

Can I write to you privately, (maybe on the forum?) to ask you some information about what you were suggesting?

[Ctrlmonster]

Hey, I ran into a similar need. I've been working on an LOD System using BatchedMesh, where each BatchedMesh represents one LOD for many geometries and then a specific instanceId refers to the same geometry across LODs, e.g.

// enable lod0 on mesh 42 (assuming 3 lod levels)
batch0.setVisibleAt(42, true);
batch1.setVisibleAt(42, false); 
batch2.setVisibleAt(42, false);

But it turns out, that this actually hurts performance, compared to having no LOD at all, as the constant batchedMesh.setVisibleAt() calls and the following uploads of the batches _indirectTexture each frame cost a lot more time than the reduced triangle count saves. So if partial updates were possible that might be help usecases like these :)

[Spiri0]

Use a large dst texture for the shader and a small src texture that only contains the data that needs to be updated in the dst. Then use copyTextureToTexture to transfer the data from the src to the dst. This works wonderfully and is very efficient for exactly what you have in mind. I speak from experience. This will update only the necessary parts of the dst texture, exactly as you want.

My LOD systems run at 120 fps. However, I don't use batched meshes for them because I don't consider them suitable for the intended purpose.

@RenaudRohlinger Do you remember that? You created this example at my request. It does exactly what this topic is about, and it works great.

"replace parts of an existing texture with all data of another texture"

https://threejs.org/examples/?q=part#webgpu_textures_partialupdate

This works the same with WebGL

[Ctrlmonster]

Hey @Spiri0, thanks for sharing your approach! Will definitely keep it in mind. The reason I'm hoping to use BatchedMesh for this, is that it solves the draw call problem much more effectively than InstancedMesh. With InstancedMesh the draw call count can still get into the hundreds (one per geometry) and since BatchedMesh already handles visibility itself, the only way around would be to fork/copy-paste the source.

[Spiri0]

If your approach with batched meshes works well for you, then it's a good solution. To each his own.

The technique of loading data into a chunk dataTexture, updating it to send the data to the GPU, and then copy it to the target texture using copyTextureToTexture is exactly what you're looking for. I'm glad if it helps.

@RenaudRohlinger @Mugen87 The example of renauldRohlinger is essentially the answer to this topic. This works equally well in WebGL and WebGPU
Here again the link:
https://threejs.org/examples/?q=part#webgpu_textures_partialupdate