Incorrect compositing for WebP with alpha

[Shnatsel]

This happens in image from git on commit 5976c195939bfbede976fe1e0a80225d192a793c with image-webp v0.2.0

Expected

convert bug.webp[0] bug.png produces an image that's fully transparent, other than the beetle:

Actual behaviour

With image extracting the first frame results in a mostly white image, with a bit of transparency around the beetle:

Reproduction steps

Image triggering the issue: bug.webp.gz

Code to reproduce it:

use std::error::Error;

fn main() -> Result<(), Box<dyn Error>> {
    use image::io::Reader as ImageReader;
    let input = std::env::args().nth(1).unwrap();
    let output = std::env::args().nth(2).unwrap();
    
    let img = ImageReader::open(input)?.with_guessed_format()?.decode()?;
    img.save(output)?;
    Ok(())
}

[Shnatsel]

Actually, here's a non-animated image with the same issue:
file_10413997_128x128.webp.gz

[fintelia]

Could you check whether set_background_color has any impact on this?

[Shnatsel]

Yes, set_background_color with [0,0,0,0] does fix it.

That is probably slightly faster as well, because we don't have to memset() ourselves and can simply use vec![] to request pre-zeroed memory from the OS.

Code used for testing, for completeness

use std::error::Error;
use image::{codecs::webp::WebPDecoder, DynamicImage, ImageFormat, ImageReader, Rgba};

fn main() -> Result<(), Box<dyn Error>> {
    let input = std::env::args().nth(1).unwrap();
    let output = std::env::args().nth(2).unwrap();
    
    let reader = ImageReader::open(input)?.into_inner();
    let mut decoder = WebPDecoder::new(reader)?;
    decoder.set_background_color(Rgba([0,0,0,0]))?;
    let image = DynamicImage::from_decoder(decoder)?;
    image.save(output)?;
    Ok(())
}

[fintelia]

My guess is that the image contains a background color set to opaque white, but ImageMagick is ignoring it and compositing onto a fully transparent canvas instead.

This is what the WebP spec says about the image's background color:

Background Color: 32 bits (uint32)
The default background color of the canvas in [Blue, Green, Red, Alpha] byte order. This color MAY be used to fill the unused space on the canvas around the frames, as well as the transparent pixels of the first frame. The background color is also used when the Disposal method is 1.

Notes:

• The background color MAY contain a non-opaque alpha value, even if the Alpha flag in the 'VP8X' Chunk is unset.
• Viewer applications SHOULD treat the background color value as a hint and are not required to use it.
• The canvas is cleared at the start of each loop. The background color MAY be used to achieve this.

[Shnatsel]

Firefox and Chrome displays the images with transparent background as well. And the images are clearly meant to be displayed like that, too.

I think we'll have to change the way background color is treated for compatibility with real-world images targeting libwebp, regardless of what the spec says.

[fintelia]

We should certainly match what other software does. The spec is clearly written to allow transparent background behavior from Firefox and Chrome, it is just frustrating that the spec's suggested approach doesn't align with that.

I propose adding a WebPDecoder::background_color_hint method that returns the background color from the image, but have the default actually match Chrome and Firefox unless overridden by WebPDecoder::set_background_color. This way users can still get the old behavior, but only if they specifically want it.

The only question left to figure out is whether browsers always use a transparent background or whether it varies based on the settings in the image. For instance, if the image doesn't have the Alpha bit set in the header, is a different background color used?

[Shnatsel]

My guess is that the image contains a background color set to opaque white, but ImageMagick is ignoring it and compositing onto a fully transparent canvas instead.

To be clear, I haven't actually inspected whether the background color is set in the file or not, because I do not know how to do that. We should verify that this is indeed the case before we implement a fix based on this assumption.

The default background color of the canvas in [Blue, Green, Red, Alpha] byte order. This color MAY be used to fill the unused space on the canvas around the frames, as well as the transparent pixels of the first frame.

(emphasis mine)

Right now compositing doesn't apply the background color to the pixels of the first frame, resulting in a transparent "hole" around the beetle. That is also a bug.

[fintelia]

You can use the webpinfo utility to dump the info:

$ webpinfo file_10413997_128x128.webp
File: file_10413997_128x128.webp
RIFF HEADER:
  File size:  90398
Chunk VP8X at offset     12, length     18
  ICCP: 0
  Alpha: 1
  EXIF: 0
  XMP: 0
  Animation: 1
  Canvas size 128 x 128
Chunk ANIM at offset     30, length     14
  Background color:(ARGB) ff ff ff ff
  Loop count      : 0
Chunk ANMF at offset     44, length   2782
  Offset_X: 14
  Offset_Y: 12
  Width: 101
  Height: 103
  Duration: 100
  Dispose: 0
  Blend: 1
Chunk ALPH at offset     68, length    858
Chunk VP8  at offset    926, length   1900
  Width: 101
  Height: 103
  Alpha: 0
  Animation: 0
  Format: Lossy (1)
Chunk ANMF at offset   2826, length   2864
  Offset_X: 18
  Offset_Y: 16
  Width: 110
  Height: 112
  Duration: 100
  Dispose: 0
  Blend: 1
Chunk ALPH at offset   2850, length    604
Chunk VP8  at offset   3454, length   2236
  Width: 110
  Height: 112
  Alpha: 0
  Animation: 0
  Format: Lossy (1)
...

[fintelia]

The default background color of the canvas in [Blue, Green, Red, Alpha] byte order. This color MAY be used to fill the unused space on the canvas around the frames, as well as the transparent pixels of the first frame.

Right now compositing doesn't apply the background color to the pixels of the first frame, resulting in a transparent "hole" around the beetle. That is also a bug.

I'm not actually sure this is a bug. Both images have Blend: 1 for the first frame. The spec doesn't specifically say how things are handled for the first frame, but this is the description of that bit:

Blending method (B): 1 bit

Indicates how transparent pixels of the current frame are to be blended with corresponding pixels of the previous canvas:

• 0: Use alpha-blending. After disposing of the previous frame, render the current frame on the canvas using alpha-blending (see below). If the current frame does not have an alpha channel, assume the alpha value is 255, effectively replacing the rectangle.

• 1: Do not blend. After disposing of the previous frame, render the current frame on the canvas by overwriting the rectangle covered by the current frame.

[Shnatsel]

I see. That makes sense.

I guess WebP background color is just a mess and we'll have to join other decoders in ignoring it 😅