Support AVIF / AV1 images

[StefanOltmann]

The AVIF closely resembles the HEIF format, with the primary distinction lying in the compression algorithm applied to the pixel data. For extracting container metadata, one can employ the same logic, as AVIF utilizes the ISO BMFF boxes outlined in the HEIC standard.

Resolves #608.

Tested and verified using this sample file: photo_74.avif.zip

[Nadahar]

This might be nitpicking, but as I understand it, AVIF is HEIF. The same goes for AVCI and potentially other compressions in the future: https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format#Variants

The "problem" is that HEIC is so closely associated with HEIF that it muddies the water, but it really isn't "more HEIF" than any of the others. I haven't studied the code closely enough to be sure what would be "most correct", but it might be a good idea to make sure that the project reflects the realities and don't add to this confusion.

As such the existing FileType defined for HEIF might really be the culprit:

metadata-extractor/Source/com/drew/imaging/FileType.java

Line 49 in 67dfddd

Heif("HEIF", "High Efficiency Image File Format", "image/heif", "heif", "heic"),

I'd argue that there is no real-world "HEIF file" since it would be pointless to make a HEIF file without a concrete compression/binary storage format implementation. The FileType definition should probably be for HEIC, and the MIME/Internet type should probably be image/heic.

I don't know if this PR is the right place for this, or if it's important at all - but it seems like a relevant place to mention it at least. AVCI could be added as well, as the parsing is the same.

@drewnoakes Is this anything to care about at all, or am I just making noise?

[StefanOltmann]

@Nadahar Yes, I agree with you. This HEIF-HEIC-thing confused me a lot at first, but in the end I decided to go with heic in my library as it seems to be more correct.

https://github.com/Ashampoo/kim/blob/01fbb7825d3ccf53170c4625aad63b713614919e/src/commonMain/kotlin/com/ashampoo/kim/model/ImageFormat.kt#L35

Also I don’t have a HeicReader, I have a BaseMediaFileFormatReader as this is what it is in the end.

Do you happen to have a AVCI test file? I will implement it in my library first to see if it works and could add it to this PR after. 🙂

[Nadahar]

@StefanOltmann Sorry, I don't have any AVCI test file. Images seems to be available from several stock photo sites, but not for free (use).

It's strange that it's hard to find, the image is simply encoded using AVC/h264 with some restrictions, which should be easily enough to do with x264. Since the container is HEIF, I would think that everything necessary to encode them should be available in for example FFmpeg. But, there seems to be so little interest in this format that nobody has bothered to put the pieces together so that you can actually encode using this combination.

FFmpeg/x264 can encode "AVCI video", which is something completely different as far as I can tell (AVC/h264 Intra) and this makes finding it even more difficult - since these video formats is collectively also referred to as "AVCI". Here is a snippet from FFmpeg's x264 documentation:

avcintra-class (class)

• Configure the encoder to generate AVC-Intra. Valid values are 50, 100 and 200

Only the video profiles are mentioned.

I found this in the source of another project, which isn't "encouraging" either: https://github.com/twitter/ios-twitter-image-pipeline/blob/fca0596d90fbc70baecfc600643a1d9f5deb4d74/TwitterImagePipeline/TIPImageTypes.h#L90-L93

That said, I'm not sure if a test image is strictly necessary. One could get one of the images available from some of the stock image sites and test on that without making the image public just to make sure, but these images should, by definition, be identical in the "metadata sections" to HEIC and AVIF.

edit: Regarding the lack of ability for FFmpeg to encode, "the mystery" seems to be solved. I assumed that they had already implemented HEIF encoding, considering that FFmpeg can encode both AV1 and HEVC used in AVIF and HEIC, but this isn't the case it seems. That certainly explains why there's no AVCI support: https://trac.ffmpeg.org/ticket/6521

[StefanOltmann]

One could get one of the images available from some of the stock image sites and test on that without making the image public

Sounds like a good idea, but regrettably, I couldn't locate one. At least not one where I could download the image without incurring a cost.

It appears that AVCI lacks any practical relevance.

Personally I wait until JPEG XL becomes a thing to move on from JPG. ;)

[drewnoakes]

@Nadahar I'm happy with the renaming proposals you've made. I feel less confused about the distinction between HEIC/HEIF after reading your comments. Did you want to file a PR? Otherwise I'll get to it as time permits. I'm on the move right now.

[drewnoakes]

Looks great, thank you @StefanOltmann! I'll run some tests and port this to .NET too in the next day or so, possibly with some API renamed in a follow up PR which I'll ping you and @Nadahar on to validate thinking.

[Nadahar]

@drewnoakes I didn't really explain it - I kind of assumed that "everybody knew" - which of course isn't true or there wouldn't be a confusion in the first place.

HEIF is the file format, it's a "generic container" based on ISOBMFF (basically the same ISO base container as MP4 etc). It does however not specify the encoding of the "payload", it's just a container. HEIC is a HEIF implementation where HEVC/h.265 is used to encode the image data. AVCI is a HEIF implementation where AVC/h.264 is used to encode the image data. AVIF is a HEIF implementation where AV1 is used to encode the image data. From the point of view of metadata extractor, these are thus all the same - and one reader should be able to handle them all. The HEIF container supports other kind of payload as well, and we could see other image formats using the HEIF container in the future, although I'm not saying that's very likely.

The confusion is probably that HEIF and HEIC are so similar, and maybe that both were kind of "introduced to the world" at the same time when Apple started using HEIC images for their phones. Whatever the reason, there seems to be a lot of mix-up between the two, and it seems to me that many use the two interchangeably without really knowing what is what.

All I wanted to do was to mention that this "confusion" maybe shouldn't be a part of metadata extractor's code. It's not like HEIC=HEIF and that AVIF and AVCI also happen to use the same metadata format. They have, by definition, the same metadata format.

So, this isn't important in any practical way, I just thought that when "expanding" the support for HEIF to other implementations than HEIC, it would be nice to keep the terms separated - and to let the logic reflect that (in how things are named etc).

[StefanOltmann]

@Nadahar Thanks, very well explained.

Indeed some news articles I read as it was first introduced also mixed that up. Maybe the writer did not know that, too. 🤷‍♂️

Apple calls it correctly (or just very specific) HEIC, but Samsung phones label it as „HEIF image“ in the settings. If it’s a HEIC that’s of course technically correct, but also adds to the confusion for end users.

What surely doesn’t help at all is files with .heif extension, which also exist and make people believe that it’s something different from .heic. This confusion does not exist with AVI & MP4 where people know that different codecs could be used. If all variants would just have .heif as extension regardless of the used codec this would be better. Now this is quite a mess.

I read some article that displays AVIF as a HEIC ripoff, but I feel now that this may have been biased when in fact AVIF is actually just a HEIF flavor. Thanks for nitpicking. 😉

[Nadahar]

@StefanOltmann It's not easy to keep things logical as long as marketing departments get their hands in the mix (as can be seen by the wild "inflation" in software version numbers of later years for example). There's actually quite a mess with for example MP4 as well, these files can be found with a variety of extensions like M4A, M4V, 3GP, 3G2 (and I'm sure there are others that I don't remember at the moment). In addition you have MOV and QT that is almost identical. M4A is a MP4 file where there is only one or more audio tracks, M4V is an MP4 file with a least one video track and as far as I can tell exactly the same as most MP4 files. 3GP and 3G2 are just subsets, more limited versions of MP4 meant for (earlier, 3G) mobile phones. You can rename them all to MP4 and play them.

The marketing people love to present something as "new", it doesn't matter if it's just repackaged in new glossy paper. I suspect that's the driver for most of this mess. Regardless, that's how the world works, unfortunately.

Regarding Samsung's use of HEIF, it's not necessarily wrong. If they have implemented support for decoding HEIF with all 3 "currently used" codecs, I think calling it HEIF is "fair". If they just support HEIC it's wildly misleading though. I don't know how wide AV1 support is in mobile devices these days, but decoding of AVC and HEVC is already there for video playback, so it should be easy for them to support those two at least.

[StefanOltmann]

@Nadahar Interesting. 🤔 Now that you mention it I recall once seeing a file with .m4v extension and wondering what it was. But I feel the mess around MP4 is not as widespread as for HEIF, because none of my cameras I ever had was producing anything other than files with .mp4 or .mov extensions.

You are definitely right about the marketing people. 😄

[drewnoakes]

Thanks again for this PR and for all the great discussion here. I'll spend some time looking at our usage of ISO BMFF and try to rationalise things across the two libraries.

Video files are the main area where the .NET and Java library outputs differ. The implementations were from the community and they take quite different approaches and produce different results. Unifying them and making them more flexible will be good for the longer term.

[StefanOltmann]

I forgot to update the README.

From what we know now HEIF (HEIC & AVIF) may be an accurate entry.

[drewnoakes]

Thanks, fixed!