dwarfs-0.11.2

Bugfixes

• macOS Ventura's version of clang appears to be missing an implementation of std::hash<std::filesystem::path, making it hard to define an unordered_map<filesystem::path>. Work around by simply using an unordered_map<string> instead.
• Installing the binaries using cmake did not honor the CMAKE_INSTALL_BINDIR or CMAKE_INSTALL_SBINDIR variables. Fixes github #253.

Full Changelog: v0.11.1...v0.11.2

SHA-256 Checksums

61fce8eaa6bbdf10917a5a12331e192748a54ab1aa175ed6f55cb26825ab3177  dwarfs-0.11.2-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
06fc4ed91ee5c348dbfc70771fe3e3ea6834277e4a58f1f99e0bc98cb16ed3d4  dwarfs-0.11.2-Linux-aarch64-clang.tar.xz
15905007cff432bb9be0bdabed93473764c1706796e0da6f3af083f0a142db6d  dwarfs-0.11.2-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
3c82708e00af9d1622e78047efd216e4e29213a60aff3afa8326bade8353ea38  dwarfs-0.11.2-Linux-x86_64-clang.tar.xz
1b38faf399a6d01cd0e5f919b176e1cab76e4a8507088d060a91b92c174d912b  dwarfs-0.11.2.tar.xz
8a028693ce0a7ab083b25dc491b100f41fbf98f28413a38f6773fe1cf27574fb  dwarfs-0.11.2-Windows-AMD64.7z
600134267dd0ad51dd9d8bd1b58fa614b0a0da9a7a3d57f5fce4dbda9bb80460  dwarfs-universal-0.11.2-Linux-aarch64-clang
a9f5f79afeff4eba5cc23893de46e4c8eaa3b51b8f5938ed7f9e6cb92560fa4f  dwarfs-universal-0.11.2-Linux-aarch64-clang-reldbg-stacktrace
1bee828de84c1a3a1c2134bc866f28bdf93a62927cb7e8c416813f389f7745ad  dwarfs-universal-0.11.2-Linux-x86_64-clang
ddbd62d3bf0bf420a1720af6c03bea21ce6a77a73cedc46f28c8d79e4ac26827  dwarfs-universal-0.11.2-Linux-x86_64-clang-reldbg-stacktrace
d95dab93a7e9d8349d4a4393213a401d9ded79040b1c48e661df7dfe118b72a7  dwarfs-universal-0.11.2-Windows-AMD64.exe

dwarfs-0.11.1

Bugfixes

• macOS Ventura's version of clang appears to be missing the <source_location> header, despite Apple claiming otherwise. Fix this by shipping a wrapper and providing a fallback implementation.

Full Changelog: v0.11.0...v0.11.1

SHA-256 Checksums

3e1b6331cf2f589d7058700aa2c5dc41f1825f3954f3828eb709034ba57a7c97  dwarfs-0.11.1-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
23b1e0b18a7c3ffeb6c5fcc97ab032a7c1c651454d0fa5cb9741918d97a14ab3  dwarfs-0.11.1-Linux-aarch64-clang.tar.xz
4ec6614e87064ac96dfdb9b7957620bd889f5f1e95416409369d00606cfff0a1  dwarfs-0.11.1-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
1eebf6e66eb5d6dc7cfb9c9b3c7c6e67084acc5ced7a018c15a511e929598f99  dwarfs-0.11.1-Linux-x86_64-clang.tar.xz
7a0cccb1ec3c2a18e9a014893c1d3e1f8f2c44ade6936c9f6d3bab5ec14b2052  dwarfs-0.11.1.tar.xz
c43fd9f2089b94ddd2819c7853dd6d8c34951e2d42cbfdc2e4470cde9c3e18fb  dwarfs-0.11.1-Windows-AMD64.7z
e9bf1f8bcccf363be25396d2f60d9f4e7765eba5bd647f071aa4d0ba5cb3785b  dwarfs-universal-0.11.1-Linux-aarch64-clang
22966f1dba98697db0cad127d1d8c50ef5952b5c9816cc2564b9410a37cdbaa3  dwarfs-universal-0.11.1-Linux-aarch64-clang-reldbg-stacktrace
0a025dc0f854ad9f3a5f9ca89ac43ada5305de14fe4b2e03088ce9ee5a23dbf4  dwarfs-universal-0.11.1-Linux-x86_64-clang
a38de846f48c7979c204699af6a1fda0d8d634caac68223b2b46fcb1c52c0e56  dwarfs-universal-0.11.1-Linux-x86_64-clang-reldbg-stacktrace
b8299e5f2102283c52c1cbd8ad14a0c3b71244937327e895e0aee809d92e4474  dwarfs-universal-0.11.1-Windows-AMD64.exe

dwarfs-0.11.0

Bugfixes

• Remove the access implementation from the FUSE driver. There's no point here trying to be more clever than FUSE's default. This makes sure DwarFS will behave more like other FUSE file systems. See github discussion #244 for details.
• Limit the number of chunks returned in inodeinfo xattr. Highly fragmented files would have megabytes in inodeinfo, which not only breaks the xattr interface, but can also dramatically slow down tools like eza who like to read xattrs for no apparent reason.
• Avoid nested indentation in manpages to work around ronn-ng bug. Fixes github #249.
• Don't link library against jemalloc. This fixes both issues with pydwarfs and issues building with jemalloc support on macOS. Only the binaries are now linked against jemalloc, which should be sufficient.

Features

• Support case-insensitive lookups. Fixes github #232.
• Allow setting image size in FUSE driver. Fixes github #239.
• Support extracting a subset of files with dwarfsextract using the new --pattern option. The same glob patterns can be used as for the filter rules in mkdwarfs. Fixes github #243.
• Allow overriding UID / GID for the whole file system when using the FUSE driver on non-Windows platforms. See github discussion #244.
• Expose more LZMA options (mode, mf, nice, depth).
• Improve filter patterns, which now support ranges and complementation.
• Improve speed of filesystem walk / walk_data_order calls by 80% / 40%. The impact of this will largely depend on what the code is being run for each inode, but, for example, the speed of listing more than 14 million files with dwarfsck will take about 16 seconds compared to 17 seconds with the previous release.
• Added an inode size cache to the metadata to speed up file size computation for large, highly fragmented files. The configuration is currently fixed using a conservative default. Only files with at least 128 chunks will be added to the cache, so in a lot of cases this cache may be completely empty and not contribute to the size of the file system image at all.
• Use bit-packing for hardlink, shared files, and chunk tables. This will consume less memory when loading a DwarFS image.
• Show total hardlink size in dwarfsck output.
• Library: return a dir_entry_view from readdir and find. This is more consistent, but was previously not easily possible due to the lack of a "self" dir entry in the metadata. The "self" entry has been added and will only impact the size of the metadata if directories metadata is not packed.
• Library: prefer std::string_view over char const*.
• Library: add directory iterator to directory_view.
• Library: support for maxiov parameter in readv call.

Other

• Lots of internal refactoring to improve overall code quality.

Full Changelog: v0.10.2...v0.11.0

SHA-256 Checksums

2040a951697ddb78a4b6ee887e06be4295f9d2e3708a311ac72ad9ad7bd28aa3  dwarfs-0.11.0-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
0db0d6bc823d26f86d47f1edf8b4ddbcf85fab24e118be7de9ee091234b5623e  dwarfs-0.11.0-Linux-aarch64-clang.tar.xz
a7214b10902653c582aa4c21e05e2476518ed1d15e4640cc3eb2bbe53a297120  dwarfs-0.11.0-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
35e851bce5ba6a17b6b53081d1305ebcee5698d8bc770b8b1a875d2986fd6d7c  dwarfs-0.11.0-Linux-x86_64-clang.tar.xz
852c96133444493eff6f03324bc2700e31859d75410a937f0714eae9f75d2dd4  dwarfs-0.11.0.tar.xz
15591223010400488c5066a864bcee3ad71c045e2aa4bf60b7c05e9d45909b9f  dwarfs-0.11.0-Windows-AMD64.7z
da197d19b3eadfea5180034765d70c050ae9b85ade58dd0aa91b65283a079236  dwarfs-universal-0.11.0-Linux-aarch64-clang
d58ad14583345d4e7efb4ddb0278ec39c836646a39868422ca1358fa22a990b7  dwarfs-universal-0.11.0-Linux-aarch64-clang-reldbg-stacktrace
72fe171dd9d9abd0bba46e52a983934affbcc9a7349d07854eda91d788ea686b  dwarfs-universal-0.11.0-Linux-x86_64-clang
1c5b19c21aca4dc6df8cff3e06358c96fb4e3bb1e969ed3ceef0eb381d84f98b  dwarfs-universal-0.11.0-Linux-x86_64-clang-reldbg-stacktrace
f2451ed0832c13157f869a3d7ba3596fcb4bb0c5c55741fc054ce6b1bdc977c8  dwarfs-universal-0.11.0-Windows-AMD64.exe

dwarfs-0.10.2

Bugfixes

• Gracefully handle localized error message on Windows. These error messages can contain characters from a Windows (non-UTF-8) code page, which could cause a fatal error in fmt::print in the logging code. Call sites that log such error messages now try to convert these from the code page to UTF-8 or, if that fails, simply replace all characters that are invalid from a UTF-8 point-of-view. Partial fix for #241.
• Handle invalid wide chars in file names on Windows. For some reason, Windows allows invalid UTF-16 characters in file names. Try to handle these gracefully when converting to UTF-8. Partial fix for #241.
• Workaround for new boost versions which have a process component.
• Workaround for a deprecated boost header.
• Support for upcoming Boost 1.87.0. io_service was deprecated and replaced by io_context in 1.66.0. The upcoming Boost 1.87.0 will remove the deprecated API. (Thanks to Michael Cho for the fix.)
• Disable extended output algorithms (shake(128|256)).
• Install libraries to CMAKE_INSTALL_LIBDIR. Fixes #240.
• mode/uid/gid checks were expecting 16-bit types.
• stricter metadata checks and improved error messages.
• Various fixes for filesystem_extractor to prevent memory leaks, correctly handle errors during extraction, and prevent creation of invalid archive outputs due to padding.
• Various minor fixes: non-virtual dtors, missing includes, std::move vs. std::forward, unused code removal.

Other

• More test cases for stricter metadata checks. Also enable the strict checks in in unit tests by default.
• Fix typos in README.md. (Thanks to Christian Clauss for the fix.)
• Fix typos in man pages.

New Contributors

• @cclauss made their first contribution in #235
• @cho-m made their first contribution in #245

Full Changelog: v0.10.1...v0.10.2

SHA-256 Checksums

2f4d275d006228acb2280c3bf5327f02098c2ba110d81fe3854a80f5fd848058  dwarfs-0.10.2-Linux-aarch64-clang-reldbg-stacktrace.tar.xz
75878252ef0bfc490e5bd6ad5870bc5a02531650ceacf1258807e09606069561  dwarfs-0.10.2-Linux-aarch64-clang.tar.xz
74b52460ebd2d8e752ad7fbe976c683be542a8a581fdf25ac59ba1dea5bc5d0c  dwarfs-0.10.2-Linux-x86_64-clang-reldbg-stacktrace.tar.xz
a018bfe2531763a273a2d78bc507b1c89fe58a44f7955c980c854a55f9adbaea  dwarfs-0.10.2-Linux-x86_64-clang.tar.xz
36767290a39f92782e41daaa3eb45e39550ad1a4294a6d8365bc0f456f75f00c  dwarfs-0.10.2.tar.xz
c15280d920b67b51b42117612bd8a959eb5ca9ed0202fd765e19743aad61a728  dwarfs-0.10.2-Windows-AMD64.7z
36f72f1ff049a1d955e68547540b932539beab44b0cba23efbdb7a1b0bfd32d4  dwarfs-universal-0.10.2-Linux-aarch64-clang
4d55e783e352a5aafc321f7ac36964b0493601320d3d93d021634e78e743505d  dwarfs-universal-0.10.2-Linux-aarch64-clang-reldbg-stacktrace
b565399a0a671d06be3e078376e02b388ee14133680b8d19483fc93c294b12d2  dwarfs-universal-0.10.2-Linux-x86_64-clang
cb374fc2d64bbf3bd4dd4714f1be37e3d6fc6ecffc7afd93714b6897e9d3751a  dwarfs-universal-0.10.2-Linux-x86_64-clang-reldbg-stacktrace
eb69b1bf4703d28bd3d5f477dca1ab3460dda4250c7ce1899eb4192c2c1bef69  dwarfs-universal-0.10.2-Windows-AMD64.exe

dwarfs-0.10.1

Bugfixes

• Allow building utils_test against a non-compatible, system-installed version of gtest. This is a common issue when trying to integrate dwarfs into a package manager, as these generally disallow fetching external dependencies at build time.
• dwarfsck was always reporting a block size of 1 byte rather than the actual block size of the image.
• DWARFS_HAVE_LIBBROTLI was not set correctly in the config file, causing build errors if the library was built without brotli.
• Several small fixes for building with Homebrew.

Full Changelog: v0.10.0...v0.10.1

SHA-256 Checksums

53bb766f3a22f019c4bac7cbf995376cb4f3f0ad69e4793471af11c954185227  dwarfs-0.10.1-Linux-aarch64-clang-stacktrace.tar.xz
f272f667649d71ec7d29d6822ad4198e13a33e997d722f74f2bca23b239de72f  dwarfs-0.10.1-Linux-aarch64-clang.tar.xz
671ce264938ab4cacc8af0aabcacb1ecfffa01284b4959441e921264ae19b47e  dwarfs-0.10.1-Linux-x86_64-clang-stacktrace.tar.xz
84894bf6a26cac2eb2c8d43d6fccf1ece7665c4c15050cec494d09199bd8310e  dwarfs-0.10.1-Linux-x86_64-clang.tar.xz
4041ed9aa19e03f44dbe69b470f31423a3c358bcd07e78230311b859629785b6  dwarfs-0.10.1-Windows-AMD64.7z
db785e0e0f257fa4363d90153db34127add4552791a72998b30ded787840d039  dwarfs-0.10.1.tar.xz
3e003c9a5fbf31b75548c11a2c2c1958f606ce2c2022db4baa6d62b80201c76d  dwarfs-universal-0.10.1-Linux-aarch64-clang
44ad0a3f2d89e373b0279d1db7c19aeca46879972a2db226e31ec7ebe8ff103e  dwarfs-universal-0.10.1-Linux-aarch64-clang-stacktrace
18f99297c7425bd1bea87d47a2046bfc7e00fe9cc566f024e631ed39a6bb1913  dwarfs-universal-0.10.1-Linux-x86_64-clang
c60821be4a248be2feb54905b5bb6c5cd323014bcb7107f0d586ba7f70deb488  dwarfs-universal-0.10.1-Linux-x86_64-clang-stacktrace
768d013d55cd030c1fbabd35ad522648581c79435da4994cc39de75b3a7eda30  dwarfs-universal-0.10.1-Windows-AMD64.exe

dwarfs-0.10.0

This release doesn't bring a lot of new features or bugfixes to the command-line tools. However, large parts of the code base have been refactored (in more than 300 commits since the last release) and a couple of long-standing issues have been resolved:

• It is finally possible to properly install the libraries that implement most of DwarFS. These libraries come with all necessary headers and a CMake config script so you can start building your own tools.

• Shared library builds are now explicitly supported. These have caused some trouble in the past, but the problematic code has been refactored to avoid issue like missing compression / categorizer code.

• The code now builds about twice as fast. This is due to shipping generated code in the source tarball and removing a lot of dependencies into the folly library.

• It is now possible to do modular builds, i.e. the library, tools and FUSE driver can be built independently.

Bugfixes

• Fixed a race condition identified by ThreadSanitizer in the root node name processing.

• The terminal abstraction code did not check any errors when trying to determine the terminal width, leading to a random terminal width value. This caused the manual page tests to occasionally crash.

• Fixed some flaky tests, e.g. unmounting the FUSE driver on macOS.

Features

• Two sets of universal binaries and binary tarballs are provided for Linux platforms: one without any debug symbols, the other with minimal debug symbols and support for stack traces. For the universal binary, only the version without debug symbols will be UPX-compressed, as the stack trace functionality doesn't work with a compressed binary.

• Symbolic links to the universal binary may now be suffixed with a version (i.e. any part of the name starting with - and followed by a digit will be ignored, e.g. the symlink could be mkdwarfs-0.10 and it would be treated as mkdwarfs).

• Introduced support for extended attributes on Windows, including a new utility for cross-platform xattr manipulation (pxattr, for portable xattr).

• Enhanced file system API, adding error-code based and exception-safe versions for getattr, access, and similar functions.

• Filter rules now consistently use Unix path separators, even for the root path component. Addresses a comment in github discussion #228.

Improvements

• Extensive refactoring to improve code modularity, maintainability and to provide proper libraries. The library code has been moved to different namespaces to make it easier to understand the role of different components (e.g. reader, writer, extractor).

• Replaced all folly library dependencies in the public DwarFS library interface with alternatives from libraries like e.g. boost or nlohmann::json which are more broadly available. folly and fbthrift are still used as implementation details, but no longer leak into the public library interfaces.

• A much smaller subset of folly is now used in DwarFS and only the necessary components are built, significantly reducing the number of compilation units when building DwarFS.

• It is now possible to do modular builds in addition to the default monolithic build, i.e. you can build and install just the DwarFS libraries and later build/install the tools (mkdwarfs, ...) and/or the FUSE driver against these libraries. This is particularly useful for packaging (e.g. in Homebrew, which has removed all FUSE support from the core formulae).

• Shared library builds are now explicitly supported. This fixes issues such as github #184.

• The source tarball now contains all auto-generated code, e.g. manual pages or generated thrift code. This reduces the number of build-time dependencies (e.g. ronn or mistletoe are no longer required) and significantly reduces the build steps (it is no longer necessary to build the thrift compiler). The build is now roughly twice as fast as in the 0.9.x releases.

• The parallel-hashmap, xxHash and zstd submodules have been removed from the git repo and are no longer added to the source tarball. Both xxHash and zstd are now widely available. If a suitable version of parallel-hashmap is found on the system, this will be used, otherwise it will be fetched during the build. Being a header-only library and only used internally, there's no need for it to be installed.

• A lot of GCC warnings have been fixed and upstreamed to folly / fbthrift.

New Contributors

• @mindfocus made their first contribution in #222

Full Changelog: v0.9.10...v0.10.0

SHA-256 Checksums

52045506839249823a9b05b711f89e3fd18a729b264149fbe74f557bd8f16b81  dwarfs-0.10.0-Linux-aarch64-clang-stacktrace.tar.xz
b12a33650694baa6f546ac18768b8a7b2e1dd90c1c674d26bec7f5149a285537  dwarfs-0.10.0-Linux-aarch64-clang.tar.xz
8593eaecc1bed4f570626ff07b0e56a7f6a025c3a5849f4b0056006ab9801393  dwarfs-0.10.0-Linux-x86_64-clang-stacktrace.tar.xz
53f02da7ff7fe484b3bb91e6532987c3fb59db28131309ccbb46d2c2c9cec9ce  dwarfs-0.10.0-Linux-x86_64-clang.tar.xz
f24ee132cf0c77b0c94e49b3d8108fdf7e0ff965031bbc34aa51e3c2e066fa43  dwarfs-0.10.0-Windows-AMD64.7z
c01ae59d4662e4f027a7c8a5934b7aebe6edc4f7affd836fa73e6d861d18bf35  dwarfs-0.10.0.tar.xz
573656e33a171017046a32500b9ebe50ebd117df16212e6ef8f8d4b034618210  dwarfs-system-gtest.patch
149c9518d5229bf6dd07b5c39df02ed5cbc4bddbf9bf54f6fa5617d76a594b1c  dwarfs-universal-0.10.0-Linux-aarch64-clang
bfaad42a15c687fdaf502283792322692e479c563c56e8734efd9a298956d716  dwarfs-universal-0.10.0-Linux-aarch64-clang-stacktrace
5f3401cdd7267b6eec6f758f902d7d874534cc77b5be7233944832992e2c0081  dwarfs-universal-0.10.0-Linux-x86_64-clang
307f973fd7da525c5f06f6090b4a4f7a0bea041d785dcb718fb226f6b15f942f  dwarfs-universal-0.10.0-Linux-x86_64-clang-stacktrace
99cf71972156ec6f5d88ab0331709ecdc1e80f571582fce95c89b694d3be258b  dwarfs-universal-0.10.0-Windows-AMD64.exe

dwarfs-0.9.10

Bugfixes

• When cloning LZMA compressor objects, the LZMA filter options of the cloned instance would still point to an options object in the original instance. This could lead to LZMA errors when initializing a new compressor. Fixes github #224.
• Fetch range-v3 if no suitable version is found. Fixes github #221.
• Filter rules did not work correctly when input is root dir
• duf reports odd sizes due to using bsize instead of frsize

dwarfs-0.9.9

Bugfixes

• A bug introduced by an optimization to skip hashing of large files if they already differ in the first 4 KiB could, under rare circumstances, lead to an unexpected "inode has no file" exception after the scanning phase. This bug did not cause any file system inconsistency issues; mkdwarfs either crashes with the exception, or its output will be correct. Fixes github #217 (see also for more details).

Features

• A sequential access detector was added to the block cache, which can trigger a prefetch of blocks assumed to be read in the future. This improves sequential read throughput roughly by a factor of two. Random access should typically be unaffected. Can be configured / disabled using -o seq_detector.

• Added tracing support in FUSE driver and dwarfsextract, which allows simple performance analysis using chrome://tracing. Traces can be enabled using -o perfmon_trace and --perfmon-trace.

• Added performance monitoring and tracing support for the block cache.

Performance

• Significantly improved the speed of dwarfsck --checksum.

dwarfs-0.9.8

Bugfixes

• Build custom version of libcrypto to link with the release binaries in order for them to run properly on FIPS-enabled setups. Fixes github #210.

• When mounting a DwarFS image on macOS and viewing the volume in Finder, only the directories were shown, but no files. The root cause was that a non-existent extended attribute is reported via a different error code in macOS (ENOATTR) compared to Linux (ENODATA) and the wrong error code was returned for certain Finder-related attributes. Fixes github #211.

• macOS builds using jemalloc were crashing when calling mallctl("version", ...). The root cause of the crash is still unclear, but as a workaround, the jemalloc version is compiled in from a preprocessor constant rather than using mallctl.

dwarfs-0.9.7

Bugfixes

• Handle root uid correctly in access() implementation. Fixes github #204.

Features

• Show and track library dependencies. Dependencies will be displayed in the command line help; they will also be tracked in the history metadata of a DwarFS image. See also github #207.

Documentation

• Describe nilsimsa ordering algorithm more accurately.

Performance

• Reorder branches to improve ricepp speed with real world data.
• Some tweaks to improve segmenter speed.