Comparison with EroFS #28

cipriancraciun · 2020-12-16T12:27:09Z

cipriancraciun
Dec 16, 2020

So I've tried to compare DwarFS with EroFS in the use-case of CDNJS (i.e. creating an image from the ~260 GiB that is the entire CDNJS repository checkout).

In what follows I'll paste the output of the time for the following format (I use the external time command as it supports more reporting fields):

/usr/bin/time -f '[git-clone]  (( elapsed: %E (user: %U, kernel: %S), CPU: %P, memory: %M (faults: %F, swapped: %W), I/O: %I / %O (waits: %w) ))' -- \
{command}

The following are times it took Git to do various tasks:

# -- [git-clone]  (( elapsed: 15:33.19 (user: 865.51, kernel: 69.15), CPU: 100%, memory: 2905824 (faults: 1240, swapped: 0), I/O: 19984224 / 27031808 (waits: 4136221) ))

# -- [git-tar-archive:git]  (( elapsed: 1:25:20 (user: 476.63, kernel: 85.42), CPU: 10%, memory: 10851244 (faults: 5135, swapped: 0), I/O: 20730952 / 0 (waits: 1281401) ))
# -- [git-tar-archive:zstd]  (( elapsed: 1:25:20 (user: 5148.13, kernel: 105.85), CPU: 102%, memory: 57908 (faults: 1, swapped: 0), I/O: 1704 / 129921936 (waits: 3979491) ))

# -- [git-untar-archive:tar]  (( elapsed: 20:27.51 (user: 30.61, kernel: 323.40), CPU: 28%, memory: 2348 (faults: 1, swapped: 0), I/O: 1785664 / 536887264 (waits: 1209226) ))
# -- [git-untar-archive:zstd]  (( elapsed: 20:27.51 (user: 239.27, kernel: 82.46), CPU: 26%, memory: 4508 (faults: 1, swapped: 0), I/O: 126295464 / 0 (waits: 7138372) ))

# -- [git-optimize:repack]  (( elapsed: 1:21:52 (user: 8016.07, kernel: 21.64), CPU: 163%, memory: 15655560 (faults: 36554, swapped: 0), I/O: 36914824 / 18209488 (waits: 129070) ))
# -- [git-optimize:fsck]  (( elapsed: 0:11.08 (user: 3.22, kernel: 1.00), CPU: 38%, memory: 2264212 (faults: 11167, swapped: 0), I/O: 9205024 / 8 (waits: 11224) ))

Basically from these commands the git-optimize:repack would be the equivalent of the mkdwarfs. The same for git-optimize:fsck equivalent to dwarfsck.

The following is the time that it took to create the DwarFS image:

# -- [mkfs-dwarfs:default-bp]  (( elapsed: 5:26:46 (user: 63479.34, kernel: 1335.10), CPU: 330%, memory: 8760668 (faults: 8791191, swapped: 0), I/O: 889590520 / 19573744 (waits: 26082909) ))
# -- [fsck-dwarfs:default-bp]  (( elapsed: 0:05.12 (user: 0.23, kernel: 0.55), CPU: 15%, memory: 553328 (faults: 6077, swapped: 0), I/O: 6180896 / 0 (waits: 6107) ))

mkdwarfs \
        --input "$( exec -- readlink -e -- ./cdnjs/sources )" \
        --output ./cdnjs/sources.images/dwarfs--default-bp \
        --order path \
        --num-workers 16 \
        --max-scanner-workers 128 \
        --set-owner 0 \
        --set-group 0 \
        --set-time 0 \
        --time-resolution day \
        --progress ascii \
#

The following is the time that it took to create the EroFS image:

# -- [fuse-dwarfs:default-bp-w16]  (( elapsed: 24:31:37 (user: 258.91, kernel: 239.32), CPU: 0%, memory: 6076272 (faults: 6626, swapped: 0), I/O: 23879912 / 0 (waits: 17746668) ))
# -- [mkfs-erofs:lz4-12]  (( elapsed: 23:10:39 (user: 82703.50, kernel: 269.63), CPU: 99%, memory: 11944428 (faults: 1, swapped: 0), I/O: 1778984 / 147655224 (waits: 14500992) ))

mkfs.erofs \
        -z lz4,12 \
        -x -1 \
        -E force-inode-compact \
        -T 0 \
        -- \
        ./cdnjs/sources.images/erofs--lz4-12 \
        ./cdnjs/sources.mount \
#

Unfortunately I wasn't able to mount the EroFS image (it mounted, but accessing files it gave an error, which most likely means incompatible tools with the kernel).

However I was able to test DwarFS:

# -- [fuse-dwarfs:default-bp-w16]  (( elapsed: 25:29.12 (user: 475.04, kernel: 509.45), CPU: 64%, memory: 12352984 (faults: 6650, swapped: 0), I/O: 19584200 / 0 (waits: 59628800) ))
# -- [fuse-dwarfs:hash-sha2-256:w16-walk]  (( elapsed: 11:15.28 (user: 2404.53, kernel: 511.98), CPU: 431%, memory: 1213916 (faults: 6, swapped: 0), I/O: 399951856 / 0 (waits: 63730409) ))

# -- [fuse-dwarfs:default-bp-w16]  (( elapsed: 10:17.43 (user: 470.03, kernel: 518.59), CPU: 160%, memory: 11972360 (faults: 0, swapped: 0), I/O: 0 / 0 (waits: 49470967) ))
# -- [fuse-dwarfs:hash-sha1:w16-walk]  (( elapsed: 7:57.87 (user: 1051.00, kernel: 479.67), CPU: 320%, memory: 1000884 (faults: 0, swapped: 0), I/O: 399770944 / 0 (waits: 68152794) ))

# -- [fuse-dwarfs:default-bp-w16]  (( elapsed: 7:48.02 (user: 473.39, kernel: 535.25), CPU: 215%, memory: 11870380 (faults: 0, swapped: 0), I/O: 0 / 0 (waits: 42157393) ))
# -- [fuse-dwarfs:hash-md5:w16-walk]  (( elapsed: 7:06.56 (user: 637.87, kernel: 472.92), CPU: 260%, memory: 987408 (faults: 0, swapped: 0), I/O: 399689064 / 0 (waits: 69013228) ))

# -- [fuse-dwarfs:default-bp-w16]  (( elapsed: 10:36.74 (user: 1419.90, kernel: 542.82), CPU: 308%, memory: 12341944 (faults: 6391, swapped: 0), I/O: 19584304 / 0 (waits: 53667533) ))
# -- [fuse-dwarfs:hash-sha1:w16-random]  (( elapsed: 10:24.24 (user: 1043.07, kernel: 509.49), CPU: 248%, memory: 1100188 (faults: 6, swapped: 0), I/O: 389169368 / 0 (waits: 67453010) ))

For comparison the following are equivalent runs for the files on Ext4:

# -- [hash-sha2-256:w64a-extent]  (( elapsed: 32:48.67 (user: 1932.38, kernel: 454.65), CPU: 121%, memory: 1399872 (faults: 2, swapped: 0), I/O: 549280576 / 0 (waits: 14511150) ))
# -- [hash-sha2-256:w64s-extent]  (( elapsed: 34:30.94 (user: 2010.10, kernel: 618.92), CPU: 126%, memory: 532920 (faults: 7, swapped: 0), I/O: 547546232 / 0 (waits: 17152518) ))

The hashing was done with my own md5-tools (https://github.com/volution/volution-md5-tools), which run 64 I/O threads (the w64), either in parallel (the a after w64) or in alternate (the s after the w64) with the walking thread.

The following are the sizes for the various folders, images, archives:

259.0G    ./cdnjs/sources
  8.7G    ./cdnjs/sources.git
 62.0G    ./cdnjs/sources.tar.zst
  9.4G    ./cdnjs/sources.images/dwarfs--default-bp
 70.0G    ./cdnjs/sources.images/erofs--lz4-12

My short conclusion based on this data is:

DwarFS obtains indeed a very good compression rate;
on the other side, reading is also efficient, basically it is able to saturate SHA2-256; (i.e. DwarFS is able to decompress faster than SHA2-256 is able to hash;)
on the downside, the FUSE driver eats quite a lot of memory; (this has been discussed in FUSE daemon -o cachesize issue #9);
also on the downside, the time it took to create the DwarFS image is quite a lot, ~5h, however it pales in comparison to the almost ~24h taken by EroFS;

mhx · 2020-12-16T13:55:34Z

mhx
Dec 16, 2020
Maintainer

I'm actually surprised you got EROFS to produce an image on such a large corpus. I gave up after 15 hours, concluding that there might be some sort of accidentally quadratic complexity built into mkfs.erofs.

on the downside, the FUSE driver eats quite a lot of memory; (this has been discussed in #9);

That should at least improve with the switch to jemalloc.

8 replies

cipriancraciun Dec 18, 2020
Author

OK, I've given dwarf-0.3.0 a try, but only the FUSE driver. Apparently after starting FUSE, without any other file access of the mount-point, the driver takes ~128 MiB:

Rss:              522380 kB
Pss:              522376 kB
Pss_Anon:         127500 kB
Pss_File:         394876 kB
Pss_Shmem:             0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:    394876 kB
Private_Dirty:    127500 kB
Referenced:       522380 kB
Anonymous:        127500 kB
LazyFree:              0 kB
AnonHugePages:    126976 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB

During the read execution, the Pss_Anon was around ~450 MiB, which isn't "much", but it isn't "small" either.

After I've finished reading the entire file-system the following is the memory consumption, namely ~380 MiB for Pss_Anon, which seems to be more than the initial ~128 MiB:

Rss:            10117832 kB
Pss:            10117828 kB
Pss_Anon:         383464 kB
Pss_File:        9734364 kB
Pss_Shmem:             0 kB
Shared_Clean:          4 kB
Shared_Dirty:          0 kB
Private_Clean:   9734364 kB
Private_Dirty:    383464 kB
Referenced:     10117832 kB
Anonymous:        383464 kB
LazyFree:              0 kB
AnonHugePages:    118784 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB

mhx Dec 18, 2020
Maintainer

The 128 MiB are the file system metadata that is read upon mounting:

./dwarfsck cdnjs.dwarfs | grep METADATA
SECTION type=METADATA_V2_SCHEMA, compression=ZSTD, length=387, blocksize=786, ratio=49.24%
SECTION type=METADATA_V2, compression=ZSTD, length=60041622, blocksize=130970464, ratio=45.84%

You can store this uncompressed, in which case it'll just be mmapd instead of taking up anon memory.

What cachesize did you run the driver with?

mhx Dec 18, 2020
Maintainer

You can store this uncompressed, in which case it'll just be mmapd instead of taking up anon memory.

With the release candidate of mkdwarfs, you can rewrite the file system image with uncompressed metadata:

mkdwarfs -i old.dwarfs -o new.dwarfs --recompress metadata --metadata-compression null

cipriancraciun Dec 18, 2020
Author

You can store this uncompressed, in which case it'll just be mmapd instead of taking up anon memory.

I've not fiddled with the default compression arguments (see the invocation in my original comment above).

However now I see that the default level 7 uses also meta-data compression.

What cachesize did you run the driver with?

dwarfs \
        -f \
        -o workers=16 \
        -o cachesize=256m \
        -o decratio=25 \
        -o mlock=none \
        -o cache_image \
        -o no_cache_files \
        -o readonly \
        -- \
        ./cdnjs/sources.images/dwarfs--default-bp \
        ./cdnjs/sources.mount \
#

mhx Dec 18, 2020
Maintainer

So that makes perfect sense then: the 380 MiB you're seeing eventually are 128 MiB for the metadata + 256 MiB for the cache.

[...]
-o decratio=25

I need to add range checks for some options... The 25 means "2500%" in this case. However, it doesn't matter anyway as only LZMA currently uses streaming decompression. Adding this for ZSTD is on the list of things to do.

mhx · 2020-12-16T14:09:07Z

mhx
Dec 16, 2020
Maintainer

--order path \

This will likely reduce the achievable compression ratio and will likely make compression slower.

--set-owner 0 \
--set-group 0 \

These really only make sense if the source files do in fact belong to different uids/gids. In case of a git checkout, I would assume that it's only one user/group, so exactly zero bits will be dedicated to the storage of uid/gid per file system entry.

--set-time 0 \
--time-resolution day \

Setting the time resolution doesn't make any difference here, as the time is already set to zero. The same is true, btw, if the time is set to any other constant. That time base constant will be stored exactly once, and all file system entry times will be stored relative to the time base. If they're all the same, again, zero bits will be used per file system entry.

2 replies

cipriancraciun Dec 16, 2020
Author

The use-case I have in mind don't use user, group and timestamps. Thus in case of --set-owner / --set-group I didn't understand quite correctly: should I set them to 0 or I shouldn't specify them at all? Please note that another use-case for these is "anonymizing" the owner, or "canonicalizing" them. (In case of --set-time it is clear that it's OK to use an overriding value.)

In fact it would be nice to have exactly these options (for owner, group and timestamp) also for the FUSE driver, thus allowing one to override what is stored in the image. (Think of a use-case for repeatable builds where constant values is needed.)

Also, as mentioned in previous issues, an option to override the file and directory mode would be nice, both at creation and mounting time. Something like --set-file-mode 0440 and --set-dir-mode 0550 plus --keep-executability that in case of files would make sure to keep the executability bits. This also helps in canonicalizing the image, without mangling the actual files on the original disk.

Regarding the --order path I am aware that it reduces the compression, however my concern is access time patterns. For example in the case of CDNJS, if one accesses the file of a library, it's quite likely he'll access another file part of the same library, thus although the FUSE driver doesn't have any read-ahead capability, the kernel will read-ahead at least the backing image.

mhx Dec 16, 2020
Maintainer

The use-case I have in mind don't use user, group and timestamps. Thus in case of --set-owner / --set-group I didn't understand quite correctly: should I set them to 0 or I shouldn't specify them at all? Please note that another use-case for these is "anonymizing" the owner, or "canonicalizing" them. (In case of --set-time it is clear that it's OK to use an overriding value.)

It's totally fine to use these for anonymization/canonicalization. I was looking at this purely from a "get the best compression ratio" angle, where these options don't give you any benefit if your source data doesn't have more than one user/group.

In fact it would be nice to have exactly these options (for owner, group and timestamp) also for the FUSE driver, thus allowing one to override what is stored in the image. (Think of a use-case for repeatable builds where constant values is needed.)

This would absolutely make sense and it's been in the back of my head for a while. This will eventually get implemented.

Also, as mentioned in previous issues, an option to override the file and directory mode would be nice, both at creation and mounting time. Something like --set-file-mode 0440 and --set-dir-mode 0550 plus --keep-executability that in case of files would make sure to keep the executability bits. This also helps in canonicalizing the image, without mangling the actual files on the original disk.

Yep, again, makes sense, will eventually get done. :)

Regarding the --order path I am aware that it reduces the compression, however my concern is access time patterns. For example in the case of CDNJS, if one accesses the file of a library, it's quite likely he'll access another file part of the same library, thus although the FUSE driver doesn't have any read-ahead capability, the kernel will read-ahead at least the backing image.

This is definitely a valid concern, and it'd definitely be nice to get some hard data here, e.g. latency, CPU time spent by the FUSE driver, for certain non-artificial access patterns.

mhx · 2020-12-16T20:19:32Z

mhx
Dec 16, 2020
Maintainer

FWIW, from cdnjs @ 5b47a977

$ time ./mkdwarfs -i /tmp/test/cdnjs -o cdnjs2.dwarfs --set-time now
real    167m29.935s
user    1015m3.230s
sys     49m55.468s
$ ll -h cdnjs2.dwarfs
-rw-r--r-- 1 mhx users 6.7G Dec 16 17:47 cdnjs2.dwarfs

So it's not actually that much smaller.

0 replies

eriksjolund · 2024-04-12T08:44:23Z

eriksjolund
Apr 12, 2024

I noticed that there is ongoing development for adding multi-threading support to mkfs.erofs
https://lore.kernel.org/all/20240204103310.141249-1-zhaoyifan@sjtu.edu.cn/T/

0 replies

mhx · 2024-04-18T06:53:24Z

mhx
Apr 18, 2024
Maintainer

I've updated the comparison using the latest version of erofs-utils from the dev branch and using different options for building the file system.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comparison with EroFS #28

{{title}}

Replies: 5 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Comparison with EroFS #28

cipriancraciun Dec 16, 2020

Replies: 5 comments · 10 replies

mhx Dec 16, 2020 Maintainer

cipriancraciun Dec 18, 2020 Author

mhx Dec 18, 2020 Maintainer

mhx Dec 18, 2020 Maintainer

cipriancraciun Dec 18, 2020 Author

mhx Dec 18, 2020 Maintainer

mhx Dec 16, 2020 Maintainer

cipriancraciun Dec 16, 2020 Author

mhx Dec 16, 2020 Maintainer

mhx Dec 16, 2020 Maintainer

eriksjolund Apr 12, 2024

mhx Apr 18, 2024 Maintainer

cipriancraciun
Dec 16, 2020

Replies: 5 comments 10 replies

mhx
Dec 16, 2020
Maintainer

cipriancraciun Dec 18, 2020
Author

mhx Dec 18, 2020
Maintainer

mhx Dec 18, 2020
Maintainer

cipriancraciun Dec 18, 2020
Author

mhx Dec 18, 2020
Maintainer

mhx
Dec 16, 2020
Maintainer

cipriancraciun Dec 16, 2020
Author

mhx Dec 16, 2020
Maintainer

mhx
Dec 16, 2020
Maintainer

eriksjolund
Apr 12, 2024

mhx
Apr 18, 2024
Maintainer