FWIW, I've updated the comparison using the latest versions of EROFS and DwarFS. There definitely have been improvements to EROFS in the meantime. DwarFS is still orders of magnitude faster in creating the file system…
Thanks for the feedback! I was probably a little too excited when writing the documentation. Reading it again with some distance, I can certainly see why this might be a bit off-putting. I'll keep this in mind for the…
https://en.wikipedia.org/wiki/Locality-sensitive_hashing https://en.wikipedia.org/wiki/Nilsimsa_Hash https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md#nils...
Correct, there are no images in the data except for 68 PNGs. It's just HTML files.
It's ever so slightly smaller than a .tar.xz of the same data. The main difference being that you don't have to fully extract it in order to access the data.
That's definitely a fair point! DwarFS actually works perfectly fine on 64-bit ARM and would likely work fine on a smartphone as well. Still (and I might be completely wrong about this), I think the primary goal of…
I've got a (relatively old) snapshot of the English Wikipedia that I'm using for testing. The snapshot is around 200 GiB in 14,000,000 files and compresses down to an 11 GiB DwarFS image.
https://github.com/mhx/dwarfs?tab=readme-ov-file#with-erofs It's been a while since I did that comparison, so the results could be significantly different now.
Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs I initially built this for having access to 1000+ Perl installations (spanning decades of Perl releases). The…
> mkdwarfs crashed with recursive links (1-level, just pointing to itself) That's odd, it shouldn't crash with links at all, as it doesn't actively follow links. Can you please file a bug if you can reproduce this? >…
Yes, and I've documented this now: https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...
> have you investigated why this might be the case? Very briefly. It looks like clang has a different strategy breaking up the code (which is mostly C++ templates) into actual functions vs. inlining it, and the hot code…
> If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations. Very true, and it's definitely a great service! However, the set of versions/configurations is still limited,…
This is really cool, I'll give squashfs-tools-ng a try! > Does the Raspbian image comparison compare XZ compression against SquashFS with Zstd? That's correct. It's not an exhaustive matrix of comparisons. > Also, is…
Precisely.
Files that you've accessed will be kept in the kernel's cache. The cache I was talking about is a cache for decompressed blocks. Single files can stretch across multiple blocks, so you need to be able to keep more than…
> @OP: Can you please explain why you keep 50 gigs of perl around? :-) Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release was tested against basically every…
> How opposed would you be to this being reworked to being able to be mainline kernel support too? I don't see any way of getting this anywhere near the kernel without a full rewrite. It's C++ and it depends on…
Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU…
Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.
Read-only media, for example. Or in general, stuff that doesn't really change. In my case: https://github.com/mhx/dwarfs#history
Author here :) I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but to make it better in certain scenarios…
FWIW, I've updated the comparison using the latest versions of EROFS and DwarFS. There definitely have been improvements to EROFS in the meantime. DwarFS is still orders of magnitude faster in creating the file system…
Thanks for the feedback! I was probably a little too excited when writing the documentation. Reading it again with some distance, I can certainly see why this might be a bit off-putting. I'll keep this in mind for the…
https://en.wikipedia.org/wiki/Locality-sensitive_hashing https://en.wikipedia.org/wiki/Nilsimsa_Hash https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md#nils...
Correct, there are no images in the data except for 68 PNGs. It's just HTML files.
It's ever so slightly smaller than a .tar.xz of the same data. The main difference being that you don't have to fully extract it in order to access the data.
That's definitely a fair point! DwarFS actually works perfectly fine on 64-bit ARM and would likely work fine on a smartphone as well. Still (and I might be completely wrong about this), I think the primary goal of…
I've got a (relatively old) snapshot of the English Wikipedia that I'm using for testing. The snapshot is around 200 GiB in 14,000,000 files and compresses down to an 11 GiB DwarFS image.
https://github.com/mhx/dwarfs?tab=readme-ov-file#with-erofs It's been a while since I did that comparison, so the results could be significantly different now.
Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs I initially built this for having access to 1000+ Perl installations (spanning decades of Perl releases). The…
> mkdwarfs crashed with recursive links (1-level, just pointing to itself) That's odd, it shouldn't crash with links at all, as it doesn't actively follow links. Can you please file a bug if you can reproduce this? >…
Yes, and I've documented this now: https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...
> have you investigated why this might be the case? Very briefly. It looks like clang has a different strategy breaking up the code (which is mostly C++ templates) into actual functions vs. inlining it, and the hot code…
> If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations. Very true, and it's definitely a great service! However, the set of versions/configurations is still limited,…
This is really cool, I'll give squashfs-tools-ng a try! > Does the Raspbian image comparison compare XZ compression against SquashFS with Zstd? That's correct. It's not an exhaustive matrix of comparisons. > Also, is…
Precisely.
Files that you've accessed will be kept in the kernel's cache. The cache I was talking about is a cache for decompressed blocks. Single files can stretch across multiple blocks, so you need to be able to keep more than…
> @OP: Can you please explain why you keep 50 gigs of perl around? :-) Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release was tested against basically every…
> How opposed would you be to this being reworked to being able to be mainline kernel support too? I don't see any way of getting this anywhere near the kernel without a full rewrite. It's C++ and it depends on…
Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU…
Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few weeks back.
Read-only media, for example. Or in general, stuff that doesn't really change. In my case: https://github.com/mhx/dwarfs#history
Author here :) I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but to make it better in certain scenarios…