Built stop the excellent SlateDB! Breaks files down into 256k chunks. Encrypted. Much much better posix compatibility than most FUSE alternatives. SlateDB has snapshots & clones, so that could be another great superpower of zerofs.
Incredible performance figures, rocketing to probably the best way to use object storage in an fs like way. There's a whole series of comparisons, & they probably need a logarithmic scale given the scale of the lead slatedb has! https://www.zerofs.net/zerofs-vs-juicefs
Speaks 9p, NFS, or NBD. Some great demos of ZFS with l2arc caches giving a near local performance while having s3 persistence.
The concept of "auto"-tiered, transparent storage is _really_ compelling! ...and the use of AGPL is really clever as an "enterprise poison pill".
It still sucks that S3 is ~$20/mo/TB basically "in perpetuity", while random SSD drives are ~$80/TB and I'd feel comfortable effectively amortizing them out at ~$20/yr for local storage instead of $20/mo for S3. :-/
Seems to be nfs v3 [0] - curious to test it out - the only userspace nfsv4 implementation I’m aware of is in buildbarn (golang) [1]. The example of their nfs v3 implementation disables locking. Still pretty cool to see all the ways the rust ecosystem is empowering stuff like this.
I’m kinda surprised someone hasn’t integrated the buildbarn nfs v4 stuff into docker/podman - the virtiofs stuff is pretty bad on osx and the buildbarn nfs 4.0 stuff is a big improvement over nfs v3.
Anyhow I digress. Can’t wait to take it for a spin.
Incredibly cool! It shows running Ubuntu and Postgres, and also supports full posix operations.
Questions:
- I see there is a diagram running multiple Postgres nodes backed by this store, very similar to horizontally distributed web-server. Doesn't Postgres use WAL replication? Or is it disabled and they are they running on same "views" of the filesystem?
- What does this mean for services that handle geo-distribution on app layer? e.g. CockroachDB?
I am in no way affiliated with JuiceFS, but I have done a lot of benchmarking and testing of it, and the numbers claimed here for JuiceFS are suspicious (basically 5 ops/second with everything mostly broken).
Seems like a really interesting project! I don't understand what's going on with latency vs durability here. The benchmarks [1] report ~1ms latency for sequential writes, but that's just not possible with S3. So presumably writes are not being confirmed to storage before confirming the write to the client.
What is the durability model? The docs don't talk about intermediate storage. Slatedb does confirm writes to S3 by default, but I assume that's not happening?
I had to laugh out loud:
"In practice, you'll encounter other constraints well before these theoretical limits, such as S3 provider limits, performance considerations with billions of objects, or simply running out of money."
At a glance, ZeroFS appears to be an overlay filesystem above a VFS. It looks like it abstracts the underlying filesystem configuration/complication and creates a simplified, unified configuration that could allow you to be more dynamic underneath.
It also appears to support a great deal of IO and permission complexity, which makes sense as it refers to itself as a filesystem. It also purports to be extremely fast. In the end it looks to function like an actual filesystem with all that entails.
rclone (my first time hearing about it) appears to be a much simpler solution with different design goals. It looks more like rsync in most instances, or a user-space link to a remote filesystem in others (like ftp or smbclient). The mount capability, while similar to zerofs, looks more like something much simpler and targeted for less-sophisticated needs.
Not saying one's better than the other; merely that they really look to be serving different needs/goals while having some overlap.
It looks bizarre to see that Azure Storage support was included. That's because there's no need to use fancy SlateDB+LSM tree code to simulate a block device on Azure Storage, it already natively provides "page blobs" that can be mounted as standard disks and used for boot disks, databases, or whatever. These are used as the backing storage for all Azure virtual machine disks and can also be used directly via the HTTP API. For example, SQL Server can store its database files on page blobs, without having to attach them as disks and formatting them as NTFS volumes.
17 comments
[ 2.9 ms ] story [ 46.5 ms ] threadIncredible performance figures, rocketing to probably the best way to use object storage in an fs like way. There's a whole series of comparisons, & they probably need a logarithmic scale given the scale of the lead slatedb has! https://www.zerofs.net/zerofs-vs-juicefs
Speaks 9p, NFS, or NBD. Some great demos of ZFS with l2arc caches giving a near local performance while having s3 persistence.
Totally what I was thinking of when in the Immich someone mentioned wanting a way to run it on cheap object storage. https://news.ycombinator.com/item?id=45169036
It still sucks that S3 is ~$20/mo/TB basically "in perpetuity", while random SSD drives are ~$80/TB and I'd feel comfortable effectively amortizing them out at ~$20/yr for local storage instead of $20/mo for S3. :-/
Looking at those benchmarks, I think you must be using a local disk to sync writes before uploading to S3?
I’m kinda surprised someone hasn’t integrated the buildbarn nfs v4 stuff into docker/podman - the virtiofs stuff is pretty bad on osx and the buildbarn nfs 4.0 stuff is a big improvement over nfs v3.
Anyhow I digress. Can’t wait to take it for a spin.
[0] https://github.com/Barre/zerofs_nfsserve
[1] https://github.com/buildbarn/bb-remote-execution/tree/master...
Questions:
- I see there is a diagram running multiple Postgres nodes backed by this store, very similar to horizontally distributed web-server. Doesn't Postgres use WAL replication? Or is it disabled and they are they running on same "views" of the filesystem?
- What does this mean for services that handle geo-distribution on app layer? e.g. CockroachDB?
Sorry if this sounds dumb.
I am in no way affiliated with JuiceFS, but I have done a lot of benchmarking and testing of it, and the numbers claimed here for JuiceFS are suspicious (basically 5 ops/second with everything mostly broken).
What is the durability model? The docs don't talk about intermediate storage. Slatedb does confirm writes to S3 by default, but I assume that's not happening?
[1] https://www.zerofs.net/zerofs-vs-juicefs
https://rclone.org/
https://github.com/rclone/rclone
It also appears to support a great deal of IO and permission complexity, which makes sense as it refers to itself as a filesystem. It also purports to be extremely fast. In the end it looks to function like an actual filesystem with all that entails.
rclone (my first time hearing about it) appears to be a much simpler solution with different design goals. It looks more like rsync in most instances, or a user-space link to a remote filesystem in others (like ftp or smbclient). The mount capability, while similar to zerofs, looks more like something much simpler and targeted for less-sophisticated needs.
Not saying one's better than the other; merely that they really look to be serving different needs/goals while having some overlap.
ZeroFS is AGPL with commercial license available.
https://www.zerofs.net/licensing
See: https://learn.microsoft.com/en-us/azure/storage/blobs/storag...
Just don’t understand what’s the point of creating things like this.