I’m Elvin from the RustFS team in the U.S. Thanks for sharing the benchmark; it’s helpful to see how RustFS performs in real-world setups.
We know trust matters, especially for a newer project, and we try to earn it through transparency and external validation. we were excited to see RustFS recently added as an optional service in Laravel Sail’s official Docker environment (PR #822). Having our implementation reviewed and accepted by a major ecosystem like Laravel was an encouraging milestone for us.
If the “non-technical reasons” you mentioned are around licensing or governance, I’m happy to discuss our long-term Apache 2.0 commitment and path to a stable GA.
I love garage. I think it has applications beyond the standard self host s3 alternative.
It's a really cool system for hyper converged architecture where storage requests can pull data from the local machine and only hit the network when needed.
Looks interesting for something like local development. I don't intend to run production object storage myself, but some of the stuff in the guide to the production setup (https://garagehq.deuxfleurs.fr/documentation/cookbook/real-w...) would scare me a bit:
> For the metadata storage, Garage does not do checksumming and integrity verification on its own, so it is better to use a robust filesystem such as BTRFS or ZFS. Users have reported that when using the LMDB database engine (the default), database files have a tendency of becoming corrupted after an unclean shutdown (e.g. a power outage), so you should take regular snapshots to be able to recover from such a situation.
It seems like you can also use SQLite, but a default database that isn't robust against power failure or crashes seems suprising to me.
Depending on the underlying storage being reliable is far from unique to garage. This is what most other services do too, unless we're talking about something like Ceph which manages the physical storage itself.
Standard filesystems such as ext4 and xfs don't have data checksumming, so you'll have to rely on another layer to provide integrity. Regardless, that's not garage's job imo. It's good that they're keeping their design simple and focus their resources on implementing the S3 spec.
One really useful usecase for Garage for me has been data engineering scripts. I can just use the S3 integration that every tool has to dump to garage and then I can more easily scale up to cloud later.
Garage looks really nice: I've evaluated it with test code and benchmarks and it looks like a winner. Also, very straightforward deployment (self contained executable) and good docs.
But no tags on objects is a pretty big gap, and I had to shelve it. If Garage folk see this: please think on this. You obviously have the talent to make a killer application, but tags are table stakes in the "cloud" API world.
I use Syncthing a lot. Is Garage only really useful if you specifically want to expose an S3 drop in compatible API, or does it also provide other benefits over syncthing?
I tried it recently. Uploaded around 300 documents (1GB) and then went to delete them. Maybe my client was buggy, because the S3 service inside the container crashed and couldn't recover - I had to restart it. It's a really cool project, but I wouldn't really call it "reliable" from my experience.
How is garage for a simple local dev env ?
I recently used seaweedfs since they have a super simple minimal setup compared to garage which seemed to require a config file just to get started.
We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in comparison to our existing use of MinIO, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s, over 10 nodes)
I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.
---
In addition:
Next time we come to this we are going to look at RustFS [1], as well as Ceph/Rook [2].
We can see we're going to have to move away from MinIO in the foreseeable future. My hope is that the alternatives get a boost of interest given the direction MinIO is now taking.
I wouldn’t use rook if you solely want S3. It is a massively complex system which you really need to invest in understanding or else your cluster will croak at some point and you will have no idea on how to fix it.
They explicitly say that top performance is not a goal: «high performances constrain a lot the design and the infrastructure; we seek performances through minimalism only» (https://garagehq.deuxfleurs.fr/documentation/design/goals/)
But it might be interesting to see where the time is spent. I suspect they may be doing fewer things in parallel than MinIO, but maybe it's something entirely different.
> We can see we're going to have to move away from MinIO in the foreseeable future.
My favorite thing about all of this is that I had just invested a ton of time in understanding MinIO and its Kubernetes operator and got everything into a state that I felt good about. I was nearly ready to deploy it to production when the announcement was released that they would not be supporting it.
I’m somewhat surprised that no one is forking it (or I haven’t heard about any organizations of consequence stepping up anyway) instead of all of these projects to rebuild it from scratch.
You can also just pay for MinIO and treat it like any other commercial dependency, with support and clearer expectations around licensing and roadmap, but forks are a different story: unless there’s a well-funded company or solid consortium behind them, you’re mostly just trading one risk for another.
Somewhat unrelated, but I just looked at the RustFS docs intro[1] after seeing it here. It has this statement:
> RustFS is a high-performance, distributed object storage software developed using Rust, the world's most popular memory-safe language.
I’m actually something of a Rust booster, and have used it professionally more than once (including working on a primarily Rust codebase for a while). But it’s hard to take a project’s docs seriously when it describes Rust as “the world’s most popular memory-safe language”. Java, JavaScript, Python, even C# - these all blow it out of the water in popularity and are unambiguously memory safe. I’ve had a lot more segfaults in Rust dependencies than I have in Java dependencies (though both are minuscule in comparison to e.g. C++ dependencies).
I agree that it is a bad idea to describe rust this way but they likely meant memory safety as used in https://www.ralfj.de/blog/2025/07/24/memory-safety.html . Meaning that shared mutable is thread unsafe, I am unsure about Java and JavaScript but I think that almost every language on the popular memory safe list fails this test.
Again the statement is probably still untrue and bad marketing, but I suspect this kind of reasoning was behind it
Of course Rust technically fails too since `unsafe` is a language feature
I don't have an issue with `unsafe` - Java has the mythical unsafe object, C# has it's own unsafe keyword, Python has ffi, etc. The title of that blog post - that there is no memory safety without thread safety - is not quite true and it acknowledges how Java, C#, and Go have strong memory safety while not forbidding races. Even the "break the language" framing seems like special pleading; I'd argue that Java permitting reading back a sheared long (64-bit) integer due to a data race does not break the language the same way writing to a totally unintended memory area or smashing the stack does, and that this distinction is useful. Java data races that cause actual exploitable vulnerabilities are very, very rare.
Thanks for the reality check on our documentation. We realize that some of our phrasing sounded more like marketing hype than a technical spec. That wasn’t our intent, and we are currently refining our docs to be more precise and transparent.
A few points to clarify where we’re coming from:
1. The Technical Bet on Rust: Rust wasn’t a buzzword choice for us. We started this project two years ago with the belief that the concurrency and performance demands of modern storage—especially for AI-driven workloads—benefit from a foundation with predictable memory behavior, zero-cost abstractions, and no garbage collector. These properties matter when you care about determinism and tail latency.
2. Language Safety vs. System Design: We’re under no illusion that using a memory-safe language automatically makes a system “100% secure.” Rust gives us strong safety primitives, but the harder problems are still in distributed systems design, failure handling, and correctness under load. That’s where most of our engineering effort is focused.
3. Giving Back to the Ecosystem: We’re committed to the ecosystem we build on. RustFS is a sponsor of the Rust Foundation, and as we move toward a global, Apache 2.0 open-source model, we intend to contribute back in more concrete ways over time.
We know there’s still work to do on the polish side, and we genuinely appreciate the feedback. If you have specific questions about our implementation details or the S3 compatibility layer, I’m happy to dive into the technical details.
Those rates are peanuts considering that a decade ago saturating 40G, per core, was more than reasonable via standard userspace networking, with atleast a few copies in the datapath.
Having just finished a "hobby size" setup of Rook-Ceph on 3 n100 mini pcs, with every service to fit in a couple hundred MB of ram (one service needs up to 3Gb when starting, but then runs around 250MB) I'd ask why not ceph?
At work I'm typically a consumer of such services from large cloud providers. I read in few places how "difficult" it is, how you need "4GB minimum RAM for most services" and how "friends do not let friends run Ceph below 10Gb".
But this setup runs on a non dedicated 2.5Gb interface (there is VLAN segmentation and careful QoSing).
My benchmarks show I'm primarily network latency and bandwidth limited. By the very definition you can't get better than that.
There were many factors why I chose Ceph and not Garage, Seaweed or MinIo. (One of the biggest is that ceph does 2 birds with one stone for me - block and object).
Thanks for flagging this and for taking the time to point out the broken links. We open-sourced RustFS only a few months ago, and while we’ve been heavily focused on getting the core system to GA, that has admittedly created some documentation debt.
We’re actively reviewing the docs and cleaning up any 404s or navigation issues we can find. For the specific 404 you mentioned, we haven’t been able to reproduce it on our end so far, but we’re continuing to investigate in case it’s environment- or cache-related.
On the licensing side, we want to be clear that we’re fully committed to Apache 2.0 for the long term.
I use this for booting up S3-compatible buckets for local development and testing -- paired up with s5cmd, I can seed 15GB and over 60,000 items (seed/mock data) in < 60s... have a perfect replica of a staging environment with Docker containers (api, db, cache, objects) all up in less than 2mins.
Super simple to set up for my case and been working great.
Previously I used LocalStack S3 but ultimately didn't like the lack of persistance thats not available on the OSS verison. MinIO OSS is apparently no longer maintained? Also looked at SeaweedFS and RustFS but from a quick reading into them this once was the easiest to set up.
Anyone know if it's possible to bandwidth-limit the sync operations? I'd love to set up garage instances across my families' houses to act as a distributed backup, but I don't want to hose their (or my) down/uplink during awake hours. Having redundant selfhosted S3like storage would solve many problems for me, but I really need that capability.
55 comments
[ 1.7 ms ] story [ 112 ms ] threadhttps://www.repoflow.io/blog/benchmarking-self-hosted-s3-com... was useful.
RustFS also looks interesting but for entirely non-technical reasons we had to exclude it.
Anyone have any advice for swapping this in for Minio?
Why skipping SeaweedFS? It rank #1 on all benchmarks, and has a lot of features.
We know trust matters, especially for a newer project, and we try to earn it through transparency and external validation. we were excited to see RustFS recently added as an optional service in Laravel Sail’s official Docker environment (PR #822). Having our implementation reviewed and accepted by a major ecosystem like Laravel was an encouraging milestone for us.
If the “non-technical reasons” you mentioned are around licensing or governance, I’m happy to discuss our long-term Apache 2.0 commitment and path to a stable GA.
It's a really cool system for hyper converged architecture where storage requests can pull data from the local machine and only hit the network when needed.
> For the metadata storage, Garage does not do checksumming and integrity verification on its own, so it is better to use a robust filesystem such as BTRFS or ZFS. Users have reported that when using the LMDB database engine (the default), database files have a tendency of becoming corrupted after an unclean shutdown (e.g. a power outage), so you should take regular snapshots to be able to recover from such a situation.
It seems like you can also use SQLite, but a default database that isn't robust against power failure or crashes seems suprising to me.
Standard filesystems such as ext4 and xfs don't have data checksumming, so you'll have to rely on another layer to provide integrity. Regardless, that's not garage's job imo. It's good that they're keeping their design simple and focus their resources on implementing the S3 spec.
LMDB mode also runs with flush/syncing disabled
this is the reliability question no?
[0] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1052
[1] https://github.com/Barre/ZeroFS
Garage looks really nice: I've evaluated it with test code and benchmarks and it looks like a winner. Also, very straightforward deployment (self contained executable) and good docs.
But no tags on objects is a pretty big gap, and I had to shelve it. If Garage folk see this: please think on this. You obviously have the talent to make a killer application, but tags are table stakes in the "cloud" API world.
We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in comparison to our existing use of MinIO, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s, over 10 nodes)
I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.
---
In addition:
Next time we come to this we are going to look at RustFS [1], as well as Ceph/Rook [2].
We can see we're going to have to move away from MinIO in the foreseeable future. My hope is that the alternatives get a boost of interest given the direction MinIO is now taking.
[0]: https://news.ycombinator.com/item?id=46140342
[1]: https://rustfs.com/
[2]: https://rook.io/
But it might be interesting to see where the time is spent. I suspect they may be doing fewer things in parallel than MinIO, but maybe it's something entirely different.
I wouldn't be surprised if this will be fixed sometime in the future.
My favorite thing about all of this is that I had just invested a ton of time in understanding MinIO and its Kubernetes operator and got everything into a state that I felt good about. I was nearly ready to deploy it to production when the announcement was released that they would not be supporting it.
I’m somewhat surprised that no one is forking it (or I haven’t heard about any organizations of consequence stepping up anyway) instead of all of these projects to rebuild it from scratch.
> RustFS is a high-performance, distributed object storage software developed using Rust, the world's most popular memory-safe language.
I’m actually something of a Rust booster, and have used it professionally more than once (including working on a primarily Rust codebase for a while). But it’s hard to take a project’s docs seriously when it describes Rust as “the world’s most popular memory-safe language”. Java, JavaScript, Python, even C# - these all blow it out of the water in popularity and are unambiguously memory safe. I’ve had a lot more segfaults in Rust dependencies than I have in Java dependencies (though both are minuscule in comparison to e.g. C++ dependencies).
[1]: https://docs.rustfs.com/installation/
Again the statement is probably still untrue and bad marketing, but I suspect this kind of reasoning was behind it
Of course Rust technically fails too since `unsafe` is a language feature
Thanks for the reality check on our documentation. We realize that some of our phrasing sounded more like marketing hype than a technical spec. That wasn’t our intent, and we are currently refining our docs to be more precise and transparent.
A few points to clarify where we’re coming from: 1. The Technical Bet on Rust: Rust wasn’t a buzzword choice for us. We started this project two years ago with the belief that the concurrency and performance demands of modern storage—especially for AI-driven workloads—benefit from a foundation with predictable memory behavior, zero-cost abstractions, and no garbage collector. These properties matter when you care about determinism and tail latency. 2. Language Safety vs. System Design: We’re under no illusion that using a memory-safe language automatically makes a system “100% secure.” Rust gives us strong safety primitives, but the harder problems are still in distributed systems design, failure handling, and correctness under load. That’s where most of our engineering effort is focused. 3. Giving Back to the Ecosystem: We’re committed to the ecosystem we build on. RustFS is a sponsor of the Rust Foundation, and as we move toward a global, Apache 2.0 open-source model, we intend to contribute back in more concrete ways over time.
We know there’s still work to do on the polish side, and we genuinely appreciate the feedback. If you have specific questions about our implementation details or the S3 compatibility layer, I’m happy to dive into the technical details.
At work I'm typically a consumer of such services from large cloud providers. I read in few places how "difficult" it is, how you need "4GB minimum RAM for most services" and how "friends do not let friends run Ceph below 10Gb".
But this setup runs on a non dedicated 2.5Gb interface (there is VLAN segmentation and careful QoSing).
My benchmarks show I'm primarily network latency and bandwidth limited. By the very definition you can't get better than that.
There were many factors why I chose Ceph and not Garage, Seaweed or MinIo. (One of the biggest is that ceph does 2 birds with one stone for me - block and object).
And also it is already rigged for a rug-pull
https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens...
We’re actively reviewing the docs and cleaning up any 404s or navigation issues we can find. For the specific 404 you mentioned, we haven’t been able to reproduce it on our end so far, but we’re continuing to investigate in case it’s environment- or cache-related.
On the licensing side, we want to be clear that we’re fully committed to Apache 2.0 for the long term.
Previously I used LocalStack S3 but ultimately didn't like the lack of persistance thats not available on the OSS verison. MinIO OSS is apparently no longer maintained? Also looked at SeaweedFS and RustFS but from a quick reading into them this once was the easiest to set up.
Does anyone know a good open source S3 alternarive that's easily extendable with custom storage backends?
For example, AWS offers IA and Glacier in addition to the defaults.
This is used for ransomware resistant backups.