Apart from Minio, we tried Garage and Ceph. I think there's definitely a need for something that interfaces using S3 API but is just a simple file system underneath, for local, testing and small scale deployments. Not sure that exists? Of course a lot of stuff is being bolted onto S3 and it's not as simple as it initially claimed to be.
I'm the author of another option (https://github.com/mickael-kerjean/filestash) which has a S3 gateway that expose itself as a S3 server but is just a proxy that forward your S3 call onto anything else like SFTP, local FS, FTP, NFS, SMB, IPFS, Sharepoint, Azure, git repo, Dropbox, Google Drive, another S3, ... it's entirely stateless and act as a proxy translating S3 call onto whatever you have connected in the other end
Hi, I really like how the software is setup, but there is almost no documentation. This makes it hard to setup. I want to run it with the local storage plugin, but it keeps using / as the storage location and I want it to use /data which I volume mount. Maybe think about improving the documentation :)
rustfs have promise, supports a lot of features, even allows to bring your own secret/access keys (if you want to migrate without changing creds on clients) but it's very much still in-development; and they have already prepared for bait-and-switch in code ( https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens... )
Ceph is closest feature wise to actual S3 feature-set wise but it's a lot to setup. It pretty much wants few local servers, you can replicate to another site but each site on its own is pretty latency sensitive between storage servers. It also offers many other features aside, as S3 is just built on top of their object store that can be also used for VM storage or even FUSE-compatible FS
Garage is great but it is very much "just to store stuff", it lacks features on both S3 side (S3 have a bunch of advanced ACLs many of the alternatives don't support, and stuff for HTTP headers too) and management side (stuff like "allow access key to access only certain path on the bucket is impossible for example). Also the clustering feature is very WAN-aware, unlike ceph where you pretty much have to have all your storage servers in same rack if you want a single site to have replication.
I didn't find an alternative that I liked as much as MinIO and I, unfortunately, ended up creating a my own. It includes just the most basic features and cannot be compared to the larger projects, but is simple and it is efficient.
I just bit the bullet last week and figured we are going to migrate our self hosted minio servers to ceph instead. So far 3 server ceph cluster has been setup with cephadm and last minio server is currently mirroring its ~120TB buckets to new cluster with a whopping 420MB/s - should finish any day now. The complexity of ceph and it's cluster nature of course if a bit scary at first compared to minio - a single Go binary with minimal configuration, but after learning the basics it should be smooth sailing. What's neat is that ceph allows expanding clusters, just throw more storage servers at it, in theory at least, not sure where the ceiling is for that yet. Shame minio went that way, it had a really neat console before they cut it out. I also contemplated le garage, but it seem elasticsearch is not happy with that S3 solution for snapshots, so ceph it is.
It's complex, but Ceph's storage and consensus layer is battle-tested and a much more solid foundation for serious use. Just make sure that your nodes don't run full!
I think the landscape has changed with those hyperscalers outcompeting open-source projects with alternative profit avenues for the money available in the market.
From my experience, Ceph works well, but requires a lot more hardware and dedicated cluster monitoring versus something like more simple like Minio; in my eyes, they have a somewhat different target audience. I can throw Minio into some customer environments as a convenient add-on, which I don't think I could do with Ceph.
Hopefully one of the open-source alternatives to Minio will step in and fill that "lighter" object storage gap.
> For anyone evaluating infrastructure dependencies right now: the license matters, but the funding model matters more. Single-vendor open source projects backed by VC are essentially on a countdown timer. Either they find a sustainable model that doesn't require closing the source, or they eventually pull the rug.
I struggle to even find example of VC-backed OSS that didn't go "ok closing down time". Only ones I remember (like Gitlab) started with open core model, not fully OSS
This is the newer generations re-discovering why various flavours of Shareware and trial demos existed since the 1980's, even though sharing code under various licenses is almost as old as computing.
Redis is the odd one out here[1]: Garantia Data, later known as Redis Labs, now known as Redis, did not create Redis, nor did it maintain Redis for most of its rise to popularity (2009–2015) nor did it employ Redis’s creator and then-maintainer 'antirez at that time. (He objected; they hired him; some years later he left; then he returned. He is apparently OK with how things ended up.) What the company did do is develop OSS Redis addons, then pull the rug on them while saying that Redis proper would “always remain BSD”[2], then prove that that was a lie too[3]. As well as do various other shady (if legal) stuff with the trademarks[4] and credits[5] too.
What is especially annoying is the aistor/minio business model, either get the „free“ version or pay about 100k… How about accepting some small dollars and keeping the core concept?
However this seems to be the business type of enshitification. Instead of slapping everything with ads, you either pay ridiculous dollars or move on.
In French the adjective follows the name so AI is actually IA.
On AWS S3, you have a storage level called "Infrequent Access", shortened IA everywhere.
A few weeks ago I had to spend way too much time explaining to a customer that, no, we weren't planning on feeding their data to an AI when, on my reports, I was talking about relying on S3 IA to reduce costs...
I ran a moderately large opensource service and my chronic back pain was cured the day I stopped maintaining the project.
Working for free is not fun. Having a paid offering with a free community version is not fun. Ultimately, dealing with people who don't pay for your product is not fun. I learnt this the hard way and I guess the MinIO team learnt this as well.
There's nothing wrong at all with charging for your product. What I do take issue with, however, is convincing everyone that your product is FOSS, waiting until people undertake a lot of work to integrate your product into their infrastructure, and then doing a bait-and-switch.
Just be honest since the start that your product will eventually abandon its FOSS licence. Then people can make an informed decision. Or, if you haven't done that, do the right thing and continue to stand by what you originally promised.
> Ultimately, dealing with people who don't pay for your product is not fun.
I find it the other way around. I feel a bit embarrassed and stressed out working with people who have paid for a copy of software I've made (which admittedly is rather rare). When they haven't paid, every exchange is about what's best for humanity and the public in general, i.e. they're not supposed to get some special treatment at the expense of anyone else, and nobody has a right to lord over the other party.
Completely different situations. None of the MinIO team worked for free. MinIO is a COSS company (commercial open source software). They give a basic version of it away for free hoping that some people, usually at companies, will want to pay for the premium features. MinIO going closed source is a business decision and there is nothing wrong with that.
I highly recommend SeaweedFS. I used it in production for a long time before partnering with Wasabi. We still have SeaweedFS for a scorching hot, 1GiB/s colocated object storage, but Wasabi is our bread and butter object storage now.
Not seeing any one else comment about it, but I would caution against relying on Wasabi primarily. They actively and silently corrupted a lot of my data and still billed me for it. You'll just start seeing random 500s when trying to get data down from your bucket and it's just gone, no recovery, but it still counts as stored data so you're still paying for it.
I don’t feel that way at all. I’ve been maintaining open source storage systems for few years. I love it. Absolutely love it. I maintain TidesDB it’s a storage engine. I also have back pain but that doesn’t mean you can’t do what you love.
Maybe open source developers should stop imagining the things they choose to give away for free as "products". I maintain a small open source library. It doesn't make any money, it will never make any money, people are free to use or not as they choose. If someone doesn't like the way I maintain the repository they are free to fork it.
It's remarkable how many people wrongly assume that open source projects can't be monetized. Business models and open source are orthogonal but compatible concepts. However, if your primary goal while maintaining an open source project is profiting financially from it, your incentives are skewed. If you feel this way, you should also stop using any open source projects, unless you financially support them as well.
I've been involved with free software for coming on 30 years, have maintained several reasonably popular free software projects and have 100% enjoyed it every time. Developing relationships with the community members and working with them toward a common goal is very rewarding. Not much more to say about this as these are subjective interpretations of our experiences and the experiences could be very different. But it definitely can be fun.
We all saw that coming. For quite some time they have been all but transparent or open, vigorously removing even mild criticism towards any decisions they were making from github with no further explanation, locking comments, etc. No one that's been following the development and has been somewhat reliant on min.io is surprised. Personally the moment I saw the "maintenance" mode, I rushed to switch to garage. I have a few features I need to pack in a PR ready but I haven't had time to get to that. I should probably prioritize that.
Why should these guys bother with people who won't pay for their offering ? The community is not skilled enough to contribute to this type of project. Honestly most serious open source is industry backed and solves very challenging distributed systems problems. A run of the mill web dev doesnt know these things I am sorry to say.
No one should be immune from criticism. If you make a well established open source project WITH the help of thousands of volunteers around the world only to lock it up and say "pay up", that's called extortion.
This is timely news for me - I was just standing up some Loki infrastructure yesterday & following Grafana's own guides on object storage (they recommend minio for non-cloud setups). I wasn't previously experienced with minio & would have completely missed the maintenance status if it wasn't for Checkov nagging me about using latest tags for images & having to go searching for release versions.
Sofar I've switched to Rustfs which seems like a very nice project, though <24hrs is hardly an evaluation period.
Tangentially related, since we are on the subject of Minio. Minio has or rather had an option to work as an FTP server! That is kind of neat because CCTV cameras have an option to upload a picture of motion detected to an FTP server and that being a distributed minio cluster really was a neat option, since you could then generate an event of a file uploaded, kick off a pipeline job or whatever. Currently instead of that I use vsftpd and inotify to detect file uploads but that is such a major pain in the ass operate, it would be really great to find another FTP to S3 gateway.
I've moved my SaaS I'm developing to SeaweedFS, it was rather painless to do it. I should also move away from minio-go SDK to just use the generic AWS one, one day. No hard feelings from my side to MinIO team though.
If you are struggling with observability solutions which require object storage for production setups after such news (i.e. Thanos, Loki, Mimir, Tempo), then try alternatives without this requirement, such as VictoriaMetrics, VictoriaLogs and VictoriaTraces. They scale to petabytes of data on regular block storage, and they provide higher performance and availability than systems, which depend on manually managed object storage such as MinIO.
AGPL is dead as a copy-left measure. LLMs do not understand, and would not care anyway, about regurgitating code that you have published to the internet.
COSS companies want it both ways. Free community contributions and bug reports during the growth phase. Then closed source once they've captured enough users. The code you run today belongs to you. The roadmap belongs to their investors.
This has been on the cards for at least a year, with the increasingly doomy commits noted by HN.
Unfortunately I don't know of any other open projects that can obviously scale to the same degree. I built up around 100PiB of storage under minio with a former employer. It's very robust in the face of drive & server failure, is simple to manage on bare hardware with ansible. We got 180Gbps sustained writes out of it, with some part time hardware maintenance.
Don't know if there's an opportunity here for larger users of minio to band together and fund some continued maintenance?
I definitely had a wishlist and some hardware management scripts around it that could be integrated into it.
90 comments
[ 2.6 ms ] story [ 128 ms ] threadIt was pretty clear they pivoted to their closed source repo back then.
Other alternatives:
https://github.com/deuxfleurs-org/garage
https://github.com/rustfs/rustfs
https://github.com/seaweedfs/seaweedfs
https://github.com/supabase/storage
https://github.com/scality/cloudserver
https://github.com/ceph/ceph
Among others
https://github.com/minio/minio/pull/21746
rustfs have promise, supports a lot of features, even allows to bring your own secret/access keys (if you want to migrate without changing creds on clients) but it's very much still in-development; and they have already prepared for bait-and-switch in code ( https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens... )
Ceph is closest feature wise to actual S3 feature-set wise but it's a lot to setup. It pretty much wants few local servers, you can replicate to another site but each site on its own is pretty latency sensitive between storage servers. It also offers many other features aside, as S3 is just built on top of their object store that can be also used for VM storage or even FUSE-compatible FS
Garage is great but it is very much "just to store stuff", it lacks features on both S3 side (S3 have a bunch of advanced ACLs many of the alternatives don't support, and stuff for HTTP headers too) and management side (stuff like "allow access key to access only certain path on the bucket is impossible for example). Also the clustering feature is very WAN-aware, unlike ceph where you pretty much have to have all your storage servers in same rack if you want a single site to have replication.
https://github.com/espebra/stupid-simple-s3
Just download the single binary, for most platforms, and run "weed mini -dir=your_data_directory", with all the configuration optimized.
From my experience, Ceph works well, but requires a lot more hardware and dedicated cluster monitoring versus something like more simple like Minio; in my eyes, they have a somewhat different target audience. I can throw Minio into some customer environments as a convenient add-on, which I don't think I could do with Ceph.
Hopefully one of the open-source alternatives to Minio will step in and fill that "lighter" object storage gap.
I struggle to even find example of VC-backed OSS that didn't go "ok closing down time". Only ones I remember (like Gitlab) started with open core model, not fully OSS
Redis is the odd one out here[1]: Garantia Data, later known as Redis Labs, now known as Redis, did not create Redis, nor did it maintain Redis for most of its rise to popularity (2009–2015) nor did it employ Redis’s creator and then-maintainer 'antirez at that time. (He objected; they hired him; some years later he left; then he returned. He is apparently OK with how things ended up.) What the company did do is develop OSS Redis addons, then pull the rug on them while saying that Redis proper would “always remain BSD”[2], then prove that that was a lie too[3]. As well as do various other shady (if legal) stuff with the trademarks[4] and credits[5] too.
[1] https://www.gomomento.com/blog/rip-redis-how-garantia-data-p...
[2] https://redis.io/blog/redis-license-bsd-will-remain-bsd/
[3] https://lwn.net/Articles/966133/
[4] https://github.com/redis-rs/redis-rs/issues/1419
[5] https://github.com/valkey-io/valkey/issues/544
On AWS S3, you have a storage level called "Infrequent Access", shortened IA everywhere.
A few weeks ago I had to spend way too much time explaining to a customer that, no, we weren't planning on feeding their data to an AI when, on my reports, I was talking about relying on S3 IA to reduce costs...
Working for free is not fun. Having a paid offering with a free community version is not fun. Ultimately, dealing with people who don't pay for your product is not fun. I learnt this the hard way and I guess the MinIO team learnt this as well.
Just be honest since the start that your product will eventually abandon its FOSS licence. Then people can make an informed decision. Or, if you haven't done that, do the right thing and continue to stand by what you originally promised.
I find it the other way around. I feel a bit embarrassed and stressed out working with people who have paid for a copy of software I've made (which admittedly is rather rare). When they haven't paid, every exchange is about what's best for humanity and the public in general, i.e. they're not supposed to get some special treatment at the expense of anyone else, and nobody has a right to lord over the other party.
I highly recommend SeaweedFS. I used it in production for a long time before partnering with Wasabi. We still have SeaweedFS for a scorching hot, 1GiB/s colocated object storage, but Wasabi is our bread and butter object storage now.
Good luck with the back pain.
I loved everyone in the community though. By heart. You were the best.
Open source can be very fun if you genuinely enjoy it.
The problem is dealing with people that have wrong expectations, those need to be ignored.
Why were you doing it then?
Sofar I've switched to Rustfs which seems like a very nice project, though <24hrs is hardly an evaluation period.
I've used minio a lot but only as part of my local dev pipeline to emulate s3, and never paid.
They could just archive it there and then, at least it would be honest. What a bunch of clowns.
Unfortunately I don't know of any other open projects that can obviously scale to the same degree. I built up around 100PiB of storage under minio with a former employer. It's very robust in the face of drive & server failure, is simple to manage on bare hardware with ansible. We got 180Gbps sustained writes out of it, with some part time hardware maintenance.
Don't know if there's an opportunity here for larger users of minio to band together and fund some continued maintenance?
I definitely had a wishlist and some hardware management scripts around it that could be integrated into it.