Seeing how they emphasize using XFS for the backing disks [1], I'm pretty sure that the objects/blobs end up as actual files on the disk (same as with OpenStack Swift).
So in the end, an "object/blob" is the same as a "file". The different term emphasizes that the user does not know (or care) which disk actually holds the file, or how many replicas exist. That's abstracted away by the cluster.
But your company might have security requirements that prevent the use of S3.
For example, if you're a defense contractor that handles FOUO information, you can't just throw that on somebody else's cloud. You have to keep it entirely in-house.
Are you serious? It's definitely possible, and almost simple, to build a limited scale service that is cheaper than S3. S3 has the Amazon footprint behind it, which you can't compete against, but the price is not hard to beat.
Yes, but who will be on call to manage, monitor, and fix it when it breaks? Answer: you! S3 is one of those things that's so cheap and works so well I'd never think of leaving it.
If it came down to it, you can lease dx lines from AWS to your datacenter switch and get 10 Gbs to S3 for pretty cheap.
Maybe you decide to move from AWS to own data center to reduce costs? This way the cost of moving is cheaper, since you don't have to modify applications that rely on S3 before moving.
Any functionality or plans for deduplication? For example if I used this for an image host and a handful of users upload the same image the image only gets stored ${replication} times instead of ${replication}*${uploads} times.
I had been using Skylable's sx server for this but unfortunately it looks like they're ceasing operations, I'm in the market for a decent replacement.
Why not hash the image on upload and store them with names based on hash? That way if one copy already exists it will just get over-written on next user upload
Someone would still need to handle life-timing the objects in the presence of deletes. If the storage layer does it then it is one less thing to handle at the application level.
If you upload by hash then you need your own ref counting for deletes (or just don't support delete ... storage is cheap after all!)
Wouldn't any account based system have an indirect reference counter basically for free? Images are associated with accounts so you safely delete an image when there are no accounts that own it.
I don't trust user input and I don't trust hashing output, I definitely don't trust mixing the two!
Serious answer though in that instance it comes down to already having a naming system in place that uses hashids to generate the filename. I could re-write the whole thing but it's not really a project that grasps my attention for long bursts. As it stands I've just moved the whole lot to s3 for now, even with it's larger footprint it's not much more in terms of cost - Mainly I use the image storage project as a way to test new object storage.
As an example off the top of my head I'm thinking block level de-duplication would be good for storing backup tars from multiple machines. In my previous post I was more hinting at file-level dedupe but that probably wouldn't do as much.
I wrote a "microservice" on top of S3 (tested with Minio too) and PostgreSQL that does this, among other things (e.g. image transformations, TUS uploads, web page screenshots). I would consider open sourcing it if there's enough interest.
two questions, first, how does this work with amazon S3, for example, if I use django-storages with support for S3, is it just plug-n-play with minio?
second, deduplication feature is great, for minio and S3 usuage. if the users decided to attach same file to multiple comments (in my case) deduplication would save me a lot.
I would be more interested in your solution as I was also thinking this kind of implementation. But, mine is more related with meta-data and video file thumbnail generating.
I remember evaluating this project several years ago, (maybe 2 or 3?) as a possible S3 replacement, and it actually worked fairly well. We ended up not using it as our requirements changed.
Yosuke Hara the main developer on the project was very helpful. We needed a way for clients that didn't speak the S3 api to access the files, and after talking with him we prototyped a NFS endpoint in a few days that kind of worked, while he worked on getting an actual one added in.
On the note of distributed systems, are there systems that perform well when the a node is not connected? Eg, i'm designing my own "distributed fs" (though my needs are very different), and i'm mainly doing this because i don't see this in many places. Figure it's on topic here.
Eg, i want a single point of contact - my local node (laptop/etc) - which can behave normally whether it is connected to the network, part of the network, or none of the network. The availability of the data at large is obviously dependent on how much of it you're connected to, but that's fine for me.
This gives me the UX of being able to write to the filestore everywhere, rather than losing connection to a self-hosted s3 when i'm out of cell signal range or w/e. Then later, when i'm back and connected to the network, different nodes who are configured to own all portions of data can pull the data as needed.
Are there other distributed offline-able filesystems like this?
The performance of such a system really depends on what you can expect from it.
If your system is distributed without full replication (and no overlap between nodes), then having offline nodes is easy - you just access to your local data, and the rest of the network can't access it.
If you want to support full replication, or even partial, you'll want to detect (and maybe resolve) conflicts, and depending on how long you think a node can be offline, you'll need different mechanisms. If you expect nodes to be offline for really long periods of time, then you'll need to keep a lot of state around to be able to detect conflicts later - therefore making it perform worse (bigger latency while replicating data).
The feature list sounds extremely similar to OpenStack Swift [1]. In fact, the first sentence on there is nearly identical to the submission title: "Swift is a highly available, distributed, eventually consistent object/blob store."
Advantages of LeoFS over Swift seem to be NFS support (I think there were some efforts to expose Swift containers as filesystems, but I didn't see anything becoming stable) and regional replication (which is tedious with Swift).
I quickly skimmed through the docs for LeoFS, and I'm a bit confused: They talk about master nodes and slave nodes, while at the same moment claiming that there is no SPOF. So I would guess that there is some sort of leader election going on. Swift doesn't do anything like that; the cluster shares a ring file that determines where each file goes, and if you have N replicas, there are at most 2N locations where these replicas can be stored (N regular locations and N fallback locations).
Another thing that concerns me is that the documentation implies that the servers need to come up in the correct order (see [2], section "Method of server launch"). One thing that I like about Swift is that individual nodes and even individual services on each node can be operated completely independently, which keeps my Kubernetes setup and Helm charts simple.
> They talk about master nodes and slave nodes, while at the same moment claiming that there is no SPOF.
If I remember right (its been a while), the master and slave nodes mentioned are just in the manager which is control of the ring. Every other component can operate with nodes failing all over the place without failures.
I'm all for options in the storage service space but "FS" signals (to me) that this acts as a (POSIX) file system. Does it expose NFS rpc? If not, what does an S3-like storage service API expose that basic HTTP/1.1 with eg. range request doesn't?
42 comments
[ 2.8 ms ] story [ 106 ms ] threadSo in the end, an "object/blob" is the same as a "file". The different term emphasizes that the user does not know (or care) which disk actually holds the file, or how many replicas exist. That's abstracted away by the cluster.
[1] http://leo-project.net/leofs/docs/installation/install_2.htm...
[1] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...
[2] https://code.facebook.com/posts/685565858139515/needle-in-a-...
For example, if you're a defense contractor that handles FOUO information, you can't just throw that on somebody else's cloud. You have to keep it entirely in-house.
If it came down to it, you can lease dx lines from AWS to your datacenter switch and get 10 Gbs to S3 for pretty cheap.
I had been using Skylable's sx server for this but unfortunately it looks like they're ceasing operations, I'm in the market for a decent replacement.
If you upload by hash then you need your own ref counting for deletes (or just don't support delete ... storage is cheap after all!)
Serious answer though in that instance it comes down to already having a naming system in place that uses hashids to generate the filename. I could re-write the whole thing but it's not really a project that grasps my attention for long bursts. As it stands I've just moved the whole lot to s3 for now, even with it's larger footprint it's not much more in terms of cost - Mainly I use the image storage project as a way to test new object storage.
As an example off the top of my head I'm thinking block level de-duplication would be good for storing backup tars from multiple machines. In my previous post I was more hinting at file-level dedupe but that probably wouldn't do as much.
second, deduplication feature is great, for minio and S3 usuage. if the users decided to attach same file to multiple comments (in my case) deduplication would save me a lot.
* S3 compatibility
* NFS support (this is really neat)
* Proper distribution and fault tolerance built in
I like the benchmark numbers:
https://github.com/leo-project/notes/tree/master/leofs/bench...
Seems to be used in production at Rakuten
Leo is a large, distributed, no-SPOF thing, only makes sense if you have multiple boxes and want high reliability at the cost of some redundancy.
Both happen to talk S3, but that's it.
Eg, i want a single point of contact - my local node (laptop/etc) - which can behave normally whether it is connected to the network, part of the network, or none of the network. The availability of the data at large is obviously dependent on how much of it you're connected to, but that's fine for me.
This gives me the UX of being able to write to the filestore everywhere, rather than losing connection to a self-hosted s3 when i'm out of cell signal range or w/e. Then later, when i'm back and connected to the network, different nodes who are configured to own all portions of data can pull the data as needed.
Are there other distributed offline-able filesystems like this?
If your system is distributed without full replication (and no overlap between nodes), then having offline nodes is easy - you just access to your local data, and the rest of the network can't access it.
If you want to support full replication, or even partial, you'll want to detect (and maybe resolve) conflicts, and depending on how long you think a node can be offline, you'll need different mechanisms. If you expect nodes to be offline for really long periods of time, then you'll need to keep a lot of state around to be able to detect conflicts later - therefore making it perform worse (bigger latency while replicating data).
Advantages of LeoFS over Swift seem to be NFS support (I think there were some efforts to expose Swift containers as filesystems, but I didn't see anything becoming stable) and regional replication (which is tedious with Swift).
I quickly skimmed through the docs for LeoFS, and I'm a bit confused: They talk about master nodes and slave nodes, while at the same moment claiming that there is no SPOF. So I would guess that there is some sort of leader election going on. Swift doesn't do anything like that; the cluster shares a ring file that determines where each file goes, and if you have N replicas, there are at most 2N locations where these replicas can be stored (N regular locations and N fallback locations).
Another thing that concerns me is that the documentation implies that the servers need to come up in the correct order (see [2], section "Method of server launch"). One thing that I like about Swift is that individual nodes and even individual services on each node can be operated completely independently, which keeps my Kubernetes setup and Helm charts simple.
[1] http://docs.openstack.org/developer/swift/
[2] http://leo-project.net/leofs/docs/getting_started/getting_st...
If I remember right (its been a while), the master and slave nodes mentioned are just in the manager which is control of the ring. Every other component can operate with nodes failing all over the place without failures.
But 56% of the code is "Shell code". What's up with that?
[1] http://ceph.com/
[1] https://www.freshports.org/databases/leofs
[2] https://docs.project-fifo.net/docs/leofs-overview
I would position LeoFS as a in-house S3 compatible storage so you can access them quickly within your environment.
One of the strengths of LeoFS is handling small objects (e.g. Images, Logs, ...) you can find more at the benchmark report page.
https://github.com/leo-project/notes/tree/master/leofs/bench...
Over the past few months, we focused on improving the stability and performance of LeoFS.
Please feel free to test it, report any issues you encountered on GitHub, we highly appreciate users' voice.