HN Explorer

Ask HN: Can anyone recommend a good distributed filesystem?

10 points by someone13 ↗ HN
Hello all,

I'm currently evaluating a bunch of distributed filesystems for a personal project, and I haven't found much information. I was hoping that some of the people here could help out.

I'll be using the filesystem to store millions, (eventually scaling to trillions) of files, comprising up to petabytes of data. These files are binary data (think: executables, music, etc.), and I do not control their content.

A quick list of required features:

1. Reliability. I will not be able to reproduce the files stored after they have originally been saved, so some sort of replication is necessary.

2. Simple interface. Various applications (potentially written in multiple languages) will need access to these files. A common interface (i.e. mounting on a Linux machine, sharing via Samba, or whatever) is necessary.

3. Scalability. Adding a new machine / more storage shouldn't be like pulling teeth.

And a quick summary of what I've looked at so far:

- Ceph: Seems good, but very new, and haven't been able to find anyone using it.

- GlusterFS: Currently my best option - seems to have nearly everything I need. Anyone ever used it?

- Lustre: Has a single Metadata Server, which means a single point of failure.

- XtreemFS: Haven't looked too closely yet.

If anyone has personal experience, or knows someone who does, that'd be appreciated.

Thanks in advance!

8 comments

[ 126 ms ] story [ 2472 ms ] thread
sitkack
Disco has a tag based DDFS which is very awesome, http://discoproject.org/ .

You are asking too much for a drop in DFS solution. You will need to roll something. I would use Riak and Python FUSE. Should only take you a weekend.

I posted some notes here to get you started. http://pastebin.com/raw.php?i=Dn2JTbfd

someone13
... Wow. That's an impressive number of resources - thanks a lot! I owe you a beer or something sometime! I'll probably release my solution as open-source, too, so anyone can use it.
sitkack
I already had that in my notes. They are in no particular order, they just caught my eye at the time for whatever reason.
rckclmbr
We started with GlusterFS, but got off it because we kept having significant downtime and data loss. We've since moved to MooseFS and haven't looked back... very happy with the choice.
someone13
How do you deal with failure of the Metadata server? According to Wikipedia, there's only one per filesystem, and failover or multiple MDSs aren't supported. Had any problems with this?
webcoyote
You might consider MogileFS (developed by LiveJournal, the same company that created memcached) and HDFS (a filesystem used for Hadoop).
true_religion
My problem with MogileFS is that it has too many moving parts.

For what it does, I feel like I could cobble something like it together in a month or so using any distributed NoSQL database to store metadata and a Nginx plugin for Webdav.

true_religion
Point about GlusterFS. I've used it for the past 5 years and found out that its reliable, reasonably well updated, and very flexible.

However, you desperately need a fast network to take advantage of it even in a small sized cluster (N < 10). In a midsized cluster, latency will kill you from all angles unless you have very low throughput.

Lastly, do not rely on their NUMFA translator. If you need sharding then do it yourself. Also striping a large file system with many small files will make it slower rather than faster because of seek locality.

I've managed to hack out a system that works without maintenance now for 1 year plus, but if I had to do it again I'd go in with a lot of testing and an eye to write my own translator or at least highly customize one that they ship with.

Their code is clean. Its a saving grace.