I started learning Go a month ago and wrote goofys in it as an exercise. Looking for feedbacks from Go best practices as well as usefulness of reduced-POSIX filesystems.
* I am not working at the moment and have some free time (shameless plug: resume at my profile), so I want to bound the amount of time I need to get something useful
* many archiving/backup workloads are WORN (write once read never), that and many streaming workloads (data processing, media streaming) don't really benefit from cache. (Unless your cache is as big as your data, but that's usually not why people use S3)
* for the use cases that cache can help, I think you can just use another layer of caching filesystem. I intend to write one if one doesn't exist already. I wonder if you can use cachefs with fuse filesystems? Let me know your use cases and I will think about it some more.
As far as I know, FS-Cache does not yet work with FUSE, though we (in Gluster) and others have often considered adding such support. You can actually get some caching within FUSE itself, but I think you have to go down to the low level (inode-based) interface to get any control over it.
Yes I looked into it (I only found out about it after I started working on goofys). They have a stub flush() which does nothing (https://github.com/skoobe/riofs/blob/master/src/rfuse.c#L104...), so of course they are faster and the benchmarks won't be meaningful.
I looked at the code some more and they do handle release(), so much of my point above was invalid. I expect riofs's streaming write performance to be comparable to goofys because we both use the same implementation strategy.
They claim S3 is "designed for" 99.999999999% durability, but there's neither a guarantee nor a clear definition of when an object starts to be covered by one. While it's both intuitively obvious and conventional wisdom that an object is durable at the end of a PUT, as far as I've ever been able to tell Amazon doesn't come out and say so.
That's one of the problems with company-defined "standards" BTW. This kind of issue would surely have been noticed and discussed in any kind of open standards process. It's what makes those processes so tedious. De facto standards can be turned around a lot faster, but there's a necessary sacrifice in precision to go with that.
Not surprising that s3fs is slower, it's implementation quality is not very high. Goofys looks much better on first sight. It would be nice if anybody could do a benchmark between goofys and riofs (without cache). But honestly, if you have some proper request handling there is not that much to tune. The biggest performance gains can be achieved from a good cache implementation and make such a system useful in a production environment, that's what riofs was written for.
first off apologies for the criticism about flush(), I only skimmed the code and flush() is usually where I look for first.
Anyways goofys at this point is just a toy project. There are more optimizations that I can potentially do (proper read prefetching) it's mostly good enough.
I see what you are doing there :-P goroutines do make certain things easier but there's nothing you can do with good old pthreads. One of the motivations for this project is for me to understand better what this Go hype is about.
I applaud the "Filey System" clarification. Confusion between things that are very close to 100% POSIX compliance and things that don't even have that as a goal but adopt similar interfaces or structures has been a real problem. It's nice to see someone being up front about the distinction, and it looks like a useful project too.
Thank you! It's not a general purpose filesystem but I don't think that has to be the case to be useful. I am lazy and I want to do as little POSIX as possible.
That's wise. There are parts of POSIX that even I (a notorious purist) think are outdated, unclear, or just not worth the trouble. Appending and random writes sure would be nice, but I appreciate the difficulties involved. Maybe if I get a chance to learn more Go I'll be able to contribute in some of those areas.
I appreciate any project that makes its strengths and weaknesses very clear and documents its consistency semantics.
Reading around[1], it looks like close-to-open consistency means that all writes must be flushed before the close() call returns, and all subsequent open() calls to the file must see the changes. IO may otherwise be buffered or cached by the FS.
23 comments
[ 3.5 ms ] story [ 66.9 ms ] thread* I am not working at the moment and have some free time (shameless plug: resume at my profile), so I want to bound the amount of time I need to get something useful
* many archiving/backup workloads are WORN (write once read never), that and many streaming workloads (data processing, media streaming) don't really benefit from cache. (Unless your cache is as big as your data, but that's usually not why people use S3)
* for the use cases that cache can help, I think you can just use another layer of caching filesystem. I intend to write one if one doesn't exist already. I wonder if you can use cachefs with fuse filesystems? Let me know your use cases and I will think about it some more.
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT...
Nope. Let's try the developer guide.
http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class...
Nope again. Let's try the FAQ.
https://aws.amazon.com/s3/faqs/
They claim S3 is "designed for" 99.999999999% durability, but there's neither a guarantee nor a clear definition of when an object starts to be covered by one. While it's both intuitively obvious and conventional wisdom that an object is durable at the end of a PUT, as far as I've ever been able to tell Amazon doesn't come out and say so.
That's one of the problems with company-defined "standards" BTW. This kind of issue would surely have been noticed and discussed in any kind of open standards process. It's what makes those processes so tedious. De facto standards can be turned around a lot faster, but there's a necessary sacrifice in precision to go with that.
Disclaimer: I initiated and supervised riofs.
Anyways goofys at this point is just a toy project. There are more optimizations that I can potentially do (proper read prefetching) it's mostly good enough.
Reading around[1], it looks like close-to-open consistency means that all writes must be flushed before the close() call returns, and all subsequent open() calls to the file must see the changes. IO may otherwise be buffered or cached by the FS.
1] Best source I found were these lecture slides: http://www0.cs.ucl.ac.uk/staff/B.Karp/gz03/f2011/lectures/gz...