Filestore is basically a single google-managed instance running NFS. It's much less sophisticated and resilient than EFS, but also can be much faster since it doesn't have the overhead of a distributed system. Don't try running something that touches lots of files like mercurial on EFS, trust me...
The article says Elastifile (which I know nothing about) will become part of filestore.
EFS is... surprisingly cheap, as long as you don’t need heavy sustained IO for more than a few hours at a time. Great for a simple file store and some big data purposes.
It will be interesting to see how well this scales. I have seen AWS EFS fall over under heavy read load (lots of reads for the same TB-scale data). A lot of poor souls using Horovod with no data team use EFS to “distribute” data in data-parallel SGD.
EFS generally only falls down when you hit your IOPs limit for the measurement period -have seen this soooo many times. Buy more storage/IOPS, and monitor that puppy.
IOPs is one of the main pitfalls of cloud platforms and I'm baffled that neither GCP nor AWS make it easier to detect that you are hitting a limit there.
I've had multiple gigs where I was tasked with solving performance pitfalls, and most of the time it was as simple as upping the IOPS. In one case even though the problem was resolved in the end, the client migrated away from the cloud again because of that bad experience.
For TB-scale heavy read loads ObjectiveFS[0] is an alternative to EFS that don't have EFS's IO/bandwidth limits. It is often used for distributing data for machine learning and large scale simulations on EC2 and GCE.
Looks like Elastifile allows users to "mount" cloud buckets to their servers as NFS. This allows them to run old desktop workflows without the need to migrate these workflows to the cloud.
The upside is that Google Cloud is getting serious about the enterprise because most of them still use desktop applications.
The question is how long this will last since the future is the cloud.
17 comments
[ 0.28 ms ] story [ 49.6 ms ] threadThe article says Elastifile (which I know nothing about) will become part of filestore.
Eliminating team coordination tips the balance.
I've had multiple gigs where I was tasked with solving performance pitfalls, and most of the time it was as simple as upping the IOPS. In one case even though the problem was resolved in the end, the client migrated away from the cloud again because of that bad experience.
[0]: https://objectivefs.com
The upside is that Google Cloud is getting serious about the enterprise because most of them still use desktop applications.
The question is how long this will last since the future is the cloud.
AWS EFS implements NFS 4.1 which support has support for massive parallel IO.
http://www.pnfs.com/