How does it replicate all the data on disk or flash while maintaining the write latency of DRAM?
I'm thinking there must be a delay during which loss of power will result in loss of data that was acknowledged as written. Is this something that RAMCloud overcomes in a novel way, or is the answer simply that the data is replicated across many nodes?
How about the problem of raw bandwidth? If you keep sending writes to a node that exceeds the bandwidth of the persistent storage medium but is well within the capabilities of DRAM, the storage medium will never be able to catch up even if we allow a generous delay. Maybe this is a non-issue right now because you can't send more than a few dozen Gbps over the network anyway, and we just need to hope that flash performance will improve faster than 100GbE goes mainstream.
Given the odds of these people inventing a huge solution to a hard problem in a publishing vacuum (I am not aware of papers coming out that fundamentally solve this, for example) I think it is pretty safe to assume it's subject to the same limitations all such systems are subject to. Loss of power leads to loss of data. Commonly mitigated as you already state.
Not knowing anything about how the project works, they state their read latency is on the order of 5us and write latency is on the order of 15us. On this synopsis, they also mention that one mechanism they deal with is scale. So, I think your guess correct in that they likely replicate writes across multiple nodes. Then if any single node goes down, they still have a valid valid for a given key. Writes could then be persisted to disk asynchronously.
They don't. Writes go to ONE server's memory. A crash will lose data. Speed over durability is the tradeoff. Eventually, writes are pushed to disk but not right away.
Is this still maintained? It's a really cool project, it's just so insanely fast.
I once implemented a MariaDB storage engine that used RAMCloud for storage, and it was an eye-opening experience. With blazingly fast reads and writes, latency was the number one performance issue for us.
What would be the difference between this and an SSD-based keystore and an instance of memcached with as much RAM as was necessary to hold the whole dataset?
Here's a slide from Dormando (memcached maintainer) who spoke recently at QCon SF. He created extstore, which lets you offload values to SSD and keep keys in memory. With all hits on different types of SSDs, he's seeing near all-DRAM performance in terms of throughput and latency.
The NAND SSD (dark blue line) is still hitting good performance but at ~250K queries per second hits P99 latencies above 1ms (a typical SLA Netflix calls out for their own caching using extstore). The Optane SSD (light blue line) stays well below 1ms P99 latencies up to ~500K queries per second.
DRAM alone can handle more throughput but by then you have to take into account what your network can handle.
They just host the site, it is their wiki product called Confluence. The same way Github hosts pages at github.io but has nothing to do with content that people host there.
Oh, I know, I administer Confluence, Jira and Bitbucket. Bar from bitbucket, the other two hosted versions are woefully slow. Don't get me started about the Jira Calendar.
If they're blogging about some tech, maybe they need to dogfood that first.. especially when it's due to performance
As per a previous comment, the contents of the pages linked have as much to do with Atlassian as the contents of a page on github.io have to do with Github.
This is a stanford.edu project wiki page, hosted by Atlassian. Nothing more.
By the same logic, any performance improvement that Github might see from random projects hosted on Github, should be "dogfooded" in the same way.
It's not dogfooding when it's someone else's work.
Atlassian is not blogging about it. Someone else created a Confluence wiki site for RAMCloud using the shared *.atlassian.net domain. Anyone can create a site with a subdomain there.
> Durability: RAMCloud replicates all data on nonvolatile secondary storage such as disk or flash, so no data is lost if servers crash or the power fails.
How does this work if someone is doing multiple sequential writes? Doesn't backup-ing to disk take a lot longer than writing to _RAM_ meaning some writes could get lost?
Out of pure curiosity but wouldn't the number of disks required to guarantee 100% write-backup at 100% peak times be pretty big? A quick search says HDD is ~200 times slower than RAM (on average, consumer grade).
I guess it should be big indeed. To somewhat reduce its number, you can buffer writes in another DRAM if those peaks are expected to be short. You can overprovision if you have multiple clients and expecting that they won't peak write together for a prolonged periods of time.
Raft is more practical (as in "well specified") than Paxos and closest to its lesser known cousin, VR (Viewstamped Replication). Beyond the academic genealogy of the project, what is interesting here is the fact that usability is a first-class concern. Clearly the fact that it was born out of a real/breathing project was a driving factor.
It wasn't just an academic being creative. To pick a notorious example, have a quick look at Leslie Lamport's paper on Paxos: you are never quite sure whether what you are reading is a distributed systems paper or a vintage edition of the Holy Bible.
So Raft had great timing too. It came after decades of clumsy (because novel!) systems research on consensus algos and from a laboratory of practitioners, hence its designers knew exactly how previous attempts were deficient with respect to their own needs. And it turns out these overlapped with a lot of people's. There is wisdom to be learnt from this.
Also, on another note I think that's funny because this narrative reads exactly like a startup-story!
Raft is certainly easier to explain, but it's hard to argue that Raft is more practical.
For one thing, implementing Raft in a performant way is incredibly difficult. Where paxos has very fine-grained units of consensus (e.g., ballots), in Raft, everything happens through leases, and is totally linear. This is one of the major reasons a bunch of (all?) major Raft implementations do unprincipled things like non-quorum reads. This sort of problem also makes it very difficult to get Raft to do anything on an unreliable network. Raft is also super noisy, so you have to do stuff like piggy back heartbeats in other RPC calls.
One way of thinking about this is: in paxos much of the complexity is in the specification itself. Performant Raft pushes this complexity into external systems. Both of these are extremely hard problems. There is no free lunch.
If the problem is really just verifying the model, then I think you are better off writing a simple modeling language, writing paxos in it, and then compiling that to c++ or something. This was the approach Google took in paxos made live[1] for example.
To expand on this point, check out this talk by the CockroachDB VP of engineering[0]: they used raft but found out that it's approximately as complicated as paxos. They also do non-quorum reads, all through the leader replica)
RAMCloud sounds cool and superfast, but why is the documentation so slow? Why does it need to be a site with heavy javascript with 482 ajax requests instead of plain old HTML?
I would recommend Apache Ignite if you want a similar production-ready system today, along with extending datasets automatically to disk, built-in messaging and distributed data structures, and read/write-thru cache options.
51 comments
[ 2.8 ms ] story [ 119 ms ] threadI'm thinking there must be a delay during which loss of power will result in loss of data that was acknowledged as written. Is this something that RAMCloud overcomes in a novel way, or is the answer simply that the data is replicated across many nodes?
How about the problem of raw bandwidth? If you keep sending writes to a node that exceeds the bandwidth of the persistent storage medium but is well within the capabilities of DRAM, the storage medium will never be able to catch up even if we allow a generous delay. Maybe this is a non-issue right now because you can't send more than a few dozen Gbps over the network anyway, and we just need to hope that flash performance will improve faster than 100GbE goes mainstream.
https://dl.acm.org/citation.cfm?id=2043560
I once implemented a MariaDB storage engine that used RAMCloud for storage, and it was an eye-opening experience. With blazingly fast reads and writes, latency was the number one performance issue for us.
https://twitter.com/justincormack/status/1059540644245295105
The NAND SSD (dark blue line) is still hitting good performance but at ~250K queries per second hits P99 latencies above 1ms (a typical SLA Netflix calls out for their own caching using extstore). The Optane SSD (light blue line) stays well below 1ms P99 latencies up to ~500K queries per second.
DRAM alone can handle more throughput but by then you have to take into account what your network can handle.
It's not by Atlassian...
If they're blogging about some tech, maybe they need to dogfood that first.. especially when it's due to performance
This is a stanford.edu project wiki page, hosted by Atlassian. Nothing more.
By the same logic, any performance improvement that Github might see from random projects hosted on Github, should be "dogfooded" in the same way.
It's not dogfooding when it's someone else's work.
Let's break this apart a bit more slowly.
They did not blog about this.
Their users did.
Atlassian, the company hosting it, did not write the post.
Their customers, the people using Atlassian's servers, did write the post.
https://blog.acolyer.org/2016/01/18/ramcloud/
How does this work if someone is doing multiple sequential writes? Doesn't backup-ing to disk take a lot longer than writing to _RAM_ meaning some writes could get lost?
Edit: Double checked, and it is - somewhat - for sure. Including "2009" in the title would be misleading.
Right now, I think that the algo is used in RAMCloud via LogCabin (https://github.com/logcabin/logcabin).
Raft is more practical (as in "well specified") than Paxos and closest to its lesser known cousin, VR (Viewstamped Replication). Beyond the academic genealogy of the project, what is interesting here is the fact that usability is a first-class concern. Clearly the fact that it was born out of a real/breathing project was a driving factor.
It wasn't just an academic being creative. To pick a notorious example, have a quick look at Leslie Lamport's paper on Paxos: you are never quite sure whether what you are reading is a distributed systems paper or a vintage edition of the Holy Bible.
So Raft had great timing too. It came after decades of clumsy (because novel!) systems research on consensus algos and from a laboratory of practitioners, hence its designers knew exactly how previous attempts were deficient with respect to their own needs. And it turns out these overlapped with a lot of people's. There is wisdom to be learnt from this.
Also, on another note I think that's funny because this narrative reads exactly like a startup-story!
For one thing, implementing Raft in a performant way is incredibly difficult. Where paxos has very fine-grained units of consensus (e.g., ballots), in Raft, everything happens through leases, and is totally linear. This is one of the major reasons a bunch of (all?) major Raft implementations do unprincipled things like non-quorum reads. This sort of problem also makes it very difficult to get Raft to do anything on an unreliable network. Raft is also super noisy, so you have to do stuff like piggy back heartbeats in other RPC calls.
One way of thinking about this is: in paxos much of the complexity is in the specification itself. Performant Raft pushes this complexity into external systems. Both of these are extremely hard problems. There is no free lunch.
If the problem is really just verifying the model, then I think you are better off writing a simple modeling language, writing paxos in it, and then compiling that to c++ or something. This was the approach Google took in paxos made live[1] for example.
[1]: http://www.read.seas.harvard.edu/~kohler/class/08w-dsi/chand...
[0] https://atscaleconference.com/videos/scale-2018-run-your-dat...
Domain.com: Congratulations! your domain is available. l1cache.net $12.99
https://github.com/ticketscale/ramcloud-deb-packaging