> What about consensus? Elections? Which server is the primary for a given repository? It also doesn't matter! There's no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it's always safe for any instance of a repository to receive a push.
Again reminded of what an amaizing piece of engineering S3 is (99.999999999% - 11 nines of durability) [1]
I really enjoyed the article. What's great about the design outlined in it is that it builds on things that work, like leveraging S3 rather than improvising your own distributed storage system.
Of course S3 goes beyond just Amazon these days. It's a proven design and architecture for doing object storage that you find across different cloud providers that you can also self host if you need to.
This article will no doubt inspire people working on projects like forgejo, gitea, tangled, etc. that are all facing the same scaling challenges.
It’s hard to overstate the reputation of the author of this post. Everything good about GitHub’s internal systems seemed to have his name all over it. Our times at GH didn’t overlap much but hearing the fact that he’s working at cursor increases my estimation of their engineering org by leaps and bounds.
Well, there's going to be a lot more of it. You can't be an outright ultra-right wing racist and expect that it doesn't cause brand damage. It never made business sense for Musk to act the way he does, but he couldn't help himself. Do not expect people to just be ok with it.
I'm tired of EDS when it's just not really anything to do with the conversation. You probably own a Tesla and have stickers on it. Why don't you just burn your car.
He can do that without the "Roman salute" part. Or the "methane reactors without emission controls" part. Or the "whining about the great replacement" part.
I'm not a user of twitter/x/whatever but every. damn. time. I click on an X link it's made me sadder and stupider, and the world a slightly worse place. It's the dirty bathwater of humanity.
Cursor isn't covering itself in glory regardless. The flagship app is getting enshittified at a surprising clip. It constantly pops up and interrupts your work pushing new features, changes your model to whatever the latest Grok is without prompting, has this mystery meat UI that is constantly changing, pushes cloud agents in ways that are definitely designed to trick you. We're actively looking at alternatives, I wouldn't touch anything this company produces from here on out.
This solution with WAL and compact resembled the database internals as taught by Martin Kleppmann.
I asked Claude about this observation, and learned that the author (Vicent Martí) wasn't only core Github developer but also worked on Vitess from PlanetScale.
So no surprise these two contexts resulted in the design of Continuity as it is.
https://vitess.io/ - Vitess is a database clustering system for horizontal scaling of MySQL.
If he has such a good reputation, why doesn't he write well? He never actually introduces the point of the article, he merely alludes to it in the concluding paragraphs.
however whether or not this improves their team - who knows! team dynamics is a hard thing to achieve, not just throwing good people in the same bucket ..
Author or not - I just get a shitty machine translation of the original post into German. This is so annoying, machine translated stuff is much harder to read than the original English.
Oh, and yes the preferred language in my browser is set to English.
> The "fan-out" is synchronized with a classic consensus algorithm called 3PC (three-phase commit) so that a push is only accepted if a majority of the nodes acknowledge it.
Doesn't 3PC require all nodes to agree, not just a majority?
Ah, and then it looks like the losing nodes get marked unhealthy:
> In essence, every write operation goes through a voting protocol, and any replicas on the losing side of the vote are marked as unhealthy—unavailable for reads or writes—until they can be repaired. Repairs are automatic and quick. Because a majority agreed either to accept or to roll back the update, there are still at least two replicas available to continue accepting both reads and writes while the unhealthy replica is
repaired.
But not sure how unhealthy nodes are discovered. Maybe the coordinator marks them, and if the coordinator crashes it allows stale reads and detects invalid writes?
Same. Sometimes you have to live with inconvenience to keep a moral stand. Those down-voting don't understand why any sacrifice of convenience or tech is worth it.
Agreed. The recent privacy kerfuffle with Grok CLI (plus the untrue "it's your fault" response, although they then changed the behavior) is a really bad sign for how your code and data will be the treated.
I really wish someone would make a version of the opencode desktop application that's as responsive and well-made as Cursor so I can promptly move over.
There’s a trend of doing impressive things by pushing many of the hard problems into S3 and assuming S3 “just works”, then not bothering to explain how S3 works. I guess we do the same thing all the time with other miraculous solutions to hard problems, like assuming that superscalar out-of-order processors “just work”, but in cases like this it makes for an unsatisfying explanation. Especially because S3 is a proprietary product, not an algorithm.
at this point, s3 just means object storage and doesnt mean it actually has to be hosted on aws. theres plenty of other companies that provide s3 compatible storage apis.
Yes a good object storage service provider would. It all depends on what you are optimizing for. If you are only optimizing for cost then there are several cheap options that probably don't make sense for hot or warm storage. But if you care about availability, consistency, and latency guarantees then there are different service providers available.
Last time I looked many of the alternatives didn't have as good support for some of the transaction-friendly things like conditional writes.
Hopefully if you're trying to build something complex on top of S3 and have any expectation of potentially needing to run on top of other infra in the future, you know which magic features you need and can see which of them are supported by what vendors/OSS projects. :)
But it's not really that different than doing cool stuff by pushing a bunch of things into Postgres or Kafka or [other thing] and not explaining how those things work.
The product is proprietary, but the interface and semantics are documented and freely available. "S3-compatible object store" is a commodity at this point, so it makes sense to treat it as a primitive building block for a distributed system. Of course, each implementation will have its own set of idiosyncrasies and limitations, so you'll still have to make sure that any such divergent behavior doesn't interfere with your own system requirements.
Less of a commodity at this point than a few years ago, what with Minio immolating itself to be reborn as, excuse my French, AIStor.
Which is not to say that TFA’s approach is invalid or uninteresting, but the mechanics of how “an S3” works are really obscure to those of us not in the know.
If you're looking specifically for self hosted S3-compatible object stores, then yes, the selection of available options is smaller. But an object store's ability to self-host is orthogonal to its compatibility with S3's interface and semantics.
I'd argue that S3-compatible isn't exactly what you'd need though. You'd need S3-compatible with similar latency characteristics for read and write, with four nines of availability and 10+ nines for durability. [0]. Their metrics are, quite honestly, absolutely impossible for most engineering teams and companies to ever achieve, no matter how hard they work at it. In essence, what I mean is Cursor could migrate Origin out of Amazon's S3, but they'd lose a lot of reliability and/or performance in the process.
This was the thing that caught my attention. It does feel like a lot of "scaling git was hard and complicated, so we're just offloading a significant to a different layer"
I've got no dog in this fight and no specific rationale, but it smells like a shortcut that's going to make some assumptions that might surprise later (maybe not at the global level, but a local level)
The key part: they decided to use the unchanged git codebase as their building block. Likely assuming that the people who wrote the Linux kernel probably knew what they were doing. That explains some features that look bizarre from the generic distributed-database perspective. I think the author has had this argument many times, because he reiterates it several times in the post. The rest follows from it.
> Packfiles are the fundamental building block of Git storage and Git networking.
(emphasis mine)
> His approach was storing the objects in a distributed hash table. This was only possible thanks to JGit, a custom Git implementation in Java. Like any good ol' Java library, JGit provides enough interfaces and factories and interface factories to abstract all the details of a normal Git repository, including replacing its on-disk packfiles with a DHT. Although the system worked and results were good enough for normal Git operations, the limitations of the Git protocol (which again, require packfiles to be sent over the network regardless of how you store data on the server) made the git clone performance bad enough to discard the design altogether.
Looks like Java enterprise design patterns aren't all bad after all :-) and the git ecosystem would have profited from a bit of abstraction and separation of concerns here, where network protocol, git domain model, and storage layer are evolved somewhat independently. The domain model is what everyone in the ecosystem needs to agree on, the network protocol is what at least the given local & remote host need to agree on, but storage is mainly a local concern.
Of course the question is whether git would have today's market and mind share if they had gone down that path. The ecosystem would be a lot more heterogeneous, evolving network protocol would probably take much longer, etc.
how does git lock the repo on extreme concurrency? let's say 1000 devs writing to the same repository? eventually you'll have saturation of the resource locking everyone out.
When I've worked at big companies they had bespoke SCM that resolved this. But i'm curious about mid size teams that are big enough to overwhelm git but not so that they can rewrite it.
Took me some thinking to figure out where this design is not optimal.
The first step is to separate the problems. The content addressable object storage and storing the refs. libgit2 has two backends for those already.
Then use appropriate distributed dbs for the two backends. With large scale something like Cassandra for the objects (AP) and FoundationDB (CP) for the refs (or the equivalent Amazon services).
For performance the objects need to be cached locally and requests routed to nodes where the objects are cached locally.
Part of the problem is that the API you're supposed to provide externally depends on git packs which you might have to reconstruct on the fly from such a system.
Very cool write up. But I do have to wonder, is Git the right solution for version control given where we are heading? The commands are all muscle memory for me now at this point, but all the projects I've been working on lately have just all moved to trunk based development, rebasing, and squashing merging feature branches onto main.
I've been working on another project where I have binary files checked in, assets that move or are moved and then changed that git "loses track of" due to the significant change. This just may be me doing silly things like moving a file and then editing it and then committing it and git thinks I deleted it and created a new file.
Reading this, I can't come to a conclusion other than "git rapidly starts sucking balls for use cases outside of Linux development". Github started with ze_current_fastness but then ended up recurrently smashing into walls because of it.
They should have just failed to scale git and leave place for the less idiosyncratic bazaar or mercurial to take the lead. These were slower, but would probably have scaled much better in the long term.
Github's core innovation was re-centralizing a decentralized SCM (as per article) by giving it nice web UI and features such as PRs. Oh and also free code hosting space. None of which relied on git's peculiarities.
There's nothing Cursor can do that GitHub/Microsoft can't in 2026... And vice versa... after several years of Cursor vibecoding a GitHub clone while catching up with GitHub's new features. Git's performance it's not the issue; it's GitHub Actions, PRs, etc. The moment Cursors steals paid GitHub customers and gets the same workloads, they will start having similar issues! Stealing engineers who couldn't fix this at GitHub won't make much of a difference. GitHub is not just source control; everybody can set up Gitolite and have an amazingly configurable and secure Git setup. I did this 10 years ago, and CODEOWNERS, protected branches, and tags can't do 1% of what I had done before. Cursor is going all over the place as it can't compete with their model. So, this is a defeat, a pivot, not something glorious!
I'd argue that for whatever other faults they have, Composer 2.5 is pretty competitive as an implement-planned-work model. It's several times faster than Sonnet 5, cheaper, and performance is comparable.
Ears wide open for a model that does better for the same parameters. Pareto efficiency is important, but Anthropic doesn't seem to care about it. They desperately need a Haiku 5, IMO.
I use Claude with the Codex plugin as a critic (sometimes I switch and manually review with Claude). With the Superpowers skill (which I am trying to get rid of), it designs expensive models and implementations, uses cheaper ones for other, less important stuff, including Haiku, sometimes upgrading if it hits a roadblock, and then, finally, reviews with expensive ones. Why would I need Composer unless I'm a poor hobbyist? I care about value created, not about the affordable costs. So far, Claude and Codex have given me a huge ROI and are worth every penny. I don't even use an IDE, although I pay for Zed, or the terminal anymore - Claude and ChatGPT desktop apps give me everything and keep things simple. I just watched Theo yesterday [0], and I realized I'm not alone.
Perhaps you didn't read the article, but the implementation they describe is interesting and perhaps easier implemented in a new project from scratch.
It can't be any worse than current github, and since we've heard plenty of people express their dissatisfaction with github recently, this is actually a good direction for Cursor. A product people may want. We'll see.
Yeah, and the grass is always greener somewhere else.
The only way to finally end up in a happy state about your git hosting and automated workflows running, is by running the whole shebang yourself. Saves you so much sanity in the long-term, even if starting and scaling is slower.
I am just too old, you know. I remember Bitbucket, GitLab, and the plethora of other GitHub killers. GitHub is an ecosystem, not a product. Cursor needs at least a decade to beat that! Maybe more!
People are lazy, and when there are no standards (or when they are not followed strictly), it's no small challenge to build a generic interface that works with multiple providers transparently and interchangeably. Many moons ago, we built an LOS (Loan Origination System) for subprime mortgage companies, and I designed a service that standardized input and output for dozens of mortgage vendors - credit, title - you name it. Every vendor was adding their flavor to the standards; everybody read the standards wrongly to some extent, etc. So, when we were pitching the LOS to a bunch of lenders, many of them came back to us and asked, "Can we just buy your servicing subsystem from you?" although it wasn't a separate product at the time. So, naturally, software engineers don't like one-offs unless they support one of the vendors only (pun intended) - the GOAT. It's not so easy to create or extend a standard that works across all vendor implementations. So, it's human nature to try to avoid this process and just support GitHub. The bigger the mass, the stronger the gravity. The biggest mass has the strongest gravity.
Oh, I told my son to create an org yesterday with Cursor Origin, just in case in 10 years it becomes a thing, and he said: "Oops! Paid customers only!" A GitHub "alternative," really? Well, this is reminding me to cancel my Cursor subscription. I am paying for it but not using it because I don't need it as it offers me nothing. I won't even say that somebody already got my GitHub org name, so I'm sure there will be an aftermarket for org usernames and people who want to get 10-15-20x of their $20 investment. Cursor's shortsighted decision is definitely welcomed by parasites! They rushed out their MVP so badly that they created an identity fiasco!
This is an excellent post that explains why their version of git hosting is more scalable. But it seems like most people won't need it, assuming Github remains free to use and they get their availability problems fixed.
Will free Git hosting continue to be sustainable, though, without architectural improvements like this?
Looks like a React component tree binding to SVG nodes. Right click and inspect element. Right click on the svg element in the (chrome) inspector. Choose `break on` > `subtree modifications`. That led me to look through the network tab for this: https://cursor.com/marketing-static/_next/static/chunks/0sqf...
117 comments
[ 0.19 ms ] story [ 13.3 ms ] threadCursor Origin Code Hosting
https://news.ycombinator.com/item?id=49334209
> Continuity (cnt for short) is the git storage system [...]
I guess someone realized that turned this into even more of a joke than it already is.
[1]: https://web.archive.org/web/20260818210312/https://cursor.co...
Again reminded of what an amaizing piece of engineering S3 is (99.999999999% - 11 nines of durability) [1]
1: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...
How many startups and corporations built on top of S3 semantics?
Snowflake and the big lakehouses are one of them. They use S3 as giant disk in the sky.
Of course S3 goes beyond just Amazon these days. It's a proven design and architecture for doing object storage that you find across different cloud providers that you can also self host if you need to.
This article will no doubt inspire people working on projects like forgejo, gitea, tangled, etc. that are all facing the same scaling challenges.
He still pushes his agenda, he still has lots of followers who believe his nonsense
https://x.com/elonmusk/status/2090483357456990564
Members of he Kushner family got rich via the outrageous $60B payment for Cursor, and I'm sure Musk will be rewarded later with government deals.
That doesn't count other consequences of his election interference, of course.
You either know this, or willfully ignore. Can't say which is worse.
In what way does it improve humanity's future to promote a racist film by Uwe Boll while claiming it is banned in Germany, when it actually isn't?
How low do you have to sink to promote a Uwe Boll film?
This solution with WAL and compact resembled the database internals as taught by Martin Kleppmann.
I asked Claude about this observation, and learned that the author (Vicent Martí) wasn't only core Github developer but also worked on Vitess from PlanetScale.
So no surprise these two contexts resulted in the design of Continuity as it is.
https://vitess.io/ - Vitess is a database clustering system for horizontal scaling of MySQL.
Every large co runs into this problem. New cos have fewer obstacles to paying more… until they get big.
however whether or not this improves their team - who knows! team dynamics is a hard thing to achieve, not just throwing good people in the same bucket ..
Oh, and yes the preferred language in my browser is set to English.
Doesn't 3PC require all nodes to agree, not just a majority?
IE - the interactive diagram and note about latency being bounded by the slowest replica
Ah, and then it looks like the losing nodes get marked unhealthy:
> In essence, every write operation goes through a voting protocol, and any replicas on the losing side of the vote are marked as unhealthy—unavailable for reads or writes—until they can be repaired. Repairs are automatic and quick. Because a majority agreed either to accept or to roll back the update, there are still at least two replicas available to continue accepting both reads and writes while the unhealthy replica is repaired.
But not sure how unhealthy nodes are discovered. Maybe the coordinator marks them, and if the coordinator crashes it allows stale reads and detects invalid writes?
Last time I looked many of the alternatives didn't have as good support for some of the transaction-friendly things like conditional writes.
Hopefully if you're trying to build something complex on top of S3 and have any expectation of potentially needing to run on top of other infra in the future, you know which magic features you need and can see which of them are supported by what vendors/OSS projects. :)
But it's not really that different than doing cool stuff by pushing a bunch of things into Postgres or Kafka or [other thing] and not explaining how those things work.
Which is not to say that TFA’s approach is invalid or uninteresting, but the mechanics of how “an S3” works are really obscure to those of us not in the know.
[0]: https://aws.amazon.com/s3/ (Click on Durability and availability)
I've got no dog in this fight and no specific rationale, but it smells like a shortcut that's going to make some assumptions that might surprise later (maybe not at the global level, but a local level)
When I look at one of the recent blog posts (https://github.blog/news-insights/company-news/an-update-on-...) I see crazy charts of PR merged, Commits, Repositories created. These are all git operations and not just some slow Rails app.
> Packfiles are the fundamental building block of Git storage and Git networking.
(emphasis mine)
> His approach was storing the objects in a distributed hash table. This was only possible thanks to JGit, a custom Git implementation in Java. Like any good ol' Java library, JGit provides enough interfaces and factories and interface factories to abstract all the details of a normal Git repository, including replacing its on-disk packfiles with a DHT. Although the system worked and results were good enough for normal Git operations, the limitations of the Git protocol (which again, require packfiles to be sent over the network regardless of how you store data on the server) made the git clone performance bad enough to discard the design altogether.
Looks like Java enterprise design patterns aren't all bad after all :-) and the git ecosystem would have profited from a bit of abstraction and separation of concerns here, where network protocol, git domain model, and storage layer are evolved somewhat independently. The domain model is what everyone in the ecosystem needs to agree on, the network protocol is what at least the given local & remote host need to agree on, but storage is mainly a local concern.
Of course the question is whether git would have today's market and mind share if they had gone down that path. The ecosystem would be a lot more heterogeneous, evolving network protocol would probably take much longer, etc.
https://blog.fnil.net/s3-at-twenty/
When I've worked at big companies they had bespoke SCM that resolved this. But i'm curious about mid size teams that are big enough to overwhelm git but not so that they can rewrite it.
Not sure the last month GH had fewer than five minutes of downtime - but I think it's been a while
The first step is to separate the problems. The content addressable object storage and storing the refs. libgit2 has two backends for those already.
Then use appropriate distributed dbs for the two backends. With large scale something like Cassandra for the objects (AP) and FoundationDB (CP) for the refs (or the equivalent Amazon services).
For performance the objects need to be cached locally and requests routed to nodes where the objects are cached locally.
I've been working on another project where I have binary files checked in, assets that move or are moved and then changed that git "loses track of" due to the significant change. This just may be me doing silly things like moving a file and then editing it and then committing it and git thinks I deleted it and created a new file.
They should have just failed to scale git and leave place for the less idiosyncratic bazaar or mercurial to take the lead. These were slower, but would probably have scaled much better in the long term.
Github's core innovation was re-centralizing a decentralized SCM (as per article) by giving it nice web UI and features such as PRs. Oh and also free code hosting space. None of which relied on git's peculiarities.
I feel like the article is a bit light on the design of the WAL and maintenance operations, but maybe it's me.
Anyway, I'd love to have a friendly chat with the author.
Ears wide open for a model that does better for the same parameters. Pareto efficiency is important, but Anthropic doesn't seem to care about it. They desperately need a Haiku 5, IMO.
[0]: https://www.youtube.com/watch?v=dLhcLqoff6k
It can't be any worse than current github, and since we've heard plenty of people express their dissatisfaction with github recently, this is actually a good direction for Cursor. A product people may want. We'll see.
Yeah, and the grass is always greener somewhere else.
The only way to finally end up in a happy state about your git hosting and automated workflows running, is by running the whole shebang yourself. Saves you so much sanity in the long-term, even if starting and scaling is slower.
Will free Git hosting continue to be sustainable, though, without architectural improvements like this?
But I believe it would be easy to fix with limitation for the free accounts (CI time limit, total size of repo limit etc.)