The other question you need to ask: how good of a database is your version control system? Git definitely provided some challenges in this regard. One example: we can't simply list recently updated pages. We keep them in a Redis sorted set :)
UPDATE: Okay, Scott Chacon just popped into Campfire to school me in git: `git diff-tree --name-only HEAD~10 HEAD`
One reason is scalability and distributability, though that doesn't apply to smaller wikis. There have been attempts to load Wikipedia's full history dump into a git repository, but git doesn't seem to be able to handle that volume of stuff (and frequency of edits) nearly as well as Wikipedia's current setup of distributed MySQL servers and custom code does.
Yeah, and the biggest edit bottleneck---git's serialization of updates---is probably not an issue either, since it only applies to wikis with a lot of pages that get a lot of edits.
(By serialization of updates, I mean that, afaik, git can't apply two patches simultaneously, even if they're to completely disjoint sets of files; it has to apply one patch, then the other. That'd be like using a db that requires you to lock the whole db to do any writes. High-volume wikis instead will only lock the rows for the particular page they're editing, so edits to other pages can happen simultaneously without blocking.)
Wikipedia has branching now in the form of "approved revisions", and it has a lot of long standing TODOs like annotate that would be solved by a SCM.
The storage backend for articles is its own mini-scm with a revision table + text storage. There's no reason why a Git based backend would be any worse of a fit for a wiki.
Are approved revisions really a branch or just a tag somewhere in a linear version history? With branches you can have n parallel branches, whereas with Wikipedia it seems--though I may be wrong--that you just have one version history, with one or more revisions within that history flagged as "approved". (Having n branches of a Wikipedia article would be frowned upon, since the community decided in the early days that you can't fork an article--that only gets in the way of building an authoritative and neutral version--though it has been argued in the past that having multiple forks on controversial topics, each written from a particular point of view, would be more informative.)
Also, Git-style branches only make sense as alternative branches of the entire repository, which doesn't exist in Wikipedia terms because you can't atomically commit changes to multiple articles together. It's more than possible to have a wiki with branching and cross-page atomic commits--it's even a good idea. But it wouldn't fit Wikipedia. And to get around the atomic commit problem (given the massively concurrent editing that happens on Wikipedia) you'd have to essentially have one Git repository per article, at which point you're probably better off with a different type of backend.
> Are approved revisions really a branch or just a tag somewhere in a linear version history?
A Git branch is just a different head, but no, those things don't have
divergent histories. They're just moving tags. Anyway, semantics aside
I meant that they're doing things now that maps effortlessly to a scm.
> Having n branches of a Wikipedia article would be frowned upon
Not if it was well supported. Saying "hey, I modified the article in
my own private branch, check it out and merge it if you want" would
beat a long argument on a talk page any day.
Branch support doesn't need to entail multiple publicly accessible (as
in the stuff normal readers see) versions of articles.
> Also, Git-style branches only make sense as alternative branches of the entire repository
What? No, they'd also make sense even if there was one Git repository
per page.
> you'd have to essentially have one Git repository per article, at
> which point you're probably better off with a different type of
> backend.
We won't know until we try, will we? But there are lots of advantages
to using a well understood and simple data format with multiple
implementations over a custom schema, but of course a custom format
has its own advantages.
That's comparing apples to oranges. A proper comparison would be to create an individual Git repository for each article, not a giant repository containing all the data in Wikipedia.
That would work more like the current MediaWiki storage backend does, and would probably do delta compression of old revisions more efficiently than the current homebrew code.
Awesome! The wiki was the weakest part of the github ecosystem, and it's wonderful to see it get some much-needed love. Besides being git-backed (and therefore portable), the new wikis also finally have markdown support. The pace of releasing new features that Github is able to maintain is incredible.
Now - if we can get Issues doing the same thing by building on top of something like YABT or some other text-based approach to issue tracking, that'll be awesome.
I'll try to address that over the weekend. YABT was my original foray into Python, so it's pretty week right now and never really developed into much more than a proof of concept.
That would rock. Scott hacked on http://github.com/schacon/ticgit for a while. I'm not totally sure what happened to that. We're all in favor of a git-backed issue tracker. Though, I think it offers some challenges in filtering, labeling, etc from a git repo.
YABT popped up shortly after ticgit -- same type of thing. Scratched and itch to try something, then let it fall off. I'm going to revisit it and see where it stands though. I'd love to get it to where it serves as the backend for GitHub Issues.
One of the reasons I wrote it instead of extending ticgit was that I wanted something entirely independent of any VCS. All stuff I need to get in a readme, though. :-)
I worked on ticgit a little too. It's such a neat idea, but I'm not sure how it would work in practice. I think git repos would be best for syncing, but not the main data store on GitHub.
I'm curious about using CouchDB, actually. It has similar versioning and replication features to git.
If the storage API inside the project is good enough (something YABT's isn't yet), swapping out where it gets the data is a minor issue. You could use Git for the main storage, then have a web frontend that reads from Couch/Mongo/Redis/NewHotnessNotYetDiscovered and lazy loads from disk. A post-receive hook or two for expiring data in that cache and you're set.
+1 for distributed issue tracking. I'm a BitBucket user, where the wiki has been Mercurial-backed for some time now, but I'd like both sites to go distributed on their issue tracking. Then you'd get complete portability on any project.
I looked at Fossil for this reason (distibuted src/wiki/issues), but wasn't convinced that it was a real Mercurial or Git replacement for source control, so I gave up for now. Maybe something like Bugs Everywhere will do the trick.
29 comments
[ 3.0 ms ] story [ 69.1 ms ] threadUPDATE: Okay, Scott Chacon just popped into Campfire to school me in git: `git diff-tree --name-only HEAD~10 HEAD`
(By serialization of updates, I mean that, afaik, git can't apply two patches simultaneously, even if they're to completely disjoint sets of files; it has to apply one patch, then the other. That'd be like using a db that requires you to lock the whole db to do any writes. High-volume wikis instead will only lock the rows for the particular page they're editing, so edits to other pages can happen simultaneously without blocking.)
The storage backend for articles is its own mini-scm with a revision table + text storage. There's no reason why a Git based backend would be any worse of a fit for a wiki.
Also, Git-style branches only make sense as alternative branches of the entire repository, which doesn't exist in Wikipedia terms because you can't atomically commit changes to multiple articles together. It's more than possible to have a wiki with branching and cross-page atomic commits--it's even a good idea. But it wouldn't fit Wikipedia. And to get around the atomic commit problem (given the massively concurrent editing that happens on Wikipedia) you'd have to essentially have one Git repository per article, at which point you're probably better off with a different type of backend.
A Git branch is just a different head, but no, those things don't have divergent histories. They're just moving tags. Anyway, semantics aside I meant that they're doing things now that maps effortlessly to a scm.
> Having n branches of a Wikipedia article would be frowned upon
Not if it was well supported. Saying "hey, I modified the article in my own private branch, check it out and merge it if you want" would beat a long argument on a talk page any day.
Branch support doesn't need to entail multiple publicly accessible (as in the stuff normal readers see) versions of articles.
> Also, Git-style branches only make sense as alternative branches of the entire repository
What? No, they'd also make sense even if there was one Git repository per page.
> you'd have to essentially have one Git repository per article, at > which point you're probably better off with a different type of > backend.
We won't know until we try, will we? But there are lots of advantages to using a well understood and simple data format with multiple implementations over a custom schema, but of course a custom format has its own advantages.
That would work more like the current MediaWiki storage backend does, and would probably do delta compression of old revisions more efficiently than the current homebrew code.
One of the reasons I wrote it instead of extending ticgit was that I wanted something entirely independent of any VCS. All stuff I need to get in a readme, though. :-)
I'm curious about using CouchDB, actually. It has similar versioning and replication features to git.
http://ditz.rubyforge.org/
I looked at Fossil for this reason (distibuted src/wiki/issues), but wasn't convinced that it was a real Mercurial or Git replacement for source control, so I gave up for now. Maybe something like Bugs Everywhere will do the trick.
http://github.com/jgm/gitit
It supports Darcs and Mercurial repositories as well as Git.
Actually, the more I read, the more this sounds like Gitit-in-Ruby-rather-than-Haskell.
Defaults to Markdown while support a bunch of other input formats? Check.
Drag and drop diffs? Check.
Git-backed wikis? Check.
Gitit still has more features (apparently there's no search for Github's wiki), but then, it's much much less popular.