When I say large, I mean about 6 gigs or so. Because your company's source tree is probably that large.
Yes, if your company's source tree is that large you might have to use a different VCS. Or: Come up with a much smarter way to manage large binary assets, which presumably are the reason why your repos are so bloated. (If you've got 6GB of text files in your project, you probably have much bigger problems than your choice of VCS.)
I wonder if one could use the moral equivalent of git submodules. Don't manage giant binary blobs by committing them into git. Instead store the giant binary blobs in Amazon S3 (or the moral equivalent), and write some hooks and tools that maintain a separate local cache of those giant binary blobs, with symlinks pointing into the cache from your local working copy of the git repo. Then assign each version of each blob a UUID, maintain a big file listing those UUIDs and the associated S3 URLs, and check that file into git. You might end up having to run scripts like "blob commit" or "blob update symlinks" now and then, but even that might be able to be fully automated via git hooks.
Has someone written a library that does something like this? Looks like a job for Ruby or Python.
> When I say large, I mean about 6 gigs or so. Because your company's source tree is probably that large.
GCC's SVN repo is north of 17 gigs (ref - http://gcc.gnu.org/rsync.html) and yet neither svn nor git nor hg (when imported into them) seemed to have managed to blow up on it yet.
Scalable means different things to different people, I expect. Size of the repo is one, but the number of highly distributed users is another and for many projects is much more relevant and one better served by git or hg or darcs. And a license fee of $740 per person per year is another form of non-scalability, again depending on the context.
4 comments
[ 3.7 ms ] story [ 21.8 ms ] threadYes, if your company's source tree is that large you might have to use a different VCS. Or: Come up with a much smarter way to manage large binary assets, which presumably are the reason why your repos are so bloated. (If you've got 6GB of text files in your project, you probably have much bigger problems than your choice of VCS.)
I wonder if one could use the moral equivalent of git submodules. Don't manage giant binary blobs by committing them into git. Instead store the giant binary blobs in Amazon S3 (or the moral equivalent), and write some hooks and tools that maintain a separate local cache of those giant binary blobs, with symlinks pointing into the cache from your local working copy of the git repo. Then assign each version of each blob a UUID, maintain a big file listing those UUIDs and the associated S3 URLs, and check that file into git. You might end up having to run scripts like "blob commit" or "blob update symlinks" now and then, but even that might be able to be fully automated via git hooks.
Has someone written a library that does something like this? Looks like a job for Ruby or Python.
GCC's SVN repo is north of 17 gigs (ref - http://gcc.gnu.org/rsync.html) and yet neither svn nor git nor hg (when imported into them) seemed to have managed to blow up on it yet.
Scalable means different things to different people, I expect. Size of the repo is one, but the number of highly distributed users is another and for many projects is much more relevant and one better served by git or hg or darcs. And a license fee of $740 per person per year is another form of non-scalability, again depending on the context.