Conway's Law for infrastructure / tooling, perhaps.
If you have complete control over the organisation's dev process, and are willing to invest enough, you can build something better than the default free / distributed / popular tooling.
This is probably partly enabled by the fact that they don't have to care about trading patches easily with the outside world. Then, by rolling their own in-house workflow/infrastructure/tooling, they make it harder for outsiders to contribute, even if this were organisationally possible and desirable.
Based on some brief experience at Oracle, accepting outside contributions is a legal nightmare - it is frequently less expensive to do a clean-room re-implementation of a feature than go through the lawyers and contract negotiations now and in the future to get code ownership rights worked out.
I just read the first link (adding the others to my reading backlog), based on it it seems like trunk based development is less of a development model and more of a release model. Especially it does not seem to conflict directly with feature branch model.
Perhaps partly. But Paul also mentions "Developers do not break the build with any commit" -- that places a pretty heavy constraint upon regular development outside of releases.
See also got-dmz or atlassian bamboo's gatekeeper feature. Team city has something similar too. Surprised the guy who wrote bors hadn't seen more examples, I have seen the idea come up over and over again (and strongly support it).
Is it really a single big repo for the whole organization? What happens if you want to deploy your code on servers? Do you have to upload all the mobile app code and history as well? Doesn't the history get very polluted this way?
If there exists appropriate tooling (e.g. for filtering commit logs based on what touched the code belonging to a single project / what touched the code belonging to a single project and its transitive dependencies) then I dare say a huge monorepo history over many projects and libraries isn't polluted history, it is remarkably rich and useful history.
In big places with solid engineering teams and a long-term investment in infrastructure it seems achievable to build the tools to do this properly. (famously: google, which isn't a very fair example).
I think many people's default reaction to monorepos -- fear, loathing and disbelief, say -- is based upon working with open source tools / related cloud services / distributed workflows - that scale poorly when dealing with large repositories, at least with default configuration and without a lot of mucking about.
That said, this default reaction is arguably a healthy reaction if the reality is that most teams / organisations do not have the resources to invest in setting up tooling / infrastructure beyond what they can get for free / rent for a few hundred dollars a month.
> Build tools and build servers exist for a reason.
What's the build server going to do? Copy the Python code from the repo to the other server? It doesn't seem like setting up a build server just to do "git checkout; rm .git" is worth the effort.
What sort of def scaffolding do you mean? The benefits of just pulling the repo are that you can use ansible-pull to automatically update the servers whenever new code is merged to master, which would be more complicated if you had the CI server doing the pushing.
version control software, to start with. perhaps more.
i guess my viewpoint is coloured as the last thing i was working on was deployed to client desktop machines. there was absolutely no way we were going to be pushing to those with version control.
if you are deploying to a server you own, i guess in principle you can do whatever you want. one of my colleagues rigged a build to create a debian package for the server-side version of the code we worked on. that took a bit of mucking about to get going but was very nice when it was set up.
Building a deb is also an option, I generally don't prefer it because just pulling is easier to set up, and debs don't have that many advantages when you're dealing with interpreted server code.
I mostly work with java, so the build server is doing a clean compilation, to ensure the code word outside my dev environment. Pulling in known versions of dependencies, some packaging, unit and integration tests etc. Mostly using Maven.
Whilst some of that is somewhat specific to java, I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production. And I'd still want to run at least some integration tests, even for the simplest project.
And I really REALLY don't want my internet facing servers to be able to reach into my internal network, which means a checkout simply isn't going to work anyway.
Lastly I don't want my code on those servers either. You can set up .htaccess or equivalent to prevent access to .svn or .git (or dotfiles in general..) but those files really shouldn't have been there in the first place.
> I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production.
In my case, the production server is the build server. You can't have dev dependencies there, because you don't develop there.
> And I'd still want to run at least some integration tests, even for the simplest project.
Integration tests run on the dev or CI server, in my case.
> And I really REALLY don't want my internet facing servers to be able to reach into my internal network, which means a checkout simply isn't going to work anyway.
The code servers aren't in the internal network, in my case. They're already internet-facing, since I need to pull from them.
> Lastly I don't want my code on those servers either. You can set up .htaccess or equivalent to prevent access to .svn or .git (or dotfiles in general..) but those files really shouldn't have been there in the first place.
In my case, if my code isn't on the server, the server doesn't run. There's no need to set up .htaccess, since the HTTP server doesn't serve anything off the filesystem.
>> I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production.
>In my case, the production server is the build server. You can't have dev dependencies there, because you don't develop there.
The problem is not that dev dependencies exist, the problem is that they might be undeclared or unrecorded. I have imagemagic on my dev machine, but if it isn't installed as part of the server setup script, then it won't be on the production server. The tests might run fine on a dev server that happens to have imagemagic on it, but fail in production.
>Integration tests run on the dev
If you run your integration tests in your dev environment, you might not catch some on the fly configuration changes or utility installations - undeclared dependencies.
I'm using "dev environment" and "dev server" distinctly. The dev environment is whatever I have on my computer, the dev server is a server that mirrors production. If there's a dependency on your dev environment that isn't on production, you'll catch it when you deploy to one of the dev servers. Same with the integration tests.
Given that in the comment to which that was a reply, you opined that 'It doesn't seem like setting up a build server just to do "git checkout; rm .git" is worth the effort.', it's not completely unwarranted.
The part that the build server can handle is the "git checkout" part, maybe create a virtualenv with the dependencies. In that case, doing a "git checkout" is enough for deployment, because you're already assuming that the rest of the deployment happens elsewhere.
If you're working with Python, you can create builds of individual modules via zip archives. This lets you create a versionable, signable build artifact which can be easily deployed almost everywhere. They can even include static versions of you dependencies, making the deploy less reliant on what is installed on the target server.
> Is it really a single big repo for the whole organization?
Yes. It's incredible. It enables organization-wide refactoring. It facilitates code sharing. It provides easy access to examples of how code should be used.
> What happens if you want to deploy your code on servers?
You don't deploy source, you deploy packages. At Google we use bazel/blaze (http://bazel.io/). The build rules specify dependencies and visibility allowing for logical separation rather than the incidental separation that comes with chopping up your repository.
Having worked with a monorepo I never want to go back.
How do you mean? If I refactor the backend, for example, an iOS dev still needs to come in and refactor the iOS app, so the refactoring still happens independently.
> It facilitates code sharing.
That's handy if many or your projects are in the same language, true. That generally didn't hold for the projects I've worked on.
> It provides easy access to examples of how code should be used.
True, but that's mostly the same as having a GitLab instance (or similar) installed.
> You don't deploy source, you deploy packages.
Sounds like I just have different needs, in my area of development (Python server backends) the source is the package.
> How do you mean? ... so the refactoring still happens independently.
Not all refactors modify APIs, and even some that do can be programmatically applied. Suppose you've deprecated method Foo(..) and have a replacement Bar(..). With access to all the source, you can change things globally, it's great. One funny programming 'mistake' that creeps into code that's been around a while is dodgy looking ternary operators:
It is for the who,e organization. No you don't have to checkout the whole repo, you can check out a filtered portion, usually a sub folder or two. You can also do a shallow checkout without history.
There are compelling benefits to pulling from repo once, packaging, whether that means zipping, just putting code in a folder or putting it into a docker image, then promoting that "build" everywhere. Including running your integration, acceptance, and manual tests on it.
As soon as we started doing that we found benefits even for our Ruby code that is also a pretty much copy and paste deploy. It turns out there were differences in how UTF8 was ha fled by default on Ubuntu vs OSX. We ran into the same thing a few years back with some system clock behavior that behaved differently on our dev macs and Linux. All works the same in our docker images.
Of course we do see stronger benefits with our compiled code, and if all of your code is interpreted the. Maybe it is t worth it in your case, but I have certainly benefitted from it in my small shop of less than 10 devs so it isn't just something for huge companies.
The key is enabling maximum developer leverage - and one aspect of this is to get rid of as many systems errors and inefficiencies as possible.
I think build systems and internal code management schemes are large interconnected systems. It feels fairly pointless just to expose one facet of the entire systems since the total system encompasses all of the individual entities dealing with the codebase - including developers, platform - the whole works.
For example, I see no overhead in my daily work from the fact that our codebase is split to several libraries with dependencies through explicit built release versions. On the contrary,there monorepository would be just horrible.
So, before saying X is great one should firsr define what type of software is built, what is the specific toolchain, what are the specific configurations, how many developers modify the same codebase frequently, and so on.
hi, author of those slides here, I actually mention this during the talk, that the talk is over-emphesizing the awesome point to take a point across. It definately is not a technique to use everywhere, so I agree with you.
It works for some kind of projects and i was motiviated for this talk, because I saw so many projects using manyrepos in cases where it may not fit very well.
Hi, yeah, sure, effective communication requires simplification.
A few points got me concerned in the slides - namely, concerning emphasizing how easy this makes to change things. I would consider that it is not a positive thing that adding a library is cheap and easy, and I find that component versioning can be bypassed even more so.
It all sounds like grievous local greedy optimization without a view of the entire lifecycle of the codebase. The slides say change is the only constant. I would claim that unless the change is backed by good and strong architecture - which throwing all architectural firewalls away does not support - then the end result will be a mess of spaghetti the older the codebase gets.
But, I may just be too much set in my ways of coding and I respect the fact that these may be non-issues in the particular domain these slides address.
Everyone I've ever met who's worked at Twitter hates the monorepo.
They have a single git repo with way more files than that tool was ever designed to maintain. As such, normally instantaneous commands like status, log, ls, etc. take an inordinately long amount of time to execute. Apparently their onboarding process is "have someone copy his working copy to a flash drive, then copy it from there to your laptop" because git pull would take too long.
But even then, as you mention, Git doesn't scale well for them. This is why Facebook is using Mercurial. This is why the Chromium and Android projects are spread out over hundreds of Git repos. This is why Google's internal monorepo is using a Perforce-like homegrown tool backed by Google's distributed wizardry.
Git will start to see significant scaling problems beyond say 1 GB total data, 100,000 files, and X number of refs (there has been significant upstream work around scaling refs - a bunch from Twitter I believe - and I haven't measured how well newer versions handle thousands of refs).
The good news is that unless you are a software project with hundreds of developers, have large binary assets, or are doing something like Twitter and have all your developers push all their in-progress feature branches to the monorepo (madness if you ask me), a Git monorepo will scale for you. A Mercurial monorepo will scale even better. Keep in mind that most people aren't {Twitter, Facebook, Google, Mozilla}, so chances are you won't ever reach these scaling limits.
This has to be a sick joke. The monorepo at Google was the bane of my existence, and everyone else that I knew.
I don't know what it's like now, but at the time Google was on svn, and there had to be a specially hacked version of it. It used a home-grown internal requirements system, to pull down the tree in a sparse way. And to accomplish that you had to re-specify all your dependencies in these special text files (because specifying deps in two places is obviously better.) And then the tool would spend a lifetime figuring it all out, and then you would still download like 35% of the repo anyway, because the nature of giant repos is to get tangled together.
With languages like PHP it might not be so bad, as you typically are editing one file at a time. But most of Google is Java, and most people like using IDEs for that, so it was a disaster. An IDE likes to touch everything all the time, to re-index and re-compile. And, fun fact: at the time Google recommended you store your entire home directory on an NFS network drive, for security. Everyone quietly ignored that.
It was better in one way: everyone could see everyone else's source, re-using libraries was more common, and branching and contributing patches was more common. It prevented people from hoarding their source code, and just throwing .jars or .so's over the wall. But I think the Github era has shown that this did not require everything being in one giant repo! I mean, at the time, Google had Codesearch and other great tools; even though git wasn't super common at the time, this could have been solved in other ways.
My current company is in the process of breaking up their monorepo, even though our codebase is nowhere near the same size. Lately I get to taunt my co-workers because I created a new project in the new broken-up way. All the tests on my micro-repo pass in 5 seconds, and it takes many minutes for the big one.
44 comments
[ 3.0 ms ] story [ 94.9 ms ] threadAren't these the companies all terrible at taking outside contributions, except for the projects that they don't manage in their monorepos?
Monorepos are far easier to work with from an engineering standpoint, for external contributors they are definitely prohibitive.
If you have complete control over the organisation's dev process, and are willing to invest enough, you can build something better than the default free / distributed / popular tooling.
This is probably partly enabled by the fact that they don't have to care about trading patches easily with the outside world. Then, by rolling their own in-house workflow/infrastructure/tooling, they make it harder for outsiders to contribute, even if this were organisationally possible and desirable.
This I think needs bit of expanding. Does he mean in contrast to feature-branches? Why monorepos would not be compatible with feature-branches?
http://paulhammant.com/2013/04/05/what-is-trunk-based-develo...
http://paulhammant.com/2013/05/06/googles-scaled-trunk-based...
http://paulhammant.com/2014/04/03/microsofts-trunk-based-dev...
http://paulhammant.com/2014/01/06/googlers-subset-their-trun...
https://dzone.com/articles/legacy-app-rejuvenation
See also the so-called "not rocket science rule of software engineering": https://graydon.livejournal.com/186550.html
This feels like a "Doctor, doctor - it hurts when I stick a spoon in my eye" moment - "Don't do that!"
Build tools and build servers exist for a reason.
Subversion has "svn export" if you insist on working that way. git can git clone repos --depth 1; rm -rf repos/.git if you really, really must.
And what you are calling polluted history, they are calling efficient collaboration.
In big places with solid engineering teams and a long-term investment in infrastructure it seems achievable to build the tools to do this properly. (famously: google, which isn't a very fair example).
I think many people's default reaction to monorepos -- fear, loathing and disbelief, say -- is based upon working with open source tools / related cloud services / distributed workflows - that scale poorly when dealing with large repositories, at least with default configuration and without a lot of mucking about.
That said, this default reaction is arguably a healthy reaction if the reality is that most teams / organisations do not have the resources to invest in setting up tooling / infrastructure beyond what they can get for free / rent for a few hundred dollars a month.
What's the build server going to do? Copy the Python code from the repo to the other server? It doesn't seem like setting up a build server just to do "git checkout; rm .git" is worth the effort.
Then you can just add/extend a job to create a release package, and deploy it.
Doesn't take that much effort, and lets you avoid infecting your production boxes with dev infrastructure/scaffolding, as `NickNameNick` suggests.
version control software, to start with. perhaps more.
i guess my viewpoint is coloured as the last thing i was working on was deployed to client desktop machines. there was absolutely no way we were going to be pushing to those with version control.
if you are deploying to a server you own, i guess in principle you can do whatever you want. one of my colleagues rigged a build to create a debian package for the server-side version of the code we worked on. that took a bit of mucking about to get going but was very nice when it was set up.
Whilst some of that is somewhat specific to java, I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production. And I'd still want to run at least some integration tests, even for the simplest project.
And I really REALLY don't want my internet facing servers to be able to reach into my internal network, which means a checkout simply isn't going to work anyway.
Lastly I don't want my code on those servers either. You can set up .htaccess or equivalent to prevent access to .svn or .git (or dotfiles in general..) but those files really shouldn't have been there in the first place.
In my case, the production server is the build server. You can't have dev dependencies there, because you don't develop there.
> And I'd still want to run at least some integration tests, even for the simplest project.
Integration tests run on the dev or CI server, in my case.
> And I really REALLY don't want my internet facing servers to be able to reach into my internal network, which means a checkout simply isn't going to work anyway.
The code servers aren't in the internal network, in my case. They're already internet-facing, since I need to pull from them.
> Lastly I don't want my code on those servers either. You can set up .htaccess or equivalent to prevent access to .svn or .git (or dotfiles in general..) but those files really shouldn't have been there in the first place.
In my case, if my code isn't on the server, the server doesn't run. There's no need to set up .htaccess, since the HTTP server doesn't serve anything off the filesystem.
The problem is not that dev dependencies exist, the problem is that they might be undeclared or unrecorded. I have imagemagic on my dev machine, but if it isn't installed as part of the server setup script, then it won't be on the production server. The tests might run fine on a dev server that happens to have imagemagic on it, but fail in production.
>Integration tests run on the dev
If you run your integration tests in your dev environment, you might not catch some on the fly configuration changes or utility installations - undeclared dependencies.
Or create a .deb or .rpm package out of the python code.
In a non-trivial setup, just doing a 'git checkout' isn't enough for a deployment.
Yes. It's incredible. It enables organization-wide refactoring. It facilitates code sharing. It provides easy access to examples of how code should be used.
> What happens if you want to deploy your code on servers?
You don't deploy source, you deploy packages. At Google we use bazel/blaze (http://bazel.io/). The build rules specify dependencies and visibility allowing for logical separation rather than the incidental separation that comes with chopping up your repository.
Having worked with a monorepo I never want to go back.
How do you mean? If I refactor the backend, for example, an iOS dev still needs to come in and refactor the iOS app, so the refactoring still happens independently.
> It facilitates code sharing.
That's handy if many or your projects are in the same language, true. That generally didn't hold for the projects I've worked on.
> It provides easy access to examples of how code should be used.
True, but that's mostly the same as having a GitLab instance (or similar) installed.
> You don't deploy source, you deploy packages.
Sounds like I just have different needs, in my area of development (Python server backends) the source is the package.
Not all refactors modify APIs, and even some that do can be programmatically applied. Suppose you've deprecated method Foo(..) and have a replacement Bar(..). With access to all the source, you can change things globally, it's great. One funny programming 'mistake' that creeps into code that's been around a while is dodgy looking ternary operators:
With a few minute's work, one engineer can metamorphose all of these into: > That's handy if many or your projects are in the same language, true.My day to day work is almost exclusively C++. Thousands of other engineers are in the same boat as me. We all benefit from a fabulous tools team.
There are compelling benefits to pulling from repo once, packaging, whether that means zipping, just putting code in a folder or putting it into a docker image, then promoting that "build" everywhere. Including running your integration, acceptance, and manual tests on it.
As soon as we started doing that we found benefits even for our Ruby code that is also a pretty much copy and paste deploy. It turns out there were differences in how UTF8 was ha fled by default on Ubuntu vs OSX. We ran into the same thing a few years back with some system clock behavior that behaved differently on our dev macs and Linux. All works the same in our docker images.
Of course we do see stronger benefits with our compiled code, and if all of your code is interpreted the. Maybe it is t worth it in your case, but I have certainly benefitted from it in my small shop of less than 10 devs so it isn't just something for huge companies.
I think build systems and internal code management schemes are large interconnected systems. It feels fairly pointless just to expose one facet of the entire systems since the total system encompasses all of the individual entities dealing with the codebase - including developers, platform - the whole works.
For example, I see no overhead in my daily work from the fact that our codebase is split to several libraries with dependencies through explicit built release versions. On the contrary,there monorepository would be just horrible.
So, before saying X is great one should firsr define what type of software is built, what is the specific toolchain, what are the specific configurations, how many developers modify the same codebase frequently, and so on.
It works for some kind of projects and i was motiviated for this talk, because I saw so many projects using manyrepos in cases where it may not fit very well.
A few points got me concerned in the slides - namely, concerning emphasizing how easy this makes to change things. I would consider that it is not a positive thing that adding a library is cheap and easy, and I find that component versioning can be bypassed even more so.
It all sounds like grievous local greedy optimization without a view of the entire lifecycle of the codebase. The slides say change is the only constant. I would claim that unless the change is backed by good and strong architecture - which throwing all architectural firewalls away does not support - then the end result will be a mess of spaghetti the older the codebase gets.
But, I may just be too much set in my ways of coding and I respect the fact that these may be non-issues in the particular domain these slides address.
They have a single git repo with way more files than that tool was ever designed to maintain. As such, normally instantaneous commands like status, log, ls, etc. take an inordinately long amount of time to execute. Apparently their onboarding process is "have someone copy his working copy to a flash drive, then copy it from there to your laptop" because git pull would take too long.
The list of users is value free, it is just a mention of companies I know using it.
Both the Google and FB talks about this mention the technical challenges, so i guess this is really a downside at very big scales.
But even then, as you mention, Git doesn't scale well for them. This is why Facebook is using Mercurial. This is why the Chromium and Android projects are spread out over hundreds of Git repos. This is why Google's internal monorepo is using a Perforce-like homegrown tool backed by Google's distributed wizardry.
Git will start to see significant scaling problems beyond say 1 GB total data, 100,000 files, and X number of refs (there has been significant upstream work around scaling refs - a bunch from Twitter I believe - and I haven't measured how well newer versions handle thousands of refs).
The good news is that unless you are a software project with hundreds of developers, have large binary assets, or are doing something like Twitter and have all your developers push all their in-progress feature branches to the monorepo (madness if you ask me), a Git monorepo will scale for you. A Mercurial monorepo will scale even better. Keep in mind that most people aren't {Twitter, Facebook, Google, Mozilla}, so chances are you won't ever reach these scaling limits.
I don't know what it's like now, but at the time Google was on svn, and there had to be a specially hacked version of it. It used a home-grown internal requirements system, to pull down the tree in a sparse way. And to accomplish that you had to re-specify all your dependencies in these special text files (because specifying deps in two places is obviously better.) And then the tool would spend a lifetime figuring it all out, and then you would still download like 35% of the repo anyway, because the nature of giant repos is to get tangled together.
With languages like PHP it might not be so bad, as you typically are editing one file at a time. But most of Google is Java, and most people like using IDEs for that, so it was a disaster. An IDE likes to touch everything all the time, to re-index and re-compile. And, fun fact: at the time Google recommended you store your entire home directory on an NFS network drive, for security. Everyone quietly ignored that.
It was better in one way: everyone could see everyone else's source, re-using libraries was more common, and branching and contributing patches was more common. It prevented people from hoarding their source code, and just throwing .jars or .so's over the wall. But I think the Github era has shown that this did not require everything being in one giant repo! I mean, at the time, Google had Codesearch and other great tools; even though git wasn't super common at the time, this could have been solved in other ways.
My current company is in the process of breaking up their monorepo, even though our codebase is nowhere near the same size. Lately I get to taunt my co-workers because I created a new project in the new broken-up way. All the tests on my micro-repo pass in 5 seconds, and it takes many minutes for the big one.