The biggest advantages monorepos have offered is development of tools like lerna(1) or yarn workspaces.
Before that there used to be a node_modules folder with GBs of [useless] data in all my projects. Now there is just one folder on top and that's it. Also if you're developing lots of modules or plugins it makes it super to work without committing changes since they are symlinked.
node_modules is an interesting worst case of package management systems.
There's some good exploratory work currently happening on making node_modules and the node package ecosystem better in general, but especially in the polyrepo case. Yarn "Plug'n'Play" is one, and Tink [1] the other.
There's a cost for being out of date, but there's also a cost for learning the hard way whether a new version breaks prod. Pay it down like any other tech debt.
Maybe I could test literally every release version of each of my dependencies, but that isn't really my job.
Greenkeeper (and similar systems) comes to mind, too, in the polyrepo case. You can still CI with "the latest" in the polyrepo case. We have the technology to automate that. Including situations like 'let me know when the next version of my dependency that passes this test is released and send me a PR to update my pinned version when it happens'.
NPM might not be the best package manager, but if you're using something like Lerna you can get the best of both worlds. Your local copy of an internal dependency can either be symlinked to the local source code, or a copy of the published package.
That makes it a lot easier to work on a package and its consumers at the same time.
Oh God! I had forgotten how much frustration was in my old Python team until I checked the docs and discovered that you can make pip "install" your local copy of one repo as the dependency for another. The poor developer who prompted me to check was testing by doing CI builds and pulling down the new eggs.
Of course this isn't really a point against polyrepos, since it had a solution, but it's definitely something that I could imagine catching out lots of juniors.
Someone makes an incompatible change, but you do not find out months later, because the client of that incompatibility is not using the latest versions. In the meantime, the development of the module and its downstreams has marched past any sort of easy resolution, and you essentially are now maintaining two different copies of your code.
Hmm. Monorepos strongly favor "always-good". Core library breakage gets detected and rolled back really fast. It's been months since I've been affected by one.
Known-good is in the eye of the beholder, and is just another dimension that generates breakage.
google3 breaking was a bimonthly event that left the entire team aground. Bad enough that it happened, but the worst part is that we couldn't do anything about it but wait.
At Uber, both of our iOS and Android teams are over 100 contributors each and we have a Monorepo for each app platform. I'm not on the ops team but being in a Monorepo here has been one of the best development experiences in my career.
Do you mean two separate mono-repos, one for Android, one for iOS? To me that's not a monorepo. Is there little shared code between the two platforms, or is there a third repo that is depended on?
Yeah, it's two separate monorepos. There is actually some shared code between multiple monorepos. Code like IDL's and some C mapping code are shared, but they are referenced as vendor libraries so each monorepo updates those dependencies when they need to. If you think about Android vs iOS vs a web based dispatch system vs an autonomy system; They are all totally different. All of the code, dependencies, vendor code (external dependencies), and tools in each monorepo are "platform" specific and live in each respective monorepo.
Each monorepo is solving one cognitive problem domain with all of the libraries and dependencies that go with it usually for a target platform. Every iOS app that Uber builds lives in one repository — you clone that down and you can build Rider, Driver, Eats, Freight, and Jump. They all use the same networking stack, same UI stack, same map stack, same VIPER architecture, etc. This makes it easy for someone who is an iOS dev to work in any app or spin up a new app with almost no effort.
But, you're right... it's not a monorepo in the sense that all of the code in the company is in one place. Maybe I like micro-monorepos.
It’s easier to divide modules than put them back together. Wait until you know what the real boundaries are (they tend to change as scope and functionality expand)
Could you get the best of both worlds by having a monorepo of submodules? Code would live in separate repos, but references would be declared in the monorepo. Checkins and rollbacks to the monorepo would trigger CI.
We actually did this. When I started at Uber ATG one of our devs made a submodule called `uber_monorepo` that was linked from the root of our git repo. In our repo's `.buckconfig` file we had access to everything that the mobile developers at Uber had access to by prefixing our targets with `//uber_monorepo/`
We did however run into the standard dependency resolution issue when you have any loosely coupled dependency. Updating our submodule usually required a 1-2 day effort because we were out of sync for a month or two.
You need fairly extensive tooling to make working with a repo of submodules comfortable at any scale. At large scale, that tooling can be simpler than the equivalent monorepo tooling, assuming that your individual repos remain "small" but also appropriately granular (not a given--organizing is hard, especially if you leave it to individual project teams). However, in the process of getting there, a monorepo requires no particular bespoke tooling at small or even medium scale (it's just "a repo"), and the performance intolerability pretty much scales smoothly from there. And those can be treated as technical problems if you don't want to approach social problems.
To put it another way, we're comparing asymptotic O(n) with something bigger, neglecting huge constant factors on the former. There's a lot of path-dependence, since restructuring all your repos with new tooling is hard to appreciate.
This is the approach taken by a number of projects. The ones I am most familiar with are the OpenEmbedded/Yocto/Angstrom family that build Linux for embedded devices. They have a 'root' repo that references the layer repos (using metadata files rather than submodules), and there is a tool that does the pulling. It's optimised for pulling not committing though, I don't think the tooling helps much with bumping versions.
It can be misused though - the releases of the root repository reference the children by tags usually. Someone retagged a child repo and we suddenly had build failures.
My polyrepo cautionary tale: Two repos, one for fooclient, one for fooserver, talking to each other over protocol. Fooserver can do scary dangerous permanent things to company server instances, of which there are thousands.
Fooserver sprouts a query syntax ("just do this for test servers A and B"), pushed to production. Fooclient sprouts code that relies on this, pushed to production. A bit later, Fooserver is rolled back, blowing away query syntax, pushed to production. "Just do this for test servers A and B" now becomes "Do this for every server in the company". Hilarity ensues.
The biggest gripe I have with modern day monorepos is that people are trying to use Git to interact with them, which doesn't make a tremendous amount of sense, and results in either an immense amount of pain and/or the creation of a bunch of tools to try to coerce Git into behaving basically like SVN.
Which of course begs the question, rather than trying to perform a bunch of unnatural acts, why not just use SVN to start with? It works extremely well with monorepo & subtree workflows.
Sure it has some warts in a few dimensions around branching, versioning, etc. compared to Git when using Git in ways aligned with how Git wants to work, but those warts are minimal in comparison to what's required to pretzel Git monorepos into scaling effectively.
I disagree. The thesis statement repeated several times throughout is "Monorepos don't scale in exactly the same ways that polyrepos don't scale, the tools to solve the scaling problems are the exact same except monorepos need more of them and encourage bad habits along the way."
You may disagree with that thesis, but it definitely seems to cover more than one use case.
Telling people what they should or should not do is generally absurd. Every situation is unique and you can't possibly know another project's requirements or acceptable trade-offs.
A better approach, in my opinion, is "Here's what we did and why". The author clearly has experience in the area. Great! Tell me about your problems. Tell me about your attempted solutions and what did or did not work. Tell me what you wish you had done! I'd love to use knowledge of your situation to inform my own decision making.
But don't be surprised if my circumstances are different and lead me to prefer different trade-offs and choose a different solution. That doesn't make me a zealot or an idiot.
The irony wasn't lost on me. It's a fine line. Let me try a slightly different approach.
When I blog I've had much better luck telling people "here's what I did and why". I don't know your circumstances and can't tell you how to solve your problems. You may need to choose different trade-offs than I did. With that said, here is my problem, how I solved it, and what I learned along the way. Hopefully you can learn from my experiences and make a more informed decision for how to handle problems you may encounter.
My problem with polyrepos is that often organizations end up splitting things too finely, and now I'm unable to make a single commit to introduce a feature because my changes have to live across several repositories. Which makes code review more annoying because you have to tab back and forth to see all the context. It's doubly frustrating when I'm (or my team is) the only people working on those repositories, because now it doesn't feel like it gained any advantages. I know the author addresses this, but I can't imagine projects are typically at the scale they're describing. Certainly it's not my experience.
Also I definitely miss the ability to make changes to fundamental (internal) libraries used by every project. It's too much hassle to track down all the uses of a particular function, so I end up putting that change elsewhere, which means someone else will do it a little different in their corner of the world, which utterly confuses the first person who's unlucky enough to work in both code bases (at the same time, or after moving teams).
It's an interesting social problem in how you manage those project / library / repository boundaries. On the flipside, though, it's been well documented that among many of the major monorepos those boundaries still exist, they just become far more opaque because no one has to track them. You find the weird gatekeepers in the dark that spring out only when you get late in your code review process because you touched "their" file and they got an automated notice from a hidden rules engine in your CI process you didn't even realize existed.
In the polyrepo case those boundaries have to be made explicit (otherwise no one gets anything done) and those owners should be easily visible. You may not like the friction they sometimes bring to the table, but at least it won't be a surprise.
http://wiki.c2.com/?ConwaysLaw
"Conway's Law" is something like "organisation of code will match the organisation of people". It's a neat description.
I think it's more common to merge or split modules and classes than repositories.
I wonder if there'd be less tension if repos and teams were 1:1 though.
> I wonder if there'd be less tension if repos and teams were 1:1 though
Anecdotally, yes I think it helps a lot. I was once part of an organization for which each "team" having a repo is the only thing that prevented violence :-)
Can a monorepo support module- or subdirectory-level ownership controls? Or do teams using a monorepo just do without them?
Partially answering my own question: SVN, recommended in a prior comment [0], supports path-based authorization [1]. But what about teams using another version control system?
Git submodules are not in the same repo, they are a link from one repo to another, and you need to push to both if you make a change to the submodule. Maybe you're thinking of subtrees? I've never used those.
If I remember correctly, Gitlab has introduced some sort of ownership control where you can say who owns what directories for things like approving merge requests that affect those directories.
In Google, this is pretty explicit with a plaintext OWNERS file in the directory. Internal IDEs not only have an understanding of that, but can automatically suggest a minimal set of reviewers in a possibly close time zone and not out of office.
For my clients, I use an open source monorepo submodule inside the client's proprietary monorepo. I can maintain the organizations' software while sharing common code.
It's very much possible to make changes to internal libraries used all over the place, but it does require versioning to be something that people think about, and a mechanism by which those libraries aren't just pulled from source control to depend on them. Once you've got some sort of dependency management such as an internal gem/npm/whatever source you can treat those internal dependencies the same as you'd treat external ones, instead of having to somehow coordinate a release of absolutely everything in one go.
My current team managed to break a single "component" out into a separate repository. Then that repository broke into two, then those broke into other repositories, until we've eventually have around 10 or so different repositories that we work on every day.
An average change touches 4 of them, and touching one of them triggers on average releases on 2 or 3 of them. Even building these locally is super tedious, because we don't have any automation in place (not formally plan to) for chain building these locally.
This is a nightmare scenario for myself. A simple change can require 4 pull requests and reviews, half a day to test and a couple hours to release.
Yet my team keeps identifying small pieces that can be conceptually separated from the rest of the functionality, even if they are heavily coupled, and makes new repos for these!
I’ve come to the conclusion that an organisation should ideally have no more than one primary repo, with maybe a handful of ancillary repos for stuff that really doesn’t make sense in the primary. What does ‘organisation’ mean there? Well, it could mean a company, or a team, or a division. Just as software conforms to organisational structure (Conway’s Law), so too should repo structure.
Once you start having lots of peer repos being worked on within the same organisation on a daily basis, you know that you’ve partitioned far too far, and you need to roll back.
Otherwise one ends up in exactly the position you’re in. The ultimate slippery-slope end-state would be hilariously bad: a repo for each ASCII character, with repos for each word or symbol constructed out of those characters, with repos for each function constructed out of those words & symbols, with repos for each module constructed out of those functions, with repos for each system constructed out of those modules, with any change requiring a massive, intricate, failure-prone dance in order to update anything, all while patting oneself on the back about how one has avoided complexity.
Noöne sane would argue for that situation, and yet I’ve seen smart people argue that requiring coördinated changes to half a dozen repos is fine & dandy.
That's not really that different in a monorepo since you often need reviews from the same number of people anyway.
I once had to wait for 9 months to get a complex change through in a monorepo setting because of all the people involved, the number of stuff it touched and the fact that everything was constantly in flux so I spent half my time tracking changes. I'm not saying it would have been faster in a polyrepo. I'm saying that complex changes are complex regardless of how the source is organized.
I do however think that polyrepos forces you to be more disciplined and that it is easier to slip up in a polyrepo and turn a blind eye to tighter couplings.
The multi-repository code review is an interesting concept. Here at RhodeCode we're actually working on such solution to implement. This is in first to solve our internal problem of release code-review spanning usually two projects at once.
This is a hard and complex problem. Especially how to make code-review not too messy if you target 5-8 repos at once.
Seems like the main point is that you'll still need to add additional tooling (search, local cloning, build, etc) to handle scaling, something you can do just as well with polyrepos. Conversely, for polyrepos, you can add tooling to fix issues with dependency management and multi-project changes/reviews. However, the author figures that monorepos engourage bad code culture and points out that Git is hard to build a monorepo on.
To me this message seems a bit shallow, of course we can build tooling to hide the fact that we have a polyrepo. Given well enough built tooling and consistent enough polyrepo structure (all using same VCS, all being linked from common tooling, following common coding standards and using the same build tooling, etc.) the distinction from having a monorepo is more of an implementation detail.
Given the choice between a consistent monorepo where everyone is running everything at HEAD and a polyrepo where each project have their own rules and there's no tooling to make a multi-project atomic change, I'd go for the former.
Given the choice between identical working environments but different underlying implementations I would go for whatever the tools team think is easier to maintain.
What is the tooling for multi-repo atomic synchronized commits? Monorepo's give you that for free, which is the reason why I think monorepo projects exist. SVN kind of gave you partial checkouts, which was helpful.
Yes I think so too. But of course, as the article points out, nothing is entirely free. At some point we will have to build tools to handle scaling, and then the trade offs between a mono and polyrepo becomes less obvious. I'd lean towards monorepos as a base either way, but given sufficiently well working tooling it might not matter much.
Polyrepo argues that is a non-feature and don't give it to you. You can figure out where things are, but you never get synchronization.
This is a good thing because when you have to make the multirepo commit you make the change and then update each downside one at a time. Each change is much smaller and so easier to review (and also easier to find the right reviewer).
Of course the downside is you either have to maintain both ABIs (not just API), have a rollout scheme with two version of the upstream library exist side by side, or don't release.
> Given well enough built tooling and consistent enough polyrepo structure (all using same VCS, all being linked from common tooling, following common coding standards and using the same build tooling, etc.) the distinction from having a monorepo is more of an implementation detail.
Exactly. Sure, you can manually recreate a monorepo from a multirepo system, but … why do that? That takes software engineering effort that you could spend on your product instead.
> is there any real difference between checking out a portion of the tree via a VFS or checking out multiple repositories? There is no difference.
How big is your monorepo? Assume each line of code is a full 80 characters, stored via ASCII/UTF-8. That 67 million lines of code in 5GB. I can fit five of those on a Blu-ray.
> The end result is that the realities of build/deploy management at scale are largely identical whether using a monorepo or polyrepo.
True.
> It might be deployed over a period of hours, days, or months. Thus, modern developers must think about backwards compatibility in the wild.
Depends entire on the application. Lots of changes are deployed within short periods of time with low compatibility requirements.
> Downside 1: Tight coupling
Monorepos do often have tightly coupled software. Polyrepos also often have tightly coupled software. Polyrepos look more decoupled, but pragmatically I can't say I've noticed a much of difference.
> Downside 2: VCS scalability
I've also heard Twitter engineers complain about the VCS. But what is the scope of the author's discussion? 1,000 engineer orgs? Or 20 engineer orgs? Those are vastly different levels of engineering collaboration. I assume the article was not written to cover both of those. Or was it?
---
Ultimately, I think the author implicitly assumed a universe of discourse of gigantic repos with hundreds and hundreds of daily contributors.
When people talk about the spectrum of monorepo vs polyrepo architectures, that is very extreme. For example, last I knew, Uber has more repos than it did engineers. And I don't assume that "polyrepos" always means multiple repos per engineer.
Unless you are pure OSS or pure closed source - you end up with a poly-repo strategy regardless as you split open and closed code, suffering the annoyances of both systems.
No, what you end up with is a system for mirroring open source code into your repo, and a system for mirroring commits that should be open source from your code into external repos. All active work still happens in a monorepo.
You can have large repos and it not only be code. I remember seeing repos many tens of gigs because all of VS was versioned as well for "reproducibility".
adding raw versions of binary assets (designs, video, ...) can quickly lift a repo beyond a TB. Now, you could say "don't do that", but there's valid use cases where you'd want to track all binary assets as part of the development cycle.
You can do that with content, you just partition the workspace/view of the monorepo to what each person needs rather than checking the entire thing out git style.
or use a VCS that allows partial checkouts of repositories. There's no DVCS that I know of that can do that, but for example SVN can. Git LFS might be an option, too. There are also commercial products that target that market.
I just wanted to point out that reaching a measly TB of data doesn't require much effort. (worked on a product that would version rendered clips for special effect production).
My advice is that if components need to release together, then they ought to be in the same repo. I'd probably go further and say that if you just think components might need to release together then they should go in the same repo, because you can in fact pretty easily manage projects with different release schedules from the same repo if you really need to.
On the other hand if you've got a whole bunch of components in different repos which need to release together it suddenly becomes a real pain.
If you've got components that will never need to release together, then of course you can stick them in different repositories. But if you do this and you want to share common code between the repositories then you will need to manage that code with some sort of robust versioning system, and robust versioning systems are hard. Only do something like that when the value is high enough to justify the overhead. If you're in a startup, chances are very good that the value is not high enough.
As a final observation, you can split big repositories into smaller ones quite easily (in Git anyway) but sticking small repositories together into a bigger one is a lot harder. So start out with a monorepo and only split smaller repositories out when it's clear that it really makes sense.
> As a final observation, you can split big repositories into smaller ones quite easily (in Git anyway) but sticking small repositories together into a bigger one is a lot harder. So start out with a monorepo and only split smaller repositories out when it's clear that it really makes sense.
If you only need to do this once, subtree will do the job, even retaining all your history if you want.
I'm not sure what the easier way to split big repos is.
In practice, I can tell you from first-hand experience that this isn't all that simple in bigger, organically grown cases (you'll have many other things to consider if you want to keep the history in a useful way). Especially the broken branching model of SVN and co. is a problem here: In the wild, it immediately leads to "copy&paste branching" (usually through multiple commits. Migrating that to Git or Hg and splitting it up can be a challenge.
After trying to get the best of both with Subversion Externals and Git Submodules, I'd have to agree. At least until things are so loosely coupled they're begging for a public release.
That said, some packaging solutions can bridge the gap reasonably well. Unless you need instantaneous, atomic releases.
I switched to using submodules about a year ago, and they work very well for a project + a set of 4 dependencies. I handle that zoo from VS Code + Git Lens plugin.
Funnily, I only use Code to handle commits to submodules, because Git Lens is not available for the full VS IDE.
Components might need to be released “together”, but if they are worked on by different teams, it means they’ll have a different release process, as in different timeline, different priorities.
First of all this is normal, because otherwise the development doesn’t scale.
In such a case the monorepo starts to suck. And that’s the problem with your philosophy ... it matters less how the components connect, it matters more who is working on it.
Truth of the matter is that the monorepo encourages shortcuts. You’d think that the monorepo saves you from incompatibilities, but it does so at the expense of tight coupling.
In my experience people miss the forest from the trees here. If breaking compatibility between components is the problem, one obvious solution is to no longer break compatibility.
And another issue is one of responsibility. Having different teams working on different components in different repos will lead to an interesting effect ... nobody wants to own more than they have to, so teams will defend their components against unneeded complexity.
And no, you cannot split a monorepo into a polyrepo easily. Been there, done that. The reason is that working in a monorepo versus multiple repos influences the architecture quite a lot and the monorepo leads to very unclear boundaries.
And no, you cannot split a monorepo into a polyrepo easily. Been there, done that. _The reason is that working in a monorepo versus multiple repos influences the architecture quite a lot and the monorepo leads to very unclear boundaries.
I think you are conflating a monorepo (where boundaries can still be established, e.g. via a module isolation mechanism specific to the stack used) with a "monoproject"/"monomodule", where is no modularization at all.
I work on a project structured into microservices and use both. There is one global repo with submodules in subrepositories.
So when someone only wants a submodule they can happily only clone that, but when someone wants all stuff (which is the default case), the can clone and install all at once.
i have a monorepo that contains a few different early stage frontend web projects that does not interact with each other at all. They do however uses a shared component library that is also placed inside the monorepo. Tools like yarn workspaces makes sharing the library easy if the projects are located on the same repo.
When I change something on the library, i could easily also run tests across all the projects that depends on it with the latest changes of the library and make sure that my change is not breaking things all over the place, which is also pretty nice.
I am not sure yet if using a monorepo is actually the best way deal with this kind of projects, but for now it feels better than having them on seperate repos and then having to deal with the complexity of sharing the library across repos by publishing it somewhere or using git submodules or something.
With monorepos you don't have to manage PRs for 8 different repositories when adding a feature.
In my experience it's hard to establish clear boundaries, regardless of repository kind. It may be more difficult to create features which are tightly coupled across multiple repositories, but people do it regularly. And when they do, you suddenly have to manage and maintain synced features across multiple repositories.
In fact, the repo tool for the android project makes it quite easy to develop features across repositories, thus lowering the boundaries significantly.
> Having different teams working on different components in different repos will lead to an interesting effect ... nobody wants to own more than they have to
... and so nobody really understands how all of the components tie together and as a result it takes weeks of manual testing to release.
> Components might need to be released “together”, but if they are worked on by different teams, it means they’ll have a different release process, as in different timeline, different priorities.
released "together" == part of the same feature. Timelines, release process and team priorities are all there to help to deliver features. If they stand in the way, they need to be adjusted. Not the other way around.
Multi repos encourage silos. Silos encourage focusing on the goals of the silo and discourage poking around the bigger picture. Couple that with scrum, that conveniently substitute real progress metrics with meaningless points, and soon enough you end up with an IT department, full on with processes but light on delivering value.
My rule of thumb is: if you need to do PRs in several repositories to do one features, you should probably merge the repositories. At work, we have code spread among a bunch of repositories, and having to link to the 2/3 related PRs in other repos is a major PITA, and even more so for the reviewers.
Not always. It makes absolutely sense to have a repository for the gui and one for the server.
When writing a new feature you usually write some gui code and some server code and create different pull requests.
I think monorepos are seriously wrong and I completely agree with this article.
Well... Why does that make sense? I have a repository containing both the GUI and the server, and sometimes I have to make changes to both. Locating those related changes together in the same commit and/or PR makes a lot of sense to me: the changes depend on each other, and thus should be reviewed together. What's the advantage of splitting them up?
Because obviously the changes that you make in the gui are completely isolated from the changes you make on the server.
When you are working on the gui the server code is just noise and vice versa.
And it gets even worse when you use two different languages for the gui and the server.
That only really matters if your backend developers are a different team to your frontend developers where they'd want to be working concurrently. And even then, they could work in different branches and both teams merge into a development branch when finished.
The idealistic discussions for or against monorepos often overlook the most important detail: who's working on the code and how would you want them to version control it?
If it's separate projects with their own versioning then it makes sense to have them as separate repositories. If it's a single project but with individual components you'd want to version (eg because it's developed by different teams with different release timelines) then there you also have a situation where you'd want to version the code separately so once again there is a strong argument for separate repositories. However if it's one product with a single release schedule then splitting up the frontend from the backend can often be a completely unnecessary step if you're doing it purely for arbitrary reasons such as the languages being different. (I mean Git certainly doesn't care about that. A project might have Bash scripts, systemd service files, Python bootstrapping, code for an AOT compiled language (eg Rust, Go, C++, etc), YAML for Concourse, etc. They're all just text files required for testing and compiling so you wouldn't split all of those into dozens of separate repos).
> That only really matters if your backend developers are a different team to your frontend developers
What if there is one team, but different developers (one working on the frontend, another on the back)? What if QA can test the API while the frontend development is ongoing?
What if the front and backends have different toolchains, and ultimately separate execution environments (server app backend vs JS running on client machines).
I’m not sure what your point is. There’s obviously going to be thousands of different scenarios that I didn’t cover; it would be impossible to cover every imaginable use case.
> What if there is one team, but different developers (one working on the frontend, another on the back)?
Then presumably everyone in that team are full stack?(Otherwise it would be different teams in the same department) so it still makes sense to have a monorepo because you could have a situation (holiday, sickness) where someone would be working on both the front end and back end. Thankfully got is a distributed version control solution and supports feature branches so you can still have multiple people working on the same repo and then merge back into a developement branch.
> What if QA can test the API while the frontend development is ongoing?
Testing isn’t the same as released versions. You can (and should) test code at all stages of development regardless of team structures, git repo structures nor release cycles.
> What if the front and backends have different toolchains, and ultimately separate execution environments (server app backend vs JS running on client machines).
I’d already covered that point when talking about different languages in the same repo. You’re making a distinction about something that version control doesn’t care in the slightest about.
I think it’s fair to say any significant cross-project tooling should be it’s own repo (you wouldn’t include the web browser or JVM with your frontend and backend repos). But if it’s just bootstrapping code that is used specifically by that project then of course you’d want that included. Eg you wouldn’t have Makefiles separate from C++ code. But you wouldn’t include GCC with it because that’s a separate project in itself.
Ultimately though, there is no right answer. It’s just what works best for the release schedule of a product and teams who have to work upon that project.
Hmm, that's not really obvious to me. Sometimes the server has to deliver new data that is to be used in the GUI, so it's nice to be able to present those together in the same PR. If it then happens that the server-side changes do not match what you need in the GUI, it's relatively painless to add those changes in the same branch that hasn't yet been merged. In other words: although you can make changes in one without breaking the other, that doesn't make them completely isolated.
You should always have a communication layer between the gui and the server.
For example using protobuf you would update the proto definition (that can be in a shared repo) and when building the gui and the server the protobuf layer is regenerated.
So the only place where you make your changes for the new data contract is the shared repo and the gui and server would automatically have the new changes.
So now we're at three repo's, one of which is shared by the other two, and changes will have to be coordinated over them. I fail to see how that is an improvement over having both in the same repo.
In the end, I think the other comments are right that it mostly depends on who's working on something. If it's different teams, then different repo's probably make sense. But if I'm responsible for both the back-end and the front-end, they're usually not isolated at all at least in terms of project requirements, and hence keeping them together makes sense.
(But of course, even then there's nuances. I think the article is mostly arguing against monorepo's as in company-wide monorepo's. I'm willing to believe Googlers that it works well for Google, and I'm not in a position to claim what it'd be like for other companies. Team-wide monorepo's for different parts of the same project, however, make a lot of sense to me.)
> I'm willing to believe Googlers that it works well for Google
It doesn't. In my entire career, that was the only environment in which some random would break us and we couldn't do anything about it other than hope for a rollback and then wait for hours for the retest queue to clear before we could deploy anything at all.
Maybe not all the time, but you need the escape hatch of pinning healthy deps, because HEAD of everything is not guaranteed to work.
Well, I'm willing to believe you that it didn't work well for you as well. My point is that company-wide monorepo's are largely irrelevant to my point, as I'm not arguing in favour or against those (I'm leaving that to people who've worked with it).
> Because obviously the changes that you make in the gui are completely isolated from the changes you make on the server.
In my experience, that is almost never the case. Often, the frontend requires a new endpoint or a modification to an existing endpoint. If you don't coordinate this change, you end up with a non-functional PR that cannot even be tested. Same happens when the backend proposes an endpoint change that affects the frontend.
We have moved the frontend and backend to the same repo to make coordination and testing of such cases simpler.
* Any non-backwards compatible change in the interface between the components. Yes this can be solved. But when working in a smaller team on proprietary software why use time solving a problem you don't need to solve?
A surprising number of companies are prepared to accept an hour of downtime for an internal system if it saves them money.. In my experience the best business practice is to offer the product owner/manager the costed options in such a situation and allow them to choose.
This is true for systems where there is a well-defined protocol between GUI and servers and a proper versioning process in place, i.e. most "old-school" client/server systems.
I expect lots of people on HN are working on systems with very tight coupling between client/GUI and server and no proper versioning between them, as is common in web applications. Hence the replies to the contrary: you're probably from quite different worlds :)
(Now, I personally think that maintaining sound versioning practices is a good idea even if you do have tightly coupled control of both the client and the server side. But that may just be me...)
I think what matters in the end is Conway's law. Conway's is frequently misinterpreted as an observation when it's actually advice: Structure your applications/repos like you structure your teams. You're going to end up with that code structure anyway, so might as well save some time.
Any full stack change will be represented by one PR that changes from pre change to post change. Two repos would introduce a new possible state where one has the change applied and the other doesn't.
And later you add the iOS and Android clients too.
Will those go into the same repo?
Better to keep server and clients apart, especially if release schedules are different.
Sure if the release schedules are different then have them in separate repos so things like tagging makes sense. But often people work with a single release schedule. There's just so many variables that go into these decisions that the thread here is bonkers.
Smart people can work through problems to get the job done. Monorepo vs polyrepo won't stop people from moving forward.
It'll be really typical for a gui/server to want to share some is_valid_payload() function. The client to validate it before sending, and for the server to do its own validation.
If it's a monorepo your PR might be a 2 line patch to that function, then adding the GUI and server code.
If you split it you'll first need to have a PR on the "validation-lib" repo, then once that gets in a PR on the "server" repo, bumping the "validation-lib" version dependency, and finally a PR on the "gui" repo bumping the dependency for both "validation-lib" and "server" (for testing etc.). That's before you need do deal with the circular dependency that "server" also wants "gui" for its own "I changed my server code, does the GUI work?" testing.
Better just to have them in a monorepo if they're logically the same code and want to share various components.
These are two separate functions why would you ever want a function that checks both gui and server?
The gui validation logic belongs to the gui layer, the server validation logic to the server layer. If you have a function that contains logic from both layers there is something seriously wrong with your design.
Imagine something like "curl" where a client needs to validate a manually provided request before making it.
In any case, if you're nitpicking that example you're missing the point. The same would go for any number of other shared code you could imagine between a client/server trying that logically make up one program talking over a network.
I still can’t see how you would have a shared library for a C# gui and a Java server for example.
Your communication layer would obviously live in both repositories.
Even in case you are using the same language and you do have shared libraries then what is the problem?
The shared libraries would surely be shared with other projects so it makes sense to have them in a separate repository.
In cases where there's a high degree of churn (i.e. early-stage startups) in shared libraries, updating those libraries can cause a large amount of busywork and ceremony.
If you had a `foo()` function shared between the GUI and the server (or two services on your backend, or whatever), in a monorepo your workflow is:
- Update foo()
- Merge to master
- Deploy
In a polyrepo where foo() is defined in a versioned, shared library your workflow is now:
- Update foo()
- Merge to shared library master
- Publish shared library
- Rev the version of shared library on the client
- Merge to master
- Deploy client
- Rev the version of shared library on the server
- Merge to master
- Deploy server
This problem gets even more compounded when your dependencies start to get more than one level deep.
I recently dealt with an incredibly minor bug (1 quick code change), that still required 14 separate PRs to get fully out to production in order to cover all of our dependencies. That's a lot of busywork to contend with.
It seems to me that the real problem is your toolchain.
In a previous project the workflow was like this:
Update foo()
Merge to master
Publish shared library
Deploy
So as you can see the only step added was to publish the shared library that would automatically update the version in all the projects using it.
If you are really doing everything manually I can understand that this is a pain, but this has nothing to do with the monorepo / multiple repo distinction, this is a tooling problem.
But you've just invented a sharded monorepo, and now have all the monorepo problems without the solutions.
What if updating foo() breaks something in one of the clients (say due to reliance on something not specified). Then you didn't catch that issue by running client's tests, now client is broken, and they don't necessarily know why. They know the most recent version of shared broke them, but then they have to say "you broke me" or now one of the teams needs to investigate and possibly needs to bisect across all changes in the version bump under their tests to find the breakage.
How is that handled?
(the broader point here is that monorepo or multirepo is an interface, not an implementation, its all a tooling problem. There are features you want your repo to support. Do you invest that tooling in scaling a single repo or in coordinating multiple ones? Maybe I should write that blog post).
If you have the same functionality that can't be re-used (for no reason), then I'd call that a design flaw.
I'll need a few more validation functions for each clients. I don't want to write+maintain multiple functions that do the same thing, even if it's just copy+paste.
It's "data" validation. So let's put that in the "data layer" repo.
We now have, at least:
- Server
- Web (GUI)
- Android
- iOS
- Data
- More clients?
We'll also have branches for each development task. How do we know what branch the other branches should use? One "simple" feature can easily spread over multiple repos. Does each repo refer to the repo+branch it depends upon (don't forget to update the references when we merge!), or we add a "build" repo which acts as the orchestrator?
Most PRs will need to be daisy chained - who reviews each one? Will they get comitted at the same time?
How do we make the builds reproducible? commit hashes? tags? ok, we now need to tag each repo, and update the references to point to that tag/hash... but that changes the build.
Well, I'm glad our code base is split over multiple repos because "scalability".
The classic reason for any validation is that you want the validation to be done in the frontend (to save a network roundtrip and provide better, immediate feedback), on the backend (so that if the frontend is compromised and maliciously circumvents that validation, it still gets validated), and both of the validations to be the same to prevent inconsistencies.
A good way to fulfil those requirements is to have the exact same function available in both places.
> If it's a monorepo your PR might be a 2 line patch to that function, then adding the GUI and server code.
> If you split it you'll first need to have a PR on the "validation-lib" repo, then once that gets in a PR on the "server" repo, bumping the "validation-lib" version dependency, and finally a PR on the "gui" repo bumping the dependency for both "validation-lib" and "server" (for testing etc.). That's before you need do deal with the circular dependency that "server" also wants "gui" for its own "I changed my server code, does the GUI work?" testing.
The above is exactly why I am so firmly opposed to multirepo[0]-first. And it's really just a throwaway example: a real change would involve multiple different library and executable repos, all having separate PRs. And then there's the relatively high risk of getting a circular incompatibility.
This can be worth the cost, for organisational reasons. But until you need it, don't do it. It's very easy to split a git repo into multiple repos, each retaining its history (using git filter-branch). Don't incur the pain until you need to, because honestly, you're not likely to need to. You're probably not going to grow to the size of Google. Heck, most of Google runs in one monorepo, with a few other repos on the side: if they can make it work at their scale, so can you. And if, as the odds are, you never grow to their size, then you'll never have wasted time engineering a successful multirepo system instead of delivering features to your business & customers.
My rule of thumb is: if you need to do PRs in several repositories to do one feature, your projects are either tightly coupled enough that they should be one monolithic piece of software, or your tight coupling is a problem you should work on resolving.
Requiring multiple PRs to multiple repos to roll out one user-facing feature is fine, as long as your independent modules/projects are not actually interdependent (i.e. one of those PRs will not break another independent repo that lacks a corresponding PR).
Just because things change in tandem, that does not mean that they're all the same thing. When I add a new function to my backend service, all frontends that consume its API also need to be adjusted. But that doesn't mean that the backend service, its command-line clients and its web GUI client should live in the same repo.
It's probably a matter of taste - but I think they should be in the same repo. I like tying test failures/regressions to a specific commit for documentation and admin purposes. Having a test fail or regression due to an 'unrelated' commit in another repo sounds like a nightmare waiting to happen when you try investigating.
I the difference of opinion is between developers who work on self-hosted "evergreen" products where the latest version is deployed, and others who work with multiple release branches with fixes/features constantly being cherry-picked.
Why? You are just creating more work for yourself by keeping the components in different repos. Now you need to create N commits when updating something. If your future self wants to investigate how the software has evolved there are N times as many commits to analyze.
At a past job, I had to edit roughly 5 different repositories in order to do some trivial programming task (send an email or some such). It was quite easily the least productive / most demoralizing workflow I've ever experienced.
Context switching really sucks. You should aim to reasonably avoid it
I can think of situations where components 'need' to release together because of organizational rules and not any actual binding between the components, in that case of course they do not need to be in the same repository.
I agree that you should always start with one repo and split as needed, it's the MVR way (minimum viable repository)
So why is that? Why do we ned to couple together the software development efforts with release? Based on my experience there is no difference between the monorepo vs multirepo approach from the deployment point of view.
What are you talking about! In my perfect micro services world I just have these enforced bounded contexts that are so perfectly designed they never need to change. Consequently all parts of the system are perfectly independent snowflakes that can be deployed without thinking about any other parts of the system. It’s beautiful really when you think about the mess that things were before we could do this!
While I know you are being sarcastic, I really have heard bushy tailed young “architects” say something similar who just read about Domain Driven Design and then decided they were trying to “educate us”.
I generate Coq proofs of Swagger descriptions that were compiled from a speech to text dump during a 10 person Hangout. Downside is that some of the protobufs aren't laid out as cleanly as one would like.
I haven't tried in Git, but with Mercurial merging repos is as simple as pulling from an unrelated repository and merging, that's it. It's a lot simpler than splitting a repo up unless you accept that all of the old history can remain, then you just make a clone and delete what should no longer be a part of the repository.
But monorepo leads to tight coupling, and that is just as much a pain to work on as versioning, or two teams are simultaneously working on the same shared code, and you have not only merge conflicts, but conflicting functionality.
Agree 100%. "Pardon the interruption" followed by an article with a fixed top bar asking me to "become a member" (supporting an anti-open web tech company like it's a charity) and a fixed bottom bar asking me to sign up.
I’ve found monorepos to be extremely valuable in an immature, high-churn codebase.
Need to change a function signature or interface? Cool, global find & replace.
At some point monorepos outgrow their usefulness. The sheer amount of files in something that’s 10K+ LOC ( not that large, I know ) warrants breaking apart the codebase into packages.
Still, I almost err on the side of monorepos because of the convenience that editors like vscode offer: autocomplete, auto-updating imports, etc.
Monorepos and packages are not mutually exclusive. You can and should have many different projects in subfolders I'm your monorepo, each with their own builds and tests and artifacts (though hopefully somewhat standardized). The point is that now it's easy to release changes across multiple projects, integration test between them on a specific global patch, etc, without a whole pile of complex tooling.
Agreed. When I wrote the parent comment, I was thinking of a time I prematurely abstracted an API wrapper to a private git repo and how painful simple, frequent changes were.
Though, as you say and I commented below they’re not mutually exclusive. A wrapper or even an entirely separate service can exist alongside others.
One dark side of this is being able to “reach inside” other parts of the monorepo and blur application boundaries.
This guy gets it. Software Engineering is about using the appropriate tools and techniques for the task at hand. If your repo gets so large it can't be comfortably checked out, something needs to get split apart.
Monorepos are also a great technique for tackling large legacy codebases. When the rot is all in one designated place, it becomes easier to encourage good developer habits on new code created in new, separated repo(s).
Speaking from experience I've worked on a team operating through a monorepo project that came out real well. The codebase was mostly golang, so everything lived in the GO_PATH, but for the most part the typescript in the UI side of the repo didn't complain. Testing and code quality was a higher priority, as well, which may have contributed to its success.
I have also worked on a monorepo project that had minimal tests and automation, that soon grew monstrous and ultimately needed refactoring. That was a big pile of coffeescript, es6 and java that ultimately refactored into three different node modules and two microservices.
Javascript and its module packaging tends to conform better to polyrepo patterns. golang code all wants to be in the same place, and java repos have their own desired nested directory structures. These two languages tend to encourage monorepo design patterns.
Monorepo or Polyrepo, the correct answer is whatever works for your team and task at hand.
Hold on, are we talking about monorepos, ie a set of projects with shared change history (and possibly 'build it all' type tooling) or single monolithic apps?
I'm seeing these two things conflated in this thread.
In fairness, a single repo does encourage a monolithic architecture (even though one can have multiple binaries inside a single repo), just as a monolithic app does encourage a lack of modularity (even though one can write a single app composed of well-chosen modules).
To me, a monorepo exists of a set of related or semi related services or runtimes that can operate autonomously, but have a dependency on their siblings to operate correctly.
In some cases, this could be two separate backend projects where you want to re-use the same deployment pipeline.
Often, I find that API wrappers are something that I share across frontends and backends in the JS world, so it often makes sense to separate my projects into:
- backend
- frontend
- common
In Typescript I really like this pattern and can namespace shared types so that it’s very clear to the future reader that this type is probably used outside of the current context.
So, to reply to your comment — I think the term “monorepo” can encompass a lot of different project types.
Static analysis is easier on monorepo. At least one can run it on all code. Polyrepo has the problem that some code is off the radar. That might be the only advantage of monorepo in my opinion.
My last 2 jobs have been working on developer productivity for 100+ developer organizations. One is a monorepo, one is not. Neither really seems to result in less work, or a better experience. But I've found that your choice just dictates what type of problems you have to solve.
Monorepos are going to be mostly challenges around scaling the org in a single repo.
Polyrepos are going to be mostly challenges with coordination.
But the absolute worst thing to do is not commit to a course of action and have to solve both sets of challenges (eg: having one pretty big repo with 80% of your code, and then the other 20% in a series of smaller repos)
Jesus, this. Look, you're going to run into issues either way, because you're trying to solve a difficult problem.
It's like thinking OOP or functional programming is going to solve all your issues... I mean, in some limited cases they could, but realistically you're just smooshing the difficulties around and hopefully moving them to somewhere where you are more able to deal with them.
FWIW, I've worked in a many-repo org and it sucked worse than huge companies with monorepos and good tooling, but I'm not going to make some blanket statement because it depends on the specifics of your code/release process/developer familiarity etc.
Sounds reasonable. I'll have to add, though, that the underlying technology factors into this as well.
For example: If you're stuck with a TFS monorepo (you poor soul), you actually get to deal with both problems to some extent, since TFS doesn't enforce that you check out the intire repository at once.
This can have very "funny" situations because someone forgot to checkout new changes in some folder. OTOH, at least for releases, you can remedy this by using CI everywhere.
>Scaling a single VCS to hundreds of developers, hundreds of millions lines of code...
Maybe I am way out of my element here, but is this a common problem? Do companies with only “hundreds of engineers” really have “hundreds of millions of lines of code”?
From personal experience, it can happen. At one point, I was personally responsible for about 2 million lines of code. Over several years, I was able to reduce it to about 500k through generous use of code generation for ORM type work. The generated code never ended up in VCS, but the generator and model did. Certainly helped checkout/update times as there was several thousand fewer files to deal with.
I was one of about 900 engineers at a financial company of about 1500 employees at the time.
I don't honestly know how many lines of code there were across the company, but I imagine it easily exceeded 100M. It took us a full week to do a full recompile of everything. We had no CI... Was always a problem approaching release time.
It's definitely the case that a mega monorepo doesn't, in practice, have the atomic commit property. E.g. once you add owner files and separate code reviews, you're in for a world of hurt. Case in point, Google developed an internal tool to split cross-cutting CLs into manageable pieces, wrangle all the owners and approvals, presubmits, etc, and then submit the CL piecemeal--i.e. not atomic.
Chromium uses a different model. It just DEPS's in other repos at pinned versions. That has a whole other set of problems.
It's not quite so black and white. It's true that repo-wide refactorings often get carved into little changes and so aren't made atomically, but they're the exception rather than the rule. Any small change, e.g. changing an interface and the 5 callers of it, _can_ be made atomically. And changing code that's reused a small number of times is a far more common case then changing core libraries the whole company uses, so atomic submit ends up being hugely valuable.
At least the author gave us the courtesy of italicizing his broken assumption from the outset of the post.
> Because, at scale, a monorepo must solve every problem that a polyrepo must solve, with the downside of encouraging tight coupling, and the additional herculean effort of tackling VCS scalability.
Right.
But you have to get to "scale" first (as it relates to VCSs). Most companies don't. Even if they're successful. Introducing polyrepos front loads the scaling problems for no reason whatsoever. A giant waste of time.
Checkmate! I didn't even need a snarky poll. The irony of that poll is that it clearly demonstrates his zealotry, not other people's.
I think there's a nuance to this that should be pointed out: Monorepos allow you to do very bad hacks (I need this other component over there; let me just put in a Symlink. Done.). And if people can, they will use those hacks.
If you split your repo up from the get go, the worst thing you can get that you'll have to assemble multiple distinct, well-encapsulated (in terms of project structure) things into one. In Git, that could lead to multiple root commits, but that's about it.
No. The worst case is that the engineering team spent more time working on “well encapsulated projects” than on the most important project for their business and are all now out of jobs. Most companies don’t fail because of tech debt. And certainly not because of version control tech debt.
Exactly. Whenever I see an engineer take a hardline position (eg: "no monorepos you zealots!") I always ask myself: is this person just annoyed?
Most of the time they're just annoyed.
One side effect of every successful business are annoyed worker ants that are sick of dealing with growth problems. I've been there. I know how annoying it can be.
Personally I've found comfort in embracing the chaos and learning to manage it responsibly. No dogma. No absolutes. Know how to do monorepos well. Know how to do polyrepos well. Learn the pitfalls of both. Don't assume other people are stupid zealots.
I agree. Any article (like the linked one) that states one side of a case as an absolute without giving any exceptions or caveats is going to be greeted by me with scepticism. Particularly as he keeps mentioning 3 large engineering organisations that disagree with him.
Not exactly. (At least small) companies can go out of business because of bugs. And one great way to "achieve" said bugs are implicit dependencies hidden from developers that didn't introduce them.
> The worst case is that the engineering team spent more time working on “well encapsulated projects” than on the most important project for their business
I'm not really sure how I should read this. Don't you use your repos to solve business problems? Why should that change because of the repo layout?
If you do a poly-repo approach from the start, and have dependencies between repos, you need to introduce component versioning from the start. Component versioning doesn't solve any business problems, but requires engineering effort.
Small businesses are not going to go out of business because of bugs unless those bugs aren't addressed. They'll go out of business because of poor sales and product management. Different things.
> Monorepos allow you to do very bad hacks (I need this other component over there; let me just put in a Symlink. Done.).
Why would you put in a symlink? You could just provide a path to the actual component and import it into your project.
> the worst thing you can get that you'll have to assemble multiple distinct, well-encapsulated (in terms of project structure) things into one
When you have multiple repos, you also have multiple versions and releases of things. Now every team has the following options:
1. Backport critical fixes to every version still in use (hard to scale)
2. Publish a deprecation policy, aggressively deprecate older releases, and ignore pleading and begging from any team that fails to upgrade (infeasible - there'll always be a team that can't upgrade at that moment for some reason)
You also have to solve the conflicting diamond dependency problem. This is when libfoo and libbar both depend on different versions of libbarfoodep. It's even more fun in Java because everything compiles and builds, but fails at runtime. So now you have to add rules and checks to your entire dependency tree - some package managers have this (Maven), others don't (npm IIRC).
What language and build tool is this that you're using?
I don't know anyone who has abused Maven or Cargo or Go like this. And I don't imagine Visual Studio Solutions for C# are used like this.
Is there an underlying disagreement based on JS/Ruby/Python scriptish coding (which creaks when a lot of developers work on it) vs C and C++ (which have astonishingly bad build system stories) vs big-iron languages that don't sweat when in a monorepo.
> And I don't imagine Visual Studio Solutions for C# are used like this.
At my workplace, we've just been cleaning up a whole bunch of instances of exactly that anti-pattern. Except that it's obviously not symlinks (which require specific user rights on Windows), but links to external files in VS.
Same problem, though: They're easy to introduce and a pain to deal with later on.
Could a VCS simply blacklist symlink files? Plus if you have developers doing crap like this (and their colleagues letting it slide in code reviews) you have problems that can't be solved by monorepo vs polyrepo. You have an engineering culture problem.
> Monorepos allow you to do very bad hacks (I need this other component over there; let me just put in a Symlink. Done.).
I've seen that with polyrepos as well: The entire project would require you to clone the individual repos into a specific directory structure so that things would work (no, not even submodules).
> You can't split monorepos after the fact, at least not without immense costs.
Sure you can. The difficulty of doing so depends on many (many) factors. If your team does their job well then the costs won't be immense. It might be annoying, but not that hard.
Speaking in absolutes or platitudes solves nothing. Sometimes monorepos make sense. Sometimes polyrepos make sense. It's entirely dependent on what your company does.
Of course, if everybody is very diligent in keeping things in the monorepo distinct and independent, then it's easy to split it later on. But relying on constant diligence doesn't work out in the long run in my experience.
It's not like polyrepos solve the long term diligence problem though.
E.g. if relying solely on a package manager to to keep coupled things in lock step, you need make sure that version numbers are kept up to date for every little change made to every library.
You can easily end up with a situation where someone in another team makes a small change but doesn't change the lib version number. That's a people issue but it does happen.
You can get round that by using a repo SHA but now you two things to keep up to date for every library.
Like wise you'll have to be diligent in versioning APIs. Anecdotally I've found it easier to keep things in lock step when in a single repo and using a single pull request for each story than I have where separate teams have to keep separate repos in sync.
Both work but the monorepo approach worked better for the projects I've worked on. It just lead to less moving parts and more repeatability when there's a single SHA to watch.
I also have been luck enough that I haven't worked on a project so large that we couldn't build a monorepo on a single machine with "normal" build tooling.
> You can always just put all your small repos into a big one.
It's not quite as simple as that. You'll need to avoid rebuilding the entire repo for every change - using something like Bazel. This means your build tooling has to be replaced entirely, which is a non-trivial task and not something your devops/release engineering team will thank you for.
For any 3rd party libraries used by your projects you need to either ingest those projects into your monorepo and update forever. Or keep npm/maven/pip/gem/whatever around just for managing 3rd party dependencies (+ whatever system you use to front the main language package registries, because of course you're not talking directly to NPM/Maven central are you? What if they go down or do a leftpad?).
I think either system - monorepos or polyrepos - works fine; just pick one and stick with it. Monorepos will probably give you better velocity starting out. Past a certain size, which most software shops will never hit, the already-available tools lend themselves better to polyrepo. And more devs know polyrepo tools (eg. Jenkins) than the corresponding monorepo solution (eg. Bazel). Things might swing in favor of monorepo on the VCS side if Twitter/Google/FB ever open-source their stuff.
Eh, if you small repos build separately before you put them into one big repository, they'll build separately after, even if you just have a Makefile in each. If building your software depended on building its parts first, you already have the tooling to do so.
This really does not work for all languages. Speaking from personal experience, trying to monorepo multiple JS projects managed by NPM, it's more work than that.
I know neiher JS nor NPM. Could you elaborate a bit? What's the difference between having your code in folders A B C, each their own repo, and having your code in folders A B C, subfolders of some monorepo? Does NPM try to do something clever with your VCS?
>not something your devops/release engineering team will thank you for.
It's their job. If they actively don't want to do work then you probably made a hiring mistake somewhere. By that logic what DevOps really wants is to the company to shut down since then they'd have none of that tedious work to do.
They will have to work the entire Christmas break to pull off the switch. That will not make them happy. Rolling out massive changes like this is not easy, it needs to be planed in advance, tools built, dry-runs run, and then the final move. It also requirement management to not schedule any releases for several weeks before or after the change. As a member of a tool team I wouldn't think of doing this level of change when other people are in the office which means I miss my Christmas holiday.
Sure it is their job, but is isn't an easy job and there are many opportunities for things to go wrong. It might or might not be the right choice for you, but don't overlook how hard it is.
Note that the above applies for going in either direction.
You're making a lot of assumptions about how such a move would be done which I don't feel are warranted. You're picking the hardest most painful option and then using it to claim the process is painful rather than that the option you chose is painful.
If I was moving many small repos into a single mono repo then I'd do it one repo at a time. Presumably your small repos are independent entities so there's no reason to do a single massive switch. Transition each repo to the new build system inside the existing repo. Once that works then you can transition that repo into the mono-repo and tie together the build systems. No need to stop releases, no massive chance of everything failing, no weeks of debugging while the world is stopped, etc. Rinse and repeat until everything is moved over. Process becomes more optimized and less error prone with each repo that is moved over.
Why weekends? Small moves means you do it during the week and during regular working hours. Done in branches with CI support so that it's pretty unlikely to break anything. If doing regular releases require you to be spending your weekends then you have bigger issues to fix first. Spread it over however long it takes. Why are you trying to make your life more difficult?
I have done this. Regardless of when you do the move everybody who works in the repo being moved is down for several hours at best if you do it during working hours.
Release engineering is a conservative profession. They won't thank you for upsetting all their established processes, introducing new software and systems that have to be maintained, and kept running just because you don't like how your code is laid out.
It's harder to make a business value case for this type of change - there are only vaguely worded promises of "improved developer velocity". Contrast that with a change that automates or makes faster some aspect of building and releasing - a professional release engineering team would be all over it because they can demonstrate value in that work.
In any company beyond 50 people, there are multiple engineering managers, directors or the VP of engineering that will need to back this initiative to make the release engineering team do it. It's really not as simple as "dump all the code in one repo". I'm speaking from experience.
> You can't split monorepos after the fact, at least not without immense costs.
That has not been my experience at all. At a previous employer, we did exactly that with a multi-language library. In fairness, having multiple languages enforced fairly good directory structure in our single repo. But isn’t that the real point: good structure makes life easier, period. The thing is, going into a project you often don’t know what the right structures are yet. Creating a new repo for each component you think you need ossifies those choices, making it far more difficult to walk back on them later on (first because you may not even see the architectural mistake, second because the maintainers of that component will have an investment in its existence).
> You can always just put all your small repos into a big one.
In my experience, that’s harder, precisely because over time so much tooling has been built into each repo to manage builds, images, deployment &c.
I’ve worked in monorepos & I’ve worked in multrepos, and so far my experience has been that monorepos enable faster velocity and more-maintainable software. I’ve not (yet) worked at Google- or Facebook-scale, though, and I’m completely open to the idea that at that scale a team really does need lots and lots of repos, and tooling to stitch them all together.
The author talks about proponents of monorepos, but I thought when I read it: actually they are victims of monorepos trying to explain to themselves as much as anyone why they choose to suffer with them. (Actual reason: for $$$).
Nobody would choose to drag around every historical afterthought in the development sequence of long forgotten software going back three decades that no longer builds with current tools, just so they can work on a small library off in a corner. Software is getting written and added to these monorepos at a much faster rate than hardware and networks are able to hide the bloat-upon-bloat growth of them.
>Nobody would choose to drag around every historical afterthought in the development sequence of long forgotten software going back three decades that no longer builds with current tools, just so they can work on a small library off in a corner.
If it doesn't work then it should be deleted. If it's still running somewhere then it should be maintained. Presumably you have a CI system so the monorepo actually requires everything in it to build.
In my experience, it's polyrepos that allow for dead and un-maintained code to just sit there for eternity. You forget about that unused repo right until the moment the service it deploys to (if you can track down that dependency) needs an update or goes down. Monorepos can more easily force system wide CI that checks for broken dependencies or other issues.
That old code worked 30 years ago... We have a closet in our office with a computer with Windows XP no service packs, and whatever compiler was used to build at the time. If we every have to release an update for whatever we were shipping we can do so. (assuming that computer still boots after all these years...)
In the embedded world supporting software for 30 years is not unheard of. We avoid it, but it is in the back of our mind that someday we might have to release an update. Fortunately 30 years ago nothing was internet connected, we are worried that we might be releasing security updates for our current products 50 years from now...
"The frank reality is that, at scale, how well an organization does with code sharing, collaboration, tight coupling, etc. is a direct result of engineering culture and leadership, and has nothing to do with whether a monorepo or a polyrepo is used. The two solutions end up looking identical to the developer. In the face of this, why use a monorepo in the first place?"
.....because, as the author directly stated, the type of repo has nothing to do with the product being successful. So stop bikeshedding, pick a model, and get on with the real business of delivering a successful product.
This argument boils down to people who have used Perforce, who believe in the benefits of a monorepo, and people who have only ever used git, who do not. While it's true that git is a terrible program that does not lead to conclusions about the merits of a monorepo.
The post just reads like some opinionated piece for traffic. The author has never even used a monorepo as far as I can tell, so can only argue from one side, the best one ever used: polyrepo. Then goes on to list 'theoretical' benefits and the downsides (which should also be theoretical if having never been used) of monorepos. It concludes with "The two solutions end up looking identical to the developer. In the face of this, why use a monorepo in the first place? Please don’t!" implying that 'Google, Facebook, Twitter, and others' do it for no benefit.
I'm not familiar with how monorepos work in practice, but it seems obvious to me that it's going to complicate everyday tasks.
Ready to commit? Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit. (I'm having flashbacks to Clearcase already.)
Need to roll back the last two commits you made? Sure, that takes two seconds--oh, wait, another team made multiple commits that got interleaved with yours. Have fun cherry picking the files you want to revert.
Of course, I'm apparently a curmodgeon, because as soon as someone starts talking about running a find/replace globally across multiple projects, I want to grab something sharp.
Problem is, places that use monorepos also tend to have whole teams full of people who feel entitled to f#ck with stuff across the entire repo, often without going through the normal review or other processes for each component. Thus it's not uncommon to commit code early in the day so that you can build packages for system test, find a problem in system test, then come back later the same day to find one of those randos has already "fixed" your code to adhere to some new standard they decided on at lunchtime. Now reverting only your own commit is a mess, and reverting theirs as well invites a s###storm of epic proportions because they're higher-caste than you.
> places that use monorepos also tend to have whole teams full of people who feel entitled to f#ck with stuff across the entire repo
That's only true to the extent that the statement 'places that use multirepos also tend to be full of people who feel entitled to f#ck with stuff across the entire codebase' is.
Bad colleagues can cause damage either way. I appreciate that with a single repo, bad colleagues are at least forced to have passing tests after their changes, rather than leaving it to me to pick up the pieces.
That hasn't been my experience. Yes, it's a culture thing rather than a technology thing, but with a monorepo the "core" or "foundation" or "developer experience" teams tend to act like they're the owners of all the code and everyone else is just visiting. With multiple repos that's reversed. Each repo has its owner, and the broad-mandate teams are at least aware of their visitor status. That cultural difference has practical consequences, which IMO favor separate repos. The busybodies and style pedants can go jump in a lava lake.
> with a single repo, bad colleagues are at least forced to have passing tests
Passing unit tests, big whoop. Maybe sometimes light functional tests. Integration/system/stress tests that have to run across an entire cluster for non-trivial time to get meaningful results and thus can't easily be kicked off from a commit hook? Not a chance. The coverage that results is no better (or even different) than what you'd get with separate repos.
> with a monorepo the "core" or "foundation" or "developer experience" teams tend to act like they're the owners of all the code and everyone else is just visiting.
Fwiw, this hasn't at all been my experience with this kind of thing at Google. Certainly developer experience teams and language teams will make broad changes that affect everyone but
Those changes are trivial: the change maker has to make a convincing argument that the change can't break anyone, if they can't, it will require local approval from the owner.
They don't happen all that often: ~once a month for any given leaf directory.
They require a special approvals process where you have to answer why the change is necessary, why the churn is worth it, and why it really is safe, and convince a group of approvers that this is the case.
If they do break something, the change-maker has to roll it back and then either reconsider, or fix the issues and try again.
>The busybodies and style pedants can go jump in a lava lake.
Consistent style is important. Among other things, it means that unmaintained code (ie. the stuff written 3 years ago that does its job) still gets updates to be consistent with everything else, so that you don't end up with ancient spaghetti that breaks every modern style rule (I mean you still get that, but less). It also allows deprecation. If there's a core library that does something "wrong", they can deprecate the badly behaved stuff and make sure everyone is off of it: if its not in trunk, it isn't being used.
>The coverage that results is no better (or even different) than what you'd get with separate repos.
This is untrue. If I'm the `core` team who maintains the `core` repo that everyone depends on, and I make a change that breaks you but my tests pass, you don't know it. Then when you version bump, fixing yourself is your problem. With a monorepo, it's my responsibility to fix you before I can make my change.
Many things are important. Some things are more important than others, and I'd say that "not making a thousand developers' workflows more cumbersome" is higher on the list than style issues.
> If ... I make a change that breaks you but my tests pass, you don't know it.
So don't do that. Open-source projects deal with this exact same issue across repos and owners all the time. There are responsible ways to do it. Mostly they involve learning to communicate as peers, with respect, instead of "core" teams imposing their neophile opinions on everyone else. If we're all at the same company, regardless of whether we use one repo or many, there's no excuse for you not to validate your changes against other groups' tests as well as your own. Diligence does not depend on a particular repo structure.
Elsewhere in this thread I've seen just the opposite. Tons of people claiming variants of "breaking changes should just bump the major version."
I'd argue that in the long run, not being able to update dependencies because they broke you is going to be much worse than them fixing the incompatibilities for you.
Either way, you need people to act like adults and communicate, but the multirepo problem is worse.
Why would those things be issues? At Google there multiple new commits per second and it works mostly fine, you just need a VCS which is made for a monorepo.
> Ready to commit? Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit. (I'm having flashbacks to Clearcase already.)
If they merge cleanly, it's not an issue. If they don't, you need to fix the merge conflict. The work you need to do is proportional to the number of merge conflicts, which isn't special to monorepos.
> Need to roll back the last two commits you made? Sure, that takes two seconds--oh, wait, another team made multiple commits that got interleaved with yours. Have fun cherry picking the files you want to revert.
Again, only an issue if the changes are on the same files. It can be a bit of a pain to revert a stack of diffs, but if it's just a random commit with no other relevent commits to the file, very easy.
Yeah, the rebasing complaint isn't fair if you're using a modern VCS.
I used to work on a large team at Cisco Systems that used Clearcase. Clearcase does not do merges. If anything has changed in master, you have to check out again, which obliterates all local changes.
(I have never met a developer who liked Clearcase. It was built to simplify life for system administrators and to tick the right boxes for management, not to be useful for developers.)
My general VCS experience is that you can't roll back a commit without also rolling back all subsequent commits, related or not. I'm glad to hear that modern systems have fixed that. (It looks like even Subversion does that now, cool!)
If you don't touch the same files the merge is trivial.
If you do - then it's a good thing that you see there's a conflict right away, rather than notice versioning problems between your separate subsystems in integration testing (or even worse - production).
> Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit.
If their commits didn't affect you, then … it merges cleanly, and you don't care. If their commits do affect you, then … now you know. With multiple repositories, you will have no idea that they broke something you rely on.
Even better, with a monorepo, if they break something your existing code relies on, they' have broken the build, and they have to fix it. With multiple repos, another person or team is free to break things you rely on without even knowing it, and you won't know it until you update your dependencies hours, days, weeks or months later, wondering why everything is suddenly different.
If another team's work is tightly-enough interwoven with yours that their daily commits affect you, then you're all on the same team (and/or your architecture needs work).
You're not a curmudgeon, you're just wrong. As another comment states, you'd have to fix the merge conflict anyways and if there's no merge conflict, your rebase isn't an issue. If you're getting merge conflicts and need to rebase your project and that's challenging, it turns out to be an...organizational issue.
A lot of the pain you describe boils down to insufficient tooling and bad code organisation.
Outside of mega-corps, usually only a few people (couple of teams at best) are working on a given section of code at a time. Coordinating changes between maybe 12-15 people is quite feasible. Most of the time it's enough to keep code nicely segregated by paths - something like $team/$XXX or $scope/$team/$YYY should work.
On top of that, you need two things to enable a nice workflow:
* server-side (or otherwise programmatic) merges/rebases only; no human should ever need to push to master directly. That's the job of the [pre-]CI machinery.
* comprehensive pre-merge testing before the server-side merge. Do all your development in branches, and have bot+CI test _all_ unmerged branches against MASTER+YOURBRANCH on every push to YOURBRANCH. Because no human is involved when merging to master, the tip of the master has not moved due to external factors. Also, rerun tests if master has indeed moved thanks to bot having merged another branch ahead of yours. Fix any test breakages in the branch.
To make the second item work, you have to realise that the testing steps need to be rapid enough. Usually it should be enough to test the new-to-be-branch against merge problems, code convention errors and have a run through all the usual unit tests. Most of the time you can leave any larger cross-service or integration/end-to-end tests for code that has landed in master already.[ß]
Once code has landed in master, CI can pick it up and produce the release artifacts. The tooling needs to be good enough to know how to avoid useless work (doing useless checkouts and running tests against dozens of branches will get expensive). It also needs to provide very good and easily actionable feedback. You want clear test results, with quick jumps to failure(s) and robust logging.
To give some context of where the above is coming from... We have a monorepo with more than 130 projects, and about 7% of the codebase changes in any given month. (Except for December. Understandably.) We also clock more than 40k pre-merge test runs a month. Once a merge request has been approved, it is often available for shipping in 20 minutes.
When builds against master never fail due to code conflicts, development velocity is maintained better even with multiple teams working on the same piece of code.
ß: this is a tradeoff between development friction and test coverage. A sufficiently thorough integration test can take anything from a just a few minutes to couple of hours. You want to run them against batches of changes, and in case of introduced failures, incentivise teams to put in new unit tests to cover as much of the uncovered error scope as possible.
401 comments
[ 3.1 ms ] story [ 270 ms ] threadBefore that there used to be a node_modules folder with GBs of [useless] data in all my projects. Now there is just one folder on top and that's it. Also if you're developing lots of modules or plugins it makes it super to work without committing changes since they are symlinked.
(1) https://lernajs.io
There's some good exploratory work currently happening on making node_modules and the node package ecosystem better in general, but especially in the polyrepo case. Yarn "Plug'n'Play" is one, and Tink [1] the other.
[1] https://npm.community/t/tink-faq-a-package-unwinder-for-java...
Maybe I could test literally every release version of each of my dependencies, but that isn't really my job.
That makes it a lot easier to work on a package and its consumers at the same time.
Of course this isn't really a point against polyrepos, since it had a solution, but it's definitely something that I could imagine catching out lots of juniors.
Someone makes an incompatible change, but you do not find out months later, because the client of that incompatibility is not using the latest versions. In the meantime, the development of the module and its downstreams has marched past any sort of easy resolution, and you essentially are now maintaining two different copies of your code.
Known-good is in the eye of the beholder, and is just another dimension that generates breakage.
Other products have their own repos, sometimes multiple. As of 2016.
Do you mean two separate mono-repos, one for Android, one for iOS? To me that's not a monorepo. Is there little shared code between the two platforms, or is there a third repo that is depended on?
Each monorepo is solving one cognitive problem domain with all of the libraries and dependencies that go with it usually for a target platform. Every iOS app that Uber builds lives in one repository — you clone that down and you can build Rider, Driver, Eats, Freight, and Jump. They all use the same networking stack, same UI stack, same map stack, same VIPER architecture, etc. This makes it easy for someone who is an iOS dev to work in any app or spin up a new app with almost no effort.
But, you're right... it's not a monorepo in the sense that all of the code in the company is in one place. Maybe I like micro-monorepos.
I gave a talk on it, "Python at Massive Scale", at PyData London 2018. Videos on YouTube/PyVideo.
We did however run into the standard dependency resolution issue when you have any loosely coupled dependency. Updating our submodule usually required a 1-2 day effort because we were out of sync for a month or two.
You need fairly extensive tooling to make working with a repo of submodules comfortable at any scale. At large scale, that tooling can be simpler than the equivalent monorepo tooling, assuming that your individual repos remain "small" but also appropriately granular (not a given--organizing is hard, especially if you leave it to individual project teams). However, in the process of getting there, a monorepo requires no particular bespoke tooling at small or even medium scale (it's just "a repo"), and the performance intolerability pretty much scales smoothly from there. And those can be treated as technical problems if you don't want to approach social problems.
To put it another way, we're comparing asymptotic O(n) with something bigger, neglecting huge constant factors on the former. There's a lot of path-dependence, since restructuring all your repos with new tooling is hard to appreciate.
It can be misused though - the releases of the root repository reference the children by tags usually. Someone retagged a child repo and we suddenly had build failures.
Fooserver sprouts a query syntax ("just do this for test servers A and B"), pushed to production. Fooclient sprouts code that relies on this, pushed to production. A bit later, Fooserver is rolled back, blowing away query syntax, pushed to production. "Just do this for test servers A and B" now becomes "Do this for every server in the company". Hilarity ensues.
Which of course begs the question, rather than trying to perform a bunch of unnatural acts, why not just use SVN to start with? It works extremely well with monorepo & subtree workflows.
Sure it has some warts in a few dimensions around branching, versioning, etc. compared to Git when using Git in ways aligned with how Git wants to work, but those warts are minimal in comparison to what's required to pretzel Git monorepos into scaling effectively.
You may disagree with that thesis, but it definitely seems to cover more than one use case.
Telling people what they should or should not do is generally absurd. Every situation is unique and you can't possibly know another project's requirements or acceptable trade-offs.
A better approach, in my opinion, is "Here's what we did and why". The author clearly has experience in the area. Great! Tell me about your problems. Tell me about your attempted solutions and what did or did not work. Tell me what you wish you had done! I'd love to use knowledge of your situation to inform my own decision making.
But don't be surprised if my circumstances are different and lead me to prefer different trade-offs and choose a different solution. That doesn't make me a zealot or an idiot.
When I blog I've had much better luck telling people "here's what I did and why". I don't know your circumstances and can't tell you how to solve your problems. You may need to choose different trade-offs than I did. With that said, here is my problem, how I solved it, and what I learned along the way. Hopefully you can learn from my experiences and make a more informed decision for how to handle problems you may encounter.
Also I definitely miss the ability to make changes to fundamental (internal) libraries used by every project. It's too much hassle to track down all the uses of a particular function, so I end up putting that change elsewhere, which means someone else will do it a little different in their corner of the world, which utterly confuses the first person who's unlucky enough to work in both code bases (at the same time, or after moving teams).
In the polyrepo case those boundaries have to be made explicit (otherwise no one gets anything done) and those owners should be easily visible. You may not like the friction they sometimes bring to the table, but at least it won't be a surprise.
I think it's more common to merge or split modules and classes than repositories. I wonder if there'd be less tension if repos and teams were 1:1 though.
Anecdotally, yes I think it helps a lot. I was once part of an organization for which each "team" having a repo is the only thing that prevented violence :-)
Partially answering my own question: SVN, recommended in a prior comment [0], supports path-based authorization [1]. But what about teams using another version control system?
[0] https://news.ycombinator.com/item?id=18810313
[1] http://svnbook.red-bean.com/en/1.5/svn-book.html#svn.serverc...
Git has the idea of sub-modules, but they're really just filters. (They're in the same repo). So ultimately, you don't have that kind of control.
We use service owners, so when a change spans multiple services, they are all added automatically as blocking reviewers.
[1] https://help.github.com/articles/about-codeowners/
[1] https://gitlab.com/gitlab-org/gitlab-ee/issues/1012 [2] https://docs.gitlab.com/ee/user/project/code_owners.html
So (mono)repos are composable.
https://github.com/OctoLinker/OctoLinker
You can just click the import [project] name and it will switch to the repo.
An average change touches 4 of them, and touching one of them triggers on average releases on 2 or 3 of them. Even building these locally is super tedious, because we don't have any automation in place (not formally plan to) for chain building these locally.
This is a nightmare scenario for myself. A simple change can require 4 pull requests and reviews, half a day to test and a couple hours to release.
Yet my team keeps identifying small pieces that can be conceptually separated from the rest of the functionality, even if they are heavily coupled, and makes new repos for these!
Once you start having lots of peer repos being worked on within the same organisation on a daily basis, you know that you’ve partitioned far too far, and you need to roll back.
Otherwise one ends up in exactly the position you’re in. The ultimate slippery-slope end-state would be hilariously bad: a repo for each ASCII character, with repos for each word or symbol constructed out of those characters, with repos for each function constructed out of those words & symbols, with repos for each module constructed out of those functions, with repos for each system constructed out of those modules, with any change requiring a massive, intricate, failure-prone dance in order to update anything, all while patting oneself on the back about how one has avoided complexity.
Noöne sane would argue for that situation, and yet I’ve seen smart people argue that requiring coördinated changes to half a dozen repos is fine & dandy.
So don't use polyrepos for heavily coupled projects, then. Or even better...
... try to avoid heavy coupling in the first place.
Q: Why are we debating the merits of mono-repos over poly-repos?
A: Because it's managing dependencies is really hard and needs expertise.
I once had to wait for 9 months to get a complex change through in a monorepo setting because of all the people involved, the number of stuff it touched and the fact that everything was constantly in flux so I spent half my time tracking changes. I'm not saying it would have been faster in a polyrepo. I'm saying that complex changes are complex regardless of how the source is organized.
I do however think that polyrepos forces you to be more disciplined and that it is easier to slip up in a polyrepo and turn a blind eye to tighter couplings.
This is a hard and complex problem. Especially how to make code-review not too messy if you target 5-8 repos at once.
I assume this does not apply to 99% of people using a repo.
To me this message seems a bit shallow, of course we can build tooling to hide the fact that we have a polyrepo. Given well enough built tooling and consistent enough polyrepo structure (all using same VCS, all being linked from common tooling, following common coding standards and using the same build tooling, etc.) the distinction from having a monorepo is more of an implementation detail.
Given the choice between a consistent monorepo where everyone is running everything at HEAD and a polyrepo where each project have their own rules and there's no tooling to make a multi-project atomic change, I'd go for the former.
Given the choice between identical working environments but different underlying implementations I would go for whatever the tools team think is easier to maintain.
This is a good thing because when you have to make the multirepo commit you make the change and then update each downside one at a time. Each change is much smaller and so easier to review (and also easier to find the right reviewer).
Of course the downside is you either have to maintain both ABIs (not just API), have a rollout scheme with two version of the upstream library exist side by side, or don't release.
Nothing is perfect.
Exactly. Sure, you can manually recreate a monorepo from a multirepo system, but … why do that? That takes software engineering effort that you could spend on your product instead.
How big is your monorepo? Assume each line of code is a full 80 characters, stored via ASCII/UTF-8. That 67 million lines of code in 5GB. I can fit five of those on a Blu-ray.
> The end result is that the realities of build/deploy management at scale are largely identical whether using a monorepo or polyrepo.
True.
> It might be deployed over a period of hours, days, or months. Thus, modern developers must think about backwards compatibility in the wild.
Depends entire on the application. Lots of changes are deployed within short periods of time with low compatibility requirements.
> Downside 1: Tight coupling
Monorepos do often have tightly coupled software. Polyrepos also often have tightly coupled software. Polyrepos look more decoupled, but pragmatically I can't say I've noticed a much of difference.
> Downside 2: VCS scalability
I've also heard Twitter engineers complain about the VCS. But what is the scope of the author's discussion? 1,000 engineer orgs? Or 20 engineer orgs? Those are vastly different levels of engineering collaboration. I assume the article was not written to cover both of those. Or was it?
---
Ultimately, I think the author implicitly assumed a universe of discourse of gigantic repos with hundreds and hundreds of daily contributors.
When people talk about the spectrum of monorepo vs polyrepo architectures, that is very extreme. For example, last I knew, Uber has more repos than it did engineers. And I don't assume that "polyrepos" always means multiple repos per engineer.
Seriously, you have over 1 TB of code and 100 people wrote it?
I just wanted to point out that reaching a measly TB of data doesn't require much effort. (worked on a product that would version rendered clips for special effect production).
On the other hand if you've got a whole bunch of components in different repos which need to release together it suddenly becomes a real pain.
If you've got components that will never need to release together, then of course you can stick them in different repositories. But if you do this and you want to share common code between the repositories then you will need to manage that code with some sort of robust versioning system, and robust versioning systems are hard. Only do something like that when the value is high enough to justify the overhead. If you're in a startup, chances are very good that the value is not high enough.
As a final observation, you can split big repositories into smaller ones quite easily (in Git anyway) but sticking small repositories together into a bigger one is a lot harder. So start out with a monorepo and only split smaller repositories out when it's clear that it really makes sense.
If you only need to do this once, subtree will do the job, even retaining all your history if you want.
I'm not sure what the easier way to split big repos is.
In practice, I can tell you from first-hand experience that this isn't all that simple in bigger, organically grown cases (you'll have many other things to consider if you want to keep the history in a useful way). Especially the broken branching model of SVN and co. is a problem here: In the wild, it immediately leads to "copy&paste branching" (usually through multiple commits. Migrating that to Git or Hg and splitting it up can be a challenge.
That said, some packaging solutions can bridge the gap reasonably well. Unless you need instantaneous, atomic releases.
Funnily, I only use Code to handle commits to submodules, because Git Lens is not available for the full VS IDE.
First of all this is normal, because otherwise the development doesn’t scale.
In such a case the monorepo starts to suck. And that’s the problem with your philosophy ... it matters less how the components connect, it matters more who is working on it.
Truth of the matter is that the monorepo encourages shortcuts. You’d think that the monorepo saves you from incompatibilities, but it does so at the expense of tight coupling.
In my experience people miss the forest from the trees here. If breaking compatibility between components is the problem, one obvious solution is to no longer break compatibility.
And another issue is one of responsibility. Having different teams working on different components in different repos will lead to an interesting effect ... nobody wants to own more than they have to, so teams will defend their components against unneeded complexity.
And no, you cannot split a monorepo into a polyrepo easily. Been there, done that. The reason is that working in a monorepo versus multiple repos influences the architecture quite a lot and the monorepo leads to very unclear boundaries.
I think you are conflating a monorepo (where boundaries can still be established, e.g. via a module isolation mechanism specific to the stack used) with a "monoproject"/"monomodule", where is no modularization at all.
Edit: expanded wording
> where boundaries can still be established, e.g. via a module isolation mechanism specific to the stack used
Unfortunately this isn't a technical issue and that's the problem.
In my opinion monorepos make refactoring dependant projects much easier. However it is much harder to establish and enforce clear boundaries...
So when someone only wants a submodule they can happily only clone that, but when someone wants all stuff (which is the default case), the can clone and install all at once.
Downside is that I have to commit twice
When I change something on the library, i could easily also run tests across all the projects that depends on it with the latest changes of the library and make sure that my change is not breaking things all over the place, which is also pretty nice.
I am not sure yet if using a monorepo is actually the best way deal with this kind of projects, but for now it feels better than having them on seperate repos and then having to deal with the complexity of sharing the library across repos by publishing it somewhere or using git submodules or something.
In my experience it's hard to establish clear boundaries, regardless of repository kind. It may be more difficult to create features which are tightly coupled across multiple repositories, but people do it regularly. And when they do, you suddenly have to manage and maintain synced features across multiple repositories.
In fact, the repo tool for the android project makes it quite easy to develop features across repositories, thus lowering the boundaries significantly.
... and so nobody really understands how all of the components tie together and as a result it takes weeks of manual testing to release.
released "together" == part of the same feature. Timelines, release process and team priorities are all there to help to deliver features. If they stand in the way, they need to be adjusted. Not the other way around.
Multi repos encourage silos. Silos encourage focusing on the goals of the silo and discourage poking around the bigger picture. Couple that with scrum, that conveniently substitute real progress metrics with meaningless points, and soon enough you end up with an IT department, full on with processes but light on delivering value.
The idealistic discussions for or against monorepos often overlook the most important detail: who's working on the code and how would you want them to version control it?
If it's separate projects with their own versioning then it makes sense to have them as separate repositories. If it's a single project but with individual components you'd want to version (eg because it's developed by different teams with different release timelines) then there you also have a situation where you'd want to version the code separately so once again there is a strong argument for separate repositories. However if it's one product with a single release schedule then splitting up the frontend from the backend can often be a completely unnecessary step if you're doing it purely for arbitrary reasons such as the languages being different. (I mean Git certainly doesn't care about that. A project might have Bash scripts, systemd service files, Python bootstrapping, code for an AOT compiled language (eg Rust, Go, C++, etc), YAML for Concourse, etc. They're all just text files required for testing and compiling so you wouldn't split all of those into dozens of separate repos).
What if there is one team, but different developers (one working on the frontend, another on the back)? What if QA can test the API while the frontend development is ongoing?
What if the front and backends have different toolchains, and ultimately separate execution environments (server app backend vs JS running on client machines).
> What if there is one team, but different developers (one working on the frontend, another on the back)?
Then presumably everyone in that team are full stack?(Otherwise it would be different teams in the same department) so it still makes sense to have a monorepo because you could have a situation (holiday, sickness) where someone would be working on both the front end and back end. Thankfully got is a distributed version control solution and supports feature branches so you can still have multiple people working on the same repo and then merge back into a developement branch.
> What if QA can test the API while the frontend development is ongoing?
Testing isn’t the same as released versions. You can (and should) test code at all stages of development regardless of team structures, git repo structures nor release cycles.
> What if the front and backends have different toolchains, and ultimately separate execution environments (server app backend vs JS running on client machines).
I’d already covered that point when talking about different languages in the same repo. You’re making a distinction about something that version control doesn’t care in the slightest about.
I think it’s fair to say any significant cross-project tooling should be it’s own repo (you wouldn’t include the web browser or JVM with your frontend and backend repos). But if it’s just bootstrapping code that is used specifically by that project then of course you’d want that included. Eg you wouldn’t have Makefiles separate from C++ code. But you wouldn’t include GCC with it because that’s a separate project in itself.
Ultimately though, there is no right answer. It’s just what works best for the release schedule of a product and teams who have to work upon that project.
In the end, I think the other comments are right that it mostly depends on who's working on something. If it's different teams, then different repo's probably make sense. But if I'm responsible for both the back-end and the front-end, they're usually not isolated at all at least in terms of project requirements, and hence keeping them together makes sense.
(But of course, even then there's nuances. I think the article is mostly arguing against monorepo's as in company-wide monorepo's. I'm willing to believe Googlers that it works well for Google, and I'm not in a position to claim what it'd be like for other companies. Team-wide monorepo's for different parts of the same project, however, make a lot of sense to me.)
It doesn't. In my entire career, that was the only environment in which some random would break us and we couldn't do anything about it other than hope for a rollback and then wait for hours for the retest queue to clear before we could deploy anything at all.
Maybe not all the time, but you need the escape hatch of pinning healthy deps, because HEAD of everything is not guaranteed to work.
In my experience, that is almost never the case. Often, the frontend requires a new endpoint or a modification to an existing endpoint. If you don't coordinate this change, you end up with a non-functional PR that cannot even be tested. Same happens when the backend proposes an endpoint change that affects the frontend.
We have moved the frontend and backend to the same repo to make coordination and testing of such cases simpler.
* Any non-backwards compatible change in the interface between the components. Yes this can be solved. But when working in a smaller team on proprietary software why use time solving a problem you don't need to solve?
(This is from experience.)
Unless they're running on the same computer and deploy literally simultaneously, this is already a problem you need to solve.
I expect lots of people on HN are working on systems with very tight coupling between client/GUI and server and no proper versioning between them, as is common in web applications. Hence the replies to the contrary: you're probably from quite different worlds :)
(Now, I personally think that maintaining sound versioning practices is a good idea even if you do have tightly coupled control of both the client and the server side. But that may just be me...)
Not really. You can have a single repo with top level directories tigershark-gui and tigershark-server.
Smart people can work through problems to get the job done. Monorepo vs polyrepo won't stop people from moving forward.
If it's a monorepo your PR might be a 2 line patch to that function, then adding the GUI and server code.
If you split it you'll first need to have a PR on the "validation-lib" repo, then once that gets in a PR on the "server" repo, bumping the "validation-lib" version dependency, and finally a PR on the "gui" repo bumping the dependency for both "validation-lib" and "server" (for testing etc.). That's before you need do deal with the circular dependency that "server" also wants "gui" for its own "I changed my server code, does the GUI work?" testing.
Better just to have them in a monorepo if they're logically the same code and want to share various components.
In any case, if you're nitpicking that example you're missing the point. The same would go for any number of other shared code you could imagine between a client/server trying that logically make up one program talking over a network.
If you had a `foo()` function shared between the GUI and the server (or two services on your backend, or whatever), in a monorepo your workflow is:
In a polyrepo where foo() is defined in a versioned, shared library your workflow is now: This problem gets even more compounded when your dependencies start to get more than one level deep.I recently dealt with an incredibly minor bug (1 quick code change), that still required 14 separate PRs to get fully out to production in order to cover all of our dependencies. That's a lot of busywork to contend with.
Update foo() Merge to master Publish shared library Deploy
So as you can see the only step added was to publish the shared library that would automatically update the version in all the projects using it. If you are really doing everything manually I can understand that this is a pain, but this has nothing to do with the monorepo / multiple repo distinction, this is a tooling problem.
What if updating foo() breaks something in one of the clients (say due to reliance on something not specified). Then you didn't catch that issue by running client's tests, now client is broken, and they don't necessarily know why. They know the most recent version of shared broke them, but then they have to say "you broke me" or now one of the teams needs to investigate and possibly needs to bisect across all changes in the version bump under their tests to find the breakage.
How is that handled?
(the broader point here is that monorepo or multirepo is an interface, not an implementation, its all a tooling problem. There are features you want your repo to support. Do you invest that tooling in scaling a single repo or in coordinating multiple ones? Maybe I should write that blog post).
I'll need a few more validation functions for each clients. I don't want to write+maintain multiple functions that do the same thing, even if it's just copy+paste.
It's "data" validation. So let's put that in the "data layer" repo.
We now have, at least:
- Server
- Web (GUI)
- Android
- iOS
- Data
- More clients?
We'll also have branches for each development task. How do we know what branch the other branches should use? One "simple" feature can easily spread over multiple repos. Does each repo refer to the repo+branch it depends upon (don't forget to update the references when we merge!), or we add a "build" repo which acts as the orchestrator?
Most PRs will need to be daisy chained - who reviews each one? Will they get comitted at the same time?
How do we make the builds reproducible? commit hashes? tags? ok, we now need to tag each repo, and update the references to point to that tag/hash... but that changes the build.
Well, I'm glad our code base is split over multiple repos because "scalability".
A good way to fulfil those requirements is to have the exact same function available in both places.
> If you split it you'll first need to have a PR on the "validation-lib" repo, then once that gets in a PR on the "server" repo, bumping the "validation-lib" version dependency, and finally a PR on the "gui" repo bumping the dependency for both "validation-lib" and "server" (for testing etc.). That's before you need do deal with the circular dependency that "server" also wants "gui" for its own "I changed my server code, does the GUI work?" testing.
The above is exactly why I am so firmly opposed to multirepo[0]-first. And it's really just a throwaway example: a real change would involve multiple different library and executable repos, all having separate PRs. And then there's the relatively high risk of getting a circular incompatibility.
This can be worth the cost, for organisational reasons. But until you need it, don't do it. It's very easy to split a git repo into multiple repos, each retaining its history (using git filter-branch). Don't incur the pain until you need to, because honestly, you're not likely to need to. You're probably not going to grow to the size of Google. Heck, most of Google runs in one monorepo, with a few other repos on the side: if they can make it work at their scale, so can you. And if, as the odds are, you never grow to their size, then you'll never have wasted time engineering a successful multirepo system instead of delivering features to your business & customers.
0: 'polyrepo,' really? https://trends.google.com/trends/explore?date=all&q=multirep... clearly shows that 'multirepo' is term.
Requiring multiple PRs to multiple repos to roll out one user-facing feature is fine, as long as your independent modules/projects are not actually interdependent (i.e. one of those PRs will not break another independent repo that lacks a corresponding PR).
But in that case you could consider the change to the dependency a single release. And ingesting it into another app a separate release.
I the difference of opinion is between developers who work on self-hosted "evergreen" products where the latest version is deployed, and others who work with multiple release branches with fixes/features constantly being cherry-picked.
Context switching really sucks. You should aim to reasonably avoid it
Who is the email being sent to?
What is the content of the email?
What data does the email content and recipient depend on?
What are you tracking on the email?
How is the email visually formatted?
All those things might be in different apps as the logic gets more complicated.
I agree that you should always start with one repo and split as needed, it's the MVR way (minimum viable repository)
But monorepo leads to tight coupling, and that is just as much a pain to work on as versioning, or two teams are simultaneously working on the same shared code, and you have not only merge conflicts, but conflicting functionality.
Medium: Please don't
For a longer explanation see https://medium.com/@nikitonsky/medium-is-a-poor-choice-for-b...
Alternatives better than medium: Wordpress, Blogger, github pages, plain html files.
There is a certain irony there which betrays one of the problems left unaddressed.
Need to change a function signature or interface? Cool, global find & replace.
At some point monorepos outgrow their usefulness. The sheer amount of files in something that’s 10K+ LOC ( not that large, I know ) warrants breaking apart the codebase into packages.
Still, I almost err on the side of monorepos because of the convenience that editors like vscode offer: autocomplete, auto-updating imports, etc.
Though, as you say and I commented below they’re not mutually exclusive. A wrapper or even an entirely separate service can exist alongside others.
One dark side of this is being able to “reach inside” other parts of the monorepo and blur application boundaries.
Monorepos are also a great technique for tackling large legacy codebases. When the rot is all in one designated place, it becomes easier to encourage good developer habits on new code created in new, separated repo(s).
Speaking from experience I've worked on a team operating through a monorepo project that came out real well. The codebase was mostly golang, so everything lived in the GO_PATH, but for the most part the typescript in the UI side of the repo didn't complain. Testing and code quality was a higher priority, as well, which may have contributed to its success.
I have also worked on a monorepo project that had minimal tests and automation, that soon grew monstrous and ultimately needed refactoring. That was a big pile of coffeescript, es6 and java that ultimately refactored into three different node modules and two microservices.
Javascript and its module packaging tends to conform better to polyrepo patterns. golang code all wants to be in the same place, and java repos have their own desired nested directory structures. These two languages tend to encourage monorepo design patterns.
Monorepo or Polyrepo, the correct answer is whatever works for your team and task at hand.
I'm seeing these two things conflated in this thread.
In some cases, this could be two separate backend projects where you want to re-use the same deployment pipeline.
Often, I find that API wrappers are something that I share across frontends and backends in the JS world, so it often makes sense to separate my projects into:
- backend
- frontend
- common
In Typescript I really like this pattern and can namespace shared types so that it’s very clear to the future reader that this type is probably used outside of the current context.
So, to reply to your comment — I think the term “monorepo” can encompass a lot of different project types.
I think Dan Luu covers the bases quite well here:
https://danluu.com/monorepo/
Monorepos are going to be mostly challenges around scaling the org in a single repo.
Polyrepos are going to be mostly challenges with coordination.
But the absolute worst thing to do is not commit to a course of action and have to solve both sets of challenges (eg: having one pretty big repo with 80% of your code, and then the other 20% in a series of smaller repos)
It's like thinking OOP or functional programming is going to solve all your issues... I mean, in some limited cases they could, but realistically you're just smooshing the difficulties around and hopefully moving them to somewhere where you are more able to deal with them.
FWIW, I've worked in a many-repo org and it sucked worse than huge companies with monorepos and good tooling, but I'm not going to make some blanket statement because it depends on the specifics of your code/release process/developer familiarity etc.
For example: If you're stuck with a TFS monorepo (you poor soul), you actually get to deal with both problems to some extent, since TFS doesn't enforce that you check out the intire repository at once.
This can have very "funny" situations because someone forgot to checkout new changes in some folder. OTOH, at least for releases, you can remedy this by using CI everywhere.
Maybe I am way out of my element here, but is this a common problem? Do companies with only “hundreds of engineers” really have “hundreds of millions of lines of code”?
I was one of about 900 engineers at a financial company of about 1500 employees at the time.
I don't honestly know how many lines of code there were across the company, but I imagine it easily exceeded 100M. It took us a full week to do a full recompile of everything. We had no CI... Was always a problem approaching release time.
Can you say which company it is, or give a few more details? I'm fascinated to know which financial company can consist of 60% engineers!
It's definitely the case that a mega monorepo doesn't, in practice, have the atomic commit property. E.g. once you add owner files and separate code reviews, you're in for a world of hurt. Case in point, Google developed an internal tool to split cross-cutting CLs into manageable pieces, wrangle all the owners and approvals, presubmits, etc, and then submit the CL piecemeal--i.e. not atomic.
Chromium uses a different model. It just DEPS's in other repos at pinned versions. That has a whole other set of problems.
It's not quite so black and white. It's true that repo-wide refactorings often get carved into little changes and so aren't made atomically, but they're the exception rather than the rule. Any small change, e.g. changing an interface and the 5 callers of it, _can_ be made atomically. And changing code that's reused a small number of times is a far more common case then changing core libraries the whole company uses, so atomic submit ends up being hugely valuable.
> Because, at scale, a monorepo must solve every problem that a polyrepo must solve, with the downside of encouraging tight coupling, and the additional herculean effort of tackling VCS scalability.
Right.
But you have to get to "scale" first (as it relates to VCSs). Most companies don't. Even if they're successful. Introducing polyrepos front loads the scaling problems for no reason whatsoever. A giant waste of time.
Checkmate! I didn't even need a snarky poll. The irony of that poll is that it clearly demonstrates his zealotry, not other people's.
If you split your repo up from the get go, the worst thing you can get that you'll have to assemble multiple distinct, well-encapsulated (in terms of project structure) things into one. In Git, that could lead to multiple root commits, but that's about it.
Most of the time they're just annoyed.
One side effect of every successful business are annoyed worker ants that are sick of dealing with growth problems. I've been there. I know how annoying it can be.
Personally I've found comfort in embracing the chaos and learning to manage it responsibly. No dogma. No absolutes. Know how to do monorepos well. Know how to do polyrepos well. Learn the pitfalls of both. Don't assume other people are stupid zealots.
> The worst case is that the engineering team spent more time working on “well encapsulated projects” than on the most important project for their business
I'm not really sure how I should read this. Don't you use your repos to solve business problems? Why should that change because of the repo layout?
Why would you put in a symlink? You could just provide a path to the actual component and import it into your project.
> the worst thing you can get that you'll have to assemble multiple distinct, well-encapsulated (in terms of project structure) things into one
When you have multiple repos, you also have multiple versions and releases of things. Now every team has the following options:
1. Backport critical fixes to every version still in use (hard to scale)
2. Publish a deprecation policy, aggressively deprecate older releases, and ignore pleading and begging from any team that fails to upgrade (infeasible - there'll always be a team that can't upgrade at that moment for some reason)
You also have to solve the conflicting diamond dependency problem. This is when libfoo and libbar both depend on different versions of libbarfoodep. It's even more fun in Java because everything compiles and builds, but fails at runtime. So now you have to add rules and checks to your entire dependency tree - some package managers have this (Maven), others don't (npm IIRC).
Where do I need to put the path again? Ah what the heck, I'll just add a symlink inside a folder that's already somewhere in the build definitions.
I don't know anyone who has abused Maven or Cargo or Go like this. And I don't imagine Visual Studio Solutions for C# are used like this.
Is there an underlying disagreement based on JS/Ruby/Python scriptish coding (which creaks when a lot of developers work on it) vs C and C++ (which have astonishingly bad build system stories) vs big-iron languages that don't sweat when in a monorepo.
At my workplace, we've just been cleaning up a whole bunch of instances of exactly that anti-pattern. Except that it's obviously not symlinks (which require specific user rights on Windows), but links to external files in VS.
Same problem, though: They're easy to introduce and a pain to deal with later on.
I've seen that with polyrepos as well: The entire project would require you to clone the individual repos into a specific directory structure so that things would work (no, not even submodules).
Sure you can. The difficulty of doing so depends on many (many) factors. If your team does their job well then the costs won't be immense. It might be annoying, but not that hard.
Speaking in absolutes or platitudes solves nothing. Sometimes monorepos make sense. Sometimes polyrepos make sense. It's entirely dependent on what your company does.
E.g. if relying solely on a package manager to to keep coupled things in lock step, you need make sure that version numbers are kept up to date for every little change made to every library.
You can easily end up with a situation where someone in another team makes a small change but doesn't change the lib version number. That's a people issue but it does happen.
You can get round that by using a repo SHA but now you two things to keep up to date for every library.
Like wise you'll have to be diligent in versioning APIs. Anecdotally I've found it easier to keep things in lock step when in a single repo and using a single pull request for each story than I have where separate teams have to keep separate repos in sync.
Both work but the monorepo approach worked better for the projects I've worked on. It just lead to less moving parts and more repeatability when there's a single SHA to watch.
I also have been luck enough that I haven't worked on a project so large that we couldn't build a monorepo on a single machine with "normal" build tooling.
It's not quite as simple as that. You'll need to avoid rebuilding the entire repo for every change - using something like Bazel. This means your build tooling has to be replaced entirely, which is a non-trivial task and not something your devops/release engineering team will thank you for.
For any 3rd party libraries used by your projects you need to either ingest those projects into your monorepo and update forever. Or keep npm/maven/pip/gem/whatever around just for managing 3rd party dependencies (+ whatever system you use to front the main language package registries, because of course you're not talking directly to NPM/Maven central are you? What if they go down or do a leftpad?).
I think either system - monorepos or polyrepos - works fine; just pick one and stick with it. Monorepos will probably give you better velocity starting out. Past a certain size, which most software shops will never hit, the already-available tools lend themselves better to polyrepo. And more devs know polyrepo tools (eg. Jenkins) than the corresponding monorepo solution (eg. Bazel). Things might swing in favor of monorepo on the VCS side if Twitter/Google/FB ever open-source their stuff.
It's their job. If they actively don't want to do work then you probably made a hiring mistake somewhere. By that logic what DevOps really wants is to the company to shut down since then they'd have none of that tedious work to do.
Sure it is their job, but is isn't an easy job and there are many opportunities for things to go wrong. It might or might not be the right choice for you, but don't overlook how hard it is.
Note that the above applies for going in either direction.
If I was moving many small repos into a single mono repo then I'd do it one repo at a time. Presumably your small repos are independent entities so there's no reason to do a single massive switch. Transition each repo to the new build system inside the existing repo. Once that works then you can transition that repo into the mono-repo and tie together the build systems. No need to stop releases, no massive chance of everything failing, no weeks of debugging while the world is stopped, etc. Rinse and repeat until everything is moved over. Process becomes more optimized and less error prone with each repo that is moved over.
It's harder to make a business value case for this type of change - there are only vaguely worded promises of "improved developer velocity". Contrast that with a change that automates or makes faster some aspect of building and releasing - a professional release engineering team would be all over it because they can demonstrate value in that work.
In any company beyond 50 people, there are multiple engineering managers, directors or the VP of engineering that will need to back this initiative to make the release engineering team do it. It's really not as simple as "dump all the code in one repo". I'm speaking from experience.
That has not been my experience at all. At a previous employer, we did exactly that with a multi-language library. In fairness, having multiple languages enforced fairly good directory structure in our single repo. But isn’t that the real point: good structure makes life easier, period. The thing is, going into a project you often don’t know what the right structures are yet. Creating a new repo for each component you think you need ossifies those choices, making it far more difficult to walk back on them later on (first because you may not even see the architectural mistake, second because the maintainers of that component will have an investment in its existence).
> You can always just put all your small repos into a big one.
In my experience, that’s harder, precisely because over time so much tooling has been built into each repo to manage builds, images, deployment &c.
I’ve worked in monorepos & I’ve worked in multrepos, and so far my experience has been that monorepos enable faster velocity and more-maintainable software. I’ve not (yet) worked at Google- or Facebook-scale, though, and I’m completely open to the idea that at that scale a team really does need lots and lots of repos, and tooling to stitch them all together.
Nobody would choose to drag around every historical afterthought in the development sequence of long forgotten software going back three decades that no longer builds with current tools, just so they can work on a small library off in a corner. Software is getting written and added to these monorepos at a much faster rate than hardware and networks are able to hide the bloat-upon-bloat growth of them.
"Hey, what does this server do?"
"No idea; it hasn't been touched in years. What's deployed to it?"
"Some 'foobar-ng' thing, never heard of that. Says it was last updated 5 years ago. Pull up our source repo for that package, will you?"
"Hang on, we've got like 30 services with names containing 'foobar', let me find that one . . . oh god. You don't wanna know."
"Fuck it, I'm just gonna shut this server down and remove that ancient, dead, busted package."
"The main billing system just broke! What did you do?!"
If it doesn't work then it should be deleted. If it's still running somewhere then it should be maintained. Presumably you have a CI system so the monorepo actually requires everything in it to build.
In my experience, it's polyrepos that allow for dead and un-maintained code to just sit there for eternity. You forget about that unused repo right until the moment the service it deploys to (if you can track down that dependency) needs an update or goes down. Monorepos can more easily force system wide CI that checks for broken dependencies or other issues.
In the embedded world supporting software for 30 years is not unheard of. We avoid it, but it is in the back of our mind that someday we might have to release an update. Fortunately 30 years ago nothing was internet connected, we are worried that we might be releasing security updates for our current products 50 years from now...
.....because, as the author directly stated, the type of repo has nothing to do with the product being successful. So stop bikeshedding, pick a model, and get on with the real business of delivering a successful product.
The post just reads like some opinionated piece for traffic. The author has never even used a monorepo as far as I can tell, so can only argue from one side, the best one ever used: polyrepo. Then goes on to list 'theoretical' benefits and the downsides (which should also be theoretical if having never been used) of monorepos. It concludes with "The two solutions end up looking identical to the developer. In the face of this, why use a monorepo in the first place? Please don’t!" implying that 'Google, Facebook, Twitter, and others' do it for no benefit.
Ready to commit? Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit. (I'm having flashbacks to Clearcase already.)
Need to roll back the last two commits you made? Sure, that takes two seconds--oh, wait, another team made multiple commits that got interleaved with yours. Have fun cherry picking the files you want to revert.
Of course, I'm apparently a curmodgeon, because as soon as someone starts talking about running a find/replace globally across multiple projects, I want to grab something sharp.
https://code.fb.com/web/rapid-release-at-massive-scale/
That's only true to the extent that the statement 'places that use multirepos also tend to be full of people who feel entitled to f#ck with stuff across the entire codebase' is.
Bad colleagues can cause damage either way. I appreciate that with a single repo, bad colleagues are at least forced to have passing tests after their changes, rather than leaving it to me to pick up the pieces.
> with a single repo, bad colleagues are at least forced to have passing tests
Passing unit tests, big whoop. Maybe sometimes light functional tests. Integration/system/stress tests that have to run across an entire cluster for non-trivial time to get meaningful results and thus can't easily be kicked off from a commit hook? Not a chance. The coverage that results is no better (or even different) than what you'd get with separate repos.
Fwiw, this hasn't at all been my experience with this kind of thing at Google. Certainly developer experience teams and language teams will make broad changes that affect everyone but
Those changes are trivial: the change maker has to make a convincing argument that the change can't break anyone, if they can't, it will require local approval from the owner.
They don't happen all that often: ~once a month for any given leaf directory.
They require a special approvals process where you have to answer why the change is necessary, why the churn is worth it, and why it really is safe, and convince a group of approvers that this is the case.
If they do break something, the change-maker has to roll it back and then either reconsider, or fix the issues and try again.
>The busybodies and style pedants can go jump in a lava lake.
Consistent style is important. Among other things, it means that unmaintained code (ie. the stuff written 3 years ago that does its job) still gets updates to be consistent with everything else, so that you don't end up with ancient spaghetti that breaks every modern style rule (I mean you still get that, but less). It also allows deprecation. If there's a core library that does something "wrong", they can deprecate the badly behaved stuff and make sure everyone is off of it: if its not in trunk, it isn't being used.
>The coverage that results is no better (or even different) than what you'd get with separate repos.
This is untrue. If I'm the `core` team who maintains the `core` repo that everyone depends on, and I make a change that breaks you but my tests pass, you don't know it. Then when you version bump, fixing yourself is your problem. With a monorepo, it's my responsibility to fix you before I can make my change.
Many things are important. Some things are more important than others, and I'd say that "not making a thousand developers' workflows more cumbersome" is higher on the list than style issues.
> If ... I make a change that breaks you but my tests pass, you don't know it.
So don't do that. Open-source projects deal with this exact same issue across repos and owners all the time. There are responsible ways to do it. Mostly they involve learning to communicate as peers, with respect, instead of "core" teams imposing their neophile opinions on everyone else. If we're all at the same company, regardless of whether we use one repo or many, there's no excuse for you not to validate your changes against other groups' tests as well as your own. Diligence does not depend on a particular repo structure.
I'd argue that in the long run, not being able to update dependencies because they broke you is going to be much worse than them fixing the incompatibilities for you.
Either way, you need people to act like adults and communicate, but the multirepo problem is worse.
If they merge cleanly, it's not an issue. If they don't, you need to fix the merge conflict. The work you need to do is proportional to the number of merge conflicts, which isn't special to monorepos.
> Need to roll back the last two commits you made? Sure, that takes two seconds--oh, wait, another team made multiple commits that got interleaved with yours. Have fun cherry picking the files you want to revert.
Again, only an issue if the changes are on the same files. It can be a bit of a pain to revert a stack of diffs, but if it's just a random commit with no other relevent commits to the file, very easy.
I used to work on a large team at Cisco Systems that used Clearcase. Clearcase does not do merges. If anything has changed in master, you have to check out again, which obliterates all local changes.
(I have never met a developer who liked Clearcase. It was built to simplify life for system administrators and to tick the right boxes for management, not to be useful for developers.)
My general VCS experience is that you can't roll back a commit without also rolling back all subsequent commits, related or not. I'm glad to hear that modern systems have fixed that. (It looks like even Subversion does that now, cool!)
If you do - then it's a good thing that you see there's a conflict right away, rather than notice versioning problems between your separate subsystems in integration testing (or even worse - production).
And if you'd like to learn more about how monorepos work in practice there's a couple of papers: https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... and https://ai.google/research/pubs/pub47040
(Also worth reading: http://danluu.com/monorepo/)
If their commits didn't affect you, then … it merges cleanly, and you don't care. If their commits do affect you, then … now you know. With multiple repositories, you will have no idea that they broke something you rely on.
Even better, with a monorepo, if they break something your existing code relies on, they' have broken the build, and they have to fix it. With multiple repos, another person or team is free to break things you rely on without even knowing it, and you won't know it until you update your dependencies hours, days, weeks or months later, wondering why everything is suddenly different.
If another team's work is tightly-enough interwoven with yours that their daily commits affect you, then you're all on the same team (and/or your architecture needs work).
Outside of mega-corps, usually only a few people (couple of teams at best) are working on a given section of code at a time. Coordinating changes between maybe 12-15 people is quite feasible. Most of the time it's enough to keep code nicely segregated by paths - something like $team/$XXX or $scope/$team/$YYY should work.
On top of that, you need two things to enable a nice workflow:
* server-side (or otherwise programmatic) merges/rebases only; no human should ever need to push to master directly. That's the job of the [pre-]CI machinery.
* comprehensive pre-merge testing before the server-side merge. Do all your development in branches, and have bot+CI test _all_ unmerged branches against MASTER+YOURBRANCH on every push to YOURBRANCH. Because no human is involved when merging to master, the tip of the master has not moved due to external factors. Also, rerun tests if master has indeed moved thanks to bot having merged another branch ahead of yours. Fix any test breakages in the branch.
To make the second item work, you have to realise that the testing steps need to be rapid enough. Usually it should be enough to test the new-to-be-branch against merge problems, code convention errors and have a run through all the usual unit tests. Most of the time you can leave any larger cross-service or integration/end-to-end tests for code that has landed in master already.[ß]
Once code has landed in master, CI can pick it up and produce the release artifacts. The tooling needs to be good enough to know how to avoid useless work (doing useless checkouts and running tests against dozens of branches will get expensive). It also needs to provide very good and easily actionable feedback. You want clear test results, with quick jumps to failure(s) and robust logging.
To give some context of where the above is coming from... We have a monorepo with more than 130 projects, and about 7% of the codebase changes in any given month. (Except for December. Understandably.) We also clock more than 40k pre-merge test runs a month. Once a merge request has been approved, it is often available for shipping in 20 minutes.
When builds against master never fail due to code conflicts, development velocity is maintained better even with multiple teams working on the same piece of code.
ß: this is a tradeoff between development friction and test coverage. A sufficiently thorough integration test can take anything from a just a few minutes to couple of hours. You want to run them against batches of changes, and in case of introduced failures, incentivise teams to put in new unit tests to cover as much of the uncovered error scope as possible.