* Commits that go across components/apps are atomic.
Cons:
* When it gets big, those features matter less
* Churn from other dev's stuff gets in your merge/rebase work.
* 'git log' and other commands can be painfully slow
* Mistakes in the repo (e.g., committing a password) now affect many more people.
Use for highly-coupled source bases. Where releases together and atomic commits are very useful. Every other time, you're probably better off splitting early and taking a little time to build the coordination facilities needed for handling dependency versioning across your different repos.
Note: I used to really prefer monorepos, but I've had some time away now to take a better look at it. Now they feel like megaclasses where small things are easier (because everything is in one place) but large things are way harder (because there's no good split point).
Good list. I’ll add: If your company does feature freezes or general code freezes around holidays or corporate earnings, it can be harder to get “run the business” changes approved during these periods in a monorepo. So even in monorepo companies it’s common to have a separate “infra” repo for network and cloud configuration, sometimes for auth too. Makes the politics easier.
> Churn from other dev's stuff gets in your merge/rebase work
Wouldn't other people' work only cause issues if they are changing the same files, in which case conflicts would happen even if the work is spread in multiple repos?
Good point. I've gotten hit by this in monorepos but I probably just tried to merge instead of rebase. Both are crazy painful on large source bases.
It still takes forever!
I should replace that with 'most commits in history are irrelevant to you, so you have to dig smarter to understand what's been going on with your components'
Invest in using a gui merge tool. It makes this process 10x easier. Also your complaint is unrelated to monorepos vs. multiple repos.
Additionally rebase or merge won't make the conflict issue go away. I recommend companies stop using rebase as it produces an in accurate history of what's going on in your git history.
I'm a magit-er now. In multiple repos, the history for one repo is limited to that repo, which is usually more topical for people working on it.
If I've got changes on an old version of the repo, and try to merge against a new one, I find conflicts in the changes others have made to the same repo -- I have the 'before' version, they have the 'after'. I frankly don't investigate too much when it hits.
As for history, I love rebase for two reasons: (1) I can squash mistakes in my history so they're gone forever. (2) In reconciliation I'm watching my changes filter through the history of the repo. When I understand what's up, this is great. When I don't, it's a catastrophe of me re-applying changes that haven't actually been applied yet, then conflicting with their application later. Ugh.
1- You finished your work, so you pull from the central repository, and merge your your branch to the master
2- You do your merge work, test it a bit and commit
3- Now, you push and oops, another team did the same, you are now left with to heads
4- You merge or rebase the two heads. Probably a simple task, but you may still need to run some tests again. And if you are lucky, you are done, otherwise, back to step 3
Also, if you are rebasing before push, you should be able to keep a clean history. However, if you are merging, and there are good reasons for a "merge only" policy, you are going to have a mess of merge commits every time the previous situation happens.
That's something you can work around with good management. But the more freedom you give individual teams to push to the common branches, the more that situation will arise, the more you try to control access, the more chance you will have for teams to go in different directions, making the merges infrequent but tricky.
When you say "But the more freedom you give individual teams to push to the common branches", what do you mean? I've typically only seen CI merge to master, and disable pushing.
Imagine you're in a branch for large project P. You want to merge to trunk but there are conflicts. The upstream library project L was changed in your branch but not in merged down by the submitter.
Even if P simply relies on a binary of L that is already in the artifact store, you still have to deal with merging this code down when normally you wouldn't.
In fact, the much bigger issue is that you now must deal with the double edged sword of updating the entire org when a shared dependency must be updated.
Perhaps you even completed that task in your project branch. Now you get to deal with merging into every project in the org that was touched.
Github (might depend on settings) refuse to merge if your branch is out of date. Rebasing on master re-triggers CI/CD. Might cause lots of waiting for this case...
The big advantage of a monorepo is that dependency on changes to a "library" or component that gets reused in multiple places can be made explicit. A user can commit a change that safely modifies a library or component in a non-backward-compatible fashion AND changes all the places that use it. This is far simpler than the alternative of releasing a non-compatible version of the library/component and maintaining both in parallel for a period while the users transition.
The big advantage of separate repositories is that the work done on them is very clearly marked as independent. It becomes easier to take one component out and utilize it elsewhere... but at the cost of having to deal with dependency management. Dependency management is simple and automatic in easy cases and really terrible in hard cases (like when two components you use mandate mutually exclusive versions of a third component).
So I think the driving factor in the decision is the tradeoff between the degree to which you are willing to invest in careful packaging of each component (and the corresponding dependency management), versus your willingness to force everyone and everything to utilize the monorepo.
(Notice I didn't mention performance issues. I really don't think they are important enough the control the decision.)
I don't have a lot of experience with monorepos but I've found them very useful for:
- Simple projects with a server and an SPA component - frontend and backend code for the same feature is on the same feature branch, can be tested and reviewed together.
- Projects with a couple of microservices that share some common libraries - these libs can be directly referenced by the microservices instead of being published on an internal package server. This of course has its drawbacks too but overall, the benefits have outweighed them.
It's a tradeoff. Yes, you can only remove something if all callsites are gone, need to take care of breaking changes etc, but on the other hand you have clear insight into that and individual components can't as easily drag behind and require you to keep old versions around/maybe even backport fixes/...
And ideally a monorepo workflow gives you the tools to a) find all callsites, b) run tests to ensure your changes won't break anything, c) apply changes in lockstep. Those tools are an engineering effort, but the equivalents for a large multi-repo setup are too.
It would also be interesting to hear about the tooling available/recommended for maintaining monorepos.(ideally FOSS tooling, but good proprietary ones as well)
Thanks for sharing that. It's mind-blowing that Google runs everything from a single repo. Most places that I work with create several repos for just one project.
Yeah, it was a significant infrastructure cost. I heard at one time that the single largest computer at Google was the Perforce server. They ended up completely re-writing it (called "Piper") for scaling.
This is sort of what I was alluding to with my comment about Google having different concerns than many other companies. They can afford to dedicate a number of engineers to maintaining a monorepo system and then re-writing when it doesn't scale. That said, I personally believe that there are a lot of benefits to monorepos, and I think those tradeoffs are worth it for other companies too.
Mono repos can be useful if you have different services that are highly dependent on each other. This lets you package releases in a simple way by just tagging the repo. That tag/release gives you the exact version of all the components that you need.
But CI becomes substantially more complicated. You have to figure out with every commit what actually changed and based on that change what needs to be tested and built. You don’t want to run a build of all components if you only changed a small piece of one component.
Google, Twitter, and Facebook have open sourced their solutions to this. Bazel, Pants, and Buck. Each have some rough edges but I'm currently migrating a bunch of stuff to Bazel and the workflow and tooling for it is pretty amazing.
We just build and test everything every time. Maybe it costs us more $$$ on CI, but it costs far less in developer time. That being said, I mostly work at startups and larger companies will certainly cross a threshold where this isn't feasible.
>>> larger companies will certainly cross a threshold where this isn't feasible.
Nope. Worked at the largest US bank that happens to be mono repo. They had compute grids with thousands of cores running tests 24/7. It's not sizable compared to the costs of the developers writing and maintaining software.
For me the biggest pro and con are the same thing. With a monorepo your respective projects' codebases become tightly coupled.
Why that can be a bad thing? Good software engineering practices tend to be associated with loose coupling, modularity and small, well-defined interface boundaries. Putting each project into its own repo, encourages developers to think of it as a standalone product that will be consumed by third-party users at arms length. That's going to engender better design, more careful documentation, and less exposed surface area than the informal cross-chatter that happens when separate projects live in the same codebase.
Why that can be a good thing? The cost of that is that each project has to be treated as a standalone project with its own release schedule, lifecycle support, and backwards compatibility considerations. Let's say you want to deprecate an internal API in a monorepo? Just find all the instances where it's called and replace accordingly. With a multi-repo it's nowhere near as easy. You'll find yourself having to support old-style calling conventions well past the point you'd prefer to avoid breaking the build for downstream consumers.
If a monorepo is generating multiple binaries, there’s no reason it can’t use separate compilation units. In many languages, separate compilation units give you that arm’s length separation you need, without introducing the problem of making breaking changes that you can’t reasonably detect prior to commit, because the code is used in some obscure module you don’t even have checked out.
For that reason, monorepos favor the new programmer, which makes it easier to ramp your team size.
The only legitimate reason I have encountered so far for not using a monorepo is open-source dependencies that the company maintains patches to/contributes changes to. In this sort of situation, you'll want to have them in their own repository so that you can go from "foolib v5 plus our changes" to "foolib v6 plus our changes" without too much fuss.
Things that change together, go together.
Monorepos prevent people from having to dig in all imaginable places of your version system to find all pieces of your application.
At the same time, if you have 10 micro services which are accessed by 1 frontend, it may be a little bit messy to keep all that code in the same place.
Common sense (which is not that common) is what should be used to determine. Ask yourself some questions:
- Does these repositories change together ?
- If the code is not together, one repo should be linked to the over via what (git tag ? version number ?)
- All pieces of the application are contained within a context ? Or they can be split (example, 10 micro services and 1 frontend, but 5 of those micro services are also used by another frontend in another project).
The main goal is to make things easier, some PoC and experiments may make it more clear because it really depends on the situation.
> if you have 10 micro services which are accessed by 1 frontend, it may be a little bit messy
I see another differentiator here. If I have a typed API interface (both endpoints and payloads), and I publish a (generated) client library for the frontend, then that frontend will only use the new API version if it is also upgraded to use the new client lib. Here I think multiple repos are more suitable because the pieces of code can improve independently.
In case the API is not typed/versioned, the case for a monorepo is much more important: the change to the API on the BE should come along with the change in the FE.
Thus typing/versioning (on specific barriers) helps with breaking up a code base (along those barriers).
Even with untyped, I'd argue that you want it to be in your face that you cannot change the front and the back ends together. Getting them in one commit will not get them in one deploy.
In mono/mulirepo debates, I frame this as multirepos make hard, scary things hard. Monorepos make it easy to introduce changes that break APIs at deploy time, but are hidden by the atomic commit actually being OK.
I think API versioning is orthogonal to how you organize your code. If you have multiple repos, then you still have that "atomic commit" where you update the version of the client library and adjust the code to handle the new semantics. It then breaks when you deploy it to production because the server code isn't deployed yet (or vice versa).
Basically, if you make RPCs, breaking up your repositories or combining your repositories doesn't eliminate this problem. It's a separate problem that you have to tackle. The reason it doesn't come up as often as it should is because a lot of code is liberal in what it accepts, and most changes are strictly additive (i.e. it's "add more information to the response", not "remove information from the response", because we're generally pushed in the direction of making our applications do more). But it is something you have to attack head-on. No tool will auto-fix the problems for you.
I think they were referring to a false atomic commit to the front and back end. In a mono repo, that can be one commit. However, it has to be two deploys, so it should have been two commits.
At least, that is my argument. So, I could be reading my view into it. :)
> API versioning is orthogonal to how you organize your code
They are, but they aren't, because we have to basically increment two counters when things change. If you want to be statically correct, you now have a distributed transaction to update both systems.
There is some path through a call graph (in-proc and rpc) that is statically valid. Then there are duck-typed results that are only additive, which you are describing. Provided the system can handle this (dictionaries over structs). You see this in protobuf and other static centric serialization systems. They start to encode dynamic information using the static building blocks. GCCs internal IR is like this. I think what you are advocating for only applies to the domain of microservices, loosely connected using languages and formats that are open under union. Other systems without those properties will not fare so well.
The folks using the monorepos probably don't disagree, but having atomic commits with hashes makes the identity function a useful crutch. I believe there are other solutions where the VCS could actually make this problem a non-issue.
> Basically, if you make RPCs, breaking up your repositories or combining your repositories doesn't eliminate this problem.
I kinda agree with this.
But as someone who has always worked in a microservice environment kept by different git repos I don't understand the benefits of a mono repo. I can see the benefits of a mono repo in a massive code base that produces a binary at the end (i.e. windows). But could you elaborate more on the benefits of a mono repo in the case of many small microservices?
>At the same time, if you have 10 micro services which are accessed by 1 frontend, it may be a little bit messy to keep all that code in the same place.
Why is this messy? Using folders to separate your code has no intrinsic organizational difference than using an entire repo. There's no issue in throwing every app in your company under one repo and just using a folder to organize your stuff.
Separating code into multiple repos makes one repo less aware of changes in another repo. It actually makes things harder and worse.
This is the same issue with monoliths and microserves. You don't need to separate your code into several computers just for organizational issues. You can use folders to organize things.
Mono-repos are either a shortcut to avoid release management and dependency management, or are a way to manage development at scales approximating Google's.
First, for almost all companies copying non-selectively what Google does is harmful - you are not Google.
Second, if you are a small team, your code only produces one binary that is shared across team boundaries, then you might be able to do with a monorepo. But if you are refactoring pieces of your code into libraries (you probably should be) and sharing those libraries for use by other teams in your company (you maybe should be), then you probably shouldn't use a monorepo.
However, converting may be a painful process. Using multiple repos, one per shared product (library, SDK, web site, documentation package) is harder to manage because you need to manage your dependencies much better and pull in the correct versions for the current product.
You also will need to communicate changes on products better across your teams, ideally with automatic notification of new releases that include a changelog and examples of new/changed features.
As part and parcel of the above you'll need to have better release management, including a release repository. What you use will depend on your environment, the language(s) your team uses, and your budget. There are some good Open Source dependency repositories out there that can be used to accomodate NPM-style dependencies, JAVA dependencies, and others (e.g., Artifactory).
In summary migrating away from monorepos is going to mean: investing in good DevOps people and giving them what they need to create/install/manage good processes; learning these new processes and dealing with the added work they impose on developers. But, it also will likely give you better products and over time both speed up development time and reduce defects (in part through more well-defined API touchpoints).
Avoiding release and dependency management is not a shortcut - it's avoiding a needless detour. Management is overhead, overhead requires resources like the 'good devops' people you mention. Many companies and teams are lean and must find ways to work more efficiently - mono repos are simpler so they are the default option. Only if you have a really good reason to support multiple versions of software simultaneously or have real independent teams with boundaries should you consider making your process more complicated by breaking this up.
"The first rule of distributed systems is don’t distribute your system until you have an observable reason to."
> Mono-repos are either a shortcut to avoid release management and dependency management
In a small company, release and dependency management are pointless busy work. And in larger companies, you have enough labor to manage monorepo tooling.
> Using multiple repos, one per shared product (library, SDK, web site, documentation package) is harder to manage because you need to manage your dependencies much better and pull in the correct versions for the current product.
This is the pointless busy work I was alluding to. See also: the diamond dependency problem. Meanwhile the benefits are unclear.
Multi-repos makes sense for entirely disconnected projects or products, or for collaboration in the open-source world. Most companies aren't going to run into significant issues with scaling a monorepo and they'll save a ton of work.
Simplicity and being able to checkout and work on single folders, this is not directly related to the repo being mono but usually you wouldn't use git as your vcs when working with mono repos.
Do people big monorepo's always clone the whole thing? Our CI/CD (TeamCity) always seems to check out the whole repo even if you want just one sub directory.
We are breaking our monorepo into 3 kind-of mono repos based on release cadence. A repo for each of two large applications plus a third repo for shared library code. We import the share library code onto the app repos using a git sub module. Has anyone else used git sub-repos to import common code?
Git submodules have a comically bad user interface imho but they work really well so I use them despite this. As with git in general the core functionality is solid. There is also something to be said for making as much of core functionality without resorting to another piece of infrastructure. People can use whatever front end they prefer on their machine but in a pinch the core git package is all you need.
We've used submodules pretty extensively in the past but we're working to get rid of as many of them as possible. A friend and former colleague liked to say they were the worst solution... with the exception of all the other solutions. Practically speaking they are probably more confusing to new developers who haven't had a lot of experience with them. We would often run into problems that stemmed from someone making a change in the submoduled repo and then forgetting to update the ref in the parent. Or people would navigate to the submodule to do the work there rather than checking it out separately, and then be confused over the detached HEAD status and check out a branch (not a problem, necessarily, but confusing to some extent). For some of our dependencies we've gotten rid of submodules in favor of hosting internal pypi packages.
let’s split our project into multiple libraries so it can be reused within the company or even better if we put it on GitHub everyone can use it.
After a few months you notice nobody within the company cares about your nicely versioned libraries and on GitHub you get more and more complaints that the libraries are very limited and need more features.
After that you merge everything back into your monorepo and try to forget all the time wasted on git bureaucracy, versioning, dependency handling and syncing changes between repos.
code reviews and ownership boundaries are less clear on a large code base in a mono-repo. Depending on your team culture, this will either be a pro or a con. While careful design of the build system can mitigate build times, there is likely to be some disagreement on basic practices such as "how to launch a service" or when to use spring vs. guice vs. rails that will be tougher than simply letting everyone go their own way.
Refactors and cross-service deployments can become much simpler in a MonoRepo.
Use them whenever things are one project, only split them up when it's legitimately separate projects.
At work I set up a repo with our back-end code in one directory, the front-end in a directory, and the ansible playbooks/other deploy stuff in another..
This allows us to automatically create servers for every branch because we know that everything goes together. Working on other projects where they have all of these separate is a nightmare. Especially when they use microservices and each service has its own repo and each library has a repo... I want to smack anyone who ever thinks that is a good idea
What makes Amazon's multirepo setup "tick" isn't the multirepos themselves but the coordination mechanisms around it (version sets, brazil, and pipelines). whereas bazel/buck are open source and work ok enough to scale monorepos, i dont think there are the equivalent tools out there for multirepo and it shows in the comments for people with multirepo pain.
EDIT: that being said, I do think multirepo is better, but only when you have the above tools to manage it.
Agreed that Amazon build and deploy is vastly ahead of bazel/buck. That said, git just works so much faster on a smaller repo. There's significant overhead trying to use git or hg on a huge monorepo that really takes a toll on development.
1. I've only used monorepo professionally once in a big org and it was SVN based
2. I'm going to look at monorepos from the perspective of git and github
The main argument I can think of in favor of monorepos, is to maintain the cohesion as high as possible between different parts of your system. For example, if you want to make a change to the load balancer regarding TTL, you can also go and make the same change in your API and your mobile clients and in the end you create one single PR and you have your tests run against that single revision.
Compare and contrast the same scenario in the traditional multi-repo approach: You make your changes to your `infra` repo, then you make the same change in your `api`, `android-client`, `ios-client` repos and in you create 4 PRs, that need to be reviewed at the same time. Which one would you prefer to do?
Potential arguments against are:
1. Too much noise - If multiple people are working on the same repo, you'd be getting emails for PRs for parts of the code that you may not care about.
2. Longer clone/pull times - In the same vain as before, the initial `git clone` and maybe every `git pull` after that will be bringing in a lot of code that you may not care about, increasing the time of each operation and increasing your frustration.
3. Access control - How do you limit who has write access to which part of your repo? AFAIK this isn't possible in git but it may be possible with codeowners in github, I don't know.
4. Organization - How do you structure your monorepo? Do you know in advance how many components it's going to have? How is a restructuring going to affect your history and/or dependencies between different components of your code?
5. History and code sharing - With a monorepo it's more difficult to share/open source just part of your code and keep the history intact.
Having said all that, I think the monorepo is a good match at a service level, not at an organization level. I know Google and Facebook have gone all in at an org level, but first of all they don't use git and second, they have the luxury and the resources to make it work for their use case. At a service level you should have a good idea of your boundaries, your applications and your infrastructure and assuming a relatively small engineering team (~10-12 people) eventually everyone would be confident enough with all parts of your system.
Personally I really like the idea of having one hash define the entire state of the world. I have a few common utilities that I use across my projects and I find it too annoying to keep them all up-to-date between different repos. I hope that eventually I will find some time (or be annoyed enough) that I will converge everything in one repo.
99 comments
[ 2.9 ms ] story [ 118 ms ] thread* Single version / branching for everything
* Commits that go across components/apps are atomic.
Cons:
* When it gets big, those features matter less
* Churn from other dev's stuff gets in your merge/rebase work.
* 'git log' and other commands can be painfully slow
* Mistakes in the repo (e.g., committing a password) now affect many more people.
Use for highly-coupled source bases. Where releases together and atomic commits are very useful. Every other time, you're probably better off splitting early and taking a little time to build the coordination facilities needed for handling dependency versioning across your different repos.
Note: I used to really prefer monorepos, but I've had some time away now to take a better look at it. Now they feel like megaclasses where small things are easier (because everything is in one place) but large things are way harder (because there's no good split point).
Wouldn't other people' work only cause issues if they are changing the same files, in which case conflicts would happen even if the work is spread in multiple repos?
It still takes forever!
I should replace that with 'most commits in history are irrelevant to you, so you have to dig smarter to understand what's been going on with your components'
Additionally rebase or merge won't make the conflict issue go away. I recommend companies stop using rebase as it produces an in accurate history of what's going on in your git history.
If I've got changes on an old version of the repo, and try to merge against a new one, I find conflicts in the changes others have made to the same repo -- I have the 'before' version, they have the 'after'. I frankly don't investigate too much when it hits.
As for history, I love rebase for two reasons: (1) I can squash mistakes in my history so they're gone forever. (2) In reconciliation I'm watching my changes filter through the history of the repo. When I understand what's up, this is great. When I don't, it's a catastrophe of me re-applying changes that haven't actually been applied yet, then conflicting with their application later. Ugh.
1- You finished your work, so you pull from the central repository, and merge your your branch to the master
2- You do your merge work, test it a bit and commit
3- Now, you push and oops, another team did the same, you are now left with to heads
4- You merge or rebase the two heads. Probably a simple task, but you may still need to run some tests again. And if you are lucky, you are done, otherwise, back to step 3
Also, if you are rebasing before push, you should be able to keep a clean history. However, if you are merging, and there are good reasons for a "merge only" policy, you are going to have a mess of merge commits every time the previous situation happens.
That's something you can work around with good management. But the more freedom you give individual teams to push to the common branches, the more that situation will arise, the more you try to control access, the more chance you will have for teams to go in different directions, making the merges infrequent but tricky.
Imagine you're in a branch for large project P. You want to merge to trunk but there are conflicts. The upstream library project L was changed in your branch but not in merged down by the submitter.
Even if P simply relies on a binary of L that is already in the artifact store, you still have to deal with merging this code down when normally you wouldn't.
In fact, the much bigger issue is that you now must deal with the double edged sword of updating the entire org when a shared dependency must be updated.
Perhaps you even completed that task in your project branch. Now you get to deal with merging into every project in the org that was touched.
The big advantage of separate repositories is that the work done on them is very clearly marked as independent. It becomes easier to take one component out and utilize it elsewhere... but at the cost of having to deal with dependency management. Dependency management is simple and automatic in easy cases and really terrible in hard cases (like when two components you use mandate mutually exclusive versions of a third component).
So I think the driving factor in the decision is the tradeoff between the degree to which you are willing to invest in careful packaging of each component (and the corresponding dependency management), versus your willingness to force everyone and everything to utilize the monorepo.
(Notice I didn't mention performance issues. I really don't think they are important enough the control the decision.)
No need to sync multiple repos and have complicated dependency versioning for in house libraries.
In house libraries are easier to refactor as dependent code can be updated in a single PR.
Code tends to be more reusable and you tend to make more reusable libraries as it is as easy as creating a new directory in the repo.
Services and tools can still have separation of control with all the monorepo advantages.
The only reason to not monorepo is if you are thinking of open sourcing the lib/tool you’re building but even then you can cut it out of the monorepo.
- Simple projects with a server and an SPA component - frontend and backend code for the same feature is on the same feature branch, can be tested and reviewed together.
- Projects with a couple of microservices that share some common libraries - these libs can be directly referenced by the microservices instead of being published on an internal package server. This of course has its drawbacks too but overall, the benefits have outweighed them.
Maybe the fix for that is to never make breaking changes, but that has drawbacks as well...
And ideally a monorepo workflow gives you the tools to a) find all callsites, b) run tests to ensure your changes won't break anything, c) apply changes in lockstep. Those tools are an engineering effort, but the equivalents for a large multi-repo setup are too.
Caveat: your company probably isn't Google, so your challenges may be different.
But CI becomes substantially more complicated. You have to figure out with every commit what actually changed and based on that change what needs to be tested and built. You don’t want to run a build of all components if you only changed a small piece of one component.
I feel this problem and I wonder, is there any ready-made solution for that?
Nope. Worked at the largest US bank that happens to be mono repo. They had compute grids with thousands of cores running tests 24/7. It's not sizable compared to the costs of the developers writing and maintaining software.
Why that can be a bad thing? Good software engineering practices tend to be associated with loose coupling, modularity and small, well-defined interface boundaries. Putting each project into its own repo, encourages developers to think of it as a standalone product that will be consumed by third-party users at arms length. That's going to engender better design, more careful documentation, and less exposed surface area than the informal cross-chatter that happens when separate projects live in the same codebase.
Why that can be a good thing? The cost of that is that each project has to be treated as a standalone project with its own release schedule, lifecycle support, and backwards compatibility considerations. Let's say you want to deprecate an internal API in a monorepo? Just find all the instances where it's called and replace accordingly. With a multi-repo it's nowhere near as easy. You'll find yourself having to support old-style calling conventions well past the point you'd prefer to avoid breaking the build for downstream consumers.
For that reason, monorepos favor the new programmer, which makes it easier to ramp your team size.
The only legitimate reason I have encountered so far for not using a monorepo is open-source dependencies that the company maintains patches to/contributes changes to. In this sort of situation, you'll want to have them in their own repository so that you can go from "foolib v5 plus our changes" to "foolib v6 plus our changes" without too much fuss.
Common sense (which is not that common) is what should be used to determine. Ask yourself some questions:
- Does these repositories change together ? - If the code is not together, one repo should be linked to the over via what (git tag ? version number ?) - All pieces of the application are contained within a context ? Or they can be split (example, 10 micro services and 1 frontend, but 5 of those micro services are also used by another frontend in another project).
The main goal is to make things easier, some PoC and experiments may make it more clear because it really depends on the situation.
Yes!
> if you have 10 micro services which are accessed by 1 frontend, it may be a little bit messy
I see another differentiator here. If I have a typed API interface (both endpoints and payloads), and I publish a (generated) client library for the frontend, then that frontend will only use the new API version if it is also upgraded to use the new client lib. Here I think multiple repos are more suitable because the pieces of code can improve independently.
In case the API is not typed/versioned, the case for a monorepo is much more important: the change to the API on the BE should come along with the change in the FE.
Thus typing/versioning (on specific barriers) helps with breaking up a code base (along those barriers).
Basically, if you make RPCs, breaking up your repositories or combining your repositories doesn't eliminate this problem. It's a separate problem that you have to tackle. The reason it doesn't come up as often as it should is because a lot of code is liberal in what it accepts, and most changes are strictly additive (i.e. it's "add more information to the response", not "remove information from the response", because we're generally pushed in the direction of making our applications do more). But it is something you have to attack head-on. No tool will auto-fix the problems for you.
At least, that is my argument. So, I could be reading my view into it. :)
They are, but they aren't, because we have to basically increment two counters when things change. If you want to be statically correct, you now have a distributed transaction to update both systems.
There is some path through a call graph (in-proc and rpc) that is statically valid. Then there are duck-typed results that are only additive, which you are describing. Provided the system can handle this (dictionaries over structs). You see this in protobuf and other static centric serialization systems. They start to encode dynamic information using the static building blocks. GCCs internal IR is like this. I think what you are advocating for only applies to the domain of microservices, loosely connected using languages and formats that are open under union. Other systems without those properties will not fare so well.
The folks using the monorepos probably don't disagree, but having atomic commits with hashes makes the identity function a useful crutch. I believe there are other solutions where the VCS could actually make this problem a non-issue.
But as someone who has always worked in a microservice environment kept by different git repos I don't understand the benefits of a mono repo. I can see the benefits of a mono repo in a massive code base that produces a binary at the end (i.e. windows). But could you elaborate more on the benefits of a mono repo in the case of many small microservices?
Why is this messy? Using folders to separate your code has no intrinsic organizational difference than using an entire repo. There's no issue in throwing every app in your company under one repo and just using a folder to organize your stuff.
Separating code into multiple repos makes one repo less aware of changes in another repo. It actually makes things harder and worse.
This is the same issue with monoliths and microserves. You don't need to separate your code into several computers just for organizational issues. You can use folders to organize things.
Cons: huge blast radius, makes it difficult for groups to be truly independent
First, for almost all companies copying non-selectively what Google does is harmful - you are not Google.
Second, if you are a small team, your code only produces one binary that is shared across team boundaries, then you might be able to do with a monorepo. But if you are refactoring pieces of your code into libraries (you probably should be) and sharing those libraries for use by other teams in your company (you maybe should be), then you probably shouldn't use a monorepo.
However, converting may be a painful process. Using multiple repos, one per shared product (library, SDK, web site, documentation package) is harder to manage because you need to manage your dependencies much better and pull in the correct versions for the current product.
You also will need to communicate changes on products better across your teams, ideally with automatic notification of new releases that include a changelog and examples of new/changed features.
As part and parcel of the above you'll need to have better release management, including a release repository. What you use will depend on your environment, the language(s) your team uses, and your budget. There are some good Open Source dependency repositories out there that can be used to accomodate NPM-style dependencies, JAVA dependencies, and others (e.g., Artifactory).
In summary migrating away from monorepos is going to mean: investing in good DevOps people and giving them what they need to create/install/manage good processes; learning these new processes and dealing with the added work they impose on developers. But, it also will likely give you better products and over time both speed up development time and reduce defects (in part through more well-defined API touchpoints).
"The first rule of distributed systems is don’t distribute your system until you have an observable reason to."
In a small company, release and dependency management are pointless busy work. And in larger companies, you have enough labor to manage monorepo tooling.
> Using multiple repos, one per shared product (library, SDK, web site, documentation package) is harder to manage because you need to manage your dependencies much better and pull in the correct versions for the current product.
This is the pointless busy work I was alluding to. See also: the diamond dependency problem. Meanwhile the benefits are unclear.
Multi-repos makes sense for entirely disconnected projects or products, or for collaboration in the open-source world. Most companies aren't going to run into significant issues with scaling a monorepo and they'll save a ton of work.
Cons: it is easy to make changes across multiple parts of a system
Cons: un-separation of concerns.
let’s split our project into multiple libraries so it can be reused within the company or even better if we put it on GitHub everyone can use it.
After a few months you notice nobody within the company cares about your nicely versioned libraries and on GitHub you get more and more complaints that the libraries are very limited and need more features.
After that you merge everything back into your monorepo and try to forget all the time wasted on git bureaucracy, versioning, dependency handling and syncing changes between repos.
Yes, i know that Microsoft made it work somehow but I'm not sure if i would like to replicate that.
Even with shallow clone and sparse checkout, you have to download at least the HEAD of the repo completely.
Git cannot restrict read access within a repository.
Branches and tags are global and visible to everybody, so you cannot have many of them.
Refactors and cross-service deployments can become much simpler in a MonoRepo.
At work I set up a repo with our back-end code in one directory, the front-end in a directory, and the ansible playbooks/other deploy stuff in another..
This allows us to automatically create servers for every branch because we know that everything goes together. Working on other projects where they have all of these separate is a nightmare. Especially when they use microservices and each service has its own repo and each library has a repo... I want to smack anyone who ever thinks that is a good idea
Google is probably the only one on the fence, and yeah they are suffering in the way you suggest.
EDIT: that being said, I do think multirepo is better, but only when you have the above tools to manage it.
1. I've only used monorepo professionally once in a big org and it was SVN based 2. I'm going to look at monorepos from the perspective of git and github
The main argument I can think of in favor of monorepos, is to maintain the cohesion as high as possible between different parts of your system. For example, if you want to make a change to the load balancer regarding TTL, you can also go and make the same change in your API and your mobile clients and in the end you create one single PR and you have your tests run against that single revision.
Compare and contrast the same scenario in the traditional multi-repo approach: You make your changes to your `infra` repo, then you make the same change in your `api`, `android-client`, `ios-client` repos and in you create 4 PRs, that need to be reviewed at the same time. Which one would you prefer to do?
Potential arguments against are:
1. Too much noise - If multiple people are working on the same repo, you'd be getting emails for PRs for parts of the code that you may not care about.
2. Longer clone/pull times - In the same vain as before, the initial `git clone` and maybe every `git pull` after that will be bringing in a lot of code that you may not care about, increasing the time of each operation and increasing your frustration.
3. Access control - How do you limit who has write access to which part of your repo? AFAIK this isn't possible in git but it may be possible with codeowners in github, I don't know.
4. Organization - How do you structure your monorepo? Do you know in advance how many components it's going to have? How is a restructuring going to affect your history and/or dependencies between different components of your code?
5. History and code sharing - With a monorepo it's more difficult to share/open source just part of your code and keep the history intact.
Having said all that, I think the monorepo is a good match at a service level, not at an organization level. I know Google and Facebook have gone all in at an org level, but first of all they don't use git and second, they have the luxury and the resources to make it work for their use case. At a service level you should have a good idea of your boundaries, your applications and your infrastructure and assuming a relatively small engineering team (~10-12 people) eventually everyone would be confident enough with all parts of your system.
Personally I really like the idea of having one hash define the entire state of the world. I have a few common utilities that I use across my projects and I find it too annoying to keep them all up-to-date between different repos. I hope that eventually I will find some time (or be annoyed enough) that I will converge everything in one repo.