I can't believe how it is working in such a big structure with just GitHub alone. GitLab with groups/subgroups and also integrated sourcegraph seems such more practical at this scale.
2000 repos what the fuck. More repos than engineers sounds terrible. Having worked in a large monolithic repo, I much prefer that. Everything (Shipping, testing, debugging, etc) is much easier that way.
I think we had 200 across three teams and that was starting to become miserable. Which is to say I was miserable and others were slowly coming around.
I started eyeballing two of them to combine because they were dominating stack trace frames, and after a number of rounds of feature toggle cleanup and refactoring to use other modules, had dwindled down to about 60% of a reasonable module size.
I’ve worked at companies with large monorepos and I’ve worked on teams with more repos than engineers. At large scale, it takes good tooling to make it work well, and that’s true both for monorepos and for multirepos.
I do think that the monorepo tooling is better, but I think the difference isn’t so large. You can have a completely miserable experience in a monorepo or multirepo setup, or a good experience.
I worked with a monorepo and multiple teams dedicated to the dev experience, I had my complaints but I was spoiled in hindsight.
I know they did alot with git to make it manageable, hopefully what ever they did makes it to the open source world eventually so we can all avoid these crazy thousand repo worlds.
Having worked with both, I think it's really much more about having someone (a team) dedicated to dev experience, maintaining good onboarding tooling, CI/CD, build infrastructure, security configurations, SOP's around package repositories, container image registries, static code analysis tooling, and the works. If you have 2000 repos but finding what you need is easy, then the number itself is not an issue. If you have 1 repo and running unit tests for a small component before merging takes 8 hours, again the number of repos is probably not the issue.
Which part do you disagree with? The slowness, the Fisher-Price looking UI, the overwhelming noise around posts, or the popups to login or install the app?
They could've improved the styling so things were nicely formatted and easy to read, but they went above and beyond (in a bad way).
I always forget that the ability to expand an image link to show an image inline is a feature of Reddit Enhancement Suite, and not a stock feature built into reddit.
The hacker news crowd is overwhelmingly in support of old.reddit. I mean look at the UI of this place. Obviously this crowd appreciates substance more than Web 3.0 ad trackers and GIF chat
> If it were my site, the UI would be a little more suited to the current year.
The only things I would change is make the Reply link open a text box in-line rather than needing a new tab, and add an icon next to my username/karma in the top right whenever there's a new reply to one of my comments.
Any other "modernization" is probably a step BACKWARDS in UX. I really like the simple design of HN.
Yes, the Reddit dev team might have spawned a 2000+ repo mess, but they also host it under the snooguts.net domain name, which is objectively adorable, so all is forgiven.
I for one am sick of cute mascots being used as excuses and distraction from actual substance of conversations. Its a very reddit thing too. funny how that works.
Along these same lines, it is common at my organization for people to gleefully say "welcome to ${ORG}!" or "It's the ${ORG} way!" in response to some failure that comes down to managerial knee-jerk or some fundamentally broken process.
It always seemed so perverse to me to promote this sort of tacit approval--as you say--passing it off as "cute" instead of... improving things.
I've heard it spoken by members of upper management, which obviously serves to reinforce the culture (and was a massive red flag for me at the time). I will say, though, that management is very open to improvements in procedure so it's more that the rank-and-file have accepted the status quo. They also have a habit of promoting a spirit of entrepreneurship within the company, which seems to be the thing that keeps me coming back for more.
Let me put it another way: Are the people making those comments responsible for those procedures?
Having seen this sort of thing many times, and being in that position myself, it seems more about it not being the "rank-and-files" responsibility to fix these things. I'm not going to deal with the process and bureaucracy in order to (try to!) fix processes that are above my paygrade - and with no guarantee things will change in any way
In short, this is a sort of coping mechanism
(Obviously this is less applicable to upper management)
Throughout my career, I spent too much time trying to change organizations. I later realized that changing culture is prohibitively difficult when one person or a minority are trying it. I now know that it's better to "choose my battle". Only focus my energy on things that really matter, give a friendly comment, maybe coated in humour in all other situations. Cultural problems happen for reasons, and those reasons are usually very hard to change, like personality of the executives.
On swaying culture in any meaningful way: I thankfully learned that lesson early. It's one of the reasons I'm grateful for having worked here. As someone building a product on the side, it's been very useful to see how much influence executives can have on the business' culture. I've spent a fair amount of time reflecting on the problems my own personality might cause.
You know, big sweeping refactors deservedly get a bad rep, but as everything else in life, there are always exceptions
At some point, I don't know, maybe when you cross the 100 repos mark, you've gotta ask yourself "maybe we could try a different approach?"
It's not like reddit has been known for its wonderful stability over the years
I'm sure the scale here is completely unlike anything I've ever worked on, but how hard can it be to write a sane implementation of a message board?
I'd be curious how much of this problem is caused by the junk that is "new reddit". I've been there since 2007... The day old.reddit.com is the day I abandon it for good
My guess is that old Reddit will disappear in short order once the IPO is done. I have no inside knowledge, just a feeling. Once there are external shareholders it will be hard to justify maintaining both versions of the site.
I am honestly surprised that it is still around, I expected them to kill it off once they hopped into the "We want to be Instagram" train with the new design and give it maybe a month or three. Who knows, maybe they just can't find the correct repo for it?
A small part of me actually hopes they kill off old reddit, so I have a reason to completely stop using it :)
Opening up the homepage shows me 1 (in words: ONE) item because it insists on expanding (and autoplaying) images/videos immediately. Old Reddit shows me 9 items (plus one paid ad) on that same screen.
Maybe they got rid of some rough edges, but the fundamental problem is that it tries to be Instagram. Imagine opening Hacker News one day and instead of seeing 25 headlines at a time you can only see one or two - that would really defeat a lot of the allure of the site.
New Reddit is still trash, and will forever be trash because the problem isn't some bug here or there, but a fundamental user experience design decision to make it that way.
To those who are swinging towards monorepos, I don't think that is a good solution. The reason being is that developers simply cannot be trusted to "do the right thing" on data and module boundaries. Someone comes in new to the project and does something they don't know they shouldn't. It is the honor system backed by weak linters and tooling.
In our monorepo, everyone passes around django orm objects and boundaries are practically non-existent. N+1 queries abound. Tests are full of patching and mocking and are _slow_. Our build takes over an hour to run tests. Someone on team A can and absolutely will mess up what someone on team B is doing. We are now having to spend quarter upon quarter as we define and enforce domain boundaries within the python code base. It is all bolted on checks. Tests are getting worse and people are actively trying to figure out ways around the testing system because it sucks.
Compare to my last gig. We had several hundred production repos. Each repo starts from a template with its own build pipeline. All production repos are gated so that any PR must pass tests before it can merge. Any merge has to pass tests before it could be deployed. As the base build processes matured, teams could, at their leisure, pull their services up to the latest and greatest. We even migrated from Jenkins to Buildkite; yeah, it took N pulls into N repos. Not a big deal. Most projects' tests and builds could get code out to production in under 10 minutes, including all those checks. Due to the network boundary, you couldn't accidentally get around someone's abstraction. And if one team blew up their build doing something dumb? No problem, it only affects that one team.
The argument is "gah, managing all those services!" Keep data behind APIs. Keep APIs backwards compatible. Keep dependencies acyclic. This is _possible_ with monorepos, but you have to do extra work compared to networked services -- yes, when any particular team/service can deploy in minutes due to low build system complexity you are winning. Can you get that wrong and make strange cyclic dependencies and introduce performance issues due to network hops? Yeah, of course. However, we were processing, literally, 10s of billions of api requests on this system and teams could work untethered from one another. The new gig does eerily similar software, but is several orders of magnitude slower in their ability to process data and their ability to move new features.
yes, yes, you could have networked services and a monorepo and you can leverage tooling like Pants to minimize the testing to only account for changed files. It is just fighting what I have found to be a better model. Keep things separate. Keep things fast to change.
While I have yet to work with a large Django monorepo, it's the architecture I'm leaning towards for a product I'm building. Are you guys using packages like nplusone [0] to help with some of this ORM complexity? As I mentioned, I am headed in this direction so I'm curious to hear your insight. I'd also love to pick your brain on any other pitfalls of this approach.
You're describing a failure in leadership, not a failure in the monorepo model. If your dev leads are not bringing these issues up to management, or management are not backing them on solving these issues, you're doomed whether you're doing monorepos or not.
Monorepos have their drawbacks (and I personally don't like them either), but being unable to trust devs is not one of them.
If you can't make monorepos work, what makes you think you can get "microrepos" to work?
There's three systems an org can (and should) have in place to prevent this:
- Dev-time analysis. This is, types, linters and tests that will throw errors and block merges before they happen.
- A good code-review environment. Not one that's so tedious that everyone just seeks the stamp to move on, but also no one that's so lax there's no code review at all. One in which code-review is really an opportunity to improve the code before it's merged, people are committing to unblocking others, and everyone assumes their responsibility in having the right people look at changes.
- A better team structure. One where ownership of the code is shared, and code quality goals are present.
Usually if computer systems, peers and the org at large all agree in creating a good dev environment, dev environments tend to not be bad.
In my experience not having a monorepo doesn't fix the issue, it just hides it away. Repos don't have the same code-quality, and overall everyone tries to stay in their little island and not deal with problems and disagreements outside of it. You lose the opportunity to talk about these issues. There isn't that learning and teaching dynamic in the wider org, and the whole company is worse off as a consequence.
Plus, of course, there's a lot of extra-overhead in developing with APIs between each component.
I worked at Reddit in the not so distant past. The entire recommendation system lived in 3 repos. I'm pretty sure there are just 2000 repos because the onboarding tutorials have you create one, and that number is probably around the number of engineers that have worked there. I'd guess 100-200 have some production component.
If this is the case, then why in the world wasn't that mentioned in TFA? I'm not questioning your comment but it strikes me as odd that there would be no attempts to clean any of that up.
When I got the responsibility for our aws accounts and github repos there where over 1000. 99% are not used, and closing them down is a PITA since AWS only allows you to close 10% of your accounts per month and finding out who actually uses a github repo can be hard (if it is not commited to regularly). My predecessor didn't care as long as they don't cost too much, which is a reasonable stance but I want to know what is running.
I'm working my way through them but it takes quite a bit of time and checking in with people on what is actually used.
Besides that there is some prestige in saying that you handle 2000 amount of repos instead of it being "we have 20 prod repos and 1980 personal playground repos with one commit"
It's not just via the GUI. I have a script (which uses their API and SDK) that I run every month to delete the next batch until it runs into the deletion limit and a calendar reminder every month to run the next batch.
Why would it be? That doesn't look good at all - why not clean them up periodically, or simply after onboarding? - and this makes them seem more capable than if they were managing 1/10 the number of repos.
(This is assuming GP is correct. I have my doubts)
That's crazy. Monorepo definitely makes more sense.
Though I always wonder - how do Google, Microsoft, Facebook etc. deal with developing code near the root of their dependency tree? Utility libraries for example. Technically you're going to have every change you make there building all the code and running all the tests, which is obviously unworkable. What do they do?
Do they not have "internal" releases for these root components that are not depended on by everything, so you aren't having to wait 2 days and use a gazillion hours of computer for every single change though?
First, you prune tests to only those affected by the change. There's basically nothing (other than the build system itself) at the root of all code because of the huge number of languages and platforms in the repo.
But you still have library code that's depended on so much that you can't easily run all the tests for each change. So you run a train system so that you can run tests for a whole bunch of changes (those on the same train) at once. Developers have to schedule changes to get on the next train. Other code analyzes the failures from the train to try to apportion blame correctly.
Then, because getting on trains can be cumbersome and tests you care about can fail because of other changes, you build random test sampling and smoke test subsets so that you can get some immediate results to attach to a review, etc.
It roughly works, as well as anything can at that scale.
Are you speaking from experience or is this just a guess?
Merge trains/queues don't really help with the problem I'm describing - they just prevent race conditions between commits. You still need to run all the tests for each PR/MR first before joining the queue/train.
1. Types, tests, all the build-time analysis you can rely on
2. Gradual migrations. You run two versions for a while and make it easy for certain codebases and/or users to be switched between the new and the old as necessary.
3. Sometimes you just don't touch that stuff, slowly build new things with new technology and hope eventually the old thing will fall in use enough that you'll be able to delete it.
I don't get the hating here. With the right tooling it doesn't matter if its 10, 100 or 2000 repos. And it buys you some nice things like per-repo permission settings.
What's your preferred solution to coordinating PRs across multiple repo boundaries? This is an enormous pain to do manually and the only tooling solution to it I've seen is Gerrit, which I find hard to describe as "right".
It's one of the reasons I prefer monorepos in practice. Especially if you use something like GitHub Enterprise, the cross-repo experience in terrible. In theory, many repos is better than a monorepo. In actual practice, most tooling only works well for monorepos. (And don't get me started on CI/CD tooling like GitHub Actions)
Or you do what the .NET team at Microsoft did, create a separate monorepo that rolls up individiual repos: https://github.com/dotnet/dotnet
I believe the "right" way to do it is to have every repo only able to talk to another repo's code via versioned APIs. So e.g. a client repo can ship an update that will talk to some service repo's code at either version N or N+1, the server repo can later on leisurely ship its update that actually makes version N+1 available (and has to support however many prior versions by policy). Or vice-versa, the server side can have N and N+1 already in service while the clients roll out their updates. You have to do something like this anyway if you have a mobile app.
Of course it has consequences and makes some things harder, you really have to go in on API-first even for things like UI widgets if different ones live in different repos, but it at least alleviates the common problem of trying to get multiple repo commits merged at the same time and have everything keep agreeing. I still prefer the monorepo approach.
I'm only concerned with calling the feature done, not the details of interface compatibility. Let me give a historical example for clarity. I built a data path for an important sensor. The hardware team made their board rev (not in source control), the FPGA modifications were made, the bootloader had to be updated, the kernel had to be updated, application middleware had to be modified, the application had to be modified, and the networked application talking to that code had to be updated.
You can certainly make the argument that all of that should belong to a monorepo (and I would agree), but it's not a painless or easy argument.
I mean, with the correct amount of investment, research and work, one can build a livable house out of Lego. I'm sure that gets you some nice things too.
It doesn't change the fact other people will think: "why didn't you just build it out of normal bricks to save yourself all that trouble?".
We used to have an ecosystem like this. In our case it reflected an entrenched set of divisions between warring teams. In some ways it may have then enhanced those positions and we still bear a few of the scars today.
A lot of the old guard have left the company though and our main product moved from four repos to just one. The threat from the legal team to have enforced OWNERS files — essentially replicating the divisive politics of the old repos but in the monorepo — thankfully withered on the vine. We still audit what goes into each release but it’s no longer part of any active permissions thing. We trust our developers but verify, for legal reasons, that nothing went wrong.
You either want one engineering team to act in unison behind your company’s mission, or you want to live a divisive narrative that you are actually multiple teams “working” together with none of the advantages of living under one roof and all the disadvantages of hard repository boundaries crisscrossing your intellectual property.
So many factors threaten to curdle your team dynamic: multiple offices, multiple floors, work from home hermits, bad management, etc. It’s simply org entropy and it takes much effort to keep the weeds out of the garden. Multiple repositories is one less bullet you can keep out of your feet while fighting all the other battles that threaten to turn your team from 1990s Sun Microsystems into 2010 Sun Microsystems.
65 comments
[ 2.9 ms ] story [ 116 ms ] threadI started eyeballing two of them to combine because they were dominating stack trace frames, and after a number of rounds of feature toggle cleanup and refactoring to use other modules, had dwindled down to about 60% of a reasonable module size.
Not a lot of this sort of work gets done.
I’ve worked at companies with large monorepos and I’ve worked on teams with more repos than engineers. At large scale, it takes good tooling to make it work well, and that’s true both for monorepos and for multirepos.
I do think that the monorepo tooling is better, but I think the difference isn’t so large. You can have a completely miserable experience in a monorepo or multirepo setup, or a good experience.
I know they did alot with git to make it manageable, hopefully what ever they did makes it to the open source world eventually so we can all avoid these crazy thousand repo worlds.
They could've improved the styling so things were nicely formatted and easy to read, but they went above and beyond (in a bad way).
https://github.com/tom-james-watson/old-reddit-redirect
If everyone loved it, there'd be no Hacker News mobile apps.
The only things I would change is make the Reply link open a text box in-line rather than needing a new tab, and add an icon next to my username/karma in the top right whenever there's a new reply to one of my comments.
Any other "modernization" is probably a step BACKWARDS in UX. I really like the simple design of HN.
If it were your site, suited to the current year, I wouldn't come back here.
I don't think someone knows what "repository" means.
At least they're bringing in Sourcegraph. That tool's helped me make sense of some chaos. Not 2000 repos' worth of chaos, but still, some chaos.
The nice thing about 2024 is that it’s not 1999. But every once in a while I run into people making 25 year old mistakes.
It always seemed so perverse to me to promote this sort of tacit approval--as you say--passing it off as "cute" instead of... improving things.
Having seen this sort of thing many times, and being in that position myself, it seems more about it not being the "rank-and-files" responsibility to fix these things. I'm not going to deal with the process and bureaucracy in order to (try to!) fix processes that are above my paygrade - and with no guarantee things will change in any way
In short, this is a sort of coping mechanism
(Obviously this is less applicable to upper management)
At some point, I don't know, maybe when you cross the 100 repos mark, you've gotta ask yourself "maybe we could try a different approach?"
It's not like reddit has been known for its wonderful stability over the years
I'm sure the scale here is completely unlike anything I've ever worked on, but how hard can it be to write a sane implementation of a message board?
I'd be curious how much of this problem is caused by the junk that is "new reddit". I've been there since 2007... The day old.reddit.com is the day I abandon it for good
Hope I’m wrong.
A small part of me actually hopes they kill off old reddit, so I have a reason to completely stop using it :)
Maybe they got rid of some rough edges, but the fundamental problem is that it tries to be Instagram. Imagine opening Hacker News one day and instead of seeing 25 headlines at a time you can only see one or two - that would really defeat a lot of the allure of the site.
New Reddit is still trash, and will forever be trash because the problem isn't some bug here or there, but a fundamental user experience design decision to make it that way.
In our monorepo, everyone passes around django orm objects and boundaries are practically non-existent. N+1 queries abound. Tests are full of patching and mocking and are _slow_. Our build takes over an hour to run tests. Someone on team A can and absolutely will mess up what someone on team B is doing. We are now having to spend quarter upon quarter as we define and enforce domain boundaries within the python code base. It is all bolted on checks. Tests are getting worse and people are actively trying to figure out ways around the testing system because it sucks.
Compare to my last gig. We had several hundred production repos. Each repo starts from a template with its own build pipeline. All production repos are gated so that any PR must pass tests before it can merge. Any merge has to pass tests before it could be deployed. As the base build processes matured, teams could, at their leisure, pull their services up to the latest and greatest. We even migrated from Jenkins to Buildkite; yeah, it took N pulls into N repos. Not a big deal. Most projects' tests and builds could get code out to production in under 10 minutes, including all those checks. Due to the network boundary, you couldn't accidentally get around someone's abstraction. And if one team blew up their build doing something dumb? No problem, it only affects that one team.
The argument is "gah, managing all those services!" Keep data behind APIs. Keep APIs backwards compatible. Keep dependencies acyclic. This is _possible_ with monorepos, but you have to do extra work compared to networked services -- yes, when any particular team/service can deploy in minutes due to low build system complexity you are winning. Can you get that wrong and make strange cyclic dependencies and introduce performance issues due to network hops? Yeah, of course. However, we were processing, literally, 10s of billions of api requests on this system and teams could work untethered from one another. The new gig does eerily similar software, but is several orders of magnitude slower in their ability to process data and their ability to move new features.
yes, yes, you could have networked services and a monorepo and you can leverage tooling like Pants to minimize the testing to only account for changed files. It is just fighting what I have found to be a better model. Keep things separate. Keep things fast to change.
[0] https://github.com/jmcarp/nplusone
Monorepos have their drawbacks (and I personally don't like them either), but being unable to trust devs is not one of them.
If you can't make monorepos work, what makes you think you can get "microrepos" to work?
- Dev-time analysis. This is, types, linters and tests that will throw errors and block merges before they happen.
- A good code-review environment. Not one that's so tedious that everyone just seeks the stamp to move on, but also no one that's so lax there's no code review at all. One in which code-review is really an opportunity to improve the code before it's merged, people are committing to unblocking others, and everyone assumes their responsibility in having the right people look at changes.
- A better team structure. One where ownership of the code is shared, and code quality goals are present.
Usually if computer systems, peers and the org at large all agree in creating a good dev environment, dev environments tend to not be bad.
In my experience not having a monorepo doesn't fix the issue, it just hides it away. Repos don't have the same code-quality, and overall everyone tries to stay in their little island and not deal with problems and disagreements outside of it. You lose the opportunity to talk about these issues. There isn't that learning and teaching dynamic in the wider org, and the whole company is worse off as a consequence.
Plus, of course, there's a lot of extra-overhead in developing with APIs between each component.
I'm working my way through them but it takes quite a bit of time and checking in with people on what is actually used.
Besides that there is some prestige in saying that you handle 2000 amount of repos instead of it being "we have 20 prod repos and 1980 personal playground repos with one commit"
AWS limits are weird both up and down.
(This is assuming GP is correct. I have my doubts)
Though I always wonder - how do Google, Microsoft, Facebook etc. deal with developing code near the root of their dependency tree? Utility libraries for example. Technically you're going to have every change you make there building all the code and running all the tests, which is obviously unworkable. What do they do?
Still best to run unit tests locally first of course!
But you still have library code that's depended on so much that you can't easily run all the tests for each change. So you run a train system so that you can run tests for a whole bunch of changes (those on the same train) at once. Developers have to schedule changes to get on the next train. Other code analyzes the failures from the train to try to apportion blame correctly.
Then, because getting on trains can be cumbersome and tests you care about can fail because of other changes, you build random test sampling and smoke test subsets so that you can get some immediate results to attach to a review, etc.
It roughly works, as well as anything can at that scale.
Merge trains/queues don't really help with the problem I'm describing - they just prevent race conditions between commits. You still need to run all the tests for each PR/MR first before joining the queue/train.
Or you do what the .NET team at Microsoft did, create a separate monorepo that rolls up individiual repos: https://github.com/dotnet/dotnet
Of course it has consequences and makes some things harder, you really have to go in on API-first even for things like UI widgets if different ones live in different repos, but it at least alleviates the common problem of trying to get multiple repo commits merged at the same time and have everything keep agreeing. I still prefer the monorepo approach.
If you are having to keep two or more modules/libs/packages/repos in sync, those should be unified.
You can certainly make the argument that all of that should belong to a monorepo (and I would agree), but it's not a painless or easy argument.
It doesn't change the fact other people will think: "why didn't you just build it out of normal bricks to save yourself all that trouble?".
A lot of the old guard have left the company though and our main product moved from four repos to just one. The threat from the legal team to have enforced OWNERS files — essentially replicating the divisive politics of the old repos but in the monorepo — thankfully withered on the vine. We still audit what goes into each release but it’s no longer part of any active permissions thing. We trust our developers but verify, for legal reasons, that nothing went wrong.
You either want one engineering team to act in unison behind your company’s mission, or you want to live a divisive narrative that you are actually multiple teams “working” together with none of the advantages of living under one roof and all the disadvantages of hard repository boundaries crisscrossing your intellectual property.
So many factors threaten to curdle your team dynamic: multiple offices, multiple floors, work from home hermits, bad management, etc. It’s simply org entropy and it takes much effort to keep the weeds out of the garden. Multiple repositories is one less bullet you can keep out of your feet while fighting all the other battles that threaten to turn your team from 1990s Sun Microsystems into 2010 Sun Microsystems.