Ask HN: What is the best code base you ever worked on?

561 points by pcatach ↗ HN
And what made it so good?

Was there someone enforcing good practices top down? Just being in a group of great engineers? Or something else?

468 comments

[ 3.8 ms ] story [ 257 ms ] thread
A blank slate, so I can make the first mistake :)
I agree, the best code base is always a zero-line code base.
After that first line, it’s all downhill from there… :-)
So true!

I spend so much time obsessing over how what I am about to write ties in with what has already been written; or fuming over the stupidity of earlier decisions (usually made by myself). A blank slate is incredibly refreshing.

Although I'm only on job three and have not had that much involvement with open source, I think my current employer (Attio) has one of the best codebases I've seen.

Qualitatively, I experience this in a few ways: * Codebase quality improves over time, even as codebase and team size rapidly increase * Everything is easy to find. Sub-packages are well-organised. Files are easy to search for * Scaling is now essentially solved and engineers can put 90% of their time into feature-focused work instead of load concerns

I think there are a few reasons for this:

* We have standard patterns for our common use cases * Our hiring bar is high and everyone is expected to improve code quality over time * Critical engineering decisions have been consistently well-made. For example, we are very happy to have chosen our current DB architecture, avoided GraphQL and used Rust for some performance-critical areas * A TypeScript monorepo means code quality spreads across web/mobile/backend * Doing good migrations has become a core competency. Old systems get migrated out and replaced by better, newer ones * GCP makes infra easy * All the standard best practices: code review, appropriate unit testing, feature flagging, ...

Of course, there are still some holes. We have one or two dark forest features that will eventually need refactoring/rebuilding; testing needs a little more work. But overall, I'm confident these things will get fixed and the trajectory is very good.

google3, all the devex tooling was taken care of by other teams. Tons of useful library functions available to import, accumulated over decades.
My favourite projects are small, with very focused goals and features.

I have a Laravel project that I have maintained for a customer for seven years. The app is straightforward and allows users to create portals that list files and metadata, such as expiration dates and tags.

Every other year, they ask me to add a new batch of features or update the UI to reflect the business's branding. As the app is so small, I have the opportunity to review every part of the app and refactor or completely rewrite parts I am not happy with.

It is a joy to work on and I always welcome new requests.

This sounds like wonderful! Maybe one day I can have a few of these “babies” that I can lovingly raise, providing joy to both the dev and the customer.
Any codebase that I had complete control over.

No, but more seriously, I've found that familiarity with the codebase is more important than having it be perfectly engineered. Once you're really familiar with the codebase, you know where dragons be, and you can make changes more easily. And God (PM) forbid, if you ever find yourself with some extra free time you might even reduce the size of dragons over time.

This brings me to my final point. Any codebase that I really enjoyed working with was the one that was constantly evolving. I don't mean rewriting everything from scratch every few months, but as long as I have permission (and time) to refactor the things that have been bothering me for months as patterns emerge, I'm a happy bee.

> Any codebase that I had complete control over.

No excuses. Code ownership is important. Sometimes it works for a team, sometimes only for individuals.

But not having to submit to core teams, architects and self-proclaimed experts of all kinds is a blessing.

I now work for an organization that discourages code ownership, and it struggles on many fronts:

        1. core teams are dysfunctional
        2. people find niches and stick to them
        3. top talent is leaving, although pay is good and business creates real value for citizens
        4. there is virtually no horizontal communication
        5. mediocre ones rise to the level of their incompetence and  infest the lives of others
        6. and so on and so forth...
And I think the root cause of all this is lack of individual (code) ownership.
I’ve had exactly the same issues but because I couldn’t change anything without getting approval from 16.5 code owners on every PR submitted. It’s a real pain if you start modifying your coding for ‘least code owners hit’ instead of ‘best architecture’.
I like how my workplace does it -- there are rigorous codeowners and usually you only need approval from 1-2. if you do need approval from 5+, you can request a select 'super' codeowner review which will approve it for all.
I think core teams being helpful or harmful really comes down to the individuals.

The problem is at this level, most orgs don't have anyone to really judge or contest competency, so they hire the salesmen rather than the doers and when they don't, they tend to cheap out and just get inexperienced people.

Logically it makes a bunch of sense, though.

Why rebuild yet another platform? Why is your central platform bad? Usually it's not self-service, sometimes it's because it's built in cumbersome ways, other times its because it actually enforces good standards on you rather than just giving app your apps admin.

It's difficult for the person who hires the core team to differentiate between those complaints, unless they themselves both have the technical competency and the empathy to really understand the problem. They usually don't.

Point being, done well, it's great, but most folks can't do it well.

Also nice is when an entire community has agreed to architect a codebase more or less the same. You're basically psychic when that happens.
> Any codebase that I had complete control over.

Anyone other than myself would instantly observe it as the the worst codebase they have ever seen.

For that, there is a saying: "code as if your kids will maintain it".

But I think it does not convey the right meaning. When I code something I will have to maintain for a long time, I try to make it as simple as possible for my future, older, less motivated and weary self.

The worst codebases are written by people who landed the gig a few months before and do not expect to stay around longer than a year or two.

This is very much true. Initially when I joined the industry I used to work for a product from its inception. So I was aware of what section of the code affects what part of the product. Although I hadn't worked on all of those, I kept an eye for the all the changes that were coming in. I knew where I should look immediately a bug is reported even if it is not something related to my line of work.

Recently I switched teams and now I find myself taking up bugs that are only related to my line of work. Not being familiar with the codebase decreases productivity and wants you to rely on other people in the team for most of the time.

> Once you're really familiar with the codebase, you know where dragons be, and you can make changes more easily.

This is an interesting, and often overlooked, point. A month or two ago someone asked us, the Fossil SCM maintainers, if we'd be open to them refactoring the tree to something which better matches modern sensibilities (i'm paraphrasing here). Despite its many proverbial dragons, the long-time maintainers are comfortable with it and know where those dragons are and how to defeat them (or, in some cases, sneak around them), so, despite our collective geek tendencies to prefer "perfect code," we're happier with the dragons we know than those we don't. (That's not to say that fossil's code is awful, but its structure varies significantly from what one might write today if one was to start such a project from the ground up.)

I also echo the code ownership part.

I have a part-time gig where I maintain accounting software for a former client of mine. It takes up a few months' weekends a year.

I wrote about 60-70% of it when I was working for the owner of the software. It's something where as long as the client's happy, and they get new integrations and updates on time, they could keep using it for a decade longer.

I had almost complete ownership of the architecting of the software. It's broken down into a few microservices (think database, core business logic, reporting, auth, logging etc). The best thing I did at the time was pushing to use gRPC even though management felt it was too new tech.

The UI is in Angular, pain-free periodic upgrades. I've even rewritten some perf-sensitive code in Rust, and everyone's happy with snappier calculations.

The code hygiene is relatively good.

The only downside's that if someone else were to take over the code, they'd struggle (it's one of those things where I'm wearing many hats). I've been fortunate to be a professional accountant who moved into software engineering, so everything makes sense to me.

> Any codebase that I had complete control over.

This is probably one of the pillars of good codebases, or at least decoupling the bits that you don't control as well as you can (this includes external services). I remember needing to write a wrapper around another JWT library, but because it was quite important, I aimed for >95% test coverage, some of the tests acted as documentation for how to use the code, there was also a nice README, there was CI configuration for pushing the build artifacts to Maven and suddenly even managing dependency updates became easy because of the test suite. Years later, it's been integrated in a bunch of services and "just works".

Come to think of it, things always get less pleasant once you add a bunch of complex dependencies and libraries/frameworks. Need to make a few RESTful Web APIs in Java? Something like Dropwizard will probably give you fewer headaches than Spring (or Spring Boot) and all of its inherent complexity, in the case of the former you might even need to do configuration in XML and that has honestly never been pleasant. If you need to integrate with a bunch of other stuff and want something opinionated, going for Spring Boot will make sense then, but for simple use cases it's overkill. Same for ASP.NET, Ruby on Rails, Laravel and many others, while they might be easier to use, updates (especially across major versions) and breaking changes will give you a headache and just add a bunch of churn to the project.

Similarly, if you need a message queue, externalizing that into RabbitMQ might make a lot of sense, same with storing files in an S3 compatible store, using something like Redis or Valkey for key-value storage, as well as not trying to shove too much logic into your RDBMS. Just pick whatever tool feels best for the given task at hand, instead of shaping things into what they're not (using the database for blob storage, for example), unless you have a whole bunch of constraints to contend with. Otherwise, sometimes you just get the worst of both worlds, like needing to use Oracle for a project, not having easily launchable local environments (because Oracle XE doesn't support some features), having to share DB instances with others during development and also running into weird obscure issues like DATABASE LINK queries taking 100 times longer in some cases, even when executing the same SELECT query on the remote DB works without issues.

To not go into a rant, I'd sum it up like this: be in control, isolate the things that you cannot control, pick the correct technologies, do the simplest thing that you can get away with without overengineering and think about the person who'll have to maintain, debug and grow the system in the following months and years (which might also be yourself, sans knowledge about what the code did, if you don't make it explain itself or don't comment the non-trivial stuff).

> And God (PM) forbid, if you ever find yourself with some extra free time you might even reduce the size of dragons over time.

Honest question, what is the company like where you can do that? Everywhere I've worked (only been working in industry for 6 years) has had such rigid agile development that even when I do find myself with free time, there's no flexibility to work on things that haven't been assigned to you and the best I can do is work on profiling/debugging tools.

Usually at smaller companies with just a handful of developers. Can be a "start-up", or just a smaller business running fine with small teams. Or sometimes in a larger company where there's a small team doing its own thing.

There's up-and-downsides to everything. The pay is usually less, and "fewer processes" often also means fewer processes surrounding HR and such. This can mean you're subject to the whims and mood swings of one person. I was once fired over a pretty far-fetched misunderstanding, but 1) they didn't mention it because "I should know myself", so I didn't really have a chance to clear it up, and 2) when I got fired they agreed it was a complete misunderstanding and that they made a huge mistake, but fired me anyway because that was the road already taken now. Great ... thanks ... It's hard to imagine that happening at a larger company with an actual HR department and stuff.

Also: a Wikipedia-esque "be bold" attitude helps. Obviously refactoring things just for the craic isn't a good idea, but if there's something that can really make an impact then I'd just do it and send a PR, regardless of what I've been assigned. Obviously "refactor everything" on your own initiative isn't a good idea, but usually there's tons of smaller stuff that really helps: "introduce new function/package that helps reduce boiler plate", "split up this difficult to understand function", things like that.

Most PMs will realize they don't have a 100% in-depth understanding of the code, and will trust you when you say it helps you do your work better. Usually it's fine as long as you apply proper judgement and don't spend too much time on it. That said, there are some places where this really isn't appreciated regardless. IMHO that's pretty dysfunctional and when I start looking for somewhere else to work.

Counter-point: whitespace PR’s, refactors I didn’t ask for, aren’t attached to a larger initiative, etc actively annoy me as an owner.

Every PR takes time to manage, compounds complexity, and carries risk. We only get so many out a week, and I want them to count.

Maybe it sometimes goes in hand with disfunction, but I’m guessing there’s more going on there.

> apply proper judgement
Your successors just hope you can only remember 3 things at once so you end up with elegant designs rather than one of those monsters who can remember "to add a new wrinkle to this feature, just edit these 25 files."
Considering the biases, the one I wrote for the company I created.

When we have the opportunity to be in this context, keeping in mind what bothered us in the codebases with which we were able to work in the past, we can force ourselves not to reproduce the same errors. Like the unmaintained unit and integration tests, the lack of refactoring, other developers that use fancy technologies instead of simpler concepts more for the opportunity to play with technologies than real need..

And also, I guess, because we are more aware that the code is a reflection of the company that we want to have, that the simpler the better is a key point when we need to debug.

Generally code that hasn't been tested commercially. Unencumbered by pesky client driven features, just code for dreamt up features that are fun to code but perhaps will never be used.
Hopefully the last one you were responsible for.
Hopefully the most recent one you were responsible for.
Any code base that doesn't use the advanced features of it's language(s) are always better in my experience. Heavy usage of e.g. meta-programming in Python or perhaps uber's fx (dependency injection) in Go makes projects infinitely harder to get into.
Such a great point. I audibly groan when I come across Python meta-programming.

While not an advanced feature, I have a similar response when I see lots of decorators in Python. They quickly become a debugging nightmare.

I can see a few cases where that depends...

Really simple languages: Ruling out meta-programming is really going to limit you in Lua for example. Just being able to do `mySocket:close()` instead of `Socket.close(mySocket)` involves meta-programming.

Older languages: For C++ the "simple" features are going to include raw pointers and macros. Maybe it's not so bad to allow smart pointers and templates to avoid those

Both of these examples are examples of an under-programmed core though. Lua is notorious for lacking batteries, so everyone has to reinvent their own. There’s literally no serious Lua program without some sort of classes, but they still resist adding them into lauxlib.
Depends who is providing it.

Django and pydantic meta programming usually make the code easier to deal with.

In shop written meta programming usually sucks.

I worked at GOV.UK for a few years on what was effectively specialised CMSs all written in Rails. Mostly basic CRUD stuff. A contractor came along and built the most insane CMS I've ever seen. I found out later it was flavor of the Java Repository Pattern using dependency injection. It became so difficult to work with that the last thing I worked on there was to delete it and rebuild it using standard Rails patterns.

The KISS philosophy exists for a reason and that includes over-using advanced language features just to show off.

Besides just KISS, a lot of messes I've seen have been implementing patterns outside the framework or implementing complex patterns that didn't add value.

Besides KISS (or maybe as an extensive), try to keep framework-based codebases as close to the official documented setup as possible. You automatically get to s of free, high-quality documentation available on the Internet.

Props for gov.uk! I’ve looked at its documentation and design system and see both as peak user experience and clarity.
I've had a consistent experience in Rails where I think for a day or two I've got a legitimate use case for one the whacky things you can do... and then as I work the problem more, it turns out: nope, the simple stuff is still the Right Way.

Someday, Ruby shenanigans, someday...

PS - Being able to pull the shenanigans is super useful during the dev process, usually to skip some yak shaving during exploration, so it's nice to have anyway.

Having just removed a metaclass from my Python code I totally agree.
I think one should not use advanced language features just because, but I also think one should not avoid using advanced language features where it is useful.

Why would the code base be worse when advanced language features are used?

Because unless you hire steadily more intelligent developers you will be headed towards a mass of code that is hard and scary to change.
I disagree with this, but I understand where it's coming from. I think you have a form of whiplash from things like: - Novices overusing the new shiny. - Java/C++/etc Jr programmers overusing design patterns. - Perl programmers solving everything with regexes. - Small startups with GraphQL, or any other large enterprise tool. - Metaprogramming, Maco's, Dependency injection, Recursion, etc when a simpler solution is a better fit.

IMHO, a "best codebase" will be just a bit more advanced than I am, with good resources for me to grok it. I want to be able to learn from it. I also don't want to be so far out of my depth that I can't make a reasonable contribution.

A pinch of salt can really liven up a dish. Not every dish needs it, but when used appropriately it's almost magic in how much difference it can make.

A lot of salt always make everything disgusting.

Huh? Salt is one of the most important foundational elements of basically all cooking.
It would be cool if newer (imperative) languages had a clear protocol to delimit the simple core of the language from the advanced, library-building features. Like Rust has `unsafe` blocks, imagine a `advanced` or `metaprogramming` block that is required in order to use advanced metaprogramming features. Then you could tell a junior to implement $MUNDANE_INTEGRATION without use of `advanced`, and that constraint could be statically verified in PRs, etc.

It seems like it would vastly simplify language evolution. The core language can have rigid limits on changes and follow the Zen of Python or whatever, while the `advanced` language extensions can be a bit looser. Features could graduate from advanced to core without breaking anything. You get a single powerful+expressive language with a "porcelain interface" without necessarily giving juniors ammo to shoot themselves in the foot.

I think D language fits your very descriptions.

You have GC by default that's Python like and intuitive programming constructs. Since for the most part D compilation is much faster than C++ and Rust compilation as long as you stick to the core non-CTFE. Heck you can use D as a compiled scripting language with REPL using rdmd [1].

Then if you want to go gung-ho, you can use the other advance features like D's excellent template meta programming, CTFE, bit banging in-line assembly, etc. D has so many advanced features that can be explored later on, even modern C++ is playing catch up with D in every releases. Nowadays D compiler also support C natively [2] and GDC compiler is included inside the venerable GCC eco-system.

There are university that teach software development and engineering class with D due to its managed, on-demand and gradual complexity, not like "in your face" complexity of the big kahuna programming language like C++ and Rust [3]. Like other compiled languages you can build real-time system even firmware device drivers in D unlike Python.

[1] https://dlang.org/areas-of-d-usage.html#academia

[2] Adding ANSI C11 C compiler to D so it can import and compile C files directly:

https://news.ycombinator.com/item?id=27102584

[3] Teaching Software Engineering in DLang [pdf]:

https://news.ycombinator.com/item?id=37620298

I worked with someone who insisted on using fx for DI in go. It's so antithetical to the entire Go philosophy, I don't even think it's an "advanced feature". It's just bringing Java cruft to a language where it isn't necessary and making everything worse.
The latest Go micro-service I have built.

About once a year roughly, for the last couple years, the opportunity has arisen to greenfield a Go micro-service with pretty loose deadlines.

Each time I have come into it with more knowledge about what went well and what I wasn't particularly happy with the last time. Each one has been better than the last.

I've been building software professionally for twenty years, and these micro-services have been one of the few projects in that time that have had clear unified vision and time to build with constant adjustments in the name of code quality.

Sounds fun. I have a similar policy but only every other year. It means I know what the hell I am talking about when in meetings with people who have slow or inefficient services. It also means my job of mostly telling other people how to solve problems is still a lot of fun for me.
Go micro-service with pretty loose deadlines

Sounds like A Money Printer, congrats.

Every code base I have ever worked on was a legacy nightmare. Every "greenfield" project I joined turned into a legacy nightmare within weeks. I have never encountered enjoyable code. I had the displeasure of wading through Spring, Hibernate and Apache HTTP client code before and they were all an incomprehensible mess.

My conclusion: You know the claim "any medication that really has an effect must also have side effects". I would like to adapt that for code: Any code that does a lot of useful and complex things must be an arcane, barely maintainable mess that can only be understood by deep study.

IMHO library code is especially challenging, as cruft has a tendency to accumulate, historical behavior needs to be preserved, and APIs are set in stone once they’re built.
"Any code that does a lot of useful and complex things must be an arcane, barely maintainable mess that can only be understood by deep study."

"Every code base I have ever worked on was a legacy nightmare. Every "greenfield" project I joined turned into a legacy nightmare within weeks"

I am sorry to say this, but it really sounds like you were either really, really unlucky, or part of the reason why it became a mess.

Complicated things are complicated. Nothing can ever change that and it requires study to understand it.

But it still does matter a lot, how one organizes the whole thing. How it is structured, documented(!), refactored. Are there competent people in charge who understand it all and kick peoples asses if they make even a temporary mess or forget to document, or do random people make changes wherever they see fit, because a deadline is ticking?

Modularisation is usually the key. Small modules do one thing and are as seperated as much as possible with as little side effects as possible.

And if one has to ship things, it is not always possible to keep it pure and if the code is not intended to live forever then this is often fine. But if the codebase is supposed to stay, then there needs to be the time to clean up the hacks. Or don't and then you end up scared touching anything.

That being said, the technologies you mentioned, I would not like to touch either..

I think that's a bit cruel. Usually mess happens because lots of people with different ideas about the future needs and best structure meet up in the code - and it's hard to develop a consistent culture.

Also codebases get too large for any one person to refactor them into shape in the time they have each day. So you end up needing people who are responsible just for keeping things in shape.

Well, that is why I said with those words it sounds like this to me, not that he or she was in fact responsible. (at least this is what I meant)

"Also codebases get too large for any one person to refactor them into shape in the time they have each day."

Which is why modules, or subprojects were invented. Or however you want to call it, if one person is only responsible for a small part and not everyone for everything. And yes, there is also the non trivial problem of time management.

yup, but the style of the modules starts to diverge as different people maintain them.

IMO one does want people whose fulltime job is to looking at a codebase orthogonally to those who are just implementing a feature. People who make sure it builds fast, is secure tries to be consistent to some degree etc.

Many companies call teams/indivuduals like this a "cost centre" and disparage it because they are dolts.

If those code checkers are experienced programmers, who do not annoy people with arbitary guidelines and standards, then yes, this might also work. But I do not think they are always necessary as a seperate full time role, if everything works normal otherwise. And with working normal, I mean there is enough time to refactor and document and clean up and is actually done.
> Modularisation is usually the key. Small modules do one thing and are as seperated as much as possible with as little side effects as possible.

100% agree. The nuance I find is that a lot times, people want to draw the lines of the module along something that's more immediately intuitive, but results in longer-term complexity.

Yes, the hard part is to define what goes inside one module. There are rules of thumb, but lots of exeptions. Taking any advice too literally can also make things awkward.
> Modularisation is usually the key. Small modules do one thing and are as seperated as much as possible with as little side effects as possible.

I think the key aspect is a bit different: understandability. That is, if you open two pages of random code, then can you roughly understand and follow what it roughly does?

Everything in a single package/module/directory can be quite understandable. It doesn't necessarily need to be modular, but it often does help. "Understandability" is more broad though, and also includes things such as not having weird hard to understand functions nested 8 levels deep, and things like that.

It also means not over-engineering things. "Our code is hard to understand, therefore, we need more architecture" is a fallacy I've seen a few times. The code wasn't hard to understand due to lack of modularity or architecture, but because it was just badly written unclear code.

"It also means not over-engineering things. "Our code is hard to understand, therefore, we need more architecture" is a fallacy I've seen a few times. The code wasn't hard to understand due to lack of modularity or architecture, but because it was just badly written unclear code."

For sure, if I have 10 tightly separated micro modules, just for the sake of it, then this is not helping clarity.

But if everything is an entangled mess of spaghetti code, then modularisation helps, so I do not have to look into 10 places to get what a certain function is doing.

Something in your answer triggered a flash back to when I worked for a phone company, 20 years ago. One team, under the leadership of our questionable chief architect, had produced our new, all encompassing backend system. This was to be the corner stone of all future development and integration, dog slow and complicated as it where. I worked as a .Net developer and had the misfortune to be among the first to integrate with this monster. Try as I might, I could not get .Net to interoperate with these services. Finally I figured out that the SOAP services was using some old deprecated versions. Going back to the architect, I ask "did you build this on Apache Axis, and please say Axis2", but no, it was just Axis, a deprecated version, that would generate webservices not supported by newer .Net version. That wasn't an problem, because: "None of our project have upgraded to those .Net versions yet"... DUDE, we've launched a brand new system based on that .Net version a year ago, and what was you f-ing plan for the future, to redo every single service using Axis2?

This guy had based a brand new system on a framework/library that was no longer maintained, even before the system was launched.

My open source projects. One in particular I've been working on for about 10 years. The code is consistent and always getting better, even though there is a lot more work that could be done.
Worked on a codebase for a large safety-critical system where everything was 100% documented, and the development guide for the project was followed so closely that you couldn't tell, across millions of lines of code, that the whole thing wasn't written by one person. Absolutely impressive levels of attention to detail everywhere, down to not even being able to find typographical errors in comments or documentation (a typo in a comment was treated just as seriously as any other bug).
Let me guess, it was very well funded and there were no fake deadlines and cross-team dependencies, am I correct or am I very correct?
Cancel the rocket launch people, we have a semantic incongruity! Correct is a binary state.
Unless you're working with Quantum Correctness.
If you allow for composable statements, the whole can be half correct.
"When people thought the Earth was flat, they were wrong. When people thought the Earth was spherical, they were wrong. But if you think that thinking the Earth is spherical is just as wrong as thinking the Earth is flat, then your view is wronger than both of them put together."
I bet you are very correct
Not only was the funding effectively unlimited, but it was developed over 20+ years (so the whole program was designed to the standard that "the next person touching this code will never have touched this system before, and the last person who touched it isn't available to ask, and it still must be correct in the face of such change."

Nothing moved very quickly, sometimes a change request took a year from initial writeup to implementation and close-out. It wasn't creative, but it worked. Very well.

I am almost sure this is because the system would have to pass a certification procedure somewhere, and for that they would need this level of clarity. Am I right?
Having worked in a couple of safety critical companies, there are things you definitely HAVE to do, but some companies do it better than others.

Some companies have process and development practices down pat: things go smoothly, and meeting the qualification process objectives is easy, because the work has been done right the whole way.

Other companies have less established or less consistent process. They generally meet the process objectives and deliver a working product, but the development process is often more of a struggle, and there is often a lot of "cleaning up" of missed pieces at the end of the release process.

This is just to say: companies and products in the safety critical space don't necessarily have some intrinsic quality, just a higher minimum bar.

In my experience there is exactly 0 correlation between certification needed and code Quality. Right now working in a multi billion company doing SW that must pass many certifications. The code is absolut trash. The tooling is terrible. The people confuse make and cmake. All said. And the SW gets certified, because it is all a matter of how much it costs to certify. It is a kind of high level corruption, that is not seen as corruption.
Sorry, I was talking about safety critical system certifications, like for avionics. The systems you describe would never pass that for sure.
> The systems you describe would never pass that for sure.

I wouldn't assume that.

Right? Boeing would like to have a word.
In my experience there is a correlation. All software I tested that had to be certified had the lowest quality by far. Certification only cared for the happy path and lots of very specific error messages. It was extremely important that the correct error codes were returned in the specified cases. Anything unexpected like e.g. an out-of-memory-error was more or less ignored because the specification didn't consider such kind of error. Even if it made the software unusable - all that mattered was the certification.
I'd love to know if working like this is enjoyable or a chore.
After way too many years of working on several sprawling Ruby and Rails projects, I would enjoy the everloving fuck out of working on this.
Having worked in both kinds personally, it's different.

You spend more time on less topics: the variety of work is much lower, which some people love, and some people are frustrated by. There is often much less "drastic innovation", or when there is, it takes much longer than you might be used to in other industries. Favor is given to proven, predictable technologies and choices, even if that leaves some opportunities on the table.

That being said, it's something I often miss. The ability to have such solid confidence on the things you've built, and the ability to drop into nearly any piece of the system and have everything be consistent and predictable is a quality in itself. It makes debugging (often VERY rare) issues much more tractable, both at a higher systems level, as well as digging deeper into the code itself.

Sounds like a well maintained tractor from 1950s that is still chugging along.
> Favor is given to proven, predictable technologies and choices

I feel like this goes two ways too. Sometimes people favor technologies they’ve used before, regardless of how many problems they know it causes.

If it’s predictable that certain tech will cause you issues years down the line, do not choose it again.

Yep. Sometimes it's "known issues are avoidable issues", and other times it's "hey I wish we didn't spend xx% of our time avoiding (incl. spending time auditing syntax) or doing mitigations for the same limitations/issues over and over and over again".

The wheels do turn, just slowly.

If lives are at stake, it would probably elicit different emotions altogether.
For me, the most eye-opening codebase of my career was Cocotron, around 2007:

https://github.com/cjwl/cocotron

I was looking for a way to port my native Mac Cocoa apps to Windows. I had been already disappointed by the aimless sprawl of GNUstep.

This one-person project implemented all the essential APIs for both Foundation and AppKit. Reading the code was a revelation: can it really be this simple and at the same time this effortlessly modular for cross-platform support?

I contributed a few missing classes, and successfully used Cocotron for some complex custom GUI apps that needed the dual-platform support.

Cocotron showed me that one person with a vision can build something that will rival or even outgun large teams at the big tech companies. But focus is essential. Architecture astronauts usually never get down from their high orbits to ship something.

+1 to this.

I remember my boss asking me to build a MacOS UI we had on Windows with this. I was very sceptical that it would go anywhere.

But no, it worked. I was just open mouth shocked to see the GUI worked pretty flawlessly in Windows.

Google3 codebase. It's just so vast and it works. It's a miracle. I feel lucky to have seen it. Everytime you change it, it reruns the dependencies. Everyone has different view concurrently. Commits are efficient immutable snapshots. It's just incredible multiplayer. So massively beyond what can be done with GitHub. Really I feel it's peak codebase. I've not seen what the other big techs do but it blew my mind
It was wonderful to work with. :-) It's one of the things I truly miss from Google.
Google3 codebase very consistently has clean code, but some of the architecture there is very much not great.

Some is great, some not so much.

Some of Verizon's code was much more elegant (though much smaller scope) from an API perspective, and really leaned into advanced type systems in a way Google has not.

+1

I can not understate how much I agree with parent comment.

The opposite of move fast, build a shitty prototype and iterate is a deliberate problem solving approach undertaken by the highest caliber of engineers. The actual challenges to be addressed are effectively addressed right at the design stage.

The result is a thing of immense beauty and elegance.

I will forever be grateful for the opportunity I had to see this magnificent piece of engineering in action.

Cannot overstate.
(Aside). To expand slightly, what robertsdionne is highlighting is the changing usage of this expression. In its original sense, e.g an issue is so important that it is impossible to overstate its importance. It is now increasingly used the other way around.

Old me would have said it’s used wrongly, but this happens all the time with language. Especially things being used in the opposite of their original sense, e.g. inflammable for flammable.

In my mind, "cannot overstate" always meant "impossible to overstate", but I think some people interpret/intend "cannot understate" to mean something like "must not understate". I don't know if that's really what they're thinking, but it is how I make sense of it. I have come to just avoid such constructions.

Edit: reminds me of an ancient SNL skit with Ed Asner in which he's a retiring nuclear engineer and as he heads out the door he says to his incompetent co-workers "Just remember, you can't put too much water in a nuclear reactor".

> opposite of their original sense, e.g. inflammable for flammable

Inflammable was never the opposite of flammable. Those word have always been synonyms. The opposite was always non-flammable.

I, like many others here, echo this sentiment. While I disliked working in ads, the experience of working with that repo+tooling is unmatched by anything else in my career.
If the tooling is so far beyond anything publicly available, why don't you guys make something like that and make millions?
Perhaps they make more than that just by using the tooling internally and creating/maintaining other stuff with it — that’s a major competitive advantage
I strongly suspect it’s not that useful for a lot of businesses.

So many have their code split between a two dozen clouds, BA tools (… does Google put that in the monorepo too? Or is that, which is a lot of the code at most businesses, not “really” code at Google?), vendor platforms, low-code tools you all hate and are trying to deprecate but god damned if you aren’t still spending dev hours on new features in it, et c…

I bet achieving anywhere near the full benefits would first require retooling one’s entire business, including processes, and bringing a whole shitload of farmed-out SaaS stuff in-house at enormous expense, most places.

It’s not just tooling; processes, infrastructure and dedicated teams for centralized infrastructure are what makes Google’s monorepo what it is. FWIW, most of the tools are publicly available or have good publicly available counterparts. What’s likely missing elsewhere is funding for infrastructure teams.
I’d love to see something like that applied to a project like Debian. The tools already exist. The cost of switching is too great, and you’d need everyone to learn your new system before you’d see the benefit.

I wrote a bit about this here: https://blog.williammanley.net/2020/05/25/unlock-software-fr...

I understand that nix have made some progress in this direction, but I don’t know any more than that.

I dunno man, I work at google and this comment feels like circlejerk. It's not held to THAT high a standard. We absolutely commit janky MVPs and iterate.
He is referring to the codebase itself, not individual projects or CLs.
As a Google outsider, that's not at all the impression I got from the paragraphs.

This:

> It's not held to THAT high a standard. We absolutely commit janky MVPs and iterate.

seems to very directly address this:

> The opposite of move fast, build a shitty prototype and iterate is a deliberate problem solving approach undertaken by the highest caliber of engineers. The actual challenges to be addressed are effectively addressed right at the design stage.

If your claim is that "Well, if you look at the codebase AS A WHOLE, there's absolutely no iteration and shitty prototypes, it's all designed right from the start."... well, I can't see any way that codebase came into existence without being built up by individual projects and changesets.

So, yanno, when folks on the ground report that individual projects and changesets/PRs/whatever ARE using the "Commit something barely serviceable, test it out, and iterate." process, statements like "We always get the design right before we write even a single line of code!" definitely come off as a circlejerk.

Google's a very large software house, it has a very exclusive hiring process, and it (reportedly) has internal tooling that's very well-adapted for the problems a company of its size faces. But Google is still hiring from the same pool of programmers as everyone else, and the odds that zero of those that it hires will work best with the "Get something out there to field-test, and use the test results to make it more suited for field use." method of development are absolutely zero. Given Google's size, the odds that zero of those will never be able to negotiate to work in the way they work best are ALSO zero.

(And -frankly- I expect that this method of development gets used a lot in the company. You can burn an assload of time on simulators (and what is the "What if?" game, but a in-brain simulator?), but in the realm of software, it's not-infrequently the case that the simulator with the best ROI is real-world deployment.)

While Google is a monorepo, that doesn't mean that every part of the codebase is equally critical.

People don't yolo submit CLs to core library components for example.

OTOH submitting somewhat hacky code to a new project while you iterate on it doesn't harm the rest of the code base that much since very little will depend on it.

Okay, sure.

How does this comment address any parts of my statement?

I'll even quote what might be the most important part for you:

> If your claim is that "Well, if you look at the codebase AS A WHOLE, there's absolutely no iteration and shitty prototypes, it's all designed right from the start."... well, I can't see any way that codebase came into existence without being built up by individual projects and changesets.

He is referring to the codebase not the code?

What does that mean?

I think they are referring to the developer experience of working in the codebase, rather than the quality of the code.
I think skill is also an issue here, in both directions. I have worked with a company that followed the opposite of move fast and it just turned into a 'who is the most correct' and 'what is the most elegant code' competition. We didn't push out a single new feature in the three years i'd worked there at that point. There was so much focus on code that we gave almost no time to business requirements.

The wider implication of this was that the number of tickets we got dropped dramatically, because users knew they'd never be resolved anyway.

Balance is key.

I honestly can't tell if this comment is sarcastic...

google3 is a quagmire and it's getting worse by the day

Agreed. Once long (10+ years) ago, google3 may have been peak codebase. Nowadays, not so much.

Edit: I guess it also depends at what level of abstraction you work. High: can be easy breezy. Low: oh boy.

I agree but would add that within the codebase there are also vastly different experiences. The search ads serving stack for example is phenomenal, it is >10 years old, very complex and gets contributions from hundreds of engineers every week, yet it is quite easy to understand what is going on anywhere and it has great performance. But I also had to work with other code which had been through 3 rewrites, none of which had ever been totally completed, so there were overlapping paradigms everywhere and working with it was just a nightmare. Tests never caught anything leading to several post mortems.
It was certainly great up to like 2015, perhaps later?

You could spend all day finding fun code to read.

I always like the main signal (eg sigsev) handler.

I worked with a very experienced engineer who created his own WAF (software-based) using Java. From a purely architectural perspective, it might not be the best (I might have used an ISAPI filter back then), but the code itself was very efficient, well-written and documented. I used it several times as a teaching example.
I haven’t contributed code to it, but I’ve read lots of code in it ( for work reasons ) - I really like the golang compiler and library codebase.

About codebases I’ve written code for, the best one strived for simplicity, and was driven by very strong engineers who actively considered code hygiene ( in the broadest possible sense ) a first class citizen.

Go, SQLite, Redis are the three that came to mind for me. Simple, readable, fairly flat hierarchy.
The one in my previous job, which was an admin board for a market intelligence application. Ultimately, the reason it was good was because the engineers had zero ego on top of having excellent skills. The team that set the codebase were basically 4 seniors and 3 principals (the client actually did pay for top talent in this case) so not only everything was based on industry standards, written elegantly and organized perfectly, but every time some new requirement came up, these senior / principal engineers would discuss it in the most civilized matter I have ever seen.

E.g, "we need to come up with a way to implement X". Person A gives their idea, person B gives another idea and so on until everybody shared their thoughts. Then someone would say "I think what person C said makes the most sense" and everybody would agree and that was it. 30 minutes to hear everybody out, 3 minutes to discuss who will do it and when and the meeting was over.

I think the biggest testament to this code base was that when junior members joined the team, they were able to follow the existing code for adding new features. It was that easy to navigate and understand the big picture of.

This is the approach I tried to take as an IC frontend on a building management system dashboard about a year ago. It was basically me and one other frontend, one full-stack, and a QA or 2, plus a manager. My manager I guess had written a lot of the code we'd be interacting with, and I guess was somewhat protective and reluctant to delegate authority over decisions around it. It was a big refactoring project, and I just encouraged my colleagues to take the things they wanted, like rebuilding the state management system. We'd discuss the requirements and why it was necessary, and I'd look for reasons to agree rather than disagree, then we'd be off. Something burnout has taught me is that the marginal differences between one implementation detail or another are not worth getting hung up on unless they pose a real problem, especially when someone else gets to decide that doing it fast is priority (sprints).
> Something burnout has taught me is that the marginal differences between one implementation detail or another are not worth getting hung up on unless they pose a real problem, especially when someone else gets to decide that doing it fast is priority (sprints).

I think this is true. And a self-imposed problem (should a problem arise) is much less frustrating to fix than one that came from a decision imposed by someone else, even if the latter avoided loads of other problems. Sometimes it's better to let people make mistakes (as you believe them to be) and correct them later.

And often, the cost of indecision is much bigger than the margin in utility between the optimal and the second-to-optimal solution.
Great example as it shows that code base really is a side consideration and a proxy to team quality.
Another reminder that soft skills multiply the effectiveness of hard skills.
Exactly.

What is development if not communication? Between the clients, the coworkers, the users, and the computer

I'd take a mediocre dev with no ego and ok social skills on my project any day over an egotistical genius who can't work with other people or effectively communicate their ideas

Otherwise known as "you are not your code" and "you are not your ideas." Essentially, criticism against your code is not a personal attack (and other corollaries). Its definitely something I aspire to, but its surprisingly hard to get right.
Last year I worked for a client that gave me a lot of time, money and autonomy to lead dev on a critical software rewrite.

We got a small team of competent people, with domain experts to peer code with the devs.

It was wonderful. We could test, document and clean up. Having people who knew the trade and users at hand removed second guessing.

The result was so good we found bugs even in competitors' implementations.

We also got x5 in perfs compared to the system it was replacing and more features.

Good for you. Magical moments in careers are hard to find in my experience but they are so satisfying when you get there.

Glad whomever was over this didnt just drop the "dont rewrite" joel spolsky article and fight making it happen.

I actually was the one telling them not to rewrite, lol.

But the original code was a mess of matlab spaghetti, they couldn't find a way to hire for that. Not to mention turning it into a web service was already a big hack of java parsing a raw dump of matlab datastructures that nobody dared to touch.

I had to read the matlab code, and it tooks hours to decypher a few lines. Plus the language doesn't have great debugging and error handling capabilities and the tooling is quite terrible.

So rewriting to python won, and for once, I must say it was a good call.

Joel has since said that that he doesn't really agree with that advice anymore, at least not in the same way. Super annoying that it gets parroted over and over again as though it's the word of the lord.
I agree it shouldn't be parroted as though it's the word of the lord, like any advice it will be more or less applicable on specific situations. I've been on both sides of the let's redesign a thing plenty of times.

The balance is somewhere in the middle, it's valuable advice for what can go wrong when you don't think through the implications of the decision making process, understand how and why the system works the way it does, and what risks exist for what can go wrong with a big redesign. But like anything, if the risks are understood, then those risks can be accepted, mitigated, or rejected if appropriate, or provide guidance on why the redesign investment isn't worth the associated risks.

Similar thing.

Had time and autonomy from a client, so took sweet time examining the domain, the existing systems et al. Spent a few months writing the basis and the framework around what will be done, based on years and years of experience I had with bad frameworks and codebases, combined with working on the same domain for their parent company years ago.

And it worked. We delivered features insanely fast, hundreds of forms were migrated, feature generators would create 90% of the boilerplate code and the code was small, readable and neatly clustered. Maintaining it was a piece of cake, leading to us not having enough work after a while so I negotiated our time to half a week for the same money.

After a while, client deemed us too expensive to pay for only 2.5 days of work - after all, how does it make sense - if we are paying them that much, they should work 5 days!

So they cut us out. Two things happened:

1. Devs that got moved to other projects in the company told me they didn't know development could be so smooth and tried to replicate it in future projects, even tho they say it failed a lot of lessons they picked up from the framework were highly relevant in their future careers.

2. The company found a cheaper developer, he said "this is unusable and has to be rewritten" and rewrote it into "clean code", taking longer than the original project took. At least he works 5 days a week now.

We were going to do this. Get several months to set up a clean base for our new system with only the four most competent people working on it.

Then I went on vacation for a week.

Come back to a system that needs to be delivered to prod next month with 20 randos submitting PR’s…

WTF happened. Did you learn nothing from previous failures?

What was the lesson they where supposed to have learned?
9 women cannot have a baby in a month.

There’s a limited amount of people that can work on a system at the start of a project if you want it to be a coherent whole, so that you can have everyone iterate on it in the same style later. If you start out with 20 (junior) engineers, you get a kind of frankenstein where all issues are waved away because ‘we need to deliver in a week’

Aka "Brooks's Law" or more broadly, the idea that "adding manpower to a late software project makes it later" from Brook's "Mythical Man-Month"
My past three employers code bases: mono-repos, Bazel, lots ot C++ and Python, thousands of libraries and tools, code generation and modeling tools that are fully integrated into the build, easy cross compilation, large integration tests just one bazel test invocation away, hermetic and uniform dependencies...
Bazel Python does not cross compile IIRC. How was this achieved?
Python is only used for build-time tooling (modeling, code generators) and developer tooling. All the on-target code is C and C++, and only that is cross compiled (Linux, QNX, various RTOS and x86+aarch64)
Laravel, once you get the hang of it everything just works, and using a debug bar to optimize database calls is very satisfying.
One of them stands out, due to being super-productive, over years, and then decades.

A large system that was originally written by only two super-productive engineers (I mean real engineers, both with PhDs in an area of Engineering). And a comparably capable and essential IT person.

The reasons for the super-productivity include one of the developers choosing great technology and using it really well, to build a foundation with "force multiplier" effects, and the other developer able to build out bulk with that, while understanding the application domain.

Another reason was understanding and being pretty fully in control of the code base, so that, as needs grew and changed, over years, someone could figure out how to do whatever was needed.

One of the costs was that most things had to be built from scratch. Over time that also proved to be an advantage, because whenever they needed (put loosely) a "framework" to something it couldn't do, they effectively owned the framework, and could make dramatic changes.

When I said "costs", I mean things like, many times they needed to make a component from scratch that would be an off-the-shelf component in some other ecosystem. So if someone looked closely at how time was sometimes spent, without really understanding it or knowing how that panned out, it would look like a cost that they could optimize away. But if they looked at the bigger picture, they'd see a few people consistently, again and again, accomplishing what you'd think would take a lot more people to do.

It helped that the first programmer also became the director for that area of business, and made sure that smart engineering kept happening.

Someone might look for a reason this couldn't work, and think of bus factor. What I think helped there was the fact that the work involved one of those niche languages that attract way more super programmers than there are jobs. "Gosh, if only we had access to a secret hiring pool of super programmers who were capable of figuring out how to take up where the other person left off, and we had a way to get them to talk with us...")

It was easy to imagine a competitor with 100 developers, not able to keep up, and at many points getting stuck with a problem that none of them were able to solve.

I assume you avoided identifying (or even hinting at) this "great technology" on purpose, but could you persuaded to divulge what it was?
It started as a bespoke structured data and Web architecture, which did somewhat complex things that would've been a huge headache to build and maintain in the dominant languages of the time, but were viable for one person to figure out and implement in Scheme and PostgreSQL.

That bespoke architecture and implementation language lent itself to a lot of rapid domain-specific functionality, as well as doable architecture changes over time.

Thanks! :) I hoped it'd be a Lisp or Smalltalk - great to hear I almost guessed right, though I never imagined an RDBMS to be in the mix :)
There's actually a bespoke object metamodel mapping atop the RDBMS, and it permits customers who want to use SQL direct to the database to do so.

And, IMHO, PostgreSQL is the default choice for most storage backend purposes, unless one has figured out a good reason something else is better. :)

There were also multiple kinds of large blob storage needs, and changing architecture needs over time (e.g., multi-tenant, changing authn implications of things like browser plugins, scalability, move to cloud), so systems programming skills and controlling your framework comes in handy, for not getting stuck or blocking on vendors, but just solving the problem.

What is this niche language attracting super programmers?
Scheme, Common Lisp, Haskell, [edit] Smalltalk, probably more obscure ones...

Also, Erlang, Rust, and Clojure, though those have been rumored to be employable, so no longer get as much of the filter as you get when it was just people caring strongly enough to want to use a particular language despite the unemployability.

Whether that caring happened because of skill (to identify something good), or skill happened because of caring (to invest the time in exploration and thinking and practice), these communities seem to get more than their share of great programmers.

And so you know where to look for them, and you have a reason that they might want to talk to you.

>probably more obscure ones.

Apl/BQN worth adding for the recent CUDA/vector language explosion, and has it's fair share of extraordinary programmers!

Forth an interesting one too look, although never properly used.

miniKanren/Prolog another vein to branch down (unification/logic programming)

(comment deleted)