Ask HN: How to avoid over-engineering software design for future use cases?

requirements -> tests -> specific code

if i reach the same code more than once, refactor and bother to generalize....

The more time you spend thinking about the business and how what you're building will support it the better judgement you'll develop. You might not be able to articulate or argue it but you'll have an instinct on when an abstraction is going to be useful vs brittle.

When everything is DRY and functions follow SRP, it's hard for your code not to be "future-proof."

Tests only when they make sense, not when you are mocking half the app.

You aren’t gonna need it.

As with anything tho, ruthlessly cutting stuff or rounding corners is in tension with supporting future capabilities, sometimes you do need it it turns out.

2. Understand it's very often easier, faster and better to write something trivially but twice than writing it well once;

3. Realize no matter how hard you try, unless this is something you've already done once (which brings you back to #2 above), then your information of the problem is incomplete and therefore any all-encompassing solution you may have will essentially be partial and based on extrapolation of your knowledge and guesswork.

The above also deals with a significant cause of procrastination - an internal subconscious feeling we can't do what we're about to do well, therefore we'd rather not even start it. Just sit down and tell yourself, "today I'm going to write a bad module X", and with enough time you'll be able to sit down again and write a good X.

The one thing you should spend a bit more time on is interfaces and decoupling, but as for internal implementations of things, or even entire system blocks that you can swap later on - don't bother too much. It'll be OK at the end, and you'll get there faster. Even if you need to rewrite the whole god damn thing.

“An architect's first work is apt to be spare and clean. He knows he doesn't know what he's doing, so he does it carefully and with great restraint.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used "next time." Sooner or later the first system is finished, and the architect, with firm, confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.”

The overall advice is to practice self-discipline:

“How does the architect avoid the second-system effect? Well, obviously he can't skip his second system. But he can be conscious of the peculiar hazards of that system, and exert extra self-discipline to avoid functional ornamentation and to avoid extrapolation of functions that are obviated by changes in assumptions and purposes.

A discipline that will open an architect's eyes is to assign each little function a value: capability x is worth not more than m bytes of memory and n microseconds per invocation. These values will guide initial decisions and serve during implementation as a guide and warning to all.

How does the project manager avoid the second-system effect? By insisting on a senior architect who has at least two systems under his belt. Too, by staying aware of the special temptations, he can ask the right questions to ensure that the philosophical concepts and objectives are fully reflected in the detailed design.”

That said, I think there are two steps in designing abstractions:

The first is to split up and isolate both code and data into small, simple pieces. These can easily be non-DRY (structurally) and often are. You'll have cases where you often (always?) pass things or do things in sequence in the same way.

The second is to merge the pieces with parametrization (or DI in OO), defining common interfaces, polymorphism etc.

The first part is very often beneficial and makes code more malleable, which makes future features/changes less cumbersome.

The second part however is the dangerous but also powerful one. It can lead to code that is much more declarative, high-level and adding new features becomes faster. But the danger is that a project isn't at the stage where these abstractions are understood well enough, so you end up trying to break them up too often.

I try to default to writing dumb, simple small pieces and deferring abstractions until they become provably apparent. Fighting the urge to refactor into DRY code.

Now there is also another issue with writing "future proof" code, which doesn't involve abstractions but rather defensive programming, which is an entirely different issue.

In 2017 I started https://wordsandbuttons.online/ as an experiment in unchitecture. There are no generic things there whatsoever. There are no dependencies, no libraries, no shared code. Every page is self-sustaining, it contains its own JS and CSS, and when I want to reuse some interactive widget or some piece of visualization somewhere else, I copy-paste it or rewrite it from scratch. In rare cases when I want multiple changes, I write a script that does repetitive work for me.

This feels wrong, and I'm still waiting when it'll blow up. But so far it doesn't.

Sure, it's a relatively small project, it has about 100 K lines along with the drafts and experiments. It is although inherently modular since it consists of independent pages. But still, this kind of design (or un-design) brings more benefits I could ever hope for.

Since I only code that I want on a page, every one of my pages is smaller than 64KB. Even with animations and interactive widgets.

Since all my pages are small, I have no trouble renovating my old code. Since there is no consistent design, I forget my own code all the time, but 64 KB is small enough to reread anew.

And since there are no dependencies, even within the project, I feel very confident experimenting with visuals. Worst case scenario - I'm breaking a page I'm currently working on. Happens all the time, takes minutes to fix.

I still believe in the golden middle, of course, I'll never choose the same approach in my contract work; but my faith is slowly drifting away from "design for the future" in general and "making things generic" specifically. So far it seems that keeping things simple and adoptable for the future is slightly more effective than designing them future-ready from the start.

A junior engineer recently presented me with some completed work. There was a data schema change on one interface and an API version change on another. He had created a version of the microservice he was working on which could be configured to work on either the old or new version of each of these, with feature flags, switching functionality and maintaining the old and new model hierarchies for both. He viewed this as an achievement. While technically it was, the result was code bloat and unnecessary complexity. The API upgrade and schema change were happening at the same time, and would never be rolled back, his customisability was a net negative.

Code is a liability. Be sparse. Build with some flexibility but overall just build what is required and when the future comes, change with it. Don't build to requirements you can imagine, because the ones you don't will kick your ass.

Of course the issue you run into is that the problem software is trying to solve changes over time and then the original abstraction which were correct become wrong over time. Then you should advocate refactoring to the new abstractions if possible.

They're afraid.

Fear: If I don't plan for all these use cases, they will be impossible! I will look foolish for not anticipating them. So let's give into that fear and over-architect just to be safe. A bit of the 'condom' argument applies: better to have it and not need it than to need it and not have it.

But the reality is that if your design doesn't match the future needs really well, you're going to have to refactor anyway. Hint: there will always be a future need you didn't anticipate! Software is a living organism that we shape and evolve over time. Shopify was a snowboard store, Youtube was a dating website, and Slack was a video game.

So my answer: relentlessly cut design features you don't need. Then relentlessly refactor your code when you discover you do need them. And don't be afraid of doing either of those things because it turns out they're both fun challenges. The best you can do is to try to ensure your design doesn't make it really hard to do anything you know or suspect you'll need in the future. Just don't start building what no one has asked for yet.

Problems arise when people reach for the Cool Tools and start building Webscale things that can handle 100M operations every second. ...but the customer only needs the system to handle 4 users who type in everything crap by hand. But hey, it has a clustered database and Kafka and Kubernetes and looks REALLY good on your Resumé.

When the scale is determined, I personally like the mantra of "Make it work first, then make it pretty". First build an MVP (or an ugly Viable Product), that proves your plan actually works, then you can iterate over it to make the implementation cleaner or faster or have better UX.

If you get stuck making everything super-generic and able to handle all possible cases, you'll spend time bikeshedding and never get anything deployed. Just Repeat Yourself with wild abandon and copy/paste stuff all over until your Gizmo actually works. You can spend time figuring out which parts are actually possible to make generic later.

In practice goals start to move after horizon/2. And a new document needs to be created to capture where the business is doubling down and where it is trimming.

Better is to design for change. Keep everything modular. Keep your concerns separated. When something needs to change, you can just change that thing. When the whole basis of the system needs to change, you can still keep all the components that don't need to change. The only future use case you should design for is change, because that's the only thing you can be certain of.

So don't make things more generic than you need today, but make sure it can be made more generic in the future. And in that future, don't just add new features all over the place, but reassess your design, and look if there's a more generic way to do the thing you're adding.

I do this sort of stuff all the time on my current project, and it works quite well. There's no part of the system we didn't rewrite at some point, but all of them were easy to rewrite.

Even then, I don't dig too deeply into things. I look for two things:

* What a migration path to realistically expected features _might_ look like. If I'm debating between two column types or table setups, go with the more flexible option.

* What scenarios will be extremely hard to get out of. Avoid those when reasonable.

----

In other words, instead of proactively planning for some feature to evolve into the unknown in some way. I'm looking to make sure I'm keeping flexibility and upgradability.

Don't do work before you need it, but also don't give up optionality unless you're getting commensurate benefits from doing so.

The followup is, if we have to change/replace said system, how difficult would that be? If scaling the simplest solution would be difficult/painful, then one can start look to higher complexity solutions. The idea isn't necessarily to always choose the simplest solution, but having it in mind can be helpful. It crystalizes one of the endpoints of the simple/complex spectrum and helps weigh the pros/cons of various approaches.

As a side note, it's sort of amusing that a lot of design focused interview questions are around things like "design a twitter/instagram like system that'll scale to billions of requests per day." I've never had to do anything like that IRL, but no interviewer has ever asked me to design, say, an invoicing system that gets called rarely. So perhaps one of the reasons the arc of software engineering bends towards complexity is that we're continuously rewarding a "build massive scalable systems" mindset?

If I had to boil the first part down, I'd say you need to focus your engineering on the interface points - network protocols, schemas, API sets, and persistence formats immediately come to mind. It's a truism of the profession that those are the most expensive aspects of a system to change, and therefore they're the least likely to be mutable down the road.

That's really the easy part... the harder part is maintaining the team and self discipline to keep things simple. For better or for worse, engineering job placement (and oftentimes personal satisfaction) is highly resume-keyword driven. Organizations and people all tend to chase the next resume keyword on the assumption that it'll help them deliver more efficiently or get the next job placement or write the next blog post or whatever else. The net of this is that there's a very strong built in tendency for projects to veer in the direction of adding components, even before considering whether or not they're appropriate for use. So keeping your eye on actual, real project requirements over all else is both important, and can require convincing and political work throughout a team and organization.

I've also found that designing for unknowns can sometimes be resolved by asking questions about the unknowns until they're known. Sometimes, business stakeholders have an answer, they just weren't asked.