I wouldn't say that "over-engineering means solving the wrong problem". It's possible that the idea is basically correct, but people are directing effort towards optimizing for constraints that don't really exist or can be dealt with once a better picture is in place, whether it's the mythical PMF or just "we now understand what the users want, let's build that".
The worst clusterfuck I've ever worked on was a web application that was actually solving a real problem fairly well, but the team was spending time building an absurd Rube-Goldberg contraption of microservices when the entire platform had less MAU than my hobby website. It wasn't the wrong problem, but it certainly was over-engineered!
And sometimes optimizing for constraints happen because someone from above (perhaps someone with a title of Director who doesn’t actually write code) imposed their constraints on the team without understanding the ramification of these constraints. It takes a manager with very good people skills and/or very good writing skills to persuade that someone to drop their constraints.
I disagree, you’re optimising many different things and there is not a simple way to compare them. The very concept of Pareto optimal is about this! There is not one uniquely perfect solution to a problem most of the time.
Spending too much time gathering requirements can be a bad business decision too. Those requirements are often not even accurate.
Building a perfect system may not be over engineering to solve the problem perfectly… but what if you don’t need to solve it perfectly? What if it’s better to fail on some requirements to deliver sooner or cheaper or deliver this and some other project?
I disagree, requirements are only a part of the engineering process. Perfection will be redefined after the product, tech, team knowledge changes, which will constantly.
Spend too much time thinking into perfection, trying to achieve it and you will get a convoluted system with lava-layer patterns full of complexity astronauts.
I'm all for pushing back against "let's not make perfect the enemy of good." I hear that all the time in reference to software that is usually quite bad, and often a little evil to boot--nothing good anywhere in sight except some poor schmuck trying to find a way to make their job something worth taking pride in.
I'm not sure I agree that systems are products though. The product mindset is toxic. It means that you've got goals which are independent of the user's goals (typically to make money, which sometimes means doing something dastardly to the users on behalf of the shareholders).
All the best software is more in the "tool" category and less in the "product" category. Usually it's made by the users, only bothers with solving problems they have, and has no ulterior motives.
You're saying it in jest but this is closer to the right answer.
Features make you money. A large enough project has an infinite backlog. You will be unlikely to make much money from clearing out your backlog.
As a guy who was a part of projects with an infinitely growing backlog (multiple times), I can tell you for certain that _not_ periodically addressing parts of your backlog absolutely will stop you from making more money; even lose you some due to customer churn.
The number of times I've heard that phrase used to justify shipping absolute crap utterly dwarfs the times it's been used to prevent over-engineering something that's already in a good state.
In a thirty year career of software engineering, I haven't once come across one of these mythical perfectionists that everyone is constantly retelling cautionary tales of, who endlessly rewrites perfectly serviceable software and never releases it. I have, however, worked at multiple companies that have collapsed under the weight of their unmaintainable spaghetti tech stacks.
Nuanced, but I’ve heard “don’t let perfection be the enemy of completion”, which is real in the cases of not shipping because one is constantly polishing/perfecting, or, as I heard sitting beside phk[0] at BSDCan commenting on rejecting some environmental sensors into FreeBSD, something to the effect of “We need ‘good’, not ‘good enough’.”, which draws a line at accepting janky “solutions” that are architecturally weak, or otherwise look obviously problematic.
Yes, agreed, that quote is not as relevant to software development. Some would make the case that software isn't good unless it's perfect. Engineers will use the word 'correct' (e.g. correctness); and by that, they mean; is it perfect... And it's not wrong to demand it because the smallest gap can be a critical security vulnerability.
Its about pragmatism. The problem is that a lot of software engineering tends to be not about building a product to service the buisness need.
Lets be honest, a lot of the time engineering decisions are made because "it would be cool to use x" rather than "this is the fastest path to achieving business goal z"
"over engineering" in a lot of cases is just either engineering gate keeping "oh you don't know how to do x properly, its soooooo much more 'performant' than y" or design by CV "lets use kafka, k8s, mobile app, AI, RAG" etc etc etc.
in 99% of cases, the architecture really doesn't matter that much, you product is never going to need what ever fancy system you put in. Mainly because people over estimate scale.
The thing that matters is that the customer is able to use it, and know how to use it. Everything else is almost irrelevant.
Popular view to dunk on meaningless perfectionists, sure, but your comment conveniently does not address "let's have this architecture because we know these new requirements are coming and we know we can't accommodate them with the current one".
"We're not trying to build a perfect solution here" is not something said to assuage over-engineering or encourage sloppy work.
It's said to head off a specific complaint from a specific kind engineer who will object that the proposed solution won't work because it doesn't cover some obscure edge case which rarely comes up in production.
"We're not trying to build a perfect solution here" is saying "We acknowledge not everything will be covered, we're setting the requirements at the 90th percentile use case".
This was my initial reaction to reading this post as well.
Additionally, as I get older, I find the sentiment of "we're not trying to build a perfect system here" is less about "let's just go fast vroooom" and more akin to saying "I've been humbled before by thinking I had the perfect mental model of the universe before a single user touched the product."
I've been that engineer and with good reason. That obscure edge case which rarely comes up in production is very disruptive when it does come up. The product person waving it off is also not the one that will get paged at 2am to address the issue when it occurs. Accepting that 90th percentile use case is infuriating because it is tacit consent for an unfinished solution with the rest being made up later with additional toil, now constrained by load bearing things you cannot change. Thanks for rushing the thinking and making my life harder later for no reason.
What is the qualifier of "rarely". One in ten, one in hundred, one in thousand? What is the failure in these cases? I could accept rarely if that is one in trillion to one in quadrillion range say. Truly rare cases. But not if it might be for seen reasonably.
My grandpa used to say "we're not building a piano" when doing things like building a quick shelf or something in the basement and be done after a quick measure and a few cuts. My other grandpa would spend an entire day measuring, sanding, pulling out a router etc. Both valid, but if you just need something to put some paint cans on, it doesn't require a day of work.
I find it that in wild it is quite often used by product people to push crap ASAP because they don’t want to spend time finding out even basic constraints in the system they supposed to know like the back of their hand.
Then engineers are on the hook because they run into those constraints while building and everyone always blames „those lazy software developers” ;)
> We acknowledge not everything will be covered, we're setting the requirements at the 90th percentile use case
I'm ok with this as long as dropped requirements are documented and nobody wakes you up at night when those edge cases show up in production. Also, when the next feature needs to build on top of those dropped edge cases, you are given enough resources to redo the previous solution.
In my experience when a manager says that it means “let’s just ignore the architecture and use a short cut if we can get the security team to rubber stamp it”.
I agree with the actual principle I think this article is trying to put across (don't give up on technical quality too early by dismissing it as "perfection") but I strongly disagree with the framing.
My biggest problem comes from:
> Every constraint on the table. Tighten those enough and something interesting happens, you end up with only one possible solution. And that solution is, somewhat ironically, the perfect one.
Constraints are not rigid and unchangable [1] but flexible things which can bend to solve the problem at hand. Constraints can be: deadlines you impose, frameworks you are working within, the shape of the output schema, performance characteristics etc. And because they trade off against each other, tightening them leaves you with several solutions, each fitting a different trade-off.
Constraints also change over time as you and your team's collective understanding of the problem, the environment or priority of the problem change. This needs to be considered when building the initial system too: baking in some flexibility means giving up some of today's "perfection" to deal with future variance.
Understanding what constraints matter, which can be shaped and trading them off against each other is, to a very large degree, the job of software engineering. So no, I don't think you enumerate the constraints and find the only solution which fits; this is a very simplistic take on the complexities of building software.
[1] unless you work in a dictatorial company in which case my condolences!
"With one big caveat: you need a very clear set of requirements"
Most new product launches are an exercise in figuring out what the product requirements should be through trial and error (really: through ongoing dialogue with your users). Even mature products can have requirements change over time as the market changes.
I think internalizing this reality is why most senior engineers who work in domains that touch the messy real world will reflexively push back against perfectionism.
> Most new product launches are an exercise in figuring out what the product requirements should be through trial and error
Sure, but these have to be well-defined trials. In other words, yes, you will test several hypotheses, but your hypotheses have to be hypotheses, not hand waving.
I concur. Most of the most painfully over-engineered swamp projects I've dealt with have been a result of not understanding the requirements.
Of course, this is not to imply that it's necessarily possible to know every requirement perfectly before you begin - I find that I frequently don't really understand the requirements until I've built the damn thing and it's made first contact with the real world.
So I usually toss that first version out, and build the second one. And that one is usually good enough that I don't immediately want to throw it out the window.
And by the time I get through the third version, usually a fair bit after the second is done, it's pretty damn good.
Because now I understand the problem and the requirements so much better than I did at the start, and I better understand what the system needs to be to solve that problem.
Will it be perfect? Never, except in a static and unchanging universe. But we can approach optimality for a given moment in time, which is functionally the same thing.
Is it though? Engineering is providing the cheapest solution that corresponds to what your customer asked. As it goes: anyone can build a bridge with infinite budget, only an engineer can do it for cheap.
A system that's un-necessarily complex for what it does is over built, and under engineered.
I think it's important beyond semantics, because sometimes you have to propose designs that wouldn't be in books, yet they're the cheap and fit with all requirements, and are thus well engineered.
Now the difficulty is to figure the requirements that are not being expressed. "Allowing an occasional resurfacing" and "don't explode when there's icing" are obvious ones; the hard one for our job are "allow for adding another lane" or "allow support for trains as well"... because you know these might come.
As engineering is the act of solving technical problems, over engineering is about putting too much engineering effort on aspects, features or products that don't have a linear payoff to the budget spent.
E.g. I worked in a company that was obsessed with unit test coverage metrics and the effort of maintaining the test suite was considerably biting in the ability to move the product. The ratio was 25% of product and 75% of unit tests.
And the payoff was small if not even negative, impacted morale, productivity and actively pushed back against refactors, because any large refactor was met with a disproportionate effort in unit test rewriting. Let alone the fact that as you were mocking external dependencies, and the mocks costed engineering effort and internals digging, it also suppressed work to keep it up to date.
I never got to convince the org to shift focus on E2E testing, which answered the real questions: does the product make money and works as expected. Uni test had to be used when writing a parser, not to validate some react scroll component as browser apis and the triggers mimicking costed days of work.
In fact we often had gigantic all green test suites for broken products.
That's to me over engineering an aspect of the engineering cycle.
Let me do that for you. I think perfection is subjectively met at the end of one's consciousness about a problem. A higher being will see the problem more deeply, and its perfect solution could therefore be more perfect than that of a lesser being.
Imagine that one has to implement a function to multiply two large integers. A junior programmer will likely use the default multiplication operator. A mid-level programmer will add a couple of strategies with a weak heuristic for routing, and perhaps with basic caching. A senior will sample the incoming requests to identify which strategy might empirically be good, and ensure that it is being hit, with optimal fallbacks. A superintelligent AI might design custom kernels, and design a custom model for routing. A god will... we don't know what a god will do. Perfection, like beauty, is in the eyes of the beholder.
> Every constraint on the table. Tighten those enough and something interesting happens, you end up with only one possible solution.
Is there always a unique solution? If constraints A is tightened and then determines the outcome, could that not also happen with a tightened constraint B that determines a different outcome?
If something can be defined in rigorous mathematical terms and proven correct, that can save a lot of trouble downstream dealing with exceptions. Not all problems can be solved so neatly, but it's often worth putting in the effort to get from 99% to 100% correct if it means you never have to read or touch it again.
Perfection is not "perfectionism". Former is provable given set of constraint. Later is anxiety due to fear of criticism.
"We don't want to build the perfect solution." Can both mean "no need to cover use cases we don't care about" and "no need to deal with divizion by zero in a product, where a number is divided by the user entered number".
Anyone saying "don't make perfection the enemy of good" is using a thought terminating cliché, avoiding the conversation of what is good enough. Worse, it is often used by people to drag quality down. If you're creating the "minimum viable product" you usually create a product that isn't working.
But, perfection doesn't exist. Most solutions in the world have no global optima. There are always tradeoffs. You must choose. You must argue with your peers to figure out that tradeoff. Perfection has infinite depth in detail. You must optimize. You just learn the unknown unknowns in an every moving landscape.
But you should still chase perfection. Like you chase a utopia. Like you chase your dreams. There's always something to improve on. Chasing perfection while knowing it can't exist means you will continue to search for the flaws. It means you will continue to improve. "Over engineer", because that's just engineering. Make things actually work, while recognizing they're always broken somewhere. Don't get offended when someone points out a flaw, you already know it's not perfect, so figure out if it is a tradeoff or can be fixed. Just keep improving things, because otherwise they keep getting worse
I really like the article above in that complexity does have to live somewhere. If you have a hard problem to solve, there is a minimum level of complexity (essential complexity) that can't be further reduced, only moved from one place to another.
And yet we often add accidental complexity by making bad choices (thinking of you, microservices). If your choice is bad enough, you get all the disadvantages (challenges with distributed systems, complicated debugging), but without any of its advantages (a distributed monolith doesn't get you any deployment or team independence).
Telling one from the other, well, is something I'd like to learn and that seems to require both experience and some sort of intuition.
I think over-complicated and over-engineered are not the same thing.
Over-complicated is adding too many features, too much mechanism, too many moving parts. Over-engineering is far exceeding the requirements in an unhelpful way. For example: build me a treehouse. Said treehouse could probably be made of wood. If you made it out of concrete and steel, it might be a heck of a lot stronger and last a heck of a lot longer, without being more complicated (just more expensive). Too strong is over-engineered. If you made a treehouse with 13 bedrooms and an elevator, glass windows, solar power, and running water, that's overcomplicated.
It's not over-engineering if you are committed to what you are building for the long term, and know you'd very likely need stronger architecture at a certain time.
No one starts a project for perfectly objective reasons. Your choice of careers was not perfectly objective, you decided you liked a field, and it seemed you had enough aptitude for it, and then you did the work to get into it, learning as you went along and adjusting.
So it is with all kinds of decisions that go along with starting or changing a software project. Of course are your architectural and engineering acumen increases, you get a goo feel for how to choose good constraints and requirements, and then within the those constraints it is now up to you to find optimal solution for them, not the most over-engineered one.
Like I said before, sometimes going from A-B-C-D is much less efficient (and can actually be counter-productive) in the aggregate than going form A-D directly, even though YAGNI would dictate A-B-C-D. So long as you generally know that D would be needed sooner or later, A-B-C-D can be counter-productive because it causes to you to miss the forest for the trees. Even if you are not doing D immediately, some thought given to D can improve greatly your understanding of the context of A and the roadmap, leading to better design.
Note that string design instincts honed over time make this process work, not building out stuff just because of fads (e.g., microservices) or for some functionality you do not actually need. So here is my point of agreement with the article: the reason teams might waste time over-engineering is not because they are exploring a design space and objectives that is larger than what is strictly required for now; it is because they lack good architectural skill and experience and instincts. Again, considering D upfront can actually simplify the A you build now.
> For someone else it's the wrong choice, they don't know Python
The author ruins their entire argument with this one claim.
If "we picked this tool because we know it best" is a legitimate requirement, then every tool choice is justified as a "perfect" choice because it's what the architect was most comfortable with. If your emotions and current knowledge levels are considered reasonable justifications for a "perfect" solution, then all solutions are perfect solutions; they simply haven't had enough emotional justification yet. If all solutions are perfect solutions, then none are.
There are, ultimately, two kinds of software - those that need to ship by a deadline, and those that don't. A deadline forces you to eject dead weight that you don't need - requirements have a habit of getting clarified real fast when you need to build to a deadline. If you had time to over-engineer despite a deadline, you should consider working for a more productive organization. Meanwhile, the concept of "over-engineering" is a little vague for software that doesn't have a deadline. If you don't have a deadline, you don't have to compromise on quality. "Over-engineering" is then just a value judgement that you made poor use of your infinite timescale and built the wrong things with it. But who is making that judgement? Not the person who built it, not the person who funded it (i.e. usually self-funded as a hobby project), and not the person who uses it (since over-engineering is an implementation detail, rather than a product choice), so who cares?
edit: to clarify: "pick a stack you know already since we don't have time to learn a new one" is a totally valid requirement. But I disagree that it means that you built a "perfect" solution with it. I also disagree that you usually need to build perfect software - getting comfortable with adequate is how most people ship most software.
Perfection is always relative to a purpose or telos. So in the case of artifacts - which are the products of engineering, arts, and techne in general - the purpose isn't inherent to the object in the manner it is in oak trees and human beings, but projected onto the thing. It lives in the mind of the designer and the user. That's why we can say that technology is an "accidental" extension of human beings by which human purposes are realized and mediated in ways that inherent human powers cannot directly accomplish.
So, yes, if you have a purpose in mind, you can build something that can be used for that purpose, more or less effectively. The more general the purpose, the less specific the implementation; the more specific the purpose, the less general the implementation. Over-engineering here is something that indeed goes beyond the purpose of something, with feature creep as an example of moving goalposts and a slippery grasp of the limit.
However, production is only one layer of concern in engineering. You can talk about ideal artifacts whose forms perfectly enable the realization of some purpose, but there is the practicality of constructing one that must also be considered. Think of diminishing returns. Does it makes sense to invest a certain amount of time and resources into producing a more perfect artifact? If not, then pursing such perfection is itself an imperfection on the part of human judgement. That is, a more perfect engineer might know the limit of perfection for an idealized artifact, but he also knows what the perfect practical limit of justifiable effort is. Perfectionism is a sign of an imperfect engineer.
Returning to feature creep, the author might argue that the moving goalposts are a sign that the purpose is being refined, but feature creep can also mean that the shifting purpose is detached from real needs. So, in that case, the purpose served is no longer the purpose that ought to be served, and in that sense, while one may realize an end perfectly in a relative sense, the end pursued is itself the result of imperfect understanding, which is effectively what the author is gesturing at when discussing requirements.
76 comments
[ 0.25 ms ] story [ 50.1 ms ] threadThe worst clusterfuck I've ever worked on was a web application that was actually solving a real problem fairly well, but the team was spending time building an absurd Rube-Goldberg contraption of microservices when the entire platform had less MAU than my hobby website. It wasn't the wrong problem, but it certainly was over-engineered!
Spending too much time gathering requirements can be a bad business decision too. Those requirements are often not even accurate.
Building a perfect system may not be over engineering to solve the problem perfectly… but what if you don’t need to solve it perfectly? What if it’s better to fail on some requirements to deliver sooner or cheaper or deliver this and some other project?
Spend too much time thinking into perfection, trying to achieve it and you will get a convoluted system with lava-layer patterns full of complexity astronauts.
I'm not sure I agree that systems are products though. The product mindset is toxic. It means that you've got goals which are independent of the user's goals (typically to make money, which sometimes means doing something dastardly to the users on behalf of the shareholders).
All the best software is more in the "tool" category and less in the "product" category. Usually it's made by the users, only bothers with solving problems they have, and has no ulterior motives.
In a thirty year career of software engineering, I haven't once come across one of these mythical perfectionists that everyone is constantly retelling cautionary tales of, who endlessly rewrites perfectly serviceable software and never releases it. I have, however, worked at multiple companies that have collapsed under the weight of their unmaintainable spaghetti tech stacks.
Nuanced, but I’ve heard “don’t let perfection be the enemy of completion”, which is real in the cases of not shipping because one is constantly polishing/perfecting, or, as I heard sitting beside phk[0] at BSDCan commenting on rejecting some environmental sensors into FreeBSD, something to the effect of “We need ‘good’, not ‘good enough’.”, which draws a line at accepting janky “solutions” that are architecturally weak, or otherwise look obviously problematic.
[0] https://people.freebsd.org/~phk/
I think you're getting caught up on nomenclature and limited sample sizes.
Lets be honest, a lot of the time engineering decisions are made because "it would be cool to use x" rather than "this is the fastest path to achieving business goal z"
"over engineering" in a lot of cases is just either engineering gate keeping "oh you don't know how to do x properly, its soooooo much more 'performant' than y" or design by CV "lets use kafka, k8s, mobile app, AI, RAG" etc etc etc.
in 99% of cases, the architecture really doesn't matter that much, you product is never going to need what ever fancy system you put in. Mainly because people over estimate scale.
The thing that matters is that the customer is able to use it, and know how to use it. Everything else is almost irrelevant.
The end result is cutting corners and perfection is never even a thought.
Arguing extremes is easy.
It's said to head off a specific complaint from a specific kind engineer who will object that the proposed solution won't work because it doesn't cover some obscure edge case which rarely comes up in production.
"We're not trying to build a perfect solution here" is saying "We acknowledge not everything will be covered, we're setting the requirements at the 90th percentile use case".
Additionally, as I get older, I find the sentiment of "we're not trying to build a perfect system here" is less about "let's just go fast vroooom" and more akin to saying "I've been humbled before by thinking I had the perfect mental model of the universe before a single user touched the product."
Then engineers are on the hook because they run into those constraints while building and everyone always blames „those lazy software developers” ;)
I'm ok with this as long as dropped requirements are documented and nobody wakes you up at night when those edge cases show up in production. Also, when the next feature needs to build on top of those dropped edge cases, you are given enough resources to redo the previous solution.
My biggest problem comes from:
> Every constraint on the table. Tighten those enough and something interesting happens, you end up with only one possible solution. And that solution is, somewhat ironically, the perfect one.
Constraints are not rigid and unchangable [1] but flexible things which can bend to solve the problem at hand. Constraints can be: deadlines you impose, frameworks you are working within, the shape of the output schema, performance characteristics etc. And because they trade off against each other, tightening them leaves you with several solutions, each fitting a different trade-off.
Constraints also change over time as you and your team's collective understanding of the problem, the environment or priority of the problem change. This needs to be considered when building the initial system too: baking in some flexibility means giving up some of today's "perfection" to deal with future variance.
Understanding what constraints matter, which can be shaped and trading them off against each other is, to a very large degree, the job of software engineering. So no, I don't think you enumerate the constraints and find the only solution which fits; this is a very simplistic take on the complexities of building software.
[1] unless you work in a dictatorial company in which case my condolences!
Most new product launches are an exercise in figuring out what the product requirements should be through trial and error (really: through ongoing dialogue with your users). Even mature products can have requirements change over time as the market changes.
I think internalizing this reality is why most senior engineers who work in domains that touch the messy real world will reflexively push back against perfectionism.
Sure, but these have to be well-defined trials. In other words, yes, you will test several hypotheses, but your hypotheses have to be hypotheses, not hand waving.
Of course, this is not to imply that it's necessarily possible to know every requirement perfectly before you begin - I find that I frequently don't really understand the requirements until I've built the damn thing and it's made first contact with the real world.
So I usually toss that first version out, and build the second one. And that one is usually good enough that I don't immediately want to throw it out the window.
And by the time I get through the third version, usually a fair bit after the second is done, it's pretty damn good.
Because now I understand the problem and the requirements so much better than I did at the start, and I better understand what the system needs to be to solve that problem.
Will it be perfect? Never, except in a static and unchanging universe. But we can approach optimality for a given moment in time, which is functionally the same thing.
Is it though? Engineering is providing the cheapest solution that corresponds to what your customer asked. As it goes: anyone can build a bridge with infinite budget, only an engineer can do it for cheap.
A system that's un-necessarily complex for what it does is over built, and under engineered.
I think it's important beyond semantics, because sometimes you have to propose designs that wouldn't be in books, yet they're the cheap and fit with all requirements, and are thus well engineered.
Now the difficulty is to figure the requirements that are not being expressed. "Allowing an occasional resurfacing" and "don't explode when there's icing" are obvious ones; the hard one for our job are "allow for adding another lane" or "allow support for trains as well"... because you know these might come.
As engineering is the act of solving technical problems, over engineering is about putting too much engineering effort on aspects, features or products that don't have a linear payoff to the budget spent.
E.g. I worked in a company that was obsessed with unit test coverage metrics and the effort of maintaining the test suite was considerably biting in the ability to move the product. The ratio was 25% of product and 75% of unit tests.
And the payoff was small if not even negative, impacted morale, productivity and actively pushed back against refactors, because any large refactor was met with a disproportionate effort in unit test rewriting. Let alone the fact that as you were mocking external dependencies, and the mocks costed engineering effort and internals digging, it also suppressed work to keep it up to date.
I never got to convince the org to shift focus on E2E testing, which answered the real questions: does the product make money and works as expected. Uni test had to be used when writing a parser, not to validate some react scroll component as browser apis and the triggers mimicking costed days of work.
In fact we often had gigantic all green test suites for broken products.
That's to me over engineering an aspect of the engineering cycle.
Imagine that one has to implement a function to multiply two large integers. A junior programmer will likely use the default multiplication operator. A mid-level programmer will add a couple of strategies with a weak heuristic for routing, and perhaps with basic caching. A senior will sample the incoming requests to identify which strategy might empirically be good, and ensure that it is being hit, with optimal fallbacks. A superintelligent AI might design custom kernels, and design a custom model for routing. A god will... we don't know what a god will do. Perfection, like beauty, is in the eyes of the beholder.
Is there always a unique solution? If constraints A is tightened and then determines the outcome, could that not also happen with a tightened constraint B that determines a different outcome?
[1]: https://max.engineer/premature-optimization
"We don't want to build the perfect solution." Can both mean "no need to cover use cases we don't care about" and "no need to deal with divizion by zero in a product, where a number is divided by the user entered number".
Anyone saying "don't make perfection the enemy of good" is using a thought terminating cliché, avoiding the conversation of what is good enough. Worse, it is often used by people to drag quality down. If you're creating the "minimum viable product" you usually create a product that isn't working.
But, perfection doesn't exist. Most solutions in the world have no global optima. There are always tradeoffs. You must choose. You must argue with your peers to figure out that tradeoff. Perfection has infinite depth in detail. You must optimize. You just learn the unknown unknowns in an every moving landscape.
But you should still chase perfection. Like you chase a utopia. Like you chase your dreams. There's always something to improve on. Chasing perfection while knowing it can't exist means you will continue to search for the flaws. It means you will continue to improve. "Over engineer", because that's just engineering. Make things actually work, while recognizing they're always broken somewhere. Don't get offended when someone points out a flaw, you already know it's not perfect, so figure out if it is a tradeoff or can be fixed. Just keep improving things, because otherwise they keep getting worse
https://ferd.ca/complexity-has-to-live-somewhere.html
I really like the article above in that complexity does have to live somewhere. If you have a hard problem to solve, there is a minimum level of complexity (essential complexity) that can't be further reduced, only moved from one place to another.
And yet we often add accidental complexity by making bad choices (thinking of you, microservices). If your choice is bad enough, you get all the disadvantages (challenges with distributed systems, complicated debugging), but without any of its advantages (a distributed monolith doesn't get you any deployment or team independence).
Telling one from the other, well, is something I'd like to learn and that seems to require both experience and some sort of intuition.
Over-complicated is adding too many features, too much mechanism, too many moving parts. Over-engineering is far exceeding the requirements in an unhelpful way. For example: build me a treehouse. Said treehouse could probably be made of wood. If you made it out of concrete and steel, it might be a heck of a lot stronger and last a heck of a lot longer, without being more complicated (just more expensive). Too strong is over-engineered. If you made a treehouse with 13 bedrooms and an elevator, glass windows, solar power, and running water, that's overcomplicated.
No one starts a project for perfectly objective reasons. Your choice of careers was not perfectly objective, you decided you liked a field, and it seemed you had enough aptitude for it, and then you did the work to get into it, learning as you went along and adjusting.
So it is with all kinds of decisions that go along with starting or changing a software project. Of course are your architectural and engineering acumen increases, you get a goo feel for how to choose good constraints and requirements, and then within the those constraints it is now up to you to find optimal solution for them, not the most over-engineered one.
Like I said before, sometimes going from A-B-C-D is much less efficient (and can actually be counter-productive) in the aggregate than going form A-D directly, even though YAGNI would dictate A-B-C-D. So long as you generally know that D would be needed sooner or later, A-B-C-D can be counter-productive because it causes to you to miss the forest for the trees. Even if you are not doing D immediately, some thought given to D can improve greatly your understanding of the context of A and the roadmap, leading to better design.
Note that string design instincts honed over time make this process work, not building out stuff just because of fads (e.g., microservices) or for some functionality you do not actually need. So here is my point of agreement with the article: the reason teams might waste time over-engineering is not because they are exploring a design space and objectives that is larger than what is strictly required for now; it is because they lack good architectural skill and experience and instincts. Again, considering D upfront can actually simplify the A you build now.
The author ruins their entire argument with this one claim.
If "we picked this tool because we know it best" is a legitimate requirement, then every tool choice is justified as a "perfect" choice because it's what the architect was most comfortable with. If your emotions and current knowledge levels are considered reasonable justifications for a "perfect" solution, then all solutions are perfect solutions; they simply haven't had enough emotional justification yet. If all solutions are perfect solutions, then none are.
There are, ultimately, two kinds of software - those that need to ship by a deadline, and those that don't. A deadline forces you to eject dead weight that you don't need - requirements have a habit of getting clarified real fast when you need to build to a deadline. If you had time to over-engineer despite a deadline, you should consider working for a more productive organization. Meanwhile, the concept of "over-engineering" is a little vague for software that doesn't have a deadline. If you don't have a deadline, you don't have to compromise on quality. "Over-engineering" is then just a value judgement that you made poor use of your infinite timescale and built the wrong things with it. But who is making that judgement? Not the person who built it, not the person who funded it (i.e. usually self-funded as a hobby project), and not the person who uses it (since over-engineering is an implementation detail, rather than a product choice), so who cares?
edit: to clarify: "pick a stack you know already since we don't have time to learn a new one" is a totally valid requirement. But I disagree that it means that you built a "perfect" solution with it. I also disagree that you usually need to build perfect software - getting comfortable with adequate is how most people ship most software.
So, yes, if you have a purpose in mind, you can build something that can be used for that purpose, more or less effectively. The more general the purpose, the less specific the implementation; the more specific the purpose, the less general the implementation. Over-engineering here is something that indeed goes beyond the purpose of something, with feature creep as an example of moving goalposts and a slippery grasp of the limit.
However, production is only one layer of concern in engineering. You can talk about ideal artifacts whose forms perfectly enable the realization of some purpose, but there is the practicality of constructing one that must also be considered. Think of diminishing returns. Does it makes sense to invest a certain amount of time and resources into producing a more perfect artifact? If not, then pursing such perfection is itself an imperfection on the part of human judgement. That is, a more perfect engineer might know the limit of perfection for an idealized artifact, but he also knows what the perfect practical limit of justifiable effort is. Perfectionism is a sign of an imperfect engineer.
Returning to feature creep, the author might argue that the moving goalposts are a sign that the purpose is being refined, but feature creep can also mean that the shifting purpose is detached from real needs. So, in that case, the purpose served is no longer the purpose that ought to be served, and in that sense, while one may realize an end perfectly in a relative sense, the end pursued is itself the result of imperfect understanding, which is effectively what the author is gesturing at when discussing requirements.