I'm wondering about this now I guess it is old fashioned now to use environment variables and bare EC2 servers, managing your own APIs and websockets/DB on same server as opposed to breaking everything out. You need to use cloud formation and "oh did you know there is an AWS service for that?" Then you are using 5 services instead of 2. This twelve factor app concept. Don't know when is the right time to do this/at what scale.
my two cents : until you hit scaling wall (and when you will congratulations, you are either successfull or a video streaming platform), a big server to upgrade is the best way to go and focus on building product.
Then you hit performance problems, any sysadmin could help you handle 2x/4x/10x scaling with simple separations of service and maybe some hours of downtime. In the meantime you probably have weeks/months to think about going really crazy with your infrastructure.
What kind of numbers am I looking for? Thousands of users? Ops/sec? I realize it is partly due to what your thing is doing eg. website vs. a complicated app.
Yeah it is a good problem to have assuming you have cash flow.
When your service can no longer process requests as fast as they come in, you've hit a wall. Until then the simple solution is to allocate more resources to your service (i.e. scale vertically).
There's obviously a balance to be struck but the more often you do something the easier it is to do. If you have a simple app with a db, backend, frontend and proxy, vertically scaling every time you hit that wall is going to be very painful. A little complexity goes a long way - using a managed system adds a very small amount of complexity in exchange for some breathing room when you need it. The last thing you want when your service is in a death spiral is to start thinking about the practicality of migrating your db and taking the hours of downtime/working overnight/weekends to do it at a convenient time for your customers.
Of course it may vary a lot but i would say a hundred of thousands of users (100 000).
To explain the number, I assume a user will spend 1% of it's time on your app (that mean the average user spend around 15min per day (EVERY DAY) on your app, that may not sound impressive but it aldready is) and an nginx server on a powerfull machine could handle around 1k connection at the same time.
If you want to dig about an exemple of number far less abstract I remember stack overflow published their settings :
My last startup was spending stupid money on AWS services and horizontal scaling for an app which had, when I left, about 100 users and maybe 2 concurrent at a given time. And they had been doing this since before I joined when the numbers were much lower. The complexity was idiotic and no one but the devops guy who set it up could grok it. We still managed to have frequent downtime
Complexity kills your product - Overengineering is just one instance of complexity - Technical Debt like having state and data all over the place is another one - I quit my last job and I happily blame this article for convincing me to quit: https://itnext.io/the-origin-of-complexity-8ecb39130fc - coordination causes complexity and this killed me - we had everything not once but twice or more in different places - just one example - it's much more complex but me and my colleagues were grinded between an old codebase that nobody understood anymore and kubernetes, ci, etc.pp on the other side because management sold this without understanding that you need a process and time for digesting and applying the concepts in the team.
It depends, there’s a trade off. At some point things can become very complicated if you don’t abstract them and put them in their own lib or framework.
The less you use external dependencies, more time for you to focus on your own product instead of solving problems updating those dependencies and replacing them.
This sounds exactly like the last place where I worked. I quit in anger Not because I don't like Kubernetes but because management sold it as the fixall solution.
I interpret “early over engineering” not as adding complexity necessarily. It could also be about decreasing complexity but by taking too much time. Early on you’re supposed to rush to your MVP and add tech debt, not spend too much time to design something pretty and modular. Later on once you know your company can survive multiple quarters, then you can spend more time refactoring.
This is like saying that water is lethal - both are true in the sense that some x (water, complexity) is ok, but too much is bad.
Complexity is an unfortunate (but necessary) side-effect of (1) adding features and (2) optimizing for performance, both of which are critical for building a product.
If you try to remove all complexity from your product, you won't have a product. Instead, you have to try to minimize complexity while still delivering the same set of features and level of performance.
The basic point is that the default tendency in many environments is to run toward ever-increasing levels complexity -- often without even being aware that this is happening -- rather than be (as I believe one should be) by default always skeptical of added complexity.
If you try to remove all complexity from your product,
This wasn't what the commenter was advocating, of course.
Rather it was an attempt to simplify the basic message of: "Complexity has attendant risks which are too easily ignored -- and which in short enough order, can kill hour product."
basically I just looked for an excuse to post the link to the article - which is really a great article and describes the problems inherent to software in an - for me at that time - new and fascinating way.
Of course in practice it's always messy and rightly so, but I've learned the hard way that you better don't loose control about your state and data - because that's where the ugly bugs are.
The article is very good. Just ignore my drivel :)
The most important thing is to find the right balance. This article goes into one direction. But to be honest, most of the time I see it shift into the other direction: in product driven orgs the drive to implement features and bring them to market quickly is more important than engineering quality. But in the end you end up with something where implementing new features takes so much time because of the complexity that built up because you started to implement a product where the specs where unknown in the beginning and only materialized later. That's the point when you should re-engineer your system. But yeah... it's all about balancing and finding the sweet spot.
To put more succinctly, the least code wins. What we're talking about 2 opposite problems that cause the same end problem, too much code.
Overengineered what-if everywhere often causes way too complex systems where even simple things become hard, de-engineering is nigh impossible because it'd break things by taking away functionality (unless an alternative manifests itself).
Non-engineering hackjobs where we basically have N copy-pastes or more or less the same functionality everywhere, cleaning up anything will require a fair bit of testing to avoid breaking anything in the process, possible but causes a lot of uncertainty and risky if you won't see it unless failing in prod (because it goes hand in hand with bad deployment practices).
As mentioned in the article and above already, you need people with experience and to give those the business knowledge to find the correct balance.
Another problem is that these overly complex systems are often very fragile when something changes. In a very theoretical situation there could be a case where someone with an overengineered client consumes your JSON API and their client breaks when you add a field to a certain service response. Something that should have been no problem suddenly causes total breakage of your software and then you'll have to alter that very complex piece of software. Something that could've been prevented if you kept things simple and robust and simple chose to ignore extra fields or headers you weren't using anyway.
That's indeed the irony. The over-engineering happened in an attempt to be resilient to future change, but the outcome can often be the opposite. We've all been there, I think?
I've had this exact ticket in the past: A customer built a complex Java client for our public API which assumed no surplus fields in the responses, and as we added a new field, their entire application broke down, causing huge losses for the customer. I wasted so much time on explaining how our stability promise does not extend to added fields!
I do not understand how is this over-engineering. Over-engineering means that you worry too much for future/edge cases that might or might not come and you want them covered too early. The fields situation looks like bad engineering unless the reason that the "no surplus field" has an explanation which falls under "covering future needs" that escapes me at the moment.
If you find yourself over-engineering due to stuff you think you should be doing, things that ‘real developers’ do, understand you are vulnerable to being pimped. It’s no different than anyone else handing over their common sense and self worth in pursuit of an abstract form of validation (so abstract that you can internalize the validation cycle even in the absence of a physical superior).
If you find yourself defending your bad over-engineering and those that sold you the lifestyle, understand you are fully pimped and are now defending the pimp, even to your own detriment.
Happens to all of us from time to time, snap out of it. Don’t get pimped by ‘faangs’ or ‘notable person of interest’. Don’t listen to everything I said either, lest you want to get pimped by me.
Unless you can wholeheartedly make an objective argument for why you used a pattern, a tech stack, a process, in plain simple words, sans ‘that’s how the big boys do it’, sans ‘this makes me a real developer’, you are simply pimped and spewing out pimped out thoughts of your overlord pimps. Be ready to be wrong and backtrack and own the mistake and correct course, but don’t you dare hold on to it, or I’ll probe to see who is pimping you out.
I was never more free from my overlord developer pimps until the day I realized they evangelized impractical solutions. Never hoe’ing for anyone ever again.
It's also important to try stuff out, fail, recover, and try again. That "Code Complexity vs. Experience" graph in the article is not completely a joke. Very few people can tunnel through the complexity hump without years of failures and successes behind them. Moreover, you might not have a choice.
You might find yourself dropped into an obstacle course of complexity that other people created and that you have to keep running following their arcane patterns and practices-- while at the same time implementing new features and refactoring it into something workable before it becomes completely intractable. I think almost everyone faces this problem (except for maybe the most orderly and elite workplaces?).
My message was mostly for those that hide behind mistakes via an appeal to authority. Trying new things is a risk, which is fine, but to own the the success of the risk means you must also pay the collateral of owning the failure. The message is for those who don’t put up the collateral and hide behind ‘this is what everyone(the pros /sarcasm) is doing’ or ‘this is how it’s done’, and never reflect objectively.
No one would argue against trying things, it’s where all creativity and innovation comes from. I argue against dysfunction, the whole ‘the ship is not sinking’, when in fact it is.
Anyway, perhaps I’m speaking too personally, because I am on a literal Titanic right now, so apologies for that.
I've written a lot of complex stuff. In fact, I'm doing it right now.
There needs to be a "sweet spot," where we have enough complexity to achieve goals (which can be business and cultural, as well as technical), and as much simplicity as possible.
A lot of folks think that using dependencies makes the project "simpler."
It does, for the final implementation team, but that complexity still lives there; along with a whole bunch of cruft, tech debt, potential legal issues, security issues, etc.
Unfortunately, you can't do complex stuff, simply. Some level of complexity is always required. Managing that complexity is where we get to play adults.
Complexity is also one of the best moats in the business, if you have a "gross" problem with many edge cases there's few people who will want to eat your lunch, and most next generation up and comers often tout their simplicity compared to your old complex product because they certainly cant argue feature parity ;)
Right, using too dependencies don't make things simpler. They make them easier as long as the dependencies can do exactly what you need. However sometimes you come to a point where the dependency can't do something that's needed (often by the time I realized a third party lib can't do sth, I could have implemented it by myself without the dependency) or where you've accumulated so many dependencies that they conflict with each other or where connecting two different libraries is the really hard part. No silver bullets.
Figuring out the sweet spot in architectural design seems to me still an art/craft, that comes from intuition and by experience.
[As far as dependencies... using dependencies may sometimes be simpler than the alternative(s), and sometimes not, sure. It depends on the nature of the dependencies, the problem, and the alternatives. :) ]
> It does, for the final implementation team, but that complexity still lives there; along with a whole bunch of cruft, tech debt, potential legal issues, security issues, etc.
I call this squeezing the balloon, the complexity doesn't go anywhere, it's inherent to the requirements of the system. You just put it somewhere else.
This is just code factoring at a macro(-ish) level.
For well maintained deps there is the extra boon that it takes work off your hands though. For instance building a React app with Next.js saves you from ever having to deal with webpack and you get big upgrades for free.
> For well maintained deps there is the extra boon that it takes work off your hands though.
Absolutely. Nowadays, it's pretty much impossible to write anything without some level of dependency; even if just the development toolchain and standard libraries.
The problem is that a lot of outfits and people are releasing subpar dependencies that smell like the kinds of high-quality deps we are used to, but, under the hood, are ... not so good ...
Nowadays, it's fairly easy to write up a Web site with lots of eye candy, and gladhand a bunch of corporations, enough to get their brand onto your Step and Repeat Page, so it looks like your dependency is "big league," when it is not. In fact, the kinds of people that couldn't design a complex system to save their lives, are exactly the ones that are experts at putting up a wonderful façade.
What gets my goat, are proprietary SDKs, often released by peripheral companies. These can be of abominable quality, but are also the only way to access their products, so you need to load a bloated, sluggish, memory-leaking, thread-safety-is-optional, crashes-are-so-much-fun library into your app, and hand that library the keys to your sandbox, just so you can get at the device.
I've been writing exactly that kind of SDK for years, and know that we can do better. I'm a big proponent of open SDKs, but many peripheral manufacturers are absolutely dead-set against that.
These SDKs often mask significant complexity. That's what they are supposed to do. They also generally translate from the workflow of the device, to one that better suits application developers. Some SDKs are just a simple "glue" layer, that directly map the low-level device communication API to the software. That can be good, sometimes, but is usually not so good.
On the other hand, simplifying an already over-engineered product is almost next to impossible simply because many jobs depend on it. Maybe I'm cynical but I'm beginning to think that software complexity grows until it justifies all the head counts in the department.
This is the same phenomenon as the Peter Principle, where people Rose to the level of their incompetence. It sounds like a joke at first, but, of course they do. People get promoted when they excel. When you are in over your head, you stop getting promoted. Of course, over time, people grow into their roles, and regain their competence.
Devs will build software until they can no longer do so because the codebase is larger than their collective abilities to manage.
> Devs will build software until they can no longer do so because the codebase is larger than their collective abilities to manage.
I'm developing a principle of radical simplicity to attempt to combat this: always keep things absolutely as simple as they can be.
It is always easy to add complexity later, never to remove it. So the only conscientious choice you can make is to keep things as simple as possible while satisfying the requirements.
You should also critically evaluate requirements that introduce complexity.
Also, I don't think people mention this enough here: complexity == bugs.
Ability to keep things "simple" (basically produce code / architecture that is easy to understand) when solving problems in business domains (which usually are very complex in real live) is an art that few can successfully practice.
This is both interesting and disheartening - the evolution would concern itself more with the software organization then the product (hence the product death).
Yeah it is like this. Even simple sites require a ton of engs thrown at it. On the business side I guess they figure spending more money on tech will make them safe, but it's like of you wanted a bowl of soup and hired ten chefs to do it. But I can't complain since I am one of the soup chefs, and I help make sure we have tests that prevent the soup from becoming too salty
Making overly complex products developed in an absence of engineering input is also a major issue, at my last employer this was endemic to everything product threw at us. If you have no concept of how difficult something is to build you have no reason to not require it; by the time it gets to engineering its either too late to change or results in endless CRs to fix.
At first I thought that the article will be about putting so many features, that “in the end it will be able to send e-mail”. Because this is what is killing products - adding, and adding and redoing design as if users really were begging for it. And then comes new product that is leaner and more simple and takes over the old product and cycle starts from the beginning. I think that in capitalism it is not possible to not overengineer a product.
The insidious thing about over engineering is that it's usually committed by very experienced engineers. Experienced engineers rarely under-engineer, that tends to be fixed very early in one's career.
As we get more competent and read more books, we have the tendency to get enamored by new fancy abstractions. We get too clever and then we get in our own way.
Best real-world example: I inherited a project that was an giant "microservice" with 50-ish endpoints, and a Mongo database and dozens of collections. After probably 3 months of wrestling with this thing I had a realization: "this whole thing can be just a simple command line tool and one collection in Mongo". That reduced the code by almost half and it became so much easier to work with. It's frustrating that it could have just started this way.
OP suggests there's a hump in the curve of experience vs over-engineering; more experience correlates to over-engineering until it gets to a certain point, at which even more experience leads to less over-engineering.
Is it true? I don't know, I think it matches my... experience.
Part of it is that the "experience" needed to avoid over-engineering is helped if it's not just engineering experience, but domain experience too. If someone is constantly jumping industries/domains, it might take longer to get there. I think they still would eventually.
My own personal learning curve matches that graph almost perfectly. I'm probably not as close to the right side as I'd like to think I am, but I'm always trying to move in that direction.
Often I'll tend to over-engineer the initial solution, but as I think about all the complex edge cases that needs to be handled etc I often find myself asking "does the customer really need all this".
After discussing it with the customer, a simpler solution often emerges, where they change the requirements slightly allowing for a much simpler solution that might even solve the actual needs better.
While it's not just down to experience, I'd say it has a strong influence on being able to see beyond the given requirements towards a better outcome.
quite a bit. under engineered as in 'never really tested, is known in general not to work'. usually, yes in theory it should be easy to fix - but since the parent organization doesn't value getting a handle on what's really going on, it never does.
Over the last years I've been working with an audio library that makes several simplifying assumptions, like
- Audio interfaces are never added or removed over the app's lifetime
- Audio is never rerouted between interfaces, or at least the app doesn't need to know about it
- The details of an audio interface, like number of channels or supported sample rates, do not change while the interface is running
- Audio latency never changes and callback timing is always precise, so timestamps are not really needed and a single number is enough
Of course, none of these are really true and adding support for some of them would require rewriting a few APIs and many, many implementations. On the overengineering side, of course it has its own smart pointers, string class, thread implementation etc just in case someone needs to build for a target that doesn't support C++11.
In the world of hardware, which I realize is a parallel universe to the one in this thread where "product" apparently means "app", the answers are "all the time" and "no", respectively.
If someone didn't take the time to do actual "engineering" - which is to say, using mathematics to formalize design requirements (again, possibly a foreign concept to the app-building class) and just YOLOed the design based on intuition, you can end up with something so fundamentally broken in concept that "fixing" it requires a bottom-up redesign starting from fundamentals. A child's drawing of an airplane is not progress towards a blueprint of one!
I used to work for a company which the founder (and original engineer) was both very curious about how compilers worked and paranoid about having the company's code base stolen. That came to fruition with the following idea: "if I develop my own language and compiler, even if someone steals part of the code they still won't be able to run without the language specification and compiler!".
He designed his own programming language, compiler and, why not, the database and tools as well. It was a mess. It got all the limitations of a poorly projected side project and no benefit in the long run besides from huge technical debt.
15 years later, with dozens of clients' sites with the software installed and running locally, the company was locked in a horrible software stack that was next to impossible to move away from by gradually replacing modules, because the programming language had zero interoperability, the original creator was retired and no one besides himself had worked on it for years; while continue giving support to existing customers, because there were fires everyday, and with the same engineering team headcount.
That is one of the top 5 "oh my" in my career. I'm glad I left that behind.
Clojure's Rich Hickey? No. I'm not too familiar with him or Clojure, but it seems he put a lot more thought on it than the co-founder in my story could ever dream of.
I think there's a very specific form of overengineering afflicting products currently, which I'd classify as "endless revisiting". This is where companies build something which works well enough, but then get trapped in a cycle of endlessly tweaking that one thing. Inevitably the amount of code churn in this one area combined with the need for some semblance of backward compatibility results in something that is fragile, complex and slow.
Plus the annoyance as a user that whenever you use this thing, all your tools are in a slightly different place and work slightly differently to how you left them. IMO there's a need for better balance between "it works well enough, leave it alone" and "we haven't fully optimised this workflow yet" in product development.
Yeah, I see this happening a lot as well. A prime example is Spotify. Their app is done. It has all the features it needs. Instead of just focusing on making those features work more reliably, they seem intent on doing a big UI redesign every few months, adding bloat to the app and making it more frustrating to use. It’s really rare to find companies that just build a thing and stop adding features once the core functionality is done.
From a UI/usability perspective, one of the dangers with overengineering is the "wall of options" issue, where all users - even the majority of them that just need something simple - still need to read and understand all these advanced options. As a product manager and UI designer you have a couple ways to deal with this. You can choose an opinionated subset of features that make sense for a given niche and target only that demographic. You can go the corporate way, keep the wall of options and just require training for users. Or you can try the balancing act - choose a sane subset of features as the default, and hide the more advanced options, so they don't bother normal users, but still have them as possible options. There are many important choices regarding how much to hide, and where, and how to make it discoverable, and how to make it possible to gradually dig deeper, and for users to self-identify as someone that needs to dig deeper - and those details are often as much art as science. But get it right, and you've got one of those rare killer apps that both newbies and experienced users enjoy.
I must not be the target user. I don't want to tinker with gradients, I just want a button with some text in it. Yet, I see options for the former, but I had to carefully align two different elements for the latter.
I'm sure this can all be solved with modeling out your design system or whatever, but as a product manager who needed to make a quick mockup, I found it a little overwhelming.
Six months ago I left a company that was working on an overengineered product. Even worse than it being overengineered was that it was also under documented. Working on anything was a pain, because the CTO wanted everything to follow his well thought out, and frankly very cleverly engineered design patterns, but he couldn't clearly communicate what those patterns were. And the entire company amounted to transforming and cleaning data sets using in-house tools, which could easily be done with existing tools too. Both myself and the other senior engineer on the team left at the same time. I felt bad leaving them, because they were trying to grow and had a ton of funding and deals with FANG companies but they were struggling to find engineers that the CTO thought were smart enough. I didn't want to burn bridges, so I didn't end up telling them that the problem wasn't a lack of qualified engineers, it was an over-engineering CTO who struggled to communicate.
Being ins a similar role myself, how do I ensure that engineers stay happy working on the project that we're working on? I'm finding myself actually doing the opposite of the CTO you mentioned and pushing them towards adopting more off-the-shelf components instead of maintaining homegrown stuff but I think I'm causing a degree of upheaval by doing this. Their justifications for push back however, often smell of sunken cost fallacy to me.
Off the shelf software isn't an instant win. You're signing up for lock-in, a set way of doing something, a boundary you can NEVER cross, and domain and language you will never be able to change. This language will leak into your software and may not be a good fit for the end user you're trying to serve.
That's a lot of trade offs to avoid maintaining the subset of features you need from said software in house, being able to leverage internal knowledge, being able to streamline all your environments and tooling to suit your needs instead of catering to the needs of said software.
This isn't an argument one way or another, it's just pointing out that there are trade offs you need to make consciously or otherwise.
In your position I would be extremely concerned about pushing off the shelf software onto Devs that either lack the clout to be comfortable pushing back or lack the communication skills to clearly articulate all the trade offs really taking place.
It's very easy for a boss figure to push through requirements with off the cuff pointed questions. The reports often need to push back with orders of magnitude more research and thoughtfulness than went into the question or suggestion.
I think the best thing you can do as a CTO is define the problems/goals and desired outcomes very clearly. Think of possible solutions if you can but don't share them, and give your team the problem to solve. And let them stumble a bit, because in the level of buy in and growth you'll get is more than worth it - it'll be their solution after all.
I'd second this. Your role as a CTO is to define technical strategy and help align your team with that strategy, not to tell your team how to do their jobs. They know how to do their jobs. That's why you hired them. Share your strategy and goals with the team, and trust their judgement on the specific decisions that help you get there. There's nothing more annoying than being an engineer whose technical decisions aren't trusted. If you have a quality engineering team, most of your decisions as CTO should be about strategic direction, not specific technical choices. Specific technical choices can be entrusted to your team as long as they're in line with the strategy.
Literally the only two things leadership should do to be above the mean:
1. Clearly define a vision for the future/ goal(s) that should be achieved
2. Get out of the fucking way of your minions, and trust that you hired correctly, to let them figure out how to get to the finish line
Bonus Points:
3. Your temperament is in the Goldilocks' zone of neither being too much of a spendthrift, nor too much of a miser, when setting budgets, i.e. you're not some rando without P&L experience that was tied to your bonuses.
As someone who is miserable on a similar project, the best thing you can do is give ample breathing room for the labor. The codebase is stress inducing, so being empathetic on deadlines and and what may appear to be shifting attitudes (good days, bad days) means a lot. Give off days early, try to make sure people pace themselves.
Otherwise they’ll burnout and the resentment of the state of the codebase will be jet fuel to that brand new revolving door on your team.
Commiserate to some degree, ‘look, I know this shit sucks’. If they don’t know you empathize, they won’t trust you.
It’s a real problem, if you’ve hired ‘architects’ whose skill and passion is creating new systems, rather than connecting and modifying existing tech. Video games went through this shift about 20 years ago, and it was painful. Many architects ended up moving to the game engine companies, or to other tech areas than games.
Sorry to point this out but I find it problematic that neither you nor the engineers are making decisions out of experience..
My recommendation is that you hire someone with a TON of experience that makes these decisions not based on Medium post he read last week..
I am a developer and I always try to re-use a solution (preferably open source) instead of rolling my own (I'm happy to do the latter in case the need/situation demands but those cases are not that frequent). I also happen to enjoy devops (specifically CI/CD stuff and enjoy the integration aspects of connecting things to achieve a goal and see the entire pipeline producing the end-result). So I guess you'll have to find people that have the integrator mindset instead of those with NIH syndrome. At the same time, you do need some talented developers as well in case you do need to roll your own. It is a balance like most things.
> and pushing them towards adopting more off-the-shelf components instead of maintaining homegrown stuff
Are you spending your time second-guessing the developers and micro-managing them on the exact thing they are supposed to be experts?
Your phrasing has that "feel", but it's far from a sure conclusion. Anyway, if it is the case, you may need to reevaluate both your hiring (should you get more senior people?) and management (this is a clear "should never be done", unless your people are explicitly under training) practices.
Same been in that role and in a similar role at the moment. It's hard but this is the industry I can't change that anymore I just have to work with these people long enough so I can retire.
> they were struggling to find engineers that the CTO thought were smart enough.
If you have to find the smartest people to keep the wheels on, you’ve already lost.
Disdain for the bell curve is the fastest way to get overengineering. Very few things have to be rocket science to create a good company. Everything else should be dead simple so that anyone can work on it. That also means you have to compartmentalize the hard bits so they don’t leak across the entire codebase.
But some people get bored with mundane code and will make things interesting by replacing a boring task with an exciting one. It’s part of the Curse of Knowledge. You can’t run a company like an action movie. Everyone gets burnt out.
This overengineering happened on my team recently. I built a REST backend and the front end team was tasked with building 15 admin CRUD interfaces in react which would talk with the backend. A junior front end guy convinced the senior front end guy to build an entirely new dynamic CRUD layout system. I saw what they were doing and knew it wouldnt work out. People got upset at me for raising concerns so I kept my mouth shut. 6 months later the project was not even 25% complete, the senior dev left for another company, and the junior dev got shuffled to another job in the company.
The biggest consequence for me was not mentioned: that your carefully planned design very soon becomes an obstacle to something a user actually wants done, at which point you say "we can't do that". Which is one of the worst things you can do. In this sense almost all of the systems I interact with, modify etc. are over engineered.
The "super simple code" at both ends of the graph aren't equivalent. The latter is more "simplest code possible".
Overly simple, hacky, narrowly spec'd code produces the same tech debt as overengineering. Anybody who's worked in a move-fast-break-things type startup will know how much engineering resources are wasted on rewrites/bugfixes due to this.
Ultimately, as with many things in life, you need to find the right balance.
I think the graph needs to continue a few more years, but the author hasn't lived that yet. The "simplify" mindset is also something you can take too far without experience.
Interestingly, dang (HN mod) once mentioned in a feature request comment that he has a mental model of a complexity budget. This very closely fits my own perception: You can only add so much complexity to a system before it starts to break down under the load. Estimating that budget, and how much new features will consume, makes a good engineer in my opinion.
As others have said, it is unecessary complexity. Over-engineering is an ambiguous term.
For example, premature optimisation is frequently mentioned as over-engineering but is it premature optimisation to use a map/dictionary instead of an array for key-based access? Nope. That is just correct. Is it over-engineering to know that if your product succeeds, you could end up with X-hundred objects which will use up all the RAM you are storing them in and therefore you might want to make them smaller/store them off-board? Of course, you can come back and refactor later but it is so much easier for the person who writes that code to understand what can be done on day 1 rather than assuming it is cheaper to refactor later only when needed.
I think a better take is for people to accept that no app lasts forever. If we build that assumption into our worldview, it will help us make better decisions. I still think some engineers think there is some perfection that means the app will live forever.
> use a map/dictionary instead of an array for key-based access? Nope. That is just correct.
Maybe, maybe not. While the map/dictionary is easier to use, if the number of items is small (which it often is) the array will be substantially faster because the CPU will pre-fetch the next element into the cache while you are still checking if the current one is the right key. Maybe - check with a cache aware profiler, and you need to design your array to be cache friendly.
Of course the above assumes your map/dictionary is on the hot path (seems unlikely), your code is too slow (probably subjectively, but in some real time applications this can be objectively measured), and you are writing in an efficient language. IF All of the above are true, only then should you go to the complexity of using an array to do the job of a map/dictionary.
Also depends on expected changes to the no. of items. If you think it can shoot up, then you are better served by sticking to map/dictionary (and paying the minor penalty, assuming not in hot path) compared to starting with array for low-volume and then having to make code changes to map/dictionary to handle increased volume.
This is a true story about an entrepreneur who was suffering a bad case of perfectionism. Instead of showing his MVP to potential customers, he just kept investing in the software, chasing an almost impossible ideal of high availability
Overengineering is a non technical problem but of course most tech companies only interview devs on their technical skills.
I've found is that once a dev has an idea of how something is suppose to work it's hard to get them to think in a different way about that thing. So when that thing doesn't work out for some reason they just keep adding exceptions and adding exceptions until you have the horribly over-engineered solution.
I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written.
Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet after only writing one or two integrations on the new platform. Nobody over-engineered here. Everything was built to spec. But three years down the line the new dev is going to get assigned a bug in one of the integrations and lament how the code was over-engineered. Hindsight is 20/20.
Lots of stories from the trenches including many in this thread make this mistake. The same goes for 'tech debt'. If you weren't there, you don't know.
With the benefit of hindsight, I think you can conclude that the system was over-engineered. If you’re trying to answer only “was it over-engineered?” I think you can answer it with hindsight.
If you’re trying to answer “did the people who built it exhibit poor judgment resulting in the system being over-engineered?” then you need to know more than just the after-the-fact outcome.
My new colleague used to wonder about certain things i.e. why is this is done this way, it doesn't make any sense. As I had been there a lot longer, I would share the technical and non-technical background/restrictions we operated with. Eventually when another new colleague joined, he told the guy, "I used to wonder why some parts of the code are setup that way; now that I have the background I can say that if you are wondering about those same things, trust me, there is a reason/background - its not because the people who did it that way were stupid".
Occasionally, it is because of that though. If not 'stupid', at least inexperienced. I've had plenty of things I've done that worked, but were, in hindsight, 'stupid' (and have been called out on that). Sometimes, people try to make a lot of post-hoc justifications for a block of code or a data/tech decision that really is, just... 'stupid'. Again, that's more likely down to inexperience than anything else, but not every decision is a 'good' one just because people 'had their reasons'.
Examples
Having a 'user info' table with 190+ boolean columns for 'country' ('US','CA','DE','IT', etc) in case someone wants to indicate they're 'from' two countries.
Joining views on views of joined views which are themselves built on joins of other views is, likely, not a terribly sound data decision. (X: "It worked fine last year - you must have broken something." Me: "well... last year you had 450 users, and now there are 37000 users - this isn't a performant way of doing this". X: "Can't be true - the person who wrote it is a DBA")
> Occasionally, it is because of that though. If not 'stupid', at least inexperienced. I've had plenty of things I've done that worked, but were, in hindsight, 'stupid' (and have been called out on that). Sometimes, people try to make a lot of post-hoc justifications for a block of code or a data/tech decision that really is, just... 'stupid'. Again, that's more likely down to inexperience than anything else, but not every decision is a 'good' one just because people 'had their reasons'.
I agree on the point regarding the decision being 'stupid' but in this case it was not because of inexperience. On the contrary, it was because an overruling decision by an experienced manager. So the point is that it is not always technical or due to inexperience (though that does happen) - from what I've seen, it is quite common for such things to happen due to hierarchical/ego/political issues as well.
you're not wrong, but there's a whole class of problems that occur outside 'enterprise' structures. Often it's just a lone cowboy building something for a small business, and the business owner have absolutely no way to determine if what's being delivered is 'good' in any meaningful sense.
It's a question of framing. Nothing is ever over- or under-engineered. It's over- or under-engineered for a purpose.
What that distinction recognizes is that it's possible for something to have been well-engineered at one time, while still being over-engineered today. Header files in C and C++ are an example of this phenomenon. They solved a very real problem with technical constraints from 40, 50 years ago, both in terms of computers' capabilities and compiler technology, but, since those problems don't exist anymore, they function as over-engineering.
Agreed, but "nothing is ever over- or under-engineered" is a bit too bold. There is definitely over engineering just for the sake of over engineering, and under engineering caused by incompetency.
This is why commit messages are so important. Documentation that is external to the code its referencing will always drift but a commit message is a snapshot in time that is attached to the thing it is referring to. In my experience not enough people take commit messages seriously and just basically give a summary of what changed not why it was changed.
Commit messages get lost too easily. It’s the worse way to document reasoning around the code. There’s many proposals to document code outside of code and none of them has ever worked. Just comment your code.
I think the lesson here is that with great product market fit everything is under engineered, with poor product market fit everything is over engineered. Statistically you are more likely to be building an over engineered product with poor product market fit than you are to be building an under engineered product with great market fit.
Now is over or under engineering a bad thing? that depends on how many resources the company can muster, your customer tolerance to bugs/failures, and whether you can build a small subset of features well. In practice I've observed products with fewer, but well built features tend to succeed more so than buggy featureful products. Large companies may opt to overbuild everything simply so that they can prove that the product was bad rather than having to decide whether it was product or execution that failed.
Technically speaking, for any given situation there's no way you can be wrong.
But this post describes a distinct and, by my experience, dominant industry phenomenon. And the stakes are as high is OP says (dead products).
Industry engineers in the large are massively divorced from product outcome. This should be no surprise, as it is a seller's market:
Delivering product simply and directly can be, depending on perspective, somewhat of a mundane and Sisyphean activity.
Engineering "stuff" is far more intellectually engaging, and there's no market pressure (for engineers) against over-engineering. If my company fails to deliver a product outcome, whether a viable company or startup, in this market there is no recourse at all on my salary let alone my hireability.
I say "depending on perspective" because if you can find a way to be more intellectually and emotionally stimulated by shipping stable product over time, then ime you _can_ unite your joie de vivre with market outcomes.
But that does not at all describe most engineers in industry who generally at best shed off some market value while spinning unnecessary webs of toy abstractions (ie liabilities).
I'm currently walking away from a code-base (and the company who let it be built) that was obviously built by someone who's mental model of how the code works was incorrect. At some very basic levels, it under-performs compared to literally every other example of this sort of code I've ever seen (and it's extremely common), which results in a subpar user-experience and an EC2 instance way bigger than necessary to run this sort of code. A lot of code running doesn't need to run in the first place, and due to the incorrect mental model, there's a substantial amount of code that doesn't run.
Nobody really understands how bad it is, though, because it's barely tested (and the engineer who wrote most of the code has argued that we don't need tests, for various reasons) and all of the engineers on the team who didn't build it initially, have told me that they're not familiar enough with how this sort of product should work to understand that it doesn't work like that (again, this is a super common kind of product).
There's a lot of other anti-patterns (I started keeping a list in my first week) but I think these are the most damning. This code is in production. Nobody at the org brought up these issues earlier in development (like saying "hey, nobody else's product does [annoying thing that doesn't need to happen], why does ours?"), which leads me to believe that the whole technical org is at fault.
It sucks and I can't wait to be done with this experience.
I don't disagree, but I think that the reason that this fits here is because the poorly engineered software is unnecessarily complex (not in the "someday we might need this" way, but in ways that I'm having a hard time articulating without giving away the product because I know my coworkers are on HN). In fact, it was the ridiculous complexity that drove me to discover the disparities between the original developer's mental-model and the working-model. It just so happens that I've worked on this sort of thing before (as have thousands of others) that I was pretty quickly able to understand where the original developer went wrong (trust me, I'm just an ordinary developer, not a 10x elite hacker or anything).
You see, overengineering is exclusively of the "some day we might need this" kind of complexity. Anything else is a different problem, very likely with very different causes, even if the consequences are similar, so it's not useful to analyze together.
Some of it makes sense when the reasons were explained (e.g. "we figured this would make it easier for [business unit] to do [thing] which turned out not to be something [business unit] actually has interest in doing").
Other examples are purely of this sort (e.g. "we'll be glad we have this in place when we finally hit [x] users or enter [market]").
Interesting angle. What if the business asked for it, the dev team delivered it, but the business subsequently gave up on it, and now it's overengineered in comparison to what remains in use. To avoid this scenario, I don't try to outthink the business team, but I do push people toward temporary manual workflows sometimes, and promise to automate later if the idea actually pans out.
Yes the business team will often ask for things they don't need but I can justify that cost. It can still impede other development but we tackle that when it becomes a problem not before.
I agree completely on pushing off features that are not core because it is better to learn in production than to daydream in meetings. I would say most phase-2 features never get implemented.
Typically over-engineered software looks something like the following.
1) The product is horizontally scalable to 3 orders of magnitude more traffic than the product will ever receive.
2) Bespoke assembler/hand crafted memory management/other do not touch code to shave 5ms off a 5 ms call on an API that isn't latency sensitive.
3) Ability to handle N customer tenants while only handling 1 customer 3 years later.
4) multi-region redundant deployment for 1 customer strictly based out of the US with no availability requirements.
5) 100% coverage for integration tests, unit tests, CI/CD, for a single tenant blog that is never updated and no one reads.
6) Custom internal framework with micro-service splits to support team of 100 engineers working on app when there are 2.
7) Automated continuously trained ML ranking model which performs .001% better than sorting by popularity or other trivial heuristic on a social media app with 10 users.
The common theme in many of these cases is that these are pretty good ideas if you have unlimited people, but have fairly low impact. In some cases these may be counter-productive to successful delivery at the company/product level. A piece of software built wrongly due to poor understanding of the product domain is often considered under-engineering.
Isn't the common theme: "this is what big tech does" ? Ie, it seems a lot of small companies/startups look at "best practices" and just copy them. It's not just engineering. It happens in product, hiring, marketing, everything.
It's completely wrong given the context 99% of companies are operating in. It's no wonder a Zuckerberg can come along and build a crazy successful company in his early 20s. He likely didn't know any "best practices". Common sense was enough.
Complexity kills projects and the will to work on them. It can be caused by either over or under-engineering. With the first you end up with extraneous layers of abstraction and badly applied design patterns that make it hard to understand or reason about the code...in the second you end up with a big ball of mud that's impossible to understand or tease apart.
In both cases the code gets harder and harder to change to the point that no one really knows how it works or wants to risk touching it.
Over engineering is a form of bad engineering, at least in my book. Fun starts when some parts of a product are over engineered while others are under engineered... One of the cases good averages get you nowhere...
I think nearly everyone would agree that "over == bad" in most settings, but the reverse is being implied by the GP - that bad engineering is a form of over engineering.
No tests? Nobody knows how it works? It doesn't perform well and doesn't fit the problem it's solving? That's very much not over engineered in my book.
I understand the sentiment but there are so many situations where this is not true.
All it takes is a good enough sales and marketing team, or the right executive relationships, especially if the team being “supported” is small.
You can say the company or division is presently functioning, but the time constant on the product is different, and the coupling is too loose to say anything else with accuracy.
> All it takes is a good enough sales and marketing team, or the right executive relationships, especially if the team being “supported” is small.
That's the point though. Getting caught up in engineering purity or even whether it's objectively _good_ is a waste. The product exists to drive business (financial) metrics. If it's doing that, it's working.
The one thing that I'd challenge about this line of thought is that the product isn't necessarily "done", there are expectations to add features & fix bugs.
If the product is poorly crafted, and especially if it's also poorly tested, adding features can introduce bugs, and fixing bugs in one place can cause issues elsewhere. Development then slows to handle these occurrences, which can then impact product viability.
I'm aware of technical debt and it's consequences. The point is that the vast majority of developers treat their job as creating good software, when it's absolutely not.
I'd say it's creating software that is "good enough" (good enough to do what it needs to, right now, sufficiently frequently that paying customers return).
You've checked-out, so it's too late, but depending on the circumstances, this can be a situation for an experienced engineer to step in. The biggest challenges are political, but also triaging the issues and coming up with a migration plan. "Rewrite all the things" rarely goes well, especially if they already work, just on beefier hardware than is needed.
> and the engineer who wrote most of the code has argued that we don't need tests, for various reasons
Sounds like you made the right choice to walk away. "My code doesn't need tests" smacks of match-over-the-shoulder type of chaos creation. This person has likely never had to maintain their own code or someone else's, nor has anyone ever had to maintain their code. 0/5 would not work with.
Tests are wonderful. Tests are proof your code works. They are something to be proud of. And they defend your code against the chaos monkey of future people refactoring and rearranging.
Specs are often over engineered. This is part of the problem. Too much effort in rowing the boat, not enough effort in figuring out the right direction to point it first.
In my experience, the situation you're describing more often results in "sloppy" code -- it seems convoluted, non-obvious, poorly thought out until you realize when the pivots happened. Overengineered code usually looks neat at first glance, maybe even impressive, especially to non-technical stakeholders. Then when you actually dive into it, you notice a bunch of wrong abstractions that introduce layers of indirection without offering any sort of valuable flexibility. "This metaclass that implements a context manager could've just been a slightly longer than average function" type stuff.
A codebase that pivoted multiple times makes more sense the more time you spend in it, an overengineered one makes less.
> I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written.
The specs and the decisions surrounding the code are usually part of the over-engineering.
> Everything was built to spec.
Even with reasonable specs, I've worked with, and later managed, engineers who had a strong drive toward over-engineering solutions. I've also done it myself, and I suspect most people reading this have done the same.
The problem is that many of us engineers became developers because we enjoy building things and we enjoy solving complex problems. It can be exciting to imagine a particularly complex version of a problem and how elegant and fun a complex solution would be.
It can be as simple as a programmer who tries to write their own CMS when the spec could be solved with a simple blogging platform. If you're an ambitious programmer, writing a CMS is fun! Installing Wordpress is blech.
I didn't truly understand the ubiquity of the problem until managing ambitious engineers who wanted to do complex things. At times it felt like 1/4 of my job was just asking "If we wanted to simplify this solution, what would that look like?" Repeat for a few rounds and eventually the team has shed a lot of unnecessary complexity without sacrificing the spec at all.
I don't think you can diagnose over-engineering after the fact. Unless you were in the room, or have access to written specs from a meticulously documented team, you don't understand the conditions under which the code was written.
Can you prove it in a court of law? No. But you still can often easily diagnose it. And you'll usually be correct.
For example the conditions under which https://wiki.c2.com/?SecondSystemEffect happens are well-known, and the pattern has been repeated often in the half-century since it was named. If it is a second system, and you can find complicating abstractions in it that get used exactly once in practice, then with better than 90% odds it was over-engineered. And the farther you dig, the stronger the evidence will become.
> Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet after only writing one or two integrations on the new platform. Nobody over-engineered here. Everything was built to spec.
I would argue this is a case of over-engineering. Though, it extends beyond pure engineering.
This is poor resource management and business de-risking. An effective approach here would have been: "Let's build out a handful of integrations. Make sure they're going to deliver the value we want. Then abstract it into a repeatable pattern".
I've been in exactly this position, in fact I conceived and created the situation at my last company. I found it uncanny how on point OP was.
The trouble with "Let's build out a handful of integrations. Make sure they're going to deliver the value we want. Then abstract it into a repeatable pattern" is that when you're building an integration marketplace, you need partners, and they don't want to be your research bitches.
They just want to build their integrations once, and gtfo. If they think you're signing then up for a revolving wheel of experimentation, they'll just politely stay away until someone else has done the hard yards.
Sure if you're a Microsoft or a Google you'll have any number of willing partners who will put up with anything to be the pioneers in your integrations marketplace.
But otherwise, they're using your integrations marketplace purely for the benefits, and they don't want to be building on sand.
Well how do you know something is overengineered?
It's when you know it will never (or too far in the future) need to scale accordingly to the engineering.
210 comments
[ 75.4 ms ] story [ 346 ms ] threadThen you hit performance problems, any sysadmin could help you handle 2x/4x/10x scaling with simple separations of service and maybe some hours of downtime. In the meantime you probably have weeks/months to think about going really crazy with your infrastructure.
What kind of numbers am I looking for? Thousands of users? Ops/sec? I realize it is partly due to what your thing is doing eg. website vs. a complicated app.
Yeah it is a good problem to have assuming you have cash flow.
To explain the number, I assume a user will spend 1% of it's time on your app (that mean the average user spend around 15min per day (EVERY DAY) on your app, that may not sound impressive but it aldready is) and an nginx server on a powerfull machine could handle around 1k connection at the same time.
If you want to dig about an exemple of number far less abstract I remember stack overflow published their settings :
- https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-...
- https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
The 2013 article state they could handle 148,084,883 page load per day with a single database server (but they did use two) for exemple.
until they change - again.
This is like saying that water is lethal - both are true in the sense that some x (water, complexity) is ok, but too much is bad.
Complexity is an unfortunate (but necessary) side-effect of (1) adding features and (2) optimizing for performance, both of which are critical for building a product.
If you try to remove all complexity from your product, you won't have a product. Instead, you have to try to minimize complexity while still delivering the same set of features and level of performance.
If you try to remove all complexity from your product,
This wasn't what the commenter was advocating, of course.
Rather it was an attempt to simplify the basic message of: "Complexity has attendant risks which are too easily ignored -- and which in short enough order, can kill hour product."
Of course in practice it's always messy and rightly so, but I've learned the hard way that you better don't loose control about your state and data - because that's where the ugly bugs are.
The article is very good. Just ignore my drivel :)
Overengineered what-if everywhere often causes way too complex systems where even simple things become hard, de-engineering is nigh impossible because it'd break things by taking away functionality (unless an alternative manifests itself).
Non-engineering hackjobs where we basically have N copy-pastes or more or less the same functionality everywhere, cleaning up anything will require a fair bit of testing to avoid breaking anything in the process, possible but causes a lot of uncertainty and risky if you won't see it unless failing in prod (because it goes hand in hand with bad deployment practices).
As mentioned in the article and above already, you need people with experience and to give those the business knowledge to find the correct balance.
I've had this exact ticket in the past: A customer built a complex Java client for our public API which assumed no surplus fields in the responses, and as we added a new field, their entire application broke down, causing huge losses for the customer. I wasted so much time on explaining how our stability promise does not extend to added fields!
If you find yourself defending your bad over-engineering and those that sold you the lifestyle, understand you are fully pimped and are now defending the pimp, even to your own detriment.
Happens to all of us from time to time, snap out of it. Don’t get pimped by ‘faangs’ or ‘notable person of interest’. Don’t listen to everything I said either, lest you want to get pimped by me.
Unless you can wholeheartedly make an objective argument for why you used a pattern, a tech stack, a process, in plain simple words, sans ‘that’s how the big boys do it’, sans ‘this makes me a real developer’, you are simply pimped and spewing out pimped out thoughts of your overlord pimps. Be ready to be wrong and backtrack and own the mistake and correct course, but don’t you dare hold on to it, or I’ll probe to see who is pimping you out.
I was never more free from my overlord developer pimps until the day I realized they evangelized impractical solutions. Never hoe’ing for anyone ever again.
It's also important to try stuff out, fail, recover, and try again. That "Code Complexity vs. Experience" graph in the article is not completely a joke. Very few people can tunnel through the complexity hump without years of failures and successes behind them. Moreover, you might not have a choice.
You might find yourself dropped into an obstacle course of complexity that other people created and that you have to keep running following their arcane patterns and practices-- while at the same time implementing new features and refactoring it into something workable before it becomes completely intractable. I think almost everyone faces this problem (except for maybe the most orderly and elite workplaces?).
No one would argue against trying things, it’s where all creativity and innovation comes from. I argue against dysfunction, the whole ‘the ship is not sinking’, when in fact it is.
Anyway, perhaps I’m speaking too personally, because I am on a literal Titanic right now, so apologies for that.
There needs to be a "sweet spot," where we have enough complexity to achieve goals (which can be business and cultural, as well as technical), and as much simplicity as possible.
A lot of folks think that using dependencies makes the project "simpler."
It does, for the final implementation team, but that complexity still lives there; along with a whole bunch of cruft, tech debt, potential legal issues, security issues, etc.
Unfortunately, you can't do complex stuff, simply. Some level of complexity is always required. Managing that complexity is where we get to play adults.
T.A.N.S.T.A.A.F.L.
Figuring out the sweet spot in architectural design seems to me still an art/craft, that comes from intuition and by experience.
[As far as dependencies... using dependencies may sometimes be simpler than the alternative(s), and sometimes not, sure. It depends on the nature of the dependencies, the problem, and the alternatives. :) ]
I call this squeezing the balloon, the complexity doesn't go anywhere, it's inherent to the requirements of the system. You just put it somewhere else.
This is just code factoring at a macro(-ish) level.
For well maintained deps there is the extra boon that it takes work off your hands though. For instance building a React app with Next.js saves you from ever having to deal with webpack and you get big upgrades for free.
Absolutely. Nowadays, it's pretty much impossible to write anything without some level of dependency; even if just the development toolchain and standard libraries.
The problem is that a lot of outfits and people are releasing subpar dependencies that smell like the kinds of high-quality deps we are used to, but, under the hood, are ... not so good ...
Nowadays, it's fairly easy to write up a Web site with lots of eye candy, and gladhand a bunch of corporations, enough to get their brand onto your Step and Repeat Page, so it looks like your dependency is "big league," when it is not. In fact, the kinds of people that couldn't design a complex system to save their lives, are exactly the ones that are experts at putting up a wonderful façade.
What gets my goat, are proprietary SDKs, often released by peripheral companies. These can be of abominable quality, but are also the only way to access their products, so you need to load a bloated, sluggish, memory-leaking, thread-safety-is-optional, crashes-are-so-much-fun library into your app, and hand that library the keys to your sandbox, just so you can get at the device.
I've been writing exactly that kind of SDK for years, and know that we can do better. I'm a big proponent of open SDKs, but many peripheral manufacturers are absolutely dead-set against that.
These SDKs often mask significant complexity. That's what they are supposed to do. They also generally translate from the workflow of the device, to one that better suits application developers. Some SDKs are just a simple "glue" layer, that directly map the low-level device communication API to the software. That can be good, sometimes, but is usually not so good.
https://www.monkeyuser.com/2018/implementation/
Devs will build software until they can no longer do so because the codebase is larger than their collective abilities to manage.
I'm developing a principle of radical simplicity to attempt to combat this: always keep things absolutely as simple as they can be.
It is always easy to add complexity later, never to remove it. So the only conscientious choice you can make is to keep things as simple as possible while satisfying the requirements.
You should also critically evaluate requirements that introduce complexity.
Also, I don't think people mention this enough here: complexity == bugs.
As we get more competent and read more books, we have the tendency to get enamored by new fancy abstractions. We get too clever and then we get in our own way.
Best real-world example: I inherited a project that was an giant "microservice" with 50-ish endpoints, and a Mongo database and dozens of collections. After probably 3 months of wrestling with this thing I had a realization: "this whole thing can be just a simple command line tool and one collection in Mongo". That reduced the code by almost half and it became so much easier to work with. It's frustrating that it could have just started this way.
Is it true? I don't know, I think it matches my... experience.
Part of it is that the "experience" needed to avoid over-engineering is helped if it's not just engineering experience, but domain experience too. If someone is constantly jumping industries/domains, it might take longer to get there. I think they still would eventually.
After discussing it with the customer, a simpler solution often emerges, where they change the requirements slightly allowing for a much simpler solution that might even solve the actual needs better.
While it's not just down to experience, I'd say it has a strong influence on being able to see beyond the given requirements towards a better outcome.
And aren’t those projects easier to fix…because they are so simple?
In my experience it can be just as difficult to fix, at least if constrained by backwards compatibility.
Under-abstracted code for example often introduce cross-dependencies which can be hard to break in a backwards compatible manner.
- Audio interfaces are never added or removed over the app's lifetime
- Audio is never rerouted between interfaces, or at least the app doesn't need to know about it
- The details of an audio interface, like number of channels or supported sample rates, do not change while the interface is running
- Audio latency never changes and callback timing is always precise, so timestamps are not really needed and a single number is enough
Of course, none of these are really true and adding support for some of them would require rewriting a few APIs and many, many implementations. On the overengineering side, of course it has its own smart pointers, string class, thread implementation etc just in case someone needs to build for a target that doesn't support C++11.
If someone didn't take the time to do actual "engineering" - which is to say, using mathematics to formalize design requirements (again, possibly a foreign concept to the app-building class) and just YOLOed the design based on intuition, you can end up with something so fundamentally broken in concept that "fixing" it requires a bottom-up redesign starting from fundamentals. A child's drawing of an airplane is not progress towards a blueprint of one!
He designed his own programming language, compiler and, why not, the database and tools as well. It was a mess. It got all the limitations of a poorly projected side project and no benefit in the long run besides from huge technical debt.
15 years later, with dozens of clients' sites with the software installed and running locally, the company was locked in a horrible software stack that was next to impossible to move away from by gradually replacing modules, because the programming language had zero interoperability, the original creator was retired and no one besides himself had worked on it for years; while continue giving support to existing customers, because there were fires everyday, and with the same engineering team headcount.
That is one of the top 5 "oh my" in my career. I'm glad I left that behind.
Plus the annoyance as a user that whenever you use this thing, all your tools are in a slightly different place and work slightly differently to how you left them. IMO there's a need for better balance between "it works well enough, leave it alone" and "we haven't fully optimised this workflow yet" in product development.
The list goes on.
I'm sure this can all be solved with modeling out your design system or whatever, but as a product manager who needed to make a quick mockup, I found it a little overwhelming.
That's a lot of trade offs to avoid maintaining the subset of features you need from said software in house, being able to leverage internal knowledge, being able to streamline all your environments and tooling to suit your needs instead of catering to the needs of said software.
This isn't an argument one way or another, it's just pointing out that there are trade offs you need to make consciously or otherwise.
In your position I would be extremely concerned about pushing off the shelf software onto Devs that either lack the clout to be comfortable pushing back or lack the communication skills to clearly articulate all the trade offs really taking place.
It's very easy for a boss figure to push through requirements with off the cuff pointed questions. The reports often need to push back with orders of magnitude more research and thoughtfulness than went into the question or suggestion.
1. Clearly define a vision for the future/ goal(s) that should be achieved
2. Get out of the fucking way of your minions, and trust that you hired correctly, to let them figure out how to get to the finish line
Bonus Points:
3. Your temperament is in the Goldilocks' zone of neither being too much of a spendthrift, nor too much of a miser, when setting budgets, i.e. you're not some rando without P&L experience that was tied to your bonuses.
Otherwise they’ll burnout and the resentment of the state of the codebase will be jet fuel to that brand new revolving door on your team.
Commiserate to some degree, ‘look, I know this shit sucks’. If they don’t know you empathize, they won’t trust you.
Are you spending your time second-guessing the developers and micro-managing them on the exact thing they are supposed to be experts?
Your phrasing has that "feel", but it's far from a sure conclusion. Anyway, if it is the case, you may need to reevaluate both your hiring (should you get more senior people?) and management (this is a clear "should never be done", unless your people are explicitly under training) practices.
If you have to find the smartest people to keep the wheels on, you’ve already lost.
Disdain for the bell curve is the fastest way to get overengineering. Very few things have to be rocket science to create a good company. Everything else should be dead simple so that anyone can work on it. That also means you have to compartmentalize the hard bits so they don’t leak across the entire codebase.
But some people get bored with mundane code and will make things interesting by replacing a boring task with an exciting one. It’s part of the Curse of Knowledge. You can’t run a company like an action movie. Everyone gets burnt out.
Overly simple, hacky, narrowly spec'd code produces the same tech debt as overengineering. Anybody who's worked in a move-fast-break-things type startup will know how much engineering resources are wasted on rewrites/bugfixes due to this.
Ultimately, as with many things in life, you need to find the right balance.
For example, premature optimisation is frequently mentioned as over-engineering but is it premature optimisation to use a map/dictionary instead of an array for key-based access? Nope. That is just correct. Is it over-engineering to know that if your product succeeds, you could end up with X-hundred objects which will use up all the RAM you are storing them in and therefore you might want to make them smaller/store them off-board? Of course, you can come back and refactor later but it is so much easier for the person who writes that code to understand what can be done on day 1 rather than assuming it is cheaper to refactor later only when needed.
I think a better take is for people to accept that no app lasts forever. If we build that assumption into our worldview, it will help us make better decisions. I still think some engineers think there is some perfection that means the app will live forever.
Maybe, maybe not. While the map/dictionary is easier to use, if the number of items is small (which it often is) the array will be substantially faster because the CPU will pre-fetch the next element into the cache while you are still checking if the current one is the right key. Maybe - check with a cache aware profiler, and you need to design your array to be cache friendly.
Of course the above assumes your map/dictionary is on the hot path (seems unlikely), your code is too slow (probably subjectively, but in some real time applications this can be objectively measured), and you are writing in an efficient language. IF All of the above are true, only then should you go to the complexity of using an array to do the job of a map/dictionary.
http://www.smashcompany.com/technology/high-availability-is-...
This is a true story about an entrepreneur who was suffering a bad case of perfectionism. Instead of showing his MVP to potential customers, he just kept investing in the software, chasing an almost impossible ideal of high availability
I've found is that once a dev has an idea of how something is suppose to work it's hard to get them to think in a different way about that thing. So when that thing doesn't work out for some reason they just keep adding exceptions and adding exceptions until you have the horribly over-engineered solution.
Maybe the company was betting at the time on an integrations marketplace, or a lot of partnerships, so a robust integration platform was built. Then the company moved on to another bet after only writing one or two integrations on the new platform. Nobody over-engineered here. Everything was built to spec. But three years down the line the new dev is going to get assigned a bug in one of the integrations and lament how the code was over-engineered. Hindsight is 20/20.
Lots of stories from the trenches including many in this thread make this mistake. The same goes for 'tech debt'. If you weren't there, you don't know.
If you’re trying to answer “did the people who built it exhibit poor judgment resulting in the system being over-engineered?” then you need to know more than just the after-the-fact outcome.
Examples
Having a 'user info' table with 190+ boolean columns for 'country' ('US','CA','DE','IT', etc) in case someone wants to indicate they're 'from' two countries.
Joining views on views of joined views which are themselves built on joins of other views is, likely, not a terribly sound data decision. (X: "It worked fine last year - you must have broken something." Me: "well... last year you had 450 users, and now there are 37000 users - this isn't a performant way of doing this". X: "Can't be true - the person who wrote it is a DBA")
Not for long…
I agree on the point regarding the decision being 'stupid' but in this case it was not because of inexperience. On the contrary, it was because an overruling decision by an experienced manager. So the point is that it is not always technical or due to inexperience (though that does happen) - from what I've seen, it is quite common for such things to happen due to hierarchical/ego/political issues as well.
https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence
What that distinction recognizes is that it's possible for something to have been well-engineered at one time, while still being over-engineered today. Header files in C and C++ are an example of this phenomenon. They solved a very real problem with technical constraints from 40, 50 years ago, both in terms of computers' capabilities and compiler technology, but, since those problems don't exist anymore, they function as over-engineering.
Basically, the way every programming language newer than C++ or Objective-C does it.
Now is over or under engineering a bad thing? that depends on how many resources the company can muster, your customer tolerance to bugs/failures, and whether you can build a small subset of features well. In practice I've observed products with fewer, but well built features tend to succeed more so than buggy featureful products. Large companies may opt to overbuild everything simply so that they can prove that the product was bad rather than having to decide whether it was product or execution that failed.
Often when you do one thing really well you may find there is more value in the primitives
Technically speaking, for any given situation there's no way you can be wrong.
But this post describes a distinct and, by my experience, dominant industry phenomenon. And the stakes are as high is OP says (dead products).
Industry engineers in the large are massively divorced from product outcome. This should be no surprise, as it is a seller's market:
Delivering product simply and directly can be, depending on perspective, somewhat of a mundane and Sisyphean activity.
Engineering "stuff" is far more intellectually engaging, and there's no market pressure (for engineers) against over-engineering. If my company fails to deliver a product outcome, whether a viable company or startup, in this market there is no recourse at all on my salary let alone my hireability.
I say "depending on perspective" because if you can find a way to be more intellectually and emotionally stimulated by shipping stable product over time, then ime you _can_ unite your joie de vivre with market outcomes.
But that does not at all describe most engineers in industry who generally at best shed off some market value while spinning unnecessary webs of toy abstractions (ie liabilities).
I'm currently walking away from a code-base (and the company who let it be built) that was obviously built by someone who's mental model of how the code works was incorrect. At some very basic levels, it under-performs compared to literally every other example of this sort of code I've ever seen (and it's extremely common), which results in a subpar user-experience and an EC2 instance way bigger than necessary to run this sort of code. A lot of code running doesn't need to run in the first place, and due to the incorrect mental model, there's a substantial amount of code that doesn't run.
Nobody really understands how bad it is, though, because it's barely tested (and the engineer who wrote most of the code has argued that we don't need tests, for various reasons) and all of the engineers on the team who didn't build it initially, have told me that they're not familiar enough with how this sort of product should work to understand that it doesn't work like that (again, this is a super common kind of product).
There's a lot of other anti-patterns (I started keeping a list in my first week) but I think these are the most damning. This code is in production. Nobody at the org brought up these issues earlier in development (like saying "hey, nobody else's product does [annoying thing that doesn't need to happen], why does ours?"), which leads me to believe that the whole technical org is at fault.
It sucks and I can't wait to be done with this experience.
Some of it makes sense when the reasons were explained (e.g. "we figured this would make it easier for [business unit] to do [thing] which turned out not to be something [business unit] actually has interest in doing").
Other examples are purely of this sort (e.g. "we'll be glad we have this in place when we finally hit [x] users or enter [market]").
Design wise, I know it when I see it but other people may see different things.
I agree completely on pushing off features that are not core because it is better to learn in production than to daydream in meetings. I would say most phase-2 features never get implemented.
1) The product is horizontally scalable to 3 orders of magnitude more traffic than the product will ever receive.
2) Bespoke assembler/hand crafted memory management/other do not touch code to shave 5ms off a 5 ms call on an API that isn't latency sensitive.
3) Ability to handle N customer tenants while only handling 1 customer 3 years later.
4) multi-region redundant deployment for 1 customer strictly based out of the US with no availability requirements.
5) 100% coverage for integration tests, unit tests, CI/CD, for a single tenant blog that is never updated and no one reads.
6) Custom internal framework with micro-service splits to support team of 100 engineers working on app when there are 2.
7) Automated continuously trained ML ranking model which performs .001% better than sorting by popularity or other trivial heuristic on a social media app with 10 users.
The common theme in many of these cases is that these are pretty good ideas if you have unlimited people, but have fairly low impact. In some cases these may be counter-productive to successful delivery at the company/product level. A piece of software built wrongly due to poor understanding of the product domain is often considered under-engineering.
It's completely wrong given the context 99% of companies are operating in. It's no wonder a Zuckerberg can come along and build a crazy successful company in his early 20s. He likely didn't know any "best practices". Common sense was enough.
In my org, we love to call this out with an (often private) "We need to do it because Google Does it!" -comment.
Complexity kills projects and the will to work on them. It can be caused by either over or under-engineering. With the first you end up with extraneous layers of abstraction and badly applied design patterns that make it hard to understand or reason about the code...in the second you end up with a big ball of mud that's impossible to understand or tease apart.
In both cases the code gets harder and harder to change to the point that no one really knows how it works or wants to risk touching it.
No tests? Nobody knows how it works? It doesn't perform well and doesn't fit the problem it's solving? That's very much not over engineered in my book.
> Does the product generate the revenue for their paychecks, plus extra for the company?
You want to be comfortably generating/contributing more revenue than the expense of employing you.
All it takes is a good enough sales and marketing team, or the right executive relationships, especially if the team being “supported” is small.
You can say the company or division is presently functioning, but the time constant on the product is different, and the coupling is too loose to say anything else with accuracy.
That's the point though. Getting caught up in engineering purity or even whether it's objectively _good_ is a waste. The product exists to drive business (financial) metrics. If it's doing that, it's working.
If the product is poorly crafted, and especially if it's also poorly tested, adding features can introduce bugs, and fixing bugs in one place can cause issues elsewhere. Development then slows to handle these occurrences, which can then impact product viability.
> and the engineer who wrote most of the code has argued that we don't need tests, for various reasons
Sounds like you made the right choice to walk away. "My code doesn't need tests" smacks of match-over-the-shoulder type of chaos creation. This person has likely never had to maintain their own code or someone else's, nor has anyone ever had to maintain their code. 0/5 would not work with.
Tests are wonderful. Tests are proof your code works. They are something to be proud of. And they defend your code against the chaos monkey of future people refactoring and rearranging.
A codebase that pivoted multiple times makes more sense the more time you spend in it, an overengineered one makes less.
The specs and the decisions surrounding the code are usually part of the over-engineering.
> Everything was built to spec.
Even with reasonable specs, I've worked with, and later managed, engineers who had a strong drive toward over-engineering solutions. I've also done it myself, and I suspect most people reading this have done the same.
The problem is that many of us engineers became developers because we enjoy building things and we enjoy solving complex problems. It can be exciting to imagine a particularly complex version of a problem and how elegant and fun a complex solution would be.
It can be as simple as a programmer who tries to write their own CMS when the spec could be solved with a simple blogging platform. If you're an ambitious programmer, writing a CMS is fun! Installing Wordpress is blech.
I didn't truly understand the ubiquity of the problem until managing ambitious engineers who wanted to do complex things. At times it felt like 1/4 of my job was just asking "If we wanted to simplify this solution, what would that look like?" Repeat for a few rounds and eventually the team has shed a lot of unnecessary complexity without sacrificing the spec at all.
Can you prove it in a court of law? No. But you still can often easily diagnose it. And you'll usually be correct.
For example the conditions under which https://wiki.c2.com/?SecondSystemEffect happens are well-known, and the pattern has been repeated often in the half-century since it was named. If it is a second system, and you can find complicating abstractions in it that get used exactly once in practice, then with better than 90% odds it was over-engineered. And the farther you dig, the stronger the evidence will become.
I would argue this is a case of over-engineering. Though, it extends beyond pure engineering.
This is poor resource management and business de-risking. An effective approach here would have been: "Let's build out a handful of integrations. Make sure they're going to deliver the value we want. Then abstract it into a repeatable pattern".
The trouble with "Let's build out a handful of integrations. Make sure they're going to deliver the value we want. Then abstract it into a repeatable pattern" is that when you're building an integration marketplace, you need partners, and they don't want to be your research bitches.
They just want to build their integrations once, and gtfo. If they think you're signing then up for a revolving wheel of experimentation, they'll just politely stay away until someone else has done the hard yards.
Sure if you're a Microsoft or a Google you'll have any number of willing partners who will put up with anything to be the pioneers in your integrations marketplace.
But otherwise, they're using your integrations marketplace purely for the benefits, and they don't want to be building on sand.
First line in the second paragraph of every dead company's obituary.