I like to believe that serverless technologies and cloud services reduce complexity for the org, but obviously that's at the cost of offloading that complexity to the cloud providers (and welding an org's software to that provider for years, if not decades).
So different, but not less complex overall. But maybe there is value in having some of that complexity consistent across a small number of cloud providers.
Every time our operating costs go up (eg, cloud providers), something else in the organization gets more complicated in order to make up the shortfall. Or super simple when we go out of business.
Some developers are merely bad at math. Most are awful. Others take this flaw to epic proportions. Be afraid of the developer who confidently tells you the math works out.
It's true that a lot of organizations had operational teams that have become money pits, or pushed back on all quality of life improvements because they don't have the talent, the budget, or the imagination to pull it off. When we move to the cloud we start aspiring to these things we didn't have because they were expensive. I don't know of anyone who moved to the cloud and didn't move the goalposts. We were just talking in another thread, as we often do here, about how much YAGNI is going on out there. Yes, the price per feature goes down, but the overall price doesn't seem to. And I get stuck taking care of things someone else used to worry about, which is opportunity cost on top of it all.
What is also true is that developers can learn a lot from their operational peers and avoid expensive mistakes. With cloud we have none of those peers. We have to learn everything first hand. By someone who is all too happy to let us wrap rope around our necks and then 'rescue' us from themselves. That's a perverse incentive and quite a setup for a fairly fucked up codependent relationship. At least with interdepartmental drama some of the money stays in the company longer before going to vendors.
We used to have devs bringing in frameworks and libraries without reading the manual. Now they spin up entire services or subsystems without reading the manual. You should read through the docs for MongoDB Atlas...which plainly illustrate that you now need a DBA unless you're cool with people pressing buttons in a panic during some type of self-induced performance issue.
They absolutely do reduce complexity! For example, think about building, testing, and deployments.
In pre-cloud Internet times, you'd have an untold number of extremely brittle bash scripts, cron jobs, rsync ssh key setup, fleets of build + test boxes to manually worry about disk space, pre-provisioned dev/QA database servers with also untold brittle sql startup/teardown scripts, and all of the requisite people whose job it was to solely maintain this infrastructure along with database tuning, build fleet monitoring, the list of menial tasks just goes on and on and on.
Today, you have a yaml file in your .github/workflows directory.
Now I agree that there are "different" requirements. Understanding the complexity of your workflows etc is no small feat-- but you're replacing such a huge amount of what used to be extremely expensive and brittle architecture with, basically, a text file or two. That's a huge cost savings.
Are you counting Chef/Ansible/Puppet/etc as things that came during the cloud era or that those things were not used to solve the problems you raised before the cloud era?
Having used Puppet, the promise of Puppet was very different than the realities of Puppet and imo more accurately reflects the "different complexities" point. The point of those provision-bare-metal-on-the-fly frameworks just evaporated when, for example, you could launch a container on Fargate with, again, a single text file, or similar on Heroku.
idk, in the enterprise world all those things still exist but they're applied against AWS/other instead of your servers in the datacenter. Instead of "build+test boxes" you have "build+test instances". It takes the same level of redtape/approvals to get an AWS instance as it took to get a server in the old datacenter. All my enterprise clients have the dedicated infrastructure teams they always had, only now they're working in AWS/other and not the datacenter.
I am going to go out on a limb and guess that this is almost entirely down to your finance department attempting to understand and control the monthly cloud spend. This is obviously fine, but rather than "oh, another 2xlarge, we have to add a couple hundred bucks to our budget" the reverse should be true; you have $x to spend per month on average with a reasonable growth plan built in.
In the build and test example, the answer to "how much compute is running?" is based off developer velocity and so "it depends" is a fair answer. When nobody is shipping any builds, your cost should be $0. (I've found that this is kind of hard for enterprise-y finance departments to wrap their heads around and is why all those esoteric billing notifications AWS services even exist)
Pre-cloud services days, finance departments had a much easier time. You had racks of physical boxes that had static costs attached to them, you had a static monthly bandwidth bill that let you run at a certain speed, and you had salary costs which are also pretty static month-to-month. The idea of "scale to 0" was completely unheard of. What do you mean your QA environment doesn't cost anything on the weekends when nobody's doing anything? etc etc etc, you get my point.
Another quick example - serving up product image assets to customers. Let's say you want to ship 1TB of images to customers per month.
In pre-cloud times, you would spend untold piles of money spinning up racks of storage arrays, switches, firewalls, leasing out gigantic pipes for bandwidth, and again all of the requisite people in order to get that running. You'd rent space in multiple distributed global datacenters, so again you'd have an unreal amount of bash-script-file-sync services so that when someone uploads a new image it gets replicated all over the globe. Millions and millions of dollars. You'd probably have to write a custom resizing service with ImageMagick, hooked into your frontend, so that you were serving customers the correct size and not blowing through your bandwidth allocation. Just incredibly complex.
Today, you click a button in your CDN provider's console; most of that functionality from above just comes for free.
Again you have to do a little bit of munging your front-end to take into consideration the vagaries of your CDN provider, but overall it's such a huge savings of mental energy and time. Put a circa-1999 systems architect in front of the Cloudflare console from today and they wouldn't believe it was real.
It's hard to describe just how much of software engineering & it's enjoyability comes from attaining perspective, from reaching a point where you see & understand how a big pile of stuff works.
> One thing working in complexity’s favor, though, is that engineers like complexity. Admit it: as much as we complain about other people’s complexity, we love our own. We love sitting around and dreaming up new architectural diagrams that can comfortably sit inside our own heads
It's harder to acquire a sense of mastery & possession, of real understanding, when there's been multiple iterations of teams hacking on a piece of software, when it's many layered & has lost cohesion & concerted intentionality.
Trying to describe how relishable, how enjoyable it can be to explore, to search & quest for meaning, to push designs that hopefully make sense, that hopefully grow, that hopefully wrangle: it's an under-sung & hard to tell story. It's interesting to me having such a big part of the world running off of a knowledge-working that is so under-described, so hard to communicate personally about.
I'm waiting for the day that AWS collapses under its own weight. Heard horror stories from backend engineers working on the infrastructure/dashboard code there.
If anything were going to do it, it'd be the labor crunch that we're seeing here. Without a constant influx of fresh-faced college coal lumps to burn in the tech debt engines Amazon would have a harder time of it. They've done a good job of pivoting though, and I think they'll weather the storm here (and be better off for it). See the increased comp to boost hiring rates - especially of non college hires - and an internal acknowledgement that addressing meaningful debt or internal pain is also a potential path to promotion.
So, it'd be funny to see, but I don't think it's happening anytime soon.
I think there’s basically a zero percent chance of this happening. Aws may internally be full of technical debt, but for ten years now Amazon engineers have been automating the reliability issues of the platform. The capabilities are expanding, but the core components are some of the most rock solid engineering on the planet.
tl;dr premise = society is complex, software is complex; complex societies collapse... therefore complex software collapses, question mark, waggling eyebrows, surely makes you think etc etc.
> ...Opinions expressed in this blog are mine *and frequently wrong*.
Emphasis added.
Equating the complexities of software with the complexities of societies (the premise of this post) is a fun and provactive blog post. Which is all this is. I was really hoping for a serious treatment that meaningfully dug into this, but that's my problem.
I don't buy the explanation of societies collapsing because they're too complex, Some software systems do.
But not because some esoteric systemic property of complex systems, but just because working projects get deprived of maintainers and eventually there's nobody that understands WTF original programmers intended and how to fix the holes.
So incomprehensible portions of the system are circumvented or replaced with something that just works. Actually the same happens in societies, unless some group sabotages the workaround.
My issue lies with software engineers making associations that society is like software.
The causal factors behind why societies fail and why software projects fail are completely different. Complexity is a description of the system and a very bad measure to draw an equivalence between the two.
The risk with drawing this equivalence is it grants license to software engineers to think that similar enough models apply to "managing software" as
"managing societies". Because when all you have is a hammer... But society is not a nail.
This is a problem I want to focus my life solving. I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers. I believe that there are finite and fundamental classes of problems, that underpin the vast majority of problems, that have a distinct visual representation, and that by representing these problems visually, you let your "visual coprocessor" assist in seeing the solutions. I believe there are well-known strategies (taken from the real world) for managing hierarchies of complexity that ease the cognitive load of exploring any problem. We just need to get these approaches into a programming "langugage"!
It's a solved problem, if you are solving the right one. Most companies out there are solving problems they don't have. The FAANG disciples descended upon the world to preach the gospel of Complexity, and everyone bought it - because everyone wants to be like Google.
What, you think "microservices" is a new invention? We called it distributed systems, and we really knew not to go there unless we wanted to decimate our productivity and sleep.
Good luck but your motivations will probably not line up with the masters who dictate things that become complexity.
For example, I look at our software running on .net MVC and think as an Engineer, it simple and knowable but the Front-End Team are less worried about simplicity and more on flashy front-end stuff since that is their job. We end up bolting on a Front-end JS framework and complexity immediately ramps up by like 300%.
Are they wrong for wanting a better and more flexible front-end? Not necessarily, I mean all the other companies have cool stuff and if we don't maybe our company dies.
this reminds me of the Head First series of books. I can still recall specific pages from the Head First Statistics book not because of the content but because every topic (and probably page) had a creative visualization.
Me too, and I definitely agree that it is time to move past thinking in "languages" to a more holistic view of programming systems. The fact that as of now, only ~1% of people can access programming and wield the magic of the computer is a tragedy akin to the comparable levels of literacy in the medieval era. At the present, there seems to be a minor renaissance of thought along these lines stemming back to the pioneers of the field (McCarthy, Engelbart, Kay, etc). Everyone has a different take as to what the answer is, but I think "let 1000 flowers bloom" is the best way to find a solution to a complex and largely unknown problem such as software. Personally, I'm afraid that if we fail, the means of programming will become gated behind AIs and society will continue in its trance-like state of passivity instead of creative expression.
We will have to contend with the last bit regardless. Programming requires attention to detail and tolerance for tedium, the erosion of which will only continue as AI grows.
> The fact that as of now, only ~1% of people can access programming and wield the magic of the computer is a tragedy akin to the comparable levels of literacy in the medieval era.
I don't know why programmers think this way. We don't expect everyone to be a doctor, a botanist, a novelist, or a musician but for some reason we think anyone can be a programmer. That's just not the case -- programming is a skill like any other -- it takes some natural inclination, some training, and a lot of practice. Just like any other skill. A programming language is to a programmer what an instrument is to a musician.
No, but we expect everybody to know reading, writing, and arithmetic. Programming may be more intrinsically difficult than these, but most of the reason it's so hard today is because of solvable problems that beginners run into, to the point of half of expertise is just knowing to avoid all these pitfalls. The downside of the open source developer ecosystem today is that putting it all together is typically a painful process, which is what Replit is aiming to solve. There is a world of difference between a good IDE and a bad one for a beginner, and a lot of that comes down to integration with the language.
Anyways, there's a reason programming should be made more widely accessible - it develops thinking skills and rationality when you learn it, as Mindstorms pointed out. Between the anecdotes in the book and things like [0], I think it's a travesty that we aren't pursuing this to its fullest extent, and instead try to teach children Python or JavaScript, two decent languages for software development but not exactly forgiving with beginners.
When I was a kid we also computer class were programming was taught. And music class. And drama. And woodworking.
And like with these other things, you can make getting into them easier. With music, children start with simple instruments that no professional ever uses. And it's the same with programming, there are plenty of easier environments for children. But if you want to make programmers and you want to make musicians, eventually they have to use the real thing. My own son jumped straight into the deep end of Unity development knowing nothing because he wants to build something real. I neither encouraged or discouraged that environment and it's pretty unforgiving.
I don't think the solvable problems you speak of as are as solvable as you think they are. Also making software development out to be special both in terms of it's benefit to thinking skills and rationality and how it's merely some tools away from being professionally approachable to the masses is totally unfounded.
I think they're saying we should make the tools to program more accessible and allow people to play music easier. Sorta like an electric keyboard vs a grand piano.
We already have those tools for just about every age range. But like with music, just because you can play twinkle twinkle little star doesn't mean you're going to be able to join the orchestra. And if you do, you're going to need to able to play that grand piano.
I don't think you can solve it. (Not a comment on you - I don't think anyone can solve it.) The problem is that software at least to some degree reflects the external environment.
Let's say your software is in a financial company. Their software has to enable them to follow all the government financial regulations. Well, the government is following the larger societal "complexify to the point of collapse", and the financial regulations are certainly doing so. That means that the external behavior (the "business logic") of the software is insanely complex. You can't make that go away just by visual programming.
But maybe you're not in the financial world. Maybe you're just writing programs for internal corporate processes at some generic company. Well, your software is still subject to the complexity that builds up in the company processes. Again, the programmers can't eliminate that complexity.
Or maybe you're writing a customer-facing app - an external-facing web app, or an application that people actually install on their machines. Here you're at the mercy of the product or project manager trying to find new things for the app to do, and they still complexify the app to the point of collapse.
The problem isn't that programming is too complicated. The problem is that what we want programs to do is too complicated. Visual programming can't save us from that.
I want to believe you, but there are just so many complicated problems IN REALITY that we have to model in software that I don't really see complexity going down.
Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that convention. Storing both risks the data going out of sync. What's the "right" way to store person names? There doesn't seem to be a simple solution.
Another example: we model physical cables (both for power grid and for data transmission) in our CMDB. All works fine, until you suddenly have a Y-shaped cable with three connectors that doesn't fit into your data model. The real world always has these 1% of cases that don't fit the general pattern; if you focus on the 99%, the 1% make trouble. If you focus on modeling every case, you have 10x the complexity, even for the simple case.
And then there are things that are moderately complex and security critical, like password recovery workflows. We haven't really found a way to reuse these among different technologies. Like, if you once figured out the perfect password reset workflow with Ruby on Rails, and your next job uses Python + Django, you're back to square one.
If somebody has a good idea for how to tackle these problems, please let me know!
As you implied, I don't think the answer is to make the perfect abstractions that can handle any scenario.
But where software does fail in my humble opinion is making it easy to pull in tried and true tested solutions to the problems that we do face, even if they are not as common. Because even though they may not seem common, I'm absolutely certain many face the same scenario.
The amount of duplication solving the same problems is insane. But this is not an easy problem to solve and I don't intend to trivialize it.
I think we need to come up with better tools. code sharing through libraries/repositories (like npm) is great, but it can't be the final solution.
Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service. You could import modules, functions, data structures, anything. But that vision is still a long ways off.
The problem is "pulling in a tried and tested solution" means pulling in a slice of the whole stack: CSS, Javascript, client/server communication, API endpoints, data-flow logic, and database schema. People are very very bad at this because they believe that software engineering principles apply only to little tiny portions of this stack. If you actually apply your software engineering principles to the whole stack, you end up with something that looks very different than what everyone else is doing (so there's not a lot of tooling to help with). I have a better way to do this than anything else I've ever seen, but even with that I struggly very strongly with the JS/API side.
Kinda reminds me of what they do in some areas of the finance world with Kx System's Kdb+. That entire install is tiny tiny and you get a high performance array language for querying your SSD database with either the main language "k" or a syntactical sugar dialect sitting on top called "q" that is more like SQL.
I think of that as optimizing the entire stack for what the customer wants of rapid/high-speed analytics.
KDB is unique for translating "in house" tech to marketable CV entry status. Having Haskell and CS(FB) experience requires educating the recruitment agencies, for comparable recognition. Personally I consider every ill in computing as symptoms of the intellectual property system, from making reading potential prior art (aka learning) verboten to non disclosures making the Peter Principle look like the common cold to covid.
I had trouble following your message. You've stated that Kdb+ looks good on a resume, Haskell requires explanation, and IP is bad? I'm not a big fan of closed source myself (and Kx Systems sounds like it would have a lot of red tape), but the technology is supposedly very fast and I find the language to be very simple and charming. I'd love for our analytics database to migrate , but I think the technology would really confuse our IT folks and it would be difficult to describe why it would make our jobs easier.
Anytime! FYI we've released a new product KX Insights which is a cloud native version of kdb+ that supports ANSI-SQL (PostgreSQL) and has a bunch of ease-of-use and interoperability improvements to open the technology to a much wider user base :)
> Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service.
Well I'd argue it's more Model => String => String if you are talking about using a fully pre-trained model. Even then you could probably expand on the type signature of the model to make it more useful.
But if you look at like co-pilot for example, if that was given the ability to have type signatures serve as input you might get a lot more powerful results than what it does with raw text (which is very impressive).
But this comes down to type signature design. You can encode any function using simple types like int -> int which aren't very useful. Where Haskell shines is when using types to limit the scope of inputs & outputs. What I am getting at is that you can still write uninformative type signatures in Haskell, but it also gives you the power to write more informative ones.
I don't think Haskell is the answer, so please don't take that as what I am saying. I do think however using richer type systems could be a stepping stone towards a solution to this problem.
Type signatures (or ASTs) are structured data, and Copilot mainly works because GPT accepts any text and doesn't have to deal with your logical fully recursive concepts like ASTs.
I feel like it wouldn't work if you could actually mathematically constrain the outputs to be syntactically correct either. That's probably one of those Gödel things.
> Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service.
Type signatures don’t tell you which should be the “then” clause vs the “else” clause in any conditional.
Don't try to force schemas onto schema-less data. Store the "name" as a JSON string/blob representing the various possible attributes (given, middle, family, title, etc) and provide a variety of functions for representing that data. IF you really need to do this at all (for an internal app, you probably don't).
> Physical links
Include an Hardware Asset FK in your Link M2M table. Model each binary link explicitly, so a Y cable = 2-3 different Links that point to the same cable Asset. Or you can have single Links with M2M inputs/outputs. But definitely don't model Y cables explicitly.
> Password recovery
What is so complicated about this, specifically? Python and Ruby are completely different languages with no guarantees for interop.
That's strange, because my one (1) assumption is that the name can be represented as a schema-less dictionary of zero or more Unicode strings. Heck, one of those strings could be an ID for a bitmap or SVG of their non-textual "name".
If you can't do that, you have more pressing issues.
> Include an Hardware Asset FK in your Link M2M table. Model each binary link explicitly, so a Y cable = 2-3 different Links that point to the same cable Asset. Or you can have single Links with M2M inputs/outputs. But definitely don't model Y cables explicitly.
Our cables have stickers with cable IDs, which are stored in the CMDB (and can be used to trace the endpoints through patch fields, for example).
So if you model a Y cable as two physical cables, you have to both allow duplicate cable IDs AND multiple cables per port. Both of these have significant downsides in preventing data entry errors.
Or just have a cable type column (which could be a pointer/foreign key back to a table of cable types, so you can add the cable with 4 ends that comes out next year).
> So if you model a Y cable as two physical cables
No. Model it as two abstract "Physical Links" or "Connections" which FK to the same physical cable Asset, which is uniquely determined by its inventory ID and/or manufacturer model+serial.
If a Link needs to know whether its part of a Y cable Asset, it can join the Asset table.
If a Y cable Asset needs to know what it's connected to, it can join the Link table.
Additionally, you have a nice clean Link table for doing graph traversal (get me everything connected to this impacted Asset by degree <= 2), and a nice clean Asset table for inventory tracking.
I could be wrong but using JSON as format for name is not a solution, its abstraction over possible solution.
Suppose we store all names as JSON structure. Question - how people will create such structure from their names? Using form with dozens of fields and checkboxes? Second question: how we will display this structure? We have to find a way to convert this JSON into string to show name. And as result we will surely need function to convert this shown strings into JSON for sure. And how we can solve issue when some names could be presented as a bit different JSON structures? And how we can validate that these slightly different JSON structures are equal?
> We haven't really found a way to reuse these among different technologies. Like, if you once figured out the perfect password reset workflow with Ruby on Rails, and your next job uses Python + Django, you're back to square one.
C and C++ libraries with light language-specific wrappers largely serve this purpose, in practice. It's plausible that, say, a PHP postgres client lib and a Node postgres client lib will share nearly all their code—probably as C or C++.
This doesn't get leveraged much aside from interfacing with daemons and sometimes for extremely complex e.g. media or crypto libraries, though. No-one's doing this for high-level software workflow building-blocks, like a password reset flow.
[EDIT] as for the "why", I suspect it's because the things it's used for are far simpler than the things it isn't. A password reset flow can potentially need to interface with lots of different things, some of which may be custom to the project, some of which may vary with run-time input, et c. If you cut it down to only the parts that could truly be re-used anywhere, with all kinds of points where you can hook in as needed... you've only made about 5% of the work re-usable, so it's just about pointless.
I'd say this is more true for C vs C++. It has been my experience that the interop between C and languages like Ruby is much easier, the tooling is usually much better (just use ffi!), and the complexity is less (since that is what we're aiming for right?)
>> We haven't really found a way to reuse these among different technologies.
The Common Language Environment on VMS, or its lesser(but infinitely more available hence virtuous by effect) imitation, dot net CLR . FFIs, if you bought a good environment. Sounds like if open source implementation of good foreign interfaces existed, the OP's problem wouldn't exist, let alone system level interoperability.
Store both a unified name field and separated given name / surname fields and let the user manage both. Yeah, this risks going out of sync, but that's the user's problem, not yours. Yeah, three fields are technically more complex than one or two, but it produces less complexity down the line.
struct Screenname {
Matches<String,regex"\w+"> _;
// TODO: ensure it is illegal to ask for users' real names
// for now we'll just do the right thing unilaterally
}
Actually, it doesn't. All of those are, well, addresses, and have some mechanical structure necessary for routing (admittedly, for phone numbers, the structure amounts to "any sequence of 0-9, #, *, and possibly those four wierd extra symbols with the fourth DTMF column tone"). You probably don't (or least shouldn't) care about that structure as long as it routes correctly, but it does exist. (Ie, you'd still use a string for that data, but it's not inherently incorrect not to, just pointless and error-prone.)
I think the point is that someone else cares about the structure and sets its rules - which may change without warning or have complexity you are unaware of - and therefore if you are not the telephone company or the post office you are likely overstepping if you try to enforce structure rather than merely passing along what you were given.
The rabbit hole on names goes deep [1]. Lots of other fun rabbit holes. [2]
For how to model physical objects like those cables, go see how Grainger does it as a working practical model.
To really handle names properly, you need more context than the name in the presentation layer that many schemas take their modeling from can obtain. Government health care or similar widely-adopted encoding is sometimes Good Enough. If you want non-lossy exactitude however, then that's a much bigger scope (I'd be investigating a first-pass classifier with contextual hints taken from various geolocations, age, etc., that implies soliciting the name comes after what you normally solicit for input, and refining from there).
Password reset; this is why vendors like Okta exist to abstract authN away for us, and auth0 for authZ. Then there is the rabbit hole of what this abstraction leads to...
> Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that convention. Storing both risks the data going out of sync. What's the "right" way to store person names? There doesn't seem to be a simple solution.
The way past bureaucracies that didn't have the luxury of unlimited cheap complexity did: you set some rules and anyone who doesn't like them can either suck it up and follow them, or deal with the consequences themselves. Allowing users to specify arbitrarily complex requirements and never saying "no" is what gets us into this problem.
I try to work on the side in a spiritual successor of Foxpro/DBase (https://tablam.org).
I consider a mix of relational/array model fill a lot of ergonomics for this (and it was proved to be right by the family of DBase langs).
But what makes this much more complex today is the explosion on OS targets (Windows, Linux, MacOs, Android, iOS, Web), and the requirement to integrate with many other stuff, dealing with many formats (json, xml, ...), is harder to do UIs now and the base support is more inconsistent and mixed as ever...
So, is possible to make a simpler tool, I certain of it, but then the developer/user will say "ah, ok, so how this connect to Redis, GraphQL and Amazon Web Services, run this on Android, Windows, parse CSV, ..."
and that is what make this very hard at the end...
What you're describing is one approach, the other is that you pay a bunch of SaaS/PaaS/IaaS companies to handle a lot of the workload for you so you and a small team can just focus on the business logic of your particular niche.
Agree that this helps in some ways, but it also adds complexity. You can e.g. subscribe to a billing SaaS, but then you have to link your system to the billing system somehow
The fundamental problem of simplifying software is humans. Just consider date formats, time zones, and tax codes. Humans love to make complicated things.
My philosophy on this has been to take the complex human stuff and stick it in a black box. A professional feather in cap with this approach is called BladeRunner ( https://dl.acm.org/doi/10.1145/3477132.3483572 ) which radically simplified the distributed system aspect by putting all the gnarly glue and business logic in a V8 VM (JavaScript).
My next thing is a focus on board games where I have invented a programming language and have started to evolve a platform. It's called Adama ( https://www.adama-platform.com/ ), and I think it is pretty cool. The interesting thing is that the complexity of board games is exceptional.
I have a few clues to share. The first is that reactivity, which is found in excel, is a key to simplifying software as this makes the glue more automatic.
Another clue is figuring out bidirectional communication which relates to reactivity as a two-way street. However, this is primarily hard because we don't have great things off the shelf to deal with this beyond TCP. For more of a deep dive, check out https://www.adama-platform.com/2021/12/22/woe.html which talks about WebSocket.
My final clue is that you can't run away from state. So many people offload state because state is hard, and you have to contend with it. I'm building yet another database.
If you look from perspective of software creator that wants to handle all kind of timezones or post codes - yes humans complicate things.
If you look from a person or user perspective - time zones or date formats are used in one location or in some context. People living in that context have it a lot easier because most of the time they don't care about other time zones.
I would say people simplify things but on local scale. If you want to go global that is your problem not humans that live in one place and use single time zone all their life.
This is an admirable and worthy pursuit, but keep in mind that any process you make streamlined:
1. becomes a post from which clever and ambitious people can build complex things. You invent docker to simplify application deployments and then somebody builds a n-dimensional microservice cloud on one side and starts commercializing new hardware architectures on the other. You don't remove the complexity, you just let it move around into new domains. (not a bad thing!)
2. is more temporal than you expect. The "finite and fundamental" problems of yesterday, today, and tomorrow are of different sets -- partly because (1) opens up new problem classes and partly because most problems are inescapably cultural and therefore subject to fashion cycles.
> I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers.
Not that I want to discourage you, but my view is that anything that makes software engineering simpler just leads us to tackling more complex problems until the complexity reaches the limit that people can handle.
So in that view, you can't succeed at making software engineering always simple. Instead, you can make previously intractable problems tractable.
If history is a guide, then you are correct. But I believe complexity is fractal (I think we all know this: "turtles all the way down..."), and that you can structure and manage complexity with recursive rules, so that you only see the resolution that you care about, for the areas that you care about, and everything else is a low frequency representation.
Frameworks approximately do this, but for specific domains: they let you organize complexity into well-defined areas and put a pin in them, easing the cognitive load. Then you can handle those abstractions more easily. But yes, to your point, because frameworks typically only tackle a few levels of complexity, you still get that complexity back when you use the framework to advance the problems to the edge of what your framework is designed to address.
A recursive/fractal management of complexity will allow all levels of the hierarchy to feel similar, so you are never increasing complexity, only looking at a different resolution. I think the key to this is mapping out the fundamental organizational problems and how they relate to each other at different resolutions.
This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely.
However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That is, if we can get over the initial switching costs.
> small team of engineers
It took 2 men to over-take thousands creating Unix over Multix. It took Linus Torvalds - just about alone - a week to create git. We have already seen this prophecy come true.
As teams get bigger, communication sales factorially, and the more people you have the more mistakes you make, which increases the size exponentially with each further mistake. What Unix and git showed is that when you put everything into a small team of engineers heads, they can work through the complexity enough until they can do it themselves.
> Frameworks
One of the things I realized a while ago is that pure/impure and library/framework have a decent mapping between the two. If you have a framework, you give it code and it acts for you, just like impure code. And so the problem we keep running into, is instead of inverting the flow like 'hexagonal architecture' says, we keep piling impure onto impure onto impure. Hexagonal says to not do anything of substance, keep the adapter clean, but each framework sure is doing something. Each layer on the stack we go up, the harder it is to get down. So now we have OS and applications and containers and k8s running micro-services that ends up being run in a web browser, when all we really needed was microkernals.
Correct, sales does have to scale factorially to the development man years. That's the problem with the prevailing economic contradiction. See Jamie Dimon, JPM, three days ago.. I've never witnessed a bulge bracket chief concluding the most public statements with, roughly, "doh, I dunno, it's crazy" before. My father cut his banking teeth in the Great Depression. That was actually much more simple. (Whether because the tools of the trade were more simple or just thankfully..) "Hurricane" is the search term for Dimon's delivery...
The number of channels is exponential, that's just Metcalfe's law.
The ability to suss-out meaning behind those channels and come to a shared understanding pushes it to factorial. If A is speaking to B and C, A needs to also think about what communication is happening between B and C, and this is different to what B knows about C and C knows about B.
In a carefully balanced classroom you might manage to get everyone knowing the same things, and this will allow knowledge to scale. Think about how much better TV got once they knew that viewers would watch every episode. What happens in software is that specialization quickly comes into play such that this is impossible in any organization that doesn't use mob programming. Other people might as well be speaking a different language.
Next time you are in a Sprint meeting, think about how much you don't understand. Even as the team lead or architect who designs the entire system and whose job it is to understand everything it will be a shocking amount - it's black boxes all the way down. You'll claim that you can't know everything and that anyone who tries will fail.
And this is made worse as the only true way to learn something is to do it, and if you aren't actually challenging yourself on something, anything you learn without doing will just fade away - as Spaced Repetition shows.
The bigger the team, the bigger the software, the more inevitable the collapse.
> The number of channels is exponential, that's just Metcalfe's law.
Metcalfe's law is quadratic rather than exponential -- O(n^2), not O(2^n). And if n people all need to be aware of communication happening between every pair of them (which is a worst-case), then that should just add a factor of n, bringing it up to cubic.
(On an unrelated and less pedantically nitpicky note, one of the most valuable professional skills I've developed is an ability and willingness to dive into black boxes. It's remarkable how many bugs arise from the interaction of two or more pieces of software with no overlap between their developers. Debugging those can involve a long, agonizing back-and-forth between people who know how X works and people who know how Y works -- or it can be over in a jiffy if you can quickly familiarize yourself with the basic internal workings of both. Past a certain point nobody can know everything, it's true, but this doesn't need to be as crippling for productivity as a lot of people allow it to be.)
One of the surprises I got when doing that was realizing that Y's code was terrible because of how X was interacting with it, and a rewrite of X allowed a rewrite of Y, and I was able to delete half the code in each. This has occurred countless times.
But that doesn't stop black-boxes from being created faster than I can fix them, unfortunately.
This seems like a fantasy considering how big the development teams for these projects are.
Multics did not have thousands of developers. Linus was just more practical about software development, was maybe better at building a community, and more importantly focused on x86 commodity HW which was a bet that only Microsoft also happened to make.
The cumulative system design contributing to MULTICS, however, probably was a order of magnitude greater work than the implementation effort. On the shoulders of Giants may be clichèd , but it also seems increasingly as if the contemporary world has decided to deprecate inherited wisdom. Blame awful policy decisions starting under Regan..
Oh, you're right. I had in my head "A combination of MIT, Bell Labs and GE" and manually translated that to thousands but it was actually only about a hundred dedicated people.
Although it may have only taken a week to create the initial version of Git the amount of polish and documentation required to get it to something someone like me who's not a kernel developer was a lot more than a week's worth of work. You may be able to get 80% of the functionality in 20% of the time but it's that last 20% of functionality that gives software enough polish to actually be used by someone who doesn't have an incredibly specific reason to figure out how to use the unpolished software.
The complexities at different levels of the hierarchy are so qualitatively different that I doubt whether a unified set of abstractions could ever be practically useful. We're going all the way from CPU microcode to loosely-coupled distributed systems with multiple dependencies on third-party services. If you want to make any progress you'll need to narrow down your focus.
> If history is a guide, then you are correct. But I believe complexity is fractal
Actually, "combinatorial" or "exponential" would be better words to describe complexity explosion. The number of possible cases/scenarios/logic flows/etc grows exponentially as a system's complexity increases. It is a curse and a blessing at the same time keeping so many SWEs gainfully employed.
"Fractal" is used to describe the property of not resolving the complexity detail until you get up close ("zoom into" the complexity). The amount of use cases does expand alarmingly, but even combinatorial and exponential use cases imply they are inventoried and are thus amenable to prioritization.
In a lot of human work (not just software, but any endeavor involving technology even as "simple" as building a tunnel or simply involving many people coordinating together), complexity emerges this "unknown unknowns" property. This has happened since time immemorial. If you don't believe me, go master how to live out a subsistence life in an agrarian area of $nation during the 1500's with their tools but modern knowledge, then try to convince yourself complexity did not exist back then.
This is a very cool comment. I agree with you. We've made operating systems extremely simple from a UI standpoint then we host web frameworks on top of those operating systems. And then we make clusters on top of those frameworks and then we make datacenters out of cluster, etc.
All still manageable by humans but at continually higher and higher levels of complexity.
> I believe that software engineering can be made so simple and clear that it can be wielded by an extremely small team of engineers.
I think one thing missing from this line of thinking is that these are people. They have their own motivations and egos. They get sick, have families, take vacations, quit.
I think part of the way large teams and large companies are structured is a hedge against changing teams with highly variable capacity. It's hard to overstate how much it can hurt a project to lose a someone who has a ton of experience locked up inside their head.
The alternative is that I have worked for places that try to treat thier engineers like fungible tokens that can be shuffled and replaced at will. That environment feels extremely dehumanizing and demotivating.
I agree with you. I have other things to do, but good luck to you.
There’s an awful lot of cultural baggage in coding. Many of the concepts that seem essential - powerful text editors, devils tooling - can be completely removed. It requires rethinking from first principles and being willing to upset the Apple cart, but I believe it is possible.
I think that's an honourable but unattainable persuit.
A huge part of the problem is the messiness of the real world and costs.
While there may be a finite set of fundamental problems, the set of possible hardware and software platforms is ever increasing and each has it's own constraints and strengths.
A "universal programming environment" would need something like a "universal hardware interface". That's why things like Java and the Web became so popular despite their poor design.
Also, visual programming is a hard thing to make work and human language skills seem to be far greater than human visual skills.
Perhaps the best that can be done is "starting from zero" and making sure everything, from the ICs in your hardware to the memory models behind your software is thoroughly tested and formally verified.
This is insanely expensive with current technology. Perhaps some fancy math and AI advancements can make formal verification powerful enough for ubiquitous use. Until then, I see little hope for a "universal programming environment".
In addition to the comments below, complexity in software is just as often driven by politics/organizational structure. There is no solution in software to that problem.
This assumes that complex software problems are decomposable into discrete components defined and analyzable solely by their public interfaces. Complex software frequently cannot be modeled in this way -- it is what makes the software "complex". In reality, components have complex interactions far outside of what is captured in the component interfaces (see also: Hyrum's Law). Surfacing these implicit interactions at the component interface level just creates insanely complex interface definitions, moving the complexity somewhere else.
To tie it to something concrete, just about everyone thinks C++ template metaprogramming is unreasonably difficult to reason about even aside from the syntax, but it exists to be extremely powerful at precisely expressing behavioral contracts of a software component that are difficult to do in other languages. Even then, it barely scratches the surface of what would be useful to express for the interaction of those components to be "simple". The number of design parameters of a component that really matter in various contexts are astronomical. No one can deal with reasoning about that many component traits in practice, so the software engineers simply hide most of them -- the complexity is still there and will manifest in unexpected ways because it is not visible at the interface.
All systems engineering is complex for this reason. Any non-trivial software system has to be reasoned about as a monolith at some level to correctly address complexity, which creates an unavoidable cognitive load. This isn't a software engineering problem, it is a systems-thinking problem. In chemical engineering, for example, there are often complex system design problems that cannot be adequately addressed by decomposing them into a sequence of sub-problems, all that does is hide major complexity at the interface of the sub-problems.
There are many embellished, hand wavy statements in this comment.
Could you give real world examples of:
* "In reality, components have complex interactions far outside of what is captured in the component interfaces"
* How template metaprogramming "precisely express[es] behavioral contracts... that are difficult to do in other languages"
I think you contradict your poetry with "Any non-trivial software system has to be reasoned about as a monolith at some level to correctly address complexity". If _any_ non-trivial software system _can_ be reasoned about as a monolith then it is true there exists some model with sufficient simplicity that there are well-defined discrete components.
That's a nice belief, but you've given no rational argument for it. Consider a monolith implemented as a single, unstructured function: 200,000 lines of unstructured code, written by a goto-loving maniac. Demonstrate why some equivalent model with well-defined, discrete components must exist.
You are misunderstanding the problem. Any systems problem can be decomposed into a discrete problems with sufficient fidelity to satisfy a quality requirement. The problem is that the number of discrete components in the model required to provide sufficient fidelity to minimize complexity can frequently number in the millions. No human can reason about that. All you've done is move the abstractions around. Many physical engineering systems have this property; it is in fact a major use of high-end compute resources.
Traditional engineering disciplines that routinely have this problem of non-decomposability actually treat it as a systems problem (or a supercomputing problem), they don't shy away from it simply because the cognitive complexity is difficult. They convert the entire system into a set of simultaneous equations that needs to be solved for. In software, we would call this designing a monolith, but the reason it is done in traditional engineering disciplines is that you can't wish away the fundamental nature of the problem. In software, because it is not a physical engineering discipline, you can pretend that this issue doesn't exist, for a while at least.
Systems complexity is intrinsic, there is no trivial reduction. If there was engineers of all types would serve little purpose. Not coincidentally, the engineers that command the highest salaries are those that have the cognitive capacity to reason about the most complex systems dynamics, the dynamics that can't be decomposed into independent sub-problems before understanding the behavior of the entire system.
Isn’t this the dream since year dot? The problem is the real world is full of leaky abstractions, with different layers evolving at different rates and massive adoption inertia.
I am skeptical of visual programming in general. Some of us just aren't visual thinkers. I find it much easier to conceptualize complex concepts in terms of text and narrative.
Don't worry - I am a visual thinker and I'm skeptical of visual programming. While it's cute for toy examples, I haven't seen it handle complexity well.
I am sure every SE agrees that somehow things are always the same. How about making code pieces algebraic in order to reason and write with them much easier? Like using Haskell?
I have been developing software for over 20 years and leading teams of developers for the last 10. Please allow me to give you some insight from experience.
To your first point, in my experience, the best software, including complex platforms that handle massive amounts of traffic and data, can be built and maintained by small teams. I mean less than 20 developers. The larger the number of developers working on an application or platform has an almost inverse correlation to the speed in which new features are built or bugs fixed.
Software architecture has been done visually since perhaps its inception (tools like UML). At most places I've worked, every new project or large feature is diagrammed visually to use as a guide in breaking down the project into component parts.
In my experience complexity arises, not from lack of tools or industry knowledge, but from 3 main causes:
1. Inexperienced developer asked to create project, who just starts building without planning beforehand.
2. The main one - business demands features built that were never expected or planned for, and built as fast as possible. This causes developers to take shortcuts, make inelegant and difficult to maintain design choices. And leads to often inscrutable code that becomes technical debt especially after the original developer leaves the company. This will always happen as long as software is used to make a business money.
3. High team turnover - I've seen places where developers came and went so often that there was a myriad of things half started and never finished.
How I've solved or helped alleviate these issues: Make the business case to company owners or management that technical debt will be an ever increasing impediment to development velocity and the dev team will need a percentage of work in any given sprint to tackle tech debt issues (as opposed to having everybody work 100% on new features and bug fixes all the time).
I have successfully taken a platform that was bug-ridden, difficult to maintain, and where new feature development had slowed to a crawl due to the over-complexity of the software, to a place of stability and ease of development, simply by allowing our team to chip away at tech debt over the course of several years. Tech debt issues were rated by level of complexity, risk to the business in change (regression bugs), and impact on team velocity. We worked on the highest impact, lowest risk items first and kept going until there wasn't much left on that list.
How many clever computer scientist (including me ) had this idea and vision at least once?
And how many attempts by languages/DSLs/visual languages have already been tried to create?
I believe you are wrong, I'd wish you are right ;)
A very worthy, laudable goal. In addition to some of the other good responses here, my reason for thinking it can’t be solved with better tools alone is that, contrary to what the article suggests, engineering & tools alone are not the primary cause of our complexity, customers and consumers and management are. Business and the need to keep making money is what causes software to need to evolve and change, to support new features that weren’t planned into the architecture, and to develop software in shorter timeframes than is necessary for clean solutions. This is what I’ve seen in a few decades of professional programming, that programmers are perfectly capable of solving hierarchical and cognitively complex problems. Customers or salespeople ask engineering to implement strange inconsistent features, people aren’t happy with your software because your competitor does X, and we’re always walking a balance between time and resources to meet those demands, and there’s never enough of either. A lot of software business, for better or worse, is also driven by fads, so need complexity inducing maintenance just to not feel old and cruddy.
The other reason complexity happens is dependencies and DRY thinking, which is often good, but dependencies, centralization of code, and sharing of code are also a risk. Avoiding repetition is what causes abstractions to grow. Using other people’s libraries means that there are complex interactions you don’t understand. Most of the time it’s all fine, but occasionally sometimes it’s not. This trade is made consciously, for the reason that it’s much faster to develop your app using existing libraries, and not reinvent all wheels, and nearly everyone is doing it. The reason that tooling is unlikely to solve this part of the problem is that failures are so often caused by incorrect expectations - someone using a library hoping that it will do X when it only does Y. Even in theory, better tools could only tell us low-level information about what a library does, but I don’t think it’s possible for tools to tell you at a high level what a library can’t do.
This is essentially Alan Kay/VPRI's old research program. I'm not sure why, but when people get into this they always think dataflow programming will solve everything - we already have Excel and it does do a lot, but it doesn't do everything.
May be you are looking at a much fine grained composition/decomposition tier to arrive at breaking down "all" software engineering towards simplicity, but anything real world usecase would end up being an aggregate so big, your composition/decomposition tier itself would add to complexity rather than reduce it.
BPMN and all of the UI pallet, drag drop coding applications try to do this very thing, this is super successful at smaller scales but breaks the first real world application.
I attended a Common Lisp convention at MIT around the year they did away with the SICP course and replaced it with a course using Python to program robots. It came up in the Q&A of one of the talks and the question got kicked to Sussman who was in the audience and gave an explanation.
He said that it used to possible, practically speaking, to treat software like discrete components so that you can reason about how to combine the units and predict the result, etc. By contrast, much programming today is bodging together a bunch of libraries whose behaviors are often poorly defined or otherwise opaque so that one often enough has to treat them like black boxes against which you have to apply scientific reasoning to understand how to use and integrate it. Hence Python and robots.
Presumably the new course teaches skills around managing this complexity, but the shift in model advocated -- from the metaphor of well-behaved discrete components in composition to opaque units whose inputs and outputs (possibly depending on hidden state) must be discovered was interesting to me, and ultimately adds to the accidental complexity of doing software now.
Do you think this sort of problem may have a distinct visual representation that would fundamentally reduce the complexity of library-dominated development? Or are you imagining a whole programming ecosystem built around your idea, so that the fundamental problem Sussmen described is somehow obviated? Or are you really trying to address essential, not accidental complexity?
> there are finite and fundamental classes of problems, that underpin the vast majority of problems
We already have had, for a long time, a 'universal' model for a specific slice of information processing that consists of a very small set of components: Graphic User Interfaces.
One problem is the diversity of domain semantics and all the associated nuances. In GUI programming, this is the tedious bit of naming the components and mapping them to processing elements.
Another problem is engineering culture (du jour). A reductionist approach that attempts to leverage structural commonalities requires what is poo poo'd as "boiler plate" and all the associated warts of component oriented programming (including factory-factories). A revisionist shift in mindset is required.
Software complexity in most cases is a direct reflection of business complexity. Changing requirements, overly complex processes, “approval” steps, etc., etc. if you can build it faster and cheaper - great! But you will end up in the same place eventually.
His examples are getting a little dated, but reading this, I can’t help but think of Spolskey’s “Architecture Astronauts” essay:
“When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. And when you think about it, calling a method on an object is like sending a message to an object! It’s the same thing again! Those are all sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more. Blah.“
You're describing Mathworks Simulink, more or less. And, I also think it's an incredible shame that it's not a more commonly used tool for forms of software development outside of basically controls engineering.
While I am a big critic of current software development tools and believe they are nowhere near what they could be, I also believe that there will never be anything that makes software development easy. The reason is that software is a tri-component system: how to do something via computation, how to communicate between people, and how to think about something. We are actually somewhat decent at the first. We are absolutely terrible at the last two. Again, I think there's a lot we can do to improve things, but we will always be pushing against the ceiling of humans to communicate, collaborate, and accurately encode a domain of knowledge.
If one thinks of software programs as stored knowledge, then it makes sense why they are so buggy, error prone, etc. It's that the knowledge was never complete to begin with. Throw in the difficulties of computation, and you have the mess that is the software landscape.
I definitely share this belief. When I started delving into functional programming I was immediately struck by how whole classes of problems just disappeared once people found some useful abstractions. Programmes that took thousands of lines of code to express could be written in hundreds or even tens.
But FP seems a bit stuck nowadays (to my untrained eye) with the breakthroughs from the greybeards of the last decade.
Do you have any progress thus far? Anything people can contribute to?
The problem with software engineering is not technical complexity. The problem with software engineering is the complexity of humans and the real world. Have you ever worked on software that needs to encode 30+ years of government and employee/union agreements into rules that can be used to automatically decide how much a specific employee needs to be paid overtime in a given month? If you had you would know that no amount of tooling or software will make that problem simple.
Out of curiosity, have you watched/read Alan Kay's team work at VPRI ? They managed to make a kind of desktop OS in less than 1MLoC, tangling DSLs to generate the boilerplate from very succint rules (or parsing RFC specs for the network stack). They wanted a 100kLoC but couldn't do it, but I believe it's still doable.
Our industry's "solution" to complexity is somewhat unique: We sweep it under the carpet. The reason that actually works is that the marginal cost to reproduce an existing pile of complexity is zero.
Once we've got a sufficiently complex pile that actually has promise, everybody duplicates the pile. And then puts an additional layer on top. These many pile+layer combos then co-evolve (and co-erode ;)
At some point, something so useful has evolved that we start copying that pile... and put new layers on top. Rinse. Repeat.
The complexity rarely collapses because those new layers can add support beams, if necessary. Worst case, a layer gets ablated, and we try again from one step below. We don't ever go back to the beginning.
(This also explains, to me, why Enterprise software is what it is - the complexity piles can rarely be shared or reused, and so there's no co-evolution, little chance to settle on the best pile)
And, to be fair, it's similar in other disciplines, but maybe a bit slower. Mechanical engineering has individual components, larger components built from those, and so on. It just is more expensive to copy the piles^W components we know to work.
Total collapse is exceedingly rare.
(That even holds for human societies - they, too, often "just" collapse down to simpler societal expressions. And we do co-evolve larger and larger forms of society. It's just that the erode-and-try-again process is uniquely painful for the humans involved)
Note that Google's "Frequent rewrites" practice -- see section 2.11 of https://arxiv.org/pdf/1702.01715.pdf -- substantially reduces the severity of this problem (though it requires a good higher-level architecture).
> Most software at Google gets rewritten every few years.
> This may seem incredibly costly. Indeed, it does consume a large fraction of Google’s resources. However, it also has some crucial benefits that are key to Google’s agility and long-term success. In a period of a few years, it is typical for the requirements for a product to change significantly, as the software environment and other technology around it change, and as changes in technology or in the marketplace affect user needs, desires, and expectations. Software that is a few years old was designed around an older set of requirements and is typically not designed in a way that is optimal for current requirements. Furthermore, it has typically accumulated a lot of complexity. Rewriting code cuts away all the unnecessary accumulated complexity that was addressing requirements which are no longer so important. In addition, rewriting code is a way of transferring knowledge and a sense of ownership to newer team members. This sense of ownership is crucial for productivity: engineers naturally put more effort into developing features and fixing problems in code that they feel is “theirs”. Frequent rewrites also encourage mobility of engineers between different projects which helps to encourage cross-pollination of ideas. Frequent rewrites also help to ensure that code is written using modern technology and methodology.
> One thing working in complexity’s favor, though, is that engineers like complexity. Admit it: as much as we complain about other people’s complexity, we love our own. We love sitting around and dreaming up new architectural diagrams that can comfortably sit inside our own heads
Where does this yutz work? Because he doesn't speak for me. Generally, I write a complex program because I don't have time to write a simpler one; and when I have to deal with that complexity later I'm like arrrggh. Rube Goldberg machines are fun to watch, but no one wants to maintain, change, or build on one.
didn't Mark Twain say something like "i'm sorry this letter is so long, i didn't have time to write a shorter one". I feel like that with some of my code at times.
Complexity in software is a lot like complexity in cuisine; most chefs want to stretch their abilities and create something memorable, transcendent even, but at the end of the day unless they’re cooking for themselves or friends and family there’s a business to be run.
So complexity has a diminishing return for economical reasons.
An interesting problem is whether the complexity is unavoidable and if it is, we need to teach migration and proxying in Software Engineering 101.
We have an old app: dotnet webforms, it's clearly hanging on for its life and needs replacing but the mythical rewrite is impossible if you intend to rebuild and switch, it just involves too much time away from your core product.
Instead, we had to setup a proxy which could send all traffic between one of the two apps so we can move things a few at a time to the new system and all new work just gets built in the new system. It still takes a long time so I am now wondering whether as soon as we finish, we should immediately setup another new app alongside the current new one and repeat indefinitely.
It certainly wasn't easy to do because old and new frameworks are rarely able to share cookie formats/sessions etc so some hand-cranking was required.
> It takes a lot of discipline to resist complexity, to say “no” to new boxes and arrows. To say, “No, we won’t solve that problem, because that will just introduce 10 new problems that we haven’t imagined yet.” Or to say, “Let’s go with a much simpler design, even if it seems amateurish, because at least we can understand it.” Or to just say, “Let’s do less instead of more.”
Discipline and complexity are best friends. Laziness and procrastinating seem more valuable for avoiding complexity.
Probably many huge codebases could be reduced to a handfull of good old SQL queries and nothing more...
If you trash things like: framework boilerplate, complex graphical UI libs, devops complications, auto-testing ecc....
I highly recommend Rich Hickey's "Simple Made Easy" talk as a way to think about software complexity. We all know that complex systems are hard to work with, but I've found his presentation very useful in understanding the nature of complexity. Understanding complexity helps being able to avoid or minimize it. For example, easy is not the same as simple, and simple changes to one part of a system can introduce great complexity at the macro level.
A for loop is easy: everyone can conceptualize doing a process N times.
A map is not as easy of a concept in some ways (it's more abstract, you are taking N elements of X to N elements of Y). But a map is simpler than a for loop, because there is no state between steps: it's completely symmetrical.
The for-loop "complects" each iteration together, even if it's as little as incrementing an integer. It's a side effect, however trivial.
To kick it up a notch, now imagine a 2-D loop. Now we have 2 counters. If it's a 2-D array, each addressing operation interacts with 2 counters. But map just needs a second map chained on, and each map is completely "unaware" of the other map. You could map over N dimensions and each map is symmetrical, while the for loop needs an ever increasing number of counters. This slight difference in complexity shows how complexity compounds with other complexity.
Let's say you have a function that does a lot of different things, and you just call:
doStuff()
That's easy, right?
But "doStuff()" may be a very complicated function, that's very hard to understand, so it's not simple.
A real-world example might be something like the type conversion that JavaScript and PHP do in comparisons; "just do 0 == '0' and it'll work, easy right?", but turns out those conversion rules aren't simple at all.
I like how he states that simpler software doesn't necessarily mean fewer components (roughly quoting). Where the article argues for fewer boxes and arrows. It's hard to argue specifics, because the article isn't getting into them, but sometimes more boxes and arrows makes things simpler. If you don't have multiple boxes, what do you have? One box of spaghetti?
I also disagree with the article that engineers like complexity. I've maybe known a couple, but usually they are really junior, or really unpopular.
Well, if you compare react and angular2+, react has less stuff, and is less complex. But on the opposite side, a function that tries to do too much (like lodash pick function) look simple at first glance, but is actually complex.
The more abstract the mental model is for building the thing—at the tool/code level—the more likely it is that the thing being built will be a difficult to maintain mess, if not an absolute "burn it all down and start from scratch" failure.
Fix the tools, and you start to fix the code. Unnecessary terminology, confusing APIs, superfluous patterns...it's like watching someone blindfolded in a dark room trying to justify the high price tag on their computer science degree.
After years of running my mouth, I proved my rants about the unnecessary complexity by building my own JS framework [1] and I'll never look back. The emperor is bare-ass naked.
Among other things, it suggests that there are some incentives for complexity: That engineers just enjoy complexity, and that making simple designs might not be good for the respect of your peers/bosses/career: "A simple design might make it seem like you’re not really doing your job."
While that's true, I think the essay isn't clear enough about the fact that simplicity is just plain hard. Complexity is sort of the "natural" outcome of "adding things". It takes time, effort, and skill to create software that can do what is asked for it while maintaining an internal simplicity. Even if everyone involved really wants it. I do not think software complexity comes only, or mainlly, from people not wanting it enough (explicitly or implicity), from incentives to complexity. It's just plain hard. It's a mistake to think just wanting it harder (or having the right incentives) is sufficient.
It may be the key challenge in software engineering -- how do we make software that does everything that's asked for it (powerful, complex things), while maintaining an internal simplicity? You can not accomplish it by cargo-culting any "design patterns" or other approaches. It's still a skilled craft. That comes from years of experience with the intent of developing this skill (years of experience alone aren't enough), looking at other people's code who are also trying to do it, and just some aptitude. And domain knowledge, if you understand the nature of the problems you are working on better, you can design simple solutions for them better--contrary to some current practices treating developers as interchangeable.
Yes, you need your organization to allow you to do it (including taking the more time it will take), but that's necessary but not sufficient for the outcome.
There is the well-worn Pascal quote about having no time to write a shorter letter[1]. There is also, I think, an important point hidden in the “domain knowledge” part: the freedom to change the spec; the effusive texts on Charles Moore’s approach to Forth programming[2,3], however much scepticism they deserve otherwise, are very direct about that freedom being essential.
But that means simplicity is a cross-cutting concern, like security, and does not particularly fit into any environment that would attempt to separate design and implementation. It needs “redraw all the module boundaries” to be a plausible (if improbable) response to “this one 300-line part of this one module is awkward and three times longer than it really needs to be”. Aside from the obvious administrative problems, suggests that the complete edifice needs to fit into the mind of a couple of people at most or the iteration simply will not converge most of the time.
[3] There was an example about changing a negative-feedback control loop, written into the spec by hardware engineers emulating analog circuitry, into a couple of fixed-point operations that behaved differently but did the actual job just as well; I can’t find the reference right now but this a minor domain-specific change among the kind I have in mind.
> Aside from the obvious administrative problems, suggests that the complete edifice needs to fit into the mind of a couple of people at most or the iteration simply will not converge most of the time.
> I don’t see a good solution to this.
I think this is probably true.
Some people take that to basically give up, and say, okay, our software is always going to be a mess, but look at all it's done as a mess, so be it.
Otherwise... the commonly understood solution seems to be composing the thing of independent units (very very very decoupled) each of which can fit into the mind of a couple of people at most. And then the whole, built of these independent units considered as black boxes, can perhaps also fit into the mind of a couple of people at most. I know you said "the complete edifice", which is not the same thing, but this is what we've got: and indeed is "abstraction", basically the entire basis of computer science and what makes possible anything we do. (Even the simplest program of 20 years ago -- can probably only fit into the mind of one or two people if you exclude the hardware all the way down, which you can because it is well abstracted and decoupled; it wasn't always, 50 years ago).
And we can come up with all the challenges and barriers of that -- of course! If it were easy, we'd have simple software. :)
But that's the task, I guess. But a lot of software isn't even built realizing that's the task -- to have one or two people "driving the bus" who have a mental model of the thing they're building, at any level of abstraction (I feel like Fred Brooks wrote about this), meaning that continuity of experience matters and domain knowledge matters, treating developers as commodities to be shifted around continually has a cost to complexity too, as does continually switching technologies and platforms so you can't build up a solid repertoire of encapsulated abstracted pieces to compose.
> engineers like complexity. Admit it: as much as we complain about other people’s complexity, we love our own
No. Speak for yourself. I am happiest when my solution to a problem is simplest. I am least happy when a solution requires introducing new infrastructure or major new ideas.
I feel the same way, but we both have to accept that we're tilting at windmills. Complexity can't be measured, so there's no way to demonstrate, much less prove, that the time you spent reducing complexity was time well spent. You can demonstrate a run time or memory performance improvement. You can demonstrate a new feature. Unless you can reduce complexity in zero time and do so without introducing any unintentional side effects/bugs, you're just hurting yourself when you try.
I have no idea who first said it, but one of my favorite sayings is: "Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands."
I strongly believe that this applies to computers as well. I love it when a system neatly solves its requirements in a way that looks effortless. When an algorithm looks like it just barely works for all the potential edge cases without any special handling because those edge cases just miraculously happen to work out as they fall through the happy path just like any other case. That's when a solution "feels" right.
If I can present my solution/code/explanation to a junior engineer, a C.S. student, or even a non-technical person and they get it, then that's when I feel like I've really done my job. It's music to my ears if I can make a problem look so easy that someone asks "Is that it all it takes?" (until they've tried to solve it on their own :-)
Very nicely written. A lot of gems in this short essay. What stood
out for me was:
> It takes a lot of discipline to resist complexity
That can be taken in many ways.
Self-discipline, to resist convenience, the omniscient and omnipotent
desire to control everything and always have the right answer
immediately. That takes discipline to let go of. Also to discipline
others, to say no, push back, become the master and set firm
boundaries against crushing 'demands' for complexity. And intellectual
discipline, to see clearly, elegantly, eliminating what is
unnecessary. Over-complexity is a failure of the ordering principle
of an organism, a step on the road to chaos and entropy.
It's based on my 20+ years, 50 different teams and many hundreds, perhaps thousands of encounters with software developers.
It's almost never the domain complexity that is high.
Sending a rocket to Mars is complex. And perhaps complex every time.
Dealing with people that has to work together can only be simplified when agreements are made and agreements are kept. Otherwise you cannot trust and have faith in people.
It comes down to far and near organization leadership, mandate and empowerment in small teams with few communication links outwards, clear expectations from whomever is asking for something and process management.
Writing code is rarely hard if you know the expectations and the boundaries of the task.
Empowerment gives the team the possibility to act on what they discover and encounter.
Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
One thing is complexity in code. Another is complexity in business. Some things are not made for software to solve.
Less experienced people are generally guided by their beliefs around their own ideas. If the mind is locked onto something it's hard not to thing it's a good idea. Usually in software there is simpler approach. Look for it
I can go on because I help companies and teams for a living. But I will stop here
> Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
NO!
This mentality is short sighted by definition and leads to stagnation. It may work within large financially healthy companies that aren't trying to rock to the boat.
Making assumptions about the future is incredibly important for us to move forward. But with these caveats:
1) Acknowledge that you are taking a risk and that you may be wrong.
2) Validate your assumptions as quickly and as often as you can.
3) Be prepared to change course whenever your assumptions are invalidated.
> Less experienced people are generally guided by their beliefs around their own ideas. If the mind is locked onto something it's hard not to thing it's a good idea. Usually in software there is simpler approach.
Less experienced people are frequently the driving force behind new ideas precisely because of their belief in it. As experienced people we should be encouraging them to carry forward responsibly instead of projecting our aged and jaded perspectives.
> I can go on because I help companies and teams for a living.
Honestly, that's a weird way to phrase what you do.
I think you and the parent are talking about two different things. Building products that are looking to the future is valuable, if risky.
But I believe the parent comment was talking about software abstractions. i.e., don't build abstractions because you might need them later. Instead, build the software in the simplest possible way to solve the problems it needs to handle now.
This is because you'll likely be wrong about your assumptions about which abstractions you'll need, and ripping those abstractions out is going to be more expensive than just modifying simple code to do new things.
We're talking about the same thing. Perhaps I was just speaking more broadly.
> don't build abstractions because you might need them later ... [snip] ... This is because you'll likely be wrong about your assumptions about which abstractions you'll need, and ripping those abstractions out is going to be more expensive than just modifying simple code to do new things.
At worst, this is dangerously wrong. At best, it creates a straw man that people can use as a tool to shut down innovative ideas.
What you're describing is a very narrow/specific strategy that one might apply at a company. An example would be when a company is doing well financially and you don't need to risk amassing technical debt for some experimental side projects. Just build the MVPs, see what sticks, and only then decide whether or not to commit serious architecture to the problem.
If you're designing new systems that need to scale quickly (just one example) then you NEED to place bets on abstractions. A well designed abstraction could save you weeks (even months!) of time when you most need it.
I've seen what happens when a successful product is comprised of stitched together MVPs. It's too late for a rewrite and all of that "simple code" is creating a death-by-a-thousand-cuts nightmare.
Bad abstractions will always be bad. Bad system design will always be bad.
It's an aged/jaded mentality to claim that we should just stop designing systems ahead of time because we suck at it. Not everyone does. Some people are actually very good at it. It's possible to design an abstraction defensively so that it isn't more expensive to rip it out later.
Want a concrete example? Payment systems. If I'm writing a billing system then you bet I'm going to abstract away the interface with the payment processor. I would do this irrespective of our plans to expand to other payment processors. It shouldn't even be a debate because there's a way to build that abstraction with minimal overhead to the team.
On the other hand I would never blindly introduce an ORM into a system that doesn't need it. That would be a bad abstraction.
> A well designed abstraction could save you weeks (even months!) of time when you most need it.
Or it could cost you a lot if your guesses are wrong. It goes both ways.
It's better to write code the simplest way to do what it needs to do now, but in a way that can be pulled out into a new system/abstraction later in an easy way. This is generally done by writing data-oriented code (focus on what the data is and how it needs to be transformed) rather than object-oriented code.
The problem is that everyone's definition of "simple" and "in a way that can be pulled out" is different. And in any case... just about everything you write is an abstraction to some degree; the question is, where is the line?
It's not hard to write abstractions defensively. For example you can limit access to your abstraction to a small and strict interface. What you do behind that interface can be the wild west; as long as you can just lop it off later.
> It goes both ways.
It does, but the word "simple" is often used in a deceptive way. A cacophony of "simple" systems are actually very complex in aggregate.
There are many cases (eg: something is your core business) where you'd want to put a lot of thought into an abstraction ahead of time. You'd then want to validate that abstraction's effectiveness and adjust as you go.
I totally agree on validating ideas but that's IMO not looking in to the future. It's trying to find a fit for something you believe is a good idea. There is a big difference in testing whether something is needed in the near future or you simply stick to your guns and blindly dictate how you believe the future will look like.
Your ORM example is good and exactly what i am referring to. Too much abstraction/gold plating due to unknown requirements in the future is perhaps a better way to put what is a problem.
And you also say that "If you're designing new systems that need to scale quickly..." - if you know that it will scale quickly then of course you know you need to do certain implementations to cater that. But if you do not know whether it will scale quickly then you are looking into an unknown future - and then you should step back on abstractions.
> There is a big difference in testing whether something is needed in the near future or you simply stick to your guns and blindly dictate how you believe the future will look like.
There are innumerable successful examples of committing blindly toward a future few people believe in. These are often cases where the objective evidence of success is very limited. If the objective evidence existed then "everyone would be doing it".
This is the crux of my disagreement. Of course we'll both agree that applying the wrong/stupid abstraction is wrong. But that's not what I'm discussing.
I'm talking about this quote specifically:
> Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
Perhaps you need to reframe this point to make it more clear?
> Don't look into the future. Because you cannot.
This is silly for all the aforementioned reasons. Especially when innovating in tech!
> And you are both naive and arrogant if you believe you can.
Calling someone arrogant for attempting to predict the future in technology. Come on, man.
> Deal with what is known now!
What is known now is what everyone else knows now. Unless you're dealing with the _extremely_ rare case where you have access to information no one else does, innovation requires one to commit to the unknown.
I'm certainly not claiming you should never design anything up front. And I'm also not saying you should never design abstractions. As with everything, the devil is in the details.
What I am trying to get at is that I've seen many times where a complex abstraction was introduced, and then it was only ever used for a single implementation of something. I've also seen many times where an abstraction was introduced, to make plugging in other backends easy, but then when it came time to plugin the second backend, the API was sufficiently different that it couldn't be shoehorned into the abstraction without a ton of extra work.
Sometimes you know enough about the problem space up front to avoid those problems, but often you don't. I often the better approach is to keep things as small as possible until you need more.
E.g., for payment processors, just ensuring that the surface area between payment processing and the rest of the code is as small as possible is likely sufficient. Maybe that's done through a simple abstraction, maybe not. But the point is, if you only have one payment processor now, the cost of implementing the abstraction doesn't change between now and when you actually need it (unless, of course, you chose a bad design, which we're obviously trying to avoid with either approach). So you can pay the cost of adding the abstraction when you actually need it, and can spend your time on something else that you need now.
I guess maybe bringing up the word "abstraction" was the wrong way to word it. In my experience, it's usually better to keep the code as small as possible early on, regardless of whether or not its using abstractions, because making it do what you want later is much easier when there is less code (in terms of constructs/concepts, not necessarily lines/characters). Also, try to keep things as orthogonal as possible. I think the reason I mentioned abstractions is that complex abstractions trying to predict all of the possible future extensions to the system is where I most often see design go wrong.
I think you and I are generally on the same page, but we're reading this statement by the root commenter very differently:
> Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
See my additional response to him elsewhere in the thread where I tried to break it down sentence by sentence.
Everything you're saying is on point for good system design IMO. What he was saying was totally different and IMO is a "dangerous" way of thinking when innovation is important.
I seem to be in drastic opposition in that I hate complexity, and will do anything to avoid it. I shudder with horror when I look at the teetering stacks all over the place these days.
303 comments
[ 3.1 ms ] story [ 275 ms ] threadSo different, but not less complex overall. But maybe there is value in having some of that complexity consistent across a small number of cloud providers.
No free lunches here.
It's true that a lot of organizations had operational teams that have become money pits, or pushed back on all quality of life improvements because they don't have the talent, the budget, or the imagination to pull it off. When we move to the cloud we start aspiring to these things we didn't have because they were expensive. I don't know of anyone who moved to the cloud and didn't move the goalposts. We were just talking in another thread, as we often do here, about how much YAGNI is going on out there. Yes, the price per feature goes down, but the overall price doesn't seem to. And I get stuck taking care of things someone else used to worry about, which is opportunity cost on top of it all.
What is also true is that developers can learn a lot from their operational peers and avoid expensive mistakes. With cloud we have none of those peers. We have to learn everything first hand. By someone who is all too happy to let us wrap rope around our necks and then 'rescue' us from themselves. That's a perverse incentive and quite a setup for a fairly fucked up codependent relationship. At least with interdepartmental drama some of the money stays in the company longer before going to vendors.
In pre-cloud Internet times, you'd have an untold number of extremely brittle bash scripts, cron jobs, rsync ssh key setup, fleets of build + test boxes to manually worry about disk space, pre-provisioned dev/QA database servers with also untold brittle sql startup/teardown scripts, and all of the requisite people whose job it was to solely maintain this infrastructure along with database tuning, build fleet monitoring, the list of menial tasks just goes on and on and on.
Today, you have a yaml file in your .github/workflows directory.
Now I agree that there are "different" requirements. Understanding the complexity of your workflows etc is no small feat-- but you're replacing such a huge amount of what used to be extremely expensive and brittle architecture with, basically, a text file or two. That's a huge cost savings.
In the build and test example, the answer to "how much compute is running?" is based off developer velocity and so "it depends" is a fair answer. When nobody is shipping any builds, your cost should be $0. (I've found that this is kind of hard for enterprise-y finance departments to wrap their heads around and is why all those esoteric billing notifications AWS services even exist)
Pre-cloud services days, finance departments had a much easier time. You had racks of physical boxes that had static costs attached to them, you had a static monthly bandwidth bill that let you run at a certain speed, and you had salary costs which are also pretty static month-to-month. The idea of "scale to 0" was completely unheard of. What do you mean your QA environment doesn't cost anything on the weekends when nobody's doing anything? etc etc etc, you get my point.
In pre-cloud times, you would spend untold piles of money spinning up racks of storage arrays, switches, firewalls, leasing out gigantic pipes for bandwidth, and again all of the requisite people in order to get that running. You'd rent space in multiple distributed global datacenters, so again you'd have an unreal amount of bash-script-file-sync services so that when someone uploads a new image it gets replicated all over the globe. Millions and millions of dollars. You'd probably have to write a custom resizing service with ImageMagick, hooked into your frontend, so that you were serving customers the correct size and not blowing through your bandwidth allocation. Just incredibly complex.
Today, you click a button in your CDN provider's console; most of that functionality from above just comes for free.
Again you have to do a little bit of munging your front-end to take into consideration the vagaries of your CDN provider, but overall it's such a huge savings of mental energy and time. Put a circa-1999 systems architect in front of the Cloudflare console from today and they wouldn't believe it was real.
> One thing working in complexity’s favor, though, is that engineers like complexity. Admit it: as much as we complain about other people’s complexity, we love our own. We love sitting around and dreaming up new architectural diagrams that can comfortably sit inside our own heads
It's harder to acquire a sense of mastery & possession, of real understanding, when there's been multiple iterations of teams hacking on a piece of software, when it's many layered & has lost cohesion & concerted intentionality.
Trying to describe how relishable, how enjoyable it can be to explore, to search & quest for meaning, to push designs that hopefully make sense, that hopefully grow, that hopefully wrangle: it's an under-sung & hard to tell story. It's interesting to me having such a big part of the world running off of a knowledge-working that is so under-described, so hard to communicate personally about.
So, it'd be funny to see, but I don't think it's happening anytime soon.
made me literally LOL. That turn of phrase is gold.
> ...Opinions expressed in this blog are mine *and frequently wrong*.
Emphasis added.
Equating the complexities of software with the complexities of societies (the premise of this post) is a fun and provactive blog post. Which is all this is. I was really hoping for a serious treatment that meaningfully dug into this, but that's my problem.
But not because some esoteric systemic property of complex systems, but just because working projects get deprived of maintainers and eventually there's nobody that understands WTF original programmers intended and how to fix the holes.
So incomprehensible portions of the system are circumvented or replaced with something that just works. Actually the same happens in societies, unless some group sabotages the workaround.
https://sudonull.com/post/111636-IBM-System-360-A-Failing-St...
I think we've seen this with the rise and fall of hardware vendors. Most recently with the Mobile OS ecosystem (symbian, windows mobile)
The causal factors behind why societies fail and why software projects fail are completely different. Complexity is a description of the system and a very bad measure to draw an equivalence between the two.
The risk with drawing this equivalence is it grants license to software engineers to think that similar enough models apply to "managing software" as "managing societies". Because when all you have is a hammer... But society is not a nail.
What, you think "microservices" is a new invention? We called it distributed systems, and we really knew not to go there unless we wanted to decimate our productivity and sleep.
For example, I look at our software running on .net MVC and think as an Engineer, it simple and knowable but the Front-End Team are less worried about simplicity and more on flashy front-end stuff since that is their job. We end up bolting on a Front-end JS framework and complexity immediately ramps up by like 300%.
Are they wrong for wanting a better and more flexible front-end? Not necessarily, I mean all the other companies have cool stuff and if we don't maybe our company dies.
Ditto for lots of other examples...
I don't know why programmers think this way. We don't expect everyone to be a doctor, a botanist, a novelist, or a musician but for some reason we think anyone can be a programmer. That's just not the case -- programming is a skill like any other -- it takes some natural inclination, some training, and a lot of practice. Just like any other skill. A programming language is to a programmer what an instrument is to a musician.
Anyways, there's a reason programming should be made more widely accessible - it develops thinking skills and rationality when you learn it, as Mindstorms pointed out. Between the anecdotes in the book and things like [0], I think it's a travesty that we aren't pursuing this to its fullest extent, and instead try to teach children Python or JavaScript, two decent languages for software development but not exactly forgiving with beginners.
[0] https://medium.com/@stevekrouse/goodbye-seymour-cb712757264f
And like with these other things, you can make getting into them easier. With music, children start with simple instruments that no professional ever uses. And it's the same with programming, there are plenty of easier environments for children. But if you want to make programmers and you want to make musicians, eventually they have to use the real thing. My own son jumped straight into the deep end of Unity development knowing nothing because he wants to build something real. I neither encouraged or discouraged that environment and it's pretty unforgiving.
I don't think the solvable problems you speak of as are as solvable as you think they are. Also making software development out to be special both in terms of it's benefit to thinking skills and rationality and how it's merely some tools away from being professionally approachable to the masses is totally unfounded.
Let's say your software is in a financial company. Their software has to enable them to follow all the government financial regulations. Well, the government is following the larger societal "complexify to the point of collapse", and the financial regulations are certainly doing so. That means that the external behavior (the "business logic") of the software is insanely complex. You can't make that go away just by visual programming.
But maybe you're not in the financial world. Maybe you're just writing programs for internal corporate processes at some generic company. Well, your software is still subject to the complexity that builds up in the company processes. Again, the programmers can't eliminate that complexity.
Or maybe you're writing a customer-facing app - an external-facing web app, or an application that people actually install on their machines. Here you're at the mercy of the product or project manager trying to find new things for the app to do, and they still complexify the app to the point of collapse.
The problem isn't that programming is too complicated. The problem is that what we want programs to do is too complicated. Visual programming can't save us from that.
Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that convention. Storing both risks the data going out of sync. What's the "right" way to store person names? There doesn't seem to be a simple solution.
Another example: we model physical cables (both for power grid and for data transmission) in our CMDB. All works fine, until you suddenly have a Y-shaped cable with three connectors that doesn't fit into your data model. The real world always has these 1% of cases that don't fit the general pattern; if you focus on the 99%, the 1% make trouble. If you focus on modeling every case, you have 10x the complexity, even for the simple case.
And then there are things that are moderately complex and security critical, like password recovery workflows. We haven't really found a way to reuse these among different technologies. Like, if you once figured out the perfect password reset workflow with Ruby on Rails, and your next job uses Python + Django, you're back to square one.
If somebody has a good idea for how to tackle these problems, please let me know!
But where software does fail in my humble opinion is making it easy to pull in tried and true tested solutions to the problems that we do face, even if they are not as common. Because even though they may not seem common, I'm absolutely certain many face the same scenario.
The amount of duplication solving the same problems is insane. But this is not an easy problem to solve and I don't intend to trivialize it.
I think we need to come up with better tools. code sharing through libraries/repositories (like npm) is great, but it can't be the final solution.
Back in the day in Haskell I dreamed of a system where you could type out a type signature and a fully tested rated implementation would be imported from an "open source" service. You could import modules, functions, data structures, anything. But that vision is still a long ways off.
I think of that as optimizing the entire stack for what the customer wants of rapid/high-speed analytics.
The type signature of GPT-3 is `string -> string`
But if you look at like co-pilot for example, if that was given the ability to have type signatures serve as input you might get a lot more powerful results than what it does with raw text (which is very impressive).
But this comes down to type signature design. You can encode any function using simple types like int -> int which aren't very useful. Where Haskell shines is when using types to limit the scope of inputs & outputs. What I am getting at is that you can still write uninformative type signatures in Haskell, but it also gives you the power to write more informative ones.
I don't think Haskell is the answer, so please don't take that as what I am saying. I do think however using richer type systems could be a stepping stone towards a solution to this problem.
I feel like it wouldn't work if you could actually mathematically constrain the outputs to be syntactically correct either. That's probably one of those Gödel things.
Type signatures don’t tell you which should be the “then” clause vs the “else” clause in any conditional.
Don't try to force schemas onto schema-less data. Store the "name" as a JSON string/blob representing the various possible attributes (given, middle, family, title, etc) and provide a variety of functions for representing that data. IF you really need to do this at all (for an internal app, you probably don't).
> Physical links
Include an Hardware Asset FK in your Link M2M table. Model each binary link explicitly, so a Y cable = 2-3 different Links that point to the same cable Asset. Or you can have single Links with M2M inputs/outputs. But definitely don't model Y cables explicitly.
> Password recovery
What is so complicated about this, specifically? Python and Ruby are completely different languages with no guarantees for interop.
If you can't do that, you have more pressing issues.
Where are you getting "a dozen assumptions" from?
Our cables have stickers with cable IDs, which are stored in the CMDB (and can be used to trace the endpoints through patch fields, for example).
So if you model a Y cable as two physical cables, you have to both allow duplicate cable IDs AND multiple cables per port. Both of these have significant downsides in preventing data entry errors.
No. Model it as two abstract "Physical Links" or "Connections" which FK to the same physical cable Asset, which is uniquely determined by its inventory ID and/or manufacturer model+serial.
If a Link needs to know whether its part of a Y cable Asset, it can join the Asset table.
If a Y cable Asset needs to know what it's connected to, it can join the Link table.
Additionally, you have a nice clean Link table for doing graph traversal (get me everything connected to this impacted Asset by degree <= 2), and a nice clean Asset table for inventory tracking.
C and C++ libraries with light language-specific wrappers largely serve this purpose, in practice. It's plausible that, say, a PHP postgres client lib and a Node postgres client lib will share nearly all their code—probably as C or C++.
This doesn't get leveraged much aside from interfacing with daemons and sometimes for extremely complex e.g. media or crypto libraries, though. No-one's doing this for high-level software workflow building-blocks, like a password reset flow.
[EDIT] as for the "why", I suspect it's because the things it's used for are far simpler than the things it isn't. A password reset flow can potentially need to interface with lots of different things, some of which may be custom to the project, some of which may vary with run-time input, et c. If you cut it down to only the parts that could truly be re-used anywhere, with all kinds of points where you can hook in as needed... you've only made about 5% of the work re-usable, so it's just about pointless.
The Common Language Environment on VMS, or its lesser(but infinitely more available hence virtuous by effect) imitation, dot net CLR . FFIs, if you bought a good environment. Sounds like if open source implementation of good foreign interfaces existed, the OP's problem wouldn't exist, let alone system level interoperability.
Store both a unified name field and separated given name / surname fields and let the user manage both. Yeah, this risks going out of sync, but that's the user's problem, not yours. Yeah, three fields are technically more complex than one or two, but it produces less complexity down the line.
Falsehoods programmers believe about name: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
Then they leave that field empty.
https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
As a practical matter, the HL7 FHIR data model for names can work well enough for the vast majority of applications in almost any country.
https://www.hl7.org/fhir/datatypes.html#HumanName
For how to model physical objects like those cables, go see how Grainger does it as a working practical model.
To really handle names properly, you need more context than the name in the presentation layer that many schemas take their modeling from can obtain. Government health care or similar widely-adopted encoding is sometimes Good Enough. If you want non-lossy exactitude however, then that's a much bigger scope (I'd be investigating a first-pass classifier with contextual hints taken from various geolocations, age, etc., that implies soliciting the name comes after what you normally solicit for input, and refining from there).
Password reset; this is why vendors like Okta exist to abstract authN away for us, and auth0 for authZ. Then there is the rabbit hole of what this abstraction leads to...
[1] https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
[2] https://github.com/kdeldycke/awesome-falsehood
The way past bureaucracies that didn't have the luxury of unlimited cheap complexity did: you set some rules and anyone who doesn't like them can either suck it up and follow them, or deal with the consequences themselves. Allowing users to specify arbitrarily complex requirements and never saying "no" is what gets us into this problem.
I try to work on the side in a spiritual successor of Foxpro/DBase (https://tablam.org).
I consider a mix of relational/array model fill a lot of ergonomics for this (and it was proved to be right by the family of DBase langs).
But what makes this much more complex today is the explosion on OS targets (Windows, Linux, MacOs, Android, iOS, Web), and the requirement to integrate with many other stuff, dealing with many formats (json, xml, ...), is harder to do UIs now and the base support is more inconsistent and mixed as ever...
So, is possible to make a simpler tool, I certain of it, but then the developer/user will say "ah, ok, so how this connect to Redis, GraphQL and Amazon Web Services, run this on Android, Windows, parse CSV, ..."
and that is what make this very hard at the end...
The fundamental problem of simplifying software is humans. Just consider date formats, time zones, and tax codes. Humans love to make complicated things.
My philosophy on this has been to take the complex human stuff and stick it in a black box. A professional feather in cap with this approach is called BladeRunner ( https://dl.acm.org/doi/10.1145/3477132.3483572 ) which radically simplified the distributed system aspect by putting all the gnarly glue and business logic in a V8 VM (JavaScript).
My next thing is a focus on board games where I have invented a programming language and have started to evolve a platform. It's called Adama ( https://www.adama-platform.com/ ), and I think it is pretty cool. The interesting thing is that the complexity of board games is exceptional.
I have a few clues to share. The first is that reactivity, which is found in excel, is a key to simplifying software as this makes the glue more automatic.
Another clue is figuring out bidirectional communication which relates to reactivity as a two-way street. However, this is primarily hard because we don't have great things off the shelf to deal with this beyond TCP. For more of a deep dive, check out https://www.adama-platform.com/2021/12/22/woe.html which talks about WebSocket.
My final clue is that you can't run away from state. So many people offload state because state is hard, and you have to contend with it. I'm building yet another database.
If you look from a person or user perspective - time zones or date formats are used in one location or in some context. People living in that context have it a lot easier because most of the time they don't care about other time zones.
I would say people simplify things but on local scale. If you want to go global that is your problem not humans that live in one place and use single time zone all their life.
1. becomes a post from which clever and ambitious people can build complex things. You invent docker to simplify application deployments and then somebody builds a n-dimensional microservice cloud on one side and starts commercializing new hardware architectures on the other. You don't remove the complexity, you just let it move around into new domains. (not a bad thing!)
2. is more temporal than you expect. The "finite and fundamental" problems of yesterday, today, and tomorrow are of different sets -- partly because (1) opens up new problem classes and partly because most problems are inescapably cultural and therefore subject to fashion cycles.
Not that I want to discourage you, but my view is that anything that makes software engineering simpler just leads us to tackling more complex problems until the complexity reaches the limit that people can handle.
So in that view, you can't succeed at making software engineering always simple. Instead, you can make previously intractable problems tractable.
Frameworks approximately do this, but for specific domains: they let you organize complexity into well-defined areas and put a pin in them, easing the cognitive load. Then you can handle those abstractions more easily. But yes, to your point, because frameworks typically only tackle a few levels of complexity, you still get that complexity back when you use the framework to advance the problems to the edge of what your framework is designed to address.
A recursive/fractal management of complexity will allow all levels of the hierarchy to feel similar, so you are never increasing complexity, only looking at a different resolution. I think the key to this is mapping out the fundamental organizational problems and how they relate to each other at different resolutions.
This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely.
However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That is, if we can get over the initial switching costs.
> small team of engineers
It took 2 men to over-take thousands creating Unix over Multix. It took Linus Torvalds - just about alone - a week to create git. We have already seen this prophecy come true.
As teams get bigger, communication sales factorially, and the more people you have the more mistakes you make, which increases the size exponentially with each further mistake. What Unix and git showed is that when you put everything into a small team of engineers heads, they can work through the complexity enough until they can do it themselves.
> Frameworks
One of the things I realized a while ago is that pure/impure and library/framework have a decent mapping between the two. If you have a framework, you give it code and it acts for you, just like impure code. And so the problem we keep running into, is instead of inverting the flow like 'hexagonal architecture' says, we keep piling impure onto impure onto impure. Hexagonal says to not do anything of substance, keep the adapter clean, but each framework sure is doing something. Each layer on the stack we go up, the harder it is to get down. So now we have OS and applications and containers and k8s running micro-services that ends up being run in a web browser, when all we really needed was microkernals.
That sounds a bit drastic. Surely at worst the scaling is quadratic?
The ability to suss-out meaning behind those channels and come to a shared understanding pushes it to factorial. If A is speaking to B and C, A needs to also think about what communication is happening between B and C, and this is different to what B knows about C and C knows about B.
In a carefully balanced classroom you might manage to get everyone knowing the same things, and this will allow knowledge to scale. Think about how much better TV got once they knew that viewers would watch every episode. What happens in software is that specialization quickly comes into play such that this is impossible in any organization that doesn't use mob programming. Other people might as well be speaking a different language.
Next time you are in a Sprint meeting, think about how much you don't understand. Even as the team lead or architect who designs the entire system and whose job it is to understand everything it will be a shocking amount - it's black boxes all the way down. You'll claim that you can't know everything and that anyone who tries will fail.
And this is made worse as the only true way to learn something is to do it, and if you aren't actually challenging yourself on something, anything you learn without doing will just fade away - as Spaced Repetition shows.
The bigger the team, the bigger the software, the more inevitable the collapse.
Metcalfe's law is quadratic rather than exponential -- O(n^2), not O(2^n). And if n people all need to be aware of communication happening between every pair of them (which is a worst-case), then that should just add a factor of n, bringing it up to cubic.
(On an unrelated and less pedantically nitpicky note, one of the most valuable professional skills I've developed is an ability and willingness to dive into black boxes. It's remarkable how many bugs arise from the interaction of two or more pieces of software with no overlap between their developers. Debugging those can involve a long, agonizing back-and-forth between people who know how X works and people who know how Y works -- or it can be over in a jiffy if you can quickly familiarize yourself with the basic internal workings of both. Past a certain point nobody can know everything, it's true, but this doesn't need to be as crippling for productivity as a lot of people allow it to be.)
But that doesn't stop black-boxes from being created faster than I can fix them, unfortunately.
Multics did not have thousands of developers. Linus was just more practical about software development, was maybe better at building a community, and more importantly focused on x86 commodity HW which was a bet that only Microsoft also happened to make.
Thanks.
Actually, "combinatorial" or "exponential" would be better words to describe complexity explosion. The number of possible cases/scenarios/logic flows/etc grows exponentially as a system's complexity increases. It is a curse and a blessing at the same time keeping so many SWEs gainfully employed.
In a lot of human work (not just software, but any endeavor involving technology even as "simple" as building a tunnel or simply involving many people coordinating together), complexity emerges this "unknown unknowns" property. This has happened since time immemorial. If you don't believe me, go master how to live out a subsistence life in an agrarian area of $nation during the 1500's with their tools but modern knowledge, then try to convince yourself complexity did not exist back then.
All still manageable by humans but at continually higher and higher levels of complexity.
you seem to have thought about this a bit did you have any ideas of how things could be rearranged to a more recursive solution?
Similarly using an operating system/assembly is much easier than trying to control pulses of electricity yourself!
I think one thing missing from this line of thinking is that these are people. They have their own motivations and egos. They get sick, have families, take vacations, quit.
I think part of the way large teams and large companies are structured is a hedge against changing teams with highly variable capacity. It's hard to overstate how much it can hurt a project to lose a someone who has a ton of experience locked up inside their head.
The alternative is that I have worked for places that try to treat thier engineers like fungible tokens that can be shuffled and replaced at will. That environment feels extremely dehumanizing and demotivating.
There’s an awful lot of cultural baggage in coding. Many of the concepts that seem essential - powerful text editors, devils tooling - can be completely removed. It requires rethinking from first principles and being willing to upset the Apple cart, but I believe it is possible.
A huge part of the problem is the messiness of the real world and costs.
While there may be a finite set of fundamental problems, the set of possible hardware and software platforms is ever increasing and each has it's own constraints and strengths.
A "universal programming environment" would need something like a "universal hardware interface". That's why things like Java and the Web became so popular despite their poor design.
Also, visual programming is a hard thing to make work and human language skills seem to be far greater than human visual skills.
Perhaps the best that can be done is "starting from zero" and making sure everything, from the ICs in your hardware to the memory models behind your software is thoroughly tested and formally verified.
This is insanely expensive with current technology. Perhaps some fancy math and AI advancements can make formal verification powerful enough for ubiquitous use. Until then, I see little hope for a "universal programming environment".
> Nobody can subtract from the system; everyone just adds.
To tie it to something concrete, just about everyone thinks C++ template metaprogramming is unreasonably difficult to reason about even aside from the syntax, but it exists to be extremely powerful at precisely expressing behavioral contracts of a software component that are difficult to do in other languages. Even then, it barely scratches the surface of what would be useful to express for the interaction of those components to be "simple". The number of design parameters of a component that really matter in various contexts are astronomical. No one can deal with reasoning about that many component traits in practice, so the software engineers simply hide most of them -- the complexity is still there and will manifest in unexpected ways because it is not visible at the interface.
All systems engineering is complex for this reason. Any non-trivial software system has to be reasoned about as a monolith at some level to correctly address complexity, which creates an unavoidable cognitive load. This isn't a software engineering problem, it is a systems-thinking problem. In chemical engineering, for example, there are often complex system design problems that cannot be adequately addressed by decomposing them into a sequence of sub-problems, all that does is hide major complexity at the interface of the sub-problems.
Could you give real world examples of:
I think you contradict your poetry with "Any non-trivial software system has to be reasoned about as a monolith at some level to correctly address complexity". If _any_ non-trivial software system _can_ be reasoned about as a monolith then it is true there exists some model with sufficient simplicity that there are well-defined discrete components.Traditional engineering disciplines that routinely have this problem of non-decomposability actually treat it as a systems problem (or a supercomputing problem), they don't shy away from it simply because the cognitive complexity is difficult. They convert the entire system into a set of simultaneous equations that needs to be solved for. In software, we would call this designing a monolith, but the reason it is done in traditional engineering disciplines is that you can't wish away the fundamental nature of the problem. In software, because it is not a physical engineering discipline, you can pretend that this issue doesn't exist, for a while at least.
Systems complexity is intrinsic, there is no trivial reduction. If there was engineers of all types would serve little purpose. Not coincidentally, the engineers that command the highest salaries are those that have the cognitive capacity to reason about the most complex systems dynamics, the dynamics that can't be decomposed into independent sub-problems before understanding the behavior of the entire system.
To your first point, in my experience, the best software, including complex platforms that handle massive amounts of traffic and data, can be built and maintained by small teams. I mean less than 20 developers. The larger the number of developers working on an application or platform has an almost inverse correlation to the speed in which new features are built or bugs fixed.
Software architecture has been done visually since perhaps its inception (tools like UML). At most places I've worked, every new project or large feature is diagrammed visually to use as a guide in breaking down the project into component parts.
In my experience complexity arises, not from lack of tools or industry knowledge, but from 3 main causes:
1. Inexperienced developer asked to create project, who just starts building without planning beforehand.
2. The main one - business demands features built that were never expected or planned for, and built as fast as possible. This causes developers to take shortcuts, make inelegant and difficult to maintain design choices. And leads to often inscrutable code that becomes technical debt especially after the original developer leaves the company. This will always happen as long as software is used to make a business money.
3. High team turnover - I've seen places where developers came and went so often that there was a myriad of things half started and never finished.
How I've solved or helped alleviate these issues: Make the business case to company owners or management that technical debt will be an ever increasing impediment to development velocity and the dev team will need a percentage of work in any given sprint to tackle tech debt issues (as opposed to having everybody work 100% on new features and bug fixes all the time).
I have successfully taken a platform that was bug-ridden, difficult to maintain, and where new feature development had slowed to a crawl due to the over-complexity of the software, to a place of stability and ease of development, simply by allowing our team to chip away at tech debt over the course of several years. Tech debt issues were rated by level of complexity, risk to the business in change (regression bugs), and impact on team velocity. We worked on the highest impact, lowest risk items first and kept going until there wasn't much left on that list.
I believe you are wrong, I'd wish you are right ;)
The other reason complexity happens is dependencies and DRY thinking, which is often good, but dependencies, centralization of code, and sharing of code are also a risk. Avoiding repetition is what causes abstractions to grow. Using other people’s libraries means that there are complex interactions you don’t understand. Most of the time it’s all fine, but occasionally sometimes it’s not. This trade is made consciously, for the reason that it’s much faster to develop your app using existing libraries, and not reinvent all wheels, and nearly everyone is doing it. The reason that tooling is unlikely to solve this part of the problem is that failures are so often caused by incorrect expectations - someone using a library hoping that it will do X when it only does Y. Even in theory, better tools could only tell us low-level information about what a library does, but I don’t think it’s possible for tools to tell you at a high level what a library can’t do.
BPMN and all of the UI pallet, drag drop coding applications try to do this very thing, this is super successful at smaller scales but breaks the first real world application.
He said that it used to possible, practically speaking, to treat software like discrete components so that you can reason about how to combine the units and predict the result, etc. By contrast, much programming today is bodging together a bunch of libraries whose behaviors are often poorly defined or otherwise opaque so that one often enough has to treat them like black boxes against which you have to apply scientific reasoning to understand how to use and integrate it. Hence Python and robots.
Presumably the new course teaches skills around managing this complexity, but the shift in model advocated -- from the metaphor of well-behaved discrete components in composition to opaque units whose inputs and outputs (possibly depending on hidden state) must be discovered was interesting to me, and ultimately adds to the accidental complexity of doing software now.
Do you think this sort of problem may have a distinct visual representation that would fundamentally reduce the complexity of library-dominated development? Or are you imagining a whole programming ecosystem built around your idea, so that the fundamental problem Sussmen described is somehow obviated? Or are you really trying to address essential, not accidental complexity?
We already have had, for a long time, a 'universal' model for a specific slice of information processing that consists of a very small set of components: Graphic User Interfaces.
One problem is the diversity of domain semantics and all the associated nuances. In GUI programming, this is the tedious bit of naming the components and mapping them to processing elements.
Another problem is engineering culture (du jour). A reductionist approach that attempts to leverage structural commonalities requires what is poo poo'd as "boiler plate" and all the associated warts of component oriented programming (including factory-factories). A revisionist shift in mindset is required.
“When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. And when you think about it, calling a method on an object is like sending a message to an object! It’s the same thing again! Those are all sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more. Blah.“
https://www.joelonsoftware.com/2001/04/21/dont-let-architect...
If one thinks of software programs as stored knowledge, then it makes sense why they are so buggy, error prone, etc. It's that the knowledge was never complete to begin with. Throw in the difficulties of computation, and you have the mess that is the software landscape.
But FP seems a bit stuck nowadays (to my untrained eye) with the breakthroughs from the greybeards of the last decade.
Do you have any progress thus far? Anything people can contribute to?
Once we've got a sufficiently complex pile that actually has promise, everybody duplicates the pile. And then puts an additional layer on top. These many pile+layer combos then co-evolve (and co-erode ;)
At some point, something so useful has evolved that we start copying that pile... and put new layers on top. Rinse. Repeat.
The complexity rarely collapses because those new layers can add support beams, if necessary. Worst case, a layer gets ablated, and we try again from one step below. We don't ever go back to the beginning.
(This also explains, to me, why Enterprise software is what it is - the complexity piles can rarely be shared or reused, and so there's no co-evolution, little chance to settle on the best pile)
And, to be fair, it's similar in other disciplines, but maybe a bit slower. Mechanical engineering has individual components, larger components built from those, and so on. It just is more expensive to copy the piles^W components we know to work.
Total collapse is exceedingly rare.
(That even holds for human societies - they, too, often "just" collapse down to simpler societal expressions. And we do co-evolve larger and larger forms of society. It's just that the erode-and-try-again process is uniquely painful for the humans involved)
> Most software at Google gets rewritten every few years.
> This may seem incredibly costly. Indeed, it does consume a large fraction of Google’s resources. However, it also has some crucial benefits that are key to Google’s agility and long-term success. In a period of a few years, it is typical for the requirements for a product to change significantly, as the software environment and other technology around it change, and as changes in technology or in the marketplace affect user needs, desires, and expectations. Software that is a few years old was designed around an older set of requirements and is typically not designed in a way that is optimal for current requirements. Furthermore, it has typically accumulated a lot of complexity. Rewriting code cuts away all the unnecessary accumulated complexity that was addressing requirements which are no longer so important. In addition, rewriting code is a way of transferring knowledge and a sense of ownership to newer team members. This sense of ownership is crucial for productivity: engineers naturally put more effort into developing features and fixing problems in code that they feel is “theirs”. Frequent rewrites also encourage mobility of engineers between different projects which helps to encourage cross-pollination of ideas. Frequent rewrites also help to ensure that code is written using modern technology and methodology.
Where does this yutz work? Because he doesn't speak for me. Generally, I write a complex program because I don't have time to write a simpler one; and when I have to deal with that complexity later I'm like arrrggh. Rube Goldberg machines are fun to watch, but no one wants to maintain, change, or build on one.
So complexity has a diminishing return for economical reasons.
We have an old app: dotnet webforms, it's clearly hanging on for its life and needs replacing but the mythical rewrite is impossible if you intend to rebuild and switch, it just involves too much time away from your core product.
Instead, we had to setup a proxy which could send all traffic between one of the two apps so we can move things a few at a time to the new system and all new work just gets built in the new system. It still takes a long time so I am now wondering whether as soon as we finish, we should immediately setup another new app alongside the current new one and repeat indefinitely.
It certainly wasn't easy to do because old and new frameworks are rarely able to share cookie formats/sessions etc so some hand-cranking was required.
Discipline and complexity are best friends. Laziness and procrastinating seem more valuable for avoiding complexity.
https://www.youtube.com/watch?v=LKtk3HCgTa8
A map is not as easy of a concept in some ways (it's more abstract, you are taking N elements of X to N elements of Y). But a map is simpler than a for loop, because there is no state between steps: it's completely symmetrical.
The for-loop "complects" each iteration together, even if it's as little as incrementing an integer. It's a side effect, however trivial.
To kick it up a notch, now imagine a 2-D loop. Now we have 2 counters. If it's a 2-D array, each addressing operation interacts with 2 counters. But map just needs a second map chained on, and each map is completely "unaware" of the other map. You could map over N dimensions and each map is symmetrical, while the for loop needs an ever increasing number of counters. This slight difference in complexity shows how complexity compounds with other complexity.
But "doStuff()" may be a very complicated function, that's very hard to understand, so it's not simple.
A real-world example might be something like the type conversion that JavaScript and PHP do in comparisons; "just do 0 == '0' and it'll work, easy right?", but turns out those conversion rules aren't simple at all.
I like how he states that simpler software doesn't necessarily mean fewer components (roughly quoting). Where the article argues for fewer boxes and arrows. It's hard to argue specifics, because the article isn't getting into them, but sometimes more boxes and arrows makes things simpler. If you don't have multiple boxes, what do you have? One box of spaghetti?
I also disagree with the article that engineers like complexity. I've maybe known a couple, but usually they are really junior, or really unpopular.
They claim to be building for the next hundreds of years, but you wouldn’t know it.
Fix the tools, and you start to fix the code. Unnecessary terminology, confusing APIs, superfluous patterns...it's like watching someone blindfolded in a dark room trying to justify the high price tag on their computer science degree.
After years of running my mouth, I proved my rants about the unnecessary complexity by building my own JS framework [1] and I'll never look back. The emperor is bare-ass naked.
[1] https://github.com/cheatcode/joystick
Also talks about the risk of losing knowledge, and a way to avoid that (make multiple copies).
Among other things, it suggests that there are some incentives for complexity: That engineers just enjoy complexity, and that making simple designs might not be good for the respect of your peers/bosses/career: "A simple design might make it seem like you’re not really doing your job."
While that's true, I think the essay isn't clear enough about the fact that simplicity is just plain hard. Complexity is sort of the "natural" outcome of "adding things". It takes time, effort, and skill to create software that can do what is asked for it while maintaining an internal simplicity. Even if everyone involved really wants it. I do not think software complexity comes only, or mainlly, from people not wanting it enough (explicitly or implicity), from incentives to complexity. It's just plain hard. It's a mistake to think just wanting it harder (or having the right incentives) is sufficient.
It may be the key challenge in software engineering -- how do we make software that does everything that's asked for it (powerful, complex things), while maintaining an internal simplicity? You can not accomplish it by cargo-culting any "design patterns" or other approaches. It's still a skilled craft. That comes from years of experience with the intent of developing this skill (years of experience alone aren't enough), looking at other people's code who are also trying to do it, and just some aptitude. And domain knowledge, if you understand the nature of the problems you are working on better, you can design simple solutions for them better--contrary to some current practices treating developers as interchangeable.
Yes, you need your organization to allow you to do it (including taking the more time it will take), but that's necessary but not sufficient for the outcome.
But that means simplicity is a cross-cutting concern, like security, and does not particularly fit into any environment that would attempt to separate design and implementation. It needs “redraw all the module boundaries” to be a plausible (if improbable) response to “this one 300-line part of this one module is awkward and three times longer than it really needs to be”. Aside from the obvious administrative problems, suggests that the complete edifice needs to fit into the mind of a couple of people at most or the iteration simply will not converge most of the time.
I don’t see a good solution to this.
[1] https://quoteinvestigator.com/2012/04/28/shorter-letter/
[2] http://www.ultratechnology.com/forth.htm
[3] There was an example about changing a negative-feedback control loop, written into the spec by hardware engineers emulating analog circuitry, into a couple of fixed-point operations that behaved differently but did the actual job just as well; I can’t find the reference right now but this a minor domain-specific change among the kind I have in mind.
> I don’t see a good solution to this.
I think this is probably true.
Some people take that to basically give up, and say, okay, our software is always going to be a mess, but look at all it's done as a mess, so be it.
Otherwise... the commonly understood solution seems to be composing the thing of independent units (very very very decoupled) each of which can fit into the mind of a couple of people at most. And then the whole, built of these independent units considered as black boxes, can perhaps also fit into the mind of a couple of people at most. I know you said "the complete edifice", which is not the same thing, but this is what we've got: and indeed is "abstraction", basically the entire basis of computer science and what makes possible anything we do. (Even the simplest program of 20 years ago -- can probably only fit into the mind of one or two people if you exclude the hardware all the way down, which you can because it is well abstracted and decoupled; it wasn't always, 50 years ago).
And we can come up with all the challenges and barriers of that -- of course! If it were easy, we'd have simple software. :)
But that's the task, I guess. But a lot of software isn't even built realizing that's the task -- to have one or two people "driving the bus" who have a mental model of the thing they're building, at any level of abstraction (I feel like Fred Brooks wrote about this), meaning that continuity of experience matters and domain knowledge matters, treating developers as commodities to be shifted around continually has a cost to complexity too, as does continually switching technologies and platforms so you can't build up a solid repertoire of encapsulated abstracted pieces to compose.
No. Speak for yourself. I am happiest when my solution to a problem is simplest. I am least happy when a solution requires introducing new infrastructure or major new ideas.
I have no idea who first said it, but one of my favorite sayings is: "Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands."
I strongly believe that this applies to computers as well. I love it when a system neatly solves its requirements in a way that looks effortless. When an algorithm looks like it just barely works for all the potential edge cases without any special handling because those edge cases just miraculously happen to work out as they fall through the happy path just like any other case. That's when a solution "feels" right.
If I can present my solution/code/explanation to a junior engineer, a C.S. student, or even a non-technical person and they get it, then that's when I feel like I've really done my job. It's music to my ears if I can make a problem look so easy that someone asks "Is that it all it takes?" (until they've tried to solve it on their own :-)
> It takes a lot of discipline to resist complexity
That can be taken in many ways.
Self-discipline, to resist convenience, the omniscient and omnipotent desire to control everything and always have the right answer immediately. That takes discipline to let go of. Also to discipline others, to say no, push back, become the master and set firm boundaries against crushing 'demands' for complexity. And intellectual discipline, to see clearly, elegantly, eliminating what is unnecessary. Over-complexity is a failure of the ordering principle of an organism, a step on the road to chaos and entropy.
It's based on my 20+ years, 50 different teams and many hundreds, perhaps thousands of encounters with software developers.
It's almost never the domain complexity that is high.
Sending a rocket to Mars is complex. And perhaps complex every time.
Dealing with people that has to work together can only be simplified when agreements are made and agreements are kept. Otherwise you cannot trust and have faith in people.
It comes down to far and near organization leadership, mandate and empowerment in small teams with few communication links outwards, clear expectations from whomever is asking for something and process management.
Writing code is rarely hard if you know the expectations and the boundaries of the task.
Empowerment gives the team the possibility to act on what they discover and encounter.
Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
One thing is complexity in code. Another is complexity in business. Some things are not made for software to solve.
Less experienced people are generally guided by their beliefs around their own ideas. If the mind is locked onto something it's hard not to thing it's a good idea. Usually in software there is simpler approach. Look for it
I can go on because I help companies and teams for a living. But I will stop here
Yes! 100% this. We need less fortune tellers and more pragmatists.
NO!
This mentality is short sighted by definition and leads to stagnation. It may work within large financially healthy companies that aren't trying to rock to the boat.
Making assumptions about the future is incredibly important for us to move forward. But with these caveats:
1) Acknowledge that you are taking a risk and that you may be wrong.
2) Validate your assumptions as quickly and as often as you can.
3) Be prepared to change course whenever your assumptions are invalidated.
> Less experienced people are generally guided by their beliefs around their own ideas. If the mind is locked onto something it's hard not to thing it's a good idea. Usually in software there is simpler approach.
Less experienced people are frequently the driving force behind new ideas precisely because of their belief in it. As experienced people we should be encouraging them to carry forward responsibly instead of projecting our aged and jaded perspectives.
> I can go on because I help companies and teams for a living.
Honestly, that's a weird way to phrase what you do.
But I believe the parent comment was talking about software abstractions. i.e., don't build abstractions because you might need them later. Instead, build the software in the simplest possible way to solve the problems it needs to handle now.
This is because you'll likely be wrong about your assumptions about which abstractions you'll need, and ripping those abstractions out is going to be more expensive than just modifying simple code to do new things.
> don't build abstractions because you might need them later ... [snip] ... This is because you'll likely be wrong about your assumptions about which abstractions you'll need, and ripping those abstractions out is going to be more expensive than just modifying simple code to do new things.
At worst, this is dangerously wrong. At best, it creates a straw man that people can use as a tool to shut down innovative ideas.
What you're describing is a very narrow/specific strategy that one might apply at a company. An example would be when a company is doing well financially and you don't need to risk amassing technical debt for some experimental side projects. Just build the MVPs, see what sticks, and only then decide whether or not to commit serious architecture to the problem.
If you're designing new systems that need to scale quickly (just one example) then you NEED to place bets on abstractions. A well designed abstraction could save you weeks (even months!) of time when you most need it.
I've seen what happens when a successful product is comprised of stitched together MVPs. It's too late for a rewrite and all of that "simple code" is creating a death-by-a-thousand-cuts nightmare.
Bad abstractions will always be bad. Bad system design will always be bad.
It's an aged/jaded mentality to claim that we should just stop designing systems ahead of time because we suck at it. Not everyone does. Some people are actually very good at it. It's possible to design an abstraction defensively so that it isn't more expensive to rip it out later.
Want a concrete example? Payment systems. If I'm writing a billing system then you bet I'm going to abstract away the interface with the payment processor. I would do this irrespective of our plans to expand to other payment processors. It shouldn't even be a debate because there's a way to build that abstraction with minimal overhead to the team.
On the other hand I would never blindly introduce an ORM into a system that doesn't need it. That would be a bad abstraction.
Or it could cost you a lot if your guesses are wrong. It goes both ways.
It's better to write code the simplest way to do what it needs to do now, but in a way that can be pulled out into a new system/abstraction later in an easy way. This is generally done by writing data-oriented code (focus on what the data is and how it needs to be transformed) rather than object-oriented code.
It's not hard to write abstractions defensively. For example you can limit access to your abstraction to a small and strict interface. What you do behind that interface can be the wild west; as long as you can just lop it off later.
> It goes both ways.
It does, but the word "simple" is often used in a deceptive way. A cacophony of "simple" systems are actually very complex in aggregate.
There are many cases (eg: something is your core business) where you'd want to put a lot of thought into an abstraction ahead of time. You'd then want to validate that abstraction's effectiveness and adjust as you go.
Your ORM example is good and exactly what i am referring to. Too much abstraction/gold plating due to unknown requirements in the future is perhaps a better way to put what is a problem.
And you also say that "If you're designing new systems that need to scale quickly..." - if you know that it will scale quickly then of course you know you need to do certain implementations to cater that. But if you do not know whether it will scale quickly then you are looking into an unknown future - and then you should step back on abstractions.
There are innumerable successful examples of committing blindly toward a future few people believe in. These are often cases where the objective evidence of success is very limited. If the objective evidence existed then "everyone would be doing it".
This is the crux of my disagreement. Of course we'll both agree that applying the wrong/stupid abstraction is wrong. But that's not what I'm discussing.
I'm talking about this quote specifically:
> Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
Perhaps you need to reframe this point to make it more clear?
> Don't look into the future. Because you cannot.
This is silly for all the aforementioned reasons. Especially when innovating in tech!
> And you are both naive and arrogant if you believe you can.
Calling someone arrogant for attempting to predict the future in technology. Come on, man.
> Deal with what is known now!
What is known now is what everyone else knows now. Unless you're dealing with the _extremely_ rare case where you have access to information no one else does, innovation requires one to commit to the unknown.
What I am trying to get at is that I've seen many times where a complex abstraction was introduced, and then it was only ever used for a single implementation of something. I've also seen many times where an abstraction was introduced, to make plugging in other backends easy, but then when it came time to plugin the second backend, the API was sufficiently different that it couldn't be shoehorned into the abstraction without a ton of extra work.
Sometimes you know enough about the problem space up front to avoid those problems, but often you don't. I often the better approach is to keep things as small as possible until you need more.
E.g., for payment processors, just ensuring that the surface area between payment processing and the rest of the code is as small as possible is likely sufficient. Maybe that's done through a simple abstraction, maybe not. But the point is, if you only have one payment processor now, the cost of implementing the abstraction doesn't change between now and when you actually need it (unless, of course, you chose a bad design, which we're obviously trying to avoid with either approach). So you can pay the cost of adding the abstraction when you actually need it, and can spend your time on something else that you need now.
I guess maybe bringing up the word "abstraction" was the wrong way to word it. In my experience, it's usually better to keep the code as small as possible early on, regardless of whether or not its using abstractions, because making it do what you want later is much easier when there is less code (in terms of constructs/concepts, not necessarily lines/characters). Also, try to keep things as orthogonal as possible. I think the reason I mentioned abstractions is that complex abstractions trying to predict all of the possible future extensions to the system is where I most often see design go wrong.
> Don't look into the future. Because you cannot. And you are both naive and arrogant if you believe you can. Deal with what is known now!
See my additional response to him elsewhere in the thread where I tried to break it down sentence by sentence.
Everything you're saying is on point for good system design IMO. What he was saying was totally different and IMO is a "dangerous" way of thinking when innovation is important.
[1] https://www.youtube.com/watch?v=ZSRHeXYDLko
If we refactor a very large function into smaller sub functions we are essentially adding new boxes... but are we creating complexity, or simplicity?