Why is there so much useless and unreliable software?
Instead of learning about those achievements and aiming to program for the same reliability, clarity, and sophistication, we see an abundance of software that cannot clearly describe their own behavior nor misbehavior.
Instead of incorporating the full functionality of XML/HTML/CSS/SVG/JS/WebGL into the development experience and providing ways to control them at the fundamental level, we reinvent crude approximations like the various web frameworks.
YAML and JSON often trumps XML/XSD until things get out of control, and even then, people still don't learn the lesson. Protobuf, flatbuffer, capnproto, and the like keep reinventing ASN.1.
Naive microservices partially reimplements Erlang's BEAM VM while ignoring all the hard parts that BEAM VM got right. Many people riding the microservice bandwagon have never even heard of Paxos, not to mention TLA+.
Many programmers keep learning new shining frameworks but are reluctant to learn about the crucial fundamentals, e.g., Introduction to Parallel Algorithms and Architectures, nor how to think clearly and unambiguously in the spirit of Coq/Agda/Lean.
No wonder ChatGPT exposes how shallow most of programming is and how lacking most programmers are in actual understanding. Linear logic and dependent types are there to help us design and think with clarity at a high level, but people would rather fumble around with OOP class hierarchies (participate in the pointless is-a/has-a arguments) and "architecture" design that only complicate things.
What is this madness? This doesn't sound like engineering.
146 comments
[ 2.0 ms ] story [ 215 ms ] threadSeems like there's not that much demand for what you would like to see.
Crappy, barely working, software now is better then 1 year delayed perfect one.
However it probably can hurt a brand reputation in the long run if you have a quality level below customer expectation.
So the quality we see would seem on average in line with expectations. Especially if things work most of the time and just fail to live up to expectations some of the time.
It seems to me that OP places a high level of usefulness or value of software at lower levels of abstraction, and I don’t disagree—it’s akin to how any manufacturing business is tremendously more valuable than the variety of consumer-facing products that can be made from it—but the cost of entry into such a business domain, and actually succeeding to make a profit, is often high.
From that I conclude that higher abstraction and making consumer-facing products through vertical integration (Apple) is more valuable than straightforward manufacturing. As is the old scheme of owning land with a valuable resource under it. And starting your company's name with the letter A.
So then if a company’s products or services lasts over generations, isn’t that actually more useful than another company’s products which people only throw away after some time? If the longevity of production of a product or service is not proof of the universality of a need, what is?
I find that the learning curve to get on top of most abstractions is greater than what is required to understand the foundations. Knowing and applying foundational components tends to yield more performant and less bloated solutions. Debugging is easier because you don't need to unravel layers of abstractions which don't quite align with the domain you are working with.
( Melvin E. Conway, https://en.wikipedia.org/wiki/Conway%27s_law )
Thus, we may conclude the world has some people that choose to be "useless and unreliable". The software part is just an artifact of this ideology. =)
This is the original Amazon memo and the rationale for microservices.
The bigger picture is that if you look at the totality of the software and hardware system, from app to browser to OS to hardware, you see that it replicates the company and market structure that produces it. Which is partly why the web is so chaotic. The web is a standards battleground over which companies fight.
Ah, but the customer interface and product layer continues to degenerate regardless of the mission statement. The public always gets what they asked for... especially when it is cheap, awful, and disposable. =)
https://twitter.com/maxkreminski/status/887815522061926400
"a reminder: if inexperienced creators are using your tool to churn out loads of half-baked garbage, your tool is a phenomenal success"
Software is such a powerful concept - it basically imbues physical objects with magic - that even bad software is hugely empowering to its users and takes off very quickly. The demand is staggering.
I appreciate that when you see yourself as the most intelligent person in the world it becomes intolerable to be surrounded by unthinking muggles scratching in the dirt, but after a while you realise that life is more complicated than that, people usually have good reasons for doing the things they do, and that perfection is neither attainable nor necessary for most of that.
Unfortunately high quality software tends to be so much more expensive in various ways that almost nobody bothers. Which is understandable, but in some cases still unacceptable.
Don't get me wrong, I'm not talking about open source software, those developers don't owe me anything and often produce better results anyway.
It's a double-edged sword because certifications are complex and sometimes expensive (e.g. FIPS certification for encryption), so smaller software companies have a hard time competing, potentially leading to more quasi-monopolies and cartels.
And a more subtle question: what happens below the acceptability theshold? It doesn't seem that acceptability can be driven up costlessly at zero time investment, so what this means is either "less software is produced" or "software is more expensive".
(We see a version of this with the app stores, which enforce a really crude "quality filter" by banning apps)
Though if you're talking about the lower bound for calling one's work "software engineering" you have a point, I think it's reasonable to expect some best practices from experienced developers. One problem seems to be how easy it is to underestimate the cost of bad software and what it takes to produce good quality. This reminds me a bit of the wildly inconsistent quality of CGI effects in today's movies, which is often the result of late changes and in turn insane deadlines, to some people VFX is magic anyway so why can't it also magically be done faster?
Most tech choices are made on the basis of incomplete information and incomplete planning or even no planning ahead. Many are merely following hype, instead of truly looking at the options and making a wise choice. Hype creates awareness of products or software, which ultimately reaches the uninformed masses. People not in the business of making software themselves have usually vastly less information to base their choice on and often make questionable choices. This in turn generates more incentive to continue making software like the one they chose. This is what amounts to the quote you posted.
When you're shooting a multi-hundred million satellite into orbit it's worth the extra few million expense of formal verification because otherwise you might lose a gigantic investment and even kill people in the process.
When you're working on something without such extreme constraints, with vague SLAs, and limited to no business risk, then regular unit/integration/etc tests are good enough and exceedingly cheaper.
And even then the cost effects of that are so prohibitive that SpaceX transformed the industry with their "we cannot guarantee the hoverslam landing will work first time" approach.
Producing something imperfect quickly and then iterating beats upfront planning by such a large margin so often that it's not funny.
People can give themselves all the fancy titles they want; I don't think 99% of software development is anything approaching Engineering.
(This isn't so much an attack on my fellow programmers as a recognition that this field is very young and still very immature.)
> engineering; the discipline dealing with the art or science of applying scientific knowledge to practical problems
We do apply scientific knowledge to practical problems, all compsci material of data structures and algorithms is science being applied to practical problems.
Every engineer I know has heard something like that, and knows that it applies to them. Except for a specific kind of engineer, and I bet you can figure what job title those engineers have
We do apply scientific knowledge to practical problems. The discipline isn't there, however, for the vast majority of us. In fact, most companies prefer it that way because proper design creation/validation takes a lot of time
Changing a tyre does not make you an engineer. I mean, half of us consider the idea of having written requirements bollocks that only exists to make managers happy, and half of the rest are happy to trudge along without knowing what actually needs to happen
Calling what most of us do "engineering" cheapens the meaning of the word
The engineer that is part of an association of engineers, working in one practice of engineering that's under a licencing scheme can lose their licence, that doesn't mean that engineering is only that... You're pigeon holing engineering to only the practice performed by licenced professionals, which is not true for the meaning of "engineering" as a whole so I don't agree with your premise a priori.
> We do apply scientific knowledge to practical problems. The discipline isn't there, however, for the vast majority of us. In fact, most companies prefer it that way because proper design creation/validation takes a lot of time
And those companies don't have a mature software engineering process, throughout my 20 years in software I've worked in them (startups usually).
On the other hand I've also worked in a couple of global tech companies that do apply software engineering practices, that do designing and RFCs to gather data and feedback about the proposals. That do apply processes for software design and architecture to maintain those systems healthy through a 5+ years timespan, etc.
Other companies don't need or want the heavyweight all of this processes carry, and they might be right in doing so if their systems won't cause major pain for a large swath of the society, they can tinker with their systems and play around, I agree that it isn't hard engineering but... Is it needed for these cases? Does everything built with software need all these practices or can these practices be used by the professionals and companies that require them?
> Calling what most of us do "engineering" cheapens the meaning of the word
Let's agree then that larger orgs do some kind of software engineering at least, a lot of smaller companies are still in the tinkering phase and that is completely fine.
It's a joke that most of us heard in one form or another back in Uni. I'm sure it's been repeated for centuries before the word "software" was first uttered
But then we all were studying some kind of Engineering, and only a very specific subset of us expected not to need a license after graduation... funnily enough, only that same subset didn't have to learn the core subjects that "all" engineers have to learn (tee hee)
>which is not true for the meaning of "engineering" as a whole so I don't agree with your premise a priori
As long as you are happy to call duct taping some damaged mechanical part "engineering" too, at which point why bother with the expression?
>I agree that it isn't hard engineering but... Is it needed for these cases?
I agree. But then I don't feel the need to call it engineering when it has very little to do with everything else goes by the same name
>a lot of smaller companies are still in the tinkering phase
A lot of big companies are still in the tinkering phase too - I've briefly worked for a certain big bank, and I live with someone that works for a certain health org that moves tens of billions of dollars every year. Their level of engineering is a joke too
The discipline itself isn't mature enough to warrant the name engineering. Additionally, it's incredibly amusing how much resistance the actual practicing professionals have against changes that would nudge the discipline towards warranting the name
We live in the wild west, act like cowboys, want to continue acting like cowboys, have the responsibility level of cowboys, and expect to be called engineers. Presumably because being called an "engineer" makes you sound legit
If we could abstract away the storage so that it is user owned, or at least a "utility" like electrical power lines where everyone is playing by the same rules, we would be able to trust these platforms significantly more, and rest assured our data is safe even if the software isn't.
This also affects our people (usually our top engineers) - which is why I want to start developing our own products this year.
Cover your ass, ticking off items as launched are other areas that lead to speed over quality every time.
The OP post reads as very naive, without experience in the real world corporate politics. No one really gives a shit about most things on that list (unfortunately).
- please put this broken crap of 15 services written in 6 different programming languages on a 50 node cluster so we can read and write data with 300 kbit/s.
2 years later:
- could you do something about this cluster, it costs us too much.
Its because they are engineered with a margin of safety and then inspected for obvious issues. But yeah throw them together is the name of the game. Faster = more money.
In fact most commercial apps - banks, flight bookers, and so on - are usable. Not superb, not outstanding, but certainly good enough for the job. But the user friction comes from dark patterns and weird decisions.
Example: we booked a hotel on booking.com recently and there's an option to pick a double bed or two singles. Turns out that whatever option you choose the hotel gets a message that says "This booking needs a double or a twin room" and doesn't specify which.
There's probably a good reason for this bizarre and unreliable behaviours, but I have no idea what it might be.
The unreliability happens in the giant apps. Chrome had a problem with Streetview recently, and then it had a problem opening PDFs. OS updates are notorious across all platforms.
But these are huge, frankenstein projects with significant hardware dependencies. Even if you modularise them, formalise them, and test them, how are you going to do that for every possible feature on every possible hardware combination?
It is also "good enough" for most use cases. So it is reasonable to start out with JSON and only upgrade to XML for complex documents/strutures.
Customers will not pay for quality. Not in toasters, not in cars, and not in software. Cheap and fast wins the day.
- no engineering standards in computer science, including especially in education of computer science
- some corporate finance views which see technology as a cost center rather than a business enabler
- some corporate strategies where marketing decides what is possible and when it must be ready (tomorrow. or yesterday.)
- computing and software development as a "fun", creative endeavor - as opposed to a rigorous, formal process
I'm sure there are dozens more reasons.
The only way to avoid this complexity is by keeping programs simple or by writing everything in house in a language like Ada/Spark very carefully, including formal verification and extensive testing. This is way too expensive for non-critical software. Btw, I don't think Erlang's concepts are general solutions, fault-tolerance by restarting services only eliminates certain kinds of problems and isn't suitable for all high-integrity demands.
You see the same thing in a lot of places. When people talk about shopping at Amazon, Wal-Mart, etc., they don't talk nearly as much about quality as about getting the lowest price. Well, the lowest price means cheap junk.
Because the only thing in common between the development of intangible products (software) and the engineering of tangible products is the cost of implementation.
> This doesn't sound like engineering.
It is precisely engineering. As opposed to pure science and art. (I consider mathematics to be, above all, art.)
Based on the post itself, you come with a theoretical mindset. You may consider purity to be more important than practical applications. Yet, if people write software to be used, they focus on the latter. Sometimes it results in hacky code even within an already hacky language. There are no extra points for purity.
Purity itself is a double-edged sword. Sometimes it makes the code more reliable. Other times - it generates a lot of abstract nonsense, which makes it harder to reason about the piece of software or change it.
On the positive side, look at the Rust language (and community!). While it has lovely abstractions and safety guarantees, it is a practical language - performant for writing and execution.
Compare and contrast with pure languages that are less versatile. Sure, Haskell has its practical uses. From a popular example - Pandoc. Yet, I know many programmers who have been preaching it is the best language but could not write a performant program that solves a given task that does not focus on algorithmics.
By contrast, PHP or early JavaScript were not pure by any sane standard. Yet, they conquered the web. (Fortunately, ES6 and TypeScript made the JavaScript environment much saner.)
FWIW, I happen to have the opposite opinion. I find Rust easier to write than, say, Ruby or JS, because it is statically typed, has enums and great pattern matching, and traits are great.
I’m hoping such pressure arrives in the form of legal or regulatory stuff, even if it chills the industry and slows/shrinks it. Until then we’re still in the computing equivalent of the auto industry when people died in mild fender benders that today are just mild annoyances.
That was the seed of the madness.
Instead of supplying dialog boxes(open, save, etc) for applications to use and then open files directly, the OS could have supplied handles (capabilities) from those dialog boxes for the applications to use. This would have allowed the user interface to be almost identical, and only require a few lines of code in the applications to change, in exchange for an environment which was almost immune to confused/rogue programs.
Because expectations were so corrupted in the desktop days, the situation now is effectively hopeless.
Applications should NEVER be trusted, especially not by the operating system.
Not hopeless. MacOS, for example, added entitlements that get you there. Applications cannot list random directories or open files at will.
If an application opens a file browser, the operating system shows you your file system. When you pick a file, the operating system gives the application the right to read that file.
See https://developer.apple.com/documentation/bundleresources/en...
> Applications should NEVER be trusted, especially not by the operating system.
This was basically impossible to do in the early personal computer era, before memory protection was a thing. So MacOS, Windows pre-NT, AmigaOS etc were all built around the assumption of applications reading each other's memory.
Very true. The user had a different option for that, since DOS was basically a program loader, and only one thing ran at a time, they could boot into program A, and use it with disks Q,R,S and know that no other disks could be corrupted. Then later, boot into program B, for disks X,Y,Z. It was capability based security, where each diskette was a capability. (Before hard drives ruined that trust model)
In the world of software - especially for startups - it is completely unacceptable to spend 5 years on building a MVP. There are some protected sectors where things move slower (defense, aerospace, for example), but if you're in the consumer field, you just can't spend too much time. You want to push out a MVP as soon as possible, and build on that.
And most software companies do not get penalized on shipping buggy software. Big game studios ship broken games, and spend a couple of years patching them up to their final form. People bitch and moan, but still throw money at 'em.
If you don't want bloated, broken, slow, and unreliable software - use your pocketbook.