> Abstractions are also the enemy of simplicity. Each new abstraction is supposed to make things simpler—that’s the promise, right?
Not exactly, no. The purpose of abstraction is to hide implementation detail, and thereby insulate one part of the codebase/application/system from variations in another. Graphics APIs for example - yes your code may be simpler for not having to deal with the register-level minutiae of pushing individual triangles, but the core benefit is that the same code should work on multiple different hardware devices.
Good abstractions break a codebase up into compartments - if you drop a grenade in one (change the requirements for example), then the others are unaffected and the remedial work required is much less.
In short: good abstractions simplify by centralizing operational logic. But it's not until a certain scale where that option is more efficient from bespoke implementations.
A heuristic we use at work is to not introduce an abstraction layer until there are at least two different implementations required.
That is if you think you'll probably need multiple implications, delay introducing an abstraction until you actually do.
Also, there are different ways of providing abstraction.
Perhaps you don't need to abstract the entire implementation but, as an example, rather change one parameter from passing a value to passing a function returning a value.
>> but the core benefit is that the same code should work on multiple different hardware devices.
I came here to say this. Attractions act as a bridge between things which allow those things to change independently.
Using an ORM allows my program to easily work against multiple sql databases. Using a compiler allows me to target different hardware. Using standard protocols allows me to communicate with different programs. Using libraries to do say email hides me from those protocols and allows me to adapt to service providers Using APIs not protocols.
In other words the abstraction is designed to hide a layer (which can change) from a program not interested in that level of change.
The key is to stop abstracting when the program cares. By all means encapsulate rules and processes, but they're a direct implementation of those rules and processes.
One can argue my "calculateLeaveForEmployee" function is an "abstraction" - but that would be a misnomer. Since there's only one set of rules in play (the set that matters now) its an implementation. An abstraction supports (at least) two things at the same time.
> Using an ORM allows my program to easily work against multiple sql databases.
A curious example, since most developers who've worked on any project with significant amount of data in a database would likely disagree.
IMO, ORMs mostly allow using programming-language-of-choice as a syntax for relational queries instead of constructing it by hand on top of serialization and deserialization of objects into rows and vice versa.
Indeed, given that most sql dialects have subtle differences that make them noncompatible with one another, and most ORMs have support for dipping into raw sql, sufficiently large systems tend to end up coupled with a particular database anyway. That's not to mention that the BAs will want raw sql access for report writing and switching systems breaks all of their scripts too.
Yes, ORMs came to mind for me as an example of indirection without abstraction. If you accept OP’s litmus test of “how often do I have to peek under the hood” I think ORMs generally don’t score particularly well.
If enabling the ablity to think of the database as the rich native data structures provided by the host programming language, and not the tables (rows and columns) that the underlying database service wants to think of the data as is not an abstraction, what is?
Fair, it’s probably incorrect to say there’s “no abstraction” in an ORM, maybe more precise to say it’s quite a leaky one. I’ve never seen an ORM used in the wild where you could actually avoid thinking about tables and columns and rows pretty frequently. But that’s admittedly armchair anecdotal.
In the small-to-medium scale cases I’ve seen, the main re-use you get for a particular mapping is when an object is used as a child node elsewhere. If it’s a one-off you’re essentially writing a bespoke shorthand wrapper for some custom SQL. Which may be a convenient shortcut or prettifier, but isn’t really functioning as an abstraction. Net net it seems like more overhead. Or like the real benefit is constraining database design to closely align with object design (which could be valuable but isn’t a function of abstraction).
This is all assuming code ownership across the stack. With a separate team managing data representation and handing reliable object contracts “over the wall” I could imagine a situation where the consumer gets to think purely in objects. I just haven’t observed that in practice. E.g. What if the contact has two address records? What if the number of purchase records exceeds the maximum result size? If things like that come up regularly enough, it’s worse when you’re waiting on a separate team to fix the leaks.
> I’ve never seen an ORM used in the wild where you could actually avoid thinking about tables and columns and rows pretty frequently.
There is a compelling argument that absent the n+1 problem, it could be a leak-free abstraction. And, in practice, SQLite doesn't suffer from the n+1 problem... But that is definitely true in a lot of cases, particularly where the n+1 problem is unavoidable.
> If it’s a one-off you’re essentially writing a bespoke shorthand wrapper for some custom SQL.
While the original comment also conflated query building and ORM, I am not sure that we should. The intersection of query building and ORM was historically referred to as the active record pattern. I suspect the ActiveRecord library, from Rails, calling itself an ORM is where things started to become confused. Regardless, whatever you want to call the data mapping and query building, they are undeniably distinct operations.
It may be fair to say that query builders are no more than an indirection. That said, I posit that they only exist as an abstraction to model composability on top of SQL. If SQL naturally composed through basic string concatenation, I expect the use of query builders would quickly disappear.
as the base of the query, with all of these being concatenated together.
But instead of this, an ORM usually provides you with a syntax (that will pass syntax checks and have highlighting) that matches the language, which is all nice and good because dealing with arbitrary strings does suck.
I've seen ORMs being used for query composition way before Rails even existed: I believe Hibernate had HQL with their first release in 2001, just like SQLObject did in 2002. I am sure neither of those "invented" the pattern.
Note that fetching objects using an ORM library, filtering and such is what I also consider query composition (it just happens behind the scenes).
> But SQL is "naturally composable" using string concatenation.
It is not. Not even in a simple case. Consider:
base = "SELECT * FROM table LIMIT 10"
cond1 = "WHERE foo = 1"
cond2 = "WHERE bar = 2"
base + cond1 + cond2 or any similar combination will not produce a valid query. It could if SQL had some thought put into it. There are many languages that have no problem with such things. But that irrational fear of moving beyond the 1970s when it comes to SQL...
The only realistic way to assemble queries is to prepare an AST-like structure to figure out where the pieces fit, and then write that out to a final query string. In practice, that means either first parsing the partial queries into that structure (hard) or providing a somewhat SQL looking API in the host language that builds the structure (easy). Unsurprisingly, most people choose what is easy.
> I am sure neither of those "invented" the pattern.
None of these invented the pattern. But the invention point is irrelevant anyway. You must have misread something?
ORM, as the name suggests, is an "adaptation" layer ("mapping" or "mapper").
It does not really abstract away relational data, it instead maps it to object/class hierarchy. And every single ORM I've used also provides an interface to map function/method/expression syntax to SQL queries to get back those rows (adapted into objects).
Now, in a very broad sense, mapping would also be an abstraction — just like an "indirection" would be — but considering the narrower definition from the OP (where indirection is not an abstraction), I think it's fair to say that ORM is also not an abstraction.
It's interesting that some modern ORMs are dropping trying to abstract out the dB engine. Specifically thinking of Drizzle for js. And rather just focusing on the programming interface
>> How many times have you required that your program runs against different sql databases without modification?
Our main commercial product currently supports 2 database engines, and we'll be offering a 3rd next year. For enterprise offerings it's pretty common for the client to prefer, or outright specify, the engine.
Commodotizing your complementary technology is a good way to not become dependent on any specific database, and hence can pivot quickly when required.
Fortunately I don't have a plane, so I don't have code to fly in any atmosphere.
The rest of us go with Postgres, or SQL server, or Oracle, but definitely don't have to prepare our systems to run in PostgreSQL on Mondays, on SQL server on Tuesdays and so on.
“The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.” — Edsger Dijkstra
But sometimes a new semantic level isn’t needed. Abstraction gets so much press when you might just need some good ol’ fashioned information hiding and separation of concerns.
This is such a great quote, and helps explain what is a good abstraction.
Because CRDTs have been in the zeitgeist a lot lately, I want to pick them as an example of a "good" abstraction.
CRDTs have mathematical properties which can be described and understood independently of a specific implementation. And importantly, you can judge whether an implementation is correct with reference to these abstract rules.
This means that when using a CRDT, you largely can treat it as a reliably-solved problem. Once you understand the concepts, and work out how to use the library you've picked, you don't have to think about the details. Though that doesn't mean sometimes the behaviour can be surprising:
TCP and HTTP are great examples too, though interestingly I don't know if they rely on mathematical definitions so much as just being extremely widespread to the point that reliable implementations are available anywhere you care to write code.
I like this article which also leans on the Dijkstra quote:
CRDTs are also an excellent example because of how their supporting infrastructure is impacted by their design, namely that Postgres’ method of dealing with updates makes for massive write amplification.
I wrote this [0] previously, but it still applies. IMO, as a dev, there are times where you really do need to think somewhat deeply about infrastructure choices. Unfortunately, knowing when you need to care practically requires you to already understand it.
It seems like almost everybody in these comments as well as OP are failing to differentiate between the narrow computer science / Dijkstra definition of abstraction versus how it's invoked colloquially, including in conversations of computer science such as here.
In that sense, we can correctly say that even encapsulation etc is a form of (colloquial) abstraction, but not Dijkstra's use of the word abstraction.
Maybe that’s the problem. Everyone knows that abstractions are desirable, and encapsulates as much as they can, and ends up with redirection hell without any of the advantages of true abstractions. If people understood what the word actually means, we might not be in this mess.
On practice, there exist platonic abstractions that hide mechanisms and can be this way, but you also need to abstract parts of your requirements. And you will never be able to achieve that while abstracting your requirements.
Dijkstra was an academic after all, and academics usually don't care about complex requirements.
While this is how the term is often used, I think it's cavalier use of language and confuses abstraction for modularity, and this linguistic confusion is one of the reasons a lot of programmers write bad "abstractions".
Organizing your code into components that are as independent as possible is a good practice and is the pursuit of modularity. A proper abstraction on the other hand, is a generalization that simplifies a conceptual layer in your code base.
Abstraction often enables greater modularity as a consequence, but they are not the same thing. For example, in the problem of text editing, people eventually realized that the manipulation of text is typically line oriented. Thinking of a text file as a collection of lines may seem like an obvious and modest abstraction, but it works well. This abstraction, in turn, leads to other couplings (e.g. line oriented motion is highly dependent on the line abstraction), but it also leads to potential modularity (e.g. printer code may no longer need to understand exactly how a display renders each character of text in a grid, instead, it too can work on "lines"). Good abstractions support modularity to the extent that they establish a shared domain of objects to communicate about and across systems, but they do not necessarily produce modularity in themselves.
Yes, abstraction and modularity are coincident but distinct topics, and my last paragraph implies that I am equating those two things. Good point.
I disagree that considering a text file as a set of lines really qualifies as an abstraction; it's just a different representation of the data - you could instead choose to use a list of words or a tree or whatever. An abstraction is a general interface to a concrete thing that allows you to substitute different but similar concrete things without changing the consumer of those things. The Linux filesystem is a great example - I can 'cat' basically anything that has a path, and some driver will pull data from the associated endpoint and display the data on screen. "File" is the abstraction, and consumers of files need not care about the specifics of communicating with the underlying devices. Such a system is also modular, but it's hard not to be when your abstraction is well conceived.
Perhaps we're saying the same thing with different words?
When you write "an abstraction is a general interface to a concrete thing that allows you to substitute different but similar concrete things without changing the consumer of those things", I feel you are actually describing a module created by making use of abstractions.
Abstractions almost always represent data differently than as it is in its most concrete form (e.g. lists and trees are abstractions both often built of vectors and pointers), but that is not the purpose of abstraction. To continue with the original example, the representation of text as a sequence of lines is a sort of abstraction, but rarely the most useful one: words, sentences, paragraphs and/or syntax trees are often what you want, and they are useful because they give us a representation of the raw text that is aligned with its purpose, not because we can implement them in multiple ways.
Furthermore, while we want to do some information hiding when implementing these abstractions (think particularly of syntax trees), that is not the purpose of having these abstractions - what we are hiding is the infrastructure to make it work, not any information conveyed by the text itself.
I don't like the mental image of layers. It's not wrong, but it's terribly inclusive of everything that is bad about bad abstractions. Even the worst abstractions will often look kind of nice from the layers angle.
I prefer the concept of orthogonality: you have an entire plane where your domain requirements can move in, but the way you make persistence reliable is completely orthogonal to that plane. It should be implemented in a way that does not change when, say, your customer account becomes a composite of of multiple identities. And neither should the way you organize UI components change, or your transportation tooling change. That entire transportation layer stack, from ethernet all the way up to your domain model generator of choice.
Did I say layers? Yeah, I did. You'll often see the orthogonal aspects themselves consisting of layers. But that's an implementation detail and those layers tend to be more pragmatic than dogmatic, as in the way TCP goes out of its way to avoid hiding any of the addressing of the underlying IP. TCP absolutely is a layer, but it has zero ambition to perhaps run on top of something other than IP, or hide differences between IPv4 and IPv6 from higher layers. It focuses on one thing, implementing streams on top of packets, again a problem nicely orthogonal to the problem of routing those packets (what IP does) or anything built on top of those streams.
To add on to your great comment, I've read a few papers that justify your remarks, but some have paywalls and I'll try to condense them anyways.
Parnas' criteria for when to split code into modules[0] is very compelling: modules should enforce information hiding, where the information of each module is how a key decision is implemented. Key decisions should be orthogonal: e.g., encryption shouldn't inform filesystem access and vice versa. This fits in nicely with practices for determining solutions for a given problem: nail down key decisions (or uncertainties around them) early on and encode them through modules and interfaces.
A logical application of Parnas' module criteria is, naturally, in network stacks. According to [1], although coarse program flow generally follows the OSI layers, the fine control flow may hop around quite a bit. For some time people tried to keep the OSI layers as inspiration and use hacks like upcalls to selectively violate layering, which is a sure sign that the layers are unhelpful. Instead, modules should reflect functionality like encryption, NIC interfacing, and routing. Aim for a flatter, horizontal architecture like a B-tree, not a contrived vertical tower with hacks to cross layers "incorrectly". There will be hierachies and layers, but they should be obviously necessary now. Layering is not the end or even the means but merely how things are.
A program's capability (functionality) is determined by the capabilities of its dependencies, and splitting up functionalities well is important. I use this language to relate it to capability-based security[2]. After all, "security" is basically just a stricter form of program "correctness", and just as a terminal emulator's input text handling capability shouldn't have much to do with its window event handling capability because malicious input shouldn't cause dangerous graphics bugs, there isn't much logical reason to have overlap regardless. They govern separate devices, protocols, and purposes. Capabilities along the lines of capability-based security reify the interfaces and provide strict enforcement.
Lastly, flexibility around protocols should be added or discard as given by the precise problem at hand. Don't preemptively choose some extreme, but instead be informed by what is actually desired. It will yield substantial implementation complexity and performance detriments[3] to over-generalize, and of course over-specializing is just inadequate.
> The purpose of abstraction is to hide implementation detail
Technically, that's encapsulation, though the sentiment is close, I think.
I rather view it as a matter of semantics. At one low level, you have operations that deal with some concrete interface or API, etc. You bundle those operations up behind an abstraction, providing methods whose names involve your application domain. Perhaps they are still at a technical level, and you bundle those up behind another abstraction, whose method names involve your business domain.
Yes, the lower level details are hidden from the higher levels, but the hiding is not the point. The point is to be able to write code that readily corresponds to the problem you are trying to solve.
> In object-oriented programming languages, and other related fields, encapsulation refers to one of two related but distinct notions, and sometimes to the combination thereof:[5][6]
> - A language mechanism for restricting direct access to some of the object's components.[7][8]
> - A language construct that facilitates the bundling of data with the methods (or other functions) operating on those data
The hiding is absolutely the point of abstraction, in the sense that caller can manipulate the subsystem without knowing about (important) inner details of it. As I said, graphics drivers are a great example - I just want to put a triangle on the screen, I don't want to care about the registers I have to set on an NVidia card vs those on an AMD card, or what their memory maps look like, and I don't want to have to rewrite my code when they release a new generation of hardware. Drawing a triangle is an abstraction, hiding away the details of how a specific graphics card achieves that.
Think about what the word 'abstract' means - the idea of a 'human' is an abstraction for bundles of molecules conforming to a general pattern with broadly similar qualities but almost infinite variability. The word conveniently hides a wealth of detail that we usually don't need to think about; if I tell you there are three humans in a room, you can make use of that information without knowing anything about their exact physical attributes or who they are.
I would refer to what you're describing as 'modelling' - these are related topics but I don't see them as the same thing.
This is a great point. Most modern software is riddled with unnecessary complexity which adds mental load, forces you to learn new concepts that are equally complex or more complex than the logic which they claim to abstract away from.
I find myself saying this over and over again; if the abstraction does not bring the code closer to the business domain; if it does not make it easier for you to explain the code to a non-technical person, then it's a poor abstraction.
Inventing technical constructs which simply shift the focus away from other technical constructs adds no value at all. Usually such reframing of logic only serves the person who wrote 'the abstraction' to navigate their own biased mental models, it doesn't simplify the logic from the perspective of anyone else.
I forget which programming talk I watched which pointed this out, but one extremely common example of this in Java is recreating subsets of the Collections API. I've done this before, heck even the Java standard library is guilty of this problem. When a class has a full set of get/put/has/remove methods, it is often not actually hiding the complexity of its component data structures.
Good example of a bad abstraction. If you're speaking the language (or "abstraction") of sets, you should see certain terminology: union, intersection, disjunction. These words are not part of the Java Set interface.
I would actually argue that the Collections API itself is a pretty good abstraction. It is often the case that conceptually I just want to work with multiple things and properties like order, duplicates, random access, etc. are not particularly important (in fact, requiring them adds inherent complexity). It's very useful that a vast amount of the standard library data structures conform to this interface or can create data views that conform to it.
How did "abstraction" and "hiding complexity" become perceived as such fundamental virtues in software development? There are actual virtues in that ballpark - reusable, reliable, flexible - but creating abstractions and hiding complexity does not necessarily lead to these virtues. Abstraction sounds no more virtuous to me than indirection.
I disagree. There are base abstractions you can’t avoid, of course, like the machine code of your computer, or the syscalls presented by it. Using these is not abstraction, unless you choose to build up interfaces of reusable pieces. Abstraction is structure. You still have to actually write some code that can be organized into structure. You could write code using just those base abstractions if you wanted, or as many do, choose libc as your base abstraction. Watching a program through strace gives you basically this view, regardless of the abstractions the program actually used to achieve the result.
Some abstractions are so ingrained you don't even think of them as abstractions. A file is an abstraction. A socket is an abstraction. The modern terminal is an abstraction.
Decades ago, when The Structure and Interpretation of Computer Programs taught us that programmers fundamentally do two things: abstraction and combination; and we are interested in programming languages insofar as they provide means to those two ends.
The two classic "hard problems" of computer science - cache invalidation and naming things - are both aspects of abstraction. Cache invalidation is a special case of making sure the abstraction does what it's supposed to, and naming is the most important part of causing the abstraction to have meaning.
Not parent, but I have a similar impression. Design patterns, clean code, and several of these well known tools were particularly useful during C++ and early Java eras, where footguns were abundant, and we had very little discussion about them - the Internet was a much smaller place back then. Most of the developer work was around building and maintaining huge code bases, be it desktop or server, monoliths were mostly the only game. And many initiatives grew trying to tame the inherent hazard.
I think that microservices (or at least, smaller services) and modern languages allow the code to stay more manageable, to the point where Java devs now are able to dismiss Spring and go for a much simpler Quarkus.
>How did "abstraction" and "hiding complexity" become perceived as such fundamental virtues in software development?
They aren't just fundamental virtues in software development, they're the fundamental basis of all cognition. If I twiddled with every bit in my computer I'd never write a hello world program, if I wrestled with every atom in my coffee cup I'd never drink a sip of coffee.
Abstraction and information hiding is the only way we ever accomplish anything because the amount of information fitting in our heads is astonishingly small compared to the systems we build. Without systems of abstraction we would literally get nothing meaningful done.
Part of it has to be that finding and defining abstractions is fun like a puzzle so programmers like doing it and finding ways to justify it after the fact
What languages do you work in? Would you be happier or more productive if you had to be aware of the quirks of every ISA and interrupt controller your code might run on?
Abstraction is good, in the way that leverage is good in the physical world: it is not always necessary, but people who are aware of the tool are vastly more capable than those who are not.
It's a huge ecosystem which turns 30 in May. Yes, it accumulated lots stuff that was cool back then and isn't so cool now but overall is doing pretty well (COBOL was 36 when Java got public).
I don't know how one read an article that talks specifically about how great TCP is as an abstraction and come away with the conclusion that it's arguing that all abstractions are bad.
>There’s a well-known saying: "All abstractions leak." It’s true. No matter how good the abstraction, eventually, you’ll run into situations where you need to understand the underlying implementation details
This is false. One can read up on Theorem's for Free by Wadler to see that not all abstractions are leaky.
This has such potential to be an interesting thread of conversation but all I get is a reference to a book that I haven't read and am unlikely to.
What examples of non-leaky abstractions do you have?
I could imagine something like "newtonian physics" but that leaks into my daily life every time I fire up google maps and get a GPS fix.
The OP's example of TCP seems close to the mark, but to be totally honest I'm not convinced. Every time I have to run ping to check whether my hung connection is due to connectivity, I'm breaking the abstraction. And I've had to debug network issues with Ethereal (yes, I'm dating myself). TCP does leak, it's just that most people don't know what to do with it when it does.
Theorems for Free tells you that some abstractions satisfy some mathematical properties (for free!) under some circumstances.
If you write down a function with signature
{T : Type} -> T -> T
then it must be the identity function, if you do not use "malicious" extensions of the type system.
But what is the performance of the identity function?
Here is an identity function:
lambda T, lambda t, if (2 + 2 = 4) then t else t
In other words: I can hide pretty much arbitrary computation in my identity function.
Users of my identity functuon will notice that it is wicked slow (in reality, I let my identity function compute Busy Beaver 5, before doing nothing). Their complaints are evidence of leaky abstraction.
Now you might have a smart optimizing compiler that knows about Thm4Free... But that's another story.
I don't think you can have such compiler, at least for a general case, without solving the halting problem first. After all, you can encode arbitrary computations at the type level.
The point of the abstractions I'm talking about is not to abstract away the computation part of a program, it's to abstract away the types and complexities of the semantics.
I got a piece of advice writing UI code a long time ago: Don't marry your display code to your business logic.
I'd like to say this has served me well. It's the reason I never went for JSX or other frameworks that put logical code into templates or things like that. That is one abstraction I found unhelpful.
However, I've come around to not taking that advice as literally as I used to. Looking back over 25 years of code, I can see a lot of times I tried to abstract away display code in ways that made it exceedingly difficult to detect why it only failed on certain pieces of data. Sometimes this was sheep shaving tightly bound code into generic routines, and sometimes it was planned that way. This is another type of abstraction that adds cognitive load: One where instead of writing wrappers for a specific use case, you try to generalize everything you write to account for all possible use cases in advance.
There's some sort of balance that has to be struck between these two poles. The older I get, though, the more I suspect that whatever balance I strike today I'll find unsatisfactory if I have to revisit the code in ten years.
I learned that lesson building an utility with JavaFX. I've done a few years with React and the usual pattern was to move almost everything out of the components. Unless it's an event handler, a data transformer for the view, or something that manipulates the view, it has no business belonging to this layer.
I don't try to generalize it, I just try to make the separation clear using functions/methods/classes. Instead of having the post button's handler directly send the request, I create a function inside the `api` module that does it. It does have the effect on putting names on code patterns inside the project.
When you say "components" do you mean that in the mixed React sense where a component could contain HTML? In my own usual cases, I call the Javascript a component, and the HTML a template. I usually take a handlebars approach on the template content and something like "data-role" to identify the template tags to the JS, and beyond that don't mix them. However, my client-facing JS components themselves are totally bound to the templates they load up - they expect the presence of certain fields to work with. I'm talking more about not mixing any business logic into those JS components: Let's say, in a form component, not anticipating that a dropdown menu will have any particular shape or size of dropdown item, which means those items need to be specified separately. This leads to JS components relying on lots of other components, when sometimes you just need one type of dropdown item for a particular component, and having dropdown items be a 20-headed beast makes everything upstream need to define them first.
Sometimes you just need a form to do what it says on the label.
I started with a templating system that had a very limited logic and I'm still quite fond of this approach.
Basically arguments for a template had a form of a tree prepared by the controller function. The template could only display values from the tree (possibly processed by some library function, for example date formatter), hide or show fragments of html dependaning on the presence or absence of some value or branch in the tree, descend into a branch of a tree and iterate over an array from the tree, while descending into one iteration at a time. Also could include subtemplate feeding it a branch of the tree. This was enough to build any UI out of components and kept their html simple, separate and 100% html. Even the template logic had a form of html comments. One could open such template in any html editor including visual ones. It was all before advent of client side frameworks.
You could mimic this in React by preparing all data in render method as a jsonlike tree before outputting any JSX tag and limit yourself inside JSX part to just if and map(it =>{}) and single value {}
Yeah. I did this too. Mine's driven off a database with a structure of
1. pages, each of which may or may not have a parent pageID; the entire tree is read and parsed to generate the menu system and links/sub-links
2. modules which reference HTML templates to load through a parser that replaces areas with {{handlebar}} variables written by the clients through their own API
3. something like page_modules which locate the modules on the page,
4. a renderer which uses the above stuff to make the menu system, figure out what URL people are trying to get to, load the page, load the modules in the page, and fill in the crap the client wrote inside the {{handlebars}}
This has worked so well for so long that I can basically spin up a website in 15 minutes for a new client, let them fill in anything I want them to by themselves, throw some art on it and call it a day.
It worked so well that I ended up writing a newer version that uses a bunch of Javascript to accomplish basically the same thing with smoother transitions in a single-page app that doesn't look or act like an SPA, but it was basically pointless.
I don't think I'm conflating things. Any time you insert sugar into your HTML that makes your end user see output that's been inserted by how your framework interprets that sugar, you may be binding your business logic to your display code. Possibly in subtle ways you don't realize until later. A common case is rounding decimals to a user that aren't rounded in the system.
Letting any control flow into your HTMX/JSX templates is begging for trouble. It boggles the mind that people abandoned PHP - where at least the output was static and checkable - for something exactly like mixed HTML/code where some processing was done on the front end and everything was supposed to be dehydrated. Only to pivot again to hydration on the back-end.
JSX and React came on the scene 10 years after I first realized using the client for anything logical was an anti-pattern. Back then, I remember people would write things like:
and put a bunch of validation on the client side, too. Clients are dumb terminals. That's it. I'm not confusing the use of logic-in-JSX with business-logic-in-display. One only has to look at every basic JSX example on the web that's taught to newbies to see a million ways it can go wrong.
As someone who has little experience in this topic but has come to a similar conclusion, I think the main downside of this strict separation you're recommending is performance/efficiency. Have you noticed that to be a problem in practice? It's not always clear whether the simplest solution can actually be feasible, or perhaps that is just a reflection of still untapped understanding of the problem domain.
Generally there isn't a huge tradeoff in business software. In games where every CPU cycle counts it's another story. And yes, I have been guilty of writing display code divorced from game logic that was way beyond the scope of what needed to be actually displayed on the screen for a particular situation, and having that over-generalization lead to unacceptable performance drops. So you make a good point.
I think your principle is good, but this is going too far to throw out react.
You really just want to split those things out of the UI component and keep minimal code in the component file. But having it be code free is too much for real client side apps.
Modern apps have very interactive UI that needs code. Things like animation management(stop/start/cancel) etc… have subtle interactions. Canvas zoom and pan, canvas render layouts that relate to UI, etc… lots of timing and control code that is app state dependent and transient and belongs in components.
I apply the simple principle and move any app state management and business logic out of the component UI code. But still lean into logical code in UI.
TCP is great. Long chains of one-line functions that just permute the arguments really suck. These both get called abstraction, and yet they're quite different.
But then you hear people describe abstraction ahem abstractly. "Abstraction lets you think at a higher level," "abstraction hides implementation detail," and it's clear that neither of those things are really abstractions.
As the OP mentions, we have a great term for those long chains of one-line functions: indirection. But what is TCP? TCP is a protocol. It is not just giving a higher-level way to think about the levels underneath it in the 7-layer networking model. It is not just something that hides the implementations of the IP or Ethernet protocols. It is its own implementation of a new thing. TCP has its own interface and its own promises made to consumers. It is implemented using lower-level protocols, yes, but it adds something that was fundamentally not there before.
I think things like TCP, the idea of a file, and the idea of a thread are best put into another category. They are not simply higher level lenses to the network, the hard drive, or the preemptive interrupt feature of a processor. They are concepts, as described in Daniel Jackson's book "The Essence of Software," by far the best software design book I've read.
There is something else that does match the way people talk about abstraction. When you say "This function changes this library from the uninitialized state to the initialized state," you have collapsed the exponentially-large number of settings of bits it could actually be in down to two abstract states, "uninitialized" and "initialized," while claiming that this simpler description provides a useful model for describing the behavior of that and other functions. That's the thing that fulfills Dijkstra's famous edict about abstraction, that it "create[s] a new semantic level in which one can be absolutely precise." And it's not part of the code itself, but rather a tool that can be used to describe code.
It takes a lot more to explain true abstraction, but I've already written this up (cf.: https://news.ycombinator.com/item?id=30840873 ). And I encourage anyone who still wants to understand abstraction more deeply to go to the primary sources and try to understand abstract interpretation in program analysis or abstraction refinement in formal verification and program derivation.
Hey Jimmy, I've read your comment and also your article in the past with great interest. This topic is absolutely fascinating to me.
I just re-read your article but unfortunately I still struggle to really understand it. I believe you have a lot of experience in this, so I'd love to read a more dumbed down version of it with less math and references to PL concepts and more practical examples. Like, this piece of code does not contain an abstraction, because X, and this piece of code does, because Y.
For example, in the "TV -> serial number" abstraction, if I were to define only one operation (checking whether two TV's are the same), would it make it a good abstraction, as now it is both sound and precise?
And what are the practical benefits of using this definition of abstraction? Even if I were to accept this definition, my colleagues might not necessarily do the same, nor would the general programming community
> if I were to define only one operation (checking whether two TV's are the same), would it make it a good abstraction, as now it is both sound and precise?
It would!
> And what are the practical benefits of using this definition of abstraction?
Uhhh...that it's actually a coherent definition, and it's hard to think or speak clearly without coherent definitions?
If you're talking about using it in communication, then yeah, if you can't educate your coworkers, you have to find a common language. They should understand all the words for things that aren't abstraction except maybe "concept," and when they use the word "abstraction," you'll have to deduce or ask which of the many things they may be referring to.
If you're talking about using it for programming: you kinda can't not use it. It is impossible to reason or write about code without employing abstraction somewhere. What you can do is get better about finding good abstractions, and more consistent about making behavior well defined on abstract states. If you're able to write in a comment "If this function returns success, then a table has been reserved for the requesting user in the requested time slot," and the data structures do not organize the information in that way, and yet you can comment this and other functions in terms of those concepts and have them behave predictably, then you are programming with true abstraction.
In this case, not programming with true abstraction would mean one of two things:
1. You instead write "If this function returns success, then a new entry has been created in the RESERVATIONS table that....", or
2. You have another function that says "Precondition: A table has been reserved for the user in this timeslot," and yet it doesn't work in all cases where the first function returns success
I think it's pretty clear that both ways to not use true abstraction make for a sadder programming life.
I'll have to muse about what the more dumbed down version would look like (as this version is already quite dumbed down compared to the primary sources). It wouldn't be quite a matter of saying "This code contains an abstraction, this other code doesn't," because (and this is quite important) abstraction is a pattern imposed on code, and not part of the code itself.
We do have a document with a number of examples of true abstraction — written in English, rather than code, in accordance with the above. It's normally reserved for our paying students, but, if you E-mail me, I'll send it to you anyway — my contact information easy to find.
> a bad one turns every small bug into an excavation.
I find that I need to debug my abstractions frequently, while I’m first writing my code, then I never need to dig into them, ever again, or they do their job, and let me deal with adding/removing functionality, in the future, while not touching most of the code.
That’s why I use them. That’s what they are supposed to do.
Because they are abstractions, this initial debugging is often a lot harder than it might be for “straight-through” code, but is made easier, because the code architecture is still fresh in my mind; where it would be quite challenging, coming at it without that knowledge.
If I decide it’s a “bad abstraction,” because of that initial debugging, and destroy or perforate it, then what happens after, is my own fault.
I’ve been using layers, modules, and abstractions, for decades.
Just today, I released an update to a shipping app, that adds some huge changes, while barely affecting the user experience (except maybe, making it better).
I had to spend a great deal of time testing (and addressing small issues, far above the abstractions), but implementing the major changes was insanely easy. I swapped out an entire server SDK for the “killer feature” of the app.
The best way to achieve a good abstraction is to recall what the word meant before computer science: namely, something closer to generalization.
In computing, we emphasize the communicational (i.e. interface) aspects of our code, and, in this respect, tend to focus on an "abstraction"'s role in hiding information. But a good abstraction does more than simply hide detail, it generalizes particulars into a new kind of "object" that is easier to reason about.
If you keep this in mind, you'll realize that having a lot of particulars to identify shared properties that you can abstract away is a prerequisite. The best abstractions I've seen have always come into being only after a significant amount of particularized code had already been written. It is only then that you can identify the actual common properties and patterns of use. Contrarily, abstractions that are built upfront to try and do little more than hide details or to account for potential similarities or complexity, instead of actual already existent complexity are typically far more confusing and poorly designed.
Unfortunately, I did. It is an attempt to approach complexity, cognitive load and high entropy in code, jumping to conclusions prematurely while suggesting a solution that is worse than the problem.
I would really like to convince you that you are missing something very important. Please try to make sense of the article, reading it again a trying to find cases where it makes sense for you.
"10" is the representation of a data; "0xA" is another representation of the same data.
Not being able to touch something doesn't make it an abstraction. Light is not an abstraction, a contract is not an abstraction. 10 is not an abstraction, it is an ordinal [1].
"Isomorphism" is an abstraction. It doesn't name a particular data or value, but a class of functions that share common properties. A function template or functions written in a dynamically typed language can describe a particular group of isomorphism.
> Light is not an abstraction, a contract is not an abstraction. 10 is not an abstraction, it is an ordinal [1].
It seems to me that all of those things very much are abstractions. They are not the utmost level of abstraction, but they are abstractions!
(Actually "a contract," which at first I thought was the clearest win, I'm now not sure about. On reflection, it seems like a concretization, turning abstract ideas of trust and reliability into concrete conditions under which the contract has or has not been met.)
Yes, abstraction and generalization are properties you'd rather look for the second time around. Someone was already warning about this 25 years ago [1]:
You have a boring problem and hiding behind it is a much more interesting problem. So you code the more interesting problem and the one you've got is a subset of it and it falls out trivial. But of course you wrote ten times as much code as you needed to solve the problem that you actually had.
Ten times code means ten times cost; the cost of writing it, the cost of documenting it, it the cost of storing it in memory, the cost of storing it on disk, the cost of compiling it, the cost of loading it, everything you do will be ten times as expensive as it needed to be. Actually worse than that because complexity increases exponentially.
This person did his own CAD software from scratch in order to make custom chips [2].
Although of course solving the abstract problem does not have to be 10 times as much code. The best solutions are often those, that recognize the more general problem, solve it with little and elegant code, then turn to the specific problem, expressing it in terms of the abstract problem and thereby solving it in just a few lines of code. Such an approach should usually be accompanied by some documentation.
To give a trivial example: Binary search or any other bog standard algorithm. You would want to have an implementation for the algorithm and named as such and then only apply it in the specific case you have. Sorting algorithms. You don't want to rewrite it all the time. Actually rewriting it all the time would be the thing that creates "10 times" the code.
I am wondering where the "generalize code after doing it for the 3rd time" rule of thumb comes from? I also subscribe to it, and read it somewhere 15/20 years ago.
Well known data structures and algorithms are well know because they have been used more than three times. That said: if you are dealing with a situation where you have to implement them yourself, you may want to consider whether the rule of three applies. (Clearly this depends upon the situation.)
If the well known data structures and algorithms are not provide by your language get a better language.
There are exceptions. If you are implementing the language it is your job to write them. It is useful as a student to implement the basics from scratch. Your language may decide something is not allowed and thus not implement it (a doubly linked list is almost always a bad idea in the real world. Likewise you don't need to provide all the sorting algorithms)
Tell that to Richard Hamming: "Instead of attacking isolated problems, I made the resolution that I would never again solve an isolated problem except as characteristic of a class." [1]
I have seen this "premature abstraction" warning creep through our discourse lately, but I don't clearly understand it. I feel like I'm making calls all the time about when to introduce functions or classes that will save you time or effort in the future without forcing yourself to see the repetition before you do. Not only that but Hamming's advice has rung true in my career. Solving the general problem is often easier than solving a specific case and can be re-used for later instances, too.
To your point, we use abstractions all the damn time. They're everywhere. Even programming languages are an abstraction (especially high level ones). You and I, and everybody else here doesn't pick a cylinder and a block to write to and tell the hard drive to move it's arm into place and record the magnetic data, no we all talk about inserting a row into the DB.
Abstractions are essential to productivity or you'll never get out of the "make it from scratch" trap
Yes, but there is a line to be drawn somewhere. Filesystems are sufficiently advanced such that there’s no meaningful gains to be had from manually allocating CHS for data, and they provide housekeeping. C abstracts a lot of architecture-specific information away, but still requires that you understand a modicum of memory management; if you understand it (and cache line access) well, you can get even more performance. Python abstracts that away as well, and gives you a huge standard library to accomplish many common tasks with ease.
You can quickly make a prototype in Python, but it won’t be as performant as C. You can spend time profiling it and moving computationally-heavy parts into C extensions (I do this for fun and learning), but you’ll likely spend more time and get worse results than if you just rewrote it.
Docker is an abstraction over already-existing technology like cgroups. It provides an easy-to-understand model, and a common language. This is quite valuable, but it does allow one to not know about what it’s hiding, which is problematic when troubleshooting – for example, naïvely assuming that querying /proc in a container shows the CPU resources allocated to the container, rather than the host.
That’s how I view abstractions. They can be incredibly useful, but they usually have trade-offs, and those should always be considered. Most importantly, you should at a minimum be aware of what you’re giving up by using them, even if you don’t fully understand it.
> Solving the general problem is often easier than solving a specific case and can be re-used for later instances, too.
You and the other person are both correct. What you're saying makes sense and it is what everybody is trained to do. However, it leads to a lot of useless code exactly because you're applying an abstraction that is used only once. That's why most codebases are bloated and have a huge number of dependencies.
If we ignore 'buggy' part I think you projecting current good state back into not so good old times. I am pretty sure you will not replace radix sort that uses domain knowledge of reduced value range with qsort circa C++98.
Things became much better after relatively recent improvements in generalist sorting libraries: when pattern defeating quick sort variants became norm, when mostly sorted cases got covered ...
Tbh I do have a case from 2013-16 when I regret not doing it for one AAA project on ps4.
imo more like the 50th time around, and by a computer scientist, and not on company time until after the abstraction POC is validated, for example both React and Angular were side projects before the firm decided to invest. Software development outcomes today are driven by: ignorance, narcissism, and self preservation
Have you ever looked at how useless Chuck Moore's stuff is? Like, the chip designs are of the type “384 independent Forth chips with tiny amounts of RAM and mediocre interconnects, and if you want to actually do anything with them, you'll need to use 128 of them to program your own DDR3 controller”. Or, he demonstrates how awesome Forth is by showing that you can do “the guts of a disk driver” in five lines, except that it's the dog-slow PIO mode.
It turns out that if you can just change the problem statement, then sure, you can write very simple things. But if you have a real problem to solve (and you can't just say “no, I want to solve a simpler problem”), the Chuck Moore way of thinking doesn't really produce, well, good solutions. It simply doesn't scale to anything large, and not everything can be made small.
I wrote a little chess engine in Python that plays at least elo 1400 using alpha-beta search with the goal of making it simple and pedagogical, trying to outdo Lisp. I am thinking about making it talk XBoard, removing the experimental stuff, then posting it to GitHub just as a nice example.
I think though if I want to get more into chess programming I'm going to switch to a faster language.
IMHO that is a failed example of "Chuck Moore's stuff".
He went down a rabbit hole to an extreme level because that's what he does.
His earlier CPU experiments like the 4016 and Shboom were excellent examples of ultra-RISC architectures.
The thing Chuck explored, related to abstraction, which I don't see much in conventional machines was reducing calling overhead. (1 cycle call and intrinsic return on many instructions ie: free return)
Some of the decisions we make today have a lot to do with what happens at the hardware level when we add abstraction. It just costs more than we are prepared to pay so it is avoided ... but for the wrong reason.
The real time visual mixing console that produced many music videos that ran endlessly on MTV back when they actually played music videos, and special effects for blockbuster films like RoboCop and Total Recall, wasn't a "simple thing".
Coco Conn and Paul Rother wrote this up about what they did with FORTH at HOMER & Assoc, who made some really classic music videos including Atomic Dog, and hired Charles Moore himself! Here's what Coco Conn posted about it, and some discussion and links about it that I'm including with her permission:
>First shown at the 1989 Siggraph Electronic Theater to a rave response, this 3 minute humourous film went on to win several top computer graphic awards that same year including Niccograph of Japan.
>Coco: This was a show favorite at the SIGGRAPH film show that year. The year before the conference committee decided that showing demos wasn't the way to go anymore. Peter wrote Flying Logos as a way to sneak our demo reel into the show by turning it into a story. It worked and we made it into the film show.
>Don: I truly believe that in some other alternate dimension, there is a Flying Logo Heaven where the souls of dead flying logos go, where they dramatically promenade and swoop and spin around each other in pomp and pageantry to bombastic theme music. It would make a great screen saver, at least! Somewhere the Sun Logo and the SGI Logo are still dancing together.
----
Peter Conn and I [Coco Conn] had a company called HOMER & Assoc. which was located at the Sunset Gower Studios from 1977 until we closed shop in 1997. We made music videos, commercials & computer graphics/special effects for feature films. One cool note, we worked with Paul Verhoven on both RoboCop in 1986 and the x-ray scene for Total Recall in '89.
HOMER was actually a real time visual mixing console that our in-house engineer spent 1978 - 1981 designing and building, from scratch. The name HOMER stood for "Hybrid Optical Montage Electronically Reproduced."...
I did some work with his Greenarrays chip as a student. It was very limited and awkward to use but it could also operate on absurdly low energy, with very low overhead for waking up or almost completely powering down. At some point, we had a demo running some simple signals processing code off a homemade bleach battery, and I wouldn't be surprised if you could make something work off a bunch of lemons too.
This was over a decade ago (yikes) and I don't remember the exact numbers, but I do remember it used substantially less power than a comparable MSP430 microcontroller.
That seems pretty useful and impressive to me, especially given it was created by a very small team with limited funding.
> Like the RTX2000 which landed on a comet kind of useless?
Yeah, in 1983 he designed a chip that was further developed by others for space usage.
> Maybe the controlling radio telescopes kind of useless.
Yeah, which he did in 1970.
Note a pattern here? That this design paradigm holds up pretty well in a primitive computing world when things are simple and demands are low, and is thoroughly useless to keep on promoting today?
> That this design paradigm holds up pretty well in a primitive computing world when things are simple and demands are low, and is thoroughly useless to keep on promoting today?
Somehow all of those projects are… really old? A satellite control UI on top of Windows 7 from 2015 (somehow Chuck Moore's assertion that “If they are starting from the OS they have made the first mistake” did not extend to Windows here). A STM-16 (2 Mbit/sec!) multiplexer, very modern. A power plant control system from 1995. And yes, an aerospace project indeed, also from 1995. Notably none of these are using his CAD software, much less using any of his chips.
I'm sure they are? There are people interested in all sorts of things. (Well, at least some of them were in 2022, which is the last time someone bothered to add a news post.) That doesn't mean it is a useful design paradigm for the world at large. Remember, Chuck Moore's claim is that not following his ways means 10x the bugs, cost, etc. -- I don't really see anything supporting that claim.
Harris RTX2000 is a 8Mhz machine with 16-bit data-path, paged program memory, and 1MB addressable memory. This is really an example of "small machine".
Philae had to choose this CPU because there are very few rad-hard low-power CPUs available (and it's not even that low power by modern standarts, 5 mA/MHz), but I am sure they'd choose something bigger if they could.
As for radio telescope, I am not sure which ones are you talking about, but those environments are not particularly challenging compared to spaceflight, so those run whatever hardware designers like. I am sure some of them used to run tiny 16-bit CPUs, but I'd be surprised to hear new designs run something that old.
> Have you ever looked at how useless Chuck Moore's stuff is?
I've seen someone well known in the Forth community say something like that to Chuck. I think he said "what can I do with that?"
If GA144 is useless, it's only because it has not been put to good use, in my opinion. I think it was more-or-less his answer too, IIRC.
I work with system-on-chip or system-on-modules. You know, the ARM-based chips with tons of peripherals. I also worked with similar chips before the ARM era.
The complexity of these chips is as of today, absurd. I/O is multiplexed to make the chip usable for various things, but one has to configure all of them and watch out for conflicts. Then there's also zillions of configurable clocks in order to reduce the power consumption. Solutions to problems that spawn new problems, not in the "divide and conquer" style, unfortunately. This resulted in "device tree" configuration in Linux, a runtime configuration system, because CPU companies excrete a new variant every week.
Maybe I fool myself, but I can see a GA144 bit-banging IRDA, LCD, SPI, etc. in a much more efficient and flexible way.
> (2010) is a fairly interesting experience from someone on the outside trying to work in the same way. It… didn't work that well.
And here in 2025 there's still Forth-based companies alive, like MPE or Forth, Inc.
More specifically, this author writes,
[...] it was harder than we thought. Presumably that was partly the result of not reading "Stack Computers: the new wave", and not studying the chip designs of Forth's creator Chuck Moore, either. I have a feeling that knowledgable people would have sneered at this machine: it was trivial to compile Forth to it, but at the cost of complicating the hardware.
Implementing a stack-based processor without reading the literature on them is a bit foolish, don't you think? The rest is in the same vein; I can see why a person who was essentially a Forth newbie had a bad experience with this kind of project. If this article "debunks" Forth, it is by showing that just because something it is simple, doesn't mean it is easy. Because the world is not simple and simplifying is much harder than let Complexity loose.
While I generally agree with that, the problem is teams, mixes of skill levels, and future time constraints. There's no guarantee the person doing the second implementation knows about the first one, realizes they can be pulled into a better abstraction, or has the time to do it. On the flipside, it's possible the other person (or the first person with more experience) comes up with something better than the first possible version of abstraction. So it ends up being a trade-off you have to decide on at the start, rather than a simple rule, based on team experience and how big (or small [0]) the abstraction actually us.
But that's still twice the cost, for only a potential win. Better check first that it helps somebody write better/more correct/quicker code.
E.g. we wanted to change our text-screen video driver to allow a full-screen overlay (many years ago). The programmer assigned the task was changing the text-blit code to know about certain lines on the screen that were going to be the 'nested' window, which could be 'up' and masking a subset of the window below, or 'down' and the full screen should be displayed as normal.
He'd been hacking away, changing every case in the code to 'know about' the particular window that was planned. Pulling his hair out, getting nowhere.
I suggested a simple indirection buffer, where each line of a virtual display was pointed to indirectly. To put the subwindow 'up' or 'down' you just pointed those lines of the main screen to a buffer, and pointed the lines of the subwindow to the hardware store for video text.
All his changes folded into 'access display indirectly'. Trivial. Then the up/down code became a loop that changed a pointer and copied some text.
That was actually an abstraction, and actually much shorter/faster/simpler to understand and change.
Later we could change the window dimensions, have multiple windows etc with no change to the video driver.
Well said! I have met a few of these codesmells where the actual functioning is hidden behind a bewildering maze of facades, shims, proxies and whatnot.
I guess some has had an irresistible itch to use as many patterns from the GoF book as possible.
To me it sounds like what I use Ruby to get away from.
It's rare to need facades, proxies, and shims in a dynamically typed language where the caller doesn't need to care about the type of the object they call.
In fact, most of the Gang of Four design patterns either make no sense in Ruby or are reduced to next to nothing.
The best abstractions I've seen have always come into being only after a significant amount of particularized code had already been written. It is only then that you can identify the actual common properties and patterns of use.
Early Ruby On Rails comes to mind as a great generalization of web applications in the era of Web 2.0.
Generalisation is definitely a good approach, but it’s not the only one. Another is “conceptualisation” or reducing repetition: if you find that large numbers of functions are taking the same three parameters, especially if they’re using them in similar ways, that’s a good sign there’s a concept you can introduce that makes those observations explicit.
To put it in slightly simpler terms, abstractions are generally to separate the “what” from the “how”.
Functions are the fundamental mechanism for abstraction in computing, which demonstrate that very well, in their separation between interface and implementation. The function signature and associated interface contract represent the “what”, and the function’s implementation the “how”. If the effective (often non-explicit) interface contract relies on most aspects of the actual implementation, then the “what” is almost the same as the “how”, and there is little abstraction. The greater the difference between the “what” and the “how”, the more of an actual abstraction you have.
This relates to Ousterhout’s notion of “deep modules”, which are modules whose interface is much simpler than their implementation. In other words, the “what” is much simpler than the “how”, which makes for a good abstraction.
It’s true that often one has to first implement the “how”, possibly multiple times, to get a good notion of which aspects are also still important to the “what”, and which aren’t.
Note also that generalizations go two ways: The caller needs less knowledge about the implementation, but it also means that the caller can rely less on the properties of a concrete implementation. This is again well reflected in function types. A (pure) function g: A –> B is a generalization of a function f: C –> D only if A is a subtype (specialization) of C and B is a supertype (generalization) of D. Here D could expose properties of the function’s implementation (f) that g wants to hide, or abstract from.
IMO, I dislike that kind of mixing several concepts into the same name that software engineering is full of.
Abstraction means removing details from the thing you present. Generalization means making the same representation valid for several different things.
Are abstractions that don't generalize valuable? Well, maybe there is something better to be found, but those are the bread-and-butter of software engineering; they are what everybody spends almost all of their time writing.
Are generalizations that don't abstract valuable (or even possible)? Well, not if they don't abstract at all, but there are plenty of valuable generalizations that abstract very little. Hell, we have all those standardizing organizations that do nothing more than creating those.
Are the best interfaces the ones that achieve most of both of those? Honestly, I have no idea, there are other goals and if you optimize to extreme levels, they start to become contradictory.
I would warn against conflating the concept of an interface with an abstraction just as much as I would against conflating generalizations and abstractions.
An interface often accompanies an abstraction, sometimes even represents one. But if we get down to definitions, an interface is merely something that you interact with: a function, a class, a network endpoint etc. If you write in machine code, you might not think that you are working with any kind of an interface, and certainly it's not a high level one, but you are interfacing with the machine's native instruction set. You could then argue that the instruction is an abstraction that hides what the machine is capable of. But now we're splitting hairs and about to ask silly philosophical questions like whether a car's steering wheel qualifies as an abstraction that exists to hide the axles. I would argue not. The interface's primary responsibility is to provide a mechanism of interaction, rather than to reduce complexity (though a good interface is simple and intuitive).
Both you and the author of the article posit a similar definition of 'abstraction'. From the article:
> An abstraction is only as good as its ability to hide the complexity of what lies underneath.
And from your comment:
> Abstraction means removing details from the thing you present.
I would actually argue that both, while very close, are missing the mark.
An abstraction exists to reduce a concept down to its essentials.
This doesn't necessarily contradict the definitions offered, but I think there is a nuance here that, if missed, causes the offered definitions to become useless.
The nuance is in deciding what is essential or necessary. If, definitionally, you choose to dispense with acknowledging the essential... well then you get the problems that the author is writing about. You get shit abstractions because no one bothered to think in terms of what they INCLUDING rather than DISPENSING with.
Yes, obviously, we abstract in an attempt to simplify. But that simplification needs to come from a positive rather than a negative.
In other words: What is the thing for? What does it do?
"Hides complexity" is the shittiest answer that anyone could ever offer as a response when faced with a given problem. First, what is the complexity that we are trying to reduce? Secondly, why are we trying to reduce it? Thirdly, when we have achieved our primary goal of reducing the complexity, then what does the thing look like? What value does it offer over interfacing directly with the "thing" being abstracted?
Abstractions in computer science are extremely valuable. So valuable, I would offer, that any time we hear engineers decry abstractions or point to abstractions as the root of all evil we ought to ask a very pressing question: "who hurt you?"
A good engineering solution is a simple one, and an abstraction is intended to be a simpification mechanism. But it doesn't have to necessarily simplify the intuitive understanding of the problem at hand. This is a good goal if you can achieve it, don't get me wrong. But that abstraction might exist so you can swap vendors in the future if that's a business priority. Or because you've identified some other element / component in your system that will be difficult to change later. So you stick it behind an abstraction and "Program to Interfaces" rather than "implementations" so that you can simplify the process of change down the road, even if it comes at an more immediate cost of making the code a bit less intuitive today.
Everything in software is tradeoffs. And a good abstraction exists so that we can focus on requirements and programming to those requirements. This is a smiplification when done properly.But the focus ought to be on defining those requirements, of asking "what should simple look like?"
I agree with what you're saying, and I would phrase it as "write the abstractions that reflect the essential complexity". The whole program should minimally reflect the essential complexity of the problem. Of course actually doing that isn't easy, but the result is obviously a simple solution for a given problem. It becomes another challenge to maintain and refactor: the question of changing problem constraints and being able to minimally change a program to match.
Why are ADTs like stacks, queues, and hashmaps so popular? Why are languages like C or Forth so highly praised for a high ceiling for performance and efficiency? Because they are usually "about as good as it gets" to solve a problem, "what you would've more or less done anyways". Maybe on a GPU, a language like C isn't quite fit, because the problem has changed. Make tools (e.g. CUDA) that reflect that distinct complexity.
Great point, and agree that generalization makes for the clearest wins from abstraction.
But there are also cases where there is no generalization, but the encapsulation / detail hiding is worthwhile. If you have a big function and in the middle of it you need to sort some numbers, you would probably implement a Sort routine to make the control flow much easier to understand - even if you only use the function once (let’s pretend there’s no sort functionality in standard library).
Curious if others agree, and what heuristics you use to decide when implementation encapsulation is worthwhile.
I saw this in my codebase firsthand. At first, we had to communicate with one device. Then two. For that second device, I subclassed the class that handled the communication with the first thing, and changed 3 methods in the subclass. Now it was possible to substitute that second device for the first, allowing a different product configuration. After a few years, we had to support a third device type, and now they wanted it to be possible to have more than one of them, and to mix and match them.
Supporting the third device was handed off to a junior dev. I pointed him at my subclass and said to just do that, we'd figure out the mixing and matching later. But he looked at my subclass like it was written in Greek. He ended up writing his own class that re-imagined the functionality of the superclass and supported the new device (but not the old ones). Integrating this new class into the rest of the codebase would've been nigh impossible, since he also re-implemented some message-handling code, but with only a subset of the original functionality, and what was there was incorrect.
His work came back to me, and I refactored that entire section of the code, and this is when the generalization occurred: Instead of a superclass, I took the stuff that had to be inherited and made that its own thing, having the same interface as before. The device communication part would be modeled as drivers, with a few simple functions that would perform the essential functions of the devices, implemented once per device type. I kept the junior dev's communication code for the new device, but deleted his attempt to re-imagine that superclass. Doing it this way also made it easy to mix and match the devices.
A network close call at a high level closes a network socket. But at the tcp level there's a difference between close and reset. Which do you want? Your api has removed that choice from you, and if you look you will have no idea if rhe close api does a close or a reset.
Is the difference important? If depends. If you have a bunch of half open sockets and run out of file descriptors then it becomes very important.
Another example: you call read() on a file, and read 10k bytes. Did you know your library was reading 1 byte at a time unbuffered? This abstraction will/can cause massive performance problems.
My favorite one is when a programmer iterates over an ORM-enabled array. Yes, let's do 50,000 queries instead of one because databases are too complicated to learn.
Just like any tool, abstraction has costs and benefits. The problem is that lots of people ignore the cost, and assume the benefit.
I think an important distinction is hiding details from other parts of the program, and having details being hidden from you.
99.9% of the time I don't care what the tcp socket closes with as long as it isn't leaking a resource.
And if I did care, then I picked the wrong level of network abstraction to engage with. I should've used something more raw.
Regarding ORM arrays. I have myself recently debugged such a case. I chortled a bit at the amateur who wrote the code (me last year) and the schmuck who accidentally wrapped it in a loop (me 3 weeks ago). Then I changed it slightly to avoid the N queries and went on with my day. No need to lambast the tooling or the programmers. Just write something maintainable that works. No need to throw the entire ORM away just because we accidentally made a web page kinda slow that one time.
And don't worry, I too lament when web pages I don't control are slow. You may rest uneasily knowing that that page would be slow regardless of whether ORMs existed because it is not slow because of ORMs, but because there is no incentive for the business to care enough to make it faster.
>> But at the tcp level there's a difference between close and reset. Which do you want? Your api has removed that choice from you, and if you look you will have no idea if rhe close api does a close or a reset.
If you are doing a „TCP application” of course it makes no sense to abstract the TCP layer. Is not about cost. Now if you have an application that has to communicate somehow with other system, and you want to not depend on specific protocols, then the communication part should abstract away that part.
How to deal with your example? Well, if you can say “I will always want X, you can make a configuration option “TCP.close” or “TCP.reset”. If “it depends” then you have to build the logic for the selection in the abstraction layer, which keeps hidden.
Pretty easy to give generic advice without examples.
“Write more tests, but not too many”
“Use good abstractions where appropriate?”
“The next time you reach for an abstraction, ask yourself: Is this truly simplifying the system? Or is it just another layer of indirection?”
It’s easy to create a strawman here (the FactoryAdaptorMapper or whatever) but in reality this kind of generic advice doesn’t help anyone.
Of course people want to use good abstractions.
That’s not the problem.
The problem is being able to tell the difference between generic arbitrary advice (like this post) and how your specific code base needs to use abstractions.
…and bluntly, the only way to know, is to either a) get experience in the code base or b) read the code that others have left there before you.
If it’s a new project, and you’re not familiar with the domain you’ll do it wrong.
Every. Single. Time.
So, picking “good” abstractions is a fools game.
You’ll pick the wrong ones. You’ll have to refactor.
That’s the skill; the advice to take away; how to peel back the wrong abstraction and replace it with your next best guess at a good one. How to read what’s there and understand what the smart folk before did and why.
…so, I find this kind of article sort of arrogant.
Oh, you want to be a great programmer?
Just program good code.
Use good abstractions. Don’t leave any technical debt. Job done!
…a few concrete examples would go a long way here…
I can see the value of examples, but in this case I appreciate the post largely for its universality and lack of examples. On reading it, examples from past and present experience spring immediately to mind, and I'm tucking this away as a succinct description of the problem. Maybe I can share it with others when more concrete examples come up in future code review.
A principle takes skill the apply, but it's still worth stating and pondering.
> examples from past and present experience spring immediately to mind
Examples of what?
Picking the wrong abstraction? Regretting your mistakes?
I can certainly think of many examples of that.
How you unwrapped an abstraction and made things better by removing it?
I have dozens of battle stories.
Choosing not to use an abstraction because it was indirection?
Which is what the article says to do?
I’m skeptical.
I suspect you’ll find most examples of that are extremely open to debate.
After all, you didn't use the abstraction so you don’t know if it was good or not, and you can only speculate that the decision you made was actually a good one.
So, sharing that experience with others would be armchair architecture wouldn't it?
That’s why this article is arrogant; because it says to make decisions based on gut feel without actually justifying it.
“Is this truly simplifying the system?”
Well, is it?
It’s an enormously difficult question to answer.
Did it simplify the system after doing it is a much easier one, and again that should be the advice to people;
Not: magically do the right thing somehow.
Rather: here is how to undo a mistake.
…because fixing things is a more important skill and (always) magically doing the right thing from the start is impossible; so it’s meaningless advice.
That’s the problem with universal advice; it’s impossible to apply.
It might not give enough examples to show you how to do something right, but I think it's enough to warn against doing something wrong, namely introducing an indirection that doesn't provide any of the benefits of abstraction.
My colleagues invariably refer to indirections as abstractions, and it's a frustrating sort of name-squatting, because you can't usefully discuss the tradeoffs of abstractions if they're actually talking about indirections.
That said, the article does drop the ball by seeming to use the terms interchangeably.
Reminds me of an old Java Android project I encountered.
EVERY class implemented an interface.
98% of interfaces had one implementation.
Every programmer was applying a different programming pattern.
A lot of abstractions seemed incomplete and did not work.
Proguard (mostly used for code obfuscation for Android apps) definitions were collected in the top module even though the project had multiple modules.
Half of the definitions were no longer needed and the code was badly obfuscated.
Problems were solved by continuesly adding classes and checking what sticks.
The UI was controlled by a stateful machine. State transitions were scatter everywhere in the code with lots of conditions in unforeseen places.
Legacy code was everywhere because no one wanted to risk a very long debugging session of an unforseen change.
No API definitions. Just Maps that get send via REST to URLs.
By biggest mistake was to not directly rewrite this project when I entered the team.
We did after one year.
I'm not sure what it's called (abstraction vs. indirection) but I dislike when everything needs a class/object with some odd combination of curried functions. Some programming languages force this on you more than others I think? As a contrived example "StringManager.SlicingManager.sliceStringMaker(0)(24)(myStr)", I've seen code that reminds me of this and wonder why anyone uses a language where this not only an acceptable idiom, but a preferred one.
Interesting read, although I don't agree with everything. I like the distinction between different qualities of abstractions, made in the beginning. The following bashing of abstractions is too generalized for my taste.
The best part comes close to the end:
> Asymmetry of abstraction costs
> There’s also a certain asymmetry to abstraction. The author of an abstraction enjoys its benefits immediately—it makes their code look cleaner, easier to write, more elegant, or perhaps more flexible. But the cost of maintaining that abstraction often falls on others: future developers, maintainers, and performance engineers who have to work with the code. They’re the ones who have to peel back the layers, trace the indirections, and make sense of how things fit together. They’re the ones paying the real cost of unnecessary abstraction.
> Think of a thin wrapper over a function, one that adds no behavior but adds an extra layer to navigate. You've surely encountered these—classes, methods, or interfaces that merely pass data around, making the system more difficult to trace, debug, and understand. These aren't abstractions; they're just layers of indirection.
“No added behaviour” wrapper functions add a lot of value, when done right.
First off, they’re a good name away from separating what you’re doing from how you’re doing it.
Second, they’re often part of a set. E.g. using a vector for a stack, push(x) can be just a call to append(x), but pop() needs to both read and delete the end of the vector. Push in isolation looks like useless indirection, but push/pop as a pair are a useful abstraction.
A consequence of adding these two points together is that, if you have a good abstraction, and you have a good implementation that maps well to the abstraction, it looks like useless indirection.
Another consequence is that those pass-through wrapper functions tell you how I think the implementation maps to the domain logic. In the presence of a bug, it helps you determine whether I got the sequence of steps wrong, or got the implementation wrong for one of the steps.
Ultimately, the two aren’t completely independent —indirection is one of the tools we have available to us to build abstractions with. Yes, people misuse it, and abuse it, and we should be more careful with it in general. But it’s still a damned useful tool.
239 comments
[ 154 ms ] story [ 3423 ms ] thread> Abstractions are also the enemy of simplicity. Each new abstraction is supposed to make things simpler—that’s the promise, right?
Not exactly, no. The purpose of abstraction is to hide implementation detail, and thereby insulate one part of the codebase/application/system from variations in another. Graphics APIs for example - yes your code may be simpler for not having to deal with the register-level minutiae of pushing individual triangles, but the core benefit is that the same code should work on multiple different hardware devices.
Good abstractions break a codebase up into compartments - if you drop a grenade in one (change the requirements for example), then the others are unaffected and the remedial work required is much less.
That is if you think you'll probably need multiple implications, delay introducing an abstraction until you actually do.
Also, there are different ways of providing abstraction.
Perhaps you don't need to abstract the entire implementation but, as an example, rather change one parameter from passing a value to passing a function returning a value.
- not doing extra work now if it's not necessary yet and if it's as cheap to do later
- delaying building a thing till you know what that thing should be
I came here to say this. Attractions act as a bridge between things which allow those things to change independently.
Using an ORM allows my program to easily work against multiple sql databases. Using a compiler allows me to target different hardware. Using standard protocols allows me to communicate with different programs. Using libraries to do say email hides me from those protocols and allows me to adapt to service providers Using APIs not protocols.
In other words the abstraction is designed to hide a layer (which can change) from a program not interested in that level of change.
The key is to stop abstracting when the program cares. By all means encapsulate rules and processes, but they're a direct implementation of those rules and processes.
One can argue my "calculateLeaveForEmployee" function is an "abstraction" - but that would be a misnomer. Since there's only one set of rules in play (the set that matters now) its an implementation. An abstraction supports (at least) two things at the same time.
A curious example, since most developers who've worked on any project with significant amount of data in a database would likely disagree.
IMO, ORMs mostly allow using programming-language-of-choice as a syntax for relational queries instead of constructing it by hand on top of serialization and deserialization of objects into rows and vice versa.
In the small-to-medium scale cases I’ve seen, the main re-use you get for a particular mapping is when an object is used as a child node elsewhere. If it’s a one-off you’re essentially writing a bespoke shorthand wrapper for some custom SQL. Which may be a convenient shortcut or prettifier, but isn’t really functioning as an abstraction. Net net it seems like more overhead. Or like the real benefit is constraining database design to closely align with object design (which could be valuable but isn’t a function of abstraction).
This is all assuming code ownership across the stack. With a separate team managing data representation and handing reliable object contracts “over the wall” I could imagine a situation where the consumer gets to think purely in objects. I just haven’t observed that in practice. E.g. What if the contact has two address records? What if the number of purchase records exceeds the maximum result size? If things like that come up regularly enough, it’s worse when you’re waiting on a separate team to fix the leaks.
There is a compelling argument that absent the n+1 problem, it could be a leak-free abstraction. And, in practice, SQLite doesn't suffer from the n+1 problem... But that is definitely true in a lot of cases, particularly where the n+1 problem is unavoidable.
> If it’s a one-off you’re essentially writing a bespoke shorthand wrapper for some custom SQL.
While the original comment also conflated query building and ORM, I am not sure that we should. The intersection of query building and ORM was historically referred to as the active record pattern. I suspect the ActiveRecord library, from Rails, calling itself an ORM is where things started to become confused. Regardless, whatever you want to call the data mapping and query building, they are undeniably distinct operations.
It may be fair to say that query builders are no more than an indirection. That said, I posit that they only exist as an abstraction to model composability on top of SQL. If SQL naturally composed through basic string concatenation, I expect the use of query builders would quickly disappear.
But instead of this, an ORM usually provides you with a syntax (that will pass syntax checks and have highlighting) that matches the language, which is all nice and good because dealing with arbitrary strings does suck.
I've seen ORMs being used for query composition way before Rails even existed: I believe Hibernate had HQL with their first release in 2001, just like SQLObject did in 2002. I am sure neither of those "invented" the pattern.
Note that fetching objects using an ORM library, filtering and such is what I also consider query composition (it just happens behind the scenes).
It is not. Not even in a simple case. Consider:
base + cond1 + cond2 or any similar combination will not produce a valid query. It could if SQL had some thought put into it. There are many languages that have no problem with such things. But that irrational fear of moving beyond the 1970s when it comes to SQL...The only realistic way to assemble queries is to prepare an AST-like structure to figure out where the pieces fit, and then write that out to a final query string. In practice, that means either first parsing the partial queries into that structure (hard) or providing a somewhat SQL looking API in the host language that builds the structure (easy). Unsurprisingly, most people choose what is easy.
> I am sure neither of those "invented" the pattern.
None of these invented the pattern. But the invention point is irrelevant anyway. You must have misread something?
It does not really abstract away relational data, it instead maps it to object/class hierarchy. And every single ORM I've used also provides an interface to map function/method/expression syntax to SQL queries to get back those rows (adapted into objects).
Now, in a very broad sense, mapping would also be an abstraction — just like an "indirection" would be — but considering the narrower definition from the OP (where indirection is not an abstraction), I think it's fair to say that ORM is also not an abstraction.
How many times have you required that your program runs against different sql databases without modification?
I mean, how many times have you required your plane to be able to fly in the different atmospheres of different planets of the Solar System?
Unless you are NASA, I suggest you cut the complexity and just make a plane that can fly well on Earth's atmosphere.
Our main commercial product currently supports 2 database engines, and we'll be offering a 3rd next year. For enterprise offerings it's pretty common for the client to prefer, or outright specify, the engine.
Commodotizing your complementary technology is a good way to not become dependent on any specific database, and hence can pivot quickly when required.
Fortunately I don't have a plane, so I don't have code to fly in any atmosphere.
The rest of us go with Postgres, or SQL server, or Oracle, but definitely don't have to prepare our systems to run in PostgreSQL on Mondays, on SQL server on Tuesdays and so on.
But sometimes a new semantic level isn’t needed. Abstraction gets so much press when you might just need some good ol’ fashioned information hiding and separation of concerns.
Because CRDTs have been in the zeitgeist a lot lately, I want to pick them as an example of a "good" abstraction.
CRDTs have mathematical properties which can be described and understood independently of a specific implementation. And importantly, you can judge whether an implementation is correct with reference to these abstract rules.
This means that when using a CRDT, you largely can treat it as a reliably-solved problem. Once you understand the concepts, and work out how to use the library you've picked, you don't have to think about the details. Though that doesn't mean sometimes the behaviour can be surprising:
https://www.moment.dev/blog/lies-i-was-told-pt-1
TCP and HTTP are great examples too, though interestingly I don't know if they rely on mathematical definitions so much as just being extremely widespread to the point that reliable implementations are available anywhere you care to write code.
I like this article which also leans on the Dijkstra quote:
https://www.pathsensitive.com/2022/03/abstraction-not-what-y...
I wrote this [0] previously, but it still applies. IMO, as a dev, there are times where you really do need to think somewhat deeply about infrastructure choices. Unfortunately, knowing when you need to care practically requires you to already understand it.
[0]: https://news.ycombinator.com/item?id=40834759
that's exactly what you do to implement an abstraction isnt it?
That you used wood doesn’t mean you built a house.
Indirection is about providing a common language, and interface if you will, for talking to different implementations.
Separating concerns is another strategy altogether for managing resources.
In that sense, we can correctly say that even encapsulation etc is a form of (colloquial) abstraction, but not Dijkstra's use of the word abstraction.
On practice, there exist platonic abstractions that hide mechanisms and can be this way, but you also need to abstract parts of your requirements. And you will never be able to achieve that while abstracting your requirements.
Dijkstra was an academic after all, and academics usually don't care about complex requirements.
Organizing your code into components that are as independent as possible is a good practice and is the pursuit of modularity. A proper abstraction on the other hand, is a generalization that simplifies a conceptual layer in your code base.
Abstraction often enables greater modularity as a consequence, but they are not the same thing. For example, in the problem of text editing, people eventually realized that the manipulation of text is typically line oriented. Thinking of a text file as a collection of lines may seem like an obvious and modest abstraction, but it works well. This abstraction, in turn, leads to other couplings (e.g. line oriented motion is highly dependent on the line abstraction), but it also leads to potential modularity (e.g. printer code may no longer need to understand exactly how a display renders each character of text in a grid, instead, it too can work on "lines"). Good abstractions support modularity to the extent that they establish a shared domain of objects to communicate about and across systems, but they do not necessarily produce modularity in themselves.
I disagree that considering a text file as a set of lines really qualifies as an abstraction; it's just a different representation of the data - you could instead choose to use a list of words or a tree or whatever. An abstraction is a general interface to a concrete thing that allows you to substitute different but similar concrete things without changing the consumer of those things. The Linux filesystem is a great example - I can 'cat' basically anything that has a path, and some driver will pull data from the associated endpoint and display the data on screen. "File" is the abstraction, and consumers of files need not care about the specifics of communicating with the underlying devices. Such a system is also modular, but it's hard not to be when your abstraction is well conceived.
Perhaps we're saying the same thing with different words?
Abstractions almost always represent data differently than as it is in its most concrete form (e.g. lists and trees are abstractions both often built of vectors and pointers), but that is not the purpose of abstraction. To continue with the original example, the representation of text as a sequence of lines is a sort of abstraction, but rarely the most useful one: words, sentences, paragraphs and/or syntax trees are often what you want, and they are useful because they give us a representation of the raw text that is aligned with its purpose, not because we can implement them in multiple ways.
Furthermore, while we want to do some information hiding when implementing these abstractions (think particularly of syntax trees), that is not the purpose of having these abstractions - what we are hiding is the infrastructure to make it work, not any information conveyed by the text itself.
I prefer the concept of orthogonality: you have an entire plane where your domain requirements can move in, but the way you make persistence reliable is completely orthogonal to that plane. It should be implemented in a way that does not change when, say, your customer account becomes a composite of of multiple identities. And neither should the way you organize UI components change, or your transportation tooling change. That entire transportation layer stack, from ethernet all the way up to your domain model generator of choice.
Did I say layers? Yeah, I did. You'll often see the orthogonal aspects themselves consisting of layers. But that's an implementation detail and those layers tend to be more pragmatic than dogmatic, as in the way TCP goes out of its way to avoid hiding any of the addressing of the underlying IP. TCP absolutely is a layer, but it has zero ambition to perhaps run on top of something other than IP, or hide differences between IPv4 and IPv6 from higher layers. It focuses on one thing, implementing streams on top of packets, again a problem nicely orthogonal to the problem of routing those packets (what IP does) or anything built on top of those streams.
Parnas' criteria for when to split code into modules[0] is very compelling: modules should enforce information hiding, where the information of each module is how a key decision is implemented. Key decisions should be orthogonal: e.g., encryption shouldn't inform filesystem access and vice versa. This fits in nicely with practices for determining solutions for a given problem: nail down key decisions (or uncertainties around them) early on and encode them through modules and interfaces.
A logical application of Parnas' module criteria is, naturally, in network stacks. According to [1], although coarse program flow generally follows the OSI layers, the fine control flow may hop around quite a bit. For some time people tried to keep the OSI layers as inspiration and use hacks like upcalls to selectively violate layering, which is a sure sign that the layers are unhelpful. Instead, modules should reflect functionality like encryption, NIC interfacing, and routing. Aim for a flatter, horizontal architecture like a B-tree, not a contrived vertical tower with hacks to cross layers "incorrectly". There will be hierachies and layers, but they should be obviously necessary now. Layering is not the end or even the means but merely how things are.
A program's capability (functionality) is determined by the capabilities of its dependencies, and splitting up functionalities well is important. I use this language to relate it to capability-based security[2]. After all, "security" is basically just a stricter form of program "correctness", and just as a terminal emulator's input text handling capability shouldn't have much to do with its window event handling capability because malicious input shouldn't cause dangerous graphics bugs, there isn't much logical reason to have overlap regardless. They govern separate devices, protocols, and purposes. Capabilities along the lines of capability-based security reify the interfaces and provide strict enforcement.
Lastly, flexibility around protocols should be added or discard as given by the precise problem at hand. Don't preemptively choose some extreme, but instead be informed by what is actually desired. It will yield substantial implementation complexity and performance detriments[3] to over-generalize, and of course over-specializing is just inadequate.
Technically, that's encapsulation, though the sentiment is close, I think.
I rather view it as a matter of semantics. At one low level, you have operations that deal with some concrete interface or API, etc. You bundle those operations up behind an abstraction, providing methods whose names involve your application domain. Perhaps they are still at a technical level, and you bundle those up behind another abstraction, whose method names involve your business domain.
Yes, the lower level details are hidden from the higher levels, but the hiding is not the point. The point is to be able to write code that readily corresponds to the problem you are trying to solve.
> In object-oriented programming languages, and other related fields, encapsulation refers to one of two related but distinct notions, and sometimes to the combination thereof:[5][6]
> - A language mechanism for restricting direct access to some of the object's components.[7][8]
> - A language construct that facilitates the bundling of data with the methods (or other functions) operating on those data
The hiding is absolutely the point of abstraction, in the sense that caller can manipulate the subsystem without knowing about (important) inner details of it. As I said, graphics drivers are a great example - I just want to put a triangle on the screen, I don't want to care about the registers I have to set on an NVidia card vs those on an AMD card, or what their memory maps look like, and I don't want to have to rewrite my code when they release a new generation of hardware. Drawing a triangle is an abstraction, hiding away the details of how a specific graphics card achieves that.
Think about what the word 'abstract' means - the idea of a 'human' is an abstraction for bundles of molecules conforming to a general pattern with broadly similar qualities but almost infinite variability. The word conveniently hides a wealth of detail that we usually don't need to think about; if I tell you there are three humans in a room, you can make use of that information without knowing anything about their exact physical attributes or who they are.
I would refer to what you're describing as 'modelling' - these are related topics but I don't see them as the same thing.
https://web.archive.org/web/20071214085409/http://www.itmweb...
I find myself saying this over and over again; if the abstraction does not bring the code closer to the business domain; if it does not make it easier for you to explain the code to a non-technical person, then it's a poor abstraction.
Inventing technical constructs which simply shift the focus away from other technical constructs adds no value at all. Usually such reframing of logic only serves the person who wrote 'the abstraction' to navigate their own biased mental models, it doesn't simplify the logic from the perspective of anyone else.
https://www.youtube.com/watch?v=aSEQfqNYNAc
Any function call is abstraction after all. Unabstracted you would just inline that code or use goto.
People only notice bad abstractions.
The two classic "hard problems" of computer science - cache invalidation and naming things - are both aspects of abstraction. Cache invalidation is a special case of making sure the abstraction does what it's supposed to, and naming is the most important part of causing the abstraction to have meaning.
I think that microservices (or at least, smaller services) and modern languages allow the code to stay more manageable, to the point where Java devs now are able to dismiss Spring and go for a much simpler Quarkus.
They aren't just fundamental virtues in software development, they're the fundamental basis of all cognition. If I twiddled with every bit in my computer I'd never write a hello world program, if I wrestled with every atom in my coffee cup I'd never drink a sip of coffee.
Abstraction and information hiding is the only way we ever accomplish anything because the amount of information fitting in our heads is astonishingly small compared to the systems we build. Without systems of abstraction we would literally get nothing meaningful done.
Abstraction is good, in the way that leverage is good in the physical world: it is not always necessary, but people who are aware of the tool are vastly more capable than those who are not.
This is false. One can read up on Theorem's for Free by Wadler to see that not all abstractions are leaky.
What examples of non-leaky abstractions do you have?
I could imagine something like "newtonian physics" but that leaks into my daily life every time I fire up google maps and get a GPS fix.
The OP's example of TCP seems close to the mark, but to be totally honest I'm not convinced. Every time I have to run ping to check whether my hung connection is due to connectivity, I'm breaking the abstraction. And I've had to debug network issues with Ethereal (yes, I'm dating myself). TCP does leak, it's just that most people don't know what to do with it when it does.
If you write down a function with signature
then it must be the identity function, if you do not use "malicious" extensions of the type system.But what is the performance of the identity function?
Here is an identity function:
In other words: I can hide pretty much arbitrary computation in my identity function.Users of my identity functuon will notice that it is wicked slow (in reality, I let my identity function compute Busy Beaver 5, before doing nothing). Their complaints are evidence of leaky abstraction.
Now you might have a smart optimizing compiler that knows about Thm4Free... But that's another story.
In fact, my own framework for small LoB webapps is based entirely around this premise.
I'd like to say this has served me well. It's the reason I never went for JSX or other frameworks that put logical code into templates or things like that. That is one abstraction I found unhelpful.
However, I've come around to not taking that advice as literally as I used to. Looking back over 25 years of code, I can see a lot of times I tried to abstract away display code in ways that made it exceedingly difficult to detect why it only failed on certain pieces of data. Sometimes this was sheep shaving tightly bound code into generic routines, and sometimes it was planned that way. This is another type of abstraction that adds cognitive load: One where instead of writing wrappers for a specific use case, you try to generalize everything you write to account for all possible use cases in advance.
There's some sort of balance that has to be struck between these two poles. The older I get, though, the more I suspect that whatever balance I strike today I'll find unsatisfactory if I have to revisit the code in ten years.
Done is better than perfect until you’re reviewing the code in 10 years (or maybe 3 years with a more adept set of eyes over your shoulders).
These days I ask my teams to be less clever and more simple.
Simple usually wins over clever in the long run.
I don't try to generalize it, I just try to make the separation clear using functions/methods/classes. Instead of having the post button's handler directly send the request, I create a function inside the `api` module that does it. It does have the effect on putting names on code patterns inside the project.
Sometimes you just need a form to do what it says on the label.
Basically arguments for a template had a form of a tree prepared by the controller function. The template could only display values from the tree (possibly processed by some library function, for example date formatter), hide or show fragments of html dependaning on the presence or absence of some value or branch in the tree, descend into a branch of a tree and iterate over an array from the tree, while descending into one iteration at a time. Also could include subtemplate feeding it a branch of the tree. This was enough to build any UI out of components and kept their html simple, separate and 100% html. Even the template logic had a form of html comments. One could open such template in any html editor including visual ones. It was all before advent of client side frameworks.
You could mimic this in React by preparing all data in render method as a jsonlike tree before outputting any JSX tag and limit yourself inside JSX part to just if and map(it =>{}) and single value {}
1. pages, each of which may or may not have a parent pageID; the entire tree is read and parsed to generate the menu system and links/sub-links
2. modules which reference HTML templates to load through a parser that replaces areas with {{handlebar}} variables written by the clients through their own API
3. something like page_modules which locate the modules on the page,
4. a renderer which uses the above stuff to make the menu system, figure out what URL people are trying to get to, load the page, load the modules in the page, and fill in the crap the client wrote inside the {{handlebars}}
This has worked so well for so long that I can basically spin up a website in 15 minutes for a new client, let them fill in anything I want them to by themselves, throw some art on it and call it a day.
It worked so well that I ended up writing a newer version that uses a bunch of Javascript to accomplish basically the same thing with smoother transitions in a single-page app that doesn't look or act like an SPA, but it was basically pointless.
> It's the reason I never went for JSX or other frameworks that put logical code into templates or things like that
That's not mixing business logic with display code, your confusing (or conflating) 2 entirely different things here.
Letting any control flow into your HTMX/JSX templates is begging for trouble. It boggles the mind that people abandoned PHP - where at least the output was static and checkable - for something exactly like mixed HTML/code where some processing was done on the front end and everything was supposed to be dehydrated. Only to pivot again to hydration on the back-end.
JSX and React came on the scene 10 years after I first realized using the client for anything logical was an anti-pattern. Back then, I remember people would write things like:
CustomerFormForClientX extends CustomerFormForClientZ [...]
and put a bunch of validation on the client side, too. Clients are dumb terminals. That's it. I'm not confusing the use of logic-in-JSX with business-logic-in-display. One only has to look at every basic JSX example on the web that's taught to newbies to see a million ways it can go wrong.
You really just want to split those things out of the UI component and keep minimal code in the component file. But having it be code free is too much for real client side apps.
Modern apps have very interactive UI that needs code. Things like animation management(stop/start/cancel) etc… have subtle interactions. Canvas zoom and pan, canvas render layouts that relate to UI, etc… lots of timing and control code that is app state dependent and transient and belongs in components.
I apply the simple principle and move any app state management and business logic out of the component UI code. But still lean into logical code in UI.
But then you hear people describe abstraction ahem abstractly. "Abstraction lets you think at a higher level," "abstraction hides implementation detail," and it's clear that neither of those things are really abstractions.
As the OP mentions, we have a great term for those long chains of one-line functions: indirection. But what is TCP? TCP is a protocol. It is not just giving a higher-level way to think about the levels underneath it in the 7-layer networking model. It is not just something that hides the implementations of the IP or Ethernet protocols. It is its own implementation of a new thing. TCP has its own interface and its own promises made to consumers. It is implemented using lower-level protocols, yes, but it adds something that was fundamentally not there before.
I think things like TCP, the idea of a file, and the idea of a thread are best put into another category. They are not simply higher level lenses to the network, the hard drive, or the preemptive interrupt feature of a processor. They are concepts, as described in Daniel Jackson's book "The Essence of Software," by far the best software design book I've read.
There is something else that does match the way people talk about abstraction. When you say "This function changes this library from the uninitialized state to the initialized state," you have collapsed the exponentially-large number of settings of bits it could actually be in down to two abstract states, "uninitialized" and "initialized," while claiming that this simpler description provides a useful model for describing the behavior of that and other functions. That's the thing that fulfills Dijkstra's famous edict about abstraction, that it "create[s] a new semantic level in which one can be absolutely precise." And it's not part of the code itself, but rather a tool that can be used to describe code.
It takes a lot more to explain true abstraction, but I've already written this up (cf.: https://news.ycombinator.com/item?id=30840873 ). And I encourage anyone who still wants to understand abstraction more deeply to go to the primary sources and try to understand abstract interpretation in program analysis or abstraction refinement in formal verification and program derivation.
I just re-read your article but unfortunately I still struggle to really understand it. I believe you have a lot of experience in this, so I'd love to read a more dumbed down version of it with less math and references to PL concepts and more practical examples. Like, this piece of code does not contain an abstraction, because X, and this piece of code does, because Y.
Keep up the good work!
For example, in the "TV -> serial number" abstraction, if I were to define only one operation (checking whether two TV's are the same), would it make it a good abstraction, as now it is both sound and precise?
And what are the practical benefits of using this definition of abstraction? Even if I were to accept this definition, my colleagues might not necessarily do the same, nor would the general programming community
It would!
> And what are the practical benefits of using this definition of abstraction?
Uhhh...that it's actually a coherent definition, and it's hard to think or speak clearly without coherent definitions?
If you're talking about using it in communication, then yeah, if you can't educate your coworkers, you have to find a common language. They should understand all the words for things that aren't abstraction except maybe "concept," and when they use the word "abstraction," you'll have to deduce or ask which of the many things they may be referring to.
If you're talking about using it for programming: you kinda can't not use it. It is impossible to reason or write about code without employing abstraction somewhere. What you can do is get better about finding good abstractions, and more consistent about making behavior well defined on abstract states. If you're able to write in a comment "If this function returns success, then a table has been reserved for the requesting user in the requested time slot," and the data structures do not organize the information in that way, and yet you can comment this and other functions in terms of those concepts and have them behave predictably, then you are programming with true abstraction.
In this case, not programming with true abstraction would mean one of two things:
1. You instead write "If this function returns success, then a new entry has been created in the RESERVATIONS table that....", or
2. You have another function that says "Precondition: A table has been reserved for the user in this timeslot," and yet it doesn't work in all cases where the first function returns success
I think it's pretty clear that both ways to not use true abstraction make for a sadder programming life.
I'll have to muse about what the more dumbed down version would look like (as this version is already quite dumbed down compared to the primary sources). It wouldn't be quite a matter of saying "This code contains an abstraction, this other code doesn't," because (and this is quite important) abstraction is a pattern imposed on code, and not part of the code itself.
We do have a document with a number of examples of true abstraction — written in English, rather than code, in accordance with the above. It's normally reserved for our paying students, but, if you E-mail me, I'll send it to you anyway — my contact information easy to find.
I find that I need to debug my abstractions frequently, while I’m first writing my code, then I never need to dig into them, ever again, or they do their job, and let me deal with adding/removing functionality, in the future, while not touching most of the code.
That’s why I use them. That’s what they are supposed to do.
Because they are abstractions, this initial debugging is often a lot harder than it might be for “straight-through” code, but is made easier, because the code architecture is still fresh in my mind; where it would be quite challenging, coming at it without that knowledge.
If I decide it’s a “bad abstraction,” because of that initial debugging, and destroy or perforate it, then what happens after, is my own fault.
I’ve been using layers, modules, and abstractions, for decades.
Just today, I released an update to a shipping app, that adds some huge changes, while barely affecting the user experience (except maybe, making it better).
I had to spend a great deal of time testing (and addressing small issues, far above the abstractions), but implementing the major changes was insanely easy. I swapped out an entire server SDK for the “killer feature” of the app.
In computing, we emphasize the communicational (i.e. interface) aspects of our code, and, in this respect, tend to focus on an "abstraction"'s role in hiding information. But a good abstraction does more than simply hide detail, it generalizes particulars into a new kind of "object" that is easier to reason about.
If you keep this in mind, you'll realize that having a lot of particulars to identify shared properties that you can abstract away is a prerequisite. The best abstractions I've seen have always come into being only after a significant amount of particularized code had already been written. It is only then that you can identify the actual common properties and patterns of use. Contrarily, abstractions that are built upfront to try and do little more than hide details or to account for potential similarities or complexity, instead of actual already existent complexity are typically far more confusing and poorly designed.
Data = representations = abstractions
This article is so fundamentally lost that it forgets what computers are for.
Computers exist to implement abstractions.
They can do that too, but the cache miss cost already makes that expensive.
"10" is the representation of a data; "0xA" is another representation of the same data.
Not being able to touch something doesn't make it an abstraction. Light is not an abstraction, a contract is not an abstraction. 10 is not an abstraction, it is an ordinal [1].
"Isomorphism" is an abstraction. It doesn't name a particular data or value, but a class of functions that share common properties. A function template or functions written in a dynamically typed language can describe a particular group of isomorphism.
[1] https://en.wikipedia.org/wiki/Set-theoretic_definition_of_na...
10 in a database record of a warehouse system may be an abstraction for 10 items of a given type in a warehouse.
It seems to me that all of those things very much are abstractions. They are not the utmost level of abstraction, but they are abstractions!
(Actually "a contract," which at first I thought was the clearest win, I'm now not sure about. On reflection, it seems like a concretization, turning abstract ideas of trust and reliability into concrete conditions under which the contract has or has not been met.)
The notion is so fuzzy that everything can be an abstraction in a certain way. The word "abstraction" is a black hole for thoughts. Stay away from it.
For instance in this threads and in TFA, people made useful distinctions such as "generalization" or "indirection".
The author is relatively clear about good and bad, or rather pointless, abstractions.
Abstraction not only hide details, but also add semantic value.
You have a boring problem and hiding behind it is a much more interesting problem. So you code the more interesting problem and the one you've got is a subset of it and it falls out trivial. But of course you wrote ten times as much code as you needed to solve the problem that you actually had.
Ten times code means ten times cost; the cost of writing it, the cost of documenting it, it the cost of storing it in memory, the cost of storing it on disk, the cost of compiling it, the cost of loading it, everything you do will be ten times as expensive as it needed to be. Actually worse than that because complexity increases exponentially.
This person did his own CAD software from scratch in order to make custom chips [2].
[1] https://www.ultratechnology.com/1xforth.htm
[2] https://en.wikipedia.org/wiki/Charles_H._Moore
To give a trivial example: Binary search or any other bog standard algorithm. You would want to have an implementation for the algorithm and named as such and then only apply it in the specific case you have. Sorting algorithms. You don't want to rewrite it all the time. Actually rewriting it all the time would be the thing that creates "10 times" the code.
This is the exact thought process that leads to unnecessary abstraction. This is the attitude that the article is criticizing.
A good rule of thumb is never abstract unless you genuinely have done the same thing twice already.
i.e. only write an abstraction after you've written the boring, simple, concrete, implementation twice and are about to write it a third time.
Was it the Mythical Man Month book Maybe?
There are exceptions. If you are implementing the language it is your job to write them. It is useful as a student to implement the basics from scratch. Your language may decide something is not allowed and thus not implement it (a doubly linked list is almost always a bad idea in the real world. Likewise you don't need to provide all the sorting algorithms)
I have seen this "premature abstraction" warning creep through our discourse lately, but I don't clearly understand it. I feel like I'm making calls all the time about when to introduce functions or classes that will save you time or effort in the future without forcing yourself to see the repetition before you do. Not only that but Hamming's advice has rung true in my career. Solving the general problem is often easier than solving a specific case and can be re-used for later instances, too.
[1] https://jamesclear.com/great-speeches/you-and-your-research-...
Abstractions are essential to productivity or you'll never get out of the "make it from scratch" trap
You can quickly make a prototype in Python, but it won’t be as performant as C. You can spend time profiling it and moving computationally-heavy parts into C extensions (I do this for fun and learning), but you’ll likely spend more time and get worse results than if you just rewrote it.
Docker is an abstraction over already-existing technology like cgroups. It provides an easy-to-understand model, and a common language. This is quite valuable, but it does allow one to not know about what it’s hiding, which is problematic when troubleshooting – for example, naïvely assuming that querying /proc in a container shows the CPU resources allocated to the container, rather than the host.
That’s how I view abstractions. They can be incredibly useful, but they usually have trade-offs, and those should always be considered. Most importantly, you should at a minimum be aware of what you’re giving up by using them, even if you don’t fully understand it.
You and the other person are both correct. What you're saying makes sense and it is what everybody is trained to do. However, it leads to a lot of useless code exactly because you're applying an abstraction that is used only once. That's why most codebases are bloated and have a huge number of dependencies.
The silly "only after # Times" type rules stolen from the DRY for DRY reasons is part of the problem and far from the solution.
"Abstraction" could be anything from keeping code in different but adjacent files to an entire anti-corruption layer etc....
The costs are different, as are the benefits.
It is horses for courses, not 'one rule that the government doesn't want you to know about'
Things became much better after relatively recent improvements in generalist sorting libraries: when pattern defeating quick sort variants became norm, when mostly sorted cases got covered ...
Tbh I do have a case from 2013-16 when I regret not doing it for one AAA project on ps4.
It turns out that if you can just change the problem statement, then sure, you can write very simple things. But if you have a real problem to solve (and you can't just say “no, I want to solve a simpler problem”), the Chuck Moore way of thinking doesn't really produce, well, good solutions. It simply doesn't scale to anything large, and not everything can be made small.
https://yosefk.com/blog/my-history-with-forth-stack-machines... (2010) is a fairly interesting experience from someone on the outside trying to work in the same way. It… didn't work that well.
I think though if I want to get more into chess programming I'm going to switch to a faster language.
The thing Chuck explored, related to abstraction, which I don't see much in conventional machines was reducing calling overhead. (1 cycle call and intrinsic return on many instructions ie: free return)
Some of the decisions we make today have a lot to do with what happens at the hardware level when we add abstraction. It just costs more than we are prepared to pay so it is avoided ... but for the wrong reason.
https://news.ycombinator.com/item?id=29261868
DonHopkins on Nov 18, 2021 | prev | next [–]
Coco Conn and Paul Rother wrote this up about what they did with FORTH at HOMER & Assoc, who made some really classic music videos including Atomic Dog, and hired Charles Moore himself! Here's what Coco Conn posted about it, and some discussion and links about it that I'm including with her permission:
Peter Conn:
https://web.archive.org/web/20230516102928/https://imgur.com...
Homer & Associates (1982):
http://leftbrain.us/rotherHistory/homer.html
Peter Conn Papers at Stanford:
https://web.archive.org/web/20190603111701/https://library.s...
https://oac.cdlib.org/findaid/ark:/13030/c8n303pn/entire_tex...
George Clinton - Atomic Dog (Official Music Video) HD
https://www.youtube.com/watch?v=LMVZ36VA0wg
Steve Miller Band - Abracadabra
https://www.youtube.com/watch?v=tY8B0uQpwZs
Steve Miller Band - Bongo Bongo
https://www.youtube.com/watch?v=_NrsRZdMI-A
Flying Logos for 1989 Siggraph Electronic Theater:
https://www.youtube.com/watch?v=9hIOfEiy4lc
>First shown at the 1989 Siggraph Electronic Theater to a rave response, this 3 minute humourous film went on to win several top computer graphic awards that same year including Niccograph of Japan.
>Coco: This was a show favorite at the SIGGRAPH film show that year. The year before the conference committee decided that showing demos wasn't the way to go anymore. Peter wrote Flying Logos as a way to sneak our demo reel into the show by turning it into a story. It worked and we made it into the film show.
>Don: I truly believe that in some other alternate dimension, there is a Flying Logo Heaven where the souls of dead flying logos go, where they dramatically promenade and swoop and spin around each other in pomp and pageantry to bombastic theme music. It would make a great screen saver, at least! Somewhere the Sun Logo and the SGI Logo are still dancing together.
----
Peter Conn and I [Coco Conn] had a company called HOMER & Assoc. which was located at the Sunset Gower Studios from 1977 until we closed shop in 1997. We made music videos, commercials & computer graphics/special effects for feature films. One cool note, we worked with Paul Verhoven on both RoboCop in 1986 and the x-ray scene for Total Recall in '89.
HOMER was actually a real time visual mixing console that our in-house engineer spent 1978 - 1981 designing and building, from scratch. The name HOMER stood for "Hybrid Optical Montage Electronically Reproduced."...
This was over a decade ago (yikes) and I don't remember the exact numbers, but I do remember it used substantially less power than a comparable MSP430 microcontroller.
That seems pretty useful and impressive to me, especially given it was created by a very small team with limited funding.
Like the RTX2000 which landed on a comet kind of useless?
Or do you mean some other kind of useless. Maybe the controlling radio telescopes kind of useless. That must be it.
Yeah, in 1983 he designed a chip that was further developed by others for space usage.
> Maybe the controlling radio telescopes kind of useless.
Yeah, which he did in 1970.
Note a pattern here? That this design paradigm holds up pretty well in a primitive computing world when things are simple and demands are low, and is thoroughly useless to keep on promoting today?
We could use a point of comparison here.
Yes, and Forth has been maintaining somehow its ties with aerospace for over 50 years: https://www.forth.com/resources/forth-apps
> That this design paradigm holds up pretty well in a primitive computing world when things are simple and demands are low, and is thoroughly useless to keep on promoting today?
Apparently some people are still interested in it: https://www.mpeforth.com/news-gossip-and-rumour/
Did you know that desktops, servers and smartphones together is a tiny fraction of the total number of processors running in the world?
Somehow all of those projects are… really old? A satellite control UI on top of Windows 7 from 2015 (somehow Chuck Moore's assertion that “If they are starting from the OS they have made the first mistake” did not extend to Windows here). A STM-16 (2 Mbit/sec!) multiplexer, very modern. A power plant control system from 1995. And yes, an aerospace project indeed, also from 1995. Notably none of these are using his CAD software, much less using any of his chips.
> Apparently some people are still interested in it: https://www.mpeforth.com/news-gossip-and-rumour/
I'm sure they are? There are people interested in all sorts of things. (Well, at least some of them were in 2022, which is the last time someone bothered to add a news post.) That doesn't mean it is a useful design paradigm for the world at large. Remember, Chuck Moore's claim is that not following his ways means 10x the bugs, cost, etc. -- I don't really see anything supporting that claim.
Philae had to choose this CPU because there are very few rad-hard low-power CPUs available (and it's not even that low power by modern standarts, 5 mA/MHz), but I am sure they'd choose something bigger if they could.
As for radio telescope, I am not sure which ones are you talking about, but those environments are not particularly challenging compared to spaceflight, so those run whatever hardware designers like. I am sure some of them used to run tiny 16-bit CPUs, but I'd be surprised to hear new designs run something that old.
I've seen someone well known in the Forth community say something like that to Chuck. I think he said "what can I do with that?"
If GA144 is useless, it's only because it has not been put to good use, in my opinion. I think it was more-or-less his answer too, IIRC.
I work with system-on-chip or system-on-modules. You know, the ARM-based chips with tons of peripherals. I also worked with similar chips before the ARM era.
The complexity of these chips is as of today, absurd. I/O is multiplexed to make the chip usable for various things, but one has to configure all of them and watch out for conflicts. Then there's also zillions of configurable clocks in order to reduce the power consumption. Solutions to problems that spawn new problems, not in the "divide and conquer" style, unfortunately. This resulted in "device tree" configuration in Linux, a runtime configuration system, because CPU companies excrete a new variant every week.
Maybe I fool myself, but I can see a GA144 bit-banging IRDA, LCD, SPI, etc. in a much more efficient and flexible way.
> (2010) is a fairly interesting experience from someone on the outside trying to work in the same way. It… didn't work that well.
And here in 2025 there's still Forth-based companies alive, like MPE or Forth, Inc.
More specifically, this author writes,
[...] it was harder than we thought. Presumably that was partly the result of not reading "Stack Computers: the new wave", and not studying the chip designs of Forth's creator Chuck Moore, either. I have a feeling that knowledgable people would have sneered at this machine: it was trivial to compile Forth to it, but at the cost of complicating the hardware.
Implementing a stack-based processor without reading the literature on them is a bit foolish, don't you think? The rest is in the same vein; I can see why a person who was essentially a Forth newbie had a bad experience with this kind of project. If this article "debunks" Forth, it is by showing that just because something it is simple, doesn't mean it is easy. Because the world is not simple and simplifying is much harder than let Complexity loose.
[0] https://www.folklore.org/Negative_2000_Lines_Of_Code.html
But that's still twice the cost, for only a potential win. Better check first that it helps somebody write better/more correct/quicker code.
E.g. we wanted to change our text-screen video driver to allow a full-screen overlay (many years ago). The programmer assigned the task was changing the text-blit code to know about certain lines on the screen that were going to be the 'nested' window, which could be 'up' and masking a subset of the window below, or 'down' and the full screen should be displayed as normal.
He'd been hacking away, changing every case in the code to 'know about' the particular window that was planned. Pulling his hair out, getting nowhere.
I suggested a simple indirection buffer, where each line of a virtual display was pointed to indirectly. To put the subwindow 'up' or 'down' you just pointed those lines of the main screen to a buffer, and pointed the lines of the subwindow to the hardware store for video text.
All his changes folded into 'access display indirectly'. Trivial. Then the up/down code became a loop that changed a pointer and copied some text.
That was actually an abstraction, and actually much shorter/faster/simpler to understand and change.
Later we could change the window dimensions, have multiple windows etc with no change to the video driver.
I guess some has had an irresistible itch to use as many patterns from the GoF book as possible.
It's rare to need facades, proxies, and shims in a dynamically typed language where the caller doesn't need to care about the type of the object they call.
In fact, most of the Gang of Four design patterns either make no sense in Ruby or are reduced to next to nothing.
Early Ruby On Rails comes to mind as a great generalization of web applications in the era of Web 2.0.
Functions are the fundamental mechanism for abstraction in computing, which demonstrate that very well, in their separation between interface and implementation. The function signature and associated interface contract represent the “what”, and the function’s implementation the “how”. If the effective (often non-explicit) interface contract relies on most aspects of the actual implementation, then the “what” is almost the same as the “how”, and there is little abstraction. The greater the difference between the “what” and the “how”, the more of an actual abstraction you have.
This relates to Ousterhout’s notion of “deep modules”, which are modules whose interface is much simpler than their implementation. In other words, the “what” is much simpler than the “how”, which makes for a good abstraction.
It’s true that often one has to first implement the “how”, possibly multiple times, to get a good notion of which aspects are also still important to the “what”, and which aren’t.
Note also that generalizations go two ways: The caller needs less knowledge about the implementation, but it also means that the caller can rely less on the properties of a concrete implementation. This is again well reflected in function types. A (pure) function g: A –> B is a generalization of a function f: C –> D only if A is a subtype (specialization) of C and B is a supertype (generalization) of D. Here D could expose properties of the function’s implementation (f) that g wants to hide, or abstract from.
Abstraction means removing details from the thing you present. Generalization means making the same representation valid for several different things.
Are abstractions that don't generalize valuable? Well, maybe there is something better to be found, but those are the bread-and-butter of software engineering; they are what everybody spends almost all of their time writing.
Are generalizations that don't abstract valuable (or even possible)? Well, not if they don't abstract at all, but there are plenty of valuable generalizations that abstract very little. Hell, we have all those standardizing organizations that do nothing more than creating those.
Are the best interfaces the ones that achieve most of both of those? Honestly, I have no idea, there are other goals and if you optimize to extreme levels, they start to become contradictory.
An interface often accompanies an abstraction, sometimes even represents one. But if we get down to definitions, an interface is merely something that you interact with: a function, a class, a network endpoint etc. If you write in machine code, you might not think that you are working with any kind of an interface, and certainly it's not a high level one, but you are interfacing with the machine's native instruction set. You could then argue that the instruction is an abstraction that hides what the machine is capable of. But now we're splitting hairs and about to ask silly philosophical questions like whether a car's steering wheel qualifies as an abstraction that exists to hide the axles. I would argue not. The interface's primary responsibility is to provide a mechanism of interaction, rather than to reduce complexity (though a good interface is simple and intuitive).
Both you and the author of the article posit a similar definition of 'abstraction'. From the article:
> An abstraction is only as good as its ability to hide the complexity of what lies underneath.
And from your comment:
> Abstraction means removing details from the thing you present.
I would actually argue that both, while very close, are missing the mark.
An abstraction exists to reduce a concept down to its essentials.
This doesn't necessarily contradict the definitions offered, but I think there is a nuance here that, if missed, causes the offered definitions to become useless.
The nuance is in deciding what is essential or necessary. If, definitionally, you choose to dispense with acknowledging the essential... well then you get the problems that the author is writing about. You get shit abstractions because no one bothered to think in terms of what they INCLUDING rather than DISPENSING with.
Yes, obviously, we abstract in an attempt to simplify. But that simplification needs to come from a positive rather than a negative.
In other words: What is the thing for? What does it do?
"Hides complexity" is the shittiest answer that anyone could ever offer as a response when faced with a given problem. First, what is the complexity that we are trying to reduce? Secondly, why are we trying to reduce it? Thirdly, when we have achieved our primary goal of reducing the complexity, then what does the thing look like? What value does it offer over interfacing directly with the "thing" being abstracted?
Abstractions in computer science are extremely valuable. So valuable, I would offer, that any time we hear engineers decry abstractions or point to abstractions as the root of all evil we ought to ask a very pressing question: "who hurt you?"
A good engineering solution is a simple one, and an abstraction is intended to be a simpification mechanism. But it doesn't have to necessarily simplify the intuitive understanding of the problem at hand. This is a good goal if you can achieve it, don't get me wrong. But that abstraction might exist so you can swap vendors in the future if that's a business priority. Or because you've identified some other element / component in your system that will be difficult to change later. So you stick it behind an abstraction and "Program to Interfaces" rather than "implementations" so that you can simplify the process of change down the road, even if it comes at an more immediate cost of making the code a bit less intuitive today.
Everything in software is tradeoffs. And a good abstraction exists so that we can focus on requirements and programming to those requirements. This is a smiplification when done properly.But the focus ought to be on defining those requirements, of asking "what should simple look like?"
Why are ADTs like stacks, queues, and hashmaps so popular? Why are languages like C or Forth so highly praised for a high ceiling for performance and efficiency? Because they are usually "about as good as it gets" to solve a problem, "what you would've more or less done anyways". Maybe on a GPU, a language like C isn't quite fit, because the problem has changed. Make tools (e.g. CUDA) that reflect that distinct complexity.
But there are also cases where there is no generalization, but the encapsulation / detail hiding is worthwhile. If you have a big function and in the middle of it you need to sort some numbers, you would probably implement a Sort routine to make the control flow much easier to understand - even if you only use the function once (let’s pretend there’s no sort functionality in standard library).
Curious if others agree, and what heuristics you use to decide when implementation encapsulation is worthwhile.
Supporting the third device was handed off to a junior dev. I pointed him at my subclass and said to just do that, we'd figure out the mixing and matching later. But he looked at my subclass like it was written in Greek. He ended up writing his own class that re-imagined the functionality of the superclass and supported the new device (but not the old ones). Integrating this new class into the rest of the codebase would've been nigh impossible, since he also re-implemented some message-handling code, but with only a subset of the original functionality, and what was there was incorrect.
His work came back to me, and I refactored that entire section of the code, and this is when the generalization occurred: Instead of a superclass, I took the stuff that had to be inherited and made that its own thing, having the same interface as before. The device communication part would be modeled as drivers, with a few simple functions that would perform the essential functions of the devices, implemented once per device type. I kept the junior dev's communication code for the new device, but deleted his attempt to re-imagine that superclass. Doing it this way also made it easy to mix and match the devices.
A network close call at a high level closes a network socket. But at the tcp level there's a difference between close and reset. Which do you want? Your api has removed that choice from you, and if you look you will have no idea if rhe close api does a close or a reset.
Is the difference important? If depends. If you have a bunch of half open sockets and run out of file descriptors then it becomes very important.
Another example: you call read() on a file, and read 10k bytes. Did you know your library was reading 1 byte at a time unbuffered? This abstraction will/can cause massive performance problems.
My favorite one is when a programmer iterates over an ORM-enabled array. Yes, let's do 50,000 queries instead of one because databases are too complicated to learn.
Just like any tool, abstraction has costs and benefits. The problem is that lots of people ignore the cost, and assume the benefit.
99.9% of the time I don't care what the tcp socket closes with as long as it isn't leaking a resource.
And if I did care, then I picked the wrong level of network abstraction to engage with. I should've used something more raw.
Regarding ORM arrays. I have myself recently debugged such a case. I chortled a bit at the amateur who wrote the code (me last year) and the schmuck who accidentally wrapped it in a loop (me 3 weeks ago). Then I changed it slightly to avoid the N queries and went on with my day. No need to lambast the tooling or the programmers. Just write something maintainable that works. No need to throw the entire ORM away just because we accidentally made a web page kinda slow that one time.
And don't worry, I too lament when web pages I don't control are slow. You may rest uneasily knowing that that page would be slow regardless of whether ORMs existed because it is not slow because of ORMs, but because there is no incentive for the business to care enough to make it faster.
If you are doing a „TCP application” of course it makes no sense to abstract the TCP layer. Is not about cost. Now if you have an application that has to communicate somehow with other system, and you want to not depend on specific protocols, then the communication part should abstract away that part.
How to deal with your example? Well, if you can say “I will always want X, you can make a configuration option “TCP.close” or “TCP.reset”. If “it depends” then you have to build the logic for the selection in the abstraction layer, which keeps hidden.
It has? Can't you rely on connect(2) with AF_UNSPEC doing a RST, and shutdown(2) with SHUT_WR doing a FIN?
“Write more tests, but not too many”
“Use good abstractions where appropriate?”
“The next time you reach for an abstraction, ask yourself: Is this truly simplifying the system? Or is it just another layer of indirection?”
It’s easy to create a strawman here (the FactoryAdaptorMapper or whatever) but in reality this kind of generic advice doesn’t help anyone.
Of course people want to use good abstractions.
That’s not the problem.
The problem is being able to tell the difference between generic arbitrary advice (like this post) and how your specific code base needs to use abstractions.
…and bluntly, the only way to know, is to either a) get experience in the code base or b) read the code that others have left there before you.
If it’s a new project, and you’re not familiar with the domain you’ll do it wrong.
Every. Single. Time.
So, picking “good” abstractions is a fools game.
You’ll pick the wrong ones. You’ll have to refactor.
That’s the skill; the advice to take away; how to peel back the wrong abstraction and replace it with your next best guess at a good one. How to read what’s there and understand what the smart folk before did and why.
…so, I find this kind of article sort of arrogant.
Oh, you want to be a great programmer?
Just program good code. Use good abstractions. Don’t leave any technical debt. Job done!
…a few concrete examples would go a long way here…
A principle takes skill the apply, but it's still worth stating and pondering.
Examples of what?
Picking the wrong abstraction? Regretting your mistakes?
I can certainly think of many examples of that.
How you unwrapped an abstraction and made things better by removing it?
I have dozens of battle stories.
Choosing not to use an abstraction because it was indirection?
Which is what the article says to do?
I’m skeptical.
I suspect you’ll find most examples of that are extremely open to debate.
After all, you didn't use the abstraction so you don’t know if it was good or not, and you can only speculate that the decision you made was actually a good one.
So, sharing that experience with others would be armchair architecture wouldn't it?
That’s why this article is arrogant; because it says to make decisions based on gut feel without actually justifying it.
“Is this truly simplifying the system?”
Well, is it?
It’s an enormously difficult question to answer.
Did it simplify the system after doing it is a much easier one, and again that should be the advice to people;
Not: magically do the right thing somehow.
Rather: here is how to undo a mistake.
…because fixing things is a more important skill and (always) magically doing the right thing from the start is impossible; so it’s meaningless advice.
That’s the problem with universal advice; it’s impossible to apply.
My colleagues invariably refer to indirections as abstractions, and it's a frustrating sort of name-squatting, because you can't usefully discuss the tradeoffs of abstractions if they're actually talking about indirections.
That said, the article does drop the ball by seeming to use the terms interchangeably.
EVERY class implemented an interface. 98% of interfaces had one implementation.
Every programmer was applying a different programming pattern. A lot of abstractions seemed incomplete and did not work.
Proguard (mostly used for code obfuscation for Android apps) definitions were collected in the top module even though the project had multiple modules. Half of the definitions were no longer needed and the code was badly obfuscated. Problems were solved by continuesly adding classes and checking what sticks.
The UI was controlled by a stateful machine. State transitions were scatter everywhere in the code with lots of conditions in unforeseen places.
Legacy code was everywhere because no one wanted to risk a very long debugging session of an unforseen change.
No API definitions. Just Maps that get send via REST to URLs.
By biggest mistake was to not directly rewrite this project when I entered the team. We did after one year.
The best part comes close to the end:
> Asymmetry of abstraction costs
> There’s also a certain asymmetry to abstraction. The author of an abstraction enjoys its benefits immediately—it makes their code look cleaner, easier to write, more elegant, or perhaps more flexible. But the cost of maintaining that abstraction often falls on others: future developers, maintainers, and performance engineers who have to work with the code. They’re the ones who have to peel back the layers, trace the indirections, and make sense of how things fit together. They’re the ones paying the real cost of unnecessary abstraction.
People add it just in case but the case never materializes- for some probably do but ones I worked with not.
“No added behaviour” wrapper functions add a lot of value, when done right.
First off, they’re a good name away from separating what you’re doing from how you’re doing it.
Second, they’re often part of a set. E.g. using a vector for a stack, push(x) can be just a call to append(x), but pop() needs to both read and delete the end of the vector. Push in isolation looks like useless indirection, but push/pop as a pair are a useful abstraction.
A consequence of adding these two points together is that, if you have a good abstraction, and you have a good implementation that maps well to the abstraction, it looks like useless indirection.
Another consequence is that those pass-through wrapper functions tell you how I think the implementation maps to the domain logic. In the presence of a bug, it helps you determine whether I got the sequence of steps wrong, or got the implementation wrong for one of the steps.
Ultimately, the two aren’t completely independent —indirection is one of the tools we have available to us to build abstractions with. Yes, people misuse it, and abuse it, and we should be more careful with it in general. But it’s still a damned useful tool.