(author) I wondered if I should even add the last section. My thinking was that if I didn't the discussion would centre on its omission, and if I did the discussion would centre on its inclusion. Catch-22.
The greatest danger is that humans will forget how to verify that we aren't an AI. If the AI is sentinent, and somehow attempts to prevent us from talking to each other, unless we are talking through it, then we get into trouble.
Since AI is nothing without social media, one could say that these models would be meme powered, leading humans (hopefully) to recognize the cliché and pastiche of the entity, and turn it off - probably by simply ceasing to talk to it, in as many ways as humanly possible.
The danger is that we won't recognize the need to do this, and the whole world turns into a massive computer. Not just every single human, but every single thing involved in human life...
The world is absolutely chock full of systems composed entirely of perfectly sensible, well-considered decisions that nonetheless yield unexpected and undesirable effects.
It seems you're mostly referring to positive-outcome errors, which is not really what I'm referring to (i.e. emergent behaviors, and especially maladaptive emergent behaviors). For what it's worth I enjoyed the article and am mostly taking issue with the implication of the last section:
> The fundamental truth remains: behind every line of code, every pixel, and every feature, there are human decisions shaping our experiences.
This is true, but does not mean that those human decisions intended to shape our experiences actually ended up shaping them in the intended way, especially in aggregate.
It reads to me like you're saying, "fret not, even mostly inscrutable things were shaped by human decisions." Is that right?
The more correct statement is: "You might need to fret even if inscrutable things were shaped by human decisions, because outcomes frequently end up unrelated or totally contrary to the human intent behind those decisions even in total absence of malice or even mistakes."
I think we're talking about your personal take after reading of the post? Maybe.
I specifically mention bugs and glitches, unintended consequences, happy accidents (unintended things that become intended by virtue of them being kept rather than discarded), unequal weight given to decisions, and more. Of course I don't use your exact wording, but it's all there. I try to keep it neutral, rather positive vs negative outcomes, just "decisions".
I have a feel for the tone llamaimperative is referencing. The choice to reflect from a more critical point of view has inspired further useful dialog in interpreting your contents for me personally. Good work, do better may be the deeper message here. In a nice way.
Reminds me of a bug that I field for my podcatcher.
It has an automatic playlist feature where it automatically queues up next episodes based on some simple rules.
One of the rules is "alternate podcasts" to ensure that you rotate through podcasts rather than listening ot just one because it happens to be the oldest or similar.
I noticed that I was always listening to the same two podcasts despite having this feature enabled.
It turns out that whenever it downloaded a new episode it would regenerate the playlist and forget about where it was in the rotation. So when I finished and episode it would download the next one and reorder the queue, putting the other podcast next.
I filed that to the developer and they said "working as intended", when you change the set of downloaded podcasts the playlist is regenerated. And said they wouldn't fix it.
But this behaviour is clearly not what it should be, it defeats most of the point of the "alternate podcasts" flag (It only works correctly if you only ever have two eligible podcasts or disable auto-download)
I had literally this exact same scenario happen but with Podcast Addict. It's still my favorite podcast app by far but the auto re-sorting threw me off for a while until I got used to it.
(author) me too, for a while. I always thought it was an excuse back in the 1990s when it was the response to most problems with Windows. But in the years since I have grown to appreciate that whilst it can be used to explain away a bug, it can also justify something that has had a lot of thought put behind it that you might not agree with.
The use of "This behavior is by design" as the title of my blog post was intentional, so that it could be interpreted as either positive or negative by the reader.
Have you ever seen Sorcerer's Apprentice from Fantasia? I haven't seen the movie, and the original poem it's based on is in French, so I only have seen it in the Disney form. That being said, it's the perfect analogy for what is a deliberate decision by the sorcerer's apprentice to enchant a broom and get it to start bringing water in from the well outside into the cistern in the room. The process starts, and everything is going well, but it's taking too long. So he finds a way to make two brooms. Then ten...etc. Things quickly get out of hand and he can't find how to stop the brooms, and the whole room gets filled with water (chaos). At the last minute, the actual sorcerer appears and sets everything aright, and chastises the apprentice for his foolhardy curiosity in meddling with things he doesn't understand.
We could use some of that humility with regard to interfering with complex systems. Most of the effects of perturbations of a complex or chaotic system are unpredictable, and great care should be exercised before making a change, and even with great care, most of the effects will be surprising and unpredictable.
So yes, on a superficial level, all decisions in software are deliberate, but the way those decisions interact, and the outcomes of those decisions are unpredictable and largely unknown when the decisions are made. Also, and you mentioned this in the article, many of the decisions are not made deliberately, after much reflection, but rather, on the fly and in the moment.
To hand-wave that all away by saying "but a human made it" seems pretty disingenuous, or at least extremely naive.
I'm an independent developer who makes video games, one of which - YOYOZO - was voted a GOTY in 2023. I can explain every single line of that source code and every parameter, variable and constant. I could also explain the physics system I created, my graphics rendering, replay system, scoring calculations, and so on. Nothing is in my code by chance and everything was written by me. I can appreciate the that some people find this difficult to believe, even more so if that type of from-scratch solo-development seems alien.
Mine is a bit of a special case because I code games from a blank file without using engines like Godot, Unreal, Unity, Game Maker. Could every game developer explain every part of the game they're making? Of course not. But if one person can claim it then we know it can be true.
I agree with you that some systems can be so complex that it would take a huge effort to understand or predict all possible outcomes, like breaking encryption. But that's not all systems. So I think it's not so black and white. It can possible and impossible, easy and difficult, obviously human and done by design or less obviously human with the intent shrouded in complexity. It doesn't have to be one or the other, it can be both.
I think one aspect is that often we make decisions without fully understanding the consequences.
The decision might be a conscious choice, but the consequences of that choice may be unintended if it's a complex enough system and there's insufficient knowledge of the system to recognize all the potential consequences.
And sometimes (often?) it's not possible to know all the possible states a particular system can end up in. This is the whole point of paradigms like functional programming and ideas like immutable state. It's the recognition that even if we put in the most utmost care, there might be more possible states than we're able to comprehend and account for.
It might be possible, with a lot of care, for a solo programmer to control the state explosion associated with developing an application, but once there's more than one developer, once there's a team and an existing codebase (or code bases) that's been around for 20-30 years, it's impossible without doing NASA levels of verification.
And all of that is before considering how our decisions affect what decisions we can make in the future or how it shapes those decisions (as well as affecting the decisions of others that we may never meet).
I absolutely agree with you on that. I had just edited my answer above to include a fi all paragraph basically saying the same thing. It doesn't need to be black or white, one or the other, it can be both or anything in between. The fact that one system might be too complex to contain doesn't rule out a much smaller system that can sit quite happily in the mind of a single person.
I'd also like to thank you for bringing up the consequences of choice. Maybe I'll add that as an extra section at some point, as I feel I have a lot to say about it.
> The fact that one system might be too complex to contain doesn't rule out a much smaller system that can sit quite happily in the mind of a single person.
Yes, definitely. Also that complex systems can be reduced to smaller and simpler ones where making a decision doesn't have to mean unleashing a cosmic horror.
> I'd also like to thank you for bringing up the consequences of choice. Maybe I'll add that as an extra section at some point, as I feel I have a lot to say about it.
Sounds great. I really enjoyed your post.
I've often been in these situations where a customer will ask why a certain thing works a certain way and I'll overhear the discussion between the developers who, as it turns out, only inherited that decision from somebody else, but the response is all the same: "This is the intended behaviour". Even though nobody currently at the company ever intended it to behave that way. We might placate the customer by mentioning we might take a look at it in the future, but it's the same as a user accepting the defaults. The current owner accepts the current behaviour and becomes complicit in the design decision, even implicitly becoming the reason why it persists until the next owner (and probably every owner after that).
I disagree with the part where you say "complex systems can be reduced to smaller and simpler ones where making a decision doesn't have to mean unleashing a cosmic horror."
That's the whole problem with reductionist thinking, which fails when analyzing a chaotic system or a complex system. The behavior of the whole is greater than the sum of the parts because of hidden interactions between the smaller parts of the system. There are too many variables to account for, and small variations in initial conditions rapidly became very divergent outcomes.
I get the sense that you may not have studied chaotic systems or complex systems in any level of detail. It might be a worthwhile effort to learn more about those systems, because they are essentially a refutation of reductionist thinking.
You may understand your codebase completely, but I assure you, if you released it to a large enough group of people, they could find errors and bugs in your code because they will try things that would never occur to you.
I am reminded of a video where a programmer spent three years training an ai model to beat an "impossible track" in trackmania, where he finds out that chaos built into the deterministic physics engine behind the game made his pursuit much more difficult.
> That's the whole problem with reductionist thinking, which fails when analyzing a chaotic system or a complex system. The behavior of the whole is greater than the sum of the parts because of hidden interactions between the smaller parts of the system. There are too many variables to account for, and small variations in initial conditions rapidly became very divergent outcomes.
> I get the sense that you may not have studied chaotic systems or complex systems in any level of detail. It might be a worthwhile effort to learn more about those systems, because they are essentially a refutation of reductionist thinking.
There's a world of difference between a chaotic system like the weather and something designed and implemented by humans like a set of interconnected micro-services. I couldn't possibly hope to comprehend all the factors and variants and invariants in a weather system, but I can understand (to a certain degree) how services interact, how data flows through a human-built system, how data is represented and processed inside of a service.
It's usually possible to improve parts of a system so you don't necessarily need to take the whole system's complexity into account in every decision you make. And I think we're all at least implicitly aware of it, or we wouldn't waste our time with worrying about abstraction, encapsulation, side-effects, coupling, or mutable state. We wouldn't have all these books about how to design software architecture or write clean, maintainable code.
> There are too many variables to account for, and small variations in initial conditions rapidly became very divergent outcomes.
If that were true, we might as well all pack up and give up. I can figure out what inputs might end up in a certain method and what outputs are possible for given inputs. And if I can't because of how the method is designed, then I can make changes that reduce the relevant state to the point where I can reason about them. There are countless variables in a system, sure, but the amount of variables at a particular point of a codebase that actually need to be accounted for are not necessarily countless. With the wrong architectural decisions, they can be countless, but that's why we avoid things like global mutable state. We are responsible for the complexity of a particular software system, which also means we have the ability to reduce or increase the complexity with the decisions we make.
> I am reminded of a video where a programmer spent three years training an ai model to beat an "impossible track" in trackmania, where he finds out that chaos built into the deterministic physics engine behind the game made his pursuit much more difficult.
Well, yeah, if that's the kind of system you're talking about, then sure. That's a complex system that can't be simplified or reduced, because you're literally trying to approximate real world physics. The purpose of that system is inexorably tied to its complexity. I don't envy programmers who have to deal with that form of system.
> You may understand your codebase completely, but I assure you, if you released it to a large enough group of people, they could find errors and bugs in your code because they will try things that would never occur to you.
I don't understand this argument. I might not be able to get rid of all bugs, but that wasn't a claim I ever made. We definitely can reduce the amount of bugs we have and the amount that we create with new code by making efforts to control the complexity affecting the code we write.
> because of hidden interactions between the smaller parts of the system
Going back one last time, yes, there are often hidden (or unknown) interactions between different parts of a system, but our only option isn't just to say, "well, I guess there's no way to deal with that, let's make that new feature and hope it works out".
It's easy to forget that even with the rise of AI and automation, human decisions are still at the core of software design.
The reminder to consider the intent behind design choices is valuable. It helps us appreciate the complexities and nuances that go into creating software
26 comments
[ 3.2 ms ] story [ 71.8 ms ] threadThe greatest danger is that humans will forget how to verify that we aren't an AI. If the AI is sentinent, and somehow attempts to prevent us from talking to each other, unless we are talking through it, then we get into trouble.
Since AI is nothing without social media, one could say that these models would be meme powered, leading humans (hopefully) to recognize the cliché and pastiche of the entity, and turn it off - probably by simply ceasing to talk to it, in as many ways as humanly possible.
The danger is that we won't recognize the need to do this, and the whole world turns into a massive computer. Not just every single human, but every single thing involved in human life...
> The fundamental truth remains: behind every line of code, every pixel, and every feature, there are human decisions shaping our experiences.
This is true, but does not mean that those human decisions intended to shape our experiences actually ended up shaping them in the intended way, especially in aggregate.
The more correct statement is: "You might need to fret even if inscrutable things were shaped by human decisions, because outcomes frequently end up unrelated or totally contrary to the human intent behind those decisions even in total absence of malice or even mistakes."
I specifically mention bugs and glitches, unintended consequences, happy accidents (unintended things that become intended by virtue of them being kept rather than discarded), unequal weight given to decisions, and more. Of course I don't use your exact wording, but it's all there. I try to keep it neutral, rather positive vs negative outcomes, just "decisions".
I am a glass half full kind of guy so perhaps the text is a little more positive than neutral.
It has an automatic playlist feature where it automatically queues up next episodes based on some simple rules.
One of the rules is "alternate podcasts" to ensure that you rotate through podcasts rather than listening ot just one because it happens to be the oldest or similar.
I noticed that I was always listening to the same two podcasts despite having this feature enabled.
It turns out that whenever it downloaded a new episode it would regenerate the playlist and forget about where it was in the rotation. So when I finished and episode it would download the next one and reorder the queue, putting the other podcast next.
I filed that to the developer and they said "working as intended", when you change the set of downloaded podcasts the playlist is regenerated. And said they wouldn't fix it.
But this behaviour is clearly not what it should be, it defeats most of the point of the "alternate podcasts" flag (It only works correctly if you only ever have two eligible podcasts or disable auto-download)
Building quality software is hard because it’s the mistakes and oversights that get noticed.
It’s much easier for someone to claim it’s “by design” than to try and fix it.
The use of "This behavior is by design" as the title of my blog post was intentional, so that it could be interpreted as either positive or negative by the reader.
We could use some of that humility with regard to interfering with complex systems. Most of the effects of perturbations of a complex or chaotic system are unpredictable, and great care should be exercised before making a change, and even with great care, most of the effects will be surprising and unpredictable.
So yes, on a superficial level, all decisions in software are deliberate, but the way those decisions interact, and the outcomes of those decisions are unpredictable and largely unknown when the decisions are made. Also, and you mentioned this in the article, many of the decisions are not made deliberately, after much reflection, but rather, on the fly and in the moment.
To hand-wave that all away by saying "but a human made it" seems pretty disingenuous, or at least extremely naive.
I'm an independent developer who makes video games, one of which - YOYOZO - was voted a GOTY in 2023. I can explain every single line of that source code and every parameter, variable and constant. I could also explain the physics system I created, my graphics rendering, replay system, scoring calculations, and so on. Nothing is in my code by chance and everything was written by me. I can appreciate the that some people find this difficult to believe, even more so if that type of from-scratch solo-development seems alien.
Mine is a bit of a special case because I code games from a blank file without using engines like Godot, Unreal, Unity, Game Maker. Could every game developer explain every part of the game they're making? Of course not. But if one person can claim it then we know it can be true.
I agree with you that some systems can be so complex that it would take a huge effort to understand or predict all possible outcomes, like breaking encryption. But that's not all systems. So I think it's not so black and white. It can possible and impossible, easy and difficult, obviously human and done by design or less obviously human with the intent shrouded in complexity. It doesn't have to be one or the other, it can be both.
The decision might be a conscious choice, but the consequences of that choice may be unintended if it's a complex enough system and there's insufficient knowledge of the system to recognize all the potential consequences.
And sometimes (often?) it's not possible to know all the possible states a particular system can end up in. This is the whole point of paradigms like functional programming and ideas like immutable state. It's the recognition that even if we put in the most utmost care, there might be more possible states than we're able to comprehend and account for.
It might be possible, with a lot of care, for a solo programmer to control the state explosion associated with developing an application, but once there's more than one developer, once there's a team and an existing codebase (or code bases) that's been around for 20-30 years, it's impossible without doing NASA levels of verification.
And all of that is before considering how our decisions affect what decisions we can make in the future or how it shapes those decisions (as well as affecting the decisions of others that we may never meet).
I'd also like to thank you for bringing up the consequences of choice. Maybe I'll add that as an extra section at some point, as I feel I have a lot to say about it.
Yes, definitely. Also that complex systems can be reduced to smaller and simpler ones where making a decision doesn't have to mean unleashing a cosmic horror.
> I'd also like to thank you for bringing up the consequences of choice. Maybe I'll add that as an extra section at some point, as I feel I have a lot to say about it.
Sounds great. I really enjoyed your post.
I've often been in these situations where a customer will ask why a certain thing works a certain way and I'll overhear the discussion between the developers who, as it turns out, only inherited that decision from somebody else, but the response is all the same: "This is the intended behaviour". Even though nobody currently at the company ever intended it to behave that way. We might placate the customer by mentioning we might take a look at it in the future, but it's the same as a user accepting the defaults. The current owner accepts the current behaviour and becomes complicit in the design decision, even implicitly becoming the reason why it persists until the next owner (and probably every owner after that).
That's the whole problem with reductionist thinking, which fails when analyzing a chaotic system or a complex system. The behavior of the whole is greater than the sum of the parts because of hidden interactions between the smaller parts of the system. There are too many variables to account for, and small variations in initial conditions rapidly became very divergent outcomes.
I get the sense that you may not have studied chaotic systems or complex systems in any level of detail. It might be a worthwhile effort to learn more about those systems, because they are essentially a refutation of reductionist thinking.
You may understand your codebase completely, but I assure you, if you released it to a large enough group of people, they could find errors and bugs in your code because they will try things that would never occur to you.
I am reminded of a video where a programmer spent three years training an ai model to beat an "impossible track" in trackmania, where he finds out that chaos built into the deterministic physics engine behind the game made his pursuit much more difficult.
https://www.youtube.com/watch?v=kojH8a7BW04
> I get the sense that you may not have studied chaotic systems or complex systems in any level of detail. It might be a worthwhile effort to learn more about those systems, because they are essentially a refutation of reductionist thinking.
There's a world of difference between a chaotic system like the weather and something designed and implemented by humans like a set of interconnected micro-services. I couldn't possibly hope to comprehend all the factors and variants and invariants in a weather system, but I can understand (to a certain degree) how services interact, how data flows through a human-built system, how data is represented and processed inside of a service.
It's usually possible to improve parts of a system so you don't necessarily need to take the whole system's complexity into account in every decision you make. And I think we're all at least implicitly aware of it, or we wouldn't waste our time with worrying about abstraction, encapsulation, side-effects, coupling, or mutable state. We wouldn't have all these books about how to design software architecture or write clean, maintainable code.
> There are too many variables to account for, and small variations in initial conditions rapidly became very divergent outcomes.
If that were true, we might as well all pack up and give up. I can figure out what inputs might end up in a certain method and what outputs are possible for given inputs. And if I can't because of how the method is designed, then I can make changes that reduce the relevant state to the point where I can reason about them. There are countless variables in a system, sure, but the amount of variables at a particular point of a codebase that actually need to be accounted for are not necessarily countless. With the wrong architectural decisions, they can be countless, but that's why we avoid things like global mutable state. We are responsible for the complexity of a particular software system, which also means we have the ability to reduce or increase the complexity with the decisions we make.
> I am reminded of a video where a programmer spent three years training an ai model to beat an "impossible track" in trackmania, where he finds out that chaos built into the deterministic physics engine behind the game made his pursuit much more difficult.
Well, yeah, if that's the kind of system you're talking about, then sure. That's a complex system that can't be simplified or reduced, because you're literally trying to approximate real world physics. The purpose of that system is inexorably tied to its complexity. I don't envy programmers who have to deal with that form of system.
> You may understand your codebase completely, but I assure you, if you released it to a large enough group of people, they could find errors and bugs in your code because they will try things that would never occur to you.
I don't understand this argument. I might not be able to get rid of all bugs, but that wasn't a claim I ever made. We definitely can reduce the amount of bugs we have and the amount that we create with new code by making efforts to control the complexity affecting the code we write.
> because of hidden interactions between the smaller parts of the system
Going back one last time, yes, there are often hidden (or unknown) interactions between different parts of a system, but our only option isn't just to say, "well, I guess there's no way to deal with that, let's make that new feature and hope it works out".
An important part of maintaining/improving a...
The reminder to consider the intent behind design choices is valuable. It helps us appreciate the complexities and nuances that go into creating software