The General Journal really only works because accounting entries are immutable: corrections are always new entries. So it's not clear if "give corner cases a home" works everywhere.
But if the principle does work for your use case, it lets the rest of the system be as strict as you like!
I mostly agree with the author but I think there's a very important distinction to make. It's important to think about this in two layers - the domain and the UI (whether that be a CLI, an API, a full blown GUI, etc). The author very clearly is talking about the domain layer, but the area where I see "Make invalid states unrepresentable" having the greatest benefit is in the UI layer. That is where you want fine grain control over what capabilities of your domain layer that you expose and is where you have a lot to gain by architecting your types/system/etc to be strict. It makes for easier to reason about code and reduces the amount of edge cases you need to QA. The constraints you put in the UI layer are product constraints, not technical ones. In my opinion, when you think about it that way, that adage isn't harmful but rather a very useful guide for technical folks to use when trying to mental model what product is asking you to build.
The devil is, as always, in the details. I think "make invalid states unrepresentable" is a good thing to teach in order to push back against spaghetti code. State management is hard, and untangling bad state management is near impossible.
But of course, some flexibility is often (not always) needed in the long run, and knowing where to keep the flexibility and where to push for strictness is an important part of skills development as an engineer.
One of the main pushbacks in this article is on the difficulty of later edits once the domain changes. The "make invalid states unrepresentable" mantra really came out of the strongly typed functional programming crowd – Elm, F#, Haskell, and now adopted by Rust. These all have exceptionally strong compilers, a main advantage of which is _easier refactoring_.
Which side of the argument one falls on is likely to be heavily influenced by which language they're writing. The mantra is likely worth sticking to heavily in, say, Haskell or Rust, and I've had plenty of success with it in Swift. Go or Java on the other hand? You'd probably want to err on the side of flexibility because that suits the language more and you can rely on the compiler less during development.
It matters where the constraints live. Inside of a codebase they are easier to change. Updating the database schema would be harder. On the protocol level it may be impossible if not all parties can be updated. However, if the protocol is too loosely specified, it could create other problems.
I've been thinking about this for a while. I don't use Clojure but I do follow Rich Hickey and have watched a few of his talks, where he makes similar points. Folks often talk about types as abstractions but they're actually concretions, they are concrete choices about what information you want to keep out of all possible choices. And like concrete in real life, it's costly to change the shape of something once it's set. This may or may not be an issue depending on where you program is running, in closed systems like a compiler it can be a great benefit, for open world systems which must use non-elegant models, must deal with state and time, must have effects and be affected, and must change in ways you can't predict, the cost is much higher.
I talk about types because most of the "make invalid states unrepresentable" dogma comes from proponents of languages with extensive static type systems, and the idea is to use the type system to define all forms of legal states.
You also see on-the-wire protocols making invalid states unrepresentable through clever tricks. Consider RFC 3550 p.19:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| defined by profile | length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| header extension |
| .... |
[...] The header extension contains a 16-bit length field that
counts the number of 32-bit words in the extension, excluding the
four-octet extension header (therefore zero is a valid length).
So for RTP extension headers, actual_num_bytes = (`length` + 1) * 4. A naive `length` field might indicate the number of bytes. But this would allow a header to indicate "zero" bytes (not possible.) So the '0' value is defined as 0 more than we already have, and since packets are supposed to only contain a multiple of 4 bytes, the units of the length field are defined as 32-bits.
While it isn't strictly harmful, one drawback of this approach is that if you perfectly bit-pack every field such that random noise can be interpreted as a well-formed packet in your protocol, it will be difficult to heuristically identify the protocol.
Making invalid states unrepresentable may be a great idea or terrible idea depending on what you are doing. My experience is all in scientific simulation, data analysis, and embedded software in medical devices.
For scientific simulations, I almost always want invalid state to immediately result in a program crash. Invalid state is usually due to a bug. And invalid state is often the type of bug which may invalidate any conclusions you'd want to draw from the simulation.
For data analysis, things are looser. I'll split data up into data which is successfully cleaned to where invalid state is unrepresentable and dirty data which I then inspect manually to see if I am wrong about what is "invalid" or if I'm missing a cleaning step.
I don't write embedded software (although I've written control algorithms to be deployed on it and have been involved in testing that the design and implementation are equivalent), but while you can't exactly make every invalid state unrepresentable you definitely don't punch giant holes in your state machine. A good design has clean state machines, never has an uncovered case, and should pretty much only reach a failure state due to outside physical events or hardware failure. Even then, if possible the software should be able to provide information to intervene to fix certain physical issues. I've seen devices RMA's where the root cause was the FPU failed; when your software detects the sort of error that might be hardware failure, sometimes the best you can do is bail out very carefully. But you want to make these unknown failures be a once per thousands or millions of device years event.
Sean is writing mostly about distributed systems where it sounds like it's not a big deal if certain things are wrong or there's not a single well defined problem being solved. That's very different than the domains I'm used to, so the correct engineering in that situation may more often be to allow invalid state. (EDIT: and it also seems very relevant that there may be multiple live systems updated independently so you can't just force upgrade everything at once. You have to handle more software incompatibilities gracefully.)
The database is not your domain model, it is the storage representation of your domain model on disk. Your REST/grpc/whatever API also isn’t your domain model, but the representation of your domain model on the wire.
These tools (database, protocols) are not the place to enforce making invalid states un-representable for reasons the article mentions. You translate your domain model into and out of these tools, so code your domain model as separately and as purely as you can and reject invalid states during translation.
The key part: "For most software, domain models are not real". Indeed, if your reality is ill-defined, you need miracles, aka invalid states, to handle certain practical cases. So it's more about admitting a design failure. One type of such failure may be putting wrong constraints on the model. You have to think hard about what can and cannot be, and often it's not a luxury your boss can afford. To allow representation of invalid states is to admit that you're going to be proven wrong eventually, which is not a wild exaggeration in a wide range of circumstances, alas. It's to plan for a mess, because a mess is inevitable.
(This whole approach reminds me of "I think therefore anything, due to the false antecedent".)
The problem with making invalid states representable is that the app logic must be able to handle it, everywhere. Which means you have to always take it into account when reasoning about your app and implementing it, otherwise it will blow up in your face when the invalid state actually occurs. Which means your state machine is inherently larger and harder to reason about.
To illustrate what I mean, take the null pointer. It's essentially an Optional<T> and in a lot of languages it's involuntary, meaning any value is implicitly optional. What happens when something returns null, but in some place in your logic you don't take it into account? Null pointer dereference.
I’ve seen more errors from “pubilshed” strings than difficulties updating an enum.
The rest of it is arguing about what constitutes valid states, eg, commit messages when the user deletes or wrong-schema datagrams being partially utilized. Here’s the rub: if you don’t allow invalid states, you’re forced to actually have that discussion (“actually, messages don’t strictly require a user”; “only these fields are required, others are a maybe”) and those are then encoded in your software rules. They’re not things that happen informally, in undocumented ways.
Further, when you do allow invalid states, you have a typo disappear 1% of your messages into the “pubilshed” category.
The first example in the article of an application or whatever doesn't seem like a good example. I don't ever have some strict state machine in my data model that only has certain state transitions. I have a state, and certain transactions are available through API endpoints, and maybe internal apps have their own endpoints that can do things that the public cannot.
Hmm this is interesting food for thought, but I mainly disagree with the "Foreign Key Constraints" section:
> A foreign key constraint forces user_id to correspond to an actual row in the users table. If you try to create or update a post with user_id 999, and there is no user with that id, the foreign key constraint will cause the SQL query to fail.
> This sounds great, right? A record pointing at a non-existent user is in an invalid state. Shouldn’t we want it to be impossible to represent invalid states? However, many large tech companies - including the two I’ve worked for, GitHub and Zendesk - deliberately choose not to use foreign key constraints. Why not?
> The main reason is flexibility. In practice, it’s much easier to deal with some illegal states in application logic (like posts with no user attached) than it is to deal with the constraint. With foreign key constraints, you have to delete all related records when a parent record is deleted. That might be okay for users and posts - though it could become a very expensive operation - but what about relationships that are less solid? If a post has a reviewer_id, what happens when that reviewer’s account is deleted? It doesn’t seem right to delete the post, surely. And so on.
So if you need to keep reviews by a user when the user can be deleted, you have a few options:
- Use a foreign key constraint but also allow nullable foreign key values (I believe DBMSes such as postgres and mysql support this)
- Create a record representing an orphaned user and move reviews to it when the user is deleted (generally, the reviews will then show as posted by user "DELETED", though you can also enforce different ways to display deleted user associations)
- Soft delete the users (in this case you also have to exercise caution around what ex-user data may be displayed where)
There are tradeoffs with each one; soft deletion of users would likely run afoul of Right to be Forgotten laws in some places. I'd tend to favour option 1 for anything that references something like a user account where the user may be deleted, but option 2 might be useful in situations as well.
I'd generally recommend against omitting foreign key constraints when it can be assumed databases backing your application will support them.
Sure, losing the FK constraint gives you more "flexibility" (I guess?) but introduces way too many footguns. The database and tooling are a massive help in avoiding so many other bugs.
Sure, maybe one day you decide reviews can have authorship associated with entities which are not users of your system (say you've acquired republication data rights from another review service whose users are not your service's users); in that case you'd need an application-level refactor and migration. You might need to add an author table, where the author can be a user of your service, or a reference to a user of another service . Then users of your service become possible authors, and all reviews need to be migrated to have an FK relation to the author table now rather than the user table.
In any case, the FK doesn't prevent you from changing how your data is structured, and I'd argue it greatly helps you avoid mistakes as you move through the migration.
This reminds me that in protocol buffers you can deprecate and extend field names in a backwards compatible way and it's handled semi-sensibly. At the very least, it makes you think about schema migration (of HTTP fields in this case. DB fields and class fields are different...). If you aren't using practices that make e.g. adding another enum option to state sensible (e.g. in a language or linter that requires passing interface checks, class X must provide method Y)
I wish there were some actual stories here about tricky migrations and how they were handled, rather than defaulting to the naturally turf war framing here.
If my experience, writing code that is “future migration friendly” is one of the hallmarks of senior developer knowledge. It’s just the kind of thing that burrows into your brain, because most senior devs have seen, built, or maintained code with randomly cordoned off “WARNING NONSENSICAL THING HERE” style sections of massive codebases that are artifacts of various migration things.
For context about how we do it, we use a giant NoSQL single table design, with zero joins. All joining is handled in the application, which ends up forcing everyone to consider how to parallelize and lookahead rather than join. Additionally, any complex data is stored as row level binary blobs from a versioned schema, which also helps localize issues to a particular subtype rather than leaking it into all schemas (for example if you change what qualifies as an “update”, your “updatedAt” column doesn’t suddenly change meanings for all things.
"some invalid states" - what does this mean, please? How do I constrain the "some" part? If you can't, you might as well say "mostly invalid states", which is what tends to happen in practice.
The whole point of state machines/type constraints/foreign key constraints/protocol definitions is that there is a clear boundary between what is allowed and what is not. And I would argue that this is what makes life easier, because you can tell one from the other. And with the right tooling, the compiler or some conformance tool will tell you that the change you just introduced, breaks your implementation in 412 places.Because this allows me to stop and think whether I can make a different change that only creates 117 problems. Or estimate the cost of the changes and compare it with the business benefit, and have appropriate conversations with stake holders. Or find out that my domain model was completely wrong to begin with and start refactoring my way out of this. And all of this before the first successful compile.
For me, this gives me maximum flexibility, because I have a good idea of the impact of any changes on the first day of development. This does require appropriate tooling though, like you would find in Ocaml, F#, Rust, State Machine Compilers, ... <insert your favourite tool here>.
The protobuf example seems best handled with version numbers in the messages. Designing protocols without them is almost an antipattern, especially when the designer says "there's no version number because I'm so smart, I'm going to get it right the first time". The examples that I remembered now seem to have been scrubbed from the web ;).
The thing about making invalid states unrepresentable is that we are often overconfident in what counts as invalid. What counts as valid and invalid behaviour is given by the requirements specification, but if there's anything that changes frequently throughout development, it's the requirements. What's invalid today might be desirable tomorrow.
Thus, we have to be careful at which level we make invalid states unrepresentable.
If we follow where Parnas and Dijkstra suggested we go[1], we'll build software with an onion architecture. There, the innermost layers are still quite primitive and will certainly be capable of representing states considered invalid by the requirements specification! Then as we stack more layers on top, we impose more constraints on the functionality that can be expressed, until the outermost layer almost by accident solves the problems that needed to be solved. The outermost layers are where invalid states should be unrepresentable.
What often happens in practice when people try to make invalid states unrepresentable is they encode assumptions from the requirements specification into the innermost layers of the onion, into the fundamental building blocks of the software. That results in a lot of rework when the requirements inevitably change. The goal of good abstraction should be to write the innermost layers such that they're usable across all possible variations of the requirements – even those we haven't learned yet. Overspecifying the innermost layers runs counter to that.
In the example of the app market place, any state transitions that are well-defined should be allowed at the inner layer of the abstraction that manages state transitions, but the high-level API in which normal transitions are commanded should only allow the ones currently considered valid by the requirements.
Aside from the flaws of the article, one provided example annoys me:
> What happens when you need to account for “official” apps, which are developed internally and shouldn’t go through the normal review process?
There is a reason devs are advised to eat their own dogfood. Building a process-bypass for the users that are also the ones responsible for fixing the process is the easiest way to get a broken process.
This article pulls a sleight of hand in which it tries to have states which are "invalid" and yet which are in fact properly handled - and so there's no reasonable sense in which they were in fact invalid. So it's just an appeal to think harder when designing a system.
A state enumeration becomes just free text, a count becomes just a number, and too bad now it's impossible to reason about the state of the system because everything is unknowable. Why is this app at the top of every chart? Oh, it has recorded negative one billion users and its state is just the URL of the app store, which causes mayhem in the scoring sub-routine.
39 comments
[ 3.2 ms ] story [ 56.1 ms ] threadIn accountancy there is the "General Journal" (https://en.wikipedia.org/wiki/General_journal) - a place to correct of accounting errors, enter adjustments, etc.
The General Journal really only works because accounting entries are immutable: corrections are always new entries. So it's not clear if "give corner cases a home" works everywhere.
But if the principle does work for your use case, it lets the rest of the system be as strict as you like!
I've also seen good advice that you should never delete anything from your DB, but rather put rows in a different soft-deleted state...
But of course, some flexibility is often (not always) needed in the long run, and knowing where to keep the flexibility and where to push for strictness is an important part of skills development as an engineer.
Which side of the argument one falls on is likely to be heavily influenced by which language they're writing. The mantra is likely worth sticking to heavily in, say, Haskell or Rust, and I've had plenty of success with it in Swift. Go or Java on the other hand? You'd probably want to err on the side of flexibility because that suits the language more and you can rely on the compiler less during development.
I talk about types because most of the "make invalid states unrepresentable" dogma comes from proponents of languages with extensive static type systems, and the idea is to use the type system to define all forms of legal states.
While it isn't strictly harmful, one drawback of this approach is that if you perfectly bit-pack every field such that random noise can be interpreted as a well-formed packet in your protocol, it will be difficult to heuristically identify the protocol.
For scientific simulations, I almost always want invalid state to immediately result in a program crash. Invalid state is usually due to a bug. And invalid state is often the type of bug which may invalidate any conclusions you'd want to draw from the simulation.
For data analysis, things are looser. I'll split data up into data which is successfully cleaned to where invalid state is unrepresentable and dirty data which I then inspect manually to see if I am wrong about what is "invalid" or if I'm missing a cleaning step.
I don't write embedded software (although I've written control algorithms to be deployed on it and have been involved in testing that the design and implementation are equivalent), but while you can't exactly make every invalid state unrepresentable you definitely don't punch giant holes in your state machine. A good design has clean state machines, never has an uncovered case, and should pretty much only reach a failure state due to outside physical events or hardware failure. Even then, if possible the software should be able to provide information to intervene to fix certain physical issues. I've seen devices RMA's where the root cause was the FPU failed; when your software detects the sort of error that might be hardware failure, sometimes the best you can do is bail out very carefully. But you want to make these unknown failures be a once per thousands or millions of device years event.
Sean is writing mostly about distributed systems where it sounds like it's not a big deal if certain things are wrong or there's not a single well defined problem being solved. That's very different than the domains I'm used to, so the correct engineering in that situation may more often be to allow invalid state. (EDIT: and it also seems very relevant that there may be multiple live systems updated independently so you can't just force upgrade everything at once. You have to handle more software incompatibilities gracefully.)
The database is not your domain model, it is the storage representation of your domain model on disk. Your REST/grpc/whatever API also isn’t your domain model, but the representation of your domain model on the wire.
These tools (database, protocols) are not the place to enforce making invalid states un-representable for reasons the article mentions. You translate your domain model into and out of these tools, so code your domain model as separately and as purely as you can and reject invalid states during translation.
(This whole approach reminds me of "I think therefore anything, due to the false antecedent".)
To illustrate what I mean, take the null pointer. It's essentially an Optional<T> and in a lot of languages it's involuntary, meaning any value is implicitly optional. What happens when something returns null, but in some place in your logic you don't take it into account? Null pointer dereference.
The rest of it is arguing about what constitutes valid states, eg, commit messages when the user deletes or wrong-schema datagrams being partially utilized. Here’s the rub: if you don’t allow invalid states, you’re forced to actually have that discussion (“actually, messages don’t strictly require a user”; “only these fields are required, others are a maybe”) and those are then encoded in your software rules. They’re not things that happen informally, in undocumented ways.
Further, when you do allow invalid states, you have a typo disappear 1% of your messages into the “pubilshed” category.
> A foreign key constraint forces user_id to correspond to an actual row in the users table. If you try to create or update a post with user_id 999, and there is no user with that id, the foreign key constraint will cause the SQL query to fail.
> This sounds great, right? A record pointing at a non-existent user is in an invalid state. Shouldn’t we want it to be impossible to represent invalid states? However, many large tech companies - including the two I’ve worked for, GitHub and Zendesk - deliberately choose not to use foreign key constraints. Why not?
> The main reason is flexibility. In practice, it’s much easier to deal with some illegal states in application logic (like posts with no user attached) than it is to deal with the constraint. With foreign key constraints, you have to delete all related records when a parent record is deleted. That might be okay for users and posts - though it could become a very expensive operation - but what about relationships that are less solid? If a post has a reviewer_id, what happens when that reviewer’s account is deleted? It doesn’t seem right to delete the post, surely. And so on.
So if you need to keep reviews by a user when the user can be deleted, you have a few options:
- Use a foreign key constraint but also allow nullable foreign key values (I believe DBMSes such as postgres and mysql support this)
- Create a record representing an orphaned user and move reviews to it when the user is deleted (generally, the reviews will then show as posted by user "DELETED", though you can also enforce different ways to display deleted user associations)
- Soft delete the users (in this case you also have to exercise caution around what ex-user data may be displayed where)
There are tradeoffs with each one; soft deletion of users would likely run afoul of Right to be Forgotten laws in some places. I'd tend to favour option 1 for anything that references something like a user account where the user may be deleted, but option 2 might be useful in situations as well.
I'd generally recommend against omitting foreign key constraints when it can be assumed databases backing your application will support them.
Sure, losing the FK constraint gives you more "flexibility" (I guess?) but introduces way too many footguns. The database and tooling are a massive help in avoiding so many other bugs.
Sure, maybe one day you decide reviews can have authorship associated with entities which are not users of your system (say you've acquired republication data rights from another review service whose users are not your service's users); in that case you'd need an application-level refactor and migration. You might need to add an author table, where the author can be a user of your service, or a reference to a user of another service . Then users of your service become possible authors, and all reviews need to be migrated to have an FK relation to the author table now rather than the user table.
In any case, the FK doesn't prevent you from changing how your data is structured, and I'd argue it greatly helps you avoid mistakes as you move through the migration.
If my experience, writing code that is “future migration friendly” is one of the hallmarks of senior developer knowledge. It’s just the kind of thing that burrows into your brain, because most senior devs have seen, built, or maintained code with randomly cordoned off “WARNING NONSENSICAL THING HERE” style sections of massive codebases that are artifacts of various migration things.
For context about how we do it, we use a giant NoSQL single table design, with zero joins. All joining is handled in the application, which ends up forcing everyone to consider how to parallelize and lookahead rather than join. Additionally, any complex data is stored as row level binary blobs from a versioned schema, which also helps localize issues to a particular subtype rather than leaking it into all schemas (for example if you change what qualifies as an “update”, your “updatedAt” column doesn’t suddenly change meanings for all things.
The whole point of state machines/type constraints/foreign key constraints/protocol definitions is that there is a clear boundary between what is allowed and what is not. And I would argue that this is what makes life easier, because you can tell one from the other. And with the right tooling, the compiler or some conformance tool will tell you that the change you just introduced, breaks your implementation in 412 places.Because this allows me to stop and think whether I can make a different change that only creates 117 problems. Or estimate the cost of the changes and compare it with the business benefit, and have appropriate conversations with stake holders. Or find out that my domain model was completely wrong to begin with and start refactoring my way out of this. And all of this before the first successful compile.
For me, this gives me maximum flexibility, because I have a good idea of the impact of any changes on the first day of development. This does require appropriate tooling though, like you would find in Ocaml, F#, Rust, State Machine Compilers, ... <insert your favourite tool here>.
Thus, we have to be careful at which level we make invalid states unrepresentable.
If we follow where Parnas and Dijkstra suggested we go[1], we'll build software with an onion architecture. There, the innermost layers are still quite primitive and will certainly be capable of representing states considered invalid by the requirements specification! Then as we stack more layers on top, we impose more constraints on the functionality that can be expressed, until the outermost layer almost by accident solves the problems that needed to be solved. The outermost layers are where invalid states should be unrepresentable.
What often happens in practice when people try to make invalid states unrepresentable is they encode assumptions from the requirements specification into the innermost layers of the onion, into the fundamental building blocks of the software. That results in a lot of rework when the requirements inevitably change. The goal of good abstraction should be to write the innermost layers such that they're usable across all possible variations of the requirements – even those we haven't learned yet. Overspecifying the innermost layers runs counter to that.
In the example of the app market place, any state transitions that are well-defined should be allowed at the inner layer of the abstraction that manages state transitions, but the high-level API in which normal transitions are commanded should only allow the ones currently considered valid by the requirements.
[1]: https://entropicthoughts.com/deliberate-abstraction
An inner layer that permits lots of invalid behavious is akin to building a house on sand. Even Jesus had a parable about that!
> What happens when you need to account for “official” apps, which are developed internally and shouldn’t go through the normal review process?
There is a reason devs are advised to eat their own dogfood. Building a process-bypass for the users that are also the ones responsible for fixing the process is the easiest way to get a broken process.
A state enumeration becomes just free text, a count becomes just a number, and too bad now it's impossible to reason about the state of the system because everything is unknowable. Why is this app at the top of every chart? Oh, it has recorded negative one billion users and its state is just the URL of the app store, which causes mayhem in the scoring sub-routine.