Bertrand Meyer suggested another way to consider this that ends up in a similar place.
For concerns of code complexity and verification, code that asks a question and code that acts on the answers should be separated. Asking can be done as pure code, and if done as such, only ever needs unit tests. The doing is the imperative part, and it requires much slower tests that are much more expensive to evolve with your changing requirements and system design.
The one place this advice falls down is security - having functions that do things without verifying preconditions are exploitable, and they are easy to accidentally expose to third party code through the addition of subsequent features, even if initially they are unreachable. Sun biffed this way a couple of times with Java.
But for non crosscutting concerns this advice can also be a step toward FC/IS, both in structuring the code and acclimating devs to the paradigm. Because you can start extracting pure code sections in place.
Down at the bottom it gets into composition to make utility functions that compose several operations. Any OO system has to be careful not to expose methods that should have been private, so that’s not specific to CQS. It’s just that the opportunities to get it wrong increase and the consequences are higher.
Many times, it has confused my co-workers when an error creeps in in regards to where is the error happening and why? Of course, this could just be because I have always worked with low effort co-workers, hard to say.
I have to wonder if programming should have kept pascals distinction between functions that only return one thing and procedures that go off and manipulate other things and do not give a return value.
These chains become easy to read and understand with a small language feature like the pipe operator (elixir) or threading macro (clojure) that takes the output of one line and injects it into the left or rightmost function parameter. For example:
(Elixir)
"go "
|> String.duplicate(3) # "go go go "
|> String.upcase() # "GO GO GO "
|> String.replace_suffix(" ", "!") # "GO GO GO!"
;; Using the thread-last macro
(->> '(1 2 3 4)
(filter even?) ; The list is passed as the last argument
(map double)) ; The result of filter is passed as the last argument
;=> (4.0 8.0)
Things like this have been added to python via a library (Pipe) [1] and there is a proposal to add this to JavaScript [2]
If an error result is returned by any of the functions, it terminates immediately and returns the error there. You can write this in most languages, even imperative/oop languages. In java, they have a built in class called Optional with options to treat null returns as empty:
or something close to that, I haven't used java in a couple years.
C++ also added a std::expected type in C++23:
auto result = some_expected()
.and_then(another_expected)
.and_then(third_expected)
.transform(/* ... some function here, I'm not familiar with the syntax*/);
This works right up to the point where you try to make the code to support opening transactions functional. :D
Some things are flat out imperative in nature. Open/close/acquire/release all come to mind. Yes, the RAI pattern is nice. But it seems to imply the opposite? Functional shell over an imperative core. Indeed, the general idea of imperative assembly comes to mind as the ultimate "core" for most software.
Edit: I certainly think having some sort of affordance in place to indicate if you are in different sections is nice.
What if a FCF (functional core function) calls another FCF which calls another FCF? Or do we do we rule out such calls?
Object Orientation is only a skin-deep thing and it boils down to functions with call stack. The functions, in turn, boil down to a sequenced list of statements with IF and GOTO here and there. All that boils boils down to machine instructions.
So, at function level, it's all a tree of calls all the way down. Not just two layers of crust and core.
I invented this pattern when I was working on a small ecommerce system (written in Scheme, yay!) in the early 2000s. It just became much easier to do all the pricing calculations, which were subject to market conditions and customer choices, if I broke it up into steps and verified each step as a side-effect-free, data-in-data-out function.
Of course by "invented" I mean that far smarter people than me probably invented it far earlier, kinda like how I "invented" intrusive linked lists in my mid-teens to manage the set of sprites for a game. The idea came from my head as the most natural solution to the problem. But it did happen well before the programming blogosphere started making the pattern popular.
Even large companies are still grasping at straws when it comes to good code. Meanwhile there are articles I wrote years ago which explain clearly from first principles why the correct philosophy is "Generic core, specific shell."
I actually remember early in my career working for a small engineering/manufacturing prototyping firm which did its own software, there was a senior developer there who didn't speak very good English but he kept insisting that the "Business layer" should be on top. How right he was. I couldn't imagine how much wisdom and experience was packed in such simple, malformed sentences. Nothing else matters really. Functional vs imperative is a very minor point IMO, mostly a distraction.
Non-functional core tends to become a buggy mess, with workarounds in the shell to account for those bugs, and then one needs to know about the nature of the core to correctly use the shell and so on. Functional core will lend itself very well to unit tests. Writing them will almost be trivial, when functional core is done right. Imperative shell is then less of an issue, because the blast radius of bugs is reduced to one usage of the core. The imperative shell should be kept as small as reasonably possible of course.
In my head, and the way its describe the generic and specific are swapped. The core handles a specific, pure problem. The core is generic (do you get the data via http, databases, filesystem, etc) then becomes irrelevant to the core problem
I would even go so far to say, that large companies even struggle with this more than small ones. The amount of people needing to know how to build things properly is larger than in a small company, where one knowledgeable engineer might already be sufficient. Too many cooks spoiling the soup(/broth?). And lots of people are cooking these days.
> Meanwhile there are articles I wrote years ago which explain clearly from first principles why the correct philosophy is ...
I think this is a very common mistake. You've spent years, maybe decades, writing code and now you want to magically transfer all that experience in a few succinct articles. But no advice that you give about "the correct philosophy" is going to instantly transfer enough knowledge to make all large companies write good code, if only they followed it. Instead, I'm sure it's valuable advice, but more along the lines of a fragment within a single day of learning for a diligent developer.
A company I worked recently had a more extreme version of this mistake. It had software written in the 1980s based on a development process by Michael Jackson (no, not that one!), a software researcher that had spent his whole career trying to come up with silly processes that were meant to fix software development once and for all; he wrote whole books about it. I remember reading a recent interview with him where he mourns that developers today are interested in new programming languages but not development methodologies. (The code base I worked on was fine by the way, given that it was 40 years old, but not really because of this Jackson stuff.)
I'm reminded of the Joel on Software article [1] where he compares talented (naturally or through experience) developers as being like really talented expert chefs, and those following some methodology as being like people working at McDonald's.
Isn't this saying business layer should not be on the top?
Business layers should be accessible via an explicit interface/shape that is agnostic to the layers above it. So if the org decides to move from mailchimp to some other email provider the business logic can remain untouched and you just need to write some code mapping the new provider to the business logic's interface.
Maybe our visualizations are mixed up, but I always viewed things like cloud providers, libraries etc. as potentially short lived whereas the core logic could stick around forever.
When you're not relying on a compiler you just have to right good code.
And it's easier if the code never has to change, only grow. I know a confident Perl programming who rarely changes his mind about anything. When he codes something he keeps it.
I always feel like I have to "maintain" code so I usually get bored after 3k lines of code, but truth is code doesn't have to be maintained if we like it the way it is, which obviously includes all the functionality that comes with it.
I don't really like the example (and it's from Google) because, beyond the general concept, it seems like the trigger for sending emails is calling bulkSend with Date.now() instead of the user actually triggering an email when it's really expired: user.subscriptionEndDate change to < Date.now().
I wrote our AI agents code with a functional core + imperative shell and I have to agree: this approach yields much faster cycle times because you can run pure unit tests and it makes testing a lot easier.
We have tens of thousands of lines of code for the platform and millions of workflow runs through them with no production errors coming from the core agent runtime which manages workflow state, variables, rehydration (suspend + resume). All of the errors and fragility are at the imperative shell (usually integrations).
Some of the examples in this thread I think get it wrong.
The effect of this is that the inner logic of `bulkSend` is completely decoupled from I/O and external logic. Now there's no need for mocking or integration tests because it is possible to use pure unit tests by simply swapping out the functions. I can easily unit test `bulkSend` because I don't need to mock anything or know about the inner behavior.
I chose this approach because writing integration tests with LLM calls would make the testing run too slowly (and costly!) so most of the interaction with the LLM is simply a function passed into our core where there's a lot of logic of parsing and moving variables and state around. You can see here that you no longer need mocks and no longer need to spy on calls because in the unit test, you can pass in whatever function you need and you can simply observe if the function was called correctly without a spy.
It is easier than most folks think to adopt -- even in imperative languages -- by simply getting comfortable working with functions at the interfaces of your core API. Wherever you have I/O or a parameter that would be obtained from I/O (database call), replace it with a function that returns the data instead. Now you can write a pure unit test by just passing in a function in the test.
I am very surprised how many of the devs on the team never write code that passes a function down.
Great examples. We were taught to pass variables, scalar or compound, into API's. Most of us were never taught to pass functions.
Even Python examples in trainings that look functional might not be. They put the function calls in as arguments. The beginner thinks the function returns some data, that would be in a variable, and they are implicitly passing that variable. Might as well, for readability, do the function call first to pass a well-named variable instead.
That was my experience. That plus minimizing side effects in functions. I've yet to really learn functional programming where I'd think to pass a function in an API. What are the best articles or books for us to learn that in general or in Python?
I would argue that the real key is to have a distinct core and shell, and to hold the core to a much higher standard of quality than the shell. In this article, being "functional" is just serving as a proxy for code quality.
Haskell practically encourages this style of programming. Any function that touches IO needs to wrap outputs with an appropriate monad. It becomes easier to push all IO out to the edges of your program and keep your core purely functional with no monads
I like the idea but the example doesn't make much sense.
In what application would you load all users into memory from database and then filter them with TypeScript functions? And that is the problem with the otherwise sound idea "Functional core, imperative shell". The shell penetrates the core.
Maybe some filters don't match the way database is laid out, what if you have a lot of users, how do you deal with email batching and error handing?
So you have to write the functional core with the side effect context in mind, for example using query builder or DSL that matches the database conventions. Then weave it with the intricacies of your email sender logic, maybe you want iterator over the right size batches of emails to send at once, can it send multiple batches in parallel?
db.getUsers() I am sorry what? Who in their right mind loads all users from the database and then filters out the expired subscription ones. Shouldn't the database query do this?
The MirageOS project [0] is a great collection of functionality pure OCaml libraries that are useful outside of unikernels. I've used the DNS library with an effectful layer for various nameserver experiments [1].
48 comments
[ 3.4 ms ] story [ 60.9 ms ] threadFor concerns of code complexity and verification, code that asks a question and code that acts on the answers should be separated. Asking can be done as pure code, and if done as such, only ever needs unit tests. The doing is the imperative part, and it requires much slower tests that are much more expensive to evolve with your changing requirements and system design.
The one place this advice falls down is security - having functions that do things without verifying preconditions are exploitable, and they are easy to accidentally expose to third party code through the addition of subsequent features, even if initially they are unreachable. Sun biffed this way a couple of times with Java.
But for non crosscutting concerns this advice can also be a step toward FC/IS, both in structuring the code and acclimating devs to the paradigm. Because you can start extracting pure code sections in place.
https://hemath.dev/blog/command-query-separation/
Down at the bottom it gets into composition to make utility functions that compose several operations. Any OO system has to be careful not to expose methods that should have been private, so that’s not specific to CQS. It’s just that the opportunities to get it wrong increase and the consequences are higher.
email.bulkSend(generateExpiryEmails(getExpiredUsers(db.getUsers(), Date.now())));
Many times, it has confused my co-workers when an error creeps in in regards to where is the error happening and why? Of course, this could just be because I have always worked with low effort co-workers, hard to say.
I have to wonder if programming should have kept pascals distinction between functions that only return one thing and procedures that go off and manipulate other things and do not give a return value.
https://docs.pascal65.org/en/latest/langref/funcproc/
(Clojure) ;; Nested function calls (map double (filter even? '(1 2 3 4)))
;; Using the thread-last macro (->> '(1 2 3 4) (filter even?) ; The list is passed as the last argument (map double)) ; The result of filter is passed as the last argument ;=> (4.0 8.0)
Things like this have been added to python via a library (Pipe) [1] and there is a proposal to add this to JavaScript [2]
1: https://pypi.org/project/pipe/ 2: https://github.com/tc39/proposal-pipeline-operator
C++ also added a std::expected type in C++23:
Some things are flat out imperative in nature. Open/close/acquire/release all come to mind. Yes, the RAI pattern is nice. But it seems to imply the opposite? Functional shell over an imperative core. Indeed, the general idea of imperative assembly comes to mind as the ultimate "core" for most software.
Edit: I certainly think having some sort of affordance in place to indicate if you are in different sections is nice.
What if a FCF (functional core function) calls another FCF which calls another FCF? Or do we do we rule out such calls?
Object Orientation is only a skin-deep thing and it boils down to functions with call stack. The functions, in turn, boil down to a sequenced list of statements with IF and GOTO here and there. All that boils boils down to machine instructions.
So, at function level, it's all a tree of calls all the way down. Not just two layers of crust and core.
Of course by "invented" I mean that far smarter people than me probably invented it far earlier, kinda like how I "invented" intrusive linked lists in my mid-teens to manage the set of sprites for a game. The idea came from my head as the most natural solution to the problem. But it did happen well before the programming blogosphere started making the pattern popular.
I actually remember early in my career working for a small engineering/manufacturing prototyping firm which did its own software, there was a senior developer there who didn't speak very good English but he kept insisting that the "Business layer" should be on top. How right he was. I couldn't imagine how much wisdom and experience was packed in such simple, malformed sentences. Nothing else matters really. Functional vs imperative is a very minor point IMO, mostly a distraction.
I think this is a very common mistake. You've spent years, maybe decades, writing code and now you want to magically transfer all that experience in a few succinct articles. But no advice that you give about "the correct philosophy" is going to instantly transfer enough knowledge to make all large companies write good code, if only they followed it. Instead, I'm sure it's valuable advice, but more along the lines of a fragment within a single day of learning for a diligent developer.
A company I worked recently had a more extreme version of this mistake. It had software written in the 1980s based on a development process by Michael Jackson (no, not that one!), a software researcher that had spent his whole career trying to come up with silly processes that were meant to fix software development once and for all; he wrote whole books about it. I remember reading a recent interview with him where he mourns that developers today are interested in new programming languages but not development methodologies. (The code base I worked on was fine by the way, given that it was 40 years old, but not really because of this Jackson stuff.)
I'm reminded of the Joel on Software article [1] where he compares talented (naturally or through experience) developers as being like really talented expert chefs, and those following some methodology as being like people working at McDonald's.
[1] https://www.joelonsoftware.com/2001/01/18/big-macs-vs-the-na...
Business layers should be accessible via an explicit interface/shape that is agnostic to the layers above it. So if the org decides to move from mailchimp to some other email provider the business logic can remain untouched and you just need to write some code mapping the new provider to the business logic's interface.
Maybe our visualizations are mixed up, but I always viewed things like cloud providers, libraries etc. as potentially short lived whereas the core logic could stick around forever.
I always feel like I have to "maintain" code so I usually get bored after 3k lines of code, but truth is code doesn't have to be maintained if we like it the way it is, which obviously includes all the functionality that comes with it.
We have tens of thousands of lines of code for the platform and millions of workflow runs through them with no production errors coming from the core agent runtime which manages workflow state, variables, rehydration (suspend + resume). All of the errors and fragility are at the imperative shell (usually integrations).
Some of the examples in this thread I think get it wrong.
This is already wrong because the call already starts with I/O; flip it and it makes a lot more sense.What you really want is (in TS, as an example):
The effect of this is that the inner logic of `bulkSend` is completely decoupled from I/O and external logic. Now there's no need for mocking or integration tests because it is possible to use pure unit tests by simply swapping out the functions. I can easily unit test `bulkSend` because I don't need to mock anything or know about the inner behavior.I chose this approach because writing integration tests with LLM calls would make the testing run too slowly (and costly!) so most of the interaction with the LLM is simply a function passed into our core where there's a lot of logic of parsing and moving variables and state around. You can see here that you no longer need mocks and no longer need to spy on calls because in the unit test, you can pass in whatever function you need and you can simply observe if the function was called correctly without a spy.
It is easier than most folks think to adopt -- even in imperative languages -- by simply getting comfortable working with functions at the interfaces of your core API. Wherever you have I/O or a parameter that would be obtained from I/O (database call), replace it with a function that returns the data instead. Now you can write a pure unit test by just passing in a function in the test.
I am very surprised how many of the devs on the team never write code that passes a function down.
Even Python examples in trainings that look functional might not be. They put the function calls in as arguments. The beginner thinks the function returns some data, that would be in a variable, and they are implicitly passing that variable. Might as well, for readability, do the function call first to pass a well-named variable instead.
That was my experience. That plus minimizing side effects in functions. I've yet to really learn functional programming where I'd think to pass a function in an API. What are the best articles or books for us to learn that in general or in Python?
In what application would you load all users into memory from database and then filter them with TypeScript functions? And that is the problem with the otherwise sound idea "Functional core, imperative shell". The shell penetrates the core.
Maybe some filters don't match the way database is laid out, what if you have a lot of users, how do you deal with email batching and error handing?
So you have to write the functional core with the side effect context in mind, for example using query builder or DSL that matches the database conventions. Then weave it with the intricacies of your email sender logic, maybe you want iterator over the right size batches of emails to send at once, can it send multiple batches in parallel?
[0] https://mirage.io/
[1] https://ryan.freumh.org/eon.html