Pragmatic handover of released code and DB to a less technical CRUD team. Avoiding unproven non-anecdotically solutions like EventStore yet still shipping quickly. Avoiding becoming an F#/Scala snob or a JavaScript monkey.
Well it's pretty proven, it's just also likely overkill for what you need. Use it in your personal hobby projects so that you know the costs, and benefits. That way you don't blindly apply it where it doesn't fit (which admittedly will be most places).
Workflows for which some steps depends on things: the current real state of an entity, the permissions the current user has etc.
In a client-server system like a web API where you want to respond to the client as fast as possible having to reconstruct your entity from snapshot + some history can take some time. So you end-up with corrective events and responses which are "we have noted your request, poll us to know the result when it's done". The worst is the ES system is already in use in your RDBMS if you use one.
Most CQRS + ES demonstration are done with the simple happy path. Rarely something like user A changes the role of user B. User B was saving changes on a document they can't access anymore: what happen?
Now with software on your PC or with a constant link to the server things feel a lot more useful.
On the CQRS+ES project I worked on for two years, we were just thinking about the happy path at first before realizing that was a mistake.
For your example, here are two approaches depending on your implementation and your tolerance for edge cases and eventual consistency:
- you can enter into a saga which would do a two phase commit, ensuring that the user is indeed authorized when the edit is made. All possible states are captured in your event model and so your event history will always show what has happened
- you can load the document aggregate when you process the command and query it at that time, throwing an exception if the user is not authorized and then you can notify the user or send another command. In this case, your event model does not have to include events that describe the violation of domain rules
The first option gives you the guarantee you are looking for, but comes at the cost of complexity. The second option will give you the correct results unless a user is deauthorized before the edit command completes. Some systems process all commands in serial and so it would not be a problem. Some systems partition their commands to distribute the work and so this edge case can occur.
I think many people just dislike DDD and CQRS/ES because it's associated with over engineering.
Either you are really good and get things right without much help, which isn't often the case, or you end up in two other directions.
You over-engineer with DDD/CQRS/ES.
You under-engineer without them.
Most developers I met preferred the last solution, probably because they end up with their "own" mess and not the mess of some process.
And I have to admit, the only people I met, that used these techniques, were some "digital transformation" consultants who made their living by selling this stuff to big corps, which isn't too convincing either.
CQRS/ES definitely seems like a correct foundations for a distributed system, I just don't think we have the right abstractions to properly model this without wanting to stab yourself in the eye.
This is the sort of comments that get downvoted usually. You make a blanket statement backing it up with exactly zero characters of evidence or even academic theory. I found thinking in those terms helped me find architecture for webapps that made sense and was extendable and robust. Can you elaborate on why one should stay away?
Still, it should be noted that Fowler warns that CRQS adds significant complexity and advises great caution about using it. See https://martinfowler.com/bliki/CQRS.html
Greg Young also cautions "Do not tell me you've built an event sourced system"* (or something along those lines) meaning that you should be careful to apply the pattern only to the part(s) of the system that makes sense (due to the increased complexity of ES)
As a non-native German speaker who's lived in Germany for coming up on 10 years, aren't you concerned that it'll be pronounced like "wol-kenkit" not "volkenkit" for non-germans?
Naming things is hard but after watching UBER Inc destroy the pronunciation, spelling and etc of a German word, I'm always cautious about unusual names.
(Also because, as now I'm hung up on the name before giving your product a fair chance)
Well, yes, there's a certain risk. We had the idea that people liked the word "wolke" (German for "cloud"). Anyway, of course, I would be very happy if you gave wolkenkit a second chance ;-)
I did - reading the brochure now, the word "semantic" keeps cropping up and I wonder if it's an unsympathetic translation from German, or I'm too far away from the JS world to understand that "Semantic" means:
> That‘s why we have built wolkenkit, a semantic JavaScript backend that addresses three important aspects of software development, and that empowers you to build better software faster.
There's a bunch of things wrong with this claim, "a javascript backend" sounds like an alternative JS VM to me, a backend for javascript (I presume it's meant that this is server side JS not client side?) I also always get my back up when I read "build software faster", surely it ought to be about maintainability when it comes to the perks of DDD/CQRS.
To be fair, 'uber' had been in use as an adverb on the internet long before the company stepped on the scene, it's just that now it's in everyone's face …
Can you explain in one or two sentences what wolkenkit is doing exactly? From scrolling through the docs it sort of looks like meteor but with a different data model?
What is your reasoning for AGPL - it's a no-go for
a) indie developers because they can't afford the enterprise stack which is presumably not AGPL
b) enterprises because immediately expect a huge price tag
And last but not least: Why is this not platform agnostic (e.g. only node)?
There is a disconnect in your messaging. Here on HN the title is "Learn what DDD, CQRS, and event-sourcing are all about", but the tagline in the PDF is "The semantic JavaScript backend for event-driven development".
So, what's this about? about DDD, CQRS and event-sourcing, or about a particular Javascript framework? Those two aren't the same at all (even though I suspect there is a little overlap).
The brochure is to 2/3s an introduction to DDD, CQRS, and event-sourcing in general, not being related to a specific JavaScript framework.
The last 1/3 shows how to apply these concepts using wolkenkit, which is a specific framework.
When we created the brochure it seemed important to us to first explain the concepts in a neutral way, because we want to help people gain a better understanding of these concepts, no matter whether they are going to use wolkenkit or not.
While event sourcing is a quite alright tech for certain problems, this idea of "one solution fits all" is just ridiculous. Also the idea of "eventual consistency" is highly situational particularly in an distributed solution. Paradoxically looking at one event source log this looks like a great idea, but there are no guarantees that events propagate evenly and "in-order" so even if events happen and there's "nothing you can do about it", systems will still be out of order, when read in relation to each other because the systems won't know that any other event actually have happened. The system will happily consume it because it won't have the proper info or acted on the wrong information. No matter how you try to mitigate this this will happen and you won't have the slightest idea that it happened. Worst of all, you can't recover from it because the information is gone.
It's just fundamental aspect of asynchronous programming, conveniently forgotten trying to sell solutions.
This is a problem in how lots of people do event sourcing, but its not a fundamental problem with event sourcing. If you enforce strict ordering of events throughout your stack and do operation catchup properly, these sorts of bugs are completely avoidable.
Well that's the entire problem, you are not the one making the requirements, the solution of the problem does... You can't eat dinner before you have made it.
Edit: I would argue that you can't write a system which is open (several domains) and believe that you will have entirely independent events throughout the system and claim that your system will be "bugfree"/correct.
I could be missing something but I feel like you are inventing an impossible scenario. You may not be making the requirements, but the requirements have to be technically feasible. CQRS/DDD give you a lot of flexibility on where to choose your boundaries/aggregate roots and these are designed around the business/requirements. If you can give a valid scenario where this model breaks down but others would succeed I'd be very interested.
It's not requirements by "what to do" but the requirements of "what we need to make "what to do"" work. For example if you are going to add two numbers you have to use some sort of arithmetic to solve that problem, right? If you fail using arithmetic to solve it you will not solve the problem and your function will do something else. The requirement of solving in this particular case will be "use arithmetic".
So this means for your problem to be correct you have to use the correct tools to solve that problem. If you think that everything (all problems) can be decomposed to independent events, which per definition can happen in any order, you can safely use ES. But if they have any dependency on each other you need ordering, and ordering in a distributed system is a hard problem. And add to this is that somewhere some guy from the business will come up with an actual requirement where you must synchronize and if you then have solution which inherently where this is not possible you are screwed.
This is true for both reading and writing in that system. Most problems are dependent otherwise it won't solve/do anything, there are usually a purpose with most systems.
While you can have systems where the events are entirely independent, structurally independent, such as JS event loop and red/blue methods (just locally independent), or it doesn't matter like plain application logs, this is rarely, if not impossible between domains if you want a correct system since you won't know if all events have arrived yet to tigger another event, if there's any dependency between them, which there will be between two domains.
DDD is good but not in this context, CQRS is trying to partially to solve above problems, but not needed and is just bloating. It does fit better with C# since the (weird) implementation of async/await and potentially with JS for the same reasons. ES do have their use cases where they fit very well but not with the practical impacts of DDD.
If you look at this purely from an API design point of view, this stinks: by forcing the command to return void, the code ends up twisting its design to return the result of the command through an artificial channel (the state of the 'user' object). In the end, we are still interested in "the result of the login attempt", and the easiest way to do this would be a function that returns a value:
Reshaping data is the source of so many costly bugs and communications breakdowns in this world. CQRS is like descending to the 8th circle, pit 7 of reshaping hell. What fetish drives one to such tortures?
If your schema is any more complicated than a plain stream of characters, and you pick CQRS for your app, then you are an overpaid, bored, architecture astronaut consultant looking to pad his resume. Nevermind the multi-million dollar dumpster fire of an app you spearheaded and ditched out on before completing (and that got cancelled, of course, a complete loss.)
If I ever saw CQRS on a resume, I'd unconditionally pass, you're not setting foot in the door.
Are those objects in the queries and queues separate and distinct types then? And, what about the object store, what types go in there? How do we handle data migrations?
I like queues as much as the next developer. Queues let you scale performance Mt Everest and deal with concurrency.
You don't have to use queues, separate objects/NoSQL store or anything like that to use cqrs.
I mean you can use them, but all CQRS means is a separate code path for queries and commands.
In my system I have command objects, they operate on domain objects which contain some business logic.
I have query objects which just execute custom sql code which returns a populated view.
This is useful because views often display specific data from multiple domain objects.
This system allows you to bring back the entire view in a single query and nothing else, rather trying to retrieve multiple domain objects which are really designed for business rules rather than queries.
Often yes, people do starting adding queues, materialised views to CQRS which can make it complicated.
> Often yes, people do starting adding queues, materialised views to CQRS which can make it complicated.
You don't say?? You don't say??
Sorry to be flip, I hope you can tell that I've been burned. Badly. Thanks for your detailed response. You obviously care and are not a stupid person nor do you come off sounding vain.
But, you're not selling me on CQRS at all. What you've described is just classifications and organization for types in a sane sounding UI design that happen to have similar roles as types in a CQRS system.
If you were selling me on CQRS, you'd start by talking about data flow. Something like, "it's a realtime, live multiuser system where all commands are queued, transformed to system events, and those events are committed to log (via another queue, we have to synchronize after all) that stretches back into time. We can replay those events, rewind time, events are merged ..." And it goes on and on into the stratosphere of architecture astronaut-hood...
My problem with your response is I honestly can't trust what you say here until I do a code review. Is your actual code pathological, or are you whitewashing the constant twizzling and niggling you have to do to keep your ship afloat? How bad, honestly, do you have to work to add a new feature? What's your velocity like? I want to see how much work and layers there are in your code to do data type transformation.
Data transformation is where El Diablo lives, because data transformation requires lots of people to agree and business terms to meet code - it's the unholy marriage of garbage disposal, forest fire, and whirlwind. CQRS seems to invite needless data transforms that other architectures do not require, and that's the pukey taste in my mouth when I have discussions about CQRS.
"it's a realtime, live multiuser system where all commands are queued, transformed to system events, and those events are committed to log (via another queue, we have to synchronize after all) that stretches back into time. We can replay those events, rewind time, events are merged ..." And it goes on and on into the stratosphere of architecture astronaut-hood..."
But that isn't CQRS. That's event sourcing, a complicated implementation of event sourcing.
CQRS can work on a standard SQL system, you can design the database using a standard normalised database.
All that has to be different is that commands and queries are separate.
A simple implementation;
A command can literally be
var command= new AcceptOfferCommand(db);
command.Execute();
A query can be
var query = new AvailableOffersQuery(db);
var viewModel = query.Execute()
Inside the command you may retrieve a domain object from repository, and execute a method on it.
Inside the query method you may just perform a raw sql statement which grabs data from multiple tables, populates a viewModel and returns it. Rather trying to retrieve multiple domain objects from multiple repositories and then shuffling that data into a result..
That's one way of doing simple cqrs. It can be simpler than this. But this code just made it obvious.
CQRS allows you do very complicated things, but it doesn't mean you have to.
Ah, well CQRS is only justifiable when you're selling the work if you couple it with the ES, and the more complex it is, the better the sales job and the budget. It's like peanut butter and jelly.
My main point in replying to you was that you weren't selling me on whatever brand of CQRS you think you've got. Why? It seems like you're atomizing your business logic into a fine mist, why is that good? And you still haven't talked about all the myriad transforms you do: where's the payoff? Does YAGNI apply in your case?
Well each command is a use case. Each use case can be tested independently without having setup controllers and their dependencies, or big service layers and their dependencies.
Plus I can run the usecases from a console frontend for quick testing without much setup.
I don't really do any transforms other than mapping domain objects into the dB or database results into a query result.
Why do I need it? Because I practise DDD, and domain objects are not designed for queries. To populate a single view you have pull multiple domain objects from repos. This can be slow and adds complexity.
Query objects allow me to query the database using standard SQL or stored procedure and put the output into query result using only a single query. These queries can as complicated and optimised as they need to be.
Meaning I can avoid all DDD abstractions that I don't need for querying, but I do need for enforcing complicated business rules on the command side.
If you're able to reuse (through encapsulation or extension) your domain object types in your commands and queries without transformations, then your pattern sounds sane to me.
The entire point of the Separation in CQRS is to avoid unnecessary coupling between reporting concerns, GUI concerns, and user concerns.
The act of starting an order does not look like an order. Asking for an order cancellation does not look like an order. Using GraphQL to pull an order summary with report-specialized drill down capabilities does not look like an order. These specialized domain objects should be modeled individually. By trying to encapsulate all these facets in a single object what you get is tight coupling, compromises, domain impedance, logical errors, and slow development.
In context: One of the root principles of DDD is that each domain represents its entities as is befitting that domain. Domain entities should be meaningful in their domain and _not_ corrupted by every niggling redigestion of the same facts through tens, or hundreds, of simultaneous conflicting views... Reporting concerns should be first-class citizens when solving reporting, for example.... View concerns and high-load systems often require denormalization, materialization, transformation, and replication that has very little to do with the user of origin while also _destroying_ the premises of traditional ORM driven RMDB interactions.
At scale, under load, while supporting multiple N-tiered applications and their accordant services, unified domain models break down. Aggregate boundaries map these epistemological distinctions.
CQRS on top of that is just good engineering. In the same way modern car-tunnel builders build two separate tunnels for traffic in each direction, CQRS isolates low-frequency user interaction from high-frequency reporting (or vice versa!), allowing independent scaling and direct delivery of features without impossible coordination activities between teams or finding silver-bullet technologies that may or may not exist.
That said: these are Enterprise solutions to Enterprise issues... Scale is the reason they exist. If you aren't looking at a horrible mess around a unified domain model then these solutions might not apply. If you aren't dealing with multiple simultaneous data delivery platforms (mixing SQL server and Oracle and Mongo DB and S3 and DocumentDB and DynamoDB all at once), these solutions may not apply. If you're not balancing multiple teams working on multiple products that have a constant need for harmonic data exchange, these solutions may not apply. If you're not juggling BigData, if you're not dealing with complex business logic, if scaling up an extra ten thousand users isn't something you deal with, if you're not supporting multiple client types, and so on: these solutions may not apply.
Commands are objects that express a command. A query is something that provides a value.
You can code it up in minutes, and most of the code is pure plumbing to connect to datasources. If that sounds complicated to buy independent control over data exchange then you're simply not exposed to the complex multi-layered and distributed applications that demand a structured approach to the issues CQRS handles. Unquestioned and implicit assumptions are great for small potatoes.
Domains should model domains and reflect the mental models of the people who work on, and with, the system. If that is experienced as complicating the only conclusion is that you're working with subject material neither complex nor interesting enough to warrant a considered architectural approach.
For someone who deals with computing (currying, monads, SOLID, etc), you should know that trying to meaningfully translate superficial domain complexity to generalizations of code quality or design quality of any given solution in all domains is laughable.
Laugh away, I'm not interested in flashy code nor resume padding. I am interested in SOLID, clean code. If you have clean code that is easy to add new features to, refactor, and teach to new developers, then good for you. I know good code when I see it. It is clear to me when developers lose sight of the forest for the trees.
Your sales technique makes your project sound like torture.
I'm not selling anything, and you clearly are missing the forest for the trees while making assumptions & assertions about unfamiliar things.
CQRS boils down to a single principle: two one-way roads make for easier road maintenance and improvement that one two-way road.
Two individual channels are too complicated to code? Torturous? Inherently badly designed? Hard to add new features too? Unclean? ... Crazy talk, unsupported by reality. In fact: CQRS specifically addresses all these points you've brought up through its decoupling.
Those kinds of prejudiced assumptions do not hold water, cannot be logically supported by the underlying design nor in-production systems, and reveals only domain ignorance.
Successful engineering around distributed Enterprise systems tells a clear story: those maintenance and improvement concerns are predominant at scale.
Opinions may vary... Informed opinions on this matter do not.
And now for the coup de grace: you produce a link to sources containing a shining example of CQRS.
Truly, one look at this archetypal wonder, (which certainly every programmer worth his salt has already studied its types and patterns at length), and all doubters shall melt away. If there ever were a more killer app demonstrative of the state of the art, it should be the muse of comp-sci journals or even popular culture!
A toy Microsoft CQRS spike on github from a couple years back does not count, neither does a book reference. I want a canonical example of something real: deployed code to users, thriving and successful. I'll be a happy to learn something new.
My suspicion however is you're all talk - a super-consultant who sold some CQRS work and has an axe to grind because you have co-workers reading this thread, including some of the client's devs. You arrogantly bragged to them and then waded out to sea because you thought me an easy mark to prove your claims to expertise on the subject.
I am prepared to gracefully concede to my betters.
I suppose I crave simplicity in code, with as little magic as possible, and with as few maddening descents into "not-invented-here" coding adventures when open source modules and frameworks exist that could do the job.
My use of the word "clean" refers to Uncle Bob's definition of clean code, not some scientology thing. He preaches the necessity of writing well-tested, high quality code via TDD. You mentioned SOLID earlier, which is another Uncle Bob-ism, and I figured you would get the clean reference. Not Uncle Bob's, but YAGNI's another acronym he likes to throw out there - YAGNI comes to mind whenever CQRS is mentioned. I bet YAGNI would be your worst nightmare if anyone around you had the courage to utter it. Are you a bully or do you just have no scope control on your projects from your business? Hey coworkers of bonesss, challenge everything this guy says with 'YAGNI', and see how he reacts!
Yeah, so anyways as I said in my other reply let's see your large scale code example that necessitates CQRS. Impress me with the high quality design and implementation, I'll do a code review.
> At scale, under load, while supporting multiple N-tiered applications and their accordant services, unified domain models break down. Aggregate boundaries map these epistemological distinctions.
I mean, I get it, you're talking about DDD. You've got context-bounded models. I've explored all this for over a decade. I apply the concepts daily (as mandated by our architecture, not my choice.) In the SOA days we had canonical models that are exposed on the service interfaces and domain-specific models that were representative of data coming from data sources or were internal application specifics and we always mediated between the models. It's nothing special. I sometimes think, if we all talked a little more we could make our terminology more consistent - but I get it, there's domains and different definitions and types will get mapped.
What you wrote is too pendantic, though. Epistemological! You're writing web apps, c'mon you are killing me!
Can anyone explain why eventual consistency is so scary? The way I see it, eventual consistency is everywhere. Even in your non-distributed synchronous CRUD app. The minute you start two db transactions within one script you have eventual consistency. You have to deal with the fact that one transaction can fail and that your state can be inconsistent.
It really depends on the application. Eventual consistency means that eventually, the databases will be consistent.
Some bank transactions, for example, do not have that luxury because it might allow 2 possible answers to the same query if the eventuality has not been met.
Some trading transactions don't have that luxury either because the milliseconds that you waited for the consistency a stock might have changed its price dramatically.
It really depends on the Domain if the time to wait for the eventuality is acceptable or not.
Banks have daily lag (pending transactions). They also stop business in the evening and resume in the morning to process all these items. That isn't feasible in a lot of scenarios.
Rarely do you have an up to the microsecond view of the markets (unless you are located appropriately and pay significant amounts of money). There is always a delay in the information's accuracy and relevancy.
I'd rather argue they are there for "domain" reasons. For one, banks often re-order transactions. I know my bank will process deposits first, then debits for the day. This way you are less likely to incur overdraft fees. It's also to fight fraudulent transactions. Although with the move to "real time" debit processing (which is actually two batch jobs a day) they are placing limits on the types of transactions that can be processed.
A domain that can't handle any delay in consistency is atypical in my experience. Either way you can do DDD/CQRS without eventual consistency.
They are there because they are used to think that way. It's conways law written all over it... There's no actual reason for them to do it. They replaced their paperwork with a system that does the same thing. Fraudulent transactions can you find without the need of summarizing it.
Yes you can do DDD/CQRS without ES but CQRS is not needed, it's just a technical construct not solving any verifiable problem. DDD on the other hand is more sane and to some extend, verifiable.
"Banks have daily lag (pending transactions). They also stop business in the evening and resume in the morning to process all these items. That isn't feasible in a lot of scenarios."
I live in the US, and if I got to the ATM and get cash on my account, I can go to my iPhone app and see the transaction and the balance updated.
Yes, there is a nightly process, but not everything is a nightly process.
"Rarely do you have an up to the microsecond view of the markets (unless you are located appropriately and pay significant amounts of money). There is always a delay in the information's accuracy and relevancy."
Bank transactions are eventually consistent. How do you think you can get an overdraft fee? Ever use a gas pump and notice it took a $1 authorization on your card then later settled for what you pumped?
Eventual consistency is not all that scary, try putting on a business hat instead of a technical one. Its about risk management.
> The minute you start two db transactions within one script you have eventual consistency.
You may also have two independent items of information that have no consistency relationship or requirements.
That's why eventual consistency "works" so "well" with NoSQL: because there's an overlap between using NoSQL databases and not requiring actual consistency.
Dealing with eventual consistency adds a lot of mental complexity to almost every feature. It turns a simple rule like "Do not allow registering an username if that username has already been registered" into an entire rulebook on what should happen if a username is registered twice. It even forces you to deal with the obvious "Once a gizmo is created, it should appear in the list of gizmos". Every part of your code becomes a potential race condition that you have to detect and handle appropriately.
You can't get rid of eventual consistency, obviously, but you can set up your system so that your code expresses rules and features in an "immediately" consistent abstraction, and the system then translates those rules and features into the eventually consistent world, by applying tactics and techniques that you would have otherwise applied manually. It's a really natural approach for programmers: automate the translation from immediate to eventual, instead of doing it yourself every time.
Of course, most of these tactics come at a cost, either of performance (we won't acknowledge the creation of a gizmo until we've propagated that creation to every server capable of rendering the list of gizmos) or of availability (the registration of your username can't be acknowledged unless the persistent consensus reflects that you're the only one who registered it). But that's fine: most of these things aren't critical enough to make it worth dealing with eventual consistency manually, and when they do become critical, it's time to go down a level of abstraction and apply immediate-to-eventual translation tactics that rely on additional assumptions about what you're doing in order to improve performance or availability.
> The minute you start two db transactions within one script you have eventual consistency.
Well it's all about errors (and subsequently about control and correctness). Even if you have two transactions you are able to undo the other without anyone else (readers of the system) know that it was there because something else happened just before yours. A transaction usually happens in an atomic fashion, which makes it possible to back out on something which is no more correct so you have to fail it somehow to someone which does have the ability to recover, the one who initiated the call.
Without being too philosophical, most, if not everything depends on synchronous state (fex events themselves), computing just gives the illusion of that things happens immediately. Eventual consistency is cheating, and you can do it where the events are entirely independent. Otherwise you will have side effects when events are not ordered.
JavaScript have an "asynchronous" coding style, where (red/blue) methods are put on event-loop as "events". But the events needs to executed in order (synchronously by a thread) for the program to behave correctly so you give the methods attributes (red/blue) for them to execute in an linked order. So even the asynchronous behaviour in JS relies on synchronous features for it to be correct!
The DDD/CQRS/ES topic appears on Hacker News every so often, and never fails to stir a strong debate. In fact, I was going to "Show HN" an Event Sourcing library I open sourced yesterday, but I guess I'll wait until everything dies down a bit :-)
I believe that the core principles of DDD/CQRS/ES are flawed and are the cause of a lot of pain. If your team plows forward and ignores the pain, you will have a bad experience working there. If your team decides to fix the problems with the architecture (usually in a way that's rather specific to their situation), and still insists that they're doing DDD/CQRS/ES, then you will have a great experience. The same can be said of many approaches to SQL database design.
When I joined Lokad, most of the code had been written by a strong CQRS advocate, and there were significant pain points caused by this approach that would have been nonexistent with a plain old SQL architecture. The author himself did a retrospective that covers a few issues: https://abdullin.com/lokad-cqrs-retrospective/
Instead of dropping everything and moving back to a more conventional SQL architecture, we dropped everything and moved to a solution which, if I were to describe it as DDD/CQRS/ES, would have me branded as a heretic by the DDD/CQRS/ES community: it does fit the definition, but it does not follow the best practices. There is a single data model shared by both commands and queries. Commands can sometimes return data. Each (micro?)service has access to exactly one aggregate. There is no uncertainty on event ordering, and no eventual consistency at the aggregate level. Materialized views for each aggregate are kept in-memory at all times. All of these intentional weakenings of the CQRS architecture were driven by pragmatic needs, and made with knowledge of the consequences.
I'm CQRS advocate. None of are weakening the CQRS. CQRS is in fact pretty damn lose.
"There is a single data model shared by both commands and queries." - Nothing wrong with that, especially in simple cases.
"Commands can sometimes return data" - Well commands can succeed or fail. Greg young himself admits commands are really synchronous, and you need to communicate that.
"and no eventual consistency at the aggregate level." - This is the recommended approach. Between aggregates and it becomes loser.
"Materialized views for each aggregate are kept in-memory at all times" - Thats some advanced CQRS usage there.
That write-up sounds absolutely terrifying. I really fear for what the .NET community might be doing with event sourcing.
Event sourcing is the definitive way to write high-performance front-office trading systems. Nothing else comes close to being able to quickly capture the rich domain of finance and satisfy stringent performance requirements.
But what that article describes is unlike any system I've seen in 15 years of fintech work. It's very clear that something has been lost in translation. I suspect the people are coming into this without having the solid foundation in event-driven architecture or high-performance messaging models.
As UK-AL points out everything you wrote is standard ES best practices. How is it even possible for a single aggregate to be "eventually consistent"? Aggregates by definition are atomically updated from one valid state to the next. Of course commands should return not just success/failure information but new information that is relevant to the sender (eg, OrderIDs for the Orders created). There's absolutely no reason to make the sender poll an event stream or "view."
"There is a single data model shared by both commands and queries." -- In the finance world this is impossible. Execution systems must be separated from reporting systems. And once you make this separation the whole CQRS-aspect naturally falls out. It makes sense, for example, to keep all orders in something like Cassandra or VoltDB for reporting though you'd never do that for execution. The thing is, CQRS is not an end to itself. I'm not sure why it has blown up in the .NET world. Again, something has been lost here. The CQRS tactic is required only when reporting requirements and execution requirements are fundamentally different/incompatible. An EDA makes CQRS-style separation possible but it does not require it. And here's the thing: if you have your end-points designed correctly it's trivial to separate out querying if it's needed. Just model your system using events and aggregates and it should work out.
I wasn't even going to comment on this thread. Most of the comments here are your typical HN nonsense -- a potent mixture of ignorance and anti-innovation FUD. But your link to the writeup was enlightening (and horrifying) if only to reveal how people can go very wrong with this "new" architectural style. (Again, event-sourcing, and EDA in general, has been de rigeur in finance since the 80s.)
Commands are by definition synchronous and return a success/failure.
One of my favourite read models to use is an in memory model providing I can keep all of the data in memory (and its reasonable for the query patterns).
hi - could you explain the implementation of this "Materialized views for each aggregate are kept in-memory at all times" ? i have been considering a heretical flavor of CQRS/ES as well, where the command returns data.... and i synchronously build the latest state before I return (rather than building through the whole event stream).
when you say "materialized views" - do you really mean the current state of an Order for example ? and by "in memory", is it being persisted to a database ?
What we would do is have a single aggregate representing the entire state of a micro-service. For instance, for the "Orders" micro-service, it would contain all orders in the system, with events such as OrderCreated, OrderPaymentReceived, OrderShipped, OrderCancelled, and so on.
Then, we would have a materialized view that would be a single C# immutable object (canonically called `State`) that contains a dictionary of every order, and maybe a few indices (such as "identifiers of orders currently waiting to be shipped out" or "list of orders by customer"). Everything fits in memory, so every query is simply a bit of C# code that traverses the in-memory graph of objectd rooted at the `State`.
A command emits events, which are then applied by the system to create a new `State` from the previous one. Events emitted by other instances of the application are also fetched by the system every so often, and you can request a "force catch-up" to guarantee that the `State` you access takes into account all events appended so far (so that you're certain to see the results of your actions, even if the load balancer bounces you from one instance to another).
Having read the article: veeeeeery little of what is mentioned is related in any meaningful sense to CQRS...
Horse-before-the-cart framework design is always bad. Encapsulating untested theories in frameworks instead of lifting them from operational designs, always bad. Building frameworks as a learning exercise, bad. Abstracting away data decisions and then trying to provide opinionated answers to storage at the framework level, very bad. Conflating logical system architecture with supplier provided products (ie cloud services), is framework _cancer_.
That is simply not a reasonable starting point to provide value and scalable systems, and the end result was the same as all other similar attempts: an A for effort with a result that was at best 'close, but not quite...'
To be clear: I do not intend this as criticism leveled at the author or the company. The article is great. I very much agree with his analysis though:
> Lokad.CQRS was created with a very short-sighted design approach in mind, a reusable LEGO constructor...These days I'd try to limit the damage I inflict upon the developers and avoid writing any widely reusable frameworks
This is where all the issues lay... Premature framework development causing up-front decisions untethered to practicality based on conjecture and prognostication instead of experience in a challenging domain insufficiently understood by the framework architect.
From a high level perspective: neither CQRS, nor framework development, nor application development were sufficiently groked before being baked into a long-term production commitment. Those are organizational issues, not CQRS issues. In the same vein, coming from such an environment and basing judgment of DDD or CQRS on anything that came from that environment won't hold water.
The architectural problems you're talking about, the core principles you think are flawed, work stunningly well in other places that have not made the same underlying mistakes. CQRS advocates != CQRS experts != framework experts.
Offtopic, but was the PDF typeset with layouting software or a markup converter such as Pandoc and the like? Looks really slick, kudos. Also is there a way to tell by PDF metadata?
Edit: Had a second look and it's layouting software
Consistency here is CAP consistency[1], not ACID consistency. In short, if datum d is visible to Alice at time t1, it MUST also be visible to Bob at time t2 > t1.
So financial transactions are transactional, but there's no guarantee that every node has a fully up-to-date state at all times.
This is also not how bank transfers work inter-bank. I as Bob (a recipient) may have zero knowledge of a transaction initiated by Alice regardless of time.
Wrong. Do you seriously think that a cross-bank transfer is operating inside a distributed database transaction? Furthermore, what did banks do before they had computer systems available?
Bank transactions are usually handled via long running messaging state machines. When as example you transfer from an account in the US to one in Sweden it does not just open a 2PC transaction between the two databases. There are also often intermediaries involved.
85 comments
[ 4.0 ms ] story [ 122 ms ] threadIn a client-server system like a web API where you want to respond to the client as fast as possible having to reconstruct your entity from snapshot + some history can take some time. So you end-up with corrective events and responses which are "we have noted your request, poll us to know the result when it's done". The worst is the ES system is already in use in your RDBMS if you use one. Most CQRS + ES demonstration are done with the simple happy path. Rarely something like user A changes the role of user B. User B was saving changes on a document they can't access anymore: what happen?
Now with software on your PC or with a constant link to the server things feel a lot more useful.
For your example, here are two approaches depending on your implementation and your tolerance for edge cases and eventual consistency:
- you can enter into a saga which would do a two phase commit, ensuring that the user is indeed authorized when the edit is made. All possible states are captured in your event model and so your event history will always show what has happened
- you can load the document aggregate when you process the command and query it at that time, throwing an exception if the user is not authorized and then you can notify the user or send another command. In this case, your event model does not have to include events that describe the violation of domain rules
The first option gives you the guarantee you are looking for, but comes at the cost of complexity. The second option will give you the correct results unless a user is deauthorized before the edit command completes. Some systems process all commands in serial and so it would not be a problem. Some systems partition their commands to distribute the work and so this edge case can occur.
But you can select what makes most sense for you.
So basically don't introduce race conditions that aren't inherently part of the domain. This seems like a modelling failure to me.
Either you are really good and get things right without much help, which isn't often the case, or you end up in two other directions.
You over-engineer with DDD/CQRS/ES.
You under-engineer without them.
Most developers I met preferred the last solution, probably because they end up with their "own" mess and not the mess of some process.
And I have to admit, the only people I met, that used these techniques, were some "digital transformation" consultants who made their living by selling this stuff to big corps, which isn't too convincing either.
CQRS/ES definitely seems like a correct foundations for a distributed system, I just don't think we have the right abstractions to properly model this without wanting to stab yourself in the eye.
Still, it should be noted that Fowler warns that CRQS adds significant complexity and advises great caution about using it. See https://martinfowler.com/bliki/CQRS.html
* I think it's in this talk https://www.youtube.com/watch?v=JHGkaShoyNs
I'm one of the authors of wolkenkit, so if you have any questions, feel free to ask here :-)
Naming things is hard but after watching UBER Inc destroy the pronunciation, spelling and etc of a German word, I'm always cautious about unusual names.
(Also because, as now I'm hung up on the name before giving your product a fair chance)
> That‘s why we have built wolkenkit, a semantic JavaScript backend that addresses three important aspects of software development, and that empowers you to build better software faster.
There's a bunch of things wrong with this claim, "a javascript backend" sounds like an alternative JS VM to me, a backend for javascript (I presume it's meant that this is server side JS not client side?) I also always get my back up when I read "build software faster", surely it ought to be about maintainability when it comes to the perks of DDD/CQRS.
What is your reasoning for AGPL - it's a no-go for
a) indie developers because they can't afford the enterprise stack which is presumably not AGPL
b) enterprises because immediately expect a huge price tag
And last but not least: Why is this not platform agnostic (e.g. only node)?
So, what's this about? about DDD, CQRS and event-sourcing, or about a particular Javascript framework? Those two aren't the same at all (even though I suspect there is a little overlap).
The last 1/3 shows how to apply these concepts using wolkenkit, which is a specific framework.
When we created the brochure it seemed important to us to first explain the concepts in a neutral way, because we want to help people gain a better understanding of these concepts, no matter whether they are going to use wolkenkit or not.
You might have an easier time getting people to read it when they don't get the impression "it's just another javascript framework".
It's just fundamental aspect of asynchronous programming, conveniently forgotten trying to sell solutions.
Edit: I would argue that you can't write a system which is open (several domains) and believe that you will have entirely independent events throughout the system and claim that your system will be "bugfree"/correct.
So this means for your problem to be correct you have to use the correct tools to solve that problem. If you think that everything (all problems) can be decomposed to independent events, which per definition can happen in any order, you can safely use ES. But if they have any dependency on each other you need ordering, and ordering in a distributed system is a hard problem. And add to this is that somewhere some guy from the business will come up with an actual requirement where you must synchronize and if you then have solution which inherently where this is not possible you are screwed.
This is true for both reading and writing in that system. Most problems are dependent otherwise it won't solve/do anything, there are usually a purpose with most systems.
While you can have systems where the events are entirely independent, structurally independent, such as JS event loop and red/blue methods (just locally independent), or it doesn't matter like plain application logs, this is rarely, if not impossible between domains if you want a correct system since you won't know if all events have arrived yet to tigger another event, if there's any dependency between them, which there will be between two domains.
DDD is good but not in this context, CQRS is trying to partially to solve above problems, but not needed and is just bloating. It does fit better with C# since the (weird) implementation of async/await and potentially with JS for the same reasons. ES do have their use cases where they fit very well but not with the practical impacts of DDD.
> user.login({ login: 'user', password: 'secret'});
> const isUserLoggedIn = user.isLoggedIn();
If you look at this purely from an API design point of view, this stinks: by forcing the command to return void, the code ends up twisting its design to return the result of the command through an artificial channel (the state of the 'user' object). In the end, we are still interested in "the result of the login attempt", and the easiest way to do this would be a function that returns a value:
> const isUserLoggedIn = user.login({ login: 'user', password: 'secret' });
If your schema is any more complicated than a plain stream of characters, and you pick CQRS for your app, then you are an overpaid, bored, architecture astronaut consultant looking to pad his resume. Nevermind the multi-million dollar dumpster fire of an app you spearheaded and ditched out on before completing (and that got cancelled, of course, a complete loss.)
If I ever saw CQRS on a resume, I'd unconditionally pass, you're not setting foot in the door.
This can be as simple or as complicated as someone wants to make it.
It could be as simple as having commands operate on a domain object, but the queries returning a fully populated view object from some custom sql.
I like queues as much as the next developer. Queues let you scale performance Mt Everest and deal with concurrency.
CQRS as an architecture is like queue abuse.
I mean you can use them, but all CQRS means is a separate code path for queries and commands.
In my system I have command objects, they operate on domain objects which contain some business logic.
I have query objects which just execute custom sql code which returns a populated view.
This is useful because views often display specific data from multiple domain objects.
This system allows you to bring back the entire view in a single query and nothing else, rather trying to retrieve multiple domain objects which are really designed for business rules rather than queries.
Often yes, people do starting adding queues, materialised views to CQRS which can make it complicated.
You don't say?? You don't say??
Sorry to be flip, I hope you can tell that I've been burned. Badly. Thanks for your detailed response. You obviously care and are not a stupid person nor do you come off sounding vain.
But, you're not selling me on CQRS at all. What you've described is just classifications and organization for types in a sane sounding UI design that happen to have similar roles as types in a CQRS system.
If you were selling me on CQRS, you'd start by talking about data flow. Something like, "it's a realtime, live multiuser system where all commands are queued, transformed to system events, and those events are committed to log (via another queue, we have to synchronize after all) that stretches back into time. We can replay those events, rewind time, events are merged ..." And it goes on and on into the stratosphere of architecture astronaut-hood...
My problem with your response is I honestly can't trust what you say here until I do a code review. Is your actual code pathological, or are you whitewashing the constant twizzling and niggling you have to do to keep your ship afloat? How bad, honestly, do you have to work to add a new feature? What's your velocity like? I want to see how much work and layers there are in your code to do data type transformation.
Data transformation is where El Diablo lives, because data transformation requires lots of people to agree and business terms to meet code - it's the unholy marriage of garbage disposal, forest fire, and whirlwind. CQRS seems to invite needless data transforms that other architectures do not require, and that's the pukey taste in my mouth when I have discussions about CQRS.
But that isn't CQRS. That's event sourcing, a complicated implementation of event sourcing.
CQRS can work on a standard SQL system, you can design the database using a standard normalised database.
All that has to be different is that commands and queries are separate.
A simple implementation;
A command can literally be
A query can be Inside the command you may retrieve a domain object from repository, and execute a method on it.Inside the query method you may just perform a raw sql statement which grabs data from multiple tables, populates a viewModel and returns it. Rather trying to retrieve multiple domain objects from multiple repositories and then shuffling that data into a result..
That's one way of doing simple cqrs. It can be simpler than this. But this code just made it obvious.
CQRS allows you do very complicated things, but it doesn't mean you have to.
My main point in replying to you was that you weren't selling me on whatever brand of CQRS you think you've got. Why? It seems like you're atomizing your business logic into a fine mist, why is that good? And you still haven't talked about all the myriad transforms you do: where's the payoff? Does YAGNI apply in your case?
Plus I can run the usecases from a console frontend for quick testing without much setup.
I don't really do any transforms other than mapping domain objects into the dB or database results into a query result.
Why do I need it? Because I practise DDD, and domain objects are not designed for queries. To populate a single view you have pull multiple domain objects from repos. This can be slow and adds complexity.
Query objects allow me to query the database using standard SQL or stored procedure and put the output into query result using only a single query. These queries can as complicated and optimised as they need to be.
Meaning I can avoid all DDD abstractions that I don't need for querying, but I do need for enforcing complicated business rules on the command side.
The act of starting an order does not look like an order. Asking for an order cancellation does not look like an order. Using GraphQL to pull an order summary with report-specialized drill down capabilities does not look like an order. These specialized domain objects should be modeled individually. By trying to encapsulate all these facets in a single object what you get is tight coupling, compromises, domain impedance, logical errors, and slow development.
In context: One of the root principles of DDD is that each domain represents its entities as is befitting that domain. Domain entities should be meaningful in their domain and _not_ corrupted by every niggling redigestion of the same facts through tens, or hundreds, of simultaneous conflicting views... Reporting concerns should be first-class citizens when solving reporting, for example.... View concerns and high-load systems often require denormalization, materialization, transformation, and replication that has very little to do with the user of origin while also _destroying_ the premises of traditional ORM driven RMDB interactions.
At scale, under load, while supporting multiple N-tiered applications and their accordant services, unified domain models break down. Aggregate boundaries map these epistemological distinctions.
CQRS on top of that is just good engineering. In the same way modern car-tunnel builders build two separate tunnels for traffic in each direction, CQRS isolates low-frequency user interaction from high-frequency reporting (or vice versa!), allowing independent scaling and direct delivery of features without impossible coordination activities between teams or finding silver-bullet technologies that may or may not exist.
That said: these are Enterprise solutions to Enterprise issues... Scale is the reason they exist. If you aren't looking at a horrible mess around a unified domain model then these solutions might not apply. If you aren't dealing with multiple simultaneous data delivery platforms (mixing SQL server and Oracle and Mongo DB and S3 and DocumentDB and DynamoDB all at once), these solutions may not apply. If you're not balancing multiple teams working on multiple products that have a constant need for harmonic data exchange, these solutions may not apply. If you're not juggling BigData, if you're not dealing with complex business logic, if scaling up an extra ten thousand users isn't something you deal with, if you're not supporting multiple client types, and so on: these solutions may not apply.
GraphQL and Falcor are interesting, and I like the sort of query-as-befits-the-view aspect to working with it. Taking a wait-and-see on those.
You can code it up in minutes, and most of the code is pure plumbing to connect to datasources. If that sounds complicated to buy independent control over data exchange then you're simply not exposed to the complex multi-layered and distributed applications that demand a structured approach to the issues CQRS handles. Unquestioned and implicit assumptions are great for small potatoes.
Domains should model domains and reflect the mental models of the people who work on, and with, the system. If that is experienced as complicating the only conclusion is that you're working with subject material neither complex nor interesting enough to warrant a considered architectural approach.
For someone who deals with computing (currying, monads, SOLID, etc), you should know that trying to meaningfully translate superficial domain complexity to generalizations of code quality or design quality of any given solution in all domains is laughable.
Your sales technique makes your project sound like torture.
CQRS boils down to a single principle: two one-way roads make for easier road maintenance and improvement that one two-way road.
Two individual channels are too complicated to code? Torturous? Inherently badly designed? Hard to add new features too? Unclean? ... Crazy talk, unsupported by reality. In fact: CQRS specifically addresses all these points you've brought up through its decoupling.
Those kinds of prejudiced assumptions do not hold water, cannot be logically supported by the underlying design nor in-production systems, and reveals only domain ignorance.
Successful engineering around distributed Enterprise systems tells a clear story: those maintenance and improvement concerns are predominant at scale.
Opinions may vary... Informed opinions on this matter do not.
Truly, one look at this archetypal wonder, (which certainly every programmer worth his salt has already studied its types and patterns at length), and all doubters shall melt away. If there ever were a more killer app demonstrative of the state of the art, it should be the muse of comp-sci journals or even popular culture!
A toy Microsoft CQRS spike on github from a couple years back does not count, neither does a book reference. I want a canonical example of something real: deployed code to users, thriving and successful. I'll be a happy to learn something new.
My suspicion however is you're all talk - a super-consultant who sold some CQRS work and has an axe to grind because you have co-workers reading this thread, including some of the client's devs. You arrogantly bragged to them and then waded out to sea because you thought me an easy mark to prove your claims to expertise on the subject.
I am prepared to gracefully concede to my betters.
I suppose I crave simplicity in code, with as little magic as possible, and with as few maddening descents into "not-invented-here" coding adventures when open source modules and frameworks exist that could do the job.
My use of the word "clean" refers to Uncle Bob's definition of clean code, not some scientology thing. He preaches the necessity of writing well-tested, high quality code via TDD. You mentioned SOLID earlier, which is another Uncle Bob-ism, and I figured you would get the clean reference. Not Uncle Bob's, but YAGNI's another acronym he likes to throw out there - YAGNI comes to mind whenever CQRS is mentioned. I bet YAGNI would be your worst nightmare if anyone around you had the courage to utter it. Are you a bully or do you just have no scope control on your projects from your business? Hey coworkers of bonesss, challenge everything this guy says with 'YAGNI', and see how he reacts!
Yeah, so anyways as I said in my other reply let's see your large scale code example that necessitates CQRS. Impress me with the high quality design and implementation, I'll do a code review.
> At scale, under load, while supporting multiple N-tiered applications and their accordant services, unified domain models break down. Aggregate boundaries map these epistemological distinctions.
I mean, I get it, you're talking about DDD. You've got context-bounded models. I've explored all this for over a decade. I apply the concepts daily (as mandated by our architecture, not my choice.) In the SOA days we had canonical models that are exposed on the service interfaces and domain-specific models that were representative of data coming from data sources or were internal application specifics and we always mediated between the models. It's nothing special. I sometimes think, if we all talked a little more we could make our terminology more consistent - but I get it, there's domains and different definitions and types will get mapped.
What you wrote is too pendantic, though. Epistemological! You're writing web apps, c'mon you are killing me!
Some bank transactions, for example, do not have that luxury because it might allow 2 possible answers to the same query if the eventuality has not been met.
Some trading transactions don't have that luxury either because the milliseconds that you waited for the consistency a stock might have changed its price dramatically.
It really depends on the Domain if the time to wait for the eventuality is acceptable or not.
Rarely do you have an up to the microsecond view of the markets (unless you are located appropriately and pay significant amounts of money). There is always a delay in the information's accuracy and relevancy.
A domain that can't handle any delay in consistency is atypical in my experience. Either way you can do DDD/CQRS without eventual consistency.
Yes you can do DDD/CQRS without ES but CQRS is not needed, it's just a technical construct not solving any verifiable problem. DDD on the other hand is more sane and to some extend, verifiable.
I live in the US, and if I got to the ATM and get cash on my account, I can go to my iPhone app and see the transaction and the balance updated.
Yes, there is a nightly process, but not everything is a nightly process.
"Rarely do you have an up to the microsecond view of the markets (unless you are located appropriately and pay significant amounts of money). There is always a delay in the information's accuracy and relevancy."
Do you know what a hedge fund is?
Eventual consistency is not all that scary, try putting on a business hat instead of a technical one. Its about risk management.
You may also have two independent items of information that have no consistency relationship or requirements.
That's why eventual consistency "works" so "well" with NoSQL: because there's an overlap between using NoSQL databases and not requiring actual consistency.
You can't get rid of eventual consistency, obviously, but you can set up your system so that your code expresses rules and features in an "immediately" consistent abstraction, and the system then translates those rules and features into the eventually consistent world, by applying tactics and techniques that you would have otherwise applied manually. It's a really natural approach for programmers: automate the translation from immediate to eventual, instead of doing it yourself every time.
Of course, most of these tactics come at a cost, either of performance (we won't acknowledge the creation of a gizmo until we've propagated that creation to every server capable of rendering the list of gizmos) or of availability (the registration of your username can't be acknowledged unless the persistent consensus reflects that you're the only one who registered it). But that's fine: most of these things aren't critical enough to make it worth dealing with eventual consistency manually, and when they do become critical, it's time to go down a level of abstraction and apply immediate-to-eventual translation tactics that rely on additional assumptions about what you're doing in order to improve performance or availability.
Well it's all about errors (and subsequently about control and correctness). Even if you have two transactions you are able to undo the other without anyone else (readers of the system) know that it was there because something else happened just before yours. A transaction usually happens in an atomic fashion, which makes it possible to back out on something which is no more correct so you have to fail it somehow to someone which does have the ability to recover, the one who initiated the call.
Without being too philosophical, most, if not everything depends on synchronous state (fex events themselves), computing just gives the illusion of that things happens immediately. Eventual consistency is cheating, and you can do it where the events are entirely independent. Otherwise you will have side effects when events are not ordered.
JavaScript have an "asynchronous" coding style, where (red/blue) methods are put on event-loop as "events". But the events needs to executed in order (synchronously by a thread) for the program to behave correctly so you give the methods attributes (red/blue) for them to execute in an linked order. So even the asynchronous behaviour in JS relies on synchronous features for it to be correct!
I believe that the core principles of DDD/CQRS/ES are flawed and are the cause of a lot of pain. If your team plows forward and ignores the pain, you will have a bad experience working there. If your team decides to fix the problems with the architecture (usually in a way that's rather specific to their situation), and still insists that they're doing DDD/CQRS/ES, then you will have a great experience. The same can be said of many approaches to SQL database design.
When I joined Lokad, most of the code had been written by a strong CQRS advocate, and there were significant pain points caused by this approach that would have been nonexistent with a plain old SQL architecture. The author himself did a retrospective that covers a few issues: https://abdullin.com/lokad-cqrs-retrospective/
Instead of dropping everything and moving back to a more conventional SQL architecture, we dropped everything and moved to a solution which, if I were to describe it as DDD/CQRS/ES, would have me branded as a heretic by the DDD/CQRS/ES community: it does fit the definition, but it does not follow the best practices. There is a single data model shared by both commands and queries. Commands can sometimes return data. Each (micro?)service has access to exactly one aggregate. There is no uncertainty on event ordering, and no eventual consistency at the aggregate level. Materialized views for each aggregate are kept in-memory at all times. All of these intentional weakenings of the CQRS architecture were driven by pragmatic needs, and made with knowledge of the consequences.
"There is a single data model shared by both commands and queries." - Nothing wrong with that, especially in simple cases.
"Commands can sometimes return data" - Well commands can succeed or fail. Greg young himself admits commands are really synchronous, and you need to communicate that.
"and no eventual consistency at the aggregate level." - This is the recommended approach. Between aggregates and it becomes loser.
"Materialized views for each aggregate are kept in-memory at all times" - Thats some advanced CQRS usage there.
But a messaging system for events.
If i'm using event sourcing use a dedicated event sourcing database with catchup subscriptions etc
Can you recommend any?
[1] https://geteventstore.com
Event sourcing is the definitive way to write high-performance front-office trading systems. Nothing else comes close to being able to quickly capture the rich domain of finance and satisfy stringent performance requirements.
But what that article describes is unlike any system I've seen in 15 years of fintech work. It's very clear that something has been lost in translation. I suspect the people are coming into this without having the solid foundation in event-driven architecture or high-performance messaging models.
As UK-AL points out everything you wrote is standard ES best practices. How is it even possible for a single aggregate to be "eventually consistent"? Aggregates by definition are atomically updated from one valid state to the next. Of course commands should return not just success/failure information but new information that is relevant to the sender (eg, OrderIDs for the Orders created). There's absolutely no reason to make the sender poll an event stream or "view."
"There is a single data model shared by both commands and queries." -- In the finance world this is impossible. Execution systems must be separated from reporting systems. And once you make this separation the whole CQRS-aspect naturally falls out. It makes sense, for example, to keep all orders in something like Cassandra or VoltDB for reporting though you'd never do that for execution. The thing is, CQRS is not an end to itself. I'm not sure why it has blown up in the .NET world. Again, something has been lost here. The CQRS tactic is required only when reporting requirements and execution requirements are fundamentally different/incompatible. An EDA makes CQRS-style separation possible but it does not require it. And here's the thing: if you have your end-points designed correctly it's trivial to separate out querying if it's needed. Just model your system using events and aggregates and it should work out.
I wasn't even going to comment on this thread. Most of the comments here are your typical HN nonsense -- a potent mixture of ignorance and anti-innovation FUD. But your link to the writeup was enlightening (and horrifying) if only to reveal how people can go very wrong with this "new" architectural style. (Again, event-sourcing, and EDA in general, has been de rigeur in finance since the 80s.)
Commands are by definition synchronous and return a success/failure.
One of my favourite read models to use is an in memory model providing I can keep all of the data in memory (and its reasonable for the query patterns).
when you say "materialized views" - do you really mean the current state of an Order for example ? and by "in memory", is it being persisted to a database ?
Then, we would have a materialized view that would be a single C# immutable object (canonically called `State`) that contains a dictionary of every order, and maybe a few indices (such as "identifiers of orders currently waiting to be shipped out" or "list of orders by customer"). Everything fits in memory, so every query is simply a bit of C# code that traverses the in-memory graph of objectd rooted at the `State`.
A command emits events, which are then applied by the system to create a new `State` from the previous one. Events emitted by other instances of the application are also fetched by the system every so often, and you can request a "force catch-up" to guarantee that the `State` you access takes into account all events appended so far (so that you're certain to see the results of your actions, even if the load balancer bounces you from one instance to another).
Here's what we use: https://github.com/Lokad/AzureEventStore
Having read the article: veeeeeery little of what is mentioned is related in any meaningful sense to CQRS...
Horse-before-the-cart framework design is always bad. Encapsulating untested theories in frameworks instead of lifting them from operational designs, always bad. Building frameworks as a learning exercise, bad. Abstracting away data decisions and then trying to provide opinionated answers to storage at the framework level, very bad. Conflating logical system architecture with supplier provided products (ie cloud services), is framework _cancer_.
That is simply not a reasonable starting point to provide value and scalable systems, and the end result was the same as all other similar attempts: an A for effort with a result that was at best 'close, but not quite...'
To be clear: I do not intend this as criticism leveled at the author or the company. The article is great. I very much agree with his analysis though:
> Lokad.CQRS was created with a very short-sighted design approach in mind, a reusable LEGO constructor...These days I'd try to limit the damage I inflict upon the developers and avoid writing any widely reusable frameworks
This is where all the issues lay... Premature framework development causing up-front decisions untethered to practicality based on conjecture and prognostication instead of experience in a challenging domain insufficiently understood by the framework architect.
From a high level perspective: neither CQRS, nor framework development, nor application development were sufficiently groked before being baked into a long-term production commitment. Those are organizational issues, not CQRS issues. In the same vein, coming from such an environment and basing judgment of DDD or CQRS on anything that came from that environment won't hold water.
The architectural problems you're talking about, the core principles you think are flawed, work stunningly well in other places that have not made the same underlying mistakes. CQRS advocates != CQRS experts != framework experts.
Edit: Had a second look and it's layouting software
Is this really the case? I was always under the impression that everything regarding financial transactions needed to be transactional.
So financial transactions are transactional, but there's no guarantee that every node has a fully up-to-date state at all times.
[1]: https://en.wikipedia.org/wiki/CAP_theorem
Which is very annoying because the article about CSS can't get it's UI right.
They call Svelte "The magical disappearing UI framework" and indeed, their fonts look like they are written in an invisble ink about to dry up.