Ask HN: Why do message queue-based architectures seem less popular now?
Now, I almost never see engineering blog posts or HN posts highlighting use of message queues. I see occasional content related to Kafka, but nothing like the hype that message queues used to have.
What changed? Possible theories I'm aware of:
* Redis tackled most of the use-case, plus caching, so it no longer made sense to pay the operational cost of running a separate message broker. Kafka picked up the really high-scale applications.
* Databases (broadly defined) got a lot better at handling high scale, so system designers moved more of the "transient" application state into the main data stores.
* We collectively realize that message queues-based architectures don't work as well as we hoped, so we build most things in other ways now.
* The technology just got mature enough that it's not exciting to write about, but it's still really widely used.
If people have experience designing or implementing greenfield systems based on message queues, I'd be curious to hear about it. I'd also be interested in understanding any war stories or pain points people have had from using message queues in production systems.
375 comments
[ 3.5 ms ] story [ 356 ms ] threadA while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue.
For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just wasn't as good as the modern stuff built around redis (bullmq).
For multi-step, DAG-style processes, I prefer to KISS and just do that all in a single, large job if I can, or break it into a small number of jobs.
If I REALLY needed a DAG thing, there are tools out there that are specifically built for that (Airflow). But I hear they're difficult to debug issues in, so would avoid at most costs.
I have run into scaling issues with redis, because their multi-node architectures are just ridiculously over-complicated, and so I stick with single-node. But sharding by hand is fine for me, and works well.
Monoliths imo are better for smaller projects with a few devs, but otherwise within a few years most of the time you'll regret building a monolith.
I also disagree with the duplicated code point. I don't understand why that would be a significant problem assuming you're using the same language and sharing packages between projects. This isn't a problem I've ever had while working on microservices anyway. I'd also debate whether they're anymore more complex than monoliths on average. My favourite thing about microservice architecture is how simple individual microservices are to understand and contribute to. The architecture and provisioning of microservices can be more complicated, but from the perspective of a developer working on a microservice it should be much simpler to work on compared to a monolith.
In a technical sense it is a complexity, but IME pales in comparison with the alternative of having to manage multiple services.
I actually really like this model, it's pretty flexible. You can still have specialized server instances dedicated for certain tasks, but you have one codebase. What's very sweet is that for local development, you just run a single application which (with all enabled profiles) can fulfill all the roles at the same time.
Seperate processes, yes.
One could say "In my experience, the earth is flat", but there's not much a conversation to be had there.
One could say, "In my experience, the earth is flat - I got in my car, went for a drive in one direction, and eventually hit a cliff at the ocean instead of finding myself back where I started". Now there's something to talk about.
(To be clear: this is the internet, and a limited communication medium: I'd assume OP could relate details about their experience, and it's totally reasonable that instead of taking the time to do that, they went outside and touched grass)
If anyone doubts that, this very browser used to read and write is built all the way up with dozens of libraries from compression, network, image encoding, decoding, video encoding decoding, encryption, graphics, sound and what not where each library is totally separate and sometimes was never intended to be used to build web browsers by the original authors.
Rest assured, most of the business (or web 2.0 systems, search, curate, recommend, surface etc kind of) systems are a lot more simpler then an A class browser.
Unlike in Microservices where each process is possibly a totally different language, runtime and framework spun up individually that too possibly in totally different ways.
2. microservices living in monorepos is common
Scale is just one. There is also fault tolerance, security, organizational separation (which can be, up to a point, also be realized with libraries as you suggest), bigger ecosystem to choose from, …
The graph of dependencies between components makes for explicit separation of concerns just like you would have a graph of dependencies between different network services.
I also find monoliths a lot easier to debug. You have the whole call stack, and you get rid of a lot of potential sources of latency problems. You don't have RPC calls that might sometimes get forgotten.
Given the choice, I'd choose monolith every time. Unless, of course, microservices are needed for various other reasons. (Scale, the ability to distribute, etc.)
From a technical POV they are good for horizontally scaling different workloads which have a different resource footprint
From my experience, when a company decides to go the microservice route, it's more for the sake of solving an organizational problem (e.g. making team responsibilities and oncall escalation more clear cut) than it is to solve a technical problem. Sometimes they will retroactively cite the technical benefit as the reason for using them, but it feels like more of an afterthought
But in all honesty: microservices are very good at solving this organizational problem. If microservice x breaks, ping line manager y who manages it. Very straightforward
I see people trying to apply micro services architectures to a web app with a single developer.
As in literally taking a working monolith written by one person and having that one person split it up into tiny services.
It’s madness.
If you cannot make a clean monolith, I have never seen any evidence that the same team can make good microservices. It is just the same crap, but distributed.
The last 2 years I see more and more seasoned devs who think the opposite: monoliths are better for most projects.
Yes, but also - more difficult to refactor, more difficult to debug (good luck tracking a business transaction over multiple async services), slower with network overhead, lack of ACID transactions... Microservices solve problems which few projects have, but adds a huge amount of complexity.
Whether this is good depends on the type of changes you need to make. Just as you mentioned maintaining modularity in a monolith can be difficult with entropy tending to push the code to spaghetti, there is an equivalent risk in microservices where developers duplicate code or hack around things locally versus making the effort to change the interfaces between microservices where it would make sense.
Ultimately microservices add structure that may be useful for large enough teams, but is still overhead that has to earn its keep.
This is both true and false in a way. Sure, the same business logic is distributed across microservices, but a method call in a monolith can only fail in a couple of ways, while network calls are much more finicky - handling it in every case is pure added complexity in case of a microservice architecture.
Also, don’t forget the observability part - a mainstream language will likely have a sane debugger, profiler, a single log stream, etc. I can easily find bugs, race conditions, slow code paths in a monolith. It’s much more difficult if you have to do it in a whole environment communicating with potentially multiple instances of a single, or multiple microservices.
Lastly, we have programming languages to help us write correct, maintainable code! A monolith != spaghetti code. We have language tools to enforce boundaries, we have static analysis, etc. A refactor will work correctly across the whole codebase. We have nothing of this sort for microservices. You might understand a given microservice better, but does anyone understand the whole graph of them? Sure, monoliths might become spaghettis, but microservices can become spaghettis that are tangled with other plates of spaghettis.
You can achieve exactly the same with simple individual libraries.
I generally agree with your stance and I would add that I find the whole simplistic „microservices suck“ talking point as nonsensical as viewing them as a panacea. They do solve a few specific (for most companies, mostly organizational/human factors because scale and redundancy don’t matter that much) problems that are harder to solve with monoliths.
I still think this point is a bit misleading, because yes, the components become simpler, but their interaction becomes more complex and that complexity is now less apparent. See:
>The architecture and provisioning of microservices can be more complicated, but from the perspective of a developer working on a microservice it should be much simpler to work on compared to a monolith.
I think that perspective often doesn’t match reality. Maybe most microservice domains I‘ve seen had poorly separated domains, but changes small enough to ignore the interactions with other services have been rare in my experience.
For example, you need some ETL to happen every day. Instead of having your pipeline logic inside an airflow task, you put your logic in a library, where you can test and establish boundaries for this behavior in isolation, and compose this logic portably into any system that can accept your library code. When you need to orchestrate, you just call this function inside an airflow task.
This has a few benefits. You now decouple, to a significant extent, your logic and state transfer from your orchestration. That means if you want to debug your DAG, you don’t need to do it in Airflow. You can take the same series of function calls and run them, for example, sequentially in a notebook and you would achieve the same effect. This also can reveal just how little logic you really need in orchestration.
There are some other tricks to making this work really well, such as reducing dependency injection to primatives only where possible, and focusing on decoupling logic from configuration. Some of this is pretty standard, but I’ve seen teams not have a strong philosophy on this and then struggle with maintaining clean orchestration interfaces.
I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG.
I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things like skipping a node based on some criteria (which, it occurs to me in typing this up, can benefit from your above advice, as that can just be a configured function call that returns skip/no-skip) - and, well... thoughts? (:
If nothing else, it makes it easy to understand what actually happened and when - just look at job logs.
Closed - Passed
Closed - Failed
Closed - Waived
When you hit that Waived state, it should include a note explaining why it was waived. This could be “parent transaction dropped below threshold amount, so we don’t need this control” or “Executive X signed off on it”.
I’m not sure about the auto-skip thing you propose, just from a UX perspective. I don’t want my task list cluttered up with unnecessary things. Still, I am struggling with precisely where to store the business logic about which tasks are needed when. I’m leaning towards implementing that in a reporting layer. Validation would happen in the background and raise warnings, rather than hard stopping people.
The theory there is that the people doing the work generally know what’s needed better than the system does. Thus the system just provides gentle reminders about the typical case, which users can make the choice to suppress.
I find embedding logic into DSLs usually quite painful and less portable than having a static job graph and all the logic firmly in my own code.
Someone suggested Temporal below and that’s a good suggestion too if you’re fine with a managed service.
However, if each node is more of a data check/wait (e.g. we’re on this step until you tell me you completed some task in the real world), then it would seem rather than your DAG orchestrating nodes of logic, the DAG itself is the logic. In this case, i think you have a few options, though Airflow itself is probably not something I would recommend for such a system.
In the case of the latter, there are a lot of specifics to consider in how it’s used. Is this a universal task list, where there is exactly one run of this DAG (e.g. tracking tasks at a company level), or would you have many independent runs of this (e.g. many users use it), are runs of it regularly scheduled (e.g. users run it daily, or as needed).
Without knowing a ton about your specifics, a pattern I might consider could be isolating your logic from your state, such that you have your logical DAG code, baked into a library of reusable components (a la the above), and then allowing those to accept configuration/state inputs that allow them to route logic appropriately. As a task is completed, update your database with the state as it relates to the world, not its place in the DAG. This will keep your state isolated from the logic of the DAG itself, which may or may not be desirable, depending on your objectives and design parameters.
Great description of good orchestration design. Airflow is fairly open ended in how you can construct dags, leading to some interesting results.
The way you use Airflow, what advantage does it have over crontab? Or to put it another way, once you remove the pipeline logic, what's left?
With cron you have to be more thoughtful about failover especially when convincing others to write failure safe cron in invoked code. With airflow you shouldn’t be running code locally so you can have a mini framework for failure handling.
Cron doesn’t natively provide singleton locking so if the system bogs down you can end up running N of the same jobs at the same time which slows things down further. Airflow isn’t immune to this by default but it’s easier to setup centralized libraries that everything uses so more junior people avoid this when writing quick one off jobs.
Backends are faster and more robust when they don't need to manage the "session" so much and instead just validate the final API request data and if they queue up anything it's those actions that require database transactions or to publish a message.
Sure the network has gotten faster, but with the few exceptions like Craigslist and this site, in general "the web" has gotten way way slower in real terms, at least as I see it. These days megabytes of javascript (or webassembly) seems to be required just to display pages with a kilobyte of two of actual text content...
(in other words, by "web" I mean the network, not the content)
If you have a problem that logically has different async services, sure, use Redis or something. Databases also were able to handle this problem, but weren't as sexy, and they explicitly handle the problem better now. Just another tool in the toolbelt.
NoSQL was another solution in search of problems, but databases can handle this use-case better now too.
https://en.wikipedia.org/wiki/Gartner_hype_cycle
When you enqueue something, you eventually need to dequeue and process it. A lambda just does that in a single call. It also removes the need to run or scale a worker.
I think Kafka continues to be popular because it is used as a temporary data store, and there is a large ecosystem around ingesting from streams.
I personally use queues a lot and am building an open source SQS alternative. I wonder if an open source lambda replacement would be useful too. https://github.com/poundifdef/SmoothMQ
I think this also speaks to a related phenomenon where there are simply more tools and technologies you can buy or run “off the shelf” now. Back in the 2010s everybody was trying to roll their own super complex distributed systems. Nowadays you have a ton of options to pay for more or less polished products to handle that mess for you. No need for engineering meetups and technical blogs about tools that kinda-sorta work if you really know what you’re doing - just pay snowflake or confluent and work on other problems.
If your perception is indeed correct it'd attribute it to your 3rd point. People usually write blogposts about new shiny stuff.
I personally use queues in my design all the time, particularly to transfer data between different systems with higher decoupling. The only pain I have ever experienced was when an upstream system backfilled 7 days of data, which clogged our queues with old requests. Running normally it would have taken over 100 hours to process all the data, while massively increasing the latency of fresh data. The solution was to manually purge the queue, and manually backfill the most recent missing data.
Even if you need to be careful around unbound queue sizes I still believe they are a great tool.
In the ruby world, delayed job was getting upended by sidekiq, but redis was still a relatively new tool in a lot of tool-belts, and organizations had to approach redis at that time with (appropriate) caution. Even Kafka by the mid 10s was still a bit scary to deploy and manage yourself, so it might have been the optimal solution but you potentially wanted to avoid it to save yourself headaches.
Today, there are so many robust solutions that you can choose from many options and not shoot yourself in the foot. You might end up with a slightly over complicated architecture or some quirky challenges, but it’s just far less complex to get it right.
That means fewer blog posts. Fewer people touting their strategy. Because, to be frank, it’s more of a “solved” problem with lots of pre existing art.
All that being said, I still personally find this stuff interesting. I love the stuff getting produced by Mike Perham. Kafka is a powerful platform that can sit next to redis. Tooling getting built on top of Postgres continues to impress and show how simple even high scale applications can be——
But, maybe not everyone cares quite the way we do.
- event log (stateful multi-consumers) have taken a portion of the message queue workflow. this is more likely than moving them to redis/database
message queuing works incredibly well for many problems. it’s as critical to most companies architectures as an application database
In large enterprises, there is usually some sort of global message bus on top of Kafka, AWS Kinesis or similar.
In smaller shops, the need for dedicated message bus is over engineering and can be avoided by using the db or something like redis. It is still a message queue, just without a dedicated platform.
These days all those people who prioritise career building technical feats over solving actual business problems in pragmatic ways - they're all hyping and blogging about AI, with similar results for the companies they (allegedly) are working for: https://www.theregister.com/2024/06/12/survey_ai_projects/
As far as the question, I was thinking that queues have probably just become a standard aspect of modern distributed systems; it's considered a pretty foundational cloud service for any provider (though we just run RabbitMQ ourselves and it has worked well for us).
There are a huge number of "commercial" things that are hitting several million dollars a month in revenue running Ruby On Rails or WordPress/PHP. You can scale a long long way with "boring technology".
Way too many people think that are "the unicorn" who's user base is going to grow so quickly that then need billion simultaneous user scale right now - instead of realising that before they get even close to that they'll be generating enough revenue to have an engineering team of hundreds who will have rewritten everything two or three times with more suitable architectures already.
If you need a billion users to turn a profit, then whether you admit it or not your business model is "burn VC money until they stop giving it to us or we pivot using Doctorow's Enshittification blog posts as a guide book". That though, is a vanishingly small percentage of all dev work. Most business models have a way too make real profits of thousands or perhaps tens or hundreds of thousands of transactions a month - and they should be rolling in profit to reinvest in future higher scale development well before they run out of fairly pedestrian "boring technology" platforms. Horizontally scalable Java/PHP/Ruby/Node with vertically scaling databases on you cloud provider of choice is a well known and battle tested way to generate real value and cashflow in probably 99% of all businesses.
Order confirmation emails and sms messages are triggered by the order submission, and again usually sent off to a 3rd party bulk email or SMS service. Twilio or Campaign Monitor or Mailchimp will have queues and retry mechanisms, but again the website devs are just firing off an API call to some 3rd party that's dealing with that.
So there are no doubt message queues being used in non-sexy 3rd party services, but those companies probably consider that kind of thing to be their "secret sauce" and don't blog about it.
In my case, I need to compile templates for the e-mails to be sent, which is somewhat slow. Even if I have an in memory cache for the compiled templates that can then be quickly filled in with actual data, I don't want to make the first user to request them after a restart/flush wait like 4-7 extra seconds upon clicking on the confirm order button (in addition to any other 3rd party calls, checking payment status etc.).
Ergo, I need to carry the actual e-mail preparation logic (regardless of whether I send it myself, or use a 3rd party service) out into a separate thread. The problem then is that I most likely don't want to create a new thread for every e-mail to be processed, so I need a queue for the requests and one or multiple worker threads. There is functionality for this in the stack I'm using so no big deal, except I can't pass live DB entities across threads, so I also need to serialize the data before passing it off to the queue and deserialize it inside of the queue (or just pass some IDs and do DB calls within the queue).
Essentially I've created a simple queue system in the app code, since processing that data definitely shouldn't make the user wait on it. I can see why some might also go the extra step and opt for something like RabbitMQ, since at the end of the day my queue system is likely to be way more barebones than something that's been around for years. But until a certain point, YAGNI.
And what language are you using? E.g. in NodeJS you can fire of requests without having ti have a new thread or having to wait for it.
Razor templates: https://learn.microsoft.com/en-us/aspnet/core/mvc/views/razo... with this outside of MVC https://github.com/adoconnection/razorenginecore
Though to be honest, if it wasn't template compilation, it might as well be slow DB queries, slow external service calls, or anything else that you wouldn't want to do in the lifecycle of the user's request, if it concerns something that should happen in the background, not even related to a particular stack.
Sometimes that promotion is rewarded by going a different company while being able to say "yes, I used X in my previous role."
Yes, that is cynical. People have been building architectures off MQ for a much longer time than microservices have been around. Lots of corporates have used JMS for a long time now.
I just added a RabbitMQ-based worker to replace some jobs that Temporal.io was bad at (previous devs threw everything at it, but it's not really suited to high throughput things like email). I'd bet that Temporal took a chunk of the new greenfield apps mindshare though.
From a technology perspective, message queuing has been commodified, I can pull an SQS off the shelf and get right to work. And so maybe the ubiquity of cloud based solutions that can be wired together has just removed the need for choice. If I need mqtt, there’s an app for that. Fanout? App for that. Needs to be up 25/7? …
Message brokers need client libraries for every language and serialization support. HTTP clients and JSON Serialization have first-class support already, so many software distributors ship those APIs and clients first. Everyone got used to working with it and started writing their own APIs with it too.
That's not to say this can't also be done with message brokers, but unless there's a good reason HTTP won't work well, a lot of this stuff already works intuitively out of the box with HTTP.
* The Log is the superior solution
Along with much more mentioned in this thread, I think a lot of companies realized that they indeed are not AWS/Google/Meta/Twitter scale, won't be in the next decade, and probably never will need to be to be successful or to support their product.
There is no hype because not much news there.
That doesn’t mean it is less used.
Thats good. The documentation for eg RabbitMQ is much better and very helpful. People use it as a workhorse just like they use Postgres/MySQL. There’s not much surprising behavior needed to architect around etc.
I love boring software.