Lol this ! WRITE down that rabbitmq-web-admin passwd. After the setup and first few weeks of checking the speed of your queues you will forget about it and try to login in 1 year later :)
We started using RabbitMQ for several projects last year, and it's been a joy.
Some of that joy is surely just moving from older, creakier solutions. But it hasn't let us down, and everyone is eager to use it for new features or refactoring legacy code.
One big thing I’ve appreciated about RabbitMQ is how well it separates publishing, message routing, and subscription concerns. Plus it’s never been the issue in any infrastructure I’ve encountered it.
RabbitMQ is great. One of the few pieces of software I've used that "just works".
The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.
+1. Discovered RabbitMQ/AMQP around 2010, since then tech went through a 2015-era wave of HTTP microservices that has come, and, largely gone or moved to MQ.
When you say "gone or moved to MQ" - if not moved to messaging services like RabbitMQ/NATS/etc, where else could things have gone? At least from my experience, HTTP microservices are still very common, especially when using things like AWS Lambdas.
I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc, or more and more I see lightweight systems going completely serverless and just using lambdas - which are HTTP microservices.
To be fair, I started working post-2015, so I've actually never come face-to-face with a long running HTTP microservice backend... what would something like that even look like? I'm thinking of systems I've worked on that use a messaging queue, but that only rely on HTTP requests - is that what it would be? So like, I'd make a request to a microservice behind an endpoint, which in turn would make requests to 3 more microservices behind other endpoints? If so, I'm certainly glad that idea isn't cool anymore because that seems greatly inefficient :)
How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that?
Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers?
My main use-case is to receive status/change notifications from a service running elsewhere from the API server servicing the UI, in order to avoid polling for new data.
We use it in a fairly big scale for our slack bot system. It was just set up once, as akyu said, and since then it just works. Whenever we had troubles, it was always anything other than RabbitMQ.
I've also looked into other solutions (ActiveMQ, Google PubSub, ...) and RabbitMQ is by far the most straight-forward and quick to set up. There are some edge cases that it doesn't cover as well, for example automatic retries, but there are some "RabbitMQ patterns" to make it work. For a simple message broker/queue system, it's great and the docs are also great.
Was it easy to setup in terms of reliability and failover?
Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising!
Is there anything I should keep in mind for running it in production? Any best practices or gotchas, based on your experience (eg don't run in docker, or make sure there's lots of RAM or things like that)? I guess its all in the production checklist. I need to read through it all again!
This came up in several other threads here: Don't use RabbitMQ's clustering. It's surprisingly brittle and hard to recover from.
The accepted wisdom that I've seen is to run a single broker with a completely independent hot spare. But of course switching over to your hot spare will violate most of the guarentees that Rabbit gives you around durability, ordering etc, so you have to be very careful how you use it.
I desperately want to like Rabbit (and have used it heavily in the past) but right now I wouldn't use it if I can get away with anything else, it just has no real HA story.
> Was it easy to setup in terms of reliability and failover?
To be honest, the current setup was set up by colleagues who have even less experience than me and it's still running flawlessly. Iirc, it's just two instances that are behind a load balancer and the consumer just consumes from both, but I'm not super certain on that.
I've tested the cluster functionality to see how to set it up and it worked fine for me, but I have no experience with that in production, but other people in this thread don't seem to be too happy with it, so ymmv.
> Is there anything I should keep in mind for running it in production?
Nothing special that you wouldn't do otherwise, when getting to know a new component/microservice. Just check out the get started section[1] on their page in the appropriate language and play around with a small setup. Get familiar with the libraries to connect and send/queue/fetch stuff and the topics. Make sure to use your brainpower before you set it up to handle all eventualities and is set up exactly how you want it to act (ACK/NACK, what happens if a sender/consumer dies, etc.), because then you set it up once and probably never touch it again.
One thing I'm not really sure on and what I haven't really answered myself yet: there might be some "logic" to your RabbitMQ instance, depending on which metadata you add to each message (e.g. retries). If you have such logic, it might be better to have a service around the RabbitMQ instance, otherwise this logic ends up in your code base of you actual solution and that might not be wanted and maybe harder to maintain. But I'm not so sure myself on that one.
Oh yeah, and check out some patterns for your needs. There are for example multiple ways to implement retries, for example with a queue for queues, etc. But if you API is REST based, everything should be straightforward.
Thanks for the detailed response (and to everyone else who responded too!), I appreciate it! I will prototype something and play around and see how it handles different situations when I get time. I've also just bought the book mentioned elsewhere, so hopefully I can get up to speed quickly. It does sound that my original impression about it being complex to run/maintain was perhaps overblown. That's good, because from a features point of view, RabbitMQ seemed like a good fit for the things I want an MQ solution for.
I've been running RabbitMQ for >8 years in production, once even in a fleet of 180 buses where every bus had an instance of rabbitmq running locally.
Never had a single issue in all those years.
But, I must admit that running a HA cluster is something that I've never tried, it sounds complicated and scary once you start digging through the docs.
All my deployments have been to bare metal Ubuntu and Debian machines with durable Qs and messages.
If you need to use transactions, they are really slow, couple of orders of magnitude slower compared to regular AMQP usage.
In my experience RMQ is solid enough that for many use-cases it's reasonable to run it without a standby (especially if you're on Kubernetes where you'll get a replacement instance created automatically if your active instance fails).
A common use-case is for async tasks (Celery) that can tolerate a few minutes of downtime. If you're running a fully evented architecture then this might not apply - though if you're not targeting 4-5 nines of reliability or an RTO of < 5mins, then you might not need a standby even if RMQ is a core part of your architecture. "Avoid single points of failure" is a good heuristic, but "consider the SLAs of your dependencies" is the more granular way of thinking about this, and a single RMQ instance has a very high uptime.
For context I had an RMQ docker container running for almost two years without any interruptions. If you're in a small team then HA might well be overkill.
Fun gotcha - if you're running RMQ in Kubernetes/Docker, make sure you give it an explicit memory limit, else it will try to allocate disk space equal to 40% of your host's memory. (See "memory limits" in https://hub.docker.com/_/rabbitmq). That's a good best-practice for any containerized environment regardless what workload you're running, but this one will cause errors if you're trying to use a small disk volume on a host with lots of memory.
I’m just using the Helm manifests, but when I set this up operators were not a thing. I’d probably look into the operator approach if I was starting from scratch now.
We use Google Pub-Sub and got the whole thing up and running very quickly with Spring integration. Message durability, automatic consumer load balancing, automatic retries, some easy broadcast patterns - all out of the box and literally a click of a button on the infra side.
My go-to solution for fault-tolerant message queues is nsq (https://nsq.io/). nsq works differently from most other message queues in that it's supposed to be run in a distributed fashion, i.e. one nsqd running wherever messages are produced. That way you have a lightweight and fast local message queue that you can push messages to and not worry about network connectivity. You can use nsqlookupd to find the distributed nsqd that hold the topic you want to subscribe to, or you can run an additional nsq-to-nsq process to push messages from one broker to the next. It's a really great and very mature and stable piece of software. I'd say the only downside to using nsq is that you have to invest a little more in monitoring and you have to make sure that network connectivity between your consumer and each nsqd that carries a certain topic is possible.
RabbitMQ, even a single-node RabbitMQ, has a hard time going down. You are more likely to have your server/container go down long before RabbitMQ node goes down. That being said, if you want to have a clustered solution with nodes being in different DCs, configure shoveling (https://www.rabbitmq.com/shovel.html) or for a simpler solution, use a private VPN to interconnect the RabbitMQ nodes. I would go for the latter.
Been using RabbitMQ for a lot of projects in production. It can handle quite a lot data and this thing never fails. Sometimes it can be running for an entire year and we force restart just because.
> I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction?
In general the defaults are pretty good I think. There is a one page production deployment guide: https://www.rabbitmq.com/production-checklist.html that I followed to replace our handbuilt cluster w/ a new automated deployment, plus a few other niceties like docker logs & rmq metrics to cloudwatch and then auto clustering via autoscaling groups lookup.
I thoroughness of the docs can perhaps seem daunting, but I see it as a badge of quality and especially if you are growing it's usage organically it should "just work".
I think this is what a lot of people who complain about Slack don't get. It's just a better message queue for your business. The fact that you can funnel all your business events, regardless of whether they originate from humans or bots, into one place and then each worker (again, either human or bot) can subscribe/filter/react to relevant events is super powerful. However, if you try to use it as a corporate SMS platform or email replacement, you will very quickly feel overwhelmed because both of those message queues are designed for much lower throughput.
Using it in multiple projects: The software itself is great and provides great value.
Only pitfall are the available libs. Especially with the .NET implementation we had quite a lot of trouble. Its not following current .NET patterns and has strange quirks. Does anyone know a good alternative to the "official" one?
It's been a few years since I used it, but I used EasyNetQ for a while when I was working with RabbitMQ and it was great. A quick peek at GitHub shows that it still seems to be actively maintained. Maybe it's what you're looking for: https://github.com/EasyNetQ/EasyNetQ
You might want to look into EasyNetQ[0]. I've not played with it much but it appears to be a cleaner, more modern abstraction over the existing .NET Client. I'm not sure whether it fixes all the 'quirks' in the client however (I've run into them too :))
MassTransit is great, the maintainer is very active on Discord, and since quarantine has been streaming every Thursday night (for my UTC-5 anyway). Documentation quality has increased greatly the last few months as well.
> Especially with the .NET implementation we had quite a lot of trouble. Its not following current .NET patterns and has strange quirks.
It would be great to get specific, actionable feedback with your experience, either via a message to the rabbitmq-users mailing list or via a GitHub. The .NET client is an old library but considerable effort into improvement went into version 6.0. The plan for 7.0 is to address old patterns that remain in the library. Feedback would help guide that effort.
I just released version 6.1.0-rc.1 and would appreciate testing if you have time. Thanks!
If the library were being designed from scratch today, pretty much every method on the model would be Async. After all, if it leads to any network I/O of any kind, that can block.
Working with the current public API, Trying to implement a publish wrapper that never blocks, and returns a task that either completes when the publisher confirm is received, or faults after some provided timeout, is a lot trickier than it might sound.
Recovery from network interruptions is complicated, and auto-recovery features are limited, and in some use cases actually dangerous. For example, if you are manually acknowledging messages to ensure end-to-end at-least-once delivery, then you cannot safely use the auto-recovery, since the delivery numbers would reset when the connection does, and you can accidentally aknowlodge the wrong message with delivery tag 5. (Acknowledge the new one, when you were trying to ack the old one).
In my implementation of that included my own recovery, I ended up needing to pass around the IModel itself with the delivery tags, so I can check if the channel I am about to acknowledge on is really the same one I received the message on. (There is no unique identifier of a channel instance, since even the channel number is likely to get re-used).
Wait, what are you talking about? RabbitMQ does have persistent messages, you just need to set the queue as "durable", and the messages persist even during failures.
Indeed I'm a bit confused. I remembered about having to find a workaround because I couldn't use retained messages. It's actually not working only for subscribers with wildcards : https://github.com/rabbitmq/rabbitmq-mqtt/issues/154
I'm more familiar with SQS than RabbitMQ, but have used both, and have chosen between queue and stream based solutions.
Kafka is a stream, and can be replayed (if you have it set up to store stuff). Rabbit is simply a queue, and when the messages are gone, they're gone.
This means that queues are a lot smaller, but can only serve one set of consumers at at time. If you want to have multiple things listening to messages, you have to use fan-out patterns that place messages on multiple queues. Queues can also suffer from less than atomic delivery, especially if the system is distributed. This means you have to jump through some hoops and add an atomic layer somewhere if you want to ensure you're not double processing anything.
Kafka can have infinite retention (if you got the storage/$), and you don't need to have multiple streams to service multiple consumers. Each consumer stores where they are in the stream, and can traverse as needed. You'll need to be careful to make sure that a single consumer is handling a single partition to promise that you'll only process a message once.
Managing streams can be a headache, but less so now if you have money to have Amazon or Confluent manage it for you. They offer pretty much unlimited scalability, and are the production grade solution for a ton of problems.
Queues are really simple to understand and build and still scale pretty dang well. Just make sure your message processing is idempotent and make sure you can handle if something is processed multiple times.
Rabbit saved my life. I had a project that involved getting the AMQP Proton library working on the Xbox. Rabbit was so easy to setup and use, it gave me a reliable way to test my work. Getting into AMQP at the time was confusing and poorly documented. Rabbit did imdeed "just work".
I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies.
Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the road.
I've seen quite a lot messages going through RabbitMQ. I wouldn't worry too much about scaling, because the possibilities depend very much on the architecture. With some tuning RabbitMQ can take you a long way. I would give clustering a go and see where the limits are before exploring more complicated architectures like federation.
With clustering, you can have more nodes and you can shard (distribute) your queues over the cluster. You don't need to mirror every queue on every node. But you are right, mirroring alone will add more load.
Rabbit MQ is a traditional message broker; you use it when you have lots of messages you don't particularly want/need to be stored persistently, and where you want/need to take advantage of the routing feature--that you put keyed messages into some topic/exchange and then subscribe to only part of the messages any given application is interested in.
Kafka creates the abstraction of a persistently stored, offset-indexed log of events. You read all events in a topic. Kafka can be used to distribute messages in the way AMQP is used, but is more likely to be the centerpiece of an architecture for your entire system where system state is pushed forward/transformed by deterministically processing the event logs.
If your main concern is scalability: Each queue in rabbit gets its own thread. So if you can spread your workload across multiple different queues you can scale without too many problems.
Then the odds of you hitting the scale where RabbitMQ v. Kafka is relevant are a million to one. There is a lot of overhead with Kafka compared to RabbitMQ.
Unless you already have Kafka infrastructure, setting up Kafka for a brand new project is crazy unless your only goal is learning how to set up Kafka.
I've been running RabbitMQ on pretty small VMs for a long time. RabbitMQ doesn't need a lot of resources per message, even with very small VMs (512MB RAM, single CPU) I've seen it handle peaks of many thousands of messages a second without running into problems. Give it a bit of beefy hardware and it'll probably handle whatever load you were thinking, unless you're saturating 10gig links with messages or something.
RabbitMQ and Kafka are very different struggles when thinking of scaling and performance. Kafka is almost a database itself of messages which have routed through the system. In many configurations clients can come back and demand to replay the message stream from almost any point in time. This means you need to handle _a lot_ of disk and memory access. With RabbitMQ, messages are traditionally very ephemeral. Once a message has been ack'd, its gone. Poof. Not in memory. Not on disk. Nobody is going to come back asking for that message. This leads to a lot more efficiency in handling things per message, but at the cost of not being able to remember the messages that went through the system a few milliseconds ago.
CPU usage highly depends on the number of connected clients, not that much on message throughput. You can experiment with the excellent rabbitmq-perf-test tool to get some ballpark numbers.
I have a system that only pushes 5k messages per second but it needs 32 cores.
Yeah that sounds about right. Of course if you had 200 connections and 50 queues you'd more likely be seeing 100000 msg/s. The number of connections and queues has a big effect on total throughput.
Rabbit's federation is a good way to bridge point-to-point connections between geographically distributed systems. I'm not sure that's a great scaling pattern for throughput though.
The clustering might look tempting but it hasn't been resilient for me in the face of janky networks. Split brains and data loss can result.
In the past I've scaled my rabbits for throughput by implementing my own routing/sharding layer.
If you're tempted to use the message persistence and you care about retaining messages, kafka is a bigger but much more capable hammer.
Adding to what the sibling comment say, be careful about buying into RabbitMQ's clustering; having run it for years, I found it to be extremely brittle.
We often lost entire queues because a small network blip caused RabbitMQ to think there was a network partition, and when the other nodes became visible, RabbitMQ has no reliable way to restore its state to what it was. It has a bunch of hacks to mitigate this, but they don't solve the core problem; the only way to run mirrored queues ("classic mirrored queues", as they're not called) reliably is to disable automatic recovery, and then you have to manually repair RabbitMQ every time this happens. If you care about integrity, you can use the new quorum queues instead, which use a Raft-based consensus system, but they lack a lot of the features of the "classic" queues. No message priorities, for example.
I've never used federation or Shovel, which are different features with other pros/cons.
If you're willing to lose the occasional message under very high load, NATS [3] is absolutely fantastic, and extremely fast and easy to cluster. Alternatively, NATS Streaming [4] and Liftbridge [5] are two message brokers built on top of NATS that implement reliable delivery. I've not used them, but heard good things.
nowadays? it's actually quite simple to setup and works pretty well (source: i know two different companies that setup clustering recently and both had good experiences with no downtime).
> lost entire queues because a small network blip caused RabbitMQ to think there was a network partition, and when the other nodes became visible, RabbitMQ has no reliable way to restore its state to what it was
I can offer a similar anecdote: we started seeing rabbitmq reporting alleged cluster partitions in production after enabling TLS between rabbitmq nodes, where manual recovery was needed each time.
After a bit of investigation we noticed that cluster partition seemed to correlate with sending an unusually large message (think something dumb like 30 megs) through rabbitmq when TLS between rabbitmq nodes was enabled. What I believe was happening was Rabbitmq was so busy encrypting/decrypting large message that it delayed sending or receiving heartbeat & then the cluster falsely assumed there has been a network partition.
Mitigated that issue by rewriting system to not send 30 meg messages- there was only one message producer that sent messages anywhere near that large, and after a bit of thought realised it was not necessary to send any message at all in that case (sending large message was to hack around some other old system performance problem that had gotten fixed properly a year back, but the hack that generated a huge message was still in place)
Erlang/OTP-22 (released last year) introduced TLS distribution optimizations and message fragmentation which sound very related to the problem you saw:
The fragmentation in particular addresses the problem where a large message would block all other messages, including heartbeats, and cause nodes to look “down” when they’re not.
fantastic. thank you for sharing that -- my anecdote about this problem is slightly dated -- it would have been late 2017 early 2018 we were seeing the issue, which indeed predates OTP 22 release.
As someone who has ran a number of messaging systems in production, this is what my current take is in general:
If you are moving to a more "event-sourced" architecture, usually two main concerns (beyond basic operational stuff of uptime, scale, etc) are routing and long-term retention.
RabbitMQ has the routing but not the retention. Kafka can have the retention and the routing, but it can be complex/expensive. Apache Pulsar really shines here as the API is pub/sub but it is underpinned by a log structure that gives you long-term retention (that doesn't need to be manually re-balanced) but it's flexibility does come with some operation complexity when compared to RabbitMQ.
If your needs is pretty much just moving large amounts of data, Kafka is definitely the most mature and has a big ecosystem, but long term-retention is difficult and there are some sharp edges around consumer groups.
If you really really don't need long-term retention and need complex topologies, RabbitMQ is your best bet and is fairly reasonable to operate even up to fairly high message rates (~10k msgs/sec shouldn't be too hard to achieve)
There are a TON more options these days though, older more java solutions like activeMQ and rocketMQ or more "minimal" implementations like NATs, not to mention the hosted services on cloud providers.
Personally, I am a big fan of Apache Pulsar for it's flexibility and some nice design choices, but I don't think there is any silver bullet in this space.
I'm guessing that the pain points surrounded having to set up a Zookeeper cluster in conjunction with Pulsar. I think Pulsar has the best model of the various queuing systems at the moment for the routing flexibility of RabbitMQ, the high-throughput of Kafka (topic/partitions), as well as the ability to seamlessly integrate with cold storage (S3/GCS) and to recall messages from cold storage without extra code (unlike Kafka), I just wish that ZK wasn't an additional dependency.
If you’re trying to “rationalize” a decision, that’s already a red flag. Also, Kafka and RabbitMQ are intended for different use cases. One is (the log component of) a streaming data processing system, the other is a message queue. Figure out which kind of system you need before deciding on a particular system. BTW, if you need to really scale, Apache Pulsar is designed to handle both scenarios.
I've used both. I was introduced to Rabbit at one job and at another, was "fed" Kafka during a selection process. At that time, I was definitely not opposed to Kafka because, hey new resume item. I ended up yearning for Rabbit for three reasons.
1) Much easier to implement and maintain for small to medium architectures. However, war stories I've heard is that it starts to become a hassle for large clustering architectures.
2) Because it's a traditional message broker, the input and output ends, which I was responsible for, were much simpler to write because I didn't have to worry about replays when it came back online. Rabbit knows which client it has already routed to and where messages went. Kafka is not that sophisticated in that regard. Kafka has been described as "dumb broker/smart clients" while Rabbit is "smart broker, dumb clients."
3) The scaling. Rabbit is very scalable. Once you get to the Uber/Paypal level (like, a couple of million writes per second), then Kafka becomes the obvious choice. Rabbit handles thousands or writes per second just fine. However, at that second company and like many others, they thought they'd have to suck up all the data, so of course, Kafka was the more scalable tool long term. Spoiler: We were never, ever close to PayPal-level transactions. If the size of the sun represents paypal/Uber transactions, we were basically Manhattan.
Kafka is one of those things where if you're new to it, especially if you're coming from Rabbit or similar, you might tend to assume the happy path - exactly once delivery. This is a bad mistake (whether that's possible and to what definition is not a debate I'd like to dive into now). What you should expect from Kafka is at least once delivery.
There will be times when you lose offsets or when you actually want to replay every message, so take an hour and figure out what that means to your app. It's usually only a few lines of code in your consumer that compares source timestamps, but it's by far the most beneficial thing you can do when working with Kafka in my experience.
It's also relatively easy to hit "tens of thousands" messages/second, especially in replay or bootstrapping scenarios, and that's when Kafka becomes useful to the non-FAANG companies.
Look into Pulsar, it can function as a message queue or pub/sub like Kafka.
By default it only retains non-acked messages, multiple subscription modes, can use non-persistent messaging, dead letter queue, scheduled delivery, can use Pulsar Functions to implement custom routing etc.
Scales like Kafka (probably better) and has cluster replication built in.
Same with MassTransit[0] and .NET. We have several distributed .NET Core services running in our data center, services running on employee PCs, etc all communicating via RMQ with MassTransit and it's great. The primary maintainer is very active (streams every Thursday evening) and the documentation has gone from "pretty bad" to really good in the last few months.
MassTransit is awesome! I love what Chris Patterson (the author) did. It essentially allows you to swap out RabbitMQ for SQS or Azure Service Bus or a few others. Pretty cool stuff if you're in .NET land.
We use ZeroMQ a bit. It's been pretty much flawless as far as I can see but I get the impression that it's becoming obsolete. Is RabbitMQ a viable replacement?
You might find it interesting to note, that Peter Hintjens, was one of the core authors of the AMQP 0-9-1 Specification [1], that RabbitMQ is implementing.
ZeroMQ was born out of a frustration with complex routing patterns and the need for a broker-less architecture for maximal performance message delivery.
I'll add that the AMQP 1.0 spec (supported in Rabbit using a plugin) is a peer-to-peer protocol that supports both the traditional broker use case, 'direct' p2p messaging and opens some interesting uses of message routers like Apache Qpid Dispatch Router.
He and Martin Sustrik both created ZeroMQ. Then after that, they saw some of the limits of ZMQ and created nanomsg. It's excited to see what cool stuff they were working on. It's a little hard to see ZeroMQ become abandonware from them. That said, the community is solid and supportive around ZeroMQ which actually I would say is the best part. In other words, you can tell if a project has staying power when the original creator no longer has to be there to maintain it.
The "MQ" in "ZeroMQ" is misleading, so this is an apples-to-oranges comparison. ZeroMQ is a socket abstraction that allows you to build apps that send messages to each other. RabbitMQ is a reliable message queue broker; a central server that stores messages and that clients connect to in order to push/pop them.
I think they’re slightly different solutions — ZeroMQ works without a broker, RabbitMQ requires a server process.
If you use the brokerless model, there was a bit of drama over ZeroMQ — the original technical developer (Martin Sustrik) left and created a successor, nanomsg, with what he learned. At some point, Martin lost interest, and Garrett D’Amore took over maintenance and did a rewrite called nng. Both the old nanomsg and nng are maintained, with nng being somewhat actively developed, but also fairly “complete”, so there’s not a lot of excitement like you see with some projects. ;) nanomsg and nng are essentially wire-compatible, so you can mix and match depending on bindings availability for your language.
Yes, my handwavy reading of the situation was that he left due to issues with zeromq that he couldn't/wasn't allowed 'fix'. Then Peter Hintjens unfortunately died a few years back. I haven't heard about nng, so thanks for that, I'll check it out.
ZeroMQ certainly isn't perfect, for example there's no way to tell if a message was successfully written to a PUB socket, or if it was dropped (just one minor issue)
If the ZeroMQ community seems quieter lately it's because things work well and there's not much left to do within the project's intentionally limited scope. libzmq is certainly maintained.
Our company has been using ZeroMQ for over 8 years. We'll be putting out another ZeroMQ-based open source project soon too.
For now, I'll just address the one point -- obsolete? NOT!
We've been working with ZeroMQ a lot over the past couple of years, and have gotten to know some of the maintainers -- we've been very favorably impressed by their ability and dedication.
Pieter Hintjens was the "voice" of ZeroMQ, and with his passing things have gotten a bit quieter, but no less active. (Just take a look at the commit log: https://github.com/zeromq/libzmq/commits/master).
We've used RabbitMQ since 2010 in KAZOO. I would argue, save one or two instances in the intervening 10 years, that RabbitMQ is the most stable piece of the infrastructure. I think it might be the only open-source project we build on that we haven't committed upstream to because we haven't encountered any issues in our usage.
I used RabbitMQ together with python and celery quite extensively and it scales really well. One thing we had trouble with though was to find a nice mechanism to scheduled tasks. Eg. “Run this task 12 hours before departure”. Maybe AMQP is the wrong place to solve that problem.
I've been using something like this for exponential backoffs, but I think it'd work for this case as well.
Let's say you've got one exchange and one main queue for processing: jobs.exchange and jobs.queue respectively.
If you need to schedule something for later, you'd assert a new queue with a TTL for the target amount of time (scheduled-jobs-<time>.queue). Also set an expiry of some amount of time, so it'd get cleaned up if nothing had been scheduled for that particular time in a while. Finally, have its dead-letter-exchange set to jobs.exchange.
This could lead to a bunch of temporary queues, but the expiration should clean them up when they haven't been used for a bit.
You're usually stuck polling for stuff like that. I'm a big fan of using things like Advanced Python Scheduler for those sorts of tasks: https://apscheduler.readthedocs.io/en/stable/
Yeh I would argue don't use a message queue for this, they're really best processing many messages quickly, there are plenty of scheduling libraries that have various persistence layers to handle this depending on your ecosystem.
Would anyone be able to explain the benefits of RabbitMQ over NATS? As far as I've seen, it's really just that RabbitMQ is more feature-rich, which I personally feel like isn't that crucial, as frankly many systems are not going to take advantage of those more complex functionalities anyway.
Durability. If you need to push messages that don't get lost, RabbitMQ is a pretty solid choice. In years past the clustering situation wasn't great and there was some potential for message lost and that seems to be resolved now with quorum queues, but the biggest different between NATS and RMQ is the durability guarantees and the at-least-once delivery guarantees that RMQ has. NATS is more like ZeroMQ in that it expects the subscribers to be online. There has been some work by others using that NATS protocol to create a Kafka-like system (written in Go, I believe) called LiftBridge. So if you like NATS and it's working for you and you want durability, take a look at LiftBridge.
That's right, but I think at least since both are listed on their website as different ways to run it that it should at least be considered a native feature at this point.
I've been meaning to give RabbitMQ a try in the last few years, but our good old beanstalkd is serving us well. It has all the features we need, and it just works.
I read but I still can't understand. I would like to know a very simple example of something that can't be solved with a CRUD, and can be solved with RabbitQM
One common way queues can be used to give an async like feel to your applications and flatten out spikes of activity without having to add hardware.
So, for example, you would have a CRUD that takes requests, and when there is background work to be done, places a message on the queue, and immediately returns to the user. This frees up the server for more requests. Meanwhile in the background, a worker process chugs through the queue and does its work. During long spikes it will take longer to get through the queue, but your end users will not have disruption of service.
Really really bursty loads. You have a customer upload a data file and you have to process it. If you crud it, you have a worker chopping it apart and making sync API calls. If something fails in the middle, it has to retry, but what happens if your container/database goes down when you're halfway through? Now you have to reprocess that file again, etc.
You move this to a queue, and have a worker chop that data file up into individual records, those records go onto a queue, and you can process them however you want, no worries about something crashing and not being able to be retried. If the database goes down, everything just pauses until it can go again. You can limit the queue throughput to whatever you want to avoid having to scale your API/Database.
Can you handle stuff via all CRUD sync APIs? Sure, just like you could handle running a restaurant where you have one person who takes the order and cooks it and delivers it to a table. However, it's more efficient to have a waiter (API) take requests and give them to a cook (queue based async worker) to handle stuff that's not as time sensitive. This saves you a lot of money in certain situations.
I'm very well versed on RabbitMQ. We use it internally in a .NET codebase.
Anyone considering RabbitMQ needs to read up on "network partitions", how to build your cluster to avoid them (odd number of nodes and pause_minority), your recovery strategy for when a network partition occurs (it will occur), your personal/organizational tolerance for message loss and a plan for how you will upgrade your cluster at some later date (ensure you architect your application to handle whatever type of upgrade strategy you will pursue).
There are definitely ways to operate to minimize these failures but you SHOULD KNOW ABOUT THEM before your add this service to your environments.
Can you talk a little bit about how you've managed your RabbitMQ infrastructure? Also if you've done any comparison to Azure Message Queues and what were the pros and cons against Rabbit?
I'm looking to pitch adding a message queue to our infrastructure (at a .Net shop on Azure), and I'm sure there will be some questions about the comparisons between the two. Unfortunately, that's been tough to really track down.
>Can you talk a little bit about how you've managed your RabbitMQ infrastructure?
From a ten thousand foot view, two or three node clusters running in non-prod environments on virtual machines running Windows. In Prod, three node clusters on Windows virtual machines.
All work to install and configure RabbitMQ is done manually. Sadly enough.
I'm on the application/architecture side of this equation but I know enough about our infrastructure to perhaps answer follow-ups or more specific questions.
Our application is single tenant (so each customer is deployed in their own isolated area) so we use virtual hosts to isolate each customer within the cluster.
>Also if you've done any comparison to Azure Message Queues and what were the pros and cons against Rabbit?
Definitely looked in to the Azure native queueing options but it's been awhile. Azure Message Queues is an AMQP compliant messaging system that seems fairly robust. To be transparent, I have no production experience with this product. If your company/department is in to managing virtual machines then they might want/prefer to go with RabbitMQ. However, if they're in to PaaS systems then I'd probably roll with Azure Message Queues and never look back.
Thanks for the response. From the other responses in this thread, it seems like the admin of the nodes/cluster is not overly onerous. Would you agree with that statement? Also, being a .Net shop, the Windows VMs make sense, but is there any tradeoffs to running Rabbit on Windows, as opposed to Linux?
I think part of the sell is how we would manage the admin component of a Message Queue, which tilts things towards Azure Message Queues as it's PaaS. We're mostly IaaS at the moment, and starting to see some of the admin overhead that comes with managing that infrastructure ourselves. We're not ready to jump onto a PaaS solution for the things we've grown accustomed to managing, but for something brand new, I think my company would be open to it.
Architecturally, we'd lean on it initially for background job processing, which is currently at a scale where our homegrown, db-backed solution is starting to show it's weaknesses. Once it's in place though, I think it could leveraged as a key component to decouple subsections our application and give us more flexibility with scaling and deployment.
RabbitMQ runs perfectly fine on Windows too. As others mentioned in the comments, RabbitMQ supports a great variety of use cases. If you want to reach out for help, you can find my contact in the article.
Download and set aside a copy of the erlang and rabbitmq installers if you're running on windows... I've had issues on many occassions with the erlang installer being unavailable or very slow to download.
>From the other responses in this thread, it seems like the admin of the nodes/cluster is not overly onerous. Would you agree with that statement?
Yes.
>is there any tradeoffs to running Rabbit on Windows, as opposed to Linux?
Should be fine to run on Linux assuming you (or you have) people are who are comfortable admin'ing Linux servers. I think that a Windows admin would get frustrated to setup/configure RabbitMQ on a Linux server. There's also a container advantage as RabbitMQ is published to Docker only with officially maintained Linux images.
>We're not ready to jump onto a PaaS solution for the things we've grown accustomed to managing, but for something brand new, I think my company would be open to it.
I'd push you to figure out why Azure Messages Queues would not work for you. If there's no compelling "no" argument then you'll thank yourself later.
>Architecturally, we'd lean on it initially for background job processing, which is currently at a scale where our homegrown, db-backed solution is starting to show it's weaknesses.
We pursued RabbitMQ for very similiar reasons (queueing mechanisms via SQL Server tables and stored procedures). Keep in mind that you still need something to submit the job (initiate the background task). RabbitMQ is not going to automagically schedule anything for you. We have a couple applications that use the tool Hangfire for job scheduling and in one case, the Hangfire job simply sends a message to RabbitMQ.
I'll add one comment, if you aren't doing a really large number of queued items (under 50k messages every few minutes), Azure Storage Queues are pretty nice and easiest to use imo.
This. We used Rabbit in our platform for a few years and it was an absolute disaster. Network partitions mainly. In retrospect I'm sure we were doing it wrong but that really wasn't obvious at the time.
If you're using RabbitMQ on .Net, I highly recommend using NserviceBus. It's made working with queues so easy. It handles maintaining a connection and retrying/acknowledging messages for you.
I worked on NServiceBus years ago (not just used, but actually was an active developer on the project). It's an excellent piece of software and Udi Dahan really knows what he's talking about.
Hindsight is the best site. That's definitely what I would do if I was starting a new project using RabbitMQ. Although I'll defend myself on this front; I inherited our RabbitMQ project from the developer who left the company 7/8 of the way through the implementation. I had the "make it work" directive and not the decision making luxury he had from the beginning.
While an official list of customers can't be published, you can get some ideas from the speakers at the last two RabbitMQ summits - https://rabbitmqsummit.com/
My general problem is that it's really hard to figure out which architecture is right for which system.
There's a different architecture for:
* one queue with billions of messages
* a millions of queues with small numbers of messages per queue
* many queues with many messages per queue
There are also different topologies:
* Anyone can send a message to anyone (O(n^2) queues)
* One publisher with millions of subscribers
* One subscribed with millions of publishers
* Complex processing networks, where messages get routed in complex ways between processing nodes.
There are differences in timing:
* More-or-less instant push notifications
* Jobs which run within e.g. 5 minutes with polling
* Jobs which run in hours/days, with a cron-style architecture
And in reliability:
* Messages get delivered 100% of the time, and archived once delivered
* Messages get delivered 99.999% of the time, but might be dropped on a system outage
* ... all the way down to ephemeral pub-subs
... and so on.
I'd give my VP's right eye to get a nice chart of what supports what. For the most part, I've found build to be cheaper than buy due to lack of benchmarks and documentation for my use cases. Otherwise, you build. You benchmark. You optimize. And things melt down.
My use case right now requires a large number of queues (eventually millions). I'd like to have an archival record of messages. Peak volume is moderate (several messages per second per queue), but usage patterns are sporadic (most queues are idle most of the time). Routing is slightly complex but not supper-complex (typically, about 30 sources per sink, at most 200; most sources only go to one sink, but might go to 2-3). Messages are relatively small (typically, around 1k), but isolated messages might be much bigger (still <1MB, but not small).
My experience has been that when I throw something like that into pick-your-queue/pub-sub, things melt down at some point, and building representative benchmarks is a ton of work.
All software breaks at some point. If you're dealing with this scale of load, it's mandatory to perform synthetic load testing to validate, otherwise you're just guessing what the breakage threshold will be.
Checkout https://github.com/yevhen/Streamstone
Had similar needs, it was a good place to start. It’s just the persistence part of it though, did the messaging part using actors.
Fantastic points. We needed millions of queues with millions of items with fair queueing and scheduled release of some items and immediate release of others. 10s of thousands of messages per second. We had to build our own.
I love RabbitMQ but deploying/managing a cluster can be tricky. We had problems with network partitioning and since we didn't really need a cluster for performance reasons - only availability - we switched to a single node.
I feel like RabbitMQ is sort of the "swiss army knife" of message queues, and I mean that in the nicest way possible.
People will compare it to Kafka, claiming that its pubsub is faster than Rabbit's, but that's sort of missing the point: Rabbit thrives because it's easy to set up, will work well for 99% of cases, and handles nearly every kind of distributed problem you're likely to come across.
I recently did a project with Rabbit on my home server, and while the project had some issues, the issues were never Rabbit.
Rabbit doesn't have Kafka's ability to massively distribute and scale (it does have a distributed story but from what I hear few explore it). But Rabbit also supports more complex use cases than Kafka because its messaging protocol (AMQP) is more intelligent. Unless you're a "web-scale"/s company, Rabbit's scale even on one node is likely enough.
I've been using Rabbit in production for RPC and pub/sub for the past 5 years (single instance running on a non-dedicated VM, medium traffic) and its been pretty easy to setup and has been pretty reliable in practice.
I've always been concerned about losing messages, and I did have to learn to turn on persistence and durability for messages to survive server interruptions, but it was easy enough. Message acknowledgements are also a nice feature, and Rabbit is able to achieve at-least-once messaging semantics.
I've got a connection/channel question for those who have built solutions with rabbitmq-- how did you decide as to how many connections and channels-per-connection to use? Does connection pooling even make sense for RabbitMQ? My impression is that channel pooling may make more sense. Thoughts?
An application usually has one connection, and many channels. Our pattern is to dedicate one channel for all publishing and then N channels mapped to consumer threads.
You don't have to pool connections as channels are multiplexed by them.
Things to watch out for:
- opening too many channels - these map to Erlang processes and can overwhelm your server if you go over ulimits
- sharing consumer channels between threads - you might see weird behavior (e.g. acking wrong messages etc)
We've built own library/framework for creating resilient consumers, and it enforces mapping 1:1 channels and consumer threads, as well as automatic reconnections and channel clean ups.
+1 for everything that's been said. Another thing to consider is message throughput, if that's a concern. In the case of multiple channels per single connection, note that a connection is a single TCP connection such that multiple channels contend for the TCP stream. At the same time, connections aren't completely free either.
The general takeaway from this should be: if you've got a particular stream of messages (either a producer or a consumer) that pushes many thousands or even tens of thousands of messages per second, use a separate TCP connection. For anything else that is slower (dozens of messages per second), multiple channels on the same connection work great.
One last consideration is that when a given channel misbehaves or you perform an operation that the broker doesn't like, the only recovery that I've seen is to shut down the entire connection which can affect others channels on the same connection.
262 comments
[ 2.8 ms ] story [ 265 ms ] threadSome of that joy is surely just moving from older, creakier solutions. But it hasn't let us down, and everyone is eager to use it for new features or refactoring legacy code.
The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.
I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc, or more and more I see lightweight systems going completely serverless and just using lambdas - which are HTTP microservices.
They could have stayed trying to do continually running microservices on HTTP.
> I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc
I do too.
> more and more I see lightweight systems going completely serverless and just using lambdas - which are HTTP microservices.
Likewise.
But long running HTTP microservices are lame, and everybody realises that now, despite it being a cool idea back in 2015.
Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers?
My main use-case is to receive status/change notifications from a service running elsewhere from the API server servicing the UI, in order to avoid polling for new data.
I've also looked into other solutions (ActiveMQ, Google PubSub, ...) and RabbitMQ is by far the most straight-forward and quick to set up. There are some edge cases that it doesn't cover as well, for example automatic retries, but there are some "RabbitMQ patterns" to make it work. For a simple message broker/queue system, it's great and the docs are also great.
Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising!
Is there anything I should keep in mind for running it in production? Any best practices or gotchas, based on your experience (eg don't run in docker, or make sure there's lots of RAM or things like that)? I guess its all in the production checklist. I need to read through it all again!
The accepted wisdom that I've seen is to run a single broker with a completely independent hot spare. But of course switching over to your hot spare will violate most of the guarentees that Rabbit gives you around durability, ordering etc, so you have to be very careful how you use it.
I desperately want to like Rabbit (and have used it heavily in the past) but right now I wouldn't use it if I can get away with anything else, it just has no real HA story.
For anyone who wants to understand the potential complexities of HA RabbitMQ, spend some time reading https://jack-vanlightly.com/blog/2018/8/31/rabbitmq-vs-kafka...
To be honest, the current setup was set up by colleagues who have even less experience than me and it's still running flawlessly. Iirc, it's just two instances that are behind a load balancer and the consumer just consumes from both, but I'm not super certain on that.
I've tested the cluster functionality to see how to set it up and it worked fine for me, but I have no experience with that in production, but other people in this thread don't seem to be too happy with it, so ymmv.
> Is there anything I should keep in mind for running it in production?
Nothing special that you wouldn't do otherwise, when getting to know a new component/microservice. Just check out the get started section[1] on their page in the appropriate language and play around with a small setup. Get familiar with the libraries to connect and send/queue/fetch stuff and the topics. Make sure to use your brainpower before you set it up to handle all eventualities and is set up exactly how you want it to act (ACK/NACK, what happens if a sender/consumer dies, etc.), because then you set it up once and probably never touch it again.
One thing I'm not really sure on and what I haven't really answered myself yet: there might be some "logic" to your RabbitMQ instance, depending on which metadata you add to each message (e.g. retries). If you have such logic, it might be better to have a service around the RabbitMQ instance, otherwise this logic ends up in your code base of you actual solution and that might not be wanted and maybe harder to maintain. But I'm not so sure myself on that one.
Oh yeah, and check out some patterns for your needs. There are for example multiple ways to implement retries, for example with a queue for queues, etc. But if you API is REST based, everything should be straightforward.
[1]: https://www.rabbitmq.com/getstarted.html
Never had a single issue in all those years.
But, I must admit that running a HA cluster is something that I've never tried, it sounds complicated and scary once you start digging through the docs.
All my deployments have been to bare metal Ubuntu and Debian machines with durable Qs and messages.
If you need to use transactions, they are really slow, couple of orders of magnitude slower compared to regular AMQP usage.
A common use-case is for async tasks (Celery) that can tolerate a few minutes of downtime. If you're running a fully evented architecture then this might not apply - though if you're not targeting 4-5 nines of reliability or an RTO of < 5mins, then you might not need a standby even if RMQ is a core part of your architecture. "Avoid single points of failure" is a good heuristic, but "consider the SLAs of your dependencies" is the more granular way of thinking about this, and a single RMQ instance has a very high uptime.
For context I had an RMQ docker container running for almost two years without any interruptions. If you're in a small team then HA might well be overkill.
Fun gotcha - if you're running RMQ in Kubernetes/Docker, make sure you give it an explicit memory limit, else it will try to allocate disk space equal to 40% of your host's memory. (See "memory limits" in https://hub.docker.com/_/rabbitmq). That's a good best-practice for any containerized environment regardless what workload you're running, but this one will cause errors if you're trying to use a small disk volume on a host with lots of memory.
Has worked out quite well so far.
RabbitMQ gave us such a performance increase that we killed our database. We ended up having to rate limit RabbitMQ!
In general the defaults are pretty good I think. There is a one page production deployment guide: https://www.rabbitmq.com/production-checklist.html that I followed to replace our handbuilt cluster w/ a new automated deployment, plus a few other niceties like docker logs & rmq metrics to cloudwatch and then auto clustering via autoscaling groups lookup.
I thoroughness of the docs can perhaps seem daunting, but I see it as a badge of quality and especially if you are growing it's usage organically it should "just work".
If you're bad at hosting and need the throughput, there's cloudamqp.
So many options for pub/sub systems so use what works for you.
I think this is why email will never die. It's basically turned into a huge message queue. Even voice mails come into my inbox.
====== EDIT - I meant to say "huge universal message queue" and left out the word "universal" accidentally
There's a lot of work in mailer-daemons to ensure that email has as reliable as possible delivery in a store-and-forward system..
Thanks for catching that.
https://pypi.org/project/dirq/
Perl had the original implementation, and there are implementations in other languages.
Only pitfall are the available libs. Especially with the .NET implementation we had quite a lot of trouble. Its not following current .NET patterns and has strange quirks. Does anyone know a good alternative to the "official" one?
[0] - https://github.com/EasyNetQ/EasyNetQ
It would be great to get specific, actionable feedback with your experience, either via a message to the rabbitmq-users mailing list or via a GitHub. The .NET client is an old library but considerable effort into improvement went into version 6.0. The plan for 7.0 is to address old patterns that remain in the library. Feedback would help guide that effort.
I just released version 6.1.0-rc.1 and would appreciate testing if you have time. Thanks!
If the library were being designed from scratch today, pretty much every method on the model would be Async. After all, if it leads to any network I/O of any kind, that can block.
Working with the current public API, Trying to implement a publish wrapper that never blocks, and returns a task that either completes when the publisher confirm is received, or faults after some provided timeout, is a lot trickier than it might sound.
Recovery from network interruptions is complicated, and auto-recovery features are limited, and in some use cases actually dangerous. For example, if you are manually acknowledging messages to ensure end-to-end at-least-once delivery, then you cannot safely use the auto-recovery, since the delivery numbers would reset when the connection does, and you can accidentally aknowlodge the wrong message with delivery tag 5. (Acknowledge the new one, when you were trying to ack the old one).
In my implementation of that included my own recovery, I ended up needing to pass around the IModel itself with the delivery tags, so I can check if the channel I am about to acknowledge on is really the same one I received the message on. (There is no unique identifier of a channel instance, since even the channel number is likely to get re-used).
If you have code you can share that you used to address shortcomings in the client, we could get ideas from it for the next major release. Cheers!
RabbitMQ has excellent support for complex message flow topologies. Kafka out of the box does not provide these features.
Kafka is a stream, and can be replayed (if you have it set up to store stuff). Rabbit is simply a queue, and when the messages are gone, they're gone.
This means that queues are a lot smaller, but can only serve one set of consumers at at time. If you want to have multiple things listening to messages, you have to use fan-out patterns that place messages on multiple queues. Queues can also suffer from less than atomic delivery, especially if the system is distributed. This means you have to jump through some hoops and add an atomic layer somewhere if you want to ensure you're not double processing anything.
Kafka can have infinite retention (if you got the storage/$), and you don't need to have multiple streams to service multiple consumers. Each consumer stores where they are in the stream, and can traverse as needed. You'll need to be careful to make sure that a single consumer is handling a single partition to promise that you'll only process a message once.
Managing streams can be a headache, but less so now if you have money to have Amazon or Confluent manage it for you. They offer pretty much unlimited scalability, and are the production grade solution for a ton of problems.
Queues are really simple to understand and build and still scale pretty dang well. Just make sure your message processing is idempotent and make sure you can handle if something is processed multiple times.
Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the road.
Can anyone shine some light?
I've seen quite a lot messages going through RabbitMQ. I wouldn't worry too much about scaling, because the possibilities depend very much on the architecture. With some tuning RabbitMQ can take you a long way. I would give clustering a go and see where the limits are before exploring more complicated architectures like federation.
Kafka creates the abstraction of a persistently stored, offset-indexed log of events. You read all events in a topic. Kafka can be used to distribute messages in the way AMQP is used, but is more likely to be the centerpiece of an architecture for your entire system where system state is pushed forward/transformed by deterministically processing the event logs.
Unless you already have Kafka infrastructure, setting up Kafka for a brand new project is crazy unless your only goal is learning how to set up Kafka.
RabbitMQ and Kafka are very different struggles when thinking of scaling and performance. Kafka is almost a database itself of messages which have routed through the system. In many configurations clients can come back and demand to replay the message stream from almost any point in time. This means you need to handle _a lot_ of disk and memory access. With RabbitMQ, messages are traditionally very ephemeral. Once a message has been ack'd, its gone. Poof. Not in memory. Not on disk. Nobody is going to come back asking for that message. This leads to a lot more efficiency in handling things per message, but at the cost of not being able to remember the messages that went through the system a few milliseconds ago.
I have a system that only pushes 5k messages per second but it needs 32 cores.
The clustering might look tempting but it hasn't been resilient for me in the face of janky networks. Split brains and data loss can result.
In the past I've scaled my rabbits for throughput by implementing my own routing/sharding layer.
If you're tempted to use the message persistence and you care about retaining messages, kafka is a bigger but much more capable hammer.
We often lost entire queues because a small network blip caused RabbitMQ to think there was a network partition, and when the other nodes became visible, RabbitMQ has no reliable way to restore its state to what it was. It has a bunch of hacks to mitigate this, but they don't solve the core problem; the only way to run mirrored queues ("classic mirrored queues", as they're not called) reliably is to disable automatic recovery, and then you have to manually repair RabbitMQ every time this happens. If you care about integrity, you can use the new quorum queues instead, which use a Raft-based consensus system, but they lack a lot of the features of the "classic" queues. No message priorities, for example.
I've never used federation or Shovel, which are different features with other pros/cons.
If you're willing to lose the occasional message under very high load, NATS [3] is absolutely fantastic, and extremely fast and easy to cluster. Alternatively, NATS Streaming [4] and Liftbridge [5] are two message brokers built on top of NATS that implement reliable delivery. I've not used them, but heard good things.
[1] https://www.rabbitmq.com/partitions.html
[2] https://www.rabbitmq.com/quorum-queues.html
[3] https://nats.io/
[4] https://docs.nats.io/nats-streaming-concepts/intro
[5] https://github.com/liftbridge-io/liftbridge
nowadays? it's actually quite simple to setup and works pretty well (source: i know two different companies that setup clustering recently and both had good experiences with no downtime).
I can offer a similar anecdote: we started seeing rabbitmq reporting alleged cluster partitions in production after enabling TLS between rabbitmq nodes, where manual recovery was needed each time.
After a bit of investigation we noticed that cluster partition seemed to correlate with sending an unusually large message (think something dumb like 30 megs) through rabbitmq when TLS between rabbitmq nodes was enabled. What I believe was happening was Rabbitmq was so busy encrypting/decrypting large message that it delayed sending or receiving heartbeat & then the cluster falsely assumed there has been a network partition.
Mitigated that issue by rewriting system to not send 30 meg messages- there was only one message producer that sent messages anywhere near that large, and after a bit of thought realised it was not necessary to send any message at all in that case (sending large message was to hack around some other old system performance problem that had gotten fixed properly a year back, but the hack that generated a huge message was still in place)
http://blog.erlang.org/OTP-22-Highlights/
The fragmentation in particular addresses the problem where a large message would block all other messages, including heartbeats, and cause nodes to look “down” when they’re not.
If you are moving to a more "event-sourced" architecture, usually two main concerns (beyond basic operational stuff of uptime, scale, etc) are routing and long-term retention.
RabbitMQ has the routing but not the retention. Kafka can have the retention and the routing, but it can be complex/expensive. Apache Pulsar really shines here as the API is pub/sub but it is underpinned by a log structure that gives you long-term retention (that doesn't need to be manually re-balanced) but it's flexibility does come with some operation complexity when compared to RabbitMQ.
If your needs is pretty much just moving large amounts of data, Kafka is definitely the most mature and has a big ecosystem, but long term-retention is difficult and there are some sharp edges around consumer groups.
If you really really don't need long-term retention and need complex topologies, RabbitMQ is your best bet and is fairly reasonable to operate even up to fairly high message rates (~10k msgs/sec shouldn't be too hard to achieve)
There are a TON more options these days though, older more java solutions like activeMQ and rocketMQ or more "minimal" implementations like NATs, not to mention the hosted services on cloud providers.
Personally, I am a big fan of Apache Pulsar for it's flexibility and some nice design choices, but I don't think there is any silver bullet in this space.
I think pulsar is wonderful, but I haven't had the chance to use it for anything serious / in production yet, so I'm curious what pain points you had.
Anyone know of any Pulsar hosting providers?
1) Much easier to implement and maintain for small to medium architectures. However, war stories I've heard is that it starts to become a hassle for large clustering architectures.
2) Because it's a traditional message broker, the input and output ends, which I was responsible for, were much simpler to write because I didn't have to worry about replays when it came back online. Rabbit knows which client it has already routed to and where messages went. Kafka is not that sophisticated in that regard. Kafka has been described as "dumb broker/smart clients" while Rabbit is "smart broker, dumb clients."
3) The scaling. Rabbit is very scalable. Once you get to the Uber/Paypal level (like, a couple of million writes per second), then Kafka becomes the obvious choice. Rabbit handles thousands or writes per second just fine. However, at that second company and like many others, they thought they'd have to suck up all the data, so of course, Kafka was the more scalable tool long term. Spoiler: We were never, ever close to PayPal-level transactions. If the size of the sun represents paypal/Uber transactions, we were basically Manhattan.
There will be times when you lose offsets or when you actually want to replay every message, so take an hour and figure out what that means to your app. It's usually only a few lines of code in your consumer that compares source timestamps, but it's by far the most beneficial thing you can do when working with Kafka in my experience.
It's also relatively easy to hit "tens of thousands" messages/second, especially in replay or bootstrapping scenarios, and that's when Kafka becomes useful to the non-FAANG companies.
By default it only retains non-acked messages, multiple subscription modes, can use non-persistent messaging, dead letter queue, scheduled delivery, can use Pulsar Functions to implement custom routing etc.
Scales like Kafka (probably better) and has cluster replication built in.
[0] https://masstransit-project.com/
ZeroMQ was born out of a frustration with complex routing patterns and the need for a broker-less architecture for maximal performance message delivery.
[1] https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf
RabbitMQ is a messaging system. ZeroMQ is sockets on steroids.
If you use the brokerless model, there was a bit of drama over ZeroMQ — the original technical developer (Martin Sustrik) left and created a successor, nanomsg, with what he learned. At some point, Martin lost interest, and Garrett D’Amore took over maintenance and did a rewrite called nng. Both the old nanomsg and nng are maintained, with nng being somewhat actively developed, but also fairly “complete”, so there’s not a lot of excitement like you see with some projects. ;) nanomsg and nng are essentially wire-compatible, so you can mix and match depending on bindings availability for your language.
ZeroMQ certainly isn't perfect, for example there's no way to tell if a message was successfully written to a PUB socket, or if it was dropped (just one minor issue)
https://stackoverflow.com/questions/13891682/detect-dropped-...
Anyway, This is digressing from the main topic
[1]: https://github.com/nanomsg/nng
Our company has been using ZeroMQ for over 8 years. We'll be putting out another ZeroMQ-based open source project soon too.
We've been working with ZeroMQ a lot over the past couple of years, and have gotten to know some of the maintainers -- we've been very favorably impressed by their ability and dedication.
Pieter Hintjens was the "voice" of ZeroMQ, and with his passing things have gotten a bit quieter, but no less active. (Just take a look at the commit log: https://github.com/zeromq/libzmq/commits/master).
Let's say you've got one exchange and one main queue for processing: jobs.exchange and jobs.queue respectively.
If you need to schedule something for later, you'd assert a new queue with a TTL for the target amount of time (scheduled-jobs-<time>.queue). Also set an expiry of some amount of time, so it'd get cleaned up if nothing had been scheduled for that particular time in a while. Finally, have its dead-letter-exchange set to jobs.exchange.
This could lead to a bunch of temporary queues, but the expiration should clean them up when they haven't been used for a bit.
https://docs.celeryproject.org/en/v2.3.3/userguide/periodic-...
:)
Thanks Gavin, I learned a lot from reading it!
https://github.com/nats-io/nats-site/issues/217
(The issue is from 2017 but illustrates a distinction)
So, for example, you would have a CRUD that takes requests, and when there is background work to be done, places a message on the queue, and immediately returns to the user. This frees up the server for more requests. Meanwhile in the background, a worker process chugs through the queue and does its work. During long spikes it will take longer to get through the queue, but your end users will not have disruption of service.
You move this to a queue, and have a worker chop that data file up into individual records, those records go onto a queue, and you can process them however you want, no worries about something crashing and not being able to be retried. If the database goes down, everything just pauses until it can go again. You can limit the queue throughput to whatever you want to avoid having to scale your API/Database.
Can you handle stuff via all CRUD sync APIs? Sure, just like you could handle running a restaurant where you have one person who takes the order and cooks it and delivers it to a table. However, it's more efficient to have a waiter (API) take requests and give them to a cook (queue based async worker) to handle stuff that's not as time sensitive. This saves you a lot of money in certain situations.
Reader Mode is the answer. The creator of that deserves a Nobel Prize.
Anyone considering RabbitMQ needs to read up on "network partitions", how to build your cluster to avoid them (odd number of nodes and pause_minority), your recovery strategy for when a network partition occurs (it will occur), your personal/organizational tolerance for message loss and a plan for how you will upgrade your cluster at some later date (ensure you architect your application to handle whatever type of upgrade strategy you will pursue).
There are definitely ways to operate to minimize these failures but you SHOULD KNOW ABOUT THEM before your add this service to your environments.
I'm looking to pitch adding a message queue to our infrastructure (at a .Net shop on Azure), and I'm sure there will be some questions about the comparisons between the two. Unfortunately, that's been tough to really track down.
From a ten thousand foot view, two or three node clusters running in non-prod environments on virtual machines running Windows. In Prod, three node clusters on Windows virtual machines.
All work to install and configure RabbitMQ is done manually. Sadly enough.
I'm on the application/architecture side of this equation but I know enough about our infrastructure to perhaps answer follow-ups or more specific questions.
Our application is single tenant (so each customer is deployed in their own isolated area) so we use virtual hosts to isolate each customer within the cluster.
>Also if you've done any comparison to Azure Message Queues and what were the pros and cons against Rabbit?
Definitely looked in to the Azure native queueing options but it's been awhile. Azure Message Queues is an AMQP compliant messaging system that seems fairly robust. To be transparent, I have no production experience with this product. If your company/department is in to managing virtual machines then they might want/prefer to go with RabbitMQ. However, if they're in to PaaS systems then I'd probably roll with Azure Message Queues and never look back.
I think part of the sell is how we would manage the admin component of a Message Queue, which tilts things towards Azure Message Queues as it's PaaS. We're mostly IaaS at the moment, and starting to see some of the admin overhead that comes with managing that infrastructure ourselves. We're not ready to jump onto a PaaS solution for the things we've grown accustomed to managing, but for something brand new, I think my company would be open to it.
Architecturally, we'd lean on it initially for background job processing, which is currently at a scale where our homegrown, db-backed solution is starting to show it's weaknesses. Once it's in place though, I think it could leveraged as a key component to decouple subsections our application and give us more flexibility with scaling and deployment.
Yes.
>is there any tradeoffs to running Rabbit on Windows, as opposed to Linux?
Should be fine to run on Linux assuming you (or you have) people are who are comfortable admin'ing Linux servers. I think that a Windows admin would get frustrated to setup/configure RabbitMQ on a Linux server. There's also a container advantage as RabbitMQ is published to Docker only with officially maintained Linux images.
>We're not ready to jump onto a PaaS solution for the things we've grown accustomed to managing, but for something brand new, I think my company would be open to it.
I'd push you to figure out why Azure Messages Queues would not work for you. If there's no compelling "no" argument then you'll thank yourself later.
>Architecturally, we'd lean on it initially for background job processing, which is currently at a scale where our homegrown, db-backed solution is starting to show it's weaknesses.
We pursued RabbitMQ for very similiar reasons (queueing mechanisms via SQL Server tables and stored procedures). Keep in mind that you still need something to submit the job (initiate the background task). RabbitMQ is not going to automagically schedule anything for you. We have a couple applications that use the tool Hangfire for job scheduling and in one case, the Hangfire job simply sends a message to RabbitMQ.
Unfortunately streams were released after we introduced RabbitMQ to our application and I really wish we could just focus on Redis.
Also, see the following articles:
Laika - https://www.rabbitmq.com/blog/2019/12/16/laika-gets-creative...
Bloomberg - https://tanzu.vmware.com/content/rabbitmq/keynote-growing-a-...
Goldman Sachs - https://tanzu.vmware.com/content/rabbitmq/keynote-scaling-ra...
Softonic - https://www.cloudamqp.com/blog/2019-01-18-softonic-userstory...
There are some big companies talking about their experience.
There's a different architecture for:
* one queue with billions of messages
* a millions of queues with small numbers of messages per queue
* many queues with many messages per queue
There are also different topologies:
* Anyone can send a message to anyone (O(n^2) queues)
* One publisher with millions of subscribers
* One subscribed with millions of publishers
* Complex processing networks, where messages get routed in complex ways between processing nodes.
There are differences in timing:
* More-or-less instant push notifications
* Jobs which run within e.g. 5 minutes with polling
* Jobs which run in hours/days, with a cron-style architecture
And in reliability:
* Messages get delivered 100% of the time, and archived once delivered
* Messages get delivered 99.999% of the time, but might be dropped on a system outage
* ... all the way down to ephemeral pub-subs
... and so on.
I'd give my VP's right eye to get a nice chart of what supports what. For the most part, I've found build to be cheaper than buy due to lack of benchmarks and documentation for my use cases. Otherwise, you build. You benchmark. You optimize. And things melt down.
My use case right now requires a large number of queues (eventually millions). I'd like to have an archival record of messages. Peak volume is moderate (several messages per second per queue), but usage patterns are sporadic (most queues are idle most of the time). Routing is slightly complex but not supper-complex (typically, about 30 sources per sink, at most 200; most sources only go to one sink, but might go to 2-3). Messages are relatively small (typically, around 1k), but isolated messages might be much bigger (still <1MB, but not small).
My experience has been that when I throw something like that into pick-your-queue/pub-sub, things melt down at some point, and building representative benchmarks is a ton of work.
Likewise this configurability makes case specific benchmarks very awkward.
People will compare it to Kafka, claiming that its pubsub is faster than Rabbit's, but that's sort of missing the point: Rabbit thrives because it's easy to set up, will work well for 99% of cases, and handles nearly every kind of distributed problem you're likely to come across.
I recently did a project with Rabbit on my home server, and while the project had some issues, the issues were never Rabbit.
I've been using Rabbit in production for RPC and pub/sub for the past 5 years (single instance running on a non-dedicated VM, medium traffic) and its been pretty easy to setup and has been pretty reliable in practice.
I've always been concerned about losing messages, and I did have to learn to turn on persistence and durability for messages to survive server interruptions, but it was easy enough. Message acknowledgements are also a nice feature, and Rabbit is able to achieve at-least-once messaging semantics.
That said, for most small to medium-large tasks, Rabbit will handle things without much trouble, making it a good fit for most common usecases.
You don't have to pool connections as channels are multiplexed by them.
Things to watch out for:
- opening too many channels - these map to Erlang processes and can overwhelm your server if you go over ulimits - sharing consumer channels between threads - you might see weird behavior (e.g. acking wrong messages etc)
We've built own library/framework for creating resilient consumers, and it enforces mapping 1:1 channels and consumer threads, as well as automatic reconnections and channel clean ups.
The general takeaway from this should be: if you've got a particular stream of messages (either a producer or a consumer) that pushes many thousands or even tens of thousands of messages per second, use a separate TCP connection. For anything else that is slower (dozens of messages per second), multiple channels on the same connection work great.
One last consideration is that when a given channel misbehaves or you perform an operation that the broker doesn't like, the only recovery that I've seen is to shut down the entire connection which can affect others channels on the same connection.