Absolutely. I happen to need these bits so zeromq is a non-starter for me. I am mildly annoyed by the "we are faster" rhetoric without the "but we only support a small subset of the useful things you can do with a message queue" caveat, but it is a mostly meaningless bit of marketing so I can let it slide; anyone who really needs those sorts of speeds knows how to actually evaluate products and is unlikely to be suckered by a false dichotomy setup like the one on the zeromq home page.
It turns out that RabbitMQ's throughput of tens of thousands of transient messages per second per CPU is enough for a lot of cases. Similar story with latency.
ZeroMQ does a lot of neat stuff to make it go faster still, e.g taking out the server, and removing all routing info from the messages, but ends up being much more like an alternative to sockets as a result, which makes it less useful for general work.
In summary - RabbitMQ and ZeroMQ are complementary, rather than really being alternatives to one another.
Not much by way of technical content. Although when the original charge is that the protocol is "fundamentally flawed", I don't know how you can respond specifically beyond saying "it works and people are using it" (which it seems to and which they are).
You could say: "Yes, you are right. We, too, have noticed what huge pile of hopelessly overengineered garbage it is. Thanks for admitting your own mistake, so glad we can finally get rid of this monstrosity and move on to something sane."
This would be the automatic response from anyone who has had the misfortune to work with AMQP, thus I'm a bit baffled the RabbitMQ guys want to stick with it.
Edit: And yes, I'm not really surprised that I'm getting modded into oblivion. Still, would the downvoters mind to elaborate on what kind of exposure they've had to AMQP and what part of it they consider worth keeping?
I'm not down-modding you, but I am using AMQP in production on http://tweeteorites.com/ with RabbitMQ.
I need persistence and flow control, as my consumers are unreliable and production rate is highly variable. This will become even more important as some of the new Twitter data streams come online.
It took me about 2 days to completely reengineer my backend around RabbitMQ and a custom daemon to serve stream requests. Granted, I already had RabbitMQ set up and routing background tasks for celery ( http://celeryproject.org/ ), but it's still pretty trivial to set up and get running. The easy case, which I'm at now, is pretty easy, and I've got a good path to scalability with fan out exchanges.
That said, faster is better, and I'd be interested in zeromq if they had some of the features I need.
Wtf? You may not like what we do with AMQP but I can assure that lots of people are very happy using it in its RabbitMQ incarnation. Which version of AMQP have you "had the misfortune to work with" and how have you found it "hopelessly overengineered"? Is there something else you would recommend - and if so why?
We do actually like making our product better and appreciate constructive criticism.
Please do explain.
Also please note that RabbitMQ supports multiple protocols and one reason we like AMQP 0-8 and 0-9-1 is that they are flexible enough to make this easy (see e.g. RabbitHub on Github).
Oh interesting to hear from a RabbitMQ developer, thanks for taking the time to reply.
Well, I'll try to elaborate on my rant. Most of it from memory (been a year ago since I last touched AMQP).
Back then when I looked at AMQP (and used it for one project):
* It was very hard to find reasonable documentation.
The most accessible I found was the (excellent)
http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ blog post.
This was what eventually got us off the ground. Imho a testament to the
shortcomings of the "official" documentation (where is it?).
* RabbitMQ only implemented AMQP 0.8. I cannot remember the exact differences
between 0.9 and 0.8 (and heck, good luck trying to find a document explaining
them even today). For us it boiled down to some clients simply not working
and it took a while to figure out what the problem even was.
* Client interoperability was generally spotty, even in clients supposed to be
compatible. It seems the idea of that whole spec-file stuff was to make the
protocol pluggable and flexible. However the only thing it seems to have
achieved is to make client development hell.
The client we eventually went with (py-amqplib) doesn't even use the
spec-file because "parsing takes too long".
* Moreover the sheer existence of http://www.rabbitmq.com/interoperability.html
is a testament to a flawed protocol. If I can't rely on a 0.8 client
to talk to a 0.8 server properly, then something is fundamentally wrong.
* There's a huge pile of semantic baggage (Exchanges, Bindings, Queues, Channels) that
left us with a giant "WTF" the more we used it. Yes, it can be made
work. But hell, talk about overengineering. Much simpler constructions
could have been used to achieve the same goals.
* As I just learned during my googling for this post, apparently some of the problems have even been acknowledged
by the officials and are supposed to be simplified in AMQP 1.0
(http://www.amqp.org/confluence/display/AMQP/Background+to+AM...).
But I'm not expecting much, to be honest.
So, that's just a few points that popped into my head immediately, there's more if I'd dig deeper. It's just a fundamentally bad protocol if I have ever seen one, and I'm honestly surprised to see people defend it even after the creators themselves (iMatix) acknowledge it to be a dead-end.
To put it from an different angle: When you think about what AMQP really does then how long would it take you to whip up a near-trivial but feature-equivalent protocol from scratch? I could think of many ways to skin that cat in a better way, with a simple, hackable, plain-text protocol.
You know, one of those protocols where I, as a developer, can even sniff and see what is going on at any time. Those are possible, as demonstrated by redis, memcached, STOMP, beanstalkd and many others.
I acknowledge that some complexity is unavoidable for advanced use-cases. Still, a good protocol layers the complexity in a way so that simple stuff remains simple and complex stuff becomes manageable. To me AMQP is a complete failure in both regards; you are confronted with the full complexity even for the most trivial use-case. And it seems for complex use-cases it adds a hefty layer of complexity of its own, rather than helping to wrap up.
> Well, I'll try to elaborate on my rant. Most of it from memory (been a year ago since I last touched AMQP).
Back then when I looked at AMQP (and used it for one project):
AMQP is quite new - think about the early days of HTTP and XMPP. Some of the things you list have improved in the last year - but we have more work to do.
It's a very good post. I think the documentation at least as far as RabbitMQ goes has improved, see the link store here: http://www.rabbitmq.com/how.html and the del.icio.us repo that it links to, which is organised by tag.
We'd very much appreciate assistance in improving all this!
> Imho a testament to the shortcomings of the "official" documentation (where is it?).
The working group has not done a good job at putting out 'official' documents about AMQP, other than the specs, which do provide documentation. For some reason people choose to not read the specs, but I do recommend the 0-9-1 spec which is about 40 pages long (not too bad considering).
There will be more of a push to do 'official' AMQP docs this year.
> * RabbitMQ only implemented AMQP 0.8. I cannot remember the exact differences between 0.9 and 0.8 (and heck, good luck trying to find a document explaining them even today).
We do actually have such a doc but have not released it since almost everyone uses 0-8. The main point is that 0-8 and 0-9 are almost identical.
> For us it boiled down to some clients simply not working and it took a while to figure out what the problem even was.
> * Client interoperability was generally spotty, even in clients supposed to be compatible. It seems the idea of that whole spec-file stuff was to make the protocol pluggable and flexible. However the only thing it seems to have achieved is to make client development hell. The client we eventually went with (py-amqplib) doesn't even use the spec-file because "parsing takes too long".
Most people have a good experience with most clients. I am sorry you did not - did you try asking for help on the mailing list?
Er.. no. It's testament to our desire to be clear about semantics.
You mention STOMP - I assume you are aware that STOMP's semantics are underspecified so that interoperability is not possible without vendor agreement?
> If I can't rely on a 0.8 client to talk to a 0.8 server properly, then something is fundamentally wrong.
How long did it take for HTTP browsers to work with all web servers? Years and years - and most people only use a subset of HTTP.
> * There's a huge pile of semantic baggage (Exchanges, Bindings, Queues, Channels) that left us with a giant "WTF" the more we used it. Yes, it can be made work. But hell, talk about overengineering. Much simpler constructions could have been used to achieve the same goals.
There are a lot of messaging protocols, and none of them have managed to be complete, interoperable, open and simple. Please do help us to make RabbitMQ better, whether for AMQP or one of the other protocols it supports.
> * As I just learned during my googling for this post, apparently some of the problems have even been acknowledged by the officials and are supposed to be simplified in AMQP 1.0 (tonyg↗
> how long would it take you to whip up a near-trivial but feature-equivalent protocol from scratch?
ok, i am a bit clueless here, but if 0mq has no persistance, how is it reliable (durable)? or, if it's not reliable (durable), isn't this whole debate comparing apples and oranges?
[edited to clarify i am talking about durability and not just reliability in the tcp sense]
You are correct, there is no persistence. 0MQ is basically just a step above socket level communications, while obviating all of the tedious BS socket programming forces you to do.
thanks. this seems an awful lot like the nosql "debate" in that the two "sides" (typically) have different priorities (and the differences are similar, related to consistency and scaling).
[and also makes imatix not look that good, if any customers need the durability part, which is quite a common requirement]
20 comments
[ 18.9 ms ] story [ 188 ms ] threadZeroMQ does a lot of neat stuff to make it go faster still, e.g taking out the server, and removing all routing info from the messages, but ends up being much more like an alternative to sockets as a result, which makes it less useful for general work.
In summary - RabbitMQ and ZeroMQ are complementary, rather than really being alternatives to one another.
alexis
RabbitMQ
This would be the automatic response from anyone who has had the misfortune to work with AMQP, thus I'm a bit baffled the RabbitMQ guys want to stick with it.
Edit: And yes, I'm not really surprised that I'm getting modded into oblivion. Still, would the downvoters mind to elaborate on what kind of exposure they've had to AMQP and what part of it they consider worth keeping?
I need persistence and flow control, as my consumers are unreliable and production rate is highly variable. This will become even more important as some of the new Twitter data streams come online.
It took me about 2 days to completely reengineer my backend around RabbitMQ and a custom daemon to serve stream requests. Granted, I already had RabbitMQ set up and routing background tasks for celery ( http://celeryproject.org/ ), but it's still pretty trivial to set up and get running. The easy case, which I'm at now, is pretty easy, and I've got a good path to scalability with fan out exchanges.
That said, faster is better, and I'd be interested in zeromq if they had some of the features I need.
Cheers
alexis
Wtf? You may not like what we do with AMQP but I can assure that lots of people are very happy using it in its RabbitMQ incarnation. Which version of AMQP have you "had the misfortune to work with" and how have you found it "hopelessly overengineered"? Is there something else you would recommend - and if so why?
We do actually like making our product better and appreciate constructive criticism.
Please do explain.
Also please note that RabbitMQ supports multiple protocols and one reason we like AMQP 0-8 and 0-9-1 is that they are flexible enough to make this easy (see e.g. RabbitHub on Github).
Cheers
alexis
Well, I'll try to elaborate on my rant. Most of it from memory (been a year ago since I last touched AMQP).
Back then when I looked at AMQP (and used it for one project):
* It was very hard to find reasonable documentation. The most accessible I found was the (excellent) http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ blog post. This was what eventually got us off the ground. Imho a testament to the shortcomings of the "official" documentation (where is it?).
* RabbitMQ only implemented AMQP 0.8. I cannot remember the exact differences between 0.9 and 0.8 (and heck, good luck trying to find a document explaining them even today). For us it boiled down to some clients simply not working and it took a while to figure out what the problem even was.
* Client interoperability was generally spotty, even in clients supposed to be compatible. It seems the idea of that whole spec-file stuff was to make the protocol pluggable and flexible. However the only thing it seems to have achieved is to make client development hell. The client we eventually went with (py-amqplib) doesn't even use the spec-file because "parsing takes too long".
* Moreover the sheer existence of http://www.rabbitmq.com/interoperability.html is a testament to a flawed protocol. If I can't rely on a 0.8 client to talk to a 0.8 server properly, then something is fundamentally wrong.
* There's a huge pile of semantic baggage (Exchanges, Bindings, Queues, Channels) that left us with a giant "WTF" the more we used it. Yes, it can be made work. But hell, talk about overengineering. Much simpler constructions could have been used to achieve the same goals.
* As I just learned during my googling for this post, apparently some of the problems have even been acknowledged by the officials and are supposed to be simplified in AMQP 1.0 (http://www.amqp.org/confluence/display/AMQP/Background+to+AM...). But I'm not expecting much, to be honest.
So, that's just a few points that popped into my head immediately, there's more if I'd dig deeper. It's just a fundamentally bad protocol if I have ever seen one, and I'm honestly surprised to see people defend it even after the creators themselves (iMatix) acknowledge it to be a dead-end.
To put it from an different angle: When you think about what AMQP really does then how long would it take you to whip up a near-trivial but feature-equivalent protocol from scratch? I could think of many ways to skin that cat in a better way, with a simple, hackable, plain-text protocol.
You know, one of those protocols where I, as a developer, can even sniff and see what is going on at any time. Those are possible, as demonstrated by redis, memcached, STOMP, beanstalkd and many others.
I acknowledge that some complexity is unavoidable for advanced use-cases. Still, a good protocol layers the complexity in a way so that simple stuff remains simple and complex stuff becomes manageable. To me AMQP is a complete failure in both regards; you are confronted with the full complexity even for the most trivial use-case. And it seems for complex use-cases it adds a hefty layer of complexity of its own, rather than helping to wrap up.
> Well, I'll try to elaborate on my rant. Most of it from memory (been a year ago since I last touched AMQP). Back then when I looked at AMQP (and used it for one project):
AMQP is quite new - think about the early days of HTTP and XMPP. Some of the things you list have improved in the last year - but we have more work to do.
> * It was very hard to find reasonable documentation. The most accessible I found was the (excellent) http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ blog post. This was what eventually got us off the ground.
It's a very good post. I think the documentation at least as far as RabbitMQ goes has improved, see the link store here: http://www.rabbitmq.com/how.html and the del.icio.us repo that it links to, which is organised by tag.
We'd very much appreciate assistance in improving all this!
> Imho a testament to the shortcomings of the "official" documentation (where is it?).
The working group has not done a good job at putting out 'official' documents about AMQP, other than the specs, which do provide documentation. For some reason people choose to not read the specs, but I do recommend the 0-9-1 spec which is about 40 pages long (not too bad considering).
There will be more of a push to do 'official' AMQP docs this year.
> * RabbitMQ only implemented AMQP 0.8. I cannot remember the exact differences between 0.9 and 0.8 (and heck, good luck trying to find a document explaining them even today).
We do actually have such a doc but have not released it since almost everyone uses 0-8. The main point is that 0-8 and 0-9 are almost identical.
> For us it boiled down to some clients simply not working and it took a while to figure out what the problem even was.
This has all got better, there are hundreds of projects now that work with RabbitMQ: http://delicious.com/alexisrichardson/rabbitmq+usecase+clien...
> * Client interoperability was generally spotty, even in clients supposed to be compatible. It seems the idea of that whole spec-file stuff was to make the protocol pluggable and flexible. However the only thing it seems to have achieved is to make client development hell. The client we eventually went with (py-amqplib) doesn't even use the spec-file because "parsing takes too long".
Most people have a good experience with most clients. I am sorry you did not - did you try asking for help on the mailing list?
> * Moreover the sheer existence of http://www.rabbitmq.com/interoperability.html is a testament to a flawed protocol.
Er.. no. It's testament to our desire to be clear about semantics.
You mention STOMP - I assume you are aware that STOMP's semantics are underspecified so that interoperability is not possible without vendor agreement?
> If I can't rely on a 0.8 client to talk to a 0.8 server properly, then something is fundamentally wrong.
How long did it take for HTTP browsers to work with all web servers? Years and years - and most people only use a subset of HTTP.
> * There's a huge pile of semantic baggage (Exchanges, Bindings, Queues, Channels) that left us with a giant "WTF" the more we used it. Yes, it can be made work. But hell, talk about overengineering. Much simpler constructions could have been used to achieve the same goals.
There are a lot of messaging protocols, and none of them have managed to be complete, interoperable, open and simple. Please do help us to make RabbitMQ better, whether for AMQP or one of the other protocols it supports.
> * As I just learned during my googling for this post, apparently some of the problems have even been acknowledged by the officials and are supposed to be simplified in AMQP 1.0 ( tonyg ↗ > how long would it take you to whip up a near-trivial but feature-equivalent protocol from scratch?
Oh, about five years, I'd say.
[edited to clarify i am talking about durability and not just reliability in the tcp sense]
Here's a good intro to 0MQ from lwn.net, along with sample code: http://lwn.net/Articles/370307/
[and also makes imatix not look that good, if any customers need the durability part, which is quite a common requirement]