AMQP is so nice from an engineer's perspective. If shit goes south somewhere in the system, I get a pagerduty alerting me that a deferred queue has exceeded some depth and I can start working to resolve the issue (if a message fails to be processed it is sent to the deferred queue). Once the issue is resolved, all the messages are still waiting in the deferred queue and will get processed as soon as they are requeued, and I don't have to replay nginx logs. Anything that can be asynchronous, make asynchronous and save yourself the headache of dealing with 500s when a component of your distributed system fails.
The real issue is that most of the time is spent marshalling all the data into YAML
Using MessagePack would speed that up significantly, too. I've done a comparison of MessagePack vs. JSON vs. YAML vs. Ruby .marshal in the past[0], and I'm currently building a system using AMQP+MessagePack (as of now the only part that's public is [1]).
Basically, YAML is great for configuration files because of its flexible syntax, but terrible for communication because of its flexible syntax.
Not only is it faster than the serializations you mentioned, messages generally have a much smaller memory footprint, and it is therefore very friendly for use in contexts where bandwidth is likely to be at a premium (e.g. mobile carrier data limits). I use it exclusively in mobile apps I develop.
You're absolutely right. The performance and security implications of YAML make it an awful choice.
Unfortunately, upgrading the server component is trivial but upgrading all the clients is next to impossible. The logical next step after this project is to implement a saner set of endpoints that breaks up all the amalgamated functions. At some point in the future, there will come a day in which we'll simply ignore the requests from older clients.
You can make it so that finding a free slot is always O(1). Keep another array that is a stack of available slot indexes. When you need a free slot, pop off an index and use that slot. When you're done with a slot, push its index on the stack.
So to speed up a request, the author wants to introduce a whole message queue, and rewrite it in another language. This seems like a big maintenance burden to me.
Could the request be changed to use a faster message format instead of YAML?
If YAML can't be replaced, could the YAML marshalling be sped up?
Why not use zmq (http://zeromq.org/bindings:ruby)? And for data serialization use Google Protobuf's port ruby-protobuf; and avoid introducing a central message broker as another point of failure?
My complaint with 0mq is that you wind up needing some sort of service discovery service from what I can tell. In order to get everything working, each client needs to know about at least one servant. Ideally, each client needs to about all servants and use a load balancing algorithm across them. I'd need to keep this configuration at all clients synchronized. I'm aware there are tools to do this, but ultimately I am not the person in charge of operating this.
I'm a little curious why in this case one wouldn't choose to do "RPC over HTTP".
Assuming the following points from the post are correct:
a) the perf problem is YAML marshalling
b) that can be solved via a go rewrite
then perhaps an easy way to solve "I want to call the old code" is to access it over HTTP, using a cheaper marshalling method than YAML.
I guess the tradeoff is whether you'd prefer an AMQP broker in there mediating the requests or an HTTP load balancer (or http client pool code in the go app). And whether you see other uses for that broker in your setup.
It seems like this guy built an awful lot of functionality by hand that is available in any number of libraries...but Ruby and Go? This site, sometimes...its like the moon compared to what most people are doing.
If someone came to me and said...hey, I'm thinking about building a piece of a large app in go I would tell them to shut the fucc up. You guys do way to much of this "latest hotness" bullshit.
19 comments
[ 5.4 ms ] story [ 44.6 ms ] threadUsing MessagePack would speed that up significantly, too. I've done a comparison of MessagePack vs. JSON vs. YAML vs. Ruby .marshal in the past[0], and I'm currently building a system using AMQP+MessagePack (as of now the only part that's public is [1]).
Basically, YAML is great for configuration files because of its flexible syntax, but terrible for communication because of its flexible syntax.
[0] http://blog.mikebourgeous.com/2012/10/12/optimizing-message-... (missing numbers for MessagePack, but mentions 2x speed up beyond best other option)
[1] https://github.com/deseretbook/classy_hash (includes a benchmark script that shows how ridiculously fast MessagePack is compared to YAML)
Unfortunately, upgrading the server component is trivial but upgrading all the clients is next to impossible. The logical next step after this project is to implement a saner set of endpoints that breaks up all the amalgamated functions. At some point in the future, there will come a day in which we'll simply ignore the requests from older clients.
The documentation describes how it works here: http://docs.spring.io/spring-amqp/reference/htmlsingle/#remo...
Could the request be changed to use a faster message format instead of YAML?
If YAML can't be replaced, could the YAML marshalling be sped up?
For sure... for example, by rewriting it in another language?
My complaint with 0mq is that you wind up needing some sort of service discovery service from what I can tell. In order to get everything working, each client needs to know about at least one servant. Ideally, each client needs to about all servants and use a load balancing algorithm across them. I'd need to keep this configuration at all clients synchronized. I'm aware there are tools to do this, but ultimately I am not the person in charge of operating this.
Assuming the following points from the post are correct:
a) the perf problem is YAML marshalling b) that can be solved via a go rewrite
then perhaps an easy way to solve "I want to call the old code" is to access it over HTTP, using a cheaper marshalling method than YAML.
I guess the tradeoff is whether you'd prefer an AMQP broker in there mediating the requests or an HTTP load balancer (or http client pool code in the go app). And whether you see other uses for that broker in your setup.
If someone came to me and said...hey, I'm thinking about building a piece of a large app in go I would tell them to shut the fucc up. You guys do way to much of this "latest hotness" bullshit.
The project isn't complete but is functional enough to be used in production.