Not necessarily. No warmup time was allowed for rubinius's cache. Also we don't know how concurrency was being handled. Rubinius should be able to utilize multiple cores. More information about the hardware, rails app, way that the puma server is being started, warmup time, duration of request time spent on the database, etc should be made available before any remote conclusion like which one is faster can be made.
Not necessarily. No warmup time was allowed for rubinius's cache. Also we don't know how concurrency was being handled. Rubinius should be able to utilize multiple cores. More information about the hardware, rails app, way that the puma server is being started, warmup time, duration of request time spent on the database, etc should be made available before any remote conclusion like which one is faster can be made.
No. Rubinius' JIT has not had time to warm up. These tests are a little unreliable. I can't find the article off hand, but the way these tests were done aren't exactly telling me anything. Power of ten tests don't do much other than show me that the tester is naive.
This is in no way a good test.
Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming.
Also, this is in dev mode... did you disable Rack::Lock?
MRI can only handle 1 request at a time, Rubinius can handle whatever your concurrency level is (assuming you've removed Rack::Lock from your environment)
> This is in no way a good test. Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming.
This is a fair point. You should run ab multiple times over a longer duration of time. 5 seconds seems like a short time to declare an absolute winner.
The ACM's SIGMETRICS[1] are the main group publishing this kind of research. Some of it gets very specialised -- just picking a title from the most recent Performance Evaluation Review[2], I get "YOUQMON: a system for on-line monitoring of YouTube QoE in operational 3G networks".
However, being the ACM, it's basically inaccessible unless you sell a kidney.
An organisation with a slightly more industrial focus is the Computer Measurement Group[3]. Like the ACM ... bring money.
Sorry to go off-topic, but does anyone know if it's possible to get your hands on these pubs/journals for free at the Science, Industry and Business Library [1] in NYC?
I would also love to see an application in production measured with NewRelic or something else and compare the results. A good way to test this would be to spin up three servers.
* 1 load balancer that plays round robin on the requests.
* 1 server running MRI
* 1 server running Rubinius
Each with NewRelic installed and compare the A/B test that is taking place for a period of about a week of heavy usage. Obviously the gems that the rubinius application would have would be different than what the MRI gems would be. But the core of the application would remain the same.
This would give me and others better insight as to how the two interpreters are running in a production.
Although you may be running a single-threaded test intentionally, I generally recommend using a multi-threaded load simulation tool versus Apache's ab tool [1]. My preference is wrk [2], especially since it provides for time-limited tests rather than request-limited. Another alternative is WeigHTTP [3], which is basically a multi-threaded replacement for ab.
27 comments
[ 11.6 ms ] story [ 1521 ms ] threadAlso, this is in dev mode... did you disable Rack::Lock?
MRI can only handle 1 request at a time, Rubinius can handle whatever your concurrency level is (assuming you've removed Rack::Lock from your environment)
This most likely means Rack::Lock is used in your test.
This is a fair point. You should run ab multiple times over a longer duration of time. 5 seconds seems like a short time to declare an absolute winner.
So which is correct?
Some questions:
* How is startup time controlled for?
* Why only one run?
* Why so few requests?
* Why no warmup period?
* Why this particular configuration?
* What is the OS, hardware configuration?
* Why is it running in a development environment instead of a test or prod environment?
iMac Intel Core 2 Duo 3,06 GHz cpu 1
However, being the ACM, it's basically inaccessible unless you sell a kidney.
An organisation with a slightly more industrial focus is the Computer Measurement Group[3]. Like the ACM ... bring money.
[1] http://www.sigmetrics.org/
[2] http://dl.acm.org/citation.cfm?id=2518025&CFID=252561427&CFT...
[3] http://www.cmg.org/
[1] http://www.nypl.org/locations/sibl
* 1 load balancer that plays round robin on the requests.
* 1 server running MRI
* 1 server running Rubinius
Each with NewRelic installed and compare the A/B test that is taking place for a period of about a week of heavy usage. Obviously the gems that the rubinius application would have would be different than what the MRI gems would be. But the core of the application would remain the same.
This would give me and others better insight as to how the two interpreters are running in a production.
[1] https://news.ycombinator.com/item?id=6211883
[2] https://github.com/wg/wrk
[3] https://github.com/lighttpd/weighttp