Interesting benchmarks, ruined for me by a huge experimental no-no: when one of the tests (the Docker delete) didn't produce the result the author wanted, he investigated the underlying problem and fixed it. Sure enough, that became the headline 49x speed-up. Where the results fit the narrative, no optimization was done/reported.
One thing I'm very interested by is Docker's idea of single-process containers; but whether this can be done with KVM isolation instead of (weaker but cheaper) LXC isolation. One thing I was surprised by from this presentation is that the memory cost of a KVM vs LXC instance was only 185MB vs 45MB. So KVM's stronger isolation might not be as expensive after all...
Absolutely agree, but (at least in my mind) Docker with a full OS really is just LXC. Running a single process inside the container is something that genuinely differentiates Docker from other approaches.
It's a powerful idea which to date has been tied to LXC containers, though I think that's a false association: it's just as interesting with KVM or Xen containers.
I have successfully run KVM inside of a container, so there's no reason not to combine them if that makes sense. Or to run a container inside of KVM.
The beauty (to me!) of Docker is that it still applies whether you're talking about any linux-based infrastructure. There are just trade-offs to be made in certain cases (like performance/density for very strict isolation)
All of the data including both sets of delete / reboot tests are provided in the benchmark presentation. How others decide to create headlines or side news is out of my control. From my POV the results contain all the data points and don't seek to mislead the reader.
Also keep in mind there are currently 23 pages of mysql images on docker index right now https://index.docker.io/search?q=mysql. Choosing an image which has "proper" bootstrapping is sometimes a trial and error process. This process is a bit more cut and dry with hypervisor / VM based images where you get a standard set of init processes out of the box.
RE KVM -- I agree KVM is impressive and the low memory overhead is likely a result of KSM (memory de-duplication) which is enabled out of the box on Ubuntu 12.04 with KVM 1.0.
The problem is not that you didn't publish the data. It is that, when collecting it, you chose to fix/re-run some experiments, but not others. The ones you chose to re-run were (presumably) based on your pre-conceived notions of what things "should be". The slippery slope here is that, if you had different notions, perhaps you would have tried to fix different experiments. Perhaps KVM would have been faster if you believed it should be faster and so had used one of the boot-by-cloning-instance patches. Perhaps the memory usage would have been higher if you thought it should be and so turned off memory sharing (and then maybe the CPU load would have been much lower).
I appreciate the transparency in admitting that you re-ran the test, and appreciate anyone that tries to provide data for OpenStack. But you shouldn't claim that you're just publishing numbers for people to interpret as they will, when interpretation entered into the production of those results. It's just not valid experimental practice.
I understand your perspective and will take your comments under consideration for future efforts or subsequent iterations of the existing benchmark.
However let me reiterate the intent of the existing work... The benchmark's purpose is to educate the community on the potential benefits of a LXC based technology (primarily from a Cloudy perspective) by providing a set of data points with full disclosure as to the means / methods used to perform tests and collect results. I expect any organization, group or individual considering actions as a result will perform their own validation to assert the technology choice is beneficial for their consumption prior to adoption. I believe this criteria is met in the current publication and therefore the results will stand on their own.
As to test configuration variations such as those stated above (disable KSM, boot by clone, etc.) -- I believe additional data sets collected via variations or other benchmarks are key to obtaining a well rounded understanding of the factors / benefits involved in this space and I encourage the community to create / publish such results. As an example I would like to gauge the impact of the various KVM guest cache modes and docker LXC storage providers which I believe are fair game to test and add as a variation to the existing benchmark.
Should you wish to iterate other configurations / requirements you believe are worth exploring I can make note of them for the next round of tests (time permitting). In addition I'm more than happy to document the instructions to setup and perform these tests so that others can run it themselves. I gladly welcome additional tests and data points from the broader community at large.
One thing to keep in mind is lightweight containers work well for deployment (isolation), but they can't replace KVM for general virtualization tasks, as guest kernel has to be the same as the host OS.
If you have your hosts running CentOS and want to deploy on Windows, for example, or want to migrate some legacy server to the "cloud" this might not work well for you.
Using virtualization technology to actually to abstract away hardware hardware and OS differences? How is that an edge case. Isn't that the raison d'être of virtualization. That fact that host and guest OS kernels happen to match is the edge case then (a nice one too, that can lead to good optimizations).
Now deployment, isolation, repeatable test and dev environments is a bit of a different beast. So LXC works great there.
The point was to avoid the conclusion that KVM is useless now in all cases, throw it away tomorrow and start using LXC.
Would you consider it edge-cases to run Linux-instances on a Windows Hyper-V based server? Would you consider it a edge case running a Windows VM in Linux?
I certainly wouldn't. Those are definitely what I call "general" virtualization cases.
And as awesome as containers are, they wont be able to cover for these cases. It's definitely worth mentioning for those who are not familiar with the matters.
Agreed -- LXC as a technology is not a silver bullet and will not cover all use cases. However for those looking to do virt with Linux distro's IMHO the need to use traditional VMs becomes the edge case moving forward. I provided some details on the gaps and use cases for traditional VMs sections here: http://www.slideshare.net/BodenRussell/realizing-linux-conta...
There are some folks having a lot of success taking apps running on old versions of centos and running them on later versions, as an example.
To elaborate - Some distros / companies are planning it to be part of their upgrade strategy - i.e, containerize the application, move to <your distro of choice cough rhel 7/atomic cough> while having the userland of the previous release intact.
When security for LXC is finished, this is a perfectly reasonable option. The key in my mind is that the user shouldn't need to use or care if they are using KVM, docker/lxc or VMWare. It should be mostly abstracted by Openstack.
Using Windows? lets use KVM.... Using budget compute and linux, lets use LXC.
But isn't libcontainer just using the same mechanisms in the Linux kernel as LXC anyway? It seems a little redundant to me; although I see value in the 'execution driver' approach as I assume they will ultimately deploy Docker on Solaris and use jails there.
I'm annoyed that the new LXC execution driver appears to have broken something in the LXC networking; previously I was able to run a Docker container using LXC networking, give it its own MAC address and have it assigned an IP address by my router and be visible to all other devices on my network (use case: a container to run Plex Media Server).
Yes correct... As I mentioned below the tests are intended to show the benefits of an LXC based technology in the Cloud space. Docker was chosen b/c that's my preferred "LXC provider", but could have just as easily been libvirt-lxc, OpenVZ, etc.
If you are new to the idea of containers vs traditional virtualization, I have put together a screencast about containers, entitled: "Introduction to Containers on Linux using LXC". Can be seen @ http://sysadmincasts.com/episodes/24-introduction-to-contain...
The intent of the benchmark was to show benefits of LXC as a technology in the Cloudy space. It just so happened I chose docker as the LXC "provider" for these tests -- it could have just as easily been libvirt-lxc or something else.
Given OpenVZ is an LXC technology, comparing docker to OpenVZ would have been more of a comparison between different LXC providers. I agree this would be interesting, however not in line with the intent of the tests.. BTW -- I'm aware that OpenVZ is probably the most mature LXC solution out there.
As for docker vs Xen paravirt, that would be interesting as well and the information I presented clearly stated additional testing is warranted. I would encourage others to try such configurations and present results.
Well I, for one, want to thank you for all of this work. I have done some benchmarks in the past, and they take a ton of work!!! It's easy to sit back and armchair quarterback about performance, but it's WAY more work to sit down and do this.
I encourage anybody that has problems with your experiment to go out and retest the results and share their work!
I recently tried using both Docker and LXC and I just had to ditch Docker.
If you wanted something as "esoteric" as having normal networking in your container, you had to understand how Docker abstracted LXC in the first place, then find out how to do things the LXC way and then push your changes through, with some extra duct-tape with pipeworks.
And if that's the start, before I've even done anything, I can't imagine things getting better. So Docker? No thanks.
Then I'll just go ahead and use LXC directly. It involved a bit more research, but nothing you can't do in a few hours. And when you do, you can get to benefit from BTRFS volumes and having snapshot management tied directly to your containers.
Maybe I'm missing something, but after fooling around with this for a few days, I can't see why anyone would choose the leaky abstraction that is Docker when you can just go with the real deal.
Did you try (redhat's)-libvirt? I used that for a project. It can use lxc, but also qemu and kvm. All with a generic interface (which actually is not so generic). It was quite nice to work with.
I guess my project wasn't suited for docker. All i needed was an image to run a process in a custom environment. I didn't need the Dockerfile and versioning stuff. Which, in my opinion, is the reason to use docker.
So instead of saying "Docker, no thanks", you probably did not find a suitable project. Which is good! I see a lot of people using docker, just because it's awesome and everybody on HN is talking about it.
34 comments
[ 8.0 ms ] story [ 81.3 ms ] threadOne thing I'm very interested by is Docker's idea of single-process containers; but whether this can be done with KVM isolation instead of (weaker but cheaper) LXC isolation. One thing I was surprised by from this presentation is that the memory cost of a KVM vs LXC instance was only 185MB vs 45MB. So KVM's stronger isolation might not be as expensive after all...
See https://phusion.github.io/baseimage-docker/ as an example of a full-OS docker base image designed to be lightweight.
These days, you can run the full OS inside docker[1][2], but you'll likely be running more than you need to.
[1] https://github.com/dotcloud/docker/issues/1024 [2] https://github.com/dotcloud/docker/issues/2276
That being said, running init inside of a container is definitely a supported use case.
It's a powerful idea which to date has been tied to LXC containers, though I think that's a false association: it's just as interesting with KVM or Xen containers.
The beauty (to me!) of Docker is that it still applies whether you're talking about any linux-based infrastructure. There are just trade-offs to be made in certain cases (like performance/density for very strict isolation)
Also keep in mind there are currently 23 pages of mysql images on docker index right now https://index.docker.io/search?q=mysql. Choosing an image which has "proper" bootstrapping is sometimes a trial and error process. This process is a bit more cut and dry with hypervisor / VM based images where you get a standard set of init processes out of the box.
RE KVM -- I agree KVM is impressive and the low memory overhead is likely a result of KSM (memory de-duplication) which is enabled out of the box on Ubuntu 12.04 with KVM 1.0.
I appreciate the transparency in admitting that you re-ran the test, and appreciate anyone that tries to provide data for OpenStack. But you shouldn't claim that you're just publishing numbers for people to interpret as they will, when interpretation entered into the production of those results. It's just not valid experimental practice.
However let me reiterate the intent of the existing work... The benchmark's purpose is to educate the community on the potential benefits of a LXC based technology (primarily from a Cloudy perspective) by providing a set of data points with full disclosure as to the means / methods used to perform tests and collect results. I expect any organization, group or individual considering actions as a result will perform their own validation to assert the technology choice is beneficial for their consumption prior to adoption. I believe this criteria is met in the current publication and therefore the results will stand on their own.
As to test configuration variations such as those stated above (disable KSM, boot by clone, etc.) -- I believe additional data sets collected via variations or other benchmarks are key to obtaining a well rounded understanding of the factors / benefits involved in this space and I encourage the community to create / publish such results. As an example I would like to gauge the impact of the various KVM guest cache modes and docker LXC storage providers which I believe are fair game to test and add as a variation to the existing benchmark.
Should you wish to iterate other configurations / requirements you believe are worth exploring I can make note of them for the next round of tests (time permitting). In addition I'm more than happy to document the instructions to setup and perform these tests so that others can run it themselves. I gladly welcome additional tests and data points from the broader community at large.
If you have your hosts running CentOS and want to deploy on Windows, for example, or want to migrate some legacy server to the "cloud" this might not work well for you.
Now deployment, isolation, repeatable test and dev environments is a bit of a different beast. So LXC works great there.
The point was to avoid the conclusion that KVM is useless now in all cases, throw it away tomorrow and start using LXC.
I certainly wouldn't. Those are definitely what I call "general" virtualization cases.
And as awesome as containers are, they wont be able to cover for these cases. It's definitely worth mentioning for those who are not familiar with the matters.
There are some folks having a lot of success taking apps running on old versions of centos and running them on later versions, as an example.
To elaborate - Some distros / companies are planning it to be part of their upgrade strategy - i.e, containerize the application, move to <your distro of choice cough rhel 7/atomic cough> while having the userland of the previous release intact.
Using Windows? lets use KVM.... Using budget compute and linux, lets use LXC.
I'm annoyed that the new LXC execution driver appears to have broken something in the LXC networking; previously I was able to run a Docker container using LXC networking, give it its own MAC address and have it assigned an IP address by my router and be visible to all other devices on my network (use case: a container to run Plex Media Server).
http://www.infoq.com/news/2014/03/docker_0_9
In speaking to him he's very well aware about libcontainer.
Given OpenVZ is an LXC technology, comparing docker to OpenVZ would have been more of a comparison between different LXC providers. I agree this would be interesting, however not in line with the intent of the tests.. BTW -- I'm aware that OpenVZ is probably the most mature LXC solution out there.
As for docker vs Xen paravirt, that would be interesting as well and the information I presented clearly stated additional testing is warranted. I would encourage others to try such configurations and present results.
I encourage anybody that has problems with your experiment to go out and retest the results and share their work!
If you wanted something as "esoteric" as having normal networking in your container, you had to understand how Docker abstracted LXC in the first place, then find out how to do things the LXC way and then push your changes through, with some extra duct-tape with pipeworks.
And if that's the start, before I've even done anything, I can't imagine things getting better. So Docker? No thanks.
Then I'll just go ahead and use LXC directly. It involved a bit more research, but nothing you can't do in a few hours. And when you do, you can get to benefit from BTRFS volumes and having snapshot management tied directly to your containers.
Maybe I'm missing something, but after fooling around with this for a few days, I can't see why anyone would choose the leaky abstraction that is Docker when you can just go with the real deal.
I guess my project wasn't suited for docker. All i needed was an image to run a process in a custom environment. I didn't need the Dockerfile and versioning stuff. Which, in my opinion, is the reason to use docker.
So instead of saying "Docker, no thanks", you probably did not find a suitable project. Which is good! I see a lot of people using docker, just because it's awesome and everybody on HN is talking about it.
And then I see no need to meddle in meta-frameworks to cover for potential use-cases I don't have, no matter how well they are engineered or tested.
Docker obviously has some work to do, but, I think you do it a disservice by ignoring the rest of the cool stuff the ecosystem brings with it.