I don't think this is a very good article in general and especially not for HN where Docker has been covered in detail already. (The history of the LXC vs. libcontainer split is pretty far inside baseball and not even that interesting IMO.)
I gasped at this assertion: "Virtuozzo, by the way, is still the best container system available. Unfortunately, it isn't open source, and that means you have to pay if you want to use it. Virtuozzo, by the way, has the distinction of being the only halfway decent container solution to (currently) run natively on Windows."
I don't even know how to process a comment like that about Virtuozzo. The fragility of Virtuozzo and OpenVZ is infamous. We get more support queries about our software running on OpenVZ and Virtuozzo containers than all other virtualization types combined (our stuff runs on a few hundred thousand hosting servers, we have a really good sample size).
For years, whenever we get a mysterious "this thing stops for no reason" or "this error doesn't make any sense" or "sometimes it works and sometimes it doesn't" support request, literally our first clarifying question is, "Is this an OpenVZ or Virtuozzo system?" It is, frankly, broken by design, with regard to memory allocation.
I would recommend anything other than Virtuozzo/OpenVZ, because of broken memory allocation. Virtuozzo allows hosts to oversell RAM, without properly handling over-allocation. So, memory allocations fail while reporting available memory, making it difficult to troubleshoot, and difficult for users to predict what their VM can accommodate.
If intermittent failures without useful error messages is the worst kind of failure (and it is, IMHO), then Virtuozzo intentionally subjects its users to the worst kind of failure. Unless you know how to spot this kind of problem, it's easy to bang your head on it for days or weeks; and we've had users who were ecstatic when we diagnose their problem and provide the knowledge they need to spot it in the future (though resolving it may be more tricky than diagnosing it). Even without the host intentionally overselling, we've seen this behavior happen, because allocation seems to require contiguous blocks of memory (or something else that means that even with enough available memory allocations can fail).
I am not deeply familiar with the code, only with spotting this problem in the wild. I've seen assertions, spanning years (including from Parallels), that it's been fixed, or that as long as the host isn't overselling RAM it won't happen. But, we still get these support requests regularly. I'm not an expert on their software, but I'm an expert on spotting when it's their software that is causing our customer's despair. (Note that the support requests are usually about Postfix, Apache, MySQL or other services "crashing"; it rarely manifests with our software specifically, since our software rarely needs to allocate large amounts of memory.)
If you happen to be stuck on a Virtuozzo or OpenVZ system, and have mysterious failures of services, you can find out of it is this problem by checking /proc/bc/* or /proc/user_beancounters.
I don't really get what your problem is? You can disable the memory limitations if your use case requires it, or set the limits as high as you want.
How else would you suggest that OpenVZ should handle over-allocation? It is, after all, one of the points of virtualization (and the reason why you can get such containers for cheap).
You can't really blame OpenVZ if the users or sysadmins are unaware that the container has hard limits and that you're able to see if you've run into them using /proc/user_beancounters. Of course, if OpenVZ fails to allocate memory when you're under the limit, that's a separate problem that should be adressed (although I've yet to run into that one).
But also to be fair, OpenVZ introduced VSwap (http://wiki.openvz.org/VSwap) to address some of the confusion that occurred under the previous resource management solution. User bean counters was notoriously hard to comprehend for mere mortals, and the default limits were reasonably tight, so most commonly Java based applications were unreliable in OpenVZ.
Otherwise I agree with you that admins should detect VZ-related OOM conditions. But a good project takes ownership over the most common pitfalls and finds ways of making their software easier to use (if their goal is to remain popular or grow).
It isn't my problem, it is Parallels problem, and the problem of their customers. Though, it becomes my problem when our customers come to us for help due to Parallels failure.
You say it is working as designed, I say that design is broken. Those are not mutually exclusive assertions.
"You can't really blame OpenVZ if the users or sysadmins are unaware that the container has hard limits and that you're able to see if you've run into them using /proc/user_beancounters."
It is not the "hard" limit that is the problem...it is the "soft limit" which is arbitrary and can be unallocate-able, with the user having no way to know that it will be unallocate-able until it causes a service to exit. So-called "burst" RAM in VZ containers would better be called Schrödinger's RAM, and it has no place in a server.
Further, it doesn't even seem to exclusively be a problem of "burst" RAM. We've had support requests show up on systems that had, say, 1 GB of "guaranteed" RAM and 2 GB of "burst" RAM, but still behave as though it has significantly less than 1 GB of allocate-able RAM. I've addressed that in my comment above, but I didn't give an example. Perhaps this is more clear?
Regardless, the behavior of VZ containers even when they are working as designed is so user-hostile that I just can't recommend it. Our products support OpenVZ containers, because we had a number of customers tied to the platform from before their were better alternatives, but I've never been fond of them.
If it works for you, and you don't share your system with anyone else, go ahead and do what makes you happy. But, from my perspective, as someone supporting probably more web hosting server users than anyone else I know (except maybe some friends who work at large hosting providers and a friend who works at our largest direct competitor), I know that VZ containers cause a lot of confusion.
Been there, done that, run an illumos distro now; it's like night and day. Calling Virtuozzo the best container system is a bad joke. Same applies to OpenVZ.
I don't know when they changed from "user_beancounters" to "VSwap" but VSwap fixes all these problems. It is also a problem with FreeSwitch (the VOIP server software, like Asterisk). Under VSwap-enabled OpenVZ there is not 1 problem.
I wish they would tell their customers. We have not stopped seeing these types of problem reports. And, I've heard that the problem was solved numerous times in the past, so I'm hesitant to believe that it is finally fixed now.
I have read a lot about Docker now. Here and elsewhere. I even have been to Docker talks in my town. Still I do not get what the usecase is.
Everytime I ask that, I get the answer "When you want to run applications seperated from each other in their own environment". Yeah, I get that. But what is the real world use case for that? Who wants to do that? Now I will get answers like "Everybody who wants to run applications in containers" :) Yeah ok. But who? Do bloggers need Docker? People who run onlineshops? Would Ycombinator benefit from running Hacker News inside a Docker container?
Imagine you have 20 services that need to run with your website ( 5 database daemons, 5 nodejs apps, 5 ngnix instances, 2 dns servers, 2 mail smtpd daemons, 1 git repository ).
Your infrastructure consists of 5+ virtual servers that host your services. Each virtual server might be database only or db + dns or web + dns or ... you got the point.
What docker allows you to do is to isolate all those 20 services in separate containers that if any of them is being hacked your attacker will end up compromising only that service without any access to the main server.
What docker allows you to do is to version control the configuration and the container for the different services in a way that if you need to reinstall a service ( redeploy your app ) it is less of a hassle and since it is containeraized you will not be compromising your other services while doing that.
Maybe I misunderstood GP's question, but what you describe has been done with countless other solutions for years, why should I specifically use docker and not stay with kvm, or bsd jails, or...?
At least in the case of KVM, it's a very heavy-weight solution. Hardware virtualisation takes more RAM, and the I/O is substantially lower. I think hardware virtualisation (e.g. KVM) should only be used when the use-case demands it (e.g. Windows, OS development, or legacy applications).
As for why Docker? Dunno. While I think illumos zones are pretty awesome myself, having a standardised format is handy. Docker is also popularising the container mantra with Linux.
To add to bildung's comment, what many of us are wondering is given all the existing tools and complexity, why add another integration point?
In the scenario you're describing where you're provisioning virtual machines in a cloud, shuffling docker containers around on a fleet of VMs may not be the way to go. Many people would use some combination of disk images and configuration management to achieve the same ends.
So you have 20 containers each running their own version of the OS? Don't you have to spend a lot of time keeping all of those up to date with OS/library/app updates?
It seems like the kind of thing that makes sense if you have a big deployment anyway, so you might as well hive stuff off, but makes much less sense if you're a "small" shop.
Can you make changes in the underlying OS image, and those changes are directly visible in the application images, without restarting said applications?
What happens is you'd rearchitect (if needed) your stack so that it scales across N-nodes (for each tier). Then docker build files & snapshotting features allow you to 'recompose' the image from a base OS, one step of which is to apply all patches.
So when you are performing patching, you'd bring a container into service with the updated patches, send some traffic to it (possibly tightly controlled to minimise disruption), verify and then put into production. Iterate through your N-nodes per tier and you've patched. Every container after the first is rapidly started up (just the cost of a supervisor/init & the application process).
> What docker allows you to do is to isolate all those 20 services in separate containers that if any of them is being hacked your attacker will end up compromising only that service without any access to the main server.
Have the Docker developers changed their stance that they don't promise that Docker can safely untrusted programs running as root? (https://news.ycombinator.com/item?id=7910117)
Well most certainly container to host escalation is possible, but if you play your cards well enough it is highly unlikely.
From the security section [1] :
> Your average server (bare metal or virtual machine) needs to run a bunch of processes as root. Those typically include SSH, cron, syslogd; hardware management tools (e.g., load modules), network configuration tools (e.g., to handle DHCP, WPA, or VPNs), and much more. A container is very different, because almost all of those tasks are handled by the infrastructure around the container ...
> This means that in most cases, containers will not need "real" root privileges at all. And therefore, containers can run with a reduced capability set; meaning that "root" within a container has much less privileges than the real "root". For instance, it is possible to ...
> This means that even if an intruder manages to escalate to root within a container, it will be much harder to do serious damage, or to escalate to the host ...
Cool thanks. So while you still need to take the same steps to provide security as you would on a physical (or virtual host), Docker gives you the added benefit of a smaller attack surface.
> What docker allows you to do is to isolate all those 20 services in separate containers that if any of them is being hacked your attacker will end up compromising only that service without any access to the main server.
Except that: containers share the same kernel. If a contained application is hijacked with a privilege escalation vulnerability, all running containers and the host are compromised. Also, several resources are not namespaced. Nothing with a shared kernel is going to be very secure.
Malicious container can easily compromise a vulnerable host OS, and an already compromised host OS completely owns the container. This is no different from the failed experiments in sandboxing on PCs.
I have completely destroyed the Oracle installation on my windows machine.. just cannot get it to run anymore. So I quickly downloaded boot2docker and from there pulled in a prebuilt Docker oracle instance. In very little time I have a working oracle server on my machine again. Now if I manage to destroy that install, it won't matter - I can very quickly rebuild it again.
Somehow at the same time I also messed up the Oracle .net drivers.. I am forced to use a specific version that the rest of my team doesn't use.. God knows how, but I have spent days trying to work out what has gone wrong. I am really looking forward to Microsoft implementing Docker for Windows. Then I will be able to develop each project in isolation and if something messes up like this again - it won't matter.
It seems to me the goal is separation of many apps on a single server for the sake of hygiene, security, and reproducibility (in theory), as well as shipping all dependencies for an app to run in one "thing".
The problem I have with it is the level that it is done at. Using OS level virtualization (lxc) seems somewhat messy, because now you have to have a whole addition layer of proxies and indirection for any networked services.
I am left wondering if sandboxing/Capsicum[1], along with something like Nix[2], would be much cleaner abstractions in the long run.
> The problem I have with it is the level that it is done at. Using OS level virtualization (lxc) seems somewhat messy, because now you have to have a whole addition layer of proxies and indirection for any networked services.
Nonsense. You can have LXC containers work at directly at ethernet-level with each container having it's own regular network IP, just like a regular VM.[1]
Just because the default is to create an internal network doesn't mean LXC is limited to this only.
And then all the problems you mention go away. You get the hygenic benfits of a VM without the resource-related cost.
I have a handful of VPS's. Every once in a while I find a fantastic deal on one and add it to my "collection."
Instead of trying to manage which VPS has what running on it - I use docker. Or rather I am migrating to it, still not 100% complete.
It is quite powerful in this regard, allowing me to write small separate applications that run completely agnostic from their underlying hardware/virtualization.
If I cancel a VPS, I just move the container over with a couple simple commands.
So how is this useful? This morning I decided I wanted to have access to a HTTP-DNS tunnel. I launched iodine in a docker container and pushed it to one of my servers - if I need to move it to a different one, it's just a couple commands away from happening without having to reset or rebuild anything.
I have a handful of web scrapers, RSS monitors, an IRC bouncer, etc running in docker containers that make management much easier than my previous solutions.
Now as for a more "professional" use? I'm not as sold on it. If your server has room for multiple "things" running on it - you a.) need multiple smaller servers or b.) your application isn't large enough to even need docker to begin with. But of course this isn't applicable to everything and docker definitely has other benefits that depending on your software may or may not be useful.
1.) Packaging and configuration development. I find it nice for this purpose because a.) it makes it easier to ensure that the work I do does not depend on some quirk of my local system configuration, which has evolved over a long period of time, b.) it is easy to tear things down and start over without consequences, c.) the versioning/layering can be useful for evolving base images
2.) Encapsulating research work and working environments. I can build a full, working environment for my research, with all source and compiled binaries, and be certain that everyone else who comes along will be able to use it and replicate/reproduce my work, or extend it with their own, without a need to hassle with configurations, compilers, etc. I can index the container with tags from a paper or conference so it can be readily found and used.
I guess none of these things requires Docker [or any lxc system for that matter], but I have found it really useful and easy-to-use for these two purposes.
1) Devs get to use whatever environment they want, but the product they work with is guaranteed to be the product on the servers - no mismatched libs and a reduction in WORKSFORME.
2) For incorrigible hackers who like a finger in every pie. Want to work on project X that depends on a given package? Spin up the container. Want to shift to a different project that uses an incompatible version of that package? Spin up a different container.
I'm not seeing it live up to the hype in my ops world - to me, it's just a package-management system with it's own baroque API and a custom init system. Then there's things like the python bindings using a different workflow to the CLI (the python bindings put config in 'start', the cli cannot do this). Cryptic cli responses. The 'docker ps' output that has crazy long whitespace and I have yet to see not wrap to become an unreadable mess on any of my terminals. Downloading a new image not only takes the 'latest' tag from the previous (fair enough), but it also steals the 'repository' name - so now the previous image has neither the tag (fine) or the repo name (!?!?!). I'm also yet to figure out how to get dockerhub to send an email that the build failed. And fun stuff like the benefit of adding yourself to the 'docker' group so you can run docker commands without having root... but you can forget about looking at the logs, no access there!
Yesterday I learned that they also intend to remove usability features rather than add them, with the belief that 'third-party apps' will provide them. Because fragmentation is good when it comes to usability? The functionality is already there, and they want to remove it to 'keep the core small'... rather than perhaps move it into an 'extras' repo (that everyone will end up adding anyway, bar the die-hard minimalists). https://github.com/docker/docker/pull/5001
I'm finding that I'm fighting docker more than it's helping - I am yet to personally see the advantage to ops over a .deb package. Or an rpm, if that floats your boat. But then again, I'm already using largely single-purpose VMs. I guess you can store state in a docker image that's not as simple in a .deb, but then again docker linux images (based off distros) generally start at 100MB and grow quickly from there.
And to make matters worse, my developers who went "ooh, shiny, we HAVE to use Docker!" and pushed the issue... aren't even using it themselves to develop our product. They grew bored of it quickly and reverted to their usual methods. I get puzzled stares when I run into the docker-corner-case-of-the-day. So much for item 1) for me...
Today's docker fun: the current docker.py breaks when trying to remove images from the previous minor version of docker (works on 1.4, breaks on 1.3). I guess docker(.py) doesn't care about the philosopy of semver.
docker.errors.APIError: 404 Client Error: Not Found ("client and server don't have same version (client : 1.16, server: 1.15)")
Docker allows developers to use standard linux command-line skills to assemble and package what are essentially chrooted/jailed applications. Once the application instructions are set in the Dockerfile, it can then be uploaded to a remote server so it can be easily passed around to other machines.
In practice, the main use case for many companies is to use docker to package Ruby, Node or Python applications along with all of their module and OS library dependencies. This allows multiple applications to run on the same physical machine without conflicting with each other.
But Linux doesn't have this problem. Paths to loadable modules are given in the application configuration, and dynamic libraries have versioned file names. On top of that you can give a path to the dynamic linker. So that doesn't really make sense.
I reached part 3 and still don't know a thing about Docker other than it's history and how its the best thing since sliced bread and it's amazing that it is what it is because its amazing.
Does part 4 finally deliver on the promise in the title?
I'm sorry, if you cannot see the benefits to running programs inside a container, then you have been reading the wrong article. I have been using FreeBSD jails for years, and now on Linux deployments, docker as well.
Lets say you have a server, it runs a mail server, database server, web server (proxy) and application server.
If they all run without containers, if one service gets compromised, and a root exploit is found, that's it game over.
If you have a service that starts eating up memory, with proper configuration, it can't overload the root server. Basically each jail/container can only see itself and any exploits cannot effect the host system, or other jails/containers (when configured correctly).
It also allows for easy expansion, when one jailed/containered service gets to large for the server, just move it to another server easily and quickly.
It also allows for speedy deployment, with docker you can bundle everything on your laptop, create an image, then ship this straight to an external host like EC2/Goole Cloud (for example), with the addition of pre built containers for Django/Rails/Postgres/MySQL etc, it creates a ready working environment for developers who might not be to hot on configuring systems. The "shipping" ability to docker is a by product of the container which is another added benefit. There are loads more features than what I have stated here, this is just a very brief summary.
If they all run without containers, if one service gets compromised, and a root exploit is found, that's it game over.
To be fair, if a kernel-level root exploit is found, it's probably also game over for containers. It's possible to have root exploits that cannot escape containers due to UID virtualization or whatever, but typically(?) root exploits are based on being able to mess with kernel memory, in which case escaping a container should also be possible.
> if one service gets compromised, and a root exploit is found, that's it game over.
For root exploits isn't Docker toast as well? I haven't followed Docker in much details, but does Docker actually promise that commands run as root will be contained?
There is a certain level of isolation for root even inside containers, but in the case of a privilege escalation exploit you would most probably achieve "real" root even if inside a container.
I have a more general question for you: I am a senior developer who is starting to figure out devops because I need to deploy two different products we built. I have limited Chef and Ansible experience and pretty good Unix skills.
Should I start out with Docker or am I missing some crucial pieces I should go through first?
The use case is "media processing services" in one case, and a sports betting game in the other.
Docker has a lot of rough edges and it's most useful in large scale-out deployments. In-band devops tools like Chef and Ansible are relatively agnostic to the hypervisor or container solution, so if you only need a few OS instances & managed configuration, I suggest sticking with Chef/Ansible/Puppet (I prefer ansible :)).
You can always scale out with docker later and I know at least ansible are adressing this use case (http://www.ansible.com/docker).
I used puppet (masterless) in a small company, and in my new company I'm using Ansible and Docker. Puppet and Chef are great, but they're overkill unless you have large infrastructure. If there's only one of you doing 'ops', you don't have large enough infra (in my opinion). Ansible is easy to configure and learn, and doesn't wrap you in dependency hell like puppet can.
I would avoid Docker until you have a use-case that Docker solves for you. Use your preferred distro's packaging technique. It can be a bit of a pain (.deb in particular) to start with, but once you're familiar, it's portable and manageable with your OS's native tools. Things 'just work', and don't need an agent to be installed and kept up-to-date. If something goes wrong, it's in your OS's standard logs, not some strange arcane location. And most importantly, you have a bigger pool of interneteers to help learn from and troubleshoot with. I can't complain about the Docker IRC channel, good experience there, but there's still fewer of them.
If you are a .debber and want your own repo, check out Aptly, which makes life much easier than previous repo software on offer. http://www.aptly.info/
TL;DR: Docker is 'another thing to learn', and it's not trivial. Put it off until it solves a problem for you; there's plenty of other stuff to get up to speed with.
Don't try introducing two new technologies at one time. You pretty much have to use configuration management regardless if you're on Docker or not, so I would absolutely recommend to start with Ansible (or Salt/Puppet/Chef) and use that to its fullest potentinal. Only then do you know what your use case for containers are, and how to set them up effectively using your configuration management infrastructure.
One of the things I've found interesting about rapidly popular opensource solutions in the last 1-2 years is how quickly venture cap funding comes in and drives the direction of future development.
Docker has a whole ecosystem of bolt-on github projects that fill-gaps that the Docker project hasn't had time themselves to fill because they are too busy organising conferences, withholding features until the keynote of said conference, and working out how to spend the millions of dollars on future development.
So Docker are on this tightrope trying to innovate whilst avoiding stepping on the toes of their VC-funded 'partners', so their response was simple-core with a plugin framework for partners to integrate into. Anyone else picturing fragmentation in 6-12 months time?
52 comments
[ 3.1 ms ] story [ 111 ms ] threadSpeaking of Docker, I did find some interesting repos today: https://github.com/socketplane/socketplane and https://github.com/ibuildthecloud/only-docker
I don't even know how to process a comment like that about Virtuozzo. The fragility of Virtuozzo and OpenVZ is infamous. We get more support queries about our software running on OpenVZ and Virtuozzo containers than all other virtualization types combined (our stuff runs on a few hundred thousand hosting servers, we have a really good sample size).
For years, whenever we get a mysterious "this thing stops for no reason" or "this error doesn't make any sense" or "sometimes it works and sometimes it doesn't" support request, literally our first clarifying question is, "Is this an OpenVZ or Virtuozzo system?" It is, frankly, broken by design, with regard to memory allocation.
Well, it'd help if you offered a counterexample to the original claim refuting it.
If intermittent failures without useful error messages is the worst kind of failure (and it is, IMHO), then Virtuozzo intentionally subjects its users to the worst kind of failure. Unless you know how to spot this kind of problem, it's easy to bang your head on it for days or weeks; and we've had users who were ecstatic when we diagnose their problem and provide the knowledge they need to spot it in the future (though resolving it may be more tricky than diagnosing it). Even without the host intentionally overselling, we've seen this behavior happen, because allocation seems to require contiguous blocks of memory (or something else that means that even with enough available memory allocations can fail).
I am not deeply familiar with the code, only with spotting this problem in the wild. I've seen assertions, spanning years (including from Parallels), that it's been fixed, or that as long as the host isn't overselling RAM it won't happen. But, we still get these support requests regularly. I'm not an expert on their software, but I'm an expert on spotting when it's their software that is causing our customer's despair. (Note that the support requests are usually about Postfix, Apache, MySQL or other services "crashing"; it rarely manifests with our software specifically, since our software rarely needs to allocate large amounts of memory.)
If you happen to be stuck on a Virtuozzo or OpenVZ system, and have mysterious failures of services, you can find out of it is this problem by checking /proc/bc/* or /proc/user_beancounters.
How else would you suggest that OpenVZ should handle over-allocation? It is, after all, one of the points of virtualization (and the reason why you can get such containers for cheap).
You can't really blame OpenVZ if the users or sysadmins are unaware that the container has hard limits and that you're able to see if you've run into them using /proc/user_beancounters. Of course, if OpenVZ fails to allocate memory when you're under the limit, that's a separate problem that should be adressed (although I've yet to run into that one).
Otherwise I agree with you that admins should detect VZ-related OOM conditions. But a good project takes ownership over the most common pitfalls and finds ways of making their software easier to use (if their goal is to remain popular or grow).
It isn't my problem, it is Parallels problem, and the problem of their customers. Though, it becomes my problem when our customers come to us for help due to Parallels failure.
You say it is working as designed, I say that design is broken. Those are not mutually exclusive assertions.
"You can't really blame OpenVZ if the users or sysadmins are unaware that the container has hard limits and that you're able to see if you've run into them using /proc/user_beancounters."
It is not the "hard" limit that is the problem...it is the "soft limit" which is arbitrary and can be unallocate-able, with the user having no way to know that it will be unallocate-able until it causes a service to exit. So-called "burst" RAM in VZ containers would better be called Schrödinger's RAM, and it has no place in a server.
Further, it doesn't even seem to exclusively be a problem of "burst" RAM. We've had support requests show up on systems that had, say, 1 GB of "guaranteed" RAM and 2 GB of "burst" RAM, but still behave as though it has significantly less than 1 GB of allocate-able RAM. I've addressed that in my comment above, but I didn't give an example. Perhaps this is more clear?
Regardless, the behavior of VZ containers even when they are working as designed is so user-hostile that I just can't recommend it. Our products support OpenVZ containers, because we had a number of customers tied to the platform from before their were better alternatives, but I've never been fond of them.
If it works for you, and you don't share your system with anyone else, go ahead and do what makes you happy. But, from my perspective, as someone supporting probably more web hosting server users than anyone else I know (except maybe some friends who work at large hosting providers and a friend who works at our largest direct competitor), I know that VZ containers cause a lot of confusion.
Been there, done that, run an illumos distro now; it's like night and day. Calling Virtuozzo the best container system is a bad joke. Same applies to OpenVZ.
Everytime I ask that, I get the answer "When you want to run applications seperated from each other in their own environment". Yeah, I get that. But what is the real world use case for that? Who wants to do that? Now I will get answers like "Everybody who wants to run applications in containers" :) Yeah ok. But who? Do bloggers need Docker? People who run onlineshops? Would Ycombinator benefit from running Hacker News inside a Docker container?
the same way you separate you model from your view from your controller, you can separate your various services from one another.
Your infrastructure consists of 5+ virtual servers that host your services. Each virtual server might be database only or db + dns or web + dns or ... you got the point.
What docker allows you to do is to isolate all those 20 services in separate containers that if any of them is being hacked your attacker will end up compromising only that service without any access to the main server.
What docker allows you to do is to version control the configuration and the container for the different services in a way that if you need to reinstall a service ( redeploy your app ) it is less of a hassle and since it is containeraized you will not be compromising your other services while doing that.
etc.
As for why Docker? Dunno. While I think illumos zones are pretty awesome myself, having a standardised format is handy. Docker is also popularising the container mantra with Linux.
In the scenario you're describing where you're provisioning virtual machines in a cloud, shuffling docker containers around on a fleet of VMs may not be the way to go. Many people would use some combination of disk images and configuration management to achieve the same ends.
It seems like the kind of thing that makes sense if you have a big deployment anyway, so you might as well hive stuff off, but makes much less sense if you're a "small" shop.
So when you are performing patching, you'd bring a container into service with the updated patches, send some traffic to it (possibly tightly controlled to minimise disruption), verify and then put into production. Iterate through your N-nodes per tier and you've patched. Every container after the first is rapidly started up (just the cost of a supervisor/init & the application process).
Have the Docker developers changed their stance that they don't promise that Docker can safely untrusted programs running as root? (https://news.ycombinator.com/item?id=7910117)
From the security section [1] :
> Your average server (bare metal or virtual machine) needs to run a bunch of processes as root. Those typically include SSH, cron, syslogd; hardware management tools (e.g., load modules), network configuration tools (e.g., to handle DHCP, WPA, or VPNs), and much more. A container is very different, because almost all of those tasks are handled by the infrastructure around the container ...
> This means that in most cases, containers will not need "real" root privileges at all. And therefore, containers can run with a reduced capability set; meaning that "root" within a container has much less privileges than the real "root". For instance, it is possible to ...
> This means that even if an intruder manages to escalate to root within a container, it will be much harder to do serious damage, or to escalate to the host ...
[1] : https://docs.docker.com/articles/security/
Except that: containers share the same kernel. If a contained application is hijacked with a privilege escalation vulnerability, all running containers and the host are compromised. Also, several resources are not namespaced. Nothing with a shared kernel is going to be very secure.
Malicious container can easily compromise a vulnerable host OS, and an already compromised host OS completely owns the container. This is no different from the failed experiments in sandboxing on PCs.
Can your container share a Linux instance with containers from other customers in the cloud? Not with any confidence, no. Containers do not contain: http://opensource.com/business/14/7/docker-security-selinux
Somehow at the same time I also messed up the Oracle .net drivers.. I am forced to use a specific version that the rest of my team doesn't use.. God knows how, but I have spent days trying to work out what has gone wrong. I am really looking forward to Microsoft implementing Docker for Windows. Then I will be able to develop each project in isolation and if something messes up like this again - it won't matter.
The problem I have with it is the level that it is done at. Using OS level virtualization (lxc) seems somewhat messy, because now you have to have a whole addition layer of proxies and indirection for any networked services.
I am left wondering if sandboxing/Capsicum[1], along with something like Nix[2], would be much cleaner abstractions in the long run.
[1]: http://www.cl.cam.ac.uk/research/security/capsicum/freebsd.h...
[2]: http://nixos.org/nix/
Nonsense. You can have LXC containers work at directly at ethernet-level with each container having it's own regular network IP, just like a regular VM.[1]
Just because the default is to create an internal network doesn't mean LXC is limited to this only.
And then all the problems you mention go away. You get the hygenic benfits of a VM without the resource-related cost.
[1] Simple example on setting this up on Debian: https://wiki.debian.org/LXC/SimpleBridge
I have a handful of VPS's. Every once in a while I find a fantastic deal on one and add it to my "collection."
Instead of trying to manage which VPS has what running on it - I use docker. Or rather I am migrating to it, still not 100% complete.
It is quite powerful in this regard, allowing me to write small separate applications that run completely agnostic from their underlying hardware/virtualization.
If I cancel a VPS, I just move the container over with a couple simple commands.
So how is this useful? This morning I decided I wanted to have access to a HTTP-DNS tunnel. I launched iodine in a docker container and pushed it to one of my servers - if I need to move it to a different one, it's just a couple commands away from happening without having to reset or rebuild anything.
I have a handful of web scrapers, RSS monitors, an IRC bouncer, etc running in docker containers that make management much easier than my previous solutions.
Now as for a more "professional" use? I'm not as sold on it. If your server has room for multiple "things" running on it - you a.) need multiple smaller servers or b.) your application isn't large enough to even need docker to begin with. But of course this isn't applicable to everything and docker definitely has other benefits that depending on your software may or may not be useful.
1.) Packaging and configuration development. I find it nice for this purpose because a.) it makes it easier to ensure that the work I do does not depend on some quirk of my local system configuration, which has evolved over a long period of time, b.) it is easy to tear things down and start over without consequences, c.) the versioning/layering can be useful for evolving base images
2.) Encapsulating research work and working environments. I can build a full, working environment for my research, with all source and compiled binaries, and be certain that everyone else who comes along will be able to use it and replicate/reproduce my work, or extend it with their own, without a need to hassle with configurations, compilers, etc. I can index the container with tags from a paper or conference so it can be readily found and used.
I guess none of these things requires Docker [or any lxc system for that matter], but I have found it really useful and easy-to-use for these two purposes.
1) Devs get to use whatever environment they want, but the product they work with is guaranteed to be the product on the servers - no mismatched libs and a reduction in WORKSFORME.
2) For incorrigible hackers who like a finger in every pie. Want to work on project X that depends on a given package? Spin up the container. Want to shift to a different project that uses an incompatible version of that package? Spin up a different container.
I'm not seeing it live up to the hype in my ops world - to me, it's just a package-management system with it's own baroque API and a custom init system. Then there's things like the python bindings using a different workflow to the CLI (the python bindings put config in 'start', the cli cannot do this). Cryptic cli responses. The 'docker ps' output that has crazy long whitespace and I have yet to see not wrap to become an unreadable mess on any of my terminals. Downloading a new image not only takes the 'latest' tag from the previous (fair enough), but it also steals the 'repository' name - so now the previous image has neither the tag (fine) or the repo name (!?!?!). I'm also yet to figure out how to get dockerhub to send an email that the build failed. And fun stuff like the benefit of adding yourself to the 'docker' group so you can run docker commands without having root... but you can forget about looking at the logs, no access there!
Yesterday I learned that they also intend to remove usability features rather than add them, with the belief that 'third-party apps' will provide them. Because fragmentation is good when it comes to usability? The functionality is already there, and they want to remove it to 'keep the core small'... rather than perhaps move it into an 'extras' repo (that everyone will end up adding anyway, bar the die-hard minimalists). https://github.com/docker/docker/pull/5001
I'm finding that I'm fighting docker more than it's helping - I am yet to personally see the advantage to ops over a .deb package. Or an rpm, if that floats your boat. But then again, I'm already using largely single-purpose VMs. I guess you can store state in a docker image that's not as simple in a .deb, but then again docker linux images (based off distros) generally start at 100MB and grow quickly from there.
And to make matters worse, my developers who went "ooh, shiny, we HAVE to use Docker!" and pushed the issue... aren't even using it themselves to develop our product. They grew bored of it quickly and reverted to their usual methods. I get puzzled stares when I run into the docker-corner-case-of-the-day. So much for item 1) for me...
/rant
In practice, the main use case for many companies is to use docker to package Ruby, Node or Python applications along with all of their module and OS library dependencies. This allows multiple applications to run on the same physical machine without conflicting with each other.
I have only occasionally had Python scripts broke because of dependencies, but not since the introduction of virtualenv.
Does part 4 finally deliver on the promise in the title?
Lets say you have a server, it runs a mail server, database server, web server (proxy) and application server.
If they all run without containers, if one service gets compromised, and a root exploit is found, that's it game over.
If you have a service that starts eating up memory, with proper configuration, it can't overload the root server. Basically each jail/container can only see itself and any exploits cannot effect the host system, or other jails/containers (when configured correctly).
It also allows for easy expansion, when one jailed/containered service gets to large for the server, just move it to another server easily and quickly.
It also allows for speedy deployment, with docker you can bundle everything on your laptop, create an image, then ship this straight to an external host like EC2/Goole Cloud (for example), with the addition of pre built containers for Django/Rails/Postgres/MySQL etc, it creates a ready working environment for developers who might not be to hot on configuring systems. The "shipping" ability to docker is a by product of the container which is another added benefit. There are loads more features than what I have stated here, this is just a very brief summary.
To be fair, if a kernel-level root exploit is found, it's probably also game over for containers. It's possible to have root exploits that cannot escape containers due to UID virtualization or whatever, but typically(?) root exploits are based on being able to mess with kernel memory, in which case escaping a container should also be possible.
For root exploits isn't Docker toast as well? I haven't followed Docker in much details, but does Docker actually promise that commands run as root will be contained?
You can always scale out with docker later and I know at least ansible are adressing this use case (http://www.ansible.com/docker).
Saltstack in particular has really good Docker integration at this point, which may be useful. YMMV.
I would avoid Docker until you have a use-case that Docker solves for you. Use your preferred distro's packaging technique. It can be a bit of a pain (.deb in particular) to start with, but once you're familiar, it's portable and manageable with your OS's native tools. Things 'just work', and don't need an agent to be installed and kept up-to-date. If something goes wrong, it's in your OS's standard logs, not some strange arcane location. And most importantly, you have a bigger pool of interneteers to help learn from and troubleshoot with. I can't complain about the Docker IRC channel, good experience there, but there's still fewer of them.
If you are a .debber and want your own repo, check out Aptly, which makes life much easier than previous repo software on offer. http://www.aptly.info/
TL;DR: Docker is 'another thing to learn', and it's not trivial. Put it off until it solves a problem for you; there's plenty of other stuff to get up to speed with.
Docker has a whole ecosystem of bolt-on github projects that fill-gaps that the Docker project hasn't had time themselves to fill because they are too busy organising conferences, withholding features until the keynote of said conference, and working out how to spend the millions of dollars on future development.
Even the gap-filler github projects are getting VC funding (see http://weaveblog.com/2014/12/03/announcing-weaveworks-and-ou...). Weave will either become the standard networking stack, or be made irrelevant by an in-tree featureful implementation.
So Docker are on this tightrope trying to innovate whilst avoiding stepping on the toes of their VC-funded 'partners', so their response was simple-core with a plugin framework for partners to integrate into. Anyone else picturing fragmentation in 6-12 months time?