For starter, failure modes are not decoupled at all. If one db hits a race condition in the storage driver on one instance, it's usually 100% guarantee that all other instances are sensitive to the same bug and it will happen sooner or later.
In practice, occurrences of a bug are highly correlated and usually happen in batch.
However, if you are going to be running docker on real tin (because thats where the value/speed comes in, if you're on AWS thats a whole 'nother issue) Then you might as well use device mapper for what it was originally designed for: mapping fibre channel. (or iscsi, or SAS [another scsi])
That is assuming you want speed, and have paid enough cash to overcome SPoF in your storage layer (it'll be cheaper and faster than trying to software your way out of it.)
This article mentions offhand that the storage drivers are unreliable, even for data volumes.
Is that actually the case? Is there a serious risk that a database will be corrupted by a container crash, as the article claims? A regular crash of the computer should not be able to corrupt a database, is a container more dangerous in this regard?
I've been running a couple of petabytes in production with Docker and Cassandra for a couple years (around 2k nodes). I've rarely seen FS corruption, however I must qualify that this is on bare metal. They could be running into issues with the interaction with EBS? This is more of a screed than an argument backed by specific details and facts.
How do you handle making updates to configuration with all your nodes in containers? Do you blue green deploy the cluster or something? Run config management in the container?
Configuration comes from the environment. We store the configuration per cluster in a centralized store (C*, etcd, SimpleDB). We bake images that contain everything else.
Depending on the customer and the tech involved we'll do blue-green by doing a controlled rolling push of the config or image after it makes it through the dev/test cycle. Also depending on the type of tech we'll store actual data on network or host volumes.
I can imagine that the probability of this happening is higher than when you run without docker.
But on the other hand you have mechanisms like sharding and replications to deal with single machine/zone failure.
I have run database clusters on kubernetes in production without running into this particular problem.
The current state of container orchestrators for running databases is not optimal because one size does not fit all database types like with stateless applications.
One solution for this problem are coreos operators which introduce third party resources into kubernetes that are specific to the database type and contain logic to manage this specific database type on kubernetes.
While it is true that Docker has a variety of storage drivers for the overlay file system, some unreliable, if you are doing DB then you'd be using a host volume mount which is a bind mount. There should be no issues with the bind mount as it is not a part of Docker.
In order for a database to provide such guarantees it needs some guarantees from an underlying hardware.
Mainly that when disk say that data was written, it really was written, and that the data was written in the right order.
The danger with overlay systems is that they might not provide these guarantees, which makes database writes unsafe. Given that current overlay drivers are unstable as it is, I doubt they put any effort to enforce such guarantees.
Not exactly a database but for example with ZFS it is known that ZFS can't provide guarantees that usually provides if you run it in a VM, unless you have a hardware that supports VT-d (a.k.a. PCI pass-through) and have it enabled.
> Not exactly a database but for example with ZFS it is known that ZFS can't provide guarantees that usually provides if you run it in a VM
This is true for any and all file systems. ext4 can't provide those guarantees either because it's up to the hypervisor to do the right thing when the guest requests it to flush caches to disk.
ZFS just happens to have made it very explicit that it requires certain guarantees to provide the data reliability, but those same requirements exist for other file systems.
Scaling everything else (stateless non-persistent services) is nearly trivial, with or without Docker. It's scaling databases where things get interesting, but here we are back to dark ages of DBAs and manual deployment :(
I've been using docker in production with Elasticsearch and MySQL for 3 years in the PB scale and have never had data corruption issues occur.
Corruption occurs on data drives even without docker - you still have to plan for it. This is why you enable replication. This is why you snapshot/backup your data daily and have disaster recovery plans.
There are some major reasons why I actually think running databases in docker containers, even if you are mounting a volume for the data.
1) Development environments can be similar to production. Ensures everyone runs the same version that is running in prod.
2) You don't have to worry as much about what is installed on the host machine.
3) In a clustered setup, it's easier to ensure each node is running the same configuration, version, etc...
One of my issues with all the gripes about docker are the assertions that it causes issues. In all of my time of using docker, 99% of the time when there is an issue it has nothing to do with docker itself. Everyone loves to blame it when things go wrong though.
This article doesn't really back up any of the claims about any of its issues. It just makes blanket statements without backing them up. Don't like docker's networking? Use host networking then.
What people don't think about is the countless issues that will never come up when using containerization. I never have to worry about whether or not python 2.7 is installed on a server that I'm going to deploy a python 3 app on. I also have MUCH higher confidence that if things work on my local development env (which runs the same containers), then there is a high chance it will work in production.
People make it seem as if Docker is some bleeding edge magical technology, but in reality its most useful features are just thin wrappers around stable linux kernel features and some nice automation.
We have also been running databases in Docker (on the tb scale though) for around 3 years, we had the odd issue here and there, but nothing terrible and certainly nothing fundamental or resulting in data loss.
If your data is corrupted by a single process dying in an unclean fashion then you have other operational problems.
Oh hell no, not hating docker! I love docker ( I'm the multiplier in my company and teaching docker,and doing presentations)
But putting pb of data in docker three years ago is just insane.
There are still things not clear right now,which are not stable ( not in unstable destroying data) but unstable in: if something goes wrong with your data you need to dig deep and find out what's going on. If stuff changes every few month. You will have a hard time.
Also, enterprise concerns like:
Will there be a docker standard from Google and Facebook?
Will AUfs be Version 4 or scrapped again?
Will the composer format stay?
How to manage runtime upgrades? Ie syncronize 1000 dockers with one version update than another 1000 with another software. So that one update depends on another.
Etc etc etc.
Three years ago it just wasn't there. Two years ago I would have said: "good enough" to play around. One year ago it started to get really interesting for enterprise.
That's why I say: you handle pb of data,with docker,three years ago?
> People make it seem as if Docker is some bleeding edge magical technology, but in reality its most useful features are just thin wrappers around stable linux kernel features and some nice automation.
That's one of the things to dislike.
The company and the community are trying to sell it as the best thing since sliced bread and usually forget to assign merit to the kernel developers.
On top of that, its 180k lines of code are unwarranted for a "thin" layer.
Part of the problem is that you're using databases that can't cope with failure. In large scale production systems things fail all the time. If you've got tech that can cope with failure it's not an issue.
Additionally, Docker is pretty handy when you're attempting to manage clusters consisting of thousands of nodes. In that instance enforcing best practices, automating workflows, scaling teams, auditing and preventing configuration drift are much bigger problems than a single server failing.
There is no tech in the universe that can cope with cascading failures, like ALL instances of a docker container crashing on ALL hosts one by one in quick succession. This usually happen because an app hits an unexpected bug in the docker disk or the docker network stack and this is the major source of concerns I have with Docker.
Some systems cope with failure better than others. Everything you've said is also true of DB running on top of a uniform linux stack. From my experience (500+ large scale production deployments) this doesn't happen very often.
Does it solve all problems? No. Does it make the world a little better and is it better than monolithic single points of failure? Yes.
Sure, but if it fails often enough that you need to prepare to deal with failure, then the number of times you invoke CleanupAfterFailure() doesn't matter so much.
It's a mix. I'm a consultant that specializes in large scale distributed systems. I have some customers that have >100k production database nodes. I manage probably >50PB of data. I have designed large distributed systems for more than 100 customers.
consultant = charge > £600 a day to bring Docker to the company. Yet doesn't care when shit hits the fan 3 months later because he's already gone. In fact, he will never known about it.
By the way, How to have 100 customers => leave right after the design phase every single time. Clients add up quickly.
I do a mix of pure consulting but also managed services. I typically have a 12 hour SLA for issues, and 1 hour SLA for some customers. 24/7 support for mission critical, revenue generating systems. So no, I'm not just a talking head. It's usually me in the NOC on the hook in case things go wrong. I'm the world expert in this field, if you want things to work at scale people call me.
Individually I charge several orders of magnitude greater than what you're quoting. I'll advise and design, do deep troubleshooting etc.. Consultants that work for me (I'm the CEO) or a large SI will do the implementation.
We have been running databases in some form of containers for ages.
Oracle DB on Solaris Zones, db2 in wpars.
But yeah people have to re-implement these ideas poorly so now we have to deal with consequences.
> I’ve seen DBMS containers running on the same host with service layer containers. But these service layers are not compatible according to hardware requirements.
> Putting your database inside the container, you’re going to waste your project’s budget. Why? Because you’re putting a lot of extra resources to the single instance. And it’s going out of control. In cloud case you have to launch the instance with 64GB memory when you need a 34. In practice some of this resources will stay unused.
For some software, resource consumption is of "fixed" size, plus temporary workload-dependent growth (e.g. application-layer processes, most of the time.) Whereas some other software will take up all the space available to it (like DBMSes.) The latter are what resource quotas are for.
Containers are not meant to be treated like "Unix binaries but more easy to deploy." Containers are just lightweight VMs that don't have to do screwy things with memory balloon drivers to efficiently pack many of those "fixed plus temp growth" workloads onto a host.
But like VMs, containers still need resource quotas to ensure they don't thrash one-another. You can avoid specifying quotas for your fixed-with-temp-growth workloads, to "oversubscribe" a host, and it'll work (similarly to oversubscribing memory-ballooed VMs.) But the "all the space available" workloads need quotas.
The author might be used to public clouds, where VMs have a "size" in vCPUs + memory and that "size" is charged for, and so might not think of picking an instance size for a VM as explicitly setting a quota. But when you set up your own hypervisor cluster, you still have to decide how big each VM should be, regardless of the fact that a bigger VM doesn't "cost" anything: a VM's "size" is the compromise you make between the needs of that workload, and the ability to "fit" other workloads alongside it on a host.
But, to go further: if you're designing "instances" and running dedicated workloads on them, you're very likely "doing containers wrong." (This is probably a provocative statement; stay with me.)
Containers are to container hosts as VMs are to hypervisors: in both cases, their architecture assumes that if you want resource-efficient deployment, you've got a big generic cluster of hosts, and your guests are loaded onto them using a bin-packing algorithm (taking into account which guests need what extra resources that are only available on certain hosts, etc.)
If you don't have a big generic cluster of hosts, then your only packing options will be necessarily sub-optimal. If your container hosts are real hardware, you're out of luck; if your container hosts are themselves VMs, running on some cloud provider, then costs will be heavily in favor of taking advantage of the cloud-provider's bin-packing by wrapping each of your containers in a separate VM and then deploying those VMs.
(Which is, coincidentally, what Amazon's Elastic Beanstalk does for you, and why it's not the same as Amazon ECS. ECS is for setting up your own "big generic cluster" of container hosts to bin-pack across; Elastic Beanstalk is for wrapping containers in VMs so that AWS will bin-pack at their abstraction level.)
I still can't really figure out what a container is. Every time I think of a use case for one, I read something like this which says that's a terrible idea.
The use-case I need solved most often is the following:
Create a standalone "server" that accepts and responds to network traffic, has some way to store data, and whose dependencies (i.e. system packages, frameworks, etc) I can manage independently of any of the other "servers" I have running. Do I just want a bunch of VMs? Or docker instances that all point to some other DB (that's apparently not in a docker instance...?). But then they're no longer independent from one another because they all use the same DB. So do I need a separate DB for each serverlet? Which lives where? On its own VM?
There is nothing special about containers to really understand
Containers are a lightweight way of sandboxing a process. Think a level lower than a VM. You can run multiple containers on a single VM in the same way you can run multiple VMs on a single host.
Ideally a container should be stateless. If a container crashes, you should be able to bring it up again without anything actually caring that it is technically a different process.
A container doesn't solve a "real problem" it mostly makes it easier to manage applications and processes by abstracting out any dependencies from the host VM and keeping everything packaged into a single thing.
A container can run any application that it is configured to run on any VM regardless of the state of the VM (Assuming the VM has a kernel that supports containers)
the "stateless" bit precludes a whole bunch of use cases, for example databases as per the post. For me the issue with containers is that they don't feel very "contained" when you have files ("images") all over the shop that don't get eliminated easily, or when attached volumes have storage in some location, and compose files elsewhere, and having to inject all sorts of environment variables. In other words, files and details scattered around everywhere. It's not nearly as clean as a VM, even though I do get it that for scaling tons of identical web servers, for example, they would be great.
The reason I prefer external (mounted from the host) storage for PGDATA is so I can easily manage it from the host. Otherwise it's tied to the image, which I consider ephemeral.
I'd argue that the stateless bit is more of a Docker idiom than something intrinsic to containers. LXC/LXD, for example, treats containers as machines instead of processes.
> Containers are a lightweight way of sandboxing a process. Think a level lower than a VM.
can you go into a little more depth? my understanding of a VM is that it installs the OS in a dedicated memory partition, and allocates hardware resources separately from that of the host machine, such that resource contention between host and VM never happens. the VM allocated resources just go dark for the host machine while the VM is running.
what is a lower level than that? I've understood containers to be thin wrappers around VMs, which would make them higher level, not lower level. do I have this wrong?
Containers aren't VMs... they're more of a sandboxed model of execution with a virtual network/disk abstraction. But lower/higher you are right, VM is probably lower-level abstraction of an entire system.
But a container can be a single executable, it doesn't have to be an entire OS structure. For example, a lot of the go app containers are just the single executable by itself as a default. Many will rely on a debian/ubuntu base as they want other systems to work. This is mainly because of shared environments/libraries that some non-dockerized systems need, but isn't a requirement.
I'm doing a bit of a simplification here so please someone correct me if I'm not saying this correctly
Every single instance of a VM has its own Kernel. When a VM boots up, it gets allocated a portion of hardware and boots up a kernel and allocates memory to itself. VMs each are isolated from each other in that they don't share resources and each VM is free to do whatever it wants to do with the hardware it is given. Like you said, to the host machine, that hardware is no longer available for any other VM to use.
For Containers, they all live on a SINGLE kernel. They share resources across each other and the Kernel handles the multiple processes much like it would handle any other multithreaded process.
If you have 3 VMs that all require a specific set of resources to run an application, you need 3x that hardware. This is not true for containers. You can get away with less because the containers will share the resources that the kernel as access to.
I call them "lower level" in the sense that they do so much less than VMs. You CAN use containers as a VM in that a container can boot an entire Kernel, but generally you don't do this.
VMs share resources from the host: disk, network, memory. Just like a container shares resources from the host.
Containers re-use the running Operating System from the host, it saves memory but it can only run a single OS.
A VM can run any operating system, and each VM runs its OS independently. VMs are memory intensive, there is a base 100-500MB to pay to run any VM because of the independent OS. (Note that the advanced VM managers have evolved to have memory deduplication and COW across VMs.)
Containers exist to save memory. That was the critical pain at the time they came to existence. Memory was expensive, it was a major problem when you wanted to run 10 hello world applications as 10 separate VMs.
Containers give the OS the ability to optimally schedule processes among them. VMs are black boxes to the hypervisor, limiting its ability to optimize.
Basically, a VM pretends it has a cpu and a disk and ram, a container pretends that it has a root directory and ports and all of that stuff. Like always it's a leaky abstraction and isn't usually sufficient for security purposes (ie. two containers on the same system might be able to talk to each other if you abuse the system) but it's good enough for most purposes and provides much better resource utilization than a bunch of VMs.
old way: one chroot per user and quota (including ulimit) per user
new way: limits handled with cgroups
security with namespace related to user's profile.
namespace can do syscall limitations, and also offer unique network access per users.
Roughly simplified.
freebsd jails where known as glass jails that would eventually break, LXC/docker are the same except they hav'nt broke yet at the price of more complexity. They will all dramatically break when people will have figured the trick in 5 years.
Simple solution : understand the problem and fix it.
Actual solution: throw more complexity and obfuscation at the problem claiming users are THE problem.
Companies and devops don't care, they have a future of at least 10 years in fat paychecks and stock options.
> They will all dramatically break when people will have figured the trick in 5 years.
VM's hypervisors have also seen bugs/exploits to gain hypervisor access, and yet they're still here. These things have been fixed, and people did not stop using them.
Also, I don't really see the "more complexity and obfuscation" thrown at "the problem", can you enlighten me?
New ways: you create very complex sets of syscalls permissions that can be fine tuned by using a Role Based Accounting or any Auth/Profile Framework.
The idea is that you got rid of the risk of mischief/compromission by containing the code and people by delegating trust to external stuff : companies, datacenters, external servers, an OS you don't own. But for delegating you use software to delegate en masse :)
However, now, your surface of attack is so big that it's impossible to do a full audit of your perimeter. And people focus on code/practices/network. You have delegated a lot.
For the sake of discussion what can be the next cost efficient approach for attack with that much smoke?
The downside of containers is the physical geographical increase of the perimeter to defend, and for SV to develop their beautiful code, a lot of workers (cleaning personal, transporter, electricians, construction workers, firemen) that are required in the physical world and are so impoverished that they are becoming a vulnerability.
Keep It simple, always attack where the costs are less.
Bribing a man today to access a physically a server, a router is less expensive than writing an exploit.
The obvious problem in the containers is the idea you can trust layers you should not. Maybe, your container runs in a datacenter where a worker infected a printer with a connected cam from home because he is to poor to afford a printer? Maybe from there you can compromise a router, and have a MITM on a VLAN used between 2 servers?
Who knows? But how can you know if you cannot check?
> Containers are a lightweight way of sandboxing a process. Think a level lower than a VM.
We tend to use containers and VMs for similar purposes, but i think this the wrong way to explain what they actually are.
A container is like a much thicker-skinned process.
You know how when you run a program as a process, it can't read other processes' memory, and you can control how much CPU it uses with renice, and it can have anonymous files that no other process can read, and you can specifically kill it, and track its resource usage? Containers are like that but more so. With containers, programs can't even read each others' filesystems, and have completely separate network interfaces, and you can measure and control CPU usage for a whole tree of processes, not just one.
Containers are processes beefed up to the level where they compete with running a VM.
> I still can't really figure out what a container is.
If you mean this in a general sense...
One use case at Unbounce (where I worked in infrastructure) was to encapsulate the runtime dependencies for different services that were on a machine.
Our monolith required Ruby 2.1 and a bunch of gems. Then we were using Scout for centralized monitoring, which required 1.8 with a separate set of gems. We only noticed the problem when our monolith moved from Ruby 1.8 to 2.1.
To fix this problem of dual-Ruby runtimes, we encapsulated the Ruby 1.8 + gems into a Docker image for Scout, then ran the Scout container on the machine. It works perfectly and never conflicts with the monolith's Ruby runtime.
Docker is designed around the idea that you only have a single process running in a container. That's not an inherent property of containers though. LXD is a better tool for managing containers that are more like VMs. The kernel is shared between the host and the containers, but they can each have their own userspace. They could each have their own database right inside them, no problem.
I am still on the journey to wholesale container acceptance, but I have been finding more and more use-cases that are delightfully solved by them. My favorite so far is a WordPress hosting platform with some shared infrastructure (web server, caching reverse proxy, and database) but each PHP-FPM instance jailed in its own container. This lets me:
* easily chroot PHP (this is surprisingly difficult otherwise)
* restrict MariaDB access by IP address
* constrain the resource consumption of each application as necessary (i.e. to prevent an out-of-control PHP script from swamping the box)
* independently determine each application's PHP version
And because each managed application is (basically) a Docker image and a Caddyfile, it's easily extensible to non-PHP things. I can feel the lightbulb flickering but I'm not yet at full k8s awareness. The shared infrastructure isn't containerized, but it could easily be, and it's all running on one VM, but it could be distributed across multiple.
Containers don't solve the common problems, they just give you more tools to work with. With databases, for example, you still need to figure out whether each application gets its own database instance? schema? user?, a replication strategy, a failover strategy, a backup strategy, etc. You can use either a bind-mounted host directory or a shared-storage volume for the backing store, just like always, or a newfangled data volume container.
I am more comfortable sharing a database instance between multiple schemas and users because I can do IP-specific grants, but if I wanted to do one per application, I could do that too!
A container is just changes you make to a file system to get something running. To run a container, Docker applies changes listed in its image then executes its entry point. When it stops, changes to file systems are preserved but memory is not. When removed, everything is gone so next time it runs you're starting fresh from original image.
Magic with container are: 1) those changes are only visible from code running in the container, and 2) changes can be layered on top of each other (FROM in Dockerfile).
You're thinking about it incorrectly. Docker is not a VM. Docker is more like a chroot and a set of additional capability restrictions on top. Basically there are several things that are namespaced in Linux. Processes, network, users, IPC, mount, etc. Docker simply manages these namespaces. At a high level, when you fire up a container, a namespace gets created for it. So unless you explicitly tell Docker to expose things from the host, there's only a very limited set of things your container will see. Crucially, everything uses the same kernel, same drivers, etc, and there's zero overhead.
Think of your Linux host as simply a default namespace.
You might want to create your own containers with standard bash commands instead of using docker. You can try cgroups, which are a standard Linux kernel feature.
It's basically just a chroot but with additional levels of isolation that make a process think it's running in its own copy of an OS in addition to running on its own filesystem (as with a simple chroot). So it's similar to the concept of a virtual machine but it "virtualizes" the OS kernel instead of the hardware.
Unix has access control (ie. memory protection, FS access protection, FS root aka chroot).
Containers are process groups with access control.
The actual entity has a different name pretty much everywhere, eg. Solaris Zone, Linux namespace and Linux cgroups. Usually the OS throws in a wider bunch of stuff that is only loosely related to access control in the classic sense, eg. CPU and memory limiting, I/O rate limiting and such (so rusage access control, in a sense).
> The use-case I need solved most often is the following:
Create a standalone "server" that accepts and responds to network traffic, has some way to store data, and whose dependencies (i.e. system packages, frameworks, etc) I can manage independently of any of the other "servers" I have running.
SaltStack, Ansible, Chef are all configuration management tools that serve this purpose. They let you configure a standalone box any way you want.
The only use case I see for Docker over one of them is if you want to run multiple, independent services on one server. But why would you want to do that when you're running in the cloud? I don't see the benefit over separate instances, each tailored to be the exact size you need.
One use that I love is pre-packaged containers. I'm designing a system for a client that does headless web automation. I spent a couple of weeks trying to get various separate versions of selenium/Firefox/Chrome running along with Xvfb without much luck. The selenium project has fully tested and functioning container images that work every time.
> But what about Configuration Management systems? They’re designed to solve this kind of routine by running one command.
The problem with this for most of the developers you see praising containers, is that with a containerized setup, you've already got the rest of your deployment process down to `docker service update --image myorg/myservice:1.3.0 myservice`
(And, in fact, maybe you're even running that code against immutable-infrastructure container-host OS like CoreOS.)
And now, you're suggesting that these developers would have to add this whole other process just for managing the deployment of the DBMS package—and probably the OS it's running on, too. (Maybe they would even have to add process to manage the VM it's running on, if they were doing everything else until now using autoscaling + swarm auto-join.)
Developers put DBMSes in containers because they're developers, not DBAs. If you are a DBA, then obviously this will seem wrong to you. A DBA wants to manage a DBMS using the DBMS's tooling. Developers, meanwhile, essentially want to manage DBMSes as part of the same "release" as their apps—being able to "pin a dependency" to a specific DBMS version; update the DBMS as part of a commit and see the whole updated stack go through CI to integration-test it; etc. These are development-time concerns that—at small scale—usually override operation-time concerns.
This is why I appreciate DBaaS offerings... It makes more sense to run DBs at least closer to the hardware, outside containers, but most developers don't want to be DBAs, so it's better to pay for someone that has all the maint/update scripts written, and not have to deal with many of those issues.
Sure, but then you have other issues with black box databases:
1. Lack of control over configuration and tuning of database.
2. Little control over semantic upgrades, i.e. security updates should be automatic, but one breaking compatibility should not be, but MAY be necessary or desired, so it needs to be possible to upgrade.
3. Failover isn't easy, and the way of handling it is often app- and load-specific.
Granted, this won't matter to many people, but there are a lot of reasons not to entirely outsource database concerns from dev.
Actually they ought to matter to everyone, or rather at some point they do matter. Data is different, data has to be upgraded in place, its genuinely not immutable but this isn't also true of the software above it, Databases are software like any other and they should have a similar release cycle because if they don't its much harder to test and push updates.
If you run database containers on a bare metal docker/container os (coreos/rancher/...), your DB's will probably be running a lot closer to the hardware than they would in a VM.
And what's gonna install CoreOS? And what's gonna install Docker? And what's gonna configure Docker? And what's gonna give a configuration to run the Docker image and how?
This sounds right to me. As a DBA containers appear to be a nightmare. I'm employed as an absolute expert in my product. A developer may know how to use Docker but are they an expert? Now...
* Who is going to look after the middle ground when the database is in the container?
* Who is going to be responsible for rewriting enterprise tools to discover those instances to gather metrics? Because none of the traditional methods (WMI, registry keys, etc) are going to work. You've just broken SCCM, ServiceNow, and everything else under the sun.
* Who owns the patching? Because WSUS can't discover it and isn't going to be able to patch inside a container.
* Who owns the backups? You know backups are complicated right and not just an on/off switch? You have to schedule the backup, but also make sure you're scheduling the backups in a standard way across your hundreds of hosts (now containers), and then validate those backups are actually being taken, and test those backups regularly. Developers couldn't care less about this stuff - it's someone else's problem - my problem - until it's in a container and then nobody is going to do it.
* And when something breaks in between, and the business suffers a massive loss of data, who are they going to sue? A liability-free open source project? I don't think so.
There's more to being a DBA than just stuffing it in a container and saying, "she'll be right mate".
These are all valid concerns, but, none seem specific to databases and - for companies who have moved even parts of their infrastructure to containers - have been deemed acceptable.
(Also, while you mentioned all Microsoft tools here, the same issues apply to Linux based containers)
Most of your arguments comes down to "this doesn't fit in my world where windows is king, so it won't work for me". That however, is not a problem of containers.
While I don't consider myself to be a pure DBA, I do know Postgres quite well, and manage quite a few both "classic" deploys in a VM and containerized instances. I was the one who created a default Postgres setup/image/config that our devs use, which when it's used correctly and as documented, when it is deployed to production, it is exactly the same as managing a normal instance.
For the devs it's simple, their local env is a checkout of a sample env, copy that to their new project, docker-compose up, and they have a database running with pretty much the same config they would get in test, acceptance and production. No surprises, we both know what to expect.
Backups? Still the same. Patches? I tell my config management to a pull a new postgres image on the servers and restart the db images during a maintenance window. This makes it actually a lot easier than updating the non-containerized services.
> and the business suffers a massive loss of data
This scenario should be recoverable in the first place, and should be tested on regular basis. I'm actually setting up a process to automatically verify database recovery using containers, which makes stuff like this a lot easier and more convenient. Spin up container, restore backup to it, full vacuum analyze, pg_check, select counts from every table, select random records from every table, and if possible spin up a test instance of the application (again, very easy if that also runs in a container) where we can run unit tests against the restored database.
> who are they going to sue? A liability-free open source project?
So when have you last heard about someone suing MS or Oracle when they had data loss? I suggest you read your license agreements... Our entire business runs on such "liability-free" open-source projects. Linux, Postgres, GNU userland, Python, GCC, clang, Boost, Wildfly, Java, ... and it worked out pretty well for us. We're not some hipster startup with nodejs, angular and mongodb "cloud" apps, we provide some mission-critical services for clients that are banks, oil companies, governments, ... with corresponding SLA's. The attitude of our (very tech-focused) management is simple: we don't need liability umbrella's when we _own_ the technology and know what the hell we're doing. If something does go wrong, this would mean that yes, we would be responsible, no point in hiding.
If you limit your understanding of "containers" by not advancing past single-page tutorials produced by content marketing folks at orchestration startups, you may _feel_ like the author is right. But as it almost always the case with damn computers and generalized topics, there's no right or wrong. The world is boring and full of "it depends" but that was conveniently left out of the article because the goal, I suspect, was to back a sensationalist title and produce clicks/views. But I'll bite:
> 1. Data insecurity
The author is mixing up Docker image store with database's own data. It is true that Docker graph drivers have issues, but they don't store any data, those are binaries you distribute and you're welcome to start docker containers from a plain old directory on disk. Layers are sexy but optional and they have nothing to do with your database data.
> 2. Specific resource requirements
The author talks about running additional processes on a database machine. Why is this an argument against containers? Maybe because containers make it somehow easier? I dunno... Yeah, don't overload your database servers with other stuff, containers aren't forcing you to do it.
> 3. Network problems
This one is the most bizarre, with statements from all over the map, basically saying "networks are hard". Riding unicycles is also hard, but that's not used as an argument against containers. Here's an obvious conclusion: if you don't feel like learning software-defined networks (or don't need the benefits they provide), then don't use them and run containers with native host networking.
> 4. State in computing environment
This port is just rambling, I do not see anything specific to reply to. If the point to make was that containers don't play nice with state, it's like saying "processes do not play nice with state" because that's what a container is: a Linux process. You have full control over where (pin it to DB machines only) and how it runs, use features you need (and understand) and don't use others.
> 5. They just don’t fit major Docker features
In this part the author is basically saying that it's easy (or easier) to install a database using configuration management tools instead of using something like Docker. True, there is more than one way to skin a cat and frankly you can use both a configuration management system and the containers. I just can't see how this can be used as an argument AGAINST anything.
> 6. Extra isolation is critical at the database layer
The author again claims the containers bring in significant overhead. That's simply not true. I would recommend to mentally replacing "container" with "process" when you read the orchestration blogs to see right through FUD. Again, you can run a container from a directory on your filesystem using host networking and it will be no different from any other process on the box. Using a network namespace does not add any measurable difference to performance. [1]
> 7. Cloud platform incompatibility
The title doesn't match the paragraph of the text that follows. The author basically claims that being provider-agnostic (one of the benefits of containers) is not valuable. Well, he's a database administrator and it's not valuable to _him_. But there's a huge business value of being able to run on different infrastructures: selling $100/mo SaaS subscriptions is nice, but when the stream of early adopters dries up and you set your aim at those nice six-figure enterprise license contracts, you may find out that you will need to be able to run on a VMware cluster in a corporate colo. And containers can help.
Containers are big not because they make developers happy, they're big because they let sophisticated companies significantly consolidate their workloads (via dynamic scheduling) and shrink their infrastructure footprints. I constantly ge...
All system can fail, and failed system should be recovered. That's why we have disaster recovery plan, back up or replication etc.
(I have just shallow knowledge in docker volume, so please reply if not corrects exist)
I understand docker with local volume is just abstracted file system have mounted path that volume specified linked with host path.
So file i/o is probably not a problem in local volume.
And if network has bug that make data corruption can make difference between nodes, docker cannot be used any system. So we can think network bug may not make data corruption(but can make network separation).
Now I am building on-premise autometic deploying software using Kubernetes as a outsourcing job, so I tried to find SAN for resolving stateful data. After many searching, I realize only local path will guarantee stability of database filesystem. So we mark storage node, and all type of stateful app(limited kinds by playform) is deploied on that node. So we can easily back up and manage storages.
As a deployment manager and backup automation, container for database have a great functionality. All file produced by container are jailed where I specified and can be copied or backed up.
(for stabaility, replication is first class. pause-backup-resume or copying on running will make operational unstability. you can use both for backup, replication first and make backup using that replication node)
Of course the counter example of this is Google, which, as I understand it, runs everything on containers. It seems like if its good enough for Google, then its good enough for the rest of us.
They don't use Docker outside Cloud, true, but their container technology is the same as what Docker uses: cgroups. Brought to you by a couple of Google dudes a decade ago.
I make an exception to this.. that would be something like Redis, or memcached as a localized (or perhaps sharded) cache cluster serving the systems running on those docker hosts.
One of the things that bug me in terms of cache as a service in AWS/Azure etc, it you're dedicating compute nodes to mostly use memory... the big win for memcached early on was utilizing unused memory on existing systems. You lose that when you don't have the caching services on the same nodes as compute/data/etc.
We are considering to use Realm to replicate our data out to all our docker instances. They would essentially work like databases local to each instance, and if the instance dies we simply spin up another one which will replicate out the same data again.
So far it is only at the experimentation stage for us, but it looks very promising. It is almost like having the ultimate cache (no network latency) right within each instance.
Alright, all you brave people who run databases in docker: where do you store the actual data?
- Host mounts? So what happens when a container gets rescheduled to another node?
- Docker volumes? What happens when a container gets rescheduled to another node?
- External SAN? Congratulations on your budget. That's not easily doable in public cloud I guess?
- Shared filesystem like NFS, or Ceph? How's the performance for you? And the filesystem itself runs outside of your docker cloud I guess?
- In the containers themselves? So you probably run some clustered database. So what if disaster strikes and all your containers go down? And get moved around?
Also, databases in many cases need to be tuned for performance. How do you do that in a cloud?
Maybe most of you are not running container scheduling. Which really is only taking containers half way.
Host mounts, because they are not rescheduled. If you say that no rescheduling only "takes containers half way", you're not fully aware of how containers are used nowadays. In my company for ex, reproducibility, immutability and a uniform deployment process is key.
I think you have some pretty big misconceptions which accounts for at least half your skepticism. Most databases are designed to restore from information that's stored on disk. If you use host volumes, there is no concept of a container "moving around." Just being dead or alive. If a container starts on a host that was previously running, that container now is the previous one.
I use Linux containers extensively. They can be rolled over by the container orchestration service at any time so I do not use them for databases except in a dev cluster where we trash and reload test decks constantly.
Mount a volume on the host and make sure only one database container runs on that host. Done.
Then it's exactly like a traditional setup except better because if the process crashes or config gets corrupted it can be replaced automatically in a few seconds, good as new.
This is what we do. We use Docker Swarm and label our database host. Then add a constraint to the service/container i our docker-compose file so that the database is always scheduled to that host.
Great thing about this setup is that we can now run the entire production system locally on our laptops and test servers with or without swarm. Everything is deployed in the same way and everyone uses the same versions and configurations (even for our database).
I use ZFS under everything (including docker) and so I can use scheduled snapshots, backups, etc, to do the real data management, and not worry about losing data.
I can use host mounts too, but I prefer to just back up and manage all of docker, out of band, without thinking too much about it.
EDIT: granted this is for relatively small databases though. I think it could be scaled up however.
Mesos with placement constraints solves this problem. In case of AWS, store the data on an EBS volume, use constraints to dedicate agents with specific hardware to the DB containers. Always ensure that EBS is available to the agent on which the DB supposed to run. Easy. Same for Kafka, same for Cassandra, you name it...
Kubernetes lets you do whatever you want. For example, if you tell it to mount an AWS EBS volume into the container, if the container is rescheduled on another host, the volume will be automatically remounted there.
Kubernetes supports a whole range of volume types that all travel with the container, fully managed. You can even ask it to carve out pieces from a larger volume, so you don't have to create one volume per container.
You can pin containers to individual hosts and use host volumes, but of course that rather defeats the purpose of using containers in the first place.
Kubernetes StatefulSets [1] are intended to address this kind of use case. They provide stable network identity and stable storage.
Kubernetes enables a container to declare the resources that it requires, including things like dedicated CPU and memory requirements. There are still some rough edges (example: how do you set the amount of kernel shared memory you need), but those issues are being ironed out.
So glad I didn't read that article before I set up a Cassandra cluster on Docker to handle 1M requests/min in production. It might have discouraged me.
This article is terrible. It's a lot of wishy-washy explanations devoid of technical detail - because their isn't a technical explanation or justification for this list.
I've run extensive benchmarks of Hadoop/HBase in Docker containers, and there is no performance difference. There is no stability difference (oh a node might crash? Welcome to thing which happens every day across a 300 machine cluster).
Any clustered database setup should recover from failed nodes. Any regular relational database should be pretty close to automated failover with replicated backups and an alert email. Containerization doesn't make this better or worse, but it helps a lot with testing and deployment.
Sadly no (and I'd have to clear it with my employer). We're going to be doing some much larger scale testing in the next few months (going from 4-5 nodes to 18) in preparation for the docker rollout on said cluster.
Are you telling me that you'd trust MySQL auto replication failover enough to have it activates multiple times per day (even with Percona)? On a busy cluster with, say, 300GB?
Seems like a straw-man. Why is a process running within a container failing over more frequently than a process running directly on bare metal -- seems like more of a resource/process scheduling issue than anything to do with containers.
Leaving aside there's now yet another abstraction layer to have bugs, it is not a straw-man to imply that current container technology is not as reliable as bare metal.
Or do you mean that Docker etc. are as reliable as bare metal?
I use Docker for systems I design; I also recognize current container technologies have their limitations. It is my job to know and avoid these pitfalls.
This conversation should distinguish between Docker as a product and container technology in general.
A lot of issues that people encounter with Docker specifically disappear if you run Kubernetes (such as volume management), simply by ignoring what Docker does and doing something sane instead.
> Or do you mean that Docker etc. are as reliable as bare metal?
This doesn't really mean anything, Docker the product has a lot issues - sure. Container technology in general? No. Where do you draw the line here? Is a chroot 'as reliable as bare metal'? At what point is a container not running on bare metal anymore?
Of course Namespace in Linux kernel is very mature. But if that's all people use, then there wouldn't be a need for Docker and its extra features- people would still be using LXC (no disrespect to LXC). People have to evaluate the entire software as a whole, instead of just looking at the core technology. I personally feel that Docker is still unproven in terms of maturity. Stateless? Hell yes. Stateful app? Well...
As you said, a lot of problems would disappear if people use Kubernetes instead of Docker. At the same time, a lot of replication problems would disappear if people use PostgreSQL instead of MySQL. My point is, when a novice mixes immature technology with immature technology, he is going to have more issues than what's necessary.
At no time was the claim made that containers were more reliable than any other method of running a process, only that running a process within a container is not inherently less reliable than un-contained.
Unless you've gone out of your way, Docker (as well as other Linux container systems) are just namespacing your process. There's no extra abstraction layer, it's just a more restricted execution environment.
The numerous performance and stability issues (not just in the containers, but also affecting the main cgroup / "host" rather badly) I had with Docker, but never with LXC, which, according to you would be pretty much the same thing - "just namespacing your process". But it isn't.
Docker is ok when it works, hell when it doesn't, has lots of bugs and regularly regresses. I don't understand why you'd run production infrastructure on that and not on any of the alternatives.
> Why is a process running within a container failing over more frequently than a process running directly on bare metal
Because the way to change anything in a container is to kill it and restart it. That's a fundamental difference compared to managing/maintaining a database not in a container.
Unless you've written very poorly behaving software, you kill it by sending it a SIGTERM, and waiting for it to exit. This is true of software both within and outside of containers.
The fact `docker kill` defaults to using SIGKILL instead of SIGTERM is unfortunate, and something one should be aware of before deploying a process with docker, but again, this does not make the process running within the container inherently less reliable.
edit: Looks like `docker stop` does the right-ish thing -- sends a SIGTERM, then only resorts to SIGKILL after a timeout has expired.
Yeah volumes skip unionfs. This article is full of FUD. The author demonstrates they don't really have enough experience to make these claims. I wonder if google has database nodes in containers? Kubernetes is adding the features for containers now. I think it is stable now.
I assume these guys have their own network controller and kickass optic fiber links. Network attached storage in poor cloud environments leads to issues.
Yes stateful sets became beta in k8s 1.5 - it's very much a win to run your test suite against a recent (within seconds) production database container that was spun up by CI. Yes you can do this with VMs but that would take 30 seconds :-)
I've run Oracle in a container, ugh what a pig, though it can be done. It's great for development, since you can checkpoint state, pass it around and have 99 containers of bugs on the ground.
At the end of the day, the data and database is run on some production instance that runs on a fully bare hypervisor at TopGuy (TM) cloud provider. This is enough so that everyone feels more or less good about their situation.
Just because they write a blog on something does not mean they won't sue you for using it (and then offer to drop the suite if you subscribe to buying a cloud license)
I'm going to have to agree with the other reply. One hand doesn't know what the other is doing and the team that does audits to prop up revenue isn't going to care about that blog post unless it has legal language in the license that allows you to bypass licensing restrictions. That said, there's free versions of Oracle's databases for development and they may have exceptions for development purposes so if that's what someone is using containers for it might not be the end of the world.
Yep, licensed Oracle instance for production and docker for developers. If you're the sort to run Oracle, the expectation is that you're going to be paying.
Hadoop and HBase are very different from mysql. Those run on yarn and are designed for containers.
> Any regular relational database should be pretty close to automated failover
In my experience, most people who work with mysql would not enable automated failover. And I believe the concerns in the article are valid and important if considering a container for mysql.
Edit: Though I do think it conflates containers and things like kubernetes or mesos in an awkward way. The good arguments are more about running relational dbs in containers on some sort of cluster orchestration system.
While I agree with you I'd like to caution some users about rushing into dockerising everything in their production environment. If your environment setup is not repeatable and you don't have your configuration management under control then you have other problems and using docker is just going to add another layer of abstraction on your mess that your DBA doesn't know how to deal with when things hit the fan. In particular I can imagine improper understanding of docker volumes could bite some people, but they also have some questionable defaults for networking (user land proxy, rewriting iptables)
That being said we currently use docker for some of our production databases, mainly for almost-idle services (mongodb for graylog, zookeeper for kafka), but I have had no problem using them for some moderately sized services with a couple thousands writes per second on redis/kafka (which is nothing for them).
We're still using non-containerised versions of the databases that needs dedicated bare metal servers mostly because I don't see the risk-benefit being worth it, but I'd love to hear someones war stories about running larger scale databases in docker.
For development, I don't think there's anything better for databases, it beats manual setup, vagrant boxes, and shared development servers by a long shot. I feel that educating everyone on your team in how to use it is well worth the investment. docker-compose makes setting up even a fairly complicated development environment a breeze.
Docker is an app store. Once you reach that point of understanding everything gets easier. On your Windows box or Ubuntu box you can just install whatever you want. Docker is more like IOS or android. When was the last time you edited an .ini file on android? It works, it doesn't, take it or leave it. Or you can clear the local data or reinstall. Not much else. Docker is the same way, at least by intent. No wonder they don't want you to store data on it, if you have a SQL database running on android how much would you expect out of it? Would you really expect it to be persistent? It's easy to install, that's the point, but it takes away a lot of freedom just like the app store(s).
As others have mentioned, Docker doesn't really do any magic which might harm the smooth running of a database, but just leverages process isolation built into the Linux kernel and provides a convenient way to package and distribute bundles of software. In the case of a database, the former can be handy any time you want to run a database on a host where you want to run other software too. As for the latter, being able to run the same configuration locally as on production servers, to replicate configuration over many nodes in a cluster, to distil both configuration and software into atomic units are as advantageous for databases as for any other software.
At my company we have been using PostgreSQL on Docker for over two years without and have been sufficiently satisfied with the results that we're in the process of turning out setup into a product in its own right: http://containable.co/
Docker is a packaging tool. How you chose to deploy your software package has nothing to do with your operational and DB administration procedures. Mixing the two topics is very confusing.
Well, it's both. It's both packaging and deployment. Which is convenient, but probably a mistake. Docker is decent at being a packager, but rather terrible at deploying stuff, which is why we have better, high-level orchestration systems like Kubernetes that handle deployment the way it should be done, and reduce the Docker runtime to a mere container runner.
That is a fair point, and most of the end users we work with are using k8s and mesos to deploy and run the applications. The issue I have with the article is that it assumes that a packaging tool is what defines the rest of your operational procedures. They are two different things, as they always have been in Linux.
There is actually little difference between a process running in a container and in the host. They are using the same network and ideally the same filesystem so there should be no difference whether you run the database or app in the host or the container.
Every container article on HN seems to perpetuate more confusion on containers and often arbitrary misguded rules on what a container should be confusing new users even more.
The problem is Docker has taken fundamental technologies developed by other people and wrapped it and since they do seem to want to give credit and pretend to be more than what they are they obfuscate things in words like docker filesystem drivers, networking drivers, container drivers etc. Untill users get familiar with the underlying technologies this sorry state of affairs will persist.
A container is simply a process launched in its own namespace thanks to kernel namespaces introduced in Linux 2.6. Its got nothing to do with cgroups, cgroups 'can' be used to limit resources to container process by cpu, memory or network resources if you want. If you launch this process by chrooting (or pivot root) into a basic linux rootfs filesystem you have a container. If you launch an init in this process you have a LXC container. If you don't and prefer to launch the process directly from the host you have the Docker version of the LXC container which is a fussy hack as now your container is not contained and runs app process not designed to be run in pid 1 in pid 1 and needs to be managed from the host. Kudos. You can also add a network namespace to the container process so its has its own network layer.
The biggest problem currently is a lot of Linux subsystems are not namespace aware and you can't really do proper isolation. Even cgroups only recently got namespace support. Anyone know who these folks are who are doing all this fundamental work?
The second biggest problem is layers are oversold, their actual practical use is marginal at best. They are also complicated and buggy with multiple issues with running overlayfs or aufs on xfs, databases and btrfs. The third biggest problem is a lot of projects and teams working on Linux containers are pushed into the background or marginalized and misrepresented like LXC was by Docker devs instead of giving proper credit and explanations.
The talented developers of overlayfs, aufs for instance are virtual unknowns in the container ecosystem inspite of Docker fundamentally dependent on them. These guys can solve a a lot of the problems with containers but first users must know about them and support them so that bugs can be fixed, rather than have the Docker team create more workarounds and hacks.
183 comments
[ 4.9 ms ] story [ 781 ms ] threadFor starter, failure modes are not decoupled at all. If one db hits a race condition in the storage driver on one instance, it's usually 100% guarantee that all other instances are sensitive to the same bug and it will happen sooner or later.
In practice, occurrences of a bug are highly correlated and usually happen in batch.
However, if you are going to be running docker on real tin (because thats where the value/speed comes in, if you're on AWS thats a whole 'nother issue) Then you might as well use device mapper for what it was originally designed for: mapping fibre channel. (or iscsi, or SAS [another scsi])
That is assuming you want speed, and have paid enough cash to overcome SPoF in your storage layer (it'll be cheaper and faster than trying to software your way out of it.)
Is that actually the case? Is there a serious risk that a database will be corrupted by a container crash, as the article claims? A regular crash of the computer should not be able to corrupt a database, is a container more dangerous in this regard?
Depending on the customer and the tech involved we'll do blue-green by doing a controlled rolling push of the config or image after it makes it through the dev/test cycle. Also depending on the type of tech we'll store actual data on network or host volumes.
I have run database clusters on kubernetes in production without running into this particular problem.
The current state of container orchestrators for running databases is not optimal because one size does not fit all database types like with stateless applications. One solution for this problem are coreos operators which introduce third party resources into kubernetes that are specific to the database type and contain logic to manage this specific database type on kubernetes.
There are major issues registered with AUFS, Overlay, Overlay2 and BTRFS. It's ridiculous.
Mainly that when disk say that data was written, it really was written, and that the data was written in the right order.
The danger with overlay systems is that they might not provide these guarantees, which makes database writes unsafe. Given that current overlay drivers are unstable as it is, I doubt they put any effort to enforce such guarantees.
Not exactly a database but for example with ZFS it is known that ZFS can't provide guarantees that usually provides if you run it in a VM, unless you have a hardware that supports VT-d (a.k.a. PCI pass-through) and have it enabled.
This is true for any and all file systems. ext4 can't provide those guarantees either because it's up to the hypervisor to do the right thing when the guest requests it to flush caches to disk.
ZFS just happens to have made it very explicit that it requires certain guarantees to provide the data reliability, but those same requirements exist for other file systems.
Corruption occurs on data drives even without docker - you still have to plan for it. This is why you enable replication. This is why you snapshot/backup your data daily and have disaster recovery plans.
There are some major reasons why I actually think running databases in docker containers, even if you are mounting a volume for the data.
1) Development environments can be similar to production. Ensures everyone runs the same version that is running in prod.
2) You don't have to worry as much about what is installed on the host machine.
3) In a clustered setup, it's easier to ensure each node is running the same configuration, version, etc...
One of my issues with all the gripes about docker are the assertions that it causes issues. In all of my time of using docker, 99% of the time when there is an issue it has nothing to do with docker itself. Everyone loves to blame it when things go wrong though.
This article doesn't really back up any of the claims about any of its issues. It just makes blanket statements without backing them up. Don't like docker's networking? Use host networking then.
What people don't think about is the countless issues that will never come up when using containerization. I never have to worry about whether or not python 2.7 is installed on a server that I'm going to deploy a python 3 app on. I also have MUCH higher confidence that if things work on my local development env (which runs the same containers), then there is a high chance it will work in production.
YMMV
We have also been running databases in Docker (on the tb scale though) for around 3 years, we had the odd issue here and there, but nothing terrible and certainly nothing fundamental or resulting in data loss.
If your data is corrupted by a single process dying in an unclean fashion then you have other operational problems.
But putting pb of data in docker three years ago is just insane.
There are still things not clear right now,which are not stable ( not in unstable destroying data) but unstable in: if something goes wrong with your data you need to dig deep and find out what's going on. If stuff changes every few month. You will have a hard time.
Also, enterprise concerns like:
Will there be a docker standard from Google and Facebook? Will AUfs be Version 4 or scrapped again? Will the composer format stay? How to manage runtime upgrades? Ie syncronize 1000 dockers with one version update than another 1000 with another software. So that one update depends on another. Etc etc etc.
Three years ago it just wasn't there. Two years ago I would have said: "good enough" to play around. One year ago it started to get really interesting for enterprise.
That's why I say: you handle pb of data,with docker,three years ago?
Balls of steel.
But with that said, if you have replication and operational workflows set up, you minimize the chance of issues.
If you have mission critical data, you should set up your database to handle failovers of any type, let alone docker.
That's one of the things to dislike. The company and the community are trying to sell it as the best thing since sliced bread and usually forget to assign merit to the kernel developers.
On top of that, its 180k lines of code are unwarranted for a "thin" layer.
Additionally, Docker is pretty handy when you're attempting to manage clusters consisting of thousands of nodes. In that instance enforcing best practices, automating workflows, scaling teams, auditing and preventing configuration drift are much bigger problems than a single server failing.
Does it solve all problems? No. Does it make the world a little better and is it better than monolithic single points of failure? Yes.
This needs to be qualified....
Did you deploy a single system 500 times, or 500 different systems? Or some combination thereof.
By the way, How to have 100 customers => leave right after the design phase every single time. Clients add up quickly.
I hope you write about that somewhere.
> Putting your database inside the container, you’re going to waste your project’s budget. Why? Because you’re putting a lot of extra resources to the single instance. And it’s going out of control. In cloud case you have to launch the instance with 64GB memory when you need a 34. In practice some of this resources will stay unused.
For some software, resource consumption is of "fixed" size, plus temporary workload-dependent growth (e.g. application-layer processes, most of the time.) Whereas some other software will take up all the space available to it (like DBMSes.) The latter are what resource quotas are for.
Containers are not meant to be treated like "Unix binaries but more easy to deploy." Containers are just lightweight VMs that don't have to do screwy things with memory balloon drivers to efficiently pack many of those "fixed plus temp growth" workloads onto a host.
But like VMs, containers still need resource quotas to ensure they don't thrash one-another. You can avoid specifying quotas for your fixed-with-temp-growth workloads, to "oversubscribe" a host, and it'll work (similarly to oversubscribing memory-ballooed VMs.) But the "all the space available" workloads need quotas.
The author might be used to public clouds, where VMs have a "size" in vCPUs + memory and that "size" is charged for, and so might not think of picking an instance size for a VM as explicitly setting a quota. But when you set up your own hypervisor cluster, you still have to decide how big each VM should be, regardless of the fact that a bigger VM doesn't "cost" anything: a VM's "size" is the compromise you make between the needs of that workload, and the ability to "fit" other workloads alongside it on a host.
But, to go further: if you're designing "instances" and running dedicated workloads on them, you're very likely "doing containers wrong." (This is probably a provocative statement; stay with me.)
Containers are to container hosts as VMs are to hypervisors: in both cases, their architecture assumes that if you want resource-efficient deployment, you've got a big generic cluster of hosts, and your guests are loaded onto them using a bin-packing algorithm (taking into account which guests need what extra resources that are only available on certain hosts, etc.)
If you don't have a big generic cluster of hosts, then your only packing options will be necessarily sub-optimal. If your container hosts are real hardware, you're out of luck; if your container hosts are themselves VMs, running on some cloud provider, then costs will be heavily in favor of taking advantage of the cloud-provider's bin-packing by wrapping each of your containers in a separate VM and then deploying those VMs.
(Which is, coincidentally, what Amazon's Elastic Beanstalk does for you, and why it's not the same as Amazon ECS. ECS is for setting up your own "big generic cluster" of container hosts to bin-pack across; Elastic Beanstalk is for wrapping containers in VMs so that AWS will bin-pack at their abstraction level.)
The use-case I need solved most often is the following:
Create a standalone "server" that accepts and responds to network traffic, has some way to store data, and whose dependencies (i.e. system packages, frameworks, etc) I can manage independently of any of the other "servers" I have running. Do I just want a bunch of VMs? Or docker instances that all point to some other DB (that's apparently not in a docker instance...?). But then they're no longer independent from one another because they all use the same DB. So do I need a separate DB for each serverlet? Which lives where? On its own VM?
Containers are a lightweight way of sandboxing a process. Think a level lower than a VM. You can run multiple containers on a single VM in the same way you can run multiple VMs on a single host.
Ideally a container should be stateless. If a container crashes, you should be able to bring it up again without anything actually caring that it is technically a different process.
A container doesn't solve a "real problem" it mostly makes it easier to manage applications and processes by abstracting out any dependencies from the host VM and keeping everything packaged into a single thing.
A container can run any application that it is configured to run on any VM regardless of the state of the VM (Assuming the VM has a kernel that supports containers)
can you go into a little more depth? my understanding of a VM is that it installs the OS in a dedicated memory partition, and allocates hardware resources separately from that of the host machine, such that resource contention between host and VM never happens. the VM allocated resources just go dark for the host machine while the VM is running.
what is a lower level than that? I've understood containers to be thin wrappers around VMs, which would make them higher level, not lower level. do I have this wrong?
But a container can be a single executable, it doesn't have to be an entire OS structure. For example, a lot of the go app containers are just the single executable by itself as a default. Many will rely on a debian/ubuntu base as they want other systems to work. This is mainly because of shared environments/libraries that some non-dockerized systems need, but isn't a requirement.
Every single instance of a VM has its own Kernel. When a VM boots up, it gets allocated a portion of hardware and boots up a kernel and allocates memory to itself. VMs each are isolated from each other in that they don't share resources and each VM is free to do whatever it wants to do with the hardware it is given. Like you said, to the host machine, that hardware is no longer available for any other VM to use.
For Containers, they all live on a SINGLE kernel. They share resources across each other and the Kernel handles the multiple processes much like it would handle any other multithreaded process.
If you have 3 VMs that all require a specific set of resources to run an application, you need 3x that hardware. This is not true for containers. You can get away with less because the containers will share the resources that the kernel as access to.
I call them "lower level" in the sense that they do so much less than VMs. You CAN use containers as a VM in that a container can boot an entire Kernel, but generally you don't do this.
Containers re-use the running Operating System from the host, it saves memory but it can only run a single OS.
A VM can run any operating system, and each VM runs its OS independently. VMs are memory intensive, there is a base 100-500MB to pay to run any VM because of the independent OS. (Note that the advanced VM managers have evolved to have memory deduplication and COW across VMs.)
Containers exist to save memory. That was the critical pain at the time they came to existence. Memory was expensive, it was a major problem when you wanted to run 10 hello world applications as 10 separate VMs.
new way: limits handled with cgroups security with namespace related to user's profile.
namespace can do syscall limitations, and also offer unique network access per users.
Roughly simplified.
freebsd jails where known as glass jails that would eventually break, LXC/docker are the same except they hav'nt broke yet at the price of more complexity. They will all dramatically break when people will have figured the trick in 5 years.
Simple solution : understand the problem and fix it.
Actual solution: throw more complexity and obfuscation at the problem claiming users are THE problem.
Companies and devops don't care, they have a future of at least 10 years in fat paychecks and stock options.
VM's hypervisors have also seen bugs/exploits to gain hypervisor access, and yet they're still here. These things have been fixed, and people did not stop using them.
Also, I don't really see the "more complexity and obfuscation" thrown at "the problem", can you enlighten me?
New ways: you create very complex sets of syscalls permissions that can be fine tuned by using a Role Based Accounting or any Auth/Profile Framework.
The idea is that you got rid of the risk of mischief/compromission by containing the code and people by delegating trust to external stuff : companies, datacenters, external servers, an OS you don't own. But for delegating you use software to delegate en masse :)
However, now, your surface of attack is so big that it's impossible to do a full audit of your perimeter. And people focus on code/practices/network. You have delegated a lot.
For the sake of discussion what can be the next cost efficient approach for attack with that much smoke?
The downside of containers is the physical geographical increase of the perimeter to defend, and for SV to develop their beautiful code, a lot of workers (cleaning personal, transporter, electricians, construction workers, firemen) that are required in the physical world and are so impoverished that they are becoming a vulnerability.
Keep It simple, always attack where the costs are less.
Bribing a man today to access a physically a server, a router is less expensive than writing an exploit.
The obvious problem in the containers is the idea you can trust layers you should not. Maybe, your container runs in a datacenter where a worker infected a printer with a connected cam from home because he is to poor to afford a printer? Maybe from there you can compromise a router, and have a MITM on a VLAN used between 2 servers?
Who knows? But how can you know if you cannot check?
We tend to use containers and VMs for similar purposes, but i think this the wrong way to explain what they actually are.
A container is like a much thicker-skinned process.
You know how when you run a program as a process, it can't read other processes' memory, and you can control how much CPU it uses with renice, and it can have anonymous files that no other process can read, and you can specifically kill it, and track its resource usage? Containers are like that but more so. With containers, programs can't even read each others' filesystems, and have completely separate network interfaces, and you can measure and control CPU usage for a whole tree of processes, not just one.
Containers are processes beefed up to the level where they compete with running a VM.
If you mean this in a general sense...
One use case at Unbounce (where I worked in infrastructure) was to encapsulate the runtime dependencies for different services that were on a machine.
Our monolith required Ruby 2.1 and a bunch of gems. Then we were using Scout for centralized monitoring, which required 1.8 with a separate set of gems. We only noticed the problem when our monolith moved from Ruby 1.8 to 2.1.
To fix this problem of dual-Ruby runtimes, we encapsulated the Ruby 1.8 + gems into a Docker image for Scout, then ran the Scout container on the machine. It works perfectly and never conflicts with the monolith's Ruby runtime.
* easily chroot PHP (this is surprisingly difficult otherwise)
* restrict MariaDB access by IP address
* constrain the resource consumption of each application as necessary (i.e. to prevent an out-of-control PHP script from swamping the box)
* independently determine each application's PHP version
And because each managed application is (basically) a Docker image and a Caddyfile, it's easily extensible to non-PHP things. I can feel the lightbulb flickering but I'm not yet at full k8s awareness. The shared infrastructure isn't containerized, but it could easily be, and it's all running on one VM, but it could be distributed across multiple.
Containers don't solve the common problems, they just give you more tools to work with. With databases, for example, you still need to figure out whether each application gets its own database instance? schema? user?, a replication strategy, a failover strategy, a backup strategy, etc. You can use either a bind-mounted host directory or a shared-storage volume for the backing store, just like always, or a newfangled data volume container.
I am more comfortable sharing a database instance between multiple schemas and users because I can do IP-specific grants, but if I wanted to do one per application, I could do that too!
Magic with container are: 1) those changes are only visible from code running in the container, and 2) changes can be layered on top of each other (FROM in Dockerfile).
Think of your Linux host as simply a default namespace.
For Ubuntu https://help.ubuntu.com/lts/serverguide/cgroups.html
For RedHat https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...
Everything should be easier to understand after that.
See here: https://en.wikipedia.org/wiki/LXC
Unix has access control (ie. memory protection, FS access protection, FS root aka chroot).
Containers are process groups with access control.
The actual entity has a different name pretty much everywhere, eg. Solaris Zone, Linux namespace and Linux cgroups. Usually the OS throws in a wider bunch of stuff that is only loosely related to access control in the classic sense, eg. CPU and memory limiting, I/O rate limiting and such (so rusage access control, in a sense).
SaltStack, Ansible, Chef are all configuration management tools that serve this purpose. They let you configure a standalone box any way you want.
The only use case I see for Docker over one of them is if you want to run multiple, independent services on one server. But why would you want to do that when you're running in the cloud? I don't see the benefit over separate instances, each tailored to be the exact size you need.
2) They let you have burstable instances, but controlling all the services sharing those resources, rather than being subjected to unknown neighbours.
3) Related to the previous points, they let you take advantage of free resources by distributing batch operations over VMs not under peak capacity.
The problem with this for most of the developers you see praising containers, is that with a containerized setup, you've already got the rest of your deployment process down to `docker service update --image myorg/myservice:1.3.0 myservice`
(And, in fact, maybe you're even running that code against immutable-infrastructure container-host OS like CoreOS.)
And now, you're suggesting that these developers would have to add this whole other process just for managing the deployment of the DBMS package—and probably the OS it's running on, too. (Maybe they would even have to add process to manage the VM it's running on, if they were doing everything else until now using autoscaling + swarm auto-join.)
Developers put DBMSes in containers because they're developers, not DBAs. If you are a DBA, then obviously this will seem wrong to you. A DBA wants to manage a DBMS using the DBMS's tooling. Developers, meanwhile, essentially want to manage DBMSes as part of the same "release" as their apps—being able to "pin a dependency" to a specific DBMS version; update the DBMS as part of a commit and see the whole updated stack go through CI to integration-test it; etc. These are development-time concerns that—at small scale—usually override operation-time concerns.
1. Lack of control over configuration and tuning of database.
2. Little control over semantic upgrades, i.e. security updates should be automatic, but one breaking compatibility should not be, but MAY be necessary or desired, so it needs to be possible to upgrade.
3. Failover isn't easy, and the way of handling it is often app- and load-specific.
Granted, this won't matter to many people, but there are a lot of reasons not to entirely outsource database concerns from dev.
* Who is going to look after the middle ground when the database is in the container?
* Who is going to be responsible for rewriting enterprise tools to discover those instances to gather metrics? Because none of the traditional methods (WMI, registry keys, etc) are going to work. You've just broken SCCM, ServiceNow, and everything else under the sun.
* Who owns the patching? Because WSUS can't discover it and isn't going to be able to patch inside a container.
* Who owns the backups? You know backups are complicated right and not just an on/off switch? You have to schedule the backup, but also make sure you're scheduling the backups in a standard way across your hundreds of hosts (now containers), and then validate those backups are actually being taken, and test those backups regularly. Developers couldn't care less about this stuff - it's someone else's problem - my problem - until it's in a container and then nobody is going to do it.
* And when something breaks in between, and the business suffers a massive loss of data, who are they going to sue? A liability-free open source project? I don't think so.
There's more to being a DBA than just stuffing it in a container and saying, "she'll be right mate".
(Also, while you mentioned all Microsoft tools here, the same issues apply to Linux based containers)
While I don't consider myself to be a pure DBA, I do know Postgres quite well, and manage quite a few both "classic" deploys in a VM and containerized instances. I was the one who created a default Postgres setup/image/config that our devs use, which when it's used correctly and as documented, when it is deployed to production, it is exactly the same as managing a normal instance.
For the devs it's simple, their local env is a checkout of a sample env, copy that to their new project, docker-compose up, and they have a database running with pretty much the same config they would get in test, acceptance and production. No surprises, we both know what to expect.
Backups? Still the same. Patches? I tell my config management to a pull a new postgres image on the servers and restart the db images during a maintenance window. This makes it actually a lot easier than updating the non-containerized services.
> and the business suffers a massive loss of data
This scenario should be recoverable in the first place, and should be tested on regular basis. I'm actually setting up a process to automatically verify database recovery using containers, which makes stuff like this a lot easier and more convenient. Spin up container, restore backup to it, full vacuum analyze, pg_check, select counts from every table, select random records from every table, and if possible spin up a test instance of the application (again, very easy if that also runs in a container) where we can run unit tests against the restored database.
> who are they going to sue? A liability-free open source project?
So when have you last heard about someone suing MS or Oracle when they had data loss? I suggest you read your license agreements... Our entire business runs on such "liability-free" open-source projects. Linux, Postgres, GNU userland, Python, GCC, clang, Boost, Wildfly, Java, ... and it worked out pretty well for us. We're not some hipster startup with nodejs, angular and mongodb "cloud" apps, we provide some mission-critical services for clients that are banks, oil companies, governments, ... with corresponding SLA's. The attitude of our (very tech-focused) management is simple: we don't need liability umbrella's when we _own_ the technology and know what the hell we're doing. If something does go wrong, this would mean that yes, we would be responsible, no point in hiding.
> 1. Data insecurity
The author is mixing up Docker image store with database's own data. It is true that Docker graph drivers have issues, but they don't store any data, those are binaries you distribute and you're welcome to start docker containers from a plain old directory on disk. Layers are sexy but optional and they have nothing to do with your database data.
> 2. Specific resource requirements
The author talks about running additional processes on a database machine. Why is this an argument against containers? Maybe because containers make it somehow easier? I dunno... Yeah, don't overload your database servers with other stuff, containers aren't forcing you to do it.
> 3. Network problems
This one is the most bizarre, with statements from all over the map, basically saying "networks are hard". Riding unicycles is also hard, but that's not used as an argument against containers. Here's an obvious conclusion: if you don't feel like learning software-defined networks (or don't need the benefits they provide), then don't use them and run containers with native host networking.
> 4. State in computing environment
This port is just rambling, I do not see anything specific to reply to. If the point to make was that containers don't play nice with state, it's like saying "processes do not play nice with state" because that's what a container is: a Linux process. You have full control over where (pin it to DB machines only) and how it runs, use features you need (and understand) and don't use others.
> 5. They just don’t fit major Docker features
In this part the author is basically saying that it's easy (or easier) to install a database using configuration management tools instead of using something like Docker. True, there is more than one way to skin a cat and frankly you can use both a configuration management system and the containers. I just can't see how this can be used as an argument AGAINST anything.
> 6. Extra isolation is critical at the database layer
The author again claims the containers bring in significant overhead. That's simply not true. I would recommend to mentally replacing "container" with "process" when you read the orchestration blogs to see right through FUD. Again, you can run a container from a directory on your filesystem using host networking and it will be no different from any other process on the box. Using a network namespace does not add any measurable difference to performance. [1]
> 7. Cloud platform incompatibility
The title doesn't match the paragraph of the text that follows. The author basically claims that being provider-agnostic (one of the benefits of containers) is not valuable. Well, he's a database administrator and it's not valuable to _him_. But there's a huge business value of being able to run on different infrastructures: selling $100/mo SaaS subscriptions is nice, but when the stream of early adopters dries up and you set your aim at those nice six-figure enterprise license contracts, you may find out that you will need to be able to run on a VMware cluster in a corporate colo. And containers can help.
Containers are big not because they make developers happy, they're big because they let sophisticated companies significantly consolidate their workloads (via dynamic scheduling) and shrink their infrastructure footprints. I constantly ge...
(I have just shallow knowledge in docker volume, so please reply if not corrects exist) I understand docker with local volume is just abstracted file system have mounted path that volume specified linked with host path. So file i/o is probably not a problem in local volume.
And if network has bug that make data corruption can make difference between nodes, docker cannot be used any system. So we can think network bug may not make data corruption(but can make network separation).
Now I am building on-premise autometic deploying software using Kubernetes as a outsourcing job, so I tried to find SAN for resolving stateful data. After many searching, I realize only local path will guarantee stability of database filesystem. So we mark storage node, and all type of stateful app(limited kinds by playform) is deploied on that node. So we can easily back up and manage storages.
As a deployment manager and backup automation, container for database have a great functionality. All file produced by container are jailed where I specified and can be copied or backed up. (for stabaility, replication is first class. pause-backup-resume or copying on running will make operational unstability. you can use both for backup, replication first and make backup using that replication node)
The concept of containers is fine. The troubles lies in the implementation that is available.
Edit: did publish
Sure, if you have Google's budget and scale.
One of the things that bug me in terms of cache as a service in AWS/Azure etc, it you're dedicating compute nodes to mostly use memory... the big win for memcached early on was utilizing unused memory on existing systems. You lose that when you don't have the caching services on the same nodes as compute/data/etc.
So far it is only at the experimentation stage for us, but it looks very promising. It is almost like having the ultimate cache (no network latency) right within each instance.
- Host mounts? So what happens when a container gets rescheduled to another node?
- Docker volumes? What happens when a container gets rescheduled to another node?
- External SAN? Congratulations on your budget. That's not easily doable in public cloud I guess?
- Shared filesystem like NFS, or Ceph? How's the performance for you? And the filesystem itself runs outside of your docker cloud I guess?
- In the containers themselves? So you probably run some clustered database. So what if disaster strikes and all your containers go down? And get moved around?
Also, databases in many cases need to be tuned for performance. How do you do that in a cloud?
Maybe most of you are not running container scheduling. Which really is only taking containers half way.
>- Host mounts? So what happens when a container gets rescheduled to another node?
Persistent Volumes and Claims. Kube manages moving containers, and the associated storage.
> - External SAN? Congratulations on your budget. That's not easily doable in public cloud I guess?
Specify a storage class on the persistent volume claim. Most providers (Google, AWS, Azure,..) support SSD
> So what if disaster strikes and all your containers go down? And get moved around?
You let Kube handle rescheduling containers onto new nodes.
Then it's exactly like a traditional setup except better because if the process crashes or config gets corrupted it can be replaced automatically in a few seconds, good as new.
There's nothing brave about that.
Great thing about this setup is that we can now run the entire production system locally on our laptops and test servers with or without swarm. Everything is deployed in the same way and everyone uses the same versions and configurations (even for our database).
I.e., you can force scheduler to schedule running on fixed set of machines.
I can use host mounts too, but I prefer to just back up and manage all of docker, out of band, without thinking too much about it.
EDIT: granted this is for relatively small databases though. I think it could be scaled up however.
Kubernetes supports a whole range of volume types that all travel with the container, fully managed. You can even ask it to carve out pieces from a larger volume, so you don't have to create one volume per container.
You can pin containers to individual hosts and use host volumes, but of course that rather defeats the purpose of using containers in the first place.
I'm very interested in approaching this problem on bare metal, though.
Kubernetes enables a container to declare the resources that it requires, including things like dedicated CPU and memory requirements. There are still some rough edges (example: how do you set the amount of kernel shared memory you need), but those issues are being ironed out.
[1] https://kubernetes.io/docs/tutorials/stateful-application/ba...
I've run extensive benchmarks of Hadoop/HBase in Docker containers, and there is no performance difference. There is no stability difference (oh a node might crash? Welcome to thing which happens every day across a 300 machine cluster).
Any clustered database setup should recover from failed nodes. Any regular relational database should be pretty close to automated failover with replicated backups and an alert email. Containerization doesn't make this better or worse, but it helps a lot with testing and deployment.
did you publish your findings anywhere?
Or do you mean that Docker etc. are as reliable as bare metal?
Heck, you can go on Docker's webpage (https://www.docker.com/what-docker). Notice that while they make the claims that Docker is-
1. More lightweight
2. 'Open'/run anywhere
3. Secure
They don't say 'more reliable'.
I use Docker for systems I design; I also recognize current container technologies have their limitations. It is my job to know and avoid these pitfalls.
A lot of issues that people encounter with Docker specifically disappear if you run Kubernetes (such as volume management), simply by ignoring what Docker does and doing something sane instead.
> Or do you mean that Docker etc. are as reliable as bare metal?
This doesn't really mean anything, Docker the product has a lot issues - sure. Container technology in general? No. Where do you draw the line here? Is a chroot 'as reliable as bare metal'? At what point is a container not running on bare metal anymore?
As you said, a lot of problems would disappear if people use Kubernetes instead of Docker. At the same time, a lot of replication problems would disappear if people use PostgreSQL instead of MySQL. My point is, when a novice mixes immature technology with immature technology, he is going to have more issues than what's necessary.
Unless you've gone out of your way, Docker (as well as other Linux container systems) are just namespacing your process. There's no extra abstraction layer, it's just a more restricted execution environment.
Run like this, there is no disk or network performance difference between running the db process directly on the host or via a Docker container.
As others have mentioned, this is a really poor article.
Docker is ok when it works, hell when it doesn't, has lots of bugs and regularly regresses. I don't understand why you'd run production infrastructure on that and not on any of the alternatives.
Because the way to change anything in a container is to kill it and restart it. That's a fundamental difference compared to managing/maintaining a database not in a container.
The fact `docker kill` defaults to using SIGKILL instead of SIGTERM is unfortunate, and something one should be aware of before deploying a process with docker, but again, this does not make the process running within the container inherently less reliable.
edit: Looks like `docker stop` does the right-ish thing -- sends a SIGTERM, then only resorts to SIGKILL after a timeout has expired.
I've been wondering this for a while. I'm sure some of the big players do it, but I'd really like to see a case study from one of them.
Yes
At the end of the day, the data and database is run on some production instance that runs on a fully bare hypervisor at TopGuy (TM) cloud provider. This is enough so that everyone feels more or less good about their situation.
Danger, danger
https://www.theregister.co.uk/2016/02/24/oracle_vmware_licen...
https://blogs.oracle.com/developer/entry/creating_and_oracle...
https://github.com/oracle/docker-images/tree/master/OracleDa...
> Any regular relational database should be pretty close to automated failover
In my experience, most people who work with mysql would not enable automated failover. And I believe the concerns in the article are valid and important if considering a container for mysql.
Edit: Though I do think it conflates containers and things like kubernetes or mesos in an awkward way. The good arguments are more about running relational dbs in containers on some sort of cluster orchestration system.
That being said we currently use docker for some of our production databases, mainly for almost-idle services (mongodb for graylog, zookeeper for kafka), but I have had no problem using them for some moderately sized services with a couple thousands writes per second on redis/kafka (which is nothing for them).
We're still using non-containerised versions of the databases that needs dedicated bare metal servers mostly because I don't see the risk-benefit being worth it, but I'd love to hear someones war stories about running larger scale databases in docker.
For development, I don't think there's anything better for databases, it beats manual setup, vagrant boxes, and shared development servers by a long shot. I feel that educating everyone on your team in how to use it is well worth the investment. docker-compose makes setting up even a fairly complicated development environment a breeze.
At my company we have been using PostgreSQL on Docker for over two years without and have been sufficiently satisfied with the results that we're in the process of turning out setup into a product in its own right: http://containable.co/
e.g. Postgres (https://github.com/docker-library/postgres/issues/37)
Every container article on HN seems to perpetuate more confusion on containers and often arbitrary misguded rules on what a container should be confusing new users even more.
The problem is Docker has taken fundamental technologies developed by other people and wrapped it and since they do seem to want to give credit and pretend to be more than what they are they obfuscate things in words like docker filesystem drivers, networking drivers, container drivers etc. Untill users get familiar with the underlying technologies this sorry state of affairs will persist.
A container is simply a process launched in its own namespace thanks to kernel namespaces introduced in Linux 2.6. Its got nothing to do with cgroups, cgroups 'can' be used to limit resources to container process by cpu, memory or network resources if you want. If you launch this process by chrooting (or pivot root) into a basic linux rootfs filesystem you have a container. If you launch an init in this process you have a LXC container. If you don't and prefer to launch the process directly from the host you have the Docker version of the LXC container which is a fussy hack as now your container is not contained and runs app process not designed to be run in pid 1 in pid 1 and needs to be managed from the host. Kudos. You can also add a network namespace to the container process so its has its own network layer.
The biggest problem currently is a lot of Linux subsystems are not namespace aware and you can't really do proper isolation. Even cgroups only recently got namespace support. Anyone know who these folks are who are doing all this fundamental work?
The second biggest problem is layers are oversold, their actual practical use is marginal at best. They are also complicated and buggy with multiple issues with running overlayfs or aufs on xfs, databases and btrfs. The third biggest problem is a lot of projects and teams working on Linux containers are pushed into the background or marginalized and misrepresented like LXC was by Docker devs instead of giving proper credit and explanations.
The talented developers of overlayfs, aufs for instance are virtual unknowns in the container ecosystem inspite of Docker fundamentally dependent on them. These guys can solve a a lot of the problems with containers but first users must know about them and support them so that bugs can be fixed, rather than have the Docker team create more workarounds and hacks.