Ask HN: Why use Docker and what are the business cases for using it?

122 points by dmarg ↗ HN
I recently did a very basic tutorial on Docker. Afterward, when talking to the tech lead of the project I am working on, he asked me "What are the business cases for using Docker, how could it save us money, how could it make us more efficient?" I told him I would do some research to find the best answers and figured I would turn to you all here at HackerNews for some advice.

Thanks in advance!

Just so you know, I currently work on a project at my company that uses AWS, Codeship and BitBucket for development and production. AWS and Elasticbeanstalk host our application, Codeship runs tests on changes to branches in BitBucket and then pushes the code that passes on certain branches to different AWS environments for Development or Production.

136 comments

[ 3.8 ms ] story [ 294 ms ] thread
I talk about this a bit here:

https://www.youtube.com/watch?v=zVUPmmUU3yY

this was a year ago, so a little out of date. I now work for another company that is into Docker.

I also have various bits on my blog:

http://zwischenzugs.tk

check out the jenkins ones, for example:

http://zwischenzugs.tk/index.php/2014/11/08/taming-slaves-wi...

http://zwischenzugs.tk/index.php/2015/03/19/scale-your-jenki...

There are many ways of using Docker and obviously different companies could come up with their own business cases for adopting the technology. So let me focus on one scenario and we can talk about whether it makes sense for your environment.

Software engineering is often difficult because programmers have to deal with inconsistent environments for development and for production execution of their products. Due to mismatches between these environments, developers often found bugs that surface in one environment but not in another.

Hardware based virtualization (VMWare, HyperV, etc) helped with the inconsistency issue because it enabled developers to create dev/test environments that could later be replicated into production. However this category of virtualization requires more computational resources (esp. storage) than operating system virtualization like LXC used by Docker.

In addition to requiring less resources than hardware virtualization, Docker defined a convenient container specification format (Dockerfile) and a way to share these specifications (DockerHub). When used together, these Docker technologies accelerate the process of defining a consistent environment for both development and production in your application. Dockerfiles are easy to maintain and help reduce the need for a dedicated operations team for your application. In buzzword speak, your team can become more "DevOps".

Docker, by the virtue of relying on a thinner virtualization layer than hardware hypervisors like VMWare, also has higher performance for I/O intensive operations. When used on bare metal hardware, you going to be able to get better performance for many databases in Docker containers compared to databases running in a virtual machine guest.

So to recap, Docker can help you

- maintain consistent dev/test/prod environments

- use less resources than hardware virtualization

- free up the time your team spends on dev to ops handoffs

- improve your app I/O performance compared to running in a hardware virtualization based virtual machine guest

However if you are using AWS, note that Docker Container Service available from Amazon actually doesn't give you Docker on bare metal. That's because Docker Containers run in AWS virtual machine guests virtualized by Xen hypervisor. So with AWS you are paying a penalty in resources and performance when using Docker.

Great, but what are the benefits of running Docker in AWS? You are still running VMs and you are being charged for running them. With Docker you are simply putting yet another layer of complexity, because now you have to run more beefier VMs, you now have problem with network communication between containers running on different hosts. So you will most likely need to use overlay network. You also decrease resiliency, because now when AWS terminated a single VM, all apps running on that node suddenly disappear.

I also don't get the argument about running the same container in dev/test/prod. For example my company is working on going Docker and one of the problem with these environments is that app running there has different configuration. So the idea to solve it is to create three different versions of the same container. Genius! But now are you really running the same thing in dev/test/prod? How is it different to what we did in the past? Especially that before Docker through our continuous delivery we actually were using exact same artifact on machines set up with chef that were configured the same way as in prod, while with Docker now we plan to use three different containers.

>what are the benefits of running Docker in AWS?

I don't see benefits to running Docker in AWS. In my opinion, AWS implemented its Docker-based Container Service very poorly. I advise my customers against using AWS when want to use Docker. There are many bare metal as a service providers out in the marketplace.

>the argument about running the same container in dev/test/prod

Is this issue really caused by Docker because you said that you had consistent environments when built by chef?

Can you name a few bare metal as a service providers?
> In my opinion, AWS implemented its Docker-based Container Service very poorly.

We looked into using it earlier this year.

The web UI was flat-out busted in several ways - they only listed the first 100 security groups, and we have a magnitude more than that. The command line interface was poorly documented, and was missing some of the functionality.

It was a total waste of a week; I wonder if they've fixed any of that.

Would you mind elaborating on some of the issues you see with Elastic Container Service? I'd hoped it would be something like an AWS-specific Mesos, but I haven't looked into it closely.
I installed the Amazon Linux Installation Image (AMI) with docker and Python 3.4 for a recent project. I know ElasticBeanstalk fairly well, but configuring this was a big headache. WASTED LOT OF TIME.

Instead of making life easy, it just added unnecessary burden of learning Docker for the future project members. Documentation is poor, had to hunt for hours for solutions to simple queries.

On AWS, I would suggest you stick with the basic Linux flavour that you know. Use their Docker build only IF you know docker very well.

Robust composition. Instead of needing a separate database, message queue, app server VM, you can run all 3 containers together on one omnibus machine, or separated as scalability needs demand.

Yes, you can achieve something similar using multiple layers in OpsWorks or different deployment schemes with normal Chef, but IMO containers make provisioning and deploying combinations of components easier than most other provisioning and deployment solutions. There's less opportunity for unexpected version collisions and because the network infrastructure is virtual, you can move containers between underlying VMs, allowing capacity planning without substantial reconfiguration.

Your environment-specific config shouldn't be in the container, but described in the environment itself, whether through ENV vars exposed to the container, or a mounted volume of config files. This is a solved problem, in my (admittedly limited, compared to some other commenters) experience.
@takeda, how do you create 3 different containers in Docker for dev/test/prod? What parameters are you changing in the three stages?
- maintain consistent dev/test/prod environments

We're pretty heavily invested in Docker. To this point, it's really nice knowing that all the developers are operating in consistent environments.

An additional bonus is that using Docker makes it really easy to propagate infrastructure changes to the rest of your team for use in development. And, more importantly, know those infrastructure changes are consistent across local dev setups.

As an example, I recently incorporated Sphinx search into a project we were working on. I didn't want to require devs to install Sphinx on their own machines and get it up and running for search to function properly. I also didn't want devs to have the overhead of running Sphinx for search on their local boxes unless they were actively working on something related to search. Basically, I wanted search to be optionally configured to run on startup.

I used a DockerFile to setup Sphinx in its own container, pushed the DockerFile to a Tools source repository we use, and then incorporated the build and running of that container into our startup scripts (just some simple orchestration stuff for the local machine written in bash). Now, if a dev wants a containerized search mechanism they run a simple bash script to build that container, then run another command to spin up our dockerized web app with a connection to a running sphinx search container. We do this for all of our services: mysql, redis, sphinx, the web app itself, and anything else we might need.

As an added bonus, all the Docker CLI work and orchestration of our application is easily hidden behind a shell script. If a dev wants to run the webapp, they simply run: app server dev. If they want the app with search they run: app server dev search from the command line. Developers never need to know what's going on under the hood to get their job done. From their perspective, it just works.

> Software engineering is often difficult because programmers have to deal with inconsistent environments for development and for production execution of their products. Due to mismatches between these environments, developers often found bugs that surface in one environment but not in another.

Docker doesnt solve this, we already ran into problems with different docker client versions, different iptables and other tools used on the host.

Docker does allow you to push the problem away for a while.

I would keep an eye on this project: http://www.opencontainers.org/
Heard about this and seems like everyone and their mother are signing on. This is one of the reasons why I asked the main question is because I want to fully understand what the business case is for using Docker.
Have you ever used provisioning software like Chef to prepare a server to run your software? Have you ever used that in conjunction with Vagrant in order to test out your provisioning and software deployment locally? Docker replaces (or can replace) all of that.
I have not, as I am fairly new to this field but that makes sense in a way if it can minimize the use of other tools to just use the one tool.
Docker does NOT replace configuration management tools like Chef, Puppet and Ansible. Those are still necessary for preparing the host machine which Docker containers will run on. Where Docker does alleviate/reallocate some things is in the configuration of the containers that run on those hosts. Instead of configuring the host for Ruby/Python/etc. you would move that configuration to your Dockerfile. But I think CM tools also have support for generating Dockerfiles, so there's that too.
> Chef, Puppet and Ansible. Those are still necessary for preparing the host machine

In many cases now, they are not. Docker containers can run on CoreOS, which machines are designed to be configured entirely from a cloud-config file, organized in clusters.

With Deis for example, you can build and orchestrate your scalable web service in Docker containers without even writing a Dockerfile, or necessarily knowing anything about how the Docker container is built. The builder creates slugs with the necessary binaries to host your service, and you tell the container how to run itself with a (often one-line) Procfile.

I would still want chef scripts for my database server, but for things that can live in a container on a Fleet cluster, I most certainly do not use Chef, but I absolutely do get reproducible hands-off builds for my runtime environment, and without spending time individually preparing the host machines.

> Docker does NOT replace configuration management tools

You're right, for now, but hopefully not forever.

Docker, the company, has got an insane valuation, so is monetize anything and everything in sight to validate that valuation. But trying to give away tools to developers is an uphill battle on a good day, never mind selling to developers. So Docker is pursuing the end of the market that's likely to pay off - the data center, and that's why there's such a huge push behind putting Docker into production.

Now, my experience with current CM tools is that it's still easier to boot a VM and tinker with it to get it working 'normally' (ie, bash + editing config files directly), and then throw that VM away and play with my chosen CM tool to get it working there instead.

Dockerfiles do a good job of bridging that divide, out of the box. On a fresh machine, I install the base OS, and then install docker, and I'm up and running Dockerfiles, albiet with a cold (docker build) cache.

Hopefully the next generation of CM tools can blend the both so setting up a local target is easy as 'docker build' and ongoing maintenance of deployed machines is as easy as 'puppet agent --onetime'.

Codeship have a great series on Docker for Continuous delivery on their blog: http://blog.codeship.com/

That said I've paged the founders to this thread, they can make the case much more effectively than I can. (disclosure: I don't work for Codeship).

(comment deleted)
Maybe you have looked already and it wasn't useful to you but on the Docker website it has some pretty good marketing to explain its usefulness: https://www.docker.com/whatisdocker

Why Use Docker: "How does this help you build better software? When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software."

Business Case: "...With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker..."

Makes system administration somewhat easier since the host OS can stay the same and docker containers change, while giving developers more control. I have total control of which version of packages is installed, which OS I'm using. I don't have to create a ticket, argue over the ask, and get approval just to change an web server timeout. It sort of usurps the sys admin role, though, which might be a negative. I can move my container anywhere that's running Docker and all packages are there inside of the container. If you spend a lot of time setting up new boxes, that's a plus. Before, I had to dump all packages, figure out which ones were missing, then install all of them, and the host OS had to be exactly the same. Now I know it's exactly the same, all the time, anywhere.

My only warning is that using anything but Ubuntu for your build host is going to take way too long and you're going to be waiting hours for it to complete if you don't have any layers cached.

Side question. I'm well aware of the benefits of docker but, has anybody measured performance degradation due to lack of machine specialization? Back in the web 1.0 days it was common knowledge that you start in 1 server, then you split into 1 app server and 1 database server and you can get 4x the capacity. Did we lose all that with the docker way? Is it not so relevant anymore with modern multicore CPUs?
Docker doesn't force you to put your app and your db on the same image. That is up to you. Most have "App" images and "DB" images separate.

If we want to get really specific, Its also common to see the "DB" image split up between the image of the disk where the data is actually persisted, and the image of the actual DB process. This makes it easy to play around with your data under different versions of your DB.

I absolutely love it (Docker + docker-compose) for creating homogenous local development environments. And if your app needs a MongoDB, Elasticsearch or anything else, it's as easy as adding one line to your overall docker-compose config file to link those services to your app. No need to pollute your development machine, you can just have anything running in Docker containers and share them across your team.

I've created several repos on Github for that matter. Here's for example some boilerplate for running Node.js, Express, Nginx, Redis and Grunt/Gulp inside Docker containers:

https://github.com/b00giZm/docker-compose-nodejs-examples

We have a shitload of servers running CentOS for historical reasons. We can't change the distribution because all the services running on this servers are tight to the quirks and special cases of this distribution. So we need to live with CentOS.

Some of our newer services need a up to date version of glibc and a lot of other dependencies CentOS can't provide. So we use docker to boot up Ubuntu 14.04 containers and run the services with special needs in them.

Another great thing is isolating scripts we don't trust. We allow our customers to run scripts of all kind on our servers --> inside Docker containers. So the customers can't mess with the hostsystem.

>running CentOS for historical reasons

Is CentOS not the state-of-the-art Linux distro to run for servers (besides RHEL for support)?

If you are hosting gameservers, it just does not fit.
Maybe safe-of-the-art. It is stable (and old) which is the way many people like their servers. But, it certainly isn't the latest and greatest.
The entire point of RHEL/CentOS is that it isn't bleeding edge; it will certainly be modern though. I think it's rather unfair to call it "old" though; the latest release was in March.
Fair enough. But the mentality that picks stable over up-to-date tends to never upgrade. I'm stuck supporting rhel5.5, our "new" systems are 6.5
This is what we do. Existing stable CentOS/RHEL environment that we can deploy other OS containers to (specifically Ubuntu 14.04) without having to stand up entire new VMs or metal.
simplest case is that it can serve as a multiple staging environment, when you have loads of APP in a single code base (often startups going about prod-market fit). With docker tech shipping speed= production speed. Without docker you are slowed down by 2x or more.Without docker, either you set up a lot of staging environments, which is not great and costly. Or you use one single testing environment and let each of you tech person wait for hours, wasting time, while QA tries to test one branch and you have another idiot deploying another brand on staging. While there are more complex/useful cases, this is one simple biz value i get out of it for my team.
To tell all the cool kids you are using it.

(Ok, I know there are real use cases for Docker, but I see a lot of hype as well. People telling my mathematician friend that she needs to use docker at the start of her project - it is likely to be a one off graph she needs to produce for a research paper).

There is a big push for reproducibility in science. If you friend can package the process for building that graph in a Dockerfile, it is more likely that readers of her paper will be able to reproduce her results.
or, you know, publish the formula, so readers can reproduce in whatever language / system they want.

Reproducibility is a big push.... but not like you are suggesting. Shipping a dockerfile is the equivalent of saying "This works, if you use this flask, this pipette, this GCMS and this piece of litmus paper"

Docker is not the only solution to problems. It solves some, but you can't tack it on to everything.

Why not both? I am not in academia but I was under the impression that some academics might be publishing 'questionable' results that cannot be reproduced at all in order get their paper count up for tenure review. Not to mention puff-pieces from industry that basically serve as PR in peer reviewed journals without furthering their discipline.

So shipping working code (even if it comes with a required pipette) might be a nice requirement for a peer reviewed publication to take on in order to keep their journal relevant. Shipping in Docker or similar guarantees reproducibility.

If the code is crap, and only works on one particular data set, then putting it in a docker container ain't going to help.
This is an area where a lot of companies are focusing in terms of data science.

As you noted, reproducibility is a huge issue in the scientific community (according to docker users/vendors I've spoke to) to the extent that there are a number of funded startups trying to solve this problem (some using Docker.)

What has been also very surprising is the big companies who have read only copies of analytic data they want to run computations on - sandboxing the data scientists scripts in a container has helped them tremendously in terms of supporting the execution.

Can people elaborate on when it would be better to use a virtual machine and when it would be beneficial to use a container?
Virtual machines have a much higher level of isolation than the LXC used currently for containers. In a container all it takes to get access to the whole system is a privilege escalation exploit. Such exploits are fairly common.
If you have own hardware (e.g. Data Center) that is running your own code that you trust. By going with containers you can pack more applications into the same hardware (less overhead), therefore your costs ate lower.

If you running in AWS, you use VMs anyway so the overhead is there no matter what (and also is not your concern, because you pay for the VMs). By adding Docker there you basically adding one extra layer on top of it, so from the infrastructure point of view you making things even more complex.

If you're familiar with Java, think of a docker container like a WAR or EAR, except it can contain ANY dependency, not just Java code. Database, binaries, cache server, you name it. The implementation is vastly different, but the effect is a deployment artifact that can be configured at build time, and easily deployed to multiple servers.
To get some additional viewpoints on containerization, you could also take a look at what has been said about similar, preceeding technologies:

* Solaris Zones, see also SmartOS Zones based on that

* FreeBSD jails

I'm a sysadmin for a private K-8 school. I use docker because of the ease of deploying and upgrading a large number of tools on a limited number of servers.

I've used puppet for several years to manage our infrastructure, and puppet is still managing all our staff and student laptops, but for servers, I've switched everything to CoreOS + Docker.

I honestly don't know how much those things cost (I have heard some people say AWS is not cheap, but compared to buying your own hardware maybe all of this stuff is very cheap). The point of asking is, my company has not found a clear place to use Docker directly, but we do use it indirectly through the Deis project, and CoreOS.

My experience with Deis has been wonderful. If you ever looked at Heroku but got to the pricing page and didn't look any further, Deis has the same workflow (and much of the same stack, Cedar) as Heroku. The whole thing is built on docker containers, and designed with failover in mind.

I see that Codeship costs a fair amount of money on the higher end of usage; for the cost of a few months on their enterprise subscription, you could probably build your own CI cluster on Deis. CoreOS also targets AWS, and I don't have any idea what your AWS environment looks like, but you could likely build a Deis cluster on AWS just as easily as you could on your own hardware, if not easier.

I try not to think of Docker as an end so much as a means. For me, it doesn't even matter that it's using Docker under the hood, but if you have containerized your application, Deis can work on already built images just as easily as Heroku works on git repositories.

I can't use Heroku for serious things because it costs too much, and we're small potatoes. But I've got plenty of hardware lying around, and some slightly bigger iron that if I'm being honest is probably underutilized, this is based on knowing that it hosts multiple kernels using virtualization, and the only reason those different tasks run on different machines is to keep them nominally isolated for increased maintainability.

Containerization is "virtualization lite." If I can take those services and jobs that all run on their own virtual machines and make them all run on Deis instead (or even just the ones that don't maintain any internal state of their own), I will gain a resource boost by not having to virtualize all of that separate virtual hardware and individual Linux kernels anymore. The marginal cost of another container is lower than a full virtual machine. If it fits into CI, the maintenance cost is lower too, because that's one less individual system that needs to get apt-get upgrade. If we were better at adopting things like chef, this might not be an argument, but for us it still is.

I inherited a lot of legacy stuff. Your situation might not be anything like mine. If you are already drinking the CI kool-aid, you might not honestly have much to gain from Docker that would compel you to invest time and effort into using it to host your apps.

If Deis looks a little complicated, you might check out dokku. Your laptop probably doesn't have enough power to spin up a whole Deis cluster, but you can still get "almost like the Heroku experience" using Dokku, with Docker under the hood providing support. I'm not going to promise you that it will cut your AWS bills in half, but if you did drink the kool-aid, it might be worth checking out just how much of your currently required development infrastructure and outsourced hosting needs can just go away when you add containerization to your developer toolbelt.

Besides just actually running software, I also find it really neat when projects use docker to build their entire application. It provides an effective means of documenting all of your dependencies and making reproducible builds.

Take the docker-compose for example. You can just check the code out, run a single script that builds the project for your environment and everything is pretty much self contained in the dockerfile (https://github.com/docker/compose/blob/master/Dockerfile). You don't have to clog up your host computer with deps and you get an executable plopped into an output bin folder.

Additionally, the steps in the dockerfile get cached so subsequent builds are really fast.

A docker file is a pretty poor way of providing reproducible builds though.

First off there's the FROM line, which can contain whatever opaque image you feel like that already has dependencies inside it, and who knows how they got there or what will happen when it needs to be updated.

Then there's the fact that it's like a script but worse: every line creates a new image, and docker will try to cache the results after each line, but that cache can work against you if you're not really careful (imagine if build systems like make worked that way? No dependency tree, just refusing to execute the first half of your makefile because well, it worked last time so why do it again?

And in practice, you get to find out how many people just put an "apt-get update" in their docker file too. Now our backwards compatiblility is really just equal to Debian's. Hope there's no back ports repos in there or anything that would give a non-backwards-compatible package!

It's certainly possible to use Dockerfiles to create reproducible builds, but it's literally no better than a shell script at doing that. You have all the rope you need to hang yourself and then some.

>making reproducible builds.

Docker builds actually aren't reproducible. There are many sources of non-determinism that Docker cannot address. Do you use the base images from DockerHub as-is or do you run 'apt-get upgrade' or whatever for security patches? If you do, the result you get from building that image (as opposed to using what's in a cache) is different depending on the time it was built. The same goes for any Dockerfiles that compile from source. Hell, just extracting a source tarball results in a different hash of the source tree because of the timestamps on the files. You and I have little hope of building the same image and getting the same exact result.

Build reproducibility is a very interesting topic with some unsolved issues, but Docker isn't helping with it. See https://reproducible.debian.net for a good resource about build reproducibility.

Don't know why you were downvoted. Docker doesn't give you reproducible builds because you're still running in a raw host OS environment with all its state, but simply the subsystems partitioned into their own namespaces. Docker is more akin to a snapshot than reproducible.
our use case for docker is the following:

we're a webshop, and recently we've standardized our stack on symfony2/nginx/postgresql, so all our websites use that. but beside of that we have some that we maintain that need to run on old version of php/centos.

As we have only 1 server internally for pre-staging environments, docker does help us to save a lot of memory/cpu compare to what we had before (virtualbox, yes...), without needing a lot of machine to setup (like openstack).

Also we don't really have a guy dedicated to sysadmin, so the less time we need to spent on server administration, the better we feel. So we have a set of 3 containers (for symfony+php_fpm / postgresql / nginx ) that already tuned to meet our needs, with a ansible playbook https://github.com/allan-simon/ansible-docker-symfony2-vagra... , that we reuse for every new project we have. So that the developers can have a working stack, without needing to reinvent the wheel, they even don't need any knowledge of system adminstration "run this ansible command, done" ! without any risk to break other services.

Also the reproductability and the stateless properties of docker containers has helped to nearly eliminate the class of bugs "the guy that does not work in our company anymore made a tweak one day on the server to solve this business critical bug, but nobody know what it is but we need to redeploy and the bug has reappeared"

I can give you two first hand examples which also revolve about the apsects osipov mentioned:

#1 is in my dayjob:

We use docker in combination with vagrant for spinning up a test environment consisting of several containers with different components (both ur own products and 3rd party) to run integration tests against them.

The main reasons for this approach are: - We can supply containers with the installed producs which saves time on automated runs since you don't have to run a setup every time

- We can provide certain customizations which are only needed for testing and which we don't want to ship to customers without doing all the steps needed for that over and over again

- We have exact control and version how the environment looks like.

- Resources are better distributed than in hardware virtualization environments

#2 Is a pet project of mine, a backend for a mobile App. There are still big parts missing, but in the moment it consists of a backend application which exposes a REST API running on equinox plus a database (in an own container).

The reasons I see for using docker here:

- I have control and versioning of the environment

- I can test on my laptop in the same components as prod, but scaled down (by just spinning up the database and one backend container)

- Since more and more cloud providers are supporting Docker (I am currently having an eye on AWS and DigitalOcean, haven't decided yet), switching the provider in the future will be easier compared to having, say a proprietary image or whatever.

- If I ever scale up the poject and onboard new teammembers, the entry barriers for (at least) helping in Ops will be lower than if they have to learn the single technologies until they get at least basic knowledge of the project.

My apologies if I'm hijacking the original poster.

Does Docker handle multi-environment configuration management? For example: qa, stage and live have the same config files, but different values.

Currently we're using Ansible and we set variables for a specific environment, then we feed those variables into config files based on where we're deploying to (config files are not duplicated, only variables that feed into config files.)

All of our configuration is in git and we can quickly see and change it.

How does Docker handle this?

Absolutely. This situation is handled by using Environment variables. Depending on the environment, I use environment variables to point services at different places (ie dedicated production database server vs my micro dev mysql container).
Remember that environment variables are visible to processes outside the container (i.e. users), if they have the same or higher privileged user. They are not a great place to store passwords or any other confidential information.
The environment of a process is only available to root or the same uid.

    vagrant@monitor:/proc$ sudo -u nginx cat 1779/environ
UPSTART_INSTANCE=runlevel=2UPSTART_JOB=rcTERM=linuxPATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/binRUNLEVEL=2PREVLEVEL=NUPSTART_EVENTS=runlevelPWD=/previous=N

It's not uncommon to allow users to sudo up to particular system users for commands, nor it is uncommon for compromised programs to give the attacker a shell as the user of the compromised program.

Anything owned by that user is vulnerable. A common problem which is typically resolved by reading a config file while root and downgrading to a lower privilege user. For example, you wouldn't want anyone who could become the nginx user to get the SSL key, or the password to your S3 bucket, or...

You do not give the nginx user sudo ability, and any user who has sudo is root, and should be treated as such.
One way I've seen many people tackle this problem is to have the Dockerfile/image built in a more generic way, then the end of the Dockerfile kicks off an Ansible playbook (or some other lite CM tool) that will configure everything for the proper environment (e.g. change configuration and kick off a service, something along those lines).

Some will even go as far as using a CM tool to do the entire internal Dockerfile build, and the Dockerfile is just a wrapper around the CM tool. This does require more bloat inside the Docker image, as you need to have your CM tool or whatever other supporting files/scripts installed in the image, but it does make more complex scenarios much simpler.

> you need to have your CM tool or whatever other supporting files/scripts installed in the image

This pattern is maybe even more helpful than harmful, for making your dev environment more closely match production, when your final deploy target is not a docker container.

(You are obviously going to want to see those build scripts running in test, if not earlier; certainly once, before they should kick off in a production environment.) You could do more individual steps in the docker file, just like you could store your token credentials and database handles in the git repository. Neither way is "completely wrong" but there is a trade-off.

This is not necessarily a part of the Docker specification, but here's a best practice followed by many apps running in Docker containers: http://12factor.net/config
This. Our most recent project was engineered to leverage Docker and Ansible in this manner.

We have a single playbook to deploy everything, i.e. deploy multiple (micro)services, heavily using docker images pulled from a private registry.

With a single playbook, we have multiple Ansible inventory / hosts file for each environment: QA, prod. Sensitive information / secrets are stored in Ansible-vault groupvar files. QA people have ssh access to their own machines, while Prod ops have their own separate ssh access and machines.

The playbook was refactored to heavily use roles, wherein config template files are dynamically setup using inputs from inventory vars and groupvars.

The roles are also topology independent, meaning a QA project cluster can actually be a single big VM with mocked DBs, while the Prod cluster can be spread across multiple machines.

Docker helped simplify the code deployment. Prior to deployment, docker images are built and tested by Jenkins first prior to pushing the images to the registry.

It sounds like you're reinventing a PaaS, which is a road many people go down when they build their devops environment from the ground up.

In the long run it's a bad idea: you wind up with a snowflake PaaS that only you maintain and only you can understand or extend. The amount of engineering effort behind Heroku, Cloud Foundry or OpenShift is enormous and you can get support on a high level.

I'm biased, because I work for Pivotal (who founded Cloud Foundry), but in my view rolling your own PaaS is a strategic error at this point.

Say you're running on CentOS 6.6 (or the equivalent RHEL) and you want to run some software that won't work because you need a newer library than is installed (this recently happened to me recently trying to install Transmission).

You have two choices:

1. Upgrade to CentOS 7.x.

2. Use Docker and install the software into a container using a newer OS (CentOS 7.x or a newer Debian).

#1 is very expensive and sometimes impossible (if you need to be on CentOS 6.x for compatibility reasons).

#2 is very cheap.

There's one of your business cases right there.

Isn't that a business case for containers, rather than Docker specifically? If you want to install an entire OS into a container, LXD is more suitable, surely?
This is our current business case. Multiple CentOS/RHEL 6 systems in a global environment and we want to run an application that requires Ubuntu and newer libraries.

Instead of spinning up new VMs in each environment for one new application, we can instead run a Ubuntu container with the application within the existing environment. This brings with it all the other benefits such as continuous delivery and orchestration that we didn't have before.

Once the platform is established there is no limit what we can run within a repeatable and consistent environment.