Excellent tutorial! I'm a newcomer to Docker, and this was quite helpful. I'm currently using Vagrant to run my local dev environment and it's been OK. But I can see how using containers to split components up can make testing much easier.
One question I have though is how useful is Docker for deploying production applications?
Firstly, can plain Docker be used for the whole setup or is something like Kubernetes required to tie the containers together?
Secondly, is there a performance hit when you use multiple Docker containers on the same server? Or should it be a one-to-one relation i.e. one container per as server? If so, why not just use plain servers with Chef or Puppet for server management?
These questions are assuming that I'm using my own servers - something like DigitalOcean or AWS. I'm not a huge fan of managed container services; it feels like there's too much "magic" going on behind the scenes.
There are a few options for using Docker in production. At the simplest (in terms of least infrastructure at least) you could just have a VM somewhere running Docker, and use docker-compose or similar to bring everything up as you would in development. In the middle you have Amazon Elastic Beanstalk which can take a Docker image and deploy it with minimal hassle, and at the other end of the spectrum you have Kubernetes/Mesosphere/Amazon ECS to allow distributing containers across a cluster of machines.
Except for isolation there's no particular need to run one container per server - while there is a bit of overhead to running in containers my experience is that its offset by being able to run multiple services which don't require a lot of resources alongside each other on one server.
It's similar to elastic beanstalk in that it gives you a great pre-configured AWS architecture. But it's more transparent and configurable.
It also comes with a nice Docker based development environment that mimics production for dev / prod parity.
We have hundreds of production clusters and some at really large scale (300+ containers) so I can confidently say that Docker in production is largely solved on AWS.
In my anecdotal experience with 10 machines and 40 services, half of machines dedicated and half in amazon - i've tried:
k8s - great but for >>10 machines and you need a devops to watch and configure it + every configuration example on site is broken in one way or another. skydns is working for dns. has it's own overlay network.
nomad - great, but doesn't manage dns or network + i didn't get how to save it to file, only upload to nomad. What if it fails? Where cluster config is stored? May be it should be used with Terraform, but i did't like it. Thou for it's purposes it working good. Has concept of long running tasks and can run VM's
helios - i don't remember exactly why but i didn't like it. Something with configuration management. No network or dns.
maestro-ng - this is as raw as it gets. Basically docker compose. I was using it for a while, but it hit it's limits pretty fast. No network or dns.
rancher - after seeing it in action i fell in love. Beautiful UI (you can even view logs and start shell from web ui, and it works great!). Has DNS, manages your overlay network. Store configuration in mysql (i use amazon's rds with MZ). It has it's own perfectly capable scheduler - Cattle. And it can run k8s and mesos if you want it to. I highly recommend OpenVPN package that will connect you to internal overlay network and you will be able to work with services walled from outside world.
RancherOS is separate entity from Rancher. It's their attempt at smallest possible OS for running docker. You may use rancher and never use rancher os.
Said that i personally use RancherOS on t2.micro instances without any problems.
I found the same. It is an all-in-ram distribution, so every docker layer is fully loaded into ram (I think there are 3). The core images must be built on something like Ubuntu. Maybe they have trimmed it down more recently. With something like an Alpine base you could probably get to a few hundred MB ram footprint without any loss in functionality. It would be great to see the bare minimum -- something like busybox+runit/dumb-init. Something with < 100MB footprint would be great for those cheap 512MB vps systems.
Another thing on the k8s configuration guides is that few (none?) of them include SSL authentication. All the ones I tried were entirely insecure, allowing anyone to point the command line client at the cluster and do whatever they pleased. There was also no documentation for fixing this or any warning that the example configurations were entirely lacking in security.
If you want to use k8s, I strongly suggest that you entirely ignore their own documentation and instead use the excellent documentation from CoreOS.
Helios is probably closer to Docker Swarm more than any of the other tools mentioned. It starts containers and isn't very sophisticated beyond that.
We evaluated it for a while, but discarded for various reasons, particularly the fact that it doesn't support private container registries [1].
I'd say Helios suffers from being an internal Spotify project. They're not under pressure to support any use cases except their own. For example, Helios is tightly coupled with Zookeeper, and they have rejected the possibility of supporting anything else (e.g., etcd). Their prerogative, of course.
Rancher is an orchestrator built on Docker. It handles scheduling of containers and certain auxilary objects (load balancers, for example) within that universe. Rancher builds on the Docker Engine and can integrate with things like Mesos and Kubernetes, but doesn't require anything else to get up and running; it comes with its own container scheduler.
Deis is an entire PaaS, designed to handle end-to-end deployment and hosting of apps. It requires Kubernetes to run, which is notoriously complex (even on Google's Cloud) and probably not worth the effort if you have a small number (some people say >10) machines.
Thank you, you really clarified my view on these tools. I don't have experience with K8 and GCE, but when it comes to AWS, it doesn't seem so complex: http://kubernetes.io/docs/getting-started-guides/aws/ but I don't have production experience with this setup either, so who knows ;).
Kubernetes is complex — no matter where you run it. Docker itself is also complex; any orchestration/scheduling system layered on top of it will by definition add a dynamism that involves complicated things happening whenever something needs to be shuffled around.
Kubernetes makes some good design decisions. For example, you can kill the master and the cluster will continue to work just fine, except no scheduling will happen. You get some operational simplicity through how Kubernetes dedicates an IP to each container, too. Several other systems (including Rancher) make the mistake of relying on DNS to abstract the locations of services from clients, but Kubernetes dedicates a single, permanent IP to each service, which is then routed it at the network level. (In case you're curious, DNS is a huge mistake because it's almost never used correctly [1], and there are numerous Rancher issues filed that all stem from this design flaw.)
I wouldn't consider Kubernetes for simple setups, even on GCE, where it's "managed". (Scarequotes because GCE starts a master for you from a prebuilt VM image, but everything else is up to you; the only real UI is the standard one the master runs.) For smaller setups, you might want to consider a simpler solution such as Docker Compose, Docker Swarm or (once it's mature enough) Nomad.
Rancher is nice, but IMHO not ready for prdoctuon; it's much too unstable for an 1.0 release. I set it up to test recently, and within a few minutes I had encountered several serious bugs and several design flaws, including the fact that Rancher's load balancer (which is just HAProxy) loses traffic whenever it's reconfigured.
AWS has the EC2 Container Service (ECS), which is essentially a proprietary Docker scheduler that AWS preinstalls on your VMs. The consensus seems to be that it's one of Amazon's worst products, which my own brief attempt at playing with it confirmed.
Sorry about the long-winded reply, it got out of hand. :-)
For me one of the great things about Docker is that there doesn't necessarily have to be a great difference between dev and production. Both in dev and prod we simply use docker-compose up. Of course with slightly different docker-compose.yml files and Dockerfiles, no HaProxy in dev, no mailcatcher in prod (It's all so wonderfully plugable). Admittedly, I cannot speak for high scale projects with thousands of containers across a cluster of nodes. I guess there you need to have some more elaborate deployment scheme.
I've been using docker for a couple months but I still learned some new stuff. Awesome tutorial! Docker should be required learning in any CS program. I cannot fathom why anyone would setup environments any other way.
Docker isn't the only way, or always even a good way to do things. A better requirement would be to teach people how to write software that isn't a giant mess of dependencies that can't be recreated without the additional complexity of docker.
I admit my wording was not as good as it could have been. I also agree that good engineering fundamental should be taught; however, Docker is a tool like any other. It can be used for both good and evil.
Serious question: have you been through a CS program? In addition to the mathematical and theoretical underpinnings of the discipline, a large part of what is taught is how to engineer software. IMO Docker is a major step forward in software engineering, this is why I believe it should be taught.
>IMO Docker is a major step forward in software engineering
And a lot of people, myself included, disagree. Docker is a huge step backwards because it is covering up the major underlying issues of package management. It's a sledgehammer solution to a problem for which more composable, elegant solutions exist.
Serious answer: a good CS program does not waste time teaching a particular use of a particular technology that may or may not be in use by the time the student has graduated. Instead, it ensures that the student has the base knowledge and research technique to learn that technology in a weekend when/if he needs it.
Docker is a (useful) compromise to help people deal with unmanageable, sloppy software. If students learn how to write simple, portable software, the need for compromises like Docker is reduced.
Docker is one of the way to bind infrastructure dependencies with a version of software, which means you can be consistent across nodes, run any version of your software without suffering dependency mismatches, and it ensure dev, qa and prod use the same (version of) dependencies. Finally it makes it easy to always run the latest stable versions of your dependencies.
Docker is one way to manage deployment better REGARDLESS of the quality of your software.
Functional package managers solve the same problems, but do it better and come with additional features like unprivileged package management, transactional upgrades and rollbacks, and bit-reproducible builds.
OP Here: Wow, people really took my comment the wrong way. Learning how to set up reproducible stacks is a skill that would greatly benefit students. How would it not?
I wasn't trying to say that anyone who doesn't use Docker is an idiot; if that's how it came across, I'm sorry.
Many schools have separate software engineering majors/disciplines. When doing my MS in CompSci, there was only one required class on software engineering. I can imagine that other schools would not have that requirement at all.
I didn't actually intend for my response to sound snarky, even though as I read it now, it probably does.
I think CS degrees should teach understanding (not practice) that will be useful to you for at least, say, the next 50 years if you are a computer scientist. Trendy tools and languages we can learn ourselves.
What might be useful in a Software Engineering degree IMO is learning to do the thing yourself (e.g. creating a chroot jail for an app you developed the previous semester), together with a short tour of tools that already do that (Docker or whatever is popular at the time can make a good case study.) However, for a CS degree where you have only 1-2 SE-related subjects, I believe such a thing would just get in the way.
Awesome tutorial! Love the deep dive into building the entire service and using compose to get it running. If you're interested in how to deploy your docker apps online, I wrote a tutorial which some of you might find useful - http://prakhar.me/docker-curriculum/
> What you are seeing is the bash shell of an isolated container running Ubuntu, on your machine. It's yours to place with - you can install things on it, run software, whatever you want.
Nitpicking a bit on security, but root in a container is still root on the host right?
Yes. You're depending on the isolation provided by the cgroups to keep root inside it's cage. It's one reason there was a problem a year or so ago with root users inside a container being able to access the kernel keyring for processes outside the container.
IIRC, there are no known breakouts with the latest version of Docker and the default settings are oriented towards preventing them where possible. The concern is largely around having privileges escalated by default if a breakout were attained.
Recently docker introduced a --userns-remap flag for the daemon which allows interested users to define a user that "root" in containers should be mapped to. And it will also create one automatically if the flag is invoked without specifying one explicitly.
It's not on by default because of backwards compatibility issues. For one thing, bind mounts as most users rely on them today would likely break completely. But the option is there.
For those learning, make sure you look into removing unused images and containers otherwise you might run into some issues a little farther down the road (eventually our builds started failing).
You can run docker rm and docker rmi to remove specific containers and images respectively, but we also found passing the --rm flag with docker run helped a lot as it removes a container after it exits.
Presumably because they ran out of space on the host that was doing the building. It's very easy to never see the gobs of containers and images no longer in use with docker.
I run into this pretty frequently with Docker on Mac OS X. Passing the `--rm` flag helps, but for cleaning up old/unused containers, I usually set these two commands on a cron:
# Remove un-tagged or un-named images
for im in $(docker images | awk '/<none>/ { print $3 }'); do docker rmi $im; done
# Remove stopped containers
docker rm $(docker ps -a -q)
> We might not want to run our production database in Docker (perhaps we'll just use Amazon RDS for example), but we can spin up a clean MySQL database in no time as a Docker container for development - leaving our development machine clean and keeping everything we do controlled and repeatable.
Curious why you wouldn't want to run it in docker in production? Wouldn't running the same container in dev/prod be helpful?
AFAIK (last I looked into this was ~6ish months ago), scaling databases in docker containers is a hard problem that currently doesn't have a clean solution. Vitess[0] is a solution, but it didn't seem quite ready for production last I looked at it.
What is there to gain by putting your database in a container? A database probably has its own dedicated server so, your probably not deploying databases often and there's the added complexity of making your data persistent.
I run my production database in a container, even if it has its own dedicated server. It's nice being able to install, start and initialize the database with a simple docker command or two to get a consistent build, instead of manually setting it up (copying/editing configuration files, deploying schemas, initializing data, setting permissions, etc.) And it makes upgrading the database very easy. Having this process automated and in source control is great. I also have the awesome ability to bundle cron in my database container to schedule regular backups. Persisting data is as simple as mounting a data volume.
This is a very good tutorial! Love the end-to-end examples. Great teaching method as it gives an overview of what you want to achieve without looking at things in isolation. Wanted to add one bit of info and give a heads up about the Docker for Mac & Windows beta at https://beta.docker.com/.
> If you are on Mac or Windows, consider using a Virtual Machine. I use Parallels on Mac OS X to run an Ubuntu machine for most development activities. Being able to take snapshots, break things and then revert back is very handy when experimenting.
The beta will make is super easy to use Docker on Mac & Windows without running a heavy Virtual Machine. Here's a little video about it @ https://blog.docker.com/2016/03/docker-for-mac-windows-beta/. Basically, it feels like running Docker natively, has upgrade features build it, so you can choose to always be on the latest release (super nice). I've been using it for a few months and it is awesome to just type "docker run" on my macbook and have it work seamlessly (vs using third party virtual machine software and env bits). The idea is that you can develop on Mac & Windows using the same images you will eventually deploy. It just works (although I'm biased).
Sorry, I'm mostly up on the Mac beta (so I'm not a Windows expert), but did a little digging through the beta docs and forums and found a thread about it.
> Unfortunately, Docker for Windows will only run on new versions of Windows 10. This is because Docker for Windows uses Windows Hyper-V and while older Windows versions have Hyper-V, their Hyper-V implementations lack features critical for Docker for Windows to work. [0]
I know this isn't what you want to hear but that is the current status.
I'm trying to get on to the beta program, but it's almost 2 months and haven't received any invite. Do you have a spare invite that you could share with me?
Does anyway know if there's an easy to encrypt a docker volume in a dev environment?
So basically, I'd start a docker database container, it'd ask for a password and then decrypt and start the container and the database. When I shut down the machine and leave the office, I can then be sure the data in the container isn't readily available if someone breaks in a take the computer.
[1][2]
...instead is a whole bunch of hacking on a whole bunch of things... One would think that with as much as SmartOS has been discussed here on Hacker News, people would finally realize there is something simpler out there without all this pain, but apparently not. Crazy!
Clearly those two commands can't by themselves know how to set up a database; I'm assuming it's in the JSON file, but then not showing it is somewhat misleading. It would certainly not look as clean if you were to include a big "echo ... > vm.json".
Also, the database setup is only the first part, it then goes on to show container linking and etc.
With SmartOS, the database image can be preconfigured and published into the catalog, so when the image is pulled down with imgadm(1M), the database is already ready to take data and serve.
The complex startup he then continues to hack on is superfluous on SmartOS, as SmartOS has the service management facility for that. For brand new services, one XML file and svcadm import later, the thing is ready to be turned into an image, never to have to be revisited again.
I do not understand why people take the harder way to do the same thing (if there is no clear advantage to doing it the harder way). He picked MySQL, a database which is infamous for data corruption and bugs where basic SQL functionality does not work; the substrate, Ubuntu, provides no fault management architecture, no end-to-end data integrity, no post mortem kernel debugging and dynamic tracing facilities, which will eventually result in incidents in the dead of the night. Then there is the choice of Docker, which is really not designed to run a fully functional virtual server, but a single process in a single image (although it can run something akin to a virtual server, with lots and lots of extra system engineering put in). Why choose Docker, when with zones on SmartOS one can have a fully isolated, fully functional UNIX virtual server, but running at the speed of bare metal, and all of that in a single invocation of vmadm -f vm_definition.json (or use the SDC software, which puts a GUI on top of it all)?
If something more reliable and easier is available, why insist on doing it the harder way with an unreliable substrate? There appears to be no logic here, and if there is any, it escapes me. Is it lack of awareness of SmartOS, FMA, SMF, ZFS, and zones, or something else?
With SmartOS, the database image can be preconfigured and published into the catalog, so when the image is pulled down with imgadm(1M), the database is already ready to take data and serve.
So can with Docker. The author simply chose to use a pre-built image instead.
The complex startup he then continues to hack on is superfluous on SmartOS, as SmartOS has the service management facility for that. For brand new services, one XML file and svcadm import later, the thing is ready to be turned into an image, never to have to be revisited again.
In Docker that's called a Dockerfile. It does exactly what you describe. Again, the author simply didn't do that.
He picked MySQL [and] Ubuntu
It's an example, does it really matter? Maybe the author is simply more familiar with it.
Why choose Docker, when with zones on SmartOS one can have a fully isolated, fully functional UNIX virtual server, but running at the speed of bare metal
The fully functional containers you've described have been available for Linux for years before SmartOS was first released, first as OpenVZ and then as LXC. Docker's approach is not a technical limitation, it's a deployment philosophy. They don't want fully functional UNIX servers, they want disposable stateless containers that can killed off and cloned.
If something more reliable and easier is available, why insist on doing it the harder way with an unreliable substrate? There appears to be no logic here, and if there is any, it escapes me. Is it lack of awareness of SmartOS, FMA, SMF, ZFS, and zones, or something else?
I suggest you then read up on Docker and the motivations behind its creation and popularity before suggesting that everyone in the thread is illogical and ignorant, because - and I say this as someone who doesn't really like Docker - your advocacy for SmartOS has a vibe of the infomercial from Family Guy: https://www.youtube.com/watch?v=5ysMt_9JnyI
> In Docker that's called a Dockerfile. It does exactly what you describe.
Actually it does not, as SMF is part of the fault management architecture, which is part of the Solaris self-healing technology: unlike the Docker file, SMF will watch over services and restart them if necessary, if configuration files are specified as dependencies and they change, SMF will call the service's refresh method. Docker is just a delivery mechanism, whereas SMF is part of something bigger and far more powerful. SMF ties into FMA, which actually watches over not just software, but hardware down to the individual chips in the memory slots (when not running on virtual hardware). Even on zones, SMF provides the benefit of uniform diagnostics. Docker isn't designed for any of that, which is why he had to hack on the microservice setup.
> fmadm
Usage: fmadm [-P prog] [-q] [cmd [args ... ]]
fmadm config - display fault manager configuration
fmadm faulty [-afgiprsv] [-u <uuid>] [-n <max_fault>] - display list of faulty resources
fmadm flush <fmri> ... - flush cached state for resource
fmadm load <path> - load specified fault manager module
fmadm repaired <fmri>|label> - notify fault manager that resource has been repaired
fmadm acquit <fmri> [<uuid>] | label [<uuid>] | <uuid> - acquit resource or acquit case
fmadm replaced <fmri>|label - notify fault manager that resource has been replaced
fmadm reset [-s serd] <module> - reset module or sub-component
fmadm rotate <logname> - rotate log file
fmadm unload <module> - unload specified fault manager module
That up there is science fiction in Docker. As is something like this:
> svcs -xv bacula
svc:/site/bacula:sd (Bacula storage daemon)
State: online since Sat Apr 23 06:54:00 2016
See: man -M /opt/abcd/share/man -s 1M bacula-sd
See: /var/svc/log/site-bacula:sd.log
Impact: None.
svc:/site/bacula:fd (Bacula file daemon configuration)
State: online since Sat Apr 23 06:54:00 2016
See: man -M /opt/abcd/share/man -s 1M bacula-fd
See: /var/svc/log/site-bacula:fd.log
Impact: None.
svc:/site/bacula:dir (Bacula host specific configuration)
State: online since Sat Apr 23 06:54:00 2016
See: man -M /opt/abcd/share/man -s 1M bacula-dir
See: /var/svc/log/site-bacula:dir.log
Impact: None.
> The fully functional containers you've described have been available for Linux for years before SmartOS was first released, first as OpenVZ and then as LXC.
Actually no, since the "containers" are zones in SmartOS, and I've been using zones since 2006, when they first came out in Solaris 10 update 2. This was before OpenVZ and LXC.
Zones gave me virtual UNIX servers. Solaris containers gave me caps on those servers. Both came in one package back in 2006. SmartOS, being a freeware open source Solaris derivative leverages all that technology, plus a whole bunch more which neither Solaris nor GNU/Linux nor Docker have. Designed for exactly what the author of the original article was trying to hack up together, minus all the hacking.
Let me tell you, you are not doing smartos any favors.
People don't give a shit about how great you think smartos is. "smartos can do this", "smartos can do that". blah blah blah.
Do you know what the difference is between the posted link and your comments? The stuff in the article actually works, while your "look how simple it is" comment, does not:
Compared to the process outlined in the article that works, and shows you how to setup the entire process to, build, rebuild, and run two separate containers.
Their first command is basically
$ docker run -it --rm ubuntu
root@37ffd019060f:/#
Which anyone with docker installed can run. Your first command returns an error message.
And then you go on to say
> With SmartOS, the database image can be preconfigured and published into the catalog
> one XML file and svcadm import later, the thing is ready to be turned into an image, never to have to be revisited again
> a single invocation of vmadm -f vm_definition.json
So all you have to do is preconfigure things (how?), publish them into the catalog (whatever that is) and write both an xml and a json file (contents unknown) to start your service.
If you actually cared about getting people interested in smartos, you would write a "learn-smartos-by-building-a-microservice" article and show how to actually use the fucking thing instead of ranting how terrible docker is.
> Do you know what the difference is between the posted link and your comments? The stuff in the article actually works, while your "look how simple it is" comment, does not:
The usage I provided is a SYNOPSIS, as in synopsis from a manual page. You will note that mine is a comment, not an entire blog article (I do not have a blog).
The intent behind my comment is to provide a pointer to further exploration; it is not a how to document, but rather a starting point for those who are actually willing and able to sit down and read some documentation, which is in the links I provided, and will provide the equivalent of what the author's article does, and dividends on top of that. (Less incidents in the dead of the night equals long term dividends.) My intent was that someone will read it and go off to explore it.
That is not to say that I do not agree with your point, as I myself am a firm believer in examples which work. However, one must at some point have the experience to distinguish between a synopsis and a guide. A lone comment on Hacker News is not the place for an entire how to guide.
> Which anyone with docker installed can run.
Yes, but as with anything, there will be long term consequences: while anyone can get it up and running easily, when it comes time to put it in production, and the thing busts because there is no lifecycle management, no change management, no configuration management (sorry: dumping a bunch of files into a Docker "image" definition doesn't even come close to lifecycle management, or capability maturity model level 2[1]), no post mortem debugging facilities, no end-to-end data integrity, and performance penalty for running inside of a fully virtualized VM, the problems will start.
For example, MySQL was also "easy to set up", but that didn't change the fact that the software does not work correctly[2] and that MySQL deployments later caused many organizations many problems. Just because something is easy to set up does not mean it is a good solution to deploy. Nowadays people are realizing that and migrating to PostgreSQL because that software does work correctly, but all that time, effort and money spent because of how "easy" MySQL was to get going could have been prevented in the first place. With Docker and how it is designed, and based on what I know, I can see the same types of problems which will inevitably lead to outages.
And in the interest of disclosure, what is it to me? Well, argumentum ad populum is extremely spread these days, which means that lots of jobs would (and have) started springing up with Docker, which means I would be forced to work with what I know from experience is inferior technology. Since I see the problems of Docker, that means I would not be able to sleep through the night, and spend my nights in conference calls with clueless managers demanding that I fix it. I do not want that; I want to sleep through my nights, and work with slick and reliable technology like SmartOS, not a toy like Docker causing me problems down the road, and just because it was popular.
You are making an indirect argumentum ad populum, which is never a good thing.
> If you actually cared about getting people interested in smartos, you would write a "learn-smartos-by-building-a-microservice" article and show how to actually use the fucking thing instead of ranting how terrible docker is.
If you had bothered to read the documentation in the links, you would have a pretty good idea on "how the fucking thing works", and how to build a microservice; I am sorry if you think that clicking on the links and reading through the articles is too much overhead. How is it then, that reading through that Docker article isn't overhead, and you had no problem clicking at that?
For those following along, I've noticed a few errors/typos, and maybe others in the thread caught them...
* Container ports are exposed like so, HOST_PORT:CONTAINER_PORT, or YOUR_MACHINE:DOCKER_VM. If you already have a mysqld instance running, then try using 3308:3306
* The /search route needs a "next" parameter in users.js:
86 comments
[ 3.1 ms ] story [ 60.7 ms ] threadOne question I have though is how useful is Docker for deploying production applications?
Firstly, can plain Docker be used for the whole setup or is something like Kubernetes required to tie the containers together?
Secondly, is there a performance hit when you use multiple Docker containers on the same server? Or should it be a one-to-one relation i.e. one container per as server? If so, why not just use plain servers with Chef or Puppet for server management?
These questions are assuming that I'm using my own servers - something like DigitalOcean or AWS. I'm not a huge fan of managed container services; it feels like there's too much "magic" going on behind the scenes.
Except for isolation there's no particular need to run one container per server - while there is a bit of overhead to running in containers my experience is that its offset by being able to run multiple services which don't require a lot of resources alongside each other on one server.
http://github.com/convox/rack
It's similar to elastic beanstalk in that it gives you a great pre-configured AWS architecture. But it's more transparent and configurable.
It also comes with a nice Docker based development environment that mimics production for dev / prod parity.
We have hundreds of production clusters and some at really large scale (300+ containers) so I can confidently say that Docker in production is largely solved on AWS.
k8s - great but for >>10 machines and you need a devops to watch and configure it + every configuration example on site is broken in one way or another. skydns is working for dns. has it's own overlay network.
nomad - great, but doesn't manage dns or network + i didn't get how to save it to file, only upload to nomad. What if it fails? Where cluster config is stored? May be it should be used with Terraform, but i did't like it. Thou for it's purposes it working good. Has concept of long running tasks and can run VM's
helios - i don't remember exactly why but i didn't like it. Something with configuration management. No network or dns.
maestro-ng - this is as raw as it gets. Basically docker compose. I was using it for a while, but it hit it's limits pretty fast. No network or dns.
rancher - after seeing it in action i fell in love. Beautiful UI (you can even view logs and start shell from web ui, and it works great!). Has DNS, manages your overlay network. Store configuration in mysql (i use amazon's rds with MZ). It has it's own perfectly capable scheduler - Cattle. And it can run k8s and mesos if you want it to. I highly recommend OpenVPN package that will connect you to internal overlay network and you will be able to work with services walled from outside world.
Said that i personally use RancherOS on t2.micro instances without any problems.
If you want to use k8s, I strongly suggest that you entirely ignore their own documentation and instead use the excellent documentation from CoreOS.
We evaluated it for a while, but discarded for various reasons, particularly the fact that it doesn't support private container registries [1].
I'd say Helios suffers from being an internal Spotify project. They're not under pressure to support any use cases except their own. For example, Helios is tightly coupled with Zookeeper, and they have rejected the possibility of supporting anything else (e.g., etcd). Their prerogative, of course.
[1] https://github.com/spotify/helios/issues/462
Rancher is an orchestrator built on Docker. It handles scheduling of containers and certain auxilary objects (load balancers, for example) within that universe. Rancher builds on the Docker Engine and can integrate with things like Mesos and Kubernetes, but doesn't require anything else to get up and running; it comes with its own container scheduler.
Deis is an entire PaaS, designed to handle end-to-end deployment and hosting of apps. It requires Kubernetes to run, which is notoriously complex (even on Google's Cloud) and probably not worth the effort if you have a small number (some people say >10) machines.
Kubernetes makes some good design decisions. For example, you can kill the master and the cluster will continue to work just fine, except no scheduling will happen. You get some operational simplicity through how Kubernetes dedicates an IP to each container, too. Several other systems (including Rancher) make the mistake of relying on DNS to abstract the locations of services from clients, but Kubernetes dedicates a single, permanent IP to each service, which is then routed it at the network level. (In case you're curious, DNS is a huge mistake because it's almost never used correctly [1], and there are numerous Rancher issues filed that all stem from this design flaw.)
I wouldn't consider Kubernetes for simple setups, even on GCE, where it's "managed". (Scarequotes because GCE starts a master for you from a prebuilt VM image, but everything else is up to you; the only real UI is the standard one the master runs.) For smaller setups, you might want to consider a simpler solution such as Docker Compose, Docker Swarm or (once it's mature enough) Nomad.
Rancher is nice, but IMHO not ready for prdoctuon; it's much too unstable for an 1.0 release. I set it up to test recently, and within a few minutes I had encountered several serious bugs and several design flaws, including the fact that Rancher's load balancer (which is just HAProxy) loses traffic whenever it's reconfigured.
AWS has the EC2 Container Service (ECS), which is essentially a proprietary Docker scheduler that AWS preinstalls on your VMs. The consensus seems to be that it's one of Amazon's worst products, which my own brief attempt at playing with it confirmed.
Sorry about the long-winded reply, it got out of hand. :-)
[1] http://nerds.airbnb.com/smartstack-service-discovery-cloud/
http://ianmiell.github.io/docker-101-tutorial/
Where is the Science in using stuff other people have put together (and then written a detailed instruction on how to use it)?
And a lot of people, myself included, disagree. Docker is a huge step backwards because it is covering up the major underlying issues of package management. It's a sledgehammer solution to a problem for which more composable, elegant solutions exist.
Docker is one way to manage deployment better REGARDLESS of the quality of your software.
Are talking about something like NIX?
How do you handle other type of installs such as python pip?
Also, could such a package manager be used to deploy to clusters?
I'm not being facetious, I'm trying to understand your rationale for using a PM over Docker for similar use cases.
I wasn't trying to say that anyone who doesn't use Docker is an idiot; if that's how it came across, I'm sorry.
Nitpicking a bit on security, but root in a container is still root on the host right?
Recently docker introduced a --userns-remap flag for the daemon which allows interested users to define a user that "root" in containers should be mapped to. And it will also create one automatically if the flag is invoked without specifying one explicitly.
It's not on by default because of backwards compatibility issues. For one thing, bind mounts as most users rely on them today would likely break completely. But the option is there.
For those learning, make sure you look into removing unused images and containers otherwise you might run into some issues a little farther down the road (eventually our builds started failing).
You can run docker rm and docker rmi to remove specific containers and images respectively, but we also found passing the --rm flag with docker run helped a lot as it removes a container after it exits.
Happy learning!
Curious why you wouldn't want to run it in docker in production? Wouldn't running the same container in dev/prod be helpful?
[0] http://vitess.io/
Given that, it seems production ready.
My curiosity was more in whether there were performance (or other) bottlenecks which made it technically infeasible.
> If you are on Mac or Windows, consider using a Virtual Machine. I use Parallels on Mac OS X to run an Ubuntu machine for most development activities. Being able to take snapshots, break things and then revert back is very handy when experimenting.
The beta will make is super easy to use Docker on Mac & Windows without running a heavy Virtual Machine. Here's a little video about it @ https://blog.docker.com/2016/03/docker-for-mac-windows-beta/. Basically, it feels like running Docker natively, has upgrade features build it, so you can choose to always be on the latest release (super nice). I've been using it for a few months and it is awesome to just type "docker run" on my macbook and have it work seamlessly (vs using third party virtual machine software and env bits). The idea is that you can develop on Mac & Windows using the same images you will eventually deploy. It just works (although I'm biased).
Disclaimer: Work for Docker.
> Unfortunately, Docker for Windows will only run on new versions of Windows 10. This is because Docker for Windows uses Windows Hyper-V and while older Windows versions have Hyper-V, their Hyper-V implementations lack features critical for Docker for Windows to work. [0]
I know this isn't what you want to hear but that is the current status.
[0] https://forums.docker.com/t/install-client-on-windows-7/1340...
email: troygoode@gmail.com
So basically, I'd start a docker database container, it'd ask for a password and then decrypt and start the container and the database. When I shut down the machine and leave the office, I can then be sure the data in the container isn't readily available if someone breaks in a take the computer.
What could have been:
[1][2] ...instead is a whole bunch of hacking on a whole bunch of things... One would think that with as much as SmartOS has been discussed here on Hacker News, people would finally realize there is something simpler out there without all this pain, but apparently not. Crazy![1] https://wiki.smartos.org/display/DOC/Managing+Images#Managin...
[2] https://wiki.smartos.org/display/DOC/Using+vmadm+to+manage+v...
Also, the database setup is only the first part, it then goes on to show container linking and etc.
The complex startup he then continues to hack on is superfluous on SmartOS, as SmartOS has the service management facility for that. For brand new services, one XML file and svcadm import later, the thing is ready to be turned into an image, never to have to be revisited again.
I do not understand why people take the harder way to do the same thing (if there is no clear advantage to doing it the harder way). He picked MySQL, a database which is infamous for data corruption and bugs where basic SQL functionality does not work; the substrate, Ubuntu, provides no fault management architecture, no end-to-end data integrity, no post mortem kernel debugging and dynamic tracing facilities, which will eventually result in incidents in the dead of the night. Then there is the choice of Docker, which is really not designed to run a fully functional virtual server, but a single process in a single image (although it can run something akin to a virtual server, with lots and lots of extra system engineering put in). Why choose Docker, when with zones on SmartOS one can have a fully isolated, fully functional UNIX virtual server, but running at the speed of bare metal, and all of that in a single invocation of vmadm -f vm_definition.json (or use the SDC software, which puts a GUI on top of it all)?
If something more reliable and easier is available, why insist on doing it the harder way with an unreliable substrate? There appears to be no logic here, and if there is any, it escapes me. Is it lack of awareness of SmartOS, FMA, SMF, ZFS, and zones, or something else?
So can with Docker. The author simply chose to use a pre-built image instead.
The complex startup he then continues to hack on is superfluous on SmartOS, as SmartOS has the service management facility for that. For brand new services, one XML file and svcadm import later, the thing is ready to be turned into an image, never to have to be revisited again.
In Docker that's called a Dockerfile. It does exactly what you describe. Again, the author simply didn't do that.
He picked MySQL [and] Ubuntu
It's an example, does it really matter? Maybe the author is simply more familiar with it.
Why choose Docker, when with zones on SmartOS one can have a fully isolated, fully functional UNIX virtual server, but running at the speed of bare metal
The fully functional containers you've described have been available for Linux for years before SmartOS was first released, first as OpenVZ and then as LXC. Docker's approach is not a technical limitation, it's a deployment philosophy. They don't want fully functional UNIX servers, they want disposable stateless containers that can killed off and cloned.
If something more reliable and easier is available, why insist on doing it the harder way with an unreliable substrate? There appears to be no logic here, and if there is any, it escapes me. Is it lack of awareness of SmartOS, FMA, SMF, ZFS, and zones, or something else?
I suggest you then read up on Docker and the motivations behind its creation and popularity before suggesting that everyone in the thread is illogical and ignorant, because - and I say this as someone who doesn't really like Docker - your advocacy for SmartOS has a vibe of the infomercial from Family Guy: https://www.youtube.com/watch?v=5ysMt_9JnyI
Actually it does not, as SMF is part of the fault management architecture, which is part of the Solaris self-healing technology: unlike the Docker file, SMF will watch over services and restart them if necessary, if configuration files are specified as dependencies and they change, SMF will call the service's refresh method. Docker is just a delivery mechanism, whereas SMF is part of something bigger and far more powerful. SMF ties into FMA, which actually watches over not just software, but hardware down to the individual chips in the memory slots (when not running on virtual hardware). Even on zones, SMF provides the benefit of uniform diagnostics. Docker isn't designed for any of that, which is why he had to hack on the microservice setup.
That up there is science fiction in Docker. As is something like this: > The fully functional containers you've described have been available for Linux for years before SmartOS was first released, first as OpenVZ and then as LXC.Actually no, since the "containers" are zones in SmartOS, and I've been using zones since 2006, when they first came out in Solaris 10 update 2. This was before OpenVZ and LXC.
Zones gave me virtual UNIX servers. Solaris containers gave me caps on those servers. Both came in one package back in 2006. SmartOS, being a freeware open source Solaris derivative leverages all that technology, plus a whole bunch more which neither Solaris nor GNU/Linux nor Docker have. Designed for exactly what the author of the original article was trying to hack up together, minus all the hacking.
Let me tell you, you are not doing smartos any favors.
People don't give a shit about how great you think smartos is. "smartos can do this", "smartos can do that". blah blah blah.
Do you know what the difference is between the posted link and your comments? The stuff in the article actually works, while your "look how simple it is" comment, does not:
Compared to the process outlined in the article that works, and shows you how to setup the entire process to, build, rebuild, and run two separate containers.Their first command is basically
Which anyone with docker installed can run. Your first command returns an error message.And then you go on to say
> With SmartOS, the database image can be preconfigured and published into the catalog
> one XML file and svcadm import later, the thing is ready to be turned into an image, never to have to be revisited again
> a single invocation of vmadm -f vm_definition.json
So all you have to do is preconfigure things (how?), publish them into the catalog (whatever that is) and write both an xml and a json file (contents unknown) to start your service.
If you actually cared about getting people interested in smartos, you would write a "learn-smartos-by-building-a-microservice" article and show how to actually use the fucking thing instead of ranting how terrible docker is.
The usage I provided is a SYNOPSIS, as in synopsis from a manual page. You will note that mine is a comment, not an entire blog article (I do not have a blog).
The intent behind my comment is to provide a pointer to further exploration; it is not a how to document, but rather a starting point for those who are actually willing and able to sit down and read some documentation, which is in the links I provided, and will provide the equivalent of what the author's article does, and dividends on top of that. (Less incidents in the dead of the night equals long term dividends.) My intent was that someone will read it and go off to explore it.
That is not to say that I do not agree with your point, as I myself am a firm believer in examples which work. However, one must at some point have the experience to distinguish between a synopsis and a guide. A lone comment on Hacker News is not the place for an entire how to guide.
> Which anyone with docker installed can run.
Yes, but as with anything, there will be long term consequences: while anyone can get it up and running easily, when it comes time to put it in production, and the thing busts because there is no lifecycle management, no change management, no configuration management (sorry: dumping a bunch of files into a Docker "image" definition doesn't even come close to lifecycle management, or capability maturity model level 2[1]), no post mortem debugging facilities, no end-to-end data integrity, and performance penalty for running inside of a fully virtualized VM, the problems will start.
For example, MySQL was also "easy to set up", but that didn't change the fact that the software does not work correctly[2] and that MySQL deployments later caused many organizations many problems. Just because something is easy to set up does not mean it is a good solution to deploy. Nowadays people are realizing that and migrating to PostgreSQL because that software does work correctly, but all that time, effort and money spent because of how "easy" MySQL was to get going could have been prevented in the first place. With Docker and how it is designed, and based on what I know, I can see the same types of problems which will inevitably lead to outages. And in the interest of disclosure, what is it to me? Well, argumentum ad populum is extremely spread these days, which means that lots of jobs would (and have) started springing up with Docker, which means I would be forced to work with what I know from experience is inferior technology. Since I see the problems of Docker, that means I would not be able to sleep through the night, and spend my nights in conference calls with clueless managers demanding that I fix it. I do not want that; I want to sleep through my nights, and work with slick and reliable technology like SmartOS, not a toy like Docker causing me problems down the road, and just because it was popular.
You are making an indirect argumentum ad populum, which is never a good thing.
> If you actually cared about getting people interested in smartos, you would write a "learn-smartos-by-building-a-microservice" article and show how to actually use the fucking thing instead of ranting how terrible docker is.
If you had bothered to read the documentation in the links, you would have a pretty good idea on "how the fucking thing works", and how to build a microservice; I am sorry if you think that clicking on the links and reading through the articles is too much overhead. How is it then, that reading through that Docker article isn't overhead, and you had no problem clicking at that?
[1] https://en.wikipedia.org/wiki/Capability_Maturity_Model#Leve...
[2] [deleted] ↗ (comment deleted)
For those following along, I've noticed a few errors/typos, and maybe others in the thread caught them...
* Container ports are exposed like so, HOST_PORT:CONTAINER_PORT, or YOUR_MACHINE:DOCKER_VM. If you already have a mysqld instance running, then try using 3308:3306
* The /search route needs a "next" parameter in users.js:
"app.get('/search', (req, res, next) => {"