If this is an RC, will docker have a way in 1.0 to properly handle stdout/stderr logs captured on the host without just allowing a giant json file to fill up? I'm not aware of a way to do it right now without handling logs in the container and avoiding the container's stdout/stderr.
Yes, I've read all of that. I consider this a blocker for production use of docker and releasing a 1.0 RC without solving this is disconcerting (at least as a process wrapper, it still works fine for wrapping an init system that handles everything itself). Especially since, in that thread, shykes said it will be solved prior to 1.0.
You're right, the logging subsystem still needs work before 1.0. We're using the RC label to indicate that we don't plan on breaking APIs and file formats in any major way between now and 1.0, so toolmakers and integrators can start locking down their schedule, and trust that their tools won't be deprecated in a couple months.
If it were entirely up to me I would never call it 1.0, because part of me wants it to be flawless before anybody uses it in production. But it's not up to me - a lot of people are already using Docker in production, and there are many organizations already in process of offering commercial support and services for it. Clearly they consider Docker acceptable for their needs at some level - and at some point we'll need to acknowledge that by releasing 1.0, no matter how flawed I find it to be. The improvements won't stop at 1.0 :)
That's great to hear - I've dove pretty hard into docker and love it but the logging story has been a dealbreaker for production. Decoupling "create" and "start" has been great when using the Remote API because I can create a container, attach my IO handlers, and then start, removing the need for docker to handle the logs at all (if it supported that). Possibly something like this can be done via the CLI such that docker can just forward the streams into a custom script.
The best solution I've seen so far is bind mounting /dev/log to a syslogd on the host and making sure everything in the container logs to syslog.
I would be pretty happy if Docker just had a simple implementation of /dev/log by default, so containers could rely on it being there. Then you could easily redirect everything to a production grade syslogd.
The first thing I wanted to do with docker was cleanup the mess I made walking through the examples. Turns out there is no built-in command to do that, and it looks like it is going to stay that way.
I was surprised to see that nothing had been done about this; a comment there mentioned that someone was able to clean up 19GB of space once he figured out what was going on. The solution is easily google-able since so many have run into it, but cleanup of unused resources should be a first-class operation in every virtualization solution.
I'm still learning my way around, but based on my understanding of containers as instantiations of images, there should be a command to delete unreferenced images. I understand the need to manually specify containers to delete but not the difficulty in cleaning up unreferenced dependencies.
The problem with that design is that in a lot of use cases, images don't necessarily need to be 'referenced'. For example, I'm using docker for a build/test system; we spin up a container from an image, run some stuff in it for a few seconds, and then shut it down. 90% of the time, the images are 'unreferenced'.
The other suggestion, put forward by a docker dev in another comment, is to have a clean command for unnamed/untagged images. Our build system has 'buildbot' (python package compilation) and 'software' (test environment) images, so this sort of automated cleaning would get rid of all of that extra crap that we have lying around.
The big issue for me is stacks of changes. Like git, anything you commit, remove something from, then commit again still takes up that space forever until you consolidate down. Which you can't do automatically.
We don't plan on "keeping it that way". Disk clutter after lots of builds is a problem which we're very much addressing. It's specifically a top-level docker clean command that I'm not a fan of, because it doesn't capture a clear intent - what are you cleaning, and who determines what should be kept and what shouldn't? In general I like to avoid Docker having to guess what you want - the proverbial "magic".
A few notes:
* The reason Docker can clutter your disk is because we made an explicit tradeoff: it's better to keep too much data than to remove too much. That is why we're able to say "yes, you can run a database on Docker. Neither the container nor its data will be removed, at any point, unless you explicitly request it". Everytime I hear someone complain "there's too much stuff on my disk", I rejoice that I am not instead hearing "Docker ate my data".
* We have already introduced a few improvements. For example Docker used to keep the intermediary container for each build step (for later inspection of the build output, transient state etc). Now, by default, we don't.
* The most elegant fix, I believe, is to use naming and tagging as an indication of interest. If you are interested in keeping an image, give it a name (for example with 'docker build -t'). When an image no longer has a name referencing it, instead of keeping it around as an anonymous image like it does today, Docker can remove garbage collect it. That should take care of the problem elegantly and without requiring a new, clunky top-level command.
TLDR: the "disk clutter" problem is the result of a conscious tradeoff which we are glad we made. We have started fixing it, and there are more fixes to come.
Cool! Can you point me to where the discussion on this is occurring? I was confused because the issue I referenced above was closed without a reference to any issue open on how this problem will be addressed.
Most of the discussion has been on irc: #docker and #docker-dev on Freenode. There are several contributions underway under the general "theme" I described, but if it helps I could write a summary of how the experience will change for the user (normally there should be almost no UI change).
It's a real PITA that the first thing you do when you start experimenting with docker is to have to find out about this missing clean command and to design a custom alias to fit your needs. In my case, I was expecting it (eg. `bower cache clean`, `npm cache clean`). Regarding the unclear intent, that's what cli help is for! `docker clean` could output it by default.
Thank you for your feedback! I'm not a Docker contributor but I'm a big fan, and everyone that responded to you, forgot to thank you for your feedback. Your feedback is immensely valuable to making it better. Obviously if you think nothing has been done about it and they are saying there are fixes, it needs to be communicated better.
+1 on that, anyone who takes the time to tell us about their issues with Docker is automatically making a valuable contribution to the project. And it's appreciated, so yes, thank you :)
There is another problem of other than space. I grade programming assigments with Docker and let the students grade too, and when containers get too much, docker tries to read each of the metadata and it hangs, and I could not find the issue and could not remote reboot my server, so I walked to server room at 4 am. For a 8 core system, load avg was 33. So, be careful with your containers, remove them when not needed.
There is a difference between "messing around", and actually running docker in production. I do NOT want a 'cleanup' command to be available on my servers. Cleaning up on commandline is rather easy with a few docker commands and xargs. I wrote small shellscripts to do this for me when building images, but these will never ever be installed on a production server.
Looks good, particularly the support for host networking; this will be a boon (I think) for those of us using Docker to run lightweight VM style containers which we wish to have directly accessible on the network by other machines/devices, and resolved via an existing router's DHCP/DNS. Is there a way to assign a MAC address to a container under this mechanism so I can setup the corresponding DNS via my router (or dnsmasq)?
(comment copied from the other item about Docker today)
Agree. Couldn't believe this wasn't in there when I checked it out before. Isn't using bridged interfaces with VMs/containers a major use case? It's certainly my primary config, but maybe I'm doing it wrong.
I think Docker was originally designed for PaaS "app container" usage where containers are too cheap to waste an IP address on, hence the NATing and port forwarding. For VMs or "system containers" it makes sense to give each one its own MAC and IP address.
Thanks to everyone who contributes to Docker for all your hard work that we benefit from. You guys really are pushing Linux forward more than almost any project.
My biggest outstanding issue with Docker (Linux really) is the storage system. None of the three options make me feel like I'll be able to sleep at night. Aufs seems broken[1] and will never make it into the kernel. DeviceMapper seems okay, but I'm not sure how well tested the Docker usage is or how it performs. Btrfs seems ideal, but again, that's still an "experimental" file system.
I was even toying with the idea of implementing a Docker storage driver using rsync (rsnapshot?), for simplicity/stability, but not sure if that would even be possible.
Obviously something that will inevitably get resolved, but my biggest stumbling block so far.
Percentage-wise, that's a pretty significant improvement, but I need more than an 0.2s performance increase to justify the switch. Are there technical reasons to switch to btrfs instead?
38 comments
[ 4.8 ms ] story [ 76.9 ms ] threadIf it were entirely up to me I would never call it 1.0, because part of me wants it to be flawless before anybody uses it in production. But it's not up to me - a lot of people are already using Docker in production, and there are many organizations already in process of offering commercial support and services for it. Clearly they consider Docker acceptable for their needs at some level - and at some point we'll need to acknowledge that by releasing 1.0, no matter how flawed I find it to be. The improvements won't stop at 1.0 :)
I would be pretty happy if Docker just had a simple implementation of /dev/log by default, so containers could rely on it being there. Then you could easily redirect everything to a production grade syslogd.
https://github.com/dotcloud/docker/issues/928
I was surprised to see that nothing had been done about this; a comment there mentioned that someone was able to clean up 19GB of space once he figured out what was going on. The solution is easily google-able since so many have run into it, but cleanup of unused resources should be a first-class operation in every virtualization solution.
http://stackoverflow.com/questions/17236796/how-to-remove-ol...
The other suggestion, put forward by a docker dev in another comment, is to have a clean command for unnamed/untagged images. Our build system has 'buildbot' (python package compilation) and 'software' (test environment) images, so this sort of automated cleaning would get rid of all of that extra crap that we have lying around.
The big issue for me is stacks of changes. Like git, anything you commit, remove something from, then commit again still takes up that space forever until you consolidate down. Which you can't do automatically.
We don't plan on "keeping it that way". Disk clutter after lots of builds is a problem which we're very much addressing. It's specifically a top-level docker clean command that I'm not a fan of, because it doesn't capture a clear intent - what are you cleaning, and who determines what should be kept and what shouldn't? In general I like to avoid Docker having to guess what you want - the proverbial "magic".
A few notes:
* The reason Docker can clutter your disk is because we made an explicit tradeoff: it's better to keep too much data than to remove too much. That is why we're able to say "yes, you can run a database on Docker. Neither the container nor its data will be removed, at any point, unless you explicitly request it". Everytime I hear someone complain "there's too much stuff on my disk", I rejoice that I am not instead hearing "Docker ate my data".
* We have already introduced a few improvements. For example Docker used to keep the intermediary container for each build step (for later inspection of the build output, transient state etc). Now, by default, we don't.
* The most elegant fix, I believe, is to use naming and tagging as an indication of interest. If you are interested in keeping an image, give it a name (for example with 'docker build -t'). When an image no longer has a name referencing it, instead of keeping it around as an anonymous image like it does today, Docker can remove garbage collect it. That should take care of the problem elegantly and without requiring a new, clunky top-level command.
TLDR: the "disk clutter" problem is the result of a conscious tradeoff which we are glad we made. We have started fixing it, and there are more fixes to come.
I have a docker-cleanup script, which removes untagged images and containers: https://github.com/blueyed/dotfiles/blob/master/usr/bin/dock...
docker clean remove-untagged
docker clean remove-old 4w
It's bizarre that the Docker team is so oblivious to the issues running Docker in a real world environment.
(comment copied from the other item about Docker today)
My biggest outstanding issue with Docker (Linux really) is the storage system. None of the three options make me feel like I'll be able to sleep at night. Aufs seems broken[1] and will never make it into the kernel. DeviceMapper seems okay, but I'm not sure how well tested the Docker usage is or how it performs. Btrfs seems ideal, but again, that's still an "experimental" file system.
I was even toying with the idea of implementing a Docker storage driver using rsync (rsnapshot?), for simplicity/stability, but not sure if that would even be possible.
Obviously something that will inevitably get resolved, but my biggest stumbling block so far.
1. https://github.com/dotcloud/docker/issues/783