Being caught in the containerization craze myself I'd love to hear whether the story is exaggerating or is painfully accurate.
So far I've been bitten by the inability to clean up images of certain age.
UPDATE:
Another really annoying thing is the inability to tag an image directly into a registry (AFAIK). You need to pull, tag and push back again. Given that images can be GBs long you end up with really heavy net traffic for a simple task.
No, there is no simple command to remove old images because the creators of docker want you to compose other existing commands to delete them and it would be unnecessary to add a command. IMO this is a terrible decision but that was apparently the rationalization.
That's what I do but given that the process is run in parallel being part of a complex CI system I needed to write quite a bit of logic to do the pruning safely myself. Main problem being that I couldn't rely on docker daemon not deleting images that might actually be still needed by other processes building in parallel.
But then you could just push from the staging server. And it's probably best to keep this independent of the image tag i.e. your system knows image v1.1.2 is the latest stable, but the image name never changes. If you retag your stable image "latest", you run into issues with rollback and figuring out what you're running.
We are currently moving to using Docker swarmkit (from non-Docker environment). We thought about using tags to communicate environment but it felt like an anti-pattern given how tags work. What we ended up with is the following:
Per publishable branch (e.g. master to prod, dev to QA, shared integration that isn't ready for dev to secondary QA):
- Publish image to private docker registry with tag in the format of <branch>-latest (e.g. master-latest, dev-latest). Also add labels with git revision hash and CI build number to reconcile what is in the image both in CI and in git repo.
- Capture the digest of the published image and put in .txt file as an artifact from CI.
- Add a tag to the git repo with the build number, treated as version number in the form of v<build-number>-<branch> (e.g. v1-master).
- Maintain list of applications' docker image digests that should be running in each environment. We currently have this automatically determined by pulling the digest artifact from the passing-build digest. We do allow for manual overrides, however.
- We have a CI project that updates the docker swarm with these digests and other settings (replicas, mounts, environment variables). We wrote a small tool that does this similar to where stacks/dab files are going but with more functionality.
With the labels and published digests we can go from build to digest, digest to build, etc.
This is working really well so far and we aren't fighting the tools. We may separate the non-master registry later in life, and that's really easy given that docker treats the private registry hostname as part of the tag.
"It affects ALL systems on the planet configured with the docker repository"
OK, this is why people use AWS' container repo, or they use the open source software and maintain their own repo... this happens with any of the public repo services, and it was 7 hours.
"The registry just grows forever"
S3 is a supported backend, highly would advise anyone running their own repo to use it (or have a similar expanding-space story). There's also a config flag for the repo to allow removing images. Obviously wouldn't want to use this if you're hosting a public repo but internally, go for it, it's off by default, seems sane enough.
"Erlang applications and containers don’t go along"
I'm certain that people are running erlang containers successfully.
"Docker is a dangerous liability that could put millions at risk. It is banned from all core systems."
Daily FUD allowance exhausted.
I guess the tone of this article really bugged me because there's obviously a point to be made from this experience that running Docker is more difficult & error prone than it should be. And maybe it wasn't a great fit for this company culture. But this article crossed the "we didn't have a good experience, therefore, it's BS software" line, and frankly, that attitude may very well have been to blame for the lack of success just as much as docker's shortcomings were...
Sure, it all hinges on cleaning up on a schedule, however you want to do that, cron is probably pretty popular. But the "hackiness" was due to a rudimentary approach, not because it inevitably needed to be a hacky process.
Right, the whole argument doesn't make sense. What should Docker do instead? Removing containers after they stopped so you have no idea what happened when it crashed? "Reuse" the container?
Apparently somebody didn't grasp the concepts* and therefor considers all this BS.
*) Docker could maybe do better here, but tbh - I think documentation and getting the word out about how things work is one of the things Docker Inc does pretty ok.
Take a look at Rancher also. It is a bit more mature project, and it did save me some headache. To be honest I didn't use Docker everywhere and everywhen but where I did, do real issues.
https://github.com/Yelp/docker-custodian looks awesome, thanks for the tip! Rancher looks great too -- would love to know if you looked at other options (kubernetes etc.) & how Rancher won out
I'm writing sick from bed so my posting capabilities are slightly diminished :) .
Rancher itself is not an alternative to Kubernetes. It can control a Kubernetes, AWS, Mesos or very simple Cattle cluster. We started with Cattle with a plan to one day use something like Kubernetes, a step at a time, because we kinda felt intimidated (I left that company) by articles like these.
FYI, Rancher can use Kubernetes for orchestration (Mesos or Swarm too)[0]. No idea how well that works though, I am using it with their own Cattle and that is more than enough for my lowly needs.
I looked at Kubernetes before Rancher but it was hard for me to really understand where to start and how it all fit together. Now that I understand much better what I am doing I should probably take another look at it.
Hmm, if anything that link seems to imply it's invalid - plus it's Quora, not exactly a high quality reference.
The rule is "an" is used in front of a word that starts with a vowel sound[1]. "An hour" is correct because the "h" is silent. All modern (native) English dialects pronounce the "h" in history, ergo "an history" is incorrect grammar.
3b. supposed authorities are in complete disagreement on fundamental points of grammar (eg. oxford/cambridge disagree on has/have in third person verb of possession; oxford is right historically, cambridge teach far more 2nd language speakers)
Well, this person seems to have had a hard few months. I have a lot of problems with the post though:
- Docker do not own CoreOS and did not create the distro
- Docker did not write OverlayFS
- There are issues with storage drivers. Don't use AUFS for the reasons outlined. I've had no issues with Overlay2, but note there is also devicemapper (which Red Hat have poured a lot of time into) and BTRFS. And ZFS now IIRC.
- It's not correct to say Docker was designed "not" to do persistent data. You do need to be aware of what you're doing, and I suggest not running DBs in containers until you have the solved the FS issues and have experience with orchestration.
- Kubernetes has been around for a while now and is fairly stable in my view. Swarm isn't rubbish, but the new version is very young and you may run into issues in the short term.
- With microservices, when you are running lots of small containers, you have to expect them to crash now and again. It's just the law of averages. You need to design your application to accept this and deal with it when it happens.
If you hit this many problems with any given tech, I would suggest you should be looking for outside help from someone that has experience in the area.
I would like to add another factual error to the list
- you can get linux 4.4 on debian stable with the backports instead of switching to an unstable release, and the current ubuntu LTS (16.04) runs 4.4 as well.
It seems that they started to use docker while it was changing a lot, I didn't have any issue in the past few months with docker, but I don't use it in production.
You know that trick the post mentions about how you have to keep removing images to stop your machines running out of disk space? Yeah, removing images doesn't actually reclaim disk space on devicemapper (!!):
You have to use devicemapper in direct-lvm mode in order to get free space back. I agree it's not the best out-of-box experience, but so far we haven't had any problems running ECS on CentOS instances since switching to direct-lvm.
Yes, although loopback mode for devicemapper is the out of the box configuration for Docker even though Docker themselves doesn't recommend it for production. I had tons of issues w/devicemapper on CentOS 7 (kernel panics causing AWS to mark the host as unhealthy and kill it off) that I was finally only able to resolve by switching to AUFS under Ubuntu 16.04, which has worked pretty well for me. Knock on wood, AUFS has been pretty solid so far.
well the article had some issue's especially around coreos, but it's good that somebody points it out.
Actually CoreOS produced pretty stable stuff: etcd, fleet. Actually with fleet/etcd combined with some cgroups i see barely a reason to use docker, I mean on Cloud's its actually another useless layer, at least for me, maybe not in Google scale.
container engines provide apis that orchestration softwares use to manage containers in all kinds of ways. In small setups with not much logistic going on i guess you can manage everything with fleet and some systemd units
Having followed the history of Docker since its LXC roots and it decided to write its own LibContainer library the only thing I want to point out is:
> It's not correct to say Docker was designed "not" to do persistent data.
Actually...looking at a lot of things "missing from" or "difficult in" Docker can more or less be traced back to Docker's overriding philosophy of container runtimes. Things like not having an init process and optimizing user experience around single "Run" commands.
Having relatively recently done a container retrospective, following the path of:
I can't help but feel as if some of the "limitations" of Docker were more or less designed.
I'm not saying it's a bad thing, being a Python enthusiast I see value in simplicity/having only one way of doing things...but I just wanted to point out that where you draw the line for what is considered "designed" starts to get blurry in areas of Docker's chosen implementation.
Don't forget that DotCloud/Docker was originally basically a clone of Heroku, where "12 factor" dictated that all persistent data was stored in (non-containerized) Postgres.
That's incorrect. Dotcloud ran both stateless and stateful services on the same container fabric, unlike Heroku and every other PaaS. Specifically you could run a managed containerized postgres, mysql, mongodb, redis, rabbitmq, and custom stateful tcp services. Docker was a result of that design.
Docker was specifically designed to NOT impose a model of ephemeral containers. For example, by default 'docker run' preserves all data.
We use https://github.com/l3nz/whaleware for persistent, database-oriented workload with persistent data. We used to keep persistent data on a data volume, but now mount a native directory instead (2x faster).
Might not be the True And Approved Docker Way, but who cares? it works well, and has been in production for a number of years now. No data loss ever.
> If you hit this many problems with any given tech, I would suggest you should be looking for outside help from someone that has experience in the area.
I suggest using something that works for you and your business instead. If the only thing that works is $SOLUTION, but you have these difficulties, then yes by any means look for help.
Edit: Ok, I feel compelled to extend my reply as I've been downvoted without comment, which is terrible.
Here's the thing: You're not forced to use Docker just because it's the next hyped thing. It is great and yes, I use it. However there is more than way of doing things and the only reason to choose one technology over another is if it's better for the business.
I did not downvote you but it sounds like the author works at one of those horrible places where developers get to do things that make no sense in production without undergoing an operational review. He probably cannot change that (change would have to come from devs in a company like this), so your advice boils down to finding a new job.
Anonymous downvoting is pretty popular around here. I agree it's a very rude thing to do. Also, just an FYI, I was scolded the other day for acknowledging the downvoting in an edit.
By "most of us" who exactly are you referring to? I wasn't aware there was some kind of user survey etc that was used to determine that stuff. I thought one person just added that to the bottom of the comments guidelines. Was there a vote or something? I'm very curious how you know that is what "most of us would prefer".
Can you point to any evidence that people enjoy talking about downvoting? More people are allowed to upvote than are allowed to downvote so if your post is negative, you're clearly doing something the majority of people reading it disagree with.
Can you point to any evidence that people enjoy being down voted without an explanation? Clearly the fact that there has to be two entries in a fairly short comment guidelines points to the reality that many people have expressed that they do not like being down voted without any explanation.
And since it is such a frequent occurrence, would that not point to the fact that people do enjoy talking about being down voted?
"Issues with storage" is an understatement. Linux has major issues with storage. I see no way forward for Linux in that area unless they adopt ZFS.
How many filesystems has the Linux world created in the last 20 years? Of those how many are rotting piles of tire fire?
Filesystems are hard to get right.
ZFS is the way forward. For cross compatability, for reliability, for stability, for lots of use cases.
Btrfs is another tire to throw on the burning pile of other filesystems that won't work out in Linux.
Let the down votes begin.
But Linux has some major architectural issues that eventually the Linux faithful will have to admit to.
Containers.
Filesystems.
Observability. Though bless him Brendan Gregg is trying his hardest to help here.
Portability.
And this is what is so painful for the rest of us to watch or deal with. This inability for the Linux folks to admit these glaring architectural problems. Problems they refuse to look outside their bubble to see how others solved these problems and adopt solutions that have already solved these problems. They just want to continue to bury their heads in NIH soil. And double down on trying to solve these issues, poorly, on their own without any awareness of how others have solved these issues. People outside Linux land just might have the right ideas on how to solve these problems!
But I expect no one in that camp to acknowledge this and down vote away.
I agree with this. SmartOS would be my choice over Solaris. I think FreeBSD is getting there. With things like Tredly. The building blocks are solid and available they just lack some polish.
So if you have the skill FreeBSD. Otherwise SmartOS makes a killer setup for containers unmatched by anyone else.
A warning, though: don't try to use FreeBSD with SCSI tape drives, in my truncated experience because of what I found experience a few months ago, the driver is capable of writing without error data it then can't read back (but can be read back by Linux).
Maybe not so important in the container context, but sooner or later, somewhere you need persistence, and tape offers certain persistence features you pretty much can't get elsewhere, especially at its media price points.
(Plus, if they got this wrong, when it's really not that hard to get right (I've done SCSI at this level before), I wonder what else they have.)
And this is what is so painful for the rest of us to watch or deal with. This inability for the Linux folks to admit these glaring architectural problems.
Alright, the suspense is killing me: what OS is the one that gets it right? I ask this to "the rest of us" as you put it.
It is not just a version of FreeBSD. The base is FreeBSD but the tooling for containers is not stock. It's value added bits that are shaping up to be very nice for containers on a more stable, secure, and storage friendly stack.
I can see a lot of inflammatory content here - vague accusations about "tire fire" filesystems and how Linux users are "faithful" zealots, blind to the truth, that will have to "admit" their folly and implicitly humble themselves before "the rest of us", whoever they are. And baiting a bad response, as you ask people to "downvote away" as if you want to inspire anger. All of this is needlessly confrontational. You don't need to insult your opposition to make your point.
You may view them as inflammatory. But it's coming from a POV of pain having to deal with broken architecture, piss poor portability and general pain of hacking around poorly engineered Linux architecture.
Frustration, pain are more appropriate terms from my pov than inflammatory. But ymmv.
It's like "Come on folks! You can do better than this! Please! Just stop! Think! Please. I'm begging you."
Regardless of whether your point is true or not (I agree that the state of Linux is suboptimal), this kind of attitude in which you spend more on venting your emotions than on actually describing issues, will not help you get your point across, and will not help making the world better. In the worst case you may even give BSD people a bad name by making the public perceive them as a bunch of emotional haters.
> ZFS is the way forward. For cross compatability, for reliability, for stability, for lots of use cases.
I'm not entirely convinced we should settle on ZFS just yet. It's fantastic and quite possibly the best option right now but it has a few limitations:
- The Linux implentation seems to have issues with releasing memory from the ARC back to the system
- The only way to expand a zpool is to add a new vdev. Pools are essentially a RAID0 of vdevs so if a single vdev fails, your entire server fails. You can mirror or RAID within a vdev but this means that the reliability of a vdev is the reliability of your entire pool. The problem here is that you can't just add 1 or 2 new disks since adding a 1 or 2 disk vdev would be data-suicide. For smaller servers, this is silly.
BTRFS looked like it was getting there but it proved buggy and unreliable. Personally, I'm waiting for bcachefs: https://www.patreon.com/bcachefs
I strongly disagree. If I have an 8 disk server at home with mirroring, the pool could only take a single disk failure. The usable space would be 50% of the total disk. Further, I'd have to buy any extra disks in pairs.
If instead I ran RAID6, I'd have 80% of the disk available and I could add disks in single disk increments.
I think ZFS makes great sense for businesses that can throw money at disks but for smaller businesses or home servers it's kinda bad.
It was kind of a pain to configure (albeit quite flexible) but it's been pretty nice overall, already survived 1 disk failure and a capacity upgrade (during which I had to resilver after every individual disk upgrade, which was time-consuming, but after the last disk got upgraded, the extra space finally showed up)
I think it's not a very good idea to design storage solutions around how inexpensive it is to add capacity to them. Your 9+-drive RAID6 is going to take forever to rebuild; 4+ mirrored vdevs (or mirrored RAID of course) will not be a problem at all.
I'd extend your point on zpool expansion to ZFS's general inflexibility when it comes to reshaping pools or rebalancing ZRAIDs.
I'm sure it's not an issue in enterprise environments, but I've personally been bit by ZFS's inability to shrink or rebalance pools (even offline) several times now through personal use.
Those sort of use cases need to be handled if ZFS is to become a more general use filesystem.
Many Linux users in particular are pretty well aware of the shortcomings of Linux's filesystem scenario, as well as the fact we never had features like DTrace. (I still think the over-proliferation of Linux tracing tools that fit 10,000 different case is a bit ridiculous.) As someone who worked on a distro, I've been acutely aware of many of these things, for example, as have our users (because they inevitably come to us for advice, suggestions, or bug reports).
I'm not sure what reality you live in where Linux users like, actively deny things like lacking an appropriate COW filesystem with low latency, or things like the fact it took us a decade to get only close to DTrace. Your own mind? 6 month Ubuntu users? Internet forums where you talk to other BSD users and nobody else?
However XFS is perfectly good for storage. I've used it on 15pbs worth of fileservers with no particular nonsense.
I wouldn't touch BTRFS with a barge pole, not because of any inherent stability issues (although that is a big factor) its the utter nastyness of the tooling.
ZFS is a joy to setup, and its simple, logical and the man pages tell you useful things.
point taken. I have had xfs_check run out of memory because $reasons. This has reminded me of that. But then we also had lustre, so you know, by comparison XFS is a paragon of stability (We also had clustered XFS....)
But, mostly it happens because the fileservers dont have UPS's (The pipeline tools are almost exclusively COW, and backups are tested many times a week.)
I think we paid for support on XFS from either redhat or SGI, but I can't remember, I left that place a year or so ago.
I've never had the balls to run ZFS on large(100tb+) arrays. Last time I tried the way slab handling was translated from solaris cause many problems (but that was more than 4 years ago. ) Plus the support is a bit odd, you either go with oracle(fuck that), or one of the openZFS lot.
To get the best performance/stability you ideally need to let ZFS do everything, instead of letting the enclosure do the raid 6 (4 * 14 disk raid 6 with 4 spares). This of course is a break from the norm and to be treated with utmost suspicion.
Have you seen the GPFS raid replacement? thats quite sexy.
I run ZFS at home, because it works, and has bitrot checking.
Yeah, I flagged the article for being grossly inaccurate. I've had a lot of frustrations with Docker myself, and will happily switch to Rkt after it matures a bit more. Docker surely deserves a lot of criticism, but it should at least be valid and well-informed.
I think Docker can somewhat be seen as a generalized term like Linux. Linux doesn't write EXT4 but it is often referred to as "Linux filesystem".
Docker has been actively trying to become to household brand name for containers so it's difficult for me to give them a pass with your line of reasoning.
> Re Persistent Data: You are probably going to have a tough time if you try to do persistent data. Containers really excel at stateless tasks(or temporary state like caching). Certainly it's doable but for someone early in their adoption the proper toe in the water is probably the stateless business logic not the state layer.
Can you blame them for wanting to break more into the Microsoft ecosystem? Besides, Docker's company mentality aligns pretty closely with a company like Microsoft, there's probably a lot for them to learn. A huge plus from a business perspective, but it's easy to see why it wouldn't sit well with open source advocates.
I thought the whole Windows thing was totally stupid (I is also freedom software advocate). I've been a big fan of Docker for about three years, put loads of stuff into production at a handful of different companies, most recently using native Swarm features that came with 1.12. Personally, I thought the Windows thing was a gimmick. But what I see (as someone that has access to the Docker community through open source) is this enormous group of people who are chained to Windows technologies. And for years and years they've watched all the cool kids play with the cool toys, from the sideline. And what Docker-for-Windows has done -- with help from a new kind of Microsoft -- is expose all of these very eager people to containerization and microservices and open source and free software more generally. Having been exposed to loads of these people (and there are an enormous number of them), I think it means a lot of important things about bringing open source and free software into the fold for these people. Docker-for-Windows opens the door for a yuge number of developers to appease their enterprise leadership by sticking with the Microsoft brand, while also having the whole world of free software at their fingertips. I can see this world on the horizon in which open source and free software concepts have broken into the Microsoft ecosystem, and I believe it comes waving the Docker flag. I am still extremely suspicious of Microsoft as a company, and wouldn't pay for Windows if my life depended on it, but you can't say they aren't doing some interesting things over there at Microsoft lately.
There is time and resources spent on improving free and open source software, and then there is time and resources spent improving proprietary software for another business.
I have some sympathy for the OP here, as I have heard of a lot of people running into pain with DBs. But yes, they definitely should not be writing to the UFS, as the OP seems to be doing. However, even if you're using volumes, you need to be careful about what happens when images change etc. It's not trivial.
Nobody really remembers the Diamond Rio Karma but everyone knows about the iPod. History is only acknowledged by historians, perception is reality otherwise.
Before the Karma the Rio PMP300 was very cool! Portable mp3's! You could load up ~20-30 minutes on its 32mb. This completely blew away the first mp3 player I saw which could only hold 8mb which was still amazing in its own right.
I worked at a university then, no one was using solaris zones. I think they had the 2 big AIX boxes that had lpars enabled in a fixed configuration that never changed.
all solaris and aix was ripped out and replaced with linux.
The problem with zones was that you used them to run solaris which as a distribution was a complete disaster. Maybe it worked if the only thing you them for was to run oracle.
I had created a bunch of solaris zones and then had the great idea to patch them.. everything broke.
I'm actually using smartos at home now. kvm and debian lx zones. works great. A solid platform with a UX that doesn't make me want to kill myself.
One man's trash another man's treasure. Solaris 10 packaging was awful, but IPS in Solaris 11 (soon 12) and OmniOS is great and the integration with zones is fantastic.
Well, you did say "Solaris zones UX is still unsurpassed"
Still implies that it was always good..
It was absolutely terrible in Solaris 10. Maybe it's better now. Maybe with solaris 11 they fixed it so you didn't have to chose between a full zone that took 5GB or a 400MB sparse zone that would break in fun ways when you tried to do anything.
At the time I was using xen+debian and a 'full' vm for me was about 200MB. Basically base system+sshd+whatever daemon I needed to run. Attempting to do the same thing using zones was a complete disaster.
That's odd, we've been using Docker for about a year in development and half a year in production (on Google Container engine / Kubernetes) and haven't experienced any of the panics, crashes yet (at least not any we could not attribute as a failure on our end).
Clearly, Docker itself has a a few annoyances, like no garbage collection and poor tagging process, but we're using it on a daily basis, and nobody complains about it (nothing a few bash scripts can't fix).
Finally, the biggest benefits of not just docker but containers in general are consistency and immutability which brings me to the final point I agree with the author: Your database shouldn't be containerized. It's not your pet (from pets vs cattle), it's your BFF.
Yeah, for every article complaining about how Docker in production is just so terrible there's a huge group of people who have been running it for a while and have no problems at all.
Nobody writes rant posts when things work, I guess.
I definitely try to highlight the success stories, but there's a critically different way that audiences react to "here's why everything went great for me" on Hacker News, though I think that is understandable. Too many times they're customer success stories, or case studies from the marketing department, and many don't care for those.
Agreed and confused. Interested to find what would be wrong with this? (from my docker-compose.yml under the db service)
volumes:
- ./data/postgresql:/var/lib/postgresql/data
We host PostgreSQL on GCE with two separate instances (master/standby) replicated with repmgr (http://www.repmgr.org/). Then we have a service defined with custom endpoints that point to these two instances.
There's plenty wrong with a Pod that writes to PV - if your Pod/Container somehow gets corrupted, you're left with a corrupted database. It's unlikely, but as the origintal article states, Docker is young and things happen. Are you willing to risk it?
So in every case you need to ocnfigure replication and master selection. Are you willing to trust the pods to do the right thing without manual intervention on data that is the core of your business? I'm not.
What exactly do you mean by "container get corrupted"? One should always use the read-flag to run Docker containers and use explicit volumes for persistent data. With this setup a container can only get corrupted if a read-only bind mount gets corrupted, and this will be a bug in kernel, not Docker.
1) The conclusion of the article is to use Google Container Engine because Google are the only one who might have managed to get a working ecosystem around Docker and may be able to maintain it.
2) As disclosed by other comments, Google Container Engine doesn't use Docker. It uses containerization technology made by google for google, with a docker-interface added on top.
That's why you don't have problems with Docker. You're not using Docker at all! ;)
It seems I have seen a lot of posts on HN relating that Docker is not ready for production. I am using Docker in production on my small company for one year and it's working pretty well (maybe because we are still small).
We're successfully running Dokku instances for various dev/qa environments, but I have not attempted containers in production as of yet.
While Kubernetes seems to be the orchestration choice du jour, I had been eyeing http://deis.io/ (which I believe to be at least as mature). Can anyone comment on their experience or thoughts on Deis vs Kubernetes? TIA.
Since OP is running on AWS anyway, why not just use Amazon ECS? The orchestration (including Elastic Beanstalk) is pretty good, and we haven't had any stability issues in production with it. Same thing for registries; just use AWS ECR instead.
Here here. I operate hundreds of ECS clusters and it's great.
> The impossible challenge with Docker is to come with a working combination of kernel + distribution + docker version + filesystem.
Amazon has met this challenge with the ECS Optimized AMI. It works, gets frequent upstream updates, and you can open tickets against it and get great support.
> My first encounter with docker goes back to early 2015. Docker was experimented with to find out whether it could benefit us. At the time it wasn’t possible to run a container [in the background] and there wasn’t any command to see what was running, debug or ssh into the container.
Having worked all of 2015 in the container space, I can state this comment is a false statement. While it may not have been possible for this person, the technology has existed for quite some time to see what process are running in a container, debugging became possible with sysdig and ssh'ing into a container has been a thing for YEARS.
I work for a large enterprise 100K+ employees using Docker in production [1]
It's certainly had its challenges (different versions of Docker clients causing havoc with different manifest versions, race conditions in the engine etc etc), but this article takes things a little far.
If you take pretty much any new technology like this make it a base part of your platform, you'll want support from a big vendor. To think otherwise would be naive.
There is definitely a debate to be had about Docker Inc's approach to production/enterprise rollouts though, but as a technology I'd say it's developing pretty much as I'd expect.
I've also seen Docker succeed in production despite the technical challenges. If you don't like the heat...
The most important challenge that Docker faced was where it wanted to position itself, IWO, as a solid stable building block for container infrastructure, or as a all-in-one-alike container solution. Now that Docker has made its decision to push Swarm mode, it is getting better for productions. There are instabilities here and there but it will come around sooner or later.
From the article: "The impossible challenge with Docker is to come with a working combination of kernel + distribution + docker version + filesystem. Right now. We don’t know of ANY combination that is stable (Maybe there isn’t any?)."
What combination of Docker version + distro + kernel + filesystem are you running, and how stable is it?
I also have been steeped in Dockerland for the last couple of years. Here are my own Docker pros/cons. Caveat: I am not nearly the power user that thehftguy is or understand Linux kernel issues in any real depth. I’m a lowly high-level language programmer (elisp/Python/Clojure/Java/shell).
Pros:
When Docker works, it’s great. The concept of being able to install software on the users behalf in the container and have the user install the container really does make life easier. The unwieldy command lines are nicely abstracted with docker-compose(.yml). Software upgrades via containers becomes a nearly trivial process.
I work on scientific software that can be a huge pain to install and configuring the packages correctly often times requires the help of the software creator. The fact that you can codify this “dark knowledge” in a Dockerfile is of tremendous benefit. And generally speaking, I like the concept of the Dockerfile. Whatever container technology ends up winning, I hope there will always be something like the Dockerfile.
Cons:
- Docker version upgrades never go smoothly. Firing up docker after apt-get update && apt-get upgrade never works with respect to Docker. You get some obscure error and after spending 30 minutes in Google you end up having to rm -rf /var/lib/docker/aufs. My solution is sometimes to throwaway the VM and start over, but this is unacceptable.
- File ownership issues inside versus outside the container are (or at least used to be) a huge pain in the ass! I am referring to files mounted from the Docker host file system into the container. (And I am also referring to running docker as unprivileged user that is part of the docker group.) The solution I settled on is to enter the container as root, run chmod/chown on the appropriate directories and drop down to a “regular” user with gosu. At that point, you proceed on your merry way by starting your service, process, etc. This work flow solves permission problems inside the container, but outside the container is a different story. At least on earlier versions of Docker, you could lose permissions of files created via the container because at that point they are owned by root as seen from outside the container! More recent versions of Docker seem to solve this problem as I have gleaned empirically and from experimentation but I have yet to see adequate documentation about any of this and I have spent a long time searching! With more recent versions of docker I have observed that the files in question are no longer owned by root (again as viewed outside the container), but by the user that started the container, the behavior I would expect. I’m also not crazy about the gosu solution which seems like a bolted on rather than baked in solution.
>Making a new filesystem in 1 year is still an impossible mission. Docker just tried and failed. Yet they’re trying again! We’ll see how it turns out in a few years.
Is this just fear mongering or are there any actual issues with overlay2. we have been trying out overlay2 on our less critical systems and haven't experienced any issues so far but we have stressed it enough to know for sure.
Docker's reliance on overlay filesystems is one of the biggest problems I have with Docker. Stacking opaque disk images on top of each other just isn't a great design, and it makes for a cache strategy that is all-too-often invalidated (because a Dockerfile is linear, there is no dependency graph). The file system is just the wrong layer of abstraction for solving such an issue.
If you change paradigms from the imperative sequence of mutations in a Dockerfile to a declarative specification that produces immutable results and a package dependency graph structure, you get a much better cache, no need for disk image layering, a real GC, etc. For example, the GNU Guix project (purely functional package manager and GNU/Linux distro) has a container implementation in the works named call-with-container. Rather than using overlay file systems, it can just bind mount packages, files, etc. inside the container file system as read-only from the host. Not only is this a much simpler design, but it allows for trivial deduplication of dependencies. Multiple containers running software with overlapping dependency graphs will find that the software is on disk exactly once. Since the results of builds are "pure" and immutable, the software running inside the container cannot damage those shared components. It's nice how some problems can simply disappear when an alternative programming paradigm is used.
One can get very similar experience with Docker. The idea is to make single Docker image with all the software one needs on the server and start all containers using it with read-only flag. Then one use volumes for persistent data.
That's not really the same thing. What about different applications that share dependencies? With Docker, they would have to share an exact subset of a linear Dockerfile to take advantage of any deduplication, so you're back to image layers. It's telling to me that when proposed solutions involve completely avoiding one of Docker's primary features that maybe Docker isn't very well designed to begin with.
With Guix, any sub-graph that is shared is naturally deduplicated, because we have a complete and precise dependency graph of the software, all the way down to libc. I find myself playing lots of games with Docker to take the most advantage of its brittle cache in order to reduce build times and share as much as possible. Furthermore, Docker's cache has no knowledge of temporal changes and therefore the cache becomes stale. Guix builds aren't subject to change with time because builds are isolated from the network. Docker needs the network, otherwise nothing would work because it's just a layer on top of an imperative distro's package manager. Docker will happily cache the image resulting from 'RUN apt-get upgrade' forever, but what happens when a security update to a package is released? You won't know about it unless you purge the cache and rebuild. Docker is completely disconnected from the real dependencies of an application, and is therefore fundamentally broken.
Docker needs network only when one uses Dockerfile for deployment, a rather bad idea. Instead Docker images should be used. It allows to verify them on development/testing machine before deployment. And with this setup all "bad pieces" of Docker are located at developer's notebook. In production everything is read-only and shared among all containers.
As an LXC user since very early days, I wrote something similar to but architecturally far more generic than docker, begun earlier (~201-2015). It's called cims. It was explicitly designed to be portable to *BSD as well as arbitrary storage drivers (docker added this later), cloud provider interfaces (VM only, AWS/GCE style full APIs, existing server with LXC or docker in place, etc.), and between arbitrary logical machine and service topologies. See docs @ http://stani.sh/walter/cims/ and early architectural forethought @ http://stani.sh/walter/pfcts/ Scope was arguably broader (ease of system administration, explicit need for existing CI/CD process integration), and to date I still believe the architectural paradigm is superior. Unfortunately my former employer decided to ask me to pay them to release the code, so I can't release it open source. But the high level docs have been open since ~early so knock yourself out on a clone. I may even write one myself in future.
Storage-wise, it had LVM2, ZFS, and loopback drivers. Never bothered with overlay, just used a generic clone API to the storage driver to spin up identical VMs before modification. Very easy with snapshot/thin-provisioning capable backends, like LVM2 and ZFS. Loopback just used cp, but because you could have these in memory they could also be very fast (but memory-hungry). Cloud-wise, we'd done a few but I was also iterating an orchestration system based upon internal requirements (high security/availability) and existing, proven solutions (pacemaker/corosync). It was designed for fully repeatable builds, something docker only began to add at a later date.
Author's company should hire a proper devops person and then docker, magically —from the devs pov— will start working.
AUFS on production? Seriously? Create a BTRFS partition.
Docker isn't a shortcut to learning a linux based OS, it is a useful tool in the hands of people who know what they are doing.
If your company lost thousands of man-hours on a tool and has nothing more to show than a blog article with half the facts coming out of the author's head and the other half boasting about bad production practices, would you prefer me to sugar coat it?
A hire that knew linux and docker, would solve the problem.
I work with docker since 2013. I came to a team with devs that already used docker. I didn't know anything about it at the time. They gave me a docker image with a JVM based app, that had a two years old JDK, ran as root and the container was ran with privileges enabled.
Did I have to know docker to tell them that this was nuts? Nope, I had to know ops. For the same reason I knew AUFS before I learned about docker and I knew very well that I would trust AUFS on my NAS for media storage but not as storage for a production database.
Did I ever tell the devs how to write java? No. It isn't on my job description for a reason.
A linux guru will know that AUFS is unstable... but the problem is mentioned nowhere and Docker still uses it as the default filesystem for [most of] all cases.
The linux guru will know to avoid AUFS... but the replacement [overlay2] was rolled out very recently and it's only available in the latest OS and systems.
AUFS isn't included in the default setup of many Linux distributions. CoreOS, CentOS, RHEL do not have it. If I recall correctly, CentOS and RHEL will use the device-mapper driver by default and CoreOS the btrfs. I would speculate that the docker team gave so much attention to AUFS to lower the barrier to entry but it isn't the only option.
My choice is btrfs and it has been rock solid for me since the start. Of course one has to know a bit about btrfs and always leave a healthy percentage (I'd say 10% or more) of the filesystem free.
I think AUFS and overlay2 do have an advantage of their own though (apart the works-out-of-the-box). Due to the way they work, you can have shared memory between containers. Thus if I had to run many containers with a JVM app, I would give them a try to lower the tax on RAM.
> If I recall correctly, CentOS and RHEL will use the device-mapper driver by default and CoreOS the btrfs
CentOS & RHEL uses device-mapper with loopback device by default (and logs "Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning". Also related article [1]
CoreOS uses overlayfs by default now from a while [2]
> A hire that knew linux and docker, would solve the problem.
I'd pick a hire who also knew a platform. I like Cloud Foundry because I work on it, Red Hat folk would prefer OpenShift, there's also Deis and Convox and others I am cruelly neglecting.
Containers are a building block. An important one, they make new architectures practicable. But past a certain point, rolling a custom PaaS that you will have to maintain at your own expense, with no commercial or opensource support whatsoever, forever, doesn't make much sense.
299 comments
[ 2.0 ms ] story [ 304 ms ] threadSo far I've been bitten by the inability to clean up images of certain age.
UPDATE: Another really annoying thing is the inability to tag an image directly into a registry (AFAIK). You need to pull, tag and push back again. Given that images can be GBs long you end up with really heavy net traffic for a simple task.
You obviously have a process to build new images... why doesn't the same process prune old ones ?
For instance, you may want to run an image for a while on a staging server before re-tagging it as the current stable release.
Per publishable branch (e.g. master to prod, dev to QA, shared integration that isn't ready for dev to secondary QA):
- Publish image to private docker registry with tag in the format of <branch>-latest (e.g. master-latest, dev-latest). Also add labels with git revision hash and CI build number to reconcile what is in the image both in CI and in git repo.
- Capture the digest of the published image and put in .txt file as an artifact from CI.
- Add a tag to the git repo with the build number, treated as version number in the form of v<build-number>-<branch> (e.g. v1-master).
- Maintain list of applications' docker image digests that should be running in each environment. We currently have this automatically determined by pulling the digest artifact from the passing-build digest. We do allow for manual overrides, however.
- We have a CI project that updates the docker swarm with these digests and other settings (replicas, mounts, environment variables). We wrote a small tool that does this similar to where stacks/dab files are going but with more functionality.
With the labels and published digests we can go from build to digest, digest to build, etc.
This is working really well so far and we aren't fighting the tools. We may separate the non-master registry later in life, and that's really easy given that docker treats the private registry hostname as part of the tag.
https://github.com/spotify/docker-gc
"Many attempts can be found on the internet, none of which works well. There is no API to list images with dates"
http://portainer.io/ seems to be able to do it, docker images lists it... I mean, I don't want to call bullshit without knowing the full story, but...
"So, the docker guys wrote a new filesystem, called overlay"
https://github.com/torvalds/linux/commit/e9be9d5e76e34872f0c... written in 2014 by the linux kernal team / Linus, but OK cool story
"It affects ALL systems on the planet configured with the docker repository"
OK, this is why people use AWS' container repo, or they use the open source software and maintain their own repo... this happens with any of the public repo services, and it was 7 hours.
"The registry just grows forever"
S3 is a supported backend, highly would advise anyone running their own repo to use it (or have a similar expanding-space story). There's also a config flag for the repo to allow removing images. Obviously wouldn't want to use this if you're hosting a public repo but internally, go for it, it's off by default, seems sane enough.
"Erlang applications and containers don’t go along"
I'm certain that people are running erlang containers successfully.
"Docker is a dangerous liability that could put millions at risk. It is banned from all core systems."
Daily FUD allowance exhausted.
I guess the tone of this article really bugged me because there's obviously a point to be made from this experience that running Docker is more difficult & error prone than it should be. And maybe it wasn't a great fit for this company culture. But this article crossed the "we didn't have a good experience, therefore, it's BS software" line, and frankly, that attitude may very well have been to blame for the lack of success just as much as docker's shortcomings were...
I mean, he's right on the cron part. That's how all the various open source docker cleaners are working.
*) Docker could maybe do better here, but tbh - I think documentation and getting the word out about how things work is one of the things Docker Inc does pretty ok.
Also there is DCGC from yelp!
Rancher itself is not an alternative to Kubernetes. It can control a Kubernetes, AWS, Mesos or very simple Cattle cluster. We started with Cattle with a plan to one day use something like Kubernetes, a step at a time, because we kinda felt intimidated (I left that company) by articles like these.
I looked at Kubernetes before Rancher but it was hard for me to really understand where to start and how it all fit together. Now that I understand much better what I am doing I should probably take another look at it.
[0]: http://rancher.com/rancher-now-supports-kubernetes/
[1] http://knowyourmeme.com/memes/an-hero
https://www.quora.com/Which-one-is-correct-a-history-or-an-h...
The rule is "an" is used in front of a word that starts with a vowel sound[1]. "An hour" is correct because the "h" is silent. All modern (native) English dialects pronounce the "h" in history, ergo "an history" is incorrect grammar.
[1] https://en.wikipedia.org/wiki/English_articles#Distinction_b...
from the first sentence.
> Quora
> not a high quality reference
> proceeds to reference Wikipedia
Source: https://en.wikipedia.org/wiki/Elision
Well, the grammar rule got a name. It doesn't say whether it applies to "a[n]" history" though.
1. typos
2. not everyone is a native anglophone. the web is global.
3. nobody is an actual grammar expert, including you.
4. avoiding actual discussion is obvious
- Docker do not own CoreOS and did not create the distro
- Docker did not write OverlayFS
- There are issues with storage drivers. Don't use AUFS for the reasons outlined. I've had no issues with Overlay2, but note there is also devicemapper (which Red Hat have poured a lot of time into) and BTRFS. And ZFS now IIRC.
- It's not correct to say Docker was designed "not" to do persistent data. You do need to be aware of what you're doing, and I suggest not running DBs in containers until you have the solved the FS issues and have experience with orchestration.
- Kubernetes has been around for a while now and is fairly stable in my view. Swarm isn't rubbish, but the new version is very young and you may run into issues in the short term.
- With microservices, when you are running lots of small containers, you have to expect them to crash now and again. It's just the law of averages. You need to design your application to accept this and deal with it when it happens.
If you hit this many problems with any given tech, I would suggest you should be looking for outside help from someone that has experience in the area.
It seems that they started to use docker while it was changing a lot, I didn't have any issue in the past few months with docker, but I don't use it in production.
2) You are not running it in production. Poster is. All sorts of issues you don't care about on your dev box matter a lot in production.
IOW, your experience can best be summarized by "just trying out Docker in a totally different way and it works great!" Irrelevant to this post.
You know that trick the post mentions about how you have to keep removing images to stop your machines running out of disk space? Yeah, removing images doesn't actually reclaim disk space on devicemapper (!!):
https://github.com/docker/docker/issues/3182
Oh, and devicemapper is the only natively supported storage backend on the AWS official AMI.
Actually CoreOS produced pretty stable stuff: etcd, fleet. Actually with fleet/etcd combined with some cgroups i see barely a reason to use docker, I mean on Cloud's its actually another useless layer, at least for me, maybe not in Google scale.
> It's not correct to say Docker was designed "not" to do persistent data.
Actually...looking at a lot of things "missing from" or "difficult in" Docker can more or less be traced back to Docker's overriding philosophy of container runtimes. Things like not having an init process and optimizing user experience around single "Run" commands.
Having relatively recently done a container retrospective, following the path of:
OpenVZ -> LXC -> Docker -> now (CoreOS, Rancher, LXD, Mesos, etc.)
I can't help but feel as if some of the "limitations" of Docker were more or less designed.
I'm not saying it's a bad thing, being a Python enthusiast I see value in simplicity/having only one way of doing things...but I just wanted to point out that where you draw the line for what is considered "designed" starts to get blurry in areas of Docker's chosen implementation.
Docker was specifically designed to NOT impose a model of ephemeral containers. For example, by default 'docker run' preserves all data.
Might not be the True And Approved Docker Way, but who cares? it works well, and has been in production for a number of years now. No data loss ever.
I suggest using something that works for you and your business instead. If the only thing that works is $SOLUTION, but you have these difficulties, then yes by any means look for help.
Edit: Ok, I feel compelled to extend my reply as I've been downvoted without comment, which is terrible.
Here's the thing: You're not forced to use Docker just because it's the next hyped thing. It is great and yes, I use it. However there is more than way of doing things and the only reason to choose one technology over another is if it's better for the business.
Anything else is just ignoring reality.
Edit: case and point
> Please resist commenting about being downvoted. It never does any good, and it makes boring reading.
> Please don't bait other users by inviting them to downvote you or announce that you expect to get downvoted.
And since it is such a frequent occurrence, would that not point to the fact that people do enjoy talking about being down voted?
How many filesystems has the Linux world created in the last 20 years? Of those how many are rotting piles of tire fire?
Filesystems are hard to get right.
ZFS is the way forward. For cross compatability, for reliability, for stability, for lots of use cases.
Btrfs is another tire to throw on the burning pile of other filesystems that won't work out in Linux.
Let the down votes begin.
But Linux has some major architectural issues that eventually the Linux faithful will have to admit to.
Containers.
Filesystems.
Observability. Though bless him Brendan Gregg is trying his hardest to help here.
Portability.
And this is what is so painful for the rest of us to watch or deal with. This inability for the Linux folks to admit these glaring architectural problems. Problems they refuse to look outside their bubble to see how others solved these problems and adopt solutions that have already solved these problems. They just want to continue to bury their heads in NIH soil. And double down on trying to solve these issues, poorly, on their own without any awareness of how others have solved these issues. People outside Linux land just might have the right ideas on how to solve these problems!
But I expect no one in that camp to acknowledge this and down vote away.
Sigh.
Both get you ZFS and something lighter-weight than VMs. Need Linux? Bite the bullet and use VMs.
So if you have the skill FreeBSD. Otherwise SmartOS makes a killer setup for containers unmatched by anyone else.
Maybe not so important in the container context, but sooner or later, somewhere you need persistence, and tape offers certain persistence features you pretty much can't get elsewhere, especially at its media price points.
(Plus, if they got this wrong, when it's really not that hard to get right (I've done SCSI at this level before), I wonder what else they have.)
No killing yourself necessary, really ;)
Alright, the suspense is killing me: what OS is the one that gets it right? I ask this to "the rest of us" as you put it.
Frustration, pain are more appropriate terms from my pov than inflammatory. But ymmv.
It's like "Come on folks! You can do better than this! Please! Just stop! Think! Please. I'm begging you."
I'm not entirely convinced we should settle on ZFS just yet. It's fantastic and quite possibly the best option right now but it has a few limitations:
- The Linux implentation seems to have issues with releasing memory from the ARC back to the system
- The only way to expand a zpool is to add a new vdev. Pools are essentially a RAID0 of vdevs so if a single vdev fails, your entire server fails. You can mirror or RAID within a vdev but this means that the reliability of a vdev is the reliability of your entire pool. The problem here is that you can't just add 1 or 2 new disks since adding a 1 or 2 disk vdev would be data-suicide. For smaller servers, this is silly.
BTRFS looked like it was getting there but it proved buggy and unreliable. Personally, I'm waiting for bcachefs: https://www.patreon.com/bcachefs
If instead I ran RAID6, I'd have 80% of the disk available and I could add disks in single disk increments.
I think ZFS makes great sense for businesses that can throw money at disks but for smaller businesses or home servers it's kinda bad.
Disk is cheap. There's no reason to design like it's not.
Plus, disk may be cheap but servers to house it are not (the kind of servers you'd run in your house, I know you can get cheap SC846 off eBay).
It was kind of a pain to configure (albeit quite flexible) but it's been pretty nice overall, already survived 1 disk failure and a capacity upgrade (during which I had to resilver after every individual disk upgrade, which was time-consuming, but after the last disk got upgraded, the extra space finally showed up)
tires++
;-)
I'm sure it's not an issue in enterprise environments, but I've personally been bit by ZFS's inability to shrink or rebalance pools (even offline) several times now through personal use.
Those sort of use cases need to be handled if ZFS is to become a more general use filesystem.
I'm not sure what reality you live in where Linux users like, actively deny things like lacking an appropriate COW filesystem with low latency, or things like the fact it took us a decade to get only close to DTrace. Your own mind? 6 month Ubuntu users? Internet forums where you talk to other BSD users and nobody else?
> Let the down votes begin.
Don't worry, I'm sure people will oblige.
Just as long as it does't get anywhere near capacity.
What a useless comment. If you are running out of capacity adding disks to a zpool is incredibly mind numbingly easy.
"zfs add poolname mirror disk1 disk2"
Every filesystem has performance hit the floor when you consume all available capacity. This is not unique to ZFS but to well, reality.
However XFS is perfectly good for storage. I've used it on 15pbs worth of fileservers with no particular nonsense.
I wouldn't touch BTRFS with a barge pole, not because of any inherent stability issues (although that is a big factor) its the utter nastyness of the tooling.
ZFS is a joy to setup, and its simple, logical and the man pages tell you useful things.
BTRFS, not so much
1. filesystem is corrupted, once again
2. try to repair it
3. oh right, the repair tool can not replay the journal
4. try to mount it
5. admit after 3 hours of nothing that the journal-replay code triggered on mount actually can really not deal with corruption
6. reboot the server to get the filesystem unstuck
7. rerun repair, this time throwing away the journal
8. look at the empty filesystem with everything in lost+found
9. restore from backup
The team I'm in only runs 1'400 servers, yet this happens regularly.
But, mostly it happens because the fileservers dont have UPS's (The pipeline tools are almost exclusively COW, and backups are tested many times a week.)
I think we paid for support on XFS from either redhat or SGI, but I can't remember, I left that place a year or so ago.
I've never had the balls to run ZFS on large(100tb+) arrays. Last time I tried the way slab handling was translated from solaris cause many problems (but that was more than 4 years ago. ) Plus the support is a bit odd, you either go with oracle(fuck that), or one of the openZFS lot.
To get the best performance/stability you ideally need to let ZFS do everything, instead of letting the enclosure do the raid 6 (4 * 14 disk raid 6 with 4 spares). This of course is a break from the norm and to be treated with utmost suspicion.
Have you seen the GPFS raid replacement? thats quite sexy.
I run ZFS at home, because it works, and has bitrot checking.
Docker has been actively trying to become to household brand name for containers so it's difficult for me to give them a pass with your line of reasoning.
> Re Persistent Data: You are probably going to have a tough time if you try to do persistent data. Containers really excel at stateless tasks(or temporary state like caching). Certainly it's doable but for someone early in their adoption the proper toe in the water is probably the stateless business logic not the state layer.
Not a company of good taste in my book.
Pick one.
Any monkey can run a bunch of stateless systems. It's the stateful ones that have the worthwhile problems.
Not just that, CoreOS is basically made to push systemd-nspawn as a container framework as hard as possible.
You need to use external volumes
I was assuming he was complaining about them even with data volumes.
If this is "real" filesystem for disks, not proxy for other filesystems, mostly read-only.
I had to laugh at this one, then again I am an old dog.
I could also add the several solutions done by mainframes, or other UNIXes like Tru64.
all solaris and aix was ripped out and replaced with linux.
I had created a bunch of solaris zones and then had the great idea to patch them.. everything broke.
I'm actually using smartos at home now. kvm and debian lx zones. works great. A solid platform with a UX that doesn't make me want to kill myself.
Still implies that it was always good..
It was absolutely terrible in Solaris 10. Maybe it's better now. Maybe with solaris 11 they fixed it so you didn't have to chose between a full zone that took 5GB or a 400MB sparse zone that would break in fun ways when you tried to do anything.
At the time I was using xen+debian and a 'full' vm for me was about 200MB. Basically base system+sshd+whatever daemon I needed to run. Attempting to do the same thing using zones was a complete disaster.
Clearly, Docker itself has a a few annoyances, like no garbage collection and poor tagging process, but we're using it on a daily basis, and nobody complains about it (nothing a few bash scripts can't fix).
Finally, the biggest benefits of not just docker but containers in general are consistency and immutability which brings me to the final point I agree with the author: Your database shouldn't be containerized. It's not your pet (from pets vs cattle), it's your BFF.
For the record, orchestration in Docker (1.12.x) is actually quite good. Its built-in service discovery and mesh networking make setup a breeze.
Nobody writes rant posts when things work, I guess.
There's plenty wrong with a Pod that writes to PV - if your Pod/Container somehow gets corrupted, you're left with a corrupted database. It's unlikely, but as the origintal article states, Docker is young and things happen. Are you willing to risk it?
So in every case you need to ocnfigure replication and master selection. Are you willing to trust the pods to do the right thing without manual intervention on data that is the core of your business? I'm not.
2) As disclosed by other comments, Google Container Engine doesn't use Docker. It uses containerization technology made by google for google, with a docker-interface added on top.
That's why you don't have problems with Docker. You're not using Docker at all! ;)
This command won't help you clean up disk space if you build/pull images each with their own unique tags.
While Kubernetes seems to be the orchestration choice du jour, I had been eyeing http://deis.io/ (which I believe to be at least as mature). Can anyone comment on their experience or thoughts on Deis vs Kubernetes? TIA.
> The impossible challenge with Docker is to come with a working combination of kernel + distribution + docker version + filesystem.
Amazon has met this challenge with the ECS Optimized AMI. It works, gets frequent upstream updates, and you can open tickets against it and get great support.
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/e...
* Amazon Linux
* Linux Kernel 4.4.23-31.54.amzn1.x86_64
* Docker version 1.11.2, build b9f10c9/1.11.2
* Device mapper
No... losing a day of development is on their infra team.. not docker.
Having worked all of 2015 in the container space, I can state this comment is a false statement. While it may not have been possible for this person, the technology has existed for quite some time to see what process are running in a container, debugging became possible with sysdig and ssh'ing into a container has been a thing for YEARS.
This was a fun read :)
It's certainly had its challenges (different versions of Docker clients causing havoc with different manifest versions, race conditions in the engine etc etc), but this article takes things a little far.
If you take pretty much any new technology like this make it a base part of your platform, you'll want support from a big vendor. To think otherwise would be naive.
There is definitely a debate to be had about Docker Inc's approach to production/enterprise rollouts though, but as a technology I'd say it's developing pretty much as I'd expect.
I've also seen Docker succeed in production despite the technical challenges. If you don't like the heat...
(1) See my blog: https://medium.com/@zwischenzugs, especially: https://medium.com/zwischenzugs/a-checklist-for-docker-in-th... and https://medium.com/@zwischenzugs/docker-in-the-enterprise-ec...
What combination of Docker version + distro + kernel + filesystem are you running, and how stable is it?
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterp... [2] https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...
Pros:
When Docker works, it’s great. The concept of being able to install software on the users behalf in the container and have the user install the container really does make life easier. The unwieldy command lines are nicely abstracted with docker-compose(.yml). Software upgrades via containers becomes a nearly trivial process.
I work on scientific software that can be a huge pain to install and configuring the packages correctly often times requires the help of the software creator. The fact that you can codify this “dark knowledge” in a Dockerfile is of tremendous benefit. And generally speaking, I like the concept of the Dockerfile. Whatever container technology ends up winning, I hope there will always be something like the Dockerfile.
Cons:
- Docker version upgrades never go smoothly. Firing up docker after apt-get update && apt-get upgrade never works with respect to Docker. You get some obscure error and after spending 30 minutes in Google you end up having to rm -rf /var/lib/docker/aufs. My solution is sometimes to throwaway the VM and start over, but this is unacceptable.
- File ownership issues inside versus outside the container are (or at least used to be) a huge pain in the ass! I am referring to files mounted from the Docker host file system into the container. (And I am also referring to running docker as unprivileged user that is part of the docker group.) The solution I settled on is to enter the container as root, run chmod/chown on the appropriate directories and drop down to a “regular” user with gosu. At that point, you proceed on your merry way by starting your service, process, etc. This work flow solves permission problems inside the container, but outside the container is a different story. At least on earlier versions of Docker, you could lose permissions of files created via the container because at that point they are owned by root as seen from outside the container! More recent versions of Docker seem to solve this problem as I have gleaned empirically and from experimentation but I have yet to see adequate documentation about any of this and I have spent a long time searching! With more recent versions of docker I have observed that the files in question are no longer owned by root (again as viewed outside the container), but by the user that started the container, the behavior I would expect. I’m also not crazy about the gosu solution which seems like a bolted on rather than baked in solution.
Is this just fear mongering or are there any actual issues with overlay2. we have been trying out overlay2 on our less critical systems and haven't experienced any issues so far but we have stressed it enough to know for sure.
If you change paradigms from the imperative sequence of mutations in a Dockerfile to a declarative specification that produces immutable results and a package dependency graph structure, you get a much better cache, no need for disk image layering, a real GC, etc. For example, the GNU Guix project (purely functional package manager and GNU/Linux distro) has a container implementation in the works named call-with-container. Rather than using overlay file systems, it can just bind mount packages, files, etc. inside the container file system as read-only from the host. Not only is this a much simpler design, but it allows for trivial deduplication of dependencies. Multiple containers running software with overlapping dependency graphs will find that the software is on disk exactly once. Since the results of builds are "pure" and immutable, the software running inside the container cannot damage those shared components. It's nice how some problems can simply disappear when an alternative programming paradigm is used.
https://www.gnu.org/software/guix/news/container-provisionin...
With Guix, any sub-graph that is shared is naturally deduplicated, because we have a complete and precise dependency graph of the software, all the way down to libc. I find myself playing lots of games with Docker to take the most advantage of its brittle cache in order to reduce build times and share as much as possible. Furthermore, Docker's cache has no knowledge of temporal changes and therefore the cache becomes stale. Guix builds aren't subject to change with time because builds are isolated from the network. Docker needs the network, otherwise nothing would work because it's just a layer on top of an imperative distro's package manager. Docker will happily cache the image resulting from 'RUN apt-get upgrade' forever, but what happens when a security update to a package is released? You won't know about it unless you purge the cache and rebuild. Docker is completely disconnected from the real dependencies of an application, and is therefore fundamentally broken.
Storage-wise, it had LVM2, ZFS, and loopback drivers. Never bothered with overlay, just used a generic clone API to the storage driver to spin up identical VMs before modification. Very easy with snapshot/thin-provisioning capable backends, like LVM2 and ZFS. Loopback just used cp, but because you could have these in memory they could also be very fast (but memory-hungry). Cloud-wise, we'd done a few but I was also iterating an orchestration system based upon internal requirements (high security/availability) and existing, proven solutions (pacemaker/corosync). It was designed for fully repeatable builds, something docker only began to add at a later date.
AUFS on production? Seriously? Create a BTRFS partition. Docker isn't a shortcut to learning a linux based OS, it is a useful tool in the hands of people who know what they are doing.
If your company lost thousands of man-hours on a tool and has nothing more to show than a blog article with half the facts coming out of the author's head and the other half boasting about bad production practices, would you prefer me to sugar coat it?
A hire that knew linux and docker, would solve the problem.
I work with docker since 2013. I came to a team with devs that already used docker. I didn't know anything about it at the time. They gave me a docker image with a JVM based app, that had a two years old JDK, ran as root and the container was ran with privileges enabled. Did I have to know docker to tell them that this was nuts? Nope, I had to know ops. For the same reason I knew AUFS before I learned about docker and I knew very well that I would trust AUFS on my NAS for media storage but not as storage for a production database.
Did I ever tell the devs how to write java? No. It isn't on my job description for a reason.
A linux guru will know that AUFS is unstable... but the problem is mentioned nowhere and Docker still uses it as the default filesystem for [most of] all cases.
The linux guru will know to avoid AUFS... but the replacement [overlay2] was rolled out very recently and it's only available in the latest OS and systems.
My choice is btrfs and it has been rock solid for me since the start. Of course one has to know a bit about btrfs and always leave a healthy percentage (I'd say 10% or more) of the filesystem free.
I think AUFS and overlay2 do have an advantage of their own though (apart the works-out-of-the-box). Due to the way they work, you can have shared memory between containers. Thus if I had to run many containers with a JVM app, I would give them a try to lower the tax on RAM.
Gotta find the right combination of OS & filesystem to have a stable docker.
CentOS & RHEL uses device-mapper with loopback device by default (and logs "Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning". Also related article [1]
CoreOS uses overlayfs by default now from a while [2]
[1] http://www.projectatomic.io/blog/2015/06/notes-on-fedora-cen...
[2] https://lwn.net/Articles/627232/
I'd pick a hire who also knew a platform. I like Cloud Foundry because I work on it, Red Hat folk would prefer OpenShift, there's also Deis and Convox and others I am cruelly neglecting.
Containers are a building block. An important one, they make new architectures practicable. But past a certain point, rolling a custom PaaS that you will have to maintain at your own expense, with no commercial or opensource support whatsoever, forever, doesn't make much sense.
CoreOS is the company behind a competing container tech called rkt (rocket) and CoreOS Linux.