Ask HN: Has anyone migrated off containerized infrastructure?
A random and non-exhausting list of things that bother me from time to time:
— Must build an image before deploying and it takes time, so deploys are slow (of course we use CI to do it, it's not manual).
— If I need quick hot fix RIGHT NOW, I can't just log in, change couple of lines and restart, must go through full deploy cycle.
— Must remember that launched containers do not close when ssh breaks connection and they can easily linger for a couple of weeks.
I generally find it harder to change how things work together. It's probably possible to spend lots of effort to fix these things, but I don't remember having to do all this cruft with old school infrastructure.
Hence the question - has anyone migrated off containerized infrastructure? Are you satisfied? Or I'm misremembering things and horrible things wait for me in the old-school ways?
369 comments
[ 361 ms ] story [ 3457 ms ] threadPick the tools that suits your flow.
Nothing wrong with bare metal or virtual servers.
EDIT: to add some good years ago was managing PHP shop where all production was baremetal and development/staging was replicated in containers, everybody happy, hope it helps
We bootstrapped the startup and I'm based in a much cheaper area than SF or NY (Eastern Europe, actually). I can hire a decent developer here for $2k per month.
It's the plural that teases out the differences between the pets and the cattle.
And maybe that's the point of this thread: scalability.
If you have the need for that kind of thing, I don't know why you would use containers.
Containers is for organizations who have processes.
Unfortunately nowadays we teach every developer to have containers, ci/cd, terraform, test coverage, ... as a requirement
I’m sure a few. But mostly back in the days before we (almost) all had containers we automated that stuff with puppet etc...
It worked ok but it had its own problems, we iterated and move on to disposable workloads and infrastructure - which has a completely different set of problems. Amongst other things it makes scaling even easier - if you need it and if you do it right.
Before automation frameworks and services we had scripts, which were either nice and simple but limited, or a huge mess and highly complex (and often fragile).
Before that we either had mainframes / centralised computing or we didn’t have scale.
Maybe it was different in the windows world, but that’s pretty much what my experience has been across many clients over the past 16~ years with Linux/BSD etc...
I think containerization is another one of those things that you're told is great for everyone, but really you need to have many teams with many services that all need to act in concert in order for containerization to be worth the effort / overhead.
That being said, I conceptually prefer how with tools like K8s you can have fully declarative infra as code, rather than the hybrid imperative/declarative mix of a tool like Ansible.
[1] https://supernotes.app
Every now and then I have long discussions with friends who swear on containerizations.
All the benfits they mention are theoretical. I have never run into one of the problems that containerization would solve.
> This way of writing software is a double-edged sword of course. It works a lot better for a small team of good, trusted programmers than it would for a big company of mediocre ones, where bad ideas are caught by committees instead of the people that had them.
http://www.paulgraham.com/road.html
If you applied the amount of process pilots use to software deployment, you'd improve ops by orders of magnitude.
The benefits they are mentioning are theoretical for you and i personally have not worked in a professional env where VPSs and rsync would be enough at all.
So they don't argue with the present "This cannot work" but with the future "This will lead to catastrophic failure at some point!".
This has been going on for years now.
No, not necessarily. Computers are fast, and if you don't add complexity until you need it, you can do a hell of a lot with a half-decent VPS and some rsyncing.
For context: a couple of years ago I ran a website that was in the Alexa top 1K for a while (back when that was still relevant), and that was heavily visited and used for the time during which it was relevant. If you worked at any news organization anywhere, it was probably on your daily list to check.
Yet it was relatively crappy PHP, not even very optimized aside from some very naive memcache caching, and ran off a random VPS with 2GB of RAM - and that included the database. The biggest challenge wasn't scaling or deployment processes, but fighting off constant DDoS attacks.
Of course, the key difference between that deployment and a typical startup deployment is that it wasn't built like a startup. It wasn't "measuring engagement", it wasn't doing "big data", it wasn't collecting data for targeted advertising - it just did one thing and it did it well, with the only complexity involved being that which was actually necessary for that purpose.
Over the years I've looked at a lot of complex "devops" setups for other people, and almost without exception the vast majority of the resource requirements and complexities originate from data collection that approaches kleptomania, and their choice of tooling - which ostensibly was chosen to better handle complexity. It's just a self-fulfilling prophecy that way. Most people don't actually have this degree of complexity to manage.
That's not to say that there's no organizations or projects at all that would benefit from automated cluster orchestration (with or without containers). But it's very much a "prove that you need it" kind of thing, not a "you need it unless..." kind of thing.
(I do think that there's inherent value in deterministic deployments. But that's separate from whether you need multi-system orchestration tooling, it can be achieved without containers, and even then the deployment process should be trivial enough to make it worth your while.)
Edit: To be clear, this is not an argument to prioritize performance over everything else or avoid dependencies/tools, at all. Just an argument to not add moving parts that you don't actually need. For anything you add, you should be able to answer "what concrete problem does this solve for me, and why is it worth it?".
You've never had to migrate your app to another host, or manage dependencies for an app?
Also, running services is hard already. Adding another layer that caused existing assumptions to break (networking, storage, etc) made it even harder.
Bare VMs are crazy fast and running apt install is crazy easy. Not looking back at the moment!
Storage is ughh.
For storage, why not just mount host dirs into the container for stuff you want to persist? Then you’re no worse off than you were before.
One thing I realised with going bare-VM is that most services today are insanely stable. MySQL almost never crashes, Redis definitely almost never crashes, Rails/Passenger/Nginx never have any issues. The things that do happen is disks filling up, application bugs causing issues, or actual VM downtime, which is rare but happens when you have 30 VMs. With Docker or K8s it added a super complex layer that is in development and has issues.
The 4 months we ran our web servers on K8s, I spent at least 1 month debugging issues that ended at an existing open ticket on Github.
Amount of $$ that was spent on Docker infra to run couple dozen servers at the last place I used to contract for.. oh and more fun when those machines have GPUs on them(a lot more fun) then they decided to support Singularity as well because.. I don’t know.
So you manually create and configure your VMs?
Do you have some kind of HA for your database? If a VM goes offline, how quickly can you replace it?
Maybe you don't need to go back to Docker or Kubernetes, but at least consider using one of the hyperscale cloud providers, with its auto-scaling groups and multiple data centers per region, so you can have a system thatheals itself even while you're asleep or on a plane.
We run a read-replica on every database, so in case a hardware error occurs on the main database we can manually switch it over. It might mean up to an hour of downtime if the worst happens. Some data loss is OK and can be solved with manual customer support most of the time. It's also a lot cost effective than working towards a 100% SLA.
Keeping the read-replicas alive is plenty pain enough! I can't imagine automating everything to auto-heal itself. (Sounds super fun though)
Codifying the setup for auto-scaling would be a massive undertaking. Each new change then requires destroying VMs and bringing up new ones. That would then require a k8s-like layer of infrastructure for secrets, DNS, service discovery, not relying on ephemeral storage (which is a lot faster than volumes/block storage).
I really love doing ops/devops, and would love to have the perfect setup which is 100% automatic and scaleable. Even now I have to stop myself from spending too much time scriptifying things that can just be run manually.
and what if you have 10 customers joining every day? still gonna be running that ansible manually?
less so if you're limited to airgapped/onprem or there are some other security or regulatory considerations
The load balancers we utilize will do failover between them selfs, and do it really well, as in "you don't notice".
Many seem to underestimate the stability of modern virtualization, the build in redundancies, fail over feature and the capabilities of load balancers.
I would guess that most Kubernetes clusters are built on virtual machines, not physical hardware. Meaning that you just now have layers of redundancy.
- Deploy times are certainly slower, up to 50x slower than non-containerized. However, we're talking 30s deploys versus 20 minute deploy times, all-inclusive. The sidenote here is that you can drastically reduce containerized deploy by putting in some effort: make sure the (docker) containers inherit from other containers (preferably self-built) with executable version that you need. For instance, you might inherit version X of program A and version Y of program B before building only a container with version Z of program C, as A and B barely change (and if they do, it's just a version bump in the final container). Even better, just build a code container during deploy (so a container with essentially only code and dependencies), and keep all the executable as separate images/containers that are built during development time;
- Containers do allow high-speed fixes, in the form of extremely simplified rollbacks. It is built into the entire fabric of containers to allow this, as you just change a version number in a config (usually) and can then rollback to a non-broken situation. Worst case, the deploy of fixed code in my case does take only 20 minutes (after the time it takes to fix/mitigate the issue, which is usually much longer);
- Local environment is _so much easier_ with containers. It takes 10 minutes to setup a new machine with a working local environment with containers, versus the literal hours it can take on bare metal, disregarding even supporting multiple OS'es. On top of that, any time production wants a version bump, you can start that game all over again without containers. Most of my devs don't ever worry about the versions of PHP or Node they are running in their containerized envs, whereas the non-container system takes a day to install for a new dev.
Containers can be heavy and cumbersome, but in many cases, a good responsibility split can make them fast and easily usable. In the specific case of docker, I find treating the containers like just the executable they are (which is a fairly default way of dealing with it) works wonders for the easy-and-quick approach.
Like everything, it depends on context. Is it good to have separate dev, staging, and production environments? For many companies, the answer is "D'oh!". But if you're one guy trying to get your first p.o.c. out, by all means, deploy directly to production.
Containers is sortof the same thing: if you're a small team, and don't have many customers, the disadvantages of containers (however small) may outweigh the advantages.
So if I was now starting a new project solo, I would probably go straight for containers and never deal with conflicting versions or difficult, hacked-together rollbacks, and I don't think it would take me, personally, more time to set up.
In other words, I think it's a good investment of time to spend, say, a full work-week on understanding containers and experimenting on how to make it work for your use-case.
have you faced performance issues? running 5-6 docker images (app,redis,db,mq etc) at the same time has been causing the machine to lag for us. most of my team is on the 16 inch macbook.
- Is the container throwing errors? A running container somehow repeatedly throwing errors on its non-main process restarting will eat up all resources of any machine, any OS;
- Is the container trying to sync a folder or datasource between the container and the host? Especially on macos, using docker for mac, this will hurt your performance. Solutions are in the form of specialized syncing systems (docker-sync)[http://docker-sync.io/] or manual syncing using rsync or unison;
- If you have many running containers, it can be useful to spin up a linux VM (ubuntu, debian) in virtualbox, then run the containers in there, finally using a tool like unison to sync dynamic content (the changing code) to the vm;
- Is 1 container using far more resources than the others? Is this strictly necessary? It is possible, in docker-compose at least, to limit certain resources for containers (it's probably also possible without docker-compose, just using docker run) https://docs.docker.com/config/containers/resource_constrain....
Probably the dynamic content synchronisation is the biggest resource hog, and docker-sync has really helped with that in the past, plus it doesn't require different setups between linux and mac hosts, i.e. you could use the same docker-compose.yml file.
It doesn't hurt to inspect whether the host machines are potentially running other heavy services, such as heavy electron apps or a million open tabs in browsers. I've had to tell some devs to perhaps not use their work machines for personal chat applications (but just having their phone open then) because those applications were using >3GBs of RAM each on a 16GB ram machine, leaving very little for any work-related processes.
In production we have it done the other way - PostgreSQL and Elasticsearch are run directly, but code is in containers.
In the production environment, I want the code image to be set in stone, that way a deploy or rollback will go to the exact git commit that I expect. So the CI-script for deployment is just a `docker build` command, the dockerfile of which clones a specific commit hash and runs dependency installation (yarn install, etc.), then sets the image version in the production env variables. The code is then encapsulated in 1 image, which is used as a volume for other containers, and the runtimes are each in their own container, connected by docker-compose.
For local, it's a much heavier code image that I've prebuilt that contains our current version of every tool we use, so that the host machine needs nothing but docker installed to be able to do anything. The services that actually display stuff on the screen (Node.js) run as their own container with their own processes, but you can hop into your code container (used as a volume for the services) and try out command line Node stuff there, without fear of killing the procs that show your local environment.
It took a long time to reach this point, lots of experimentation, but it's now pretty lightweight and pretty useful too.
There are solutions like docker-sync but they have a kind of random delay, sometimes the sync happened fast sometimes it takes a few seconds.
Have seen what you call old-school maybe, the VMware era, Xen, colocated infra ,owned infra, ... The Cloud(s)... and finally docker (and later kubernetes), in those years.
Now I can say I happily switched job 3 years ago, to a place that never entered the virtualization, neither the containers vogue.
On a team of 3 and half persons (enough to cover one week of on-call each month), we manage our near 500 physical servers (all of them with 2xXeon, between 32 and 256G of ram, 10G networks (in 40G switches/uplinks) and all kind of storage layouts with SATA, SSD and NVME ) in different colocations. Sometimes, with the help of remote hands (mainly for hard disk replacements).
During those 3 years (and before) I have seen lot of drama between my friends, by all kind of issues related to container technologies.
Complexity issues, high availability issues, bugs, maintenance issues, team issues, cross-team issues, management issues, security issues, networking issues, chaos issues, nobody-knows issues, cost issues, operational issues, etc
Yes, you can have all of them with bare metal servers too, indeed, but I look my daily work, and then I talk with friends in pro-container companies, and I feel good.
Nothing stops you to use ansible in bare metal servers, indeed you need to automate everything if you want to be happy: IPMI setup, firmware updates, OS install, service management, operations, monitoring... the most of that you fully automate, the better will be your daily routine.
Also, really important, going "old-school" doesn't free you to have a full stage environment, and a well thought and fault-resistant architecture and good backups.
Regarding your random list:
> Must build an image before deploying and it takes time, so deploys are slow (of course we use CI to do it, it's not manual).
Maybe going "quick and dirty" will release you of this, but doing things well, no mater if it's container or bare metal, won't.
I need to build packages (or ansible roles), test the change effects, pass tests and CI, the same (it's not mandatory, but it's convenient)
> If I need quick hot fix RIGHT NOW, I can't just log in, change couple of lines and restart, must go through full deploy cycle.
True, but we're in the same point... "old-school" is not the same than "quick and dirty". You can change containers in a quick and dirty way too if you want (and your company allows that).
> Must remember that launched containers do not close when ssh breaks connection and they can easily linger for a couple of weeks
Well each platform has their own issues to deal with. In bare metal you could have to deal with compatibility and degradation (or even breakage) issues for example.
I think, a good balance between all this could be: develop "quick and dirty" and freely, release via CI, deploy properly. No mater the platform.
If developers don't have agile environments, and they need to commit, wait for build, wait for CI, review the results etc for every and each line of code they want to try... I get what you mean, it's a pain.
Of course! We have a full stage environment, good backups, etc. And I'm not willing to lose them.
I've worked once in a company where a senior developer dropped a couple of tables on production, back in 2008. They made backups but never tested them. Turns out that for some reason they were zero-sized. I'm testing all my backups since, so they're good :)
It is about application packaging.
For example, in some projects, I run containers via ansible.
Our new CI happens to use containers, but:
- We aren't managing the containers ourselves, because we're an arts platform not a cloud infrastructure provider.
- We don't spawn new instances to run a (expletive) linting tool.
So I rather deal with containers. However never do kubernetes, it has nice features but it's not worth it for small and medium companies.
The complexity of the configuration is huge. Zero downtime deployments and load balancing can be achieved without the complexity of kubernetes.
For example, drag and drop of .php files on a FTP server did that 15 years ago.
> drag and drop a PHP file
These two things run in conflict. It’s not about not dropping requests - It’s about a sane state and error reproducibility. If you drag the wrong file, there will be plenty of downtime...
Anyway, I'm not advising for uploading PHP files through FTP on live servers in 2020, I took it as an example of something we used to do without kubernetes.
They will surely get something weird, an error at best. Containers allow transactional deployments and FTP does not.
We shaved about 3-6 minutes off deploy by going to Lambda... Could shave another 1-2 minutes if we drop docker entirely, but we still build docker containers to run tests, and for local development work. The main time savings was in task definition rollover and health checks by the load balancer.. but will lambda there's less concern over unhealthy-ness since it's only detectable by the health check route in limited circumstances
Being able to just ssh into a machine is one of the problems that we did solve with containers. We didn't want to allow anyone to SSH into a machine and change state. Everything must come from a well define state checked into version control. That's where containers did help us a lot.
Not sure what you mean with lingering containers. Do you launch yours through SSH manually? That's terrible. We had automation that would launch containers for us. Also we had monitoring that did notify us of any discrepancies between intended and actuated state.
Maybe containers aren't the right tool for your usecase - but I wouldn't want to work with your setup.
Btw. most of this is possible with VMs, too. So if you prefer normal GCE / EC2 VMs over containers that's fine, too. But then please build the images from checked in configuration using e.g. Packer and don't SSH into them at all.
Maybe I phrased this point badly. I haven't found myself to need a hot fix RIGHT NOW for a long time. What I had in mind was this: sometimes there is an issue that is very hard to reproduce locally. If a developer doesn't understand it from the get go and have to experiment a bit, it can be frustrating to commit, wait for deploy (even to stage environment), test what happens and repeat.
> Not sure what you mean with lingering containers. Do you launch yours through SSH manually?
Of course the application itself is run automatically, web server, workers, etc.
However, we often need to do some bespoke data analysis, so we often ssh into a server, type `make shell` to launch a REPL and type/paste some stuff into it.
There are more sane ways to do this than through raw kubectl access, of course: see e.g. telepresence (https://www.telepresence.io/)
- How do you provide easy visibility into running applications?
- How do you prevent inspecting an application from affecting the behavior of an application?
I think of the phrase “adding epicycles” when this kind of stuff starts happening.
The solution is to force yourself not to use the CI/container system at all during debugging, and instead to build the binary (or whatever) standalone. That's hard because you invariably aren't tooled up to run the component outside its deployment system so you have to do some extra tricks, but in the long tun it seems to be the way to go.
Maybe your current strategy isn't working?
Taking code review away, because we all know we're not going to sit around waiting for a review while a patch needs to go out urgently, if your build, test, and deploy pipelines take three hours then you have some serious problems that you need to address, and containers aren't it.
There are methods for handling hotfixes/patches to production quickly that work well in high volume sales website setups.
I have app stacks that deploy a dozen containers over an hour because the orchestration takes time: signal the old containers to drain, pause for an app with a very long initialization time to settle, gradually roll traffic to the new one to let caches warm, and then repeat.
In both of these cases, deployment is a function of the application. There's nothing infrastructural that puts a time floor on things.
I worked at a trading firm where you could add a zero to that cost for a 3-hour outage, and I can tell you that the one thing we would absolutely never skimp on was code review. Because the cost of a bad "fix" that actually makes things worse has the potential to be greater still, and because humans are most likely to make silly mistakes when they're working under intense pressure.
What we would do instead is slightly intensify the code review process by pair programming the hotfix, and ensuring that a third developer who was familiar with the system in question was standing by to follow up with an immediate review.
We will even sometimes keep the video call open until deployment is done and production validation is complete - the devops guys get the information they need, someone else is keeping an eye on the checklist and calling out items if necessary, etc.
That's an easier solve with docker containers. And testing, including UI testing, can be integrated much more easily with ci/CD tools and docker containers that have code which goes by commit hashes and which ensure every package down to the version is controlled across environments.
And the impact of all of these on direct revenue/productivity can be immense. SSH into prod is a crutch.
There are other costs from outages which occur infrequently but are highly costly because developers/support are manually touching the servers.
The cost really should be analysed as what is the median cost on revenue (or profit) as a percentage point. One off pricing is pretty meaningless.
Without ant context that seems like an outrageously ignorant comment. (In saying that most companies probably are doing many things wrong).
Almost sounds like script kiddies when they start learning programming for the first time in their lives they assume that the test of successfully learning how-to-program is that your code should compile on first try without errors.
I think that sounds actually very plausible :)
I've got a client that has data feeds from multiple vendors. some are pulls, some are... "hey, we'll FTP this file to you". the file format has changed - unannounced - at least 3 times in the past... 15 months. Then something breaks on production, but you don't know what. You need to get on that machine and take a look.
"Redeploy the older working version" doesn't do anything except re-introduce more problems in these instances.
If that's a problem you find yourself having at all, much less regularly. You have a serious observability problem.
I mostly write python code and one very nice pattern I've found is to run a container somewhere (either locally or on a server somewhere) with an open port. You can SSH into it and use the remote interpreter as your main project interpreter. That way your dev environment is 100% reproducible. VS Code an Pycharm Pro both support doing that.
Can we please grow up beyond this kind of comment?
I suspect everybody knows that hotfixing production isn't an ideal thing to do, and the many reasons why that's the case, but lots of us nevertheless do it from time to time. Where I work now we've probably hotfixed production a handful of times over the past couple of years, amongst thousands of "proper" deployments. It's really not a big deal.
We know so little of the context behind OP's issues that picking holes isn't helpful or informative. The specific question was about migrating from containerized infrastructure back to a more traditional hosting model. I for one would be interested in reading about any experiences people have in this area so it's quite frustrating to find this as the top comment when it's substantially off-topic.
No platform that assigns quantitative virtue points to comments based on how many people agree with that comment will ever not have an abundance of people lobbing low-effort quips and dog whistles designed to appeal to the majority.
Exactly this. I'm fortunate in that where I work now it's pretty rare: the culture is fairly collegiate and friendly, and there's a strong sense of "we're all on the same side". Other places I've worked every meeting has been an exercise in point scoring, which is incredibly wearing and - try as one might - the culture does end up influencing one's own behaviour.
You can obviously get preachy about how people should be stronger characters and not so easily influenced but the phrase "Bad company corrupts good character" exists for a reason. Unless you're Gandhi it's incredibly difficult for many people to resist the culture around them day in and day out without some positive reinforcement from the behaviour of others, especially when you're incentivised to do otherwise, and may be penalised for not doing so[0].
[0] The answer is, if you can, find another job. Not always an option, but a good idea if it is.
You're absolutely right. There is no context outside of the few lines OP asked and the quote you highlighted shows a lack of real world experience.
Yes it's true the "right now" fixes indicate there is a problem, but in small shops it's generally the most reasonable approach. Now if you're on a team of 50 other people and you need to make "right now" fixes then there is certainly a problem. Neither of which any of us can know from the context.... but besides the point, that it's not even on topic.
A great example of this is using a production repl. It even served Nasa well (https://stackoverflow.com/questions/17253459/what-exactly-ha...). Having to change your system on-the-fly is not always an indication of a poorly developed or managed system.
Most of us are building CRUD apps not sending people to the moon.
Even in this thread, the arguments "against" containers, a lot of these things have alternatives that are still better. For instance, ok maybe you need to make a hotfix, you're still better off just compiling a one-off image on a dev machine and deploying it (I used to be able to do this in like, 30 seconds), as compared to trying to edit files directly on the server. Especially if you have multiple servers.
If your users are asking for something and your response is “you’re just doing it wrong!”, you’re probably the one who is wrong.
you should be able to ssh into a machine and strace a process to see why something is going wrong. If your only solution is always "restart the container" or "revert to an old container" or "only deploy containers that are known to work" you're not actually debugging extant problems.
But after working with it, it's pretty visible that the abstraction layer is really huge and you need to learn the tools well. When you deploy to linux VPS, you probably have already worked on unix system and know plenty of the commands.
Another thing, I think having a designated person to the infrastructure makes it much less trying for a team. On the other hand, you have 'code' sitting in the repos and everyone feels like they can into devops. I don't think it's exactly true, because e.g. k8s is a pretty complex solution.
If you don’t use CI, it’s easy to get fast deploys with containers. Just build the images on your dev box, tag with the branch and commit hash, and push directly to a docker registry (docker push is smart enough to only push the layers the registry doesn’t already have). Code is running 30 seconds after compiling finishes.
(Don’t want to pay for a registry? It’s trivial to run one yourself)
These aren’t foolproof, fully reproducible builds, but practically they’re pretty close if your tools require your repo to be clean and pushed before building the image, and if your build system is sane. Besides, if you’re used to editing code as it’s running on servers, you don’t care about reproducible builds.
Also, if you’re starting containers manually on the command line, you’re doing it wrong. At least use compose so your setup is declarative and lifetime-managed.
(Edit: s/swarm/compose/)
As I wrote in a nearby comment, I'm not starting containers manually - we have compose, swarm, it's declarative and lifetime-managed.
However, we often need to do some bespoke data analysis, so we often ssh into a server, type `make shell` to launch a REPL and type/paste some stuff into it.
I develop my stuff with k8s with all it's lifetime management and have 10 second deploys from my dev box.
make image push deploy
kubectl -f somewhere/deploy.yaml
Also sounds like there is some lay hanging fruit available by adding some caching/layering in the build process
In my main business I am forced (regulatory) to do it all by the book; vastly less enjoyable for me.
Docker first started getting traction when people were building their software with make. Make never quite got caching right (not really its fault), so nobody was really sure that their changes were going to show up in their release unless they ran "make clean" first. And, you had to have 700 helper utilities installed on your workstation for a build to work after "make clean". Docker to the rescue! It gives you a fresh Linux distribution, apt-get installs all the utilities needed for your build, and then builds your thing. It works 100% of the time! Celebrate!
At the same time, programming languages started moving towards being build systems. They want "mylang run foo.ml" to be fast, so they implemented their own caching. But this time they did it right; the "mylang" compiler knows about the effect every input file has on every output file, so it's guaranteed to give you the right answer with or without the cache. Some languages are so confident these days that you can't even disable the cache! They know it works perfectly every time. The result is extremely fast incremental builds that are just as reliable as clean builds, if you have access to that cache.
This, unfortunately, is not something that Docker supports -- layers have one input filesystem and one output filesystem, but now languages are producing two outputs -- one binary for the next layer, one cache directory that should be used as input (opportunistically) to the next build. The result is, to work around people writing makefiles like "foo.o: foo.c" when they actually meant "foo.o: foo.c foo.h helper.h /usr/include/third-party-library.h /usr/lib/libthird-party.so.42", EVERYONE suffers. "mylang run foo.mt" takes a few milliseconds on your workstation, but 5 minutes in Docker. Your critical fix is hamstrung by your CI infrastructure.
There are a number of solutions to this problem. You could have language-specific builders that integrate with your CI system, take the source code and a recent cache as inputs, and produce a Docker container as output. (Systems like Bazel even have a protocol for sharing cache between machines, so you don't have to copy gigabytes of data around that you probably don't need.)
But instead of doing that, people are writing articles about how their CI takes 3 hours when a build on their workstation takes 3 seconds, and it's because containers suck! But they don't actually suck -- the underlying problem is not saying "in production, we will only run .tar.gz files that contain everything the application needs". That is actually wonderful. The underlying problem is "the first build step is 'rm -rf /' and the second is 'make world'".
Docker is not the same as containers - docker is just one way to build containers. I've not seen one yet that would be easier to use (and as as result faster than docker) but it does not mean we cannot create one!
That said, 'docker build' is not the only game in town. For a long time, Red Hat's source-to-image tool has been able to do incremental builds:
This creates a new container from the mybuilderimage image, copies the source code from /path/to/mycode into the container at /tmp/src, and runs mybuilderimage's 'assemble' script (which knows how to invoke "mylang run foo.ml" and install the build result into the right place so that the 'run' script will later find it). The result is committed as the myapp container image, which typically uses mybuilderimage's 'run' script as its entry point.The next time the command is run, the first thing s2i will do is create a new container based on myapp, and invoke its 'save-artifacts' script, which has the job of copying out any build caches and other artifacts to be used in an incremental build. The container is then discarded.
Now, the build runs as before, but with the addition of s2i copying the saved build artifacts into /tmp/artifacts, so that the 'assemble' script can use them to make the build faster.
This isn't perfect: you pay for speedier builds with larger container images, since you can't delete build caches, etc. like you'd normally do at the end of a Dockerfile. But it's a good first step, and you can always have another step in your pipeline that starts with the myapp container, deletes unwanted files and then squashes the result into a single layer above the original base image that mybuilderimage was itself built from.
A result of this is that containerisation took off hardest in ecosystems without good build and deploy tools. Getting exactly the right version of the application, every library, and the runtime has traditionally been a struggle in Ruby, a nightmare in Python, torture in C, etc, but pretty easy in Java. As a result, most of the Java shops i know are still deploying by copying a zip file or some such onto a server.
You can run lxc/nspawn containers as lightweight VMs and save a lot of (runtime, management) overhead without having to worry about any of Docker's or k8s's quirks.
We're quite happy with that approach, Docker isn't production grade IMO and k8s doesn't make sense at our scale.
You write code, you push it, 5 Minutes later it is rolling out, tested, with health checks and health metrics.
Your infrastructure itself is keeping itself up to date (nightly image builds, e2e tests etc.)
It just works and runs. It doesn't make the same mistake twice, it doesn't need an expert to be used.
I'm not saying its for everyone! Put three/four VMs on AWS, add a managed database and you are good to go with your ansible. Use a Jira plugin to create reaccuring Tickets for doing your maintenance and that should work fine.
Nonethless, based on your 'random list of things' it does sound like you are not doing it right.
There is something very wrong if you really think its critical for you to be able to 'hot fix' aka playing hero by jumping on your vms and hacking around. IF you only one single server for your production env. there is no risk of forgetting a server to patch but there is still the issue of forgetting to backport it (which is probably the wrong term if you don't hotfix your release branch)
Most mistakes i do, are mistakes i do because i was able to do them.
And that might sound unfortunate but there is a feeling of trust for your setup. At least i get that feeling and i get that through automatisation. Knowing and seeing the deployprocess just working day in day out. Knowing that my monitoring and alerting is setup properly, knowing that the systems keep themselfs up to date, knowing there are proper tests in place.
Yep, I have that. It's more like 15 minutes than 5 for me, but this process has nothing to do with containers - it can be done in the same way without containers.
Except when you misconfigure something on friday night and it does the same mistake 100 times per hour until someone notices it.
My automated system will only get more resiliant over time. This is a benefit for the system itself.
Of course when you do it manually, you will learn and gain experience but thats only for YOU. It does not just get transfered to your colleagues and when you are on holiday and shit hits the fan, it will not help.
My biggest reason why i like automatisation so much is: the company becomes less reliant on me.
It is the same mechanism why the industry is getting more computer logic: Machines are complicated and you need to train people. Make it easier for people to 'just use' and you have more people available which you need to train less.
We have an internal tool that listens to a message queue, dumps a database (from a list of permitted databases), encrypts it and sends it to S3 to be investigated by developers.
When running on a container, the process takes 2-3 minutes with small databases, about an hour or more with larger ones. When running on a regular EC2 image, the process takes about 5 minutes in the worst case scenario and is borderline instant with smaller databases.
Mapping internal volumes, external volumes, editing LLVM settings, contacting AWS support etc yielded nothing. Only migrating it to a regular EC2 instance had any results and they were dramatic.
We run Docker containers for local development too but when restoring very large databases we need to use MySQL in a real VM instead of in our Docker setup because Docker crashes when faced with a disk-heavy workload.
So to conclude, the only reason I wouldn't want to containerise a workload is when the workload is very disk IO heavy, whereas most of our apps are very network IO heavy instead.
Tooling such as skaffold.dev would alleviate some of your complaints around deploy lifecycle. It will watch for code changes and automatically build, tag and deploy the image.
Paketo Buildbacks is great as well, no more Dockerfiles and a consistent build process across different languages and frameworks
It helped us solve a few pain points with deploying to VMs:
- dependencies are packaged and shipped with you app (no apt get install) so dev and prod environments are the same - local development happens against a docker container that contain all our prod services running inside of it (redis, postgres, Kafka, etc.) - built in CD and rollbacks via deployment channels
Enable rollbacks?
As for dealing with runtime issues/hot fixes, put something like supervisor in your containers that might need temporary interventions in production and use that to restart the service without having to restart the container.
I would think it’s pretty reasonable right now, but someone here said their deploy cycle is 10 seconds long and there are probably things we can improve without rebuilding whole infrastructure around different concepts.
1. "Hot fix now" -> I do just log in and enter the container, change a couple of lines and restart, not sure what's your problem here
2. "containers do not close when ssh breaks" -> I guess that's also going to save you if you run <whatever mysql management command> without screen/tmux !
3. "Harder to change how things work" -> actually, it makes it much easier to add services to a stack: just add them to the compose file and use container names for hostnames in configurations !
4. "Must remember launched containers" -> why not use NetData for monitoring ? it really does all the monitoring/alerting you need out of the box ! And will show containers, tell you before /var/lib/docker runs out of space (use BtrFS to save a lot of time and space)
I'll add that containers make it easy to go from DevOps to eXtreme DevOps which will let you maintain a clean master branch, and that is priceless ! Details -> https://blog.yourlabs.org/posts/2020-02-08-bigsudo-extreme-d...
Where I'm migrating to : replacing Ansible/Docker/Compose with a new micro-framework that lets you do all of these with a single script per repo, instead of a bunch of files, but that's just because I can :)