Ask HN: Who operates at scale without containers?
In other words, who runs operations at a scale where distributed systems are absolutely necessary, without using any sort of container runtime or container orchestration tool?
If so, what does their technology stack look like? Are you aware of any good blog posts?
edit : While I do appreciate all the replies, I'd like to know if there are any organizations out there who operate at web scale without relying on the specific practice of shipping software with heaps of dependencies. Whether that be in a container or in a single-use VM. Thank you in advance and sorry for the confusion.
445 comments
[ 3.7 ms ] story [ 310 ms ] threadThe build process would just create VM images with the required binaries in there and then deploy that to an autoscaling group.
It worked well and if you only ever intend to run a single service per machine then is the right solution.
Even without orchestration, I argue containers are useful. They abstract the operating system from the application and allow you to manage each independently. Much more easily than you'd be able to otherwise, anyway.
Plus you can run that image locally using something like docker-compose for an improved developer experience.
That does sound bit weird setup to me; with something like Packer building an AMI is almost as easy as building a Docker image with Dockerfiles so the benefits of using Docker seem quite slim?
That takes a TON of variables out of the equation.
I’ve been trying to pick the right Linux distro to base my images on. Ubuntu Server is the low effort route but a bit big to redeploy constantly.
I’ve also been looking at the possibility of using Alpine Linux which feels like a better fit but a bit more tweaking needed for compatibility across cloud providers.
Unikernels are also interesting but I think that might be going too far in the other direction for my use case.
For others doing this, I’d be interested in what distros you’re using?
Debian (and, formerly, CentOS) is a good standard: it occupies a sweet spot between ubuntu server and alpine, in the sense that it's batteries-included and very well-supported (apt/yum), but not particularly bloated.
I use debian for all my personal stuff. The BSD flavors (https://en.wikipedia.org/wiki/Comparison_of_BSD_operating_sy...) are tempting, though. Perhaps one weekend!
For other distros, if you're working on non-x86, or have uncommon dependencies, you should be prepared to pull in and build your dependencies from scratch, which is not much fun to maintain.
I tend to gravitate towards projects like BSD, since they align with my principles, but you do pay a cost in terms of compatibility with common software.
For example, Firefox does not treat BSD as a first class build target, so it's up to community members to build and deploy Firefox binaries, and report feedback on bugs that break BSD installations.
If access to the most up-to-date compiled versions of popular software packages is a big sticking point, BSD-land may not be the right choice. But if you're living mostly in `vim`, `bash` and `man`, and are willing to roll up the sleeves, BSD feels cozy.
It's maybe a 20% correct analogy (don't read too deeply into it, or you'll draw incorrect conclusions), but C++ is to GNU/Linux, as D/Rust/Zig is to BSD.
[1]: https://unixsheikh.com/articles/technical-reasons-to-choose-...
Thanks for the link - I’m taking a look at that now!
Note that my personal infrastructure is a mixture of Debian and FreeBSD and I dearly love both - if you forced me to pick one of the two to keep I'd be horribly torn.
In paper form (more gory details): https://www.usenix.org/system/files/conference/lisa13/lisa13...
They are using CentOS Stream now.
Our Jenkins creates a debian package, and deploying involves installing this package on a machine and creating an image. This image is then deployed to the ASGs. We operate in 5 regions in GCP.
No idea how much has changed since then
If your company builds - it's no garuntee containers are used (but it's a choice).
If your company buys software - I highly doubt containers are used at all.
I have a vendor who pitched their new "cloud-native" re-platforming project and it really spooked me. It's data management-type tool that was migrating from a traditional on-prem client/server architecture to an AWS-hosted Angular interface with a MongoDB backend. I got the same pitch a year later and the entire stack had changed. I was spooked the first time; now I'm really spooked and thinking about migrating off the platform.
Whether or not this is a good idea depends on your situation and their level of competence.
The point I'm trying to make is that if you're aware of different ways to run an OCI image, you can run things in virtually the same way as the old way of having things packaged as a VM image (AMIs on EC2).
It does prevent you from setting up your own OS distribution and integrating the app directly with it, but so do AMIs.
We used a bash script to handle what we now use GitLab's CI system for. Deployments were handled through CodeDeploy and infrastructure would be replaced in a blue/green fashion.
For example, Google's Borg absolutely uses Linux namespacing for its workloads, and these workloads get scheduled automatically on arbitrary nodes, but this doesn't feel at all like Docker/OCI containers (ie., no whole-filesystem image, no private IP address to bind to, no UID 0, no control over passwd...). Instead, it feels much closer to just getting your binary/package installed and started on a traditional Linux server.
At least in the past, almost all jobs ran in their own private filesystem - it was stitched together in userspace via bind mounts rather than having the kernel do it with an overlayfs extracted from layer tar files (since overlayfs didn't exist back then), but the result was fairly similar.
Most jobs didn't actually request any customization so they ended up with a filesystem that looked a lot like the node's filesystem but with most of it mounted read-only. But e.g. for a while anything running Java needed to include in their job definition an overlay that updated glibc to an appropriate version since the stock Google redhat image was really old.
Using nomad as a job scheduler and deployer allows you to use various modules for jobs: java, shell, ec2, apps (and containers).
I use it in my homelab and it’s great. That said, I don’t use it professionally.
I think Cloudflare is running this stack alongside firecracker for some amazing edge stuff.
https://www.nomadproject.io/docs/who-uses-nomad
(have advocated for its use in multiple financial orgs as part of my day gig, no affiliation with Hashicorp)
Homelab would be fine =)
Many companies just rely on the practice of shipping software with heaps of dependencies.
I worked at a place that simply spun up blank AWS images in autoscaling groups and allowed configuration management to install literally everything: security/infra/observability agents, the code and dependencies (via AWS CodeDeploy), and any other needed instance configuration.
The downside of this practice was slow startup times. The upside was...I don't know, I think this pattern happened by accident. Packaging these AWS instances into images beforehand would be smarter. Newly created services were generally moved over to k8s.
These were stateless web services for the most part.
I think the lesson I learned from this was "nearly any operating paradigm can be made reliable enough to tolerate in production."
At the time they would bake full machine images, which is really just a heavyweight way of making a container.
However, Docker is essentially dead; the future is CRI-O or something similar which has no daemon and runs as an unprivileged user. And you still get the flexibility and process isolation, but with more security.
Use docker on your laptop, sure. Its time in prod is over.
1.20 is when docker mode got deprecated, IIRC, but many of us were already happily running in containerd for some time.
Writes to what?
That said, I'm using Docker for my current side project. Even if it never runs at scale, I just don't want to have to muck around with system administration, not to mention how nice it is to have dev and prod be identical.
[0]: https://www.music-map.com/
I still remember opening the site one day and reading the weird appology letter. I think that was the first time I saw something I really cared about just dissappear off of the internet.
This is why I use docker, at work and for my own stuff. No longer having to give a shit whether the hosting server is LTS or latest-release is wonderful. I barely even have to care which distro it is. Much faster and easier than doing something similar with scripted-configuration VMs, plus the hit to performance is much lower.
The gist of their approach was radical uniformity. For the most part, all VMs ran identical images. Developers didn't get to pick dependencies willy-nilly; we had to coordinate closely with ops. (Tangentially, at subsequent employers I've been amazed to see how just a few hours of developers handling things for themselves can save many minutes of talking to ops.) All services and applications were developed and packaged to be xcopy deployable, and they all had to obey some company standards on how their CLI worked, what signals they would respond to and how, stuff like that. That standard interface allowed it all to be orchestrated with a surprisingly small volume - in terms of SLOC, not capability - of homegrown devops infrastructure.
We use Chef and Terraform for the most part.
Running 1B valuation with manually going to instances and docker pull xxx && docker-compose down && docker-compose up -d. EC2 created by hand. No issues.
We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is that it frees us from troubleshooting "works on my machine" issues, or from someone pushing an update somewhere and breaking our build. That's not important to everyone if they have few dependencies that don't change often, but for an internet company, the trend is accelerating towards bigger and more complex dependency graphs.
Kubernetes has mostly focused on stateless applications so far. That's the easy part! The hard part is managing databases. We don't use Kubernetes, but there's little attraction because it would be addressing something that's already effortless for us to manage.
What works for us is to do the simplest thing that works, then iterate. I remember being really intimidated about all the big data technologies coming out a decade ago, thinking they are so complex that they must know what they're doing! But I'd so often dive in to understand the details and be disillusioned about how much complexity there is for relatively little benefit. I was in a sort of paralysis of what we'd do after we outgrew postgresql, and never found a good answer. Here we are years later, with a dozen+ postgresql databases, some measuring up to 30 terabytes each, and it's still the best solution for us.
Perhaps I've read too far into the intent of the question, but maybe you can afford to drop the research project into containers and kubernetes, and do something simple that works for now, and get back to focusing on product?
Ding ding ding ding ding.
"But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime) is what's tricky. Spinning up new workers, reliable- and repeatable-enough deployment with rollbacks and such, that's easy, until it touches the disk (or the DB schema, that's its own whole messy dance to get right)
You can ask k8s for a persistent volume on local storage, but at this point you have to treat that pod exactly the same way you'd treat a snowflake database server, because the local storage is what ultimately matters.
You can replicate to other pods with their own persistent volumes, but the replication overhead adds up too both in terms of performance and complexity.
I know there's stuff like vitess and crunchy data, which try to abstract this away, but the amount of layers buried here is very high for benefits that seem kind of nebulous compared to "big ol server" mode.
Edit: somewhat* disagrees with you. It's a good listen.
k8s _can_ do stateful, but if a managed service exists for this workload, use it. It is not about _can_ I run it on k8s, it is a _should_ question. Is it worth the cumulative effort required to achieve the same degree of quality.
Simple topologies and NoSQL databases (or databases they can handle replication/partitioning/node failures automatically) are pretty easy to stick in StatefulSets.
There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet.
In addition, the patch and lifecycle cadence of k8s is pretty quick so rebooting SQL databases with 100ks TPS more than 1-2 times a year tends to be problematic
This isn't a great example to cite, since traditional circular replication in MySQL is a massive anti-pattern... it's incredibly fragile and pretty much has no valid use-case.
That's especially true today when other options like Galera or Group Replication are available. But even 10-15+ years ago, mentioning circular replication was a great way to give a DBA an aneurysm. (Well, I suppose that's arguably a use-case, if you profoundly dislike your company's DBAs...)
For instance, you put a RDBMS in a StatefulSet. You figure out how to replicate between pods. Now a piece of hardware fails and a replica needs rebuilt--what does that? So you add an operator into the mix. What happens if bin logs have been purged and a replica can't just "rejoin"? Now you have to figure out how to transfer data between pods (PVs) to bootstrap a new replica. Now you probably need some sort of init container solution to keep the pod running and volume mounted while you're copying in data. Now that you have all that, how do you handle master/primary failures and replica promotion? How do you handle backups? How do you handle restores?
Once you've solved all that, how do you performance tune your database? At some point, you're probably going to start looking at kernel tuning which you'll need some additional customizations to enable (probably dedicated node per database pod, then you can either skip k8s and configure the host directly or wire up allowed sysctls through k8s).
Or, you can skip all that and require human intervention for everything, but now humans need to wade through the k8s abstractions and fix things. With that route, you break basic k8s functionality like restarting pods since it can take down your database topology
fwiw my team eventually automated pretty much "everything" for Facebook's database fleet, and it was hundreds of thousands of lines of custom automation code (non-k8s), many years of work for 10+ extremely experienced engineers. In a k8s environment I suspect it would have been even more work.
The open source mysql k8s operators are certainly getting better with time, but there's still a lot of stuff they don't handle.
(I used to work there)
But you can run the software in k8s while using an external data store for data. With microservices you can also use small databases for each and push the data you want to persist like finished transactions and invoices to external systems.
No reason to have only one giant database to do everything in. At my last place of work we moved from a monolithic software using a giant database to microservices and many small databases, some of them NoSQL.
>No reason to have only one giant database to do everything in
It's the reality for a lot of "legacy" services. Some of these things have 50-100+ apps all connecting to the same DB with incredibly complex schemas, functions, triggers, etc. It's potentially a multi-year project to undo that. When you do that, you're potentially introducing significantly more network hops. If a business transaction spanned multiple pieces of data in the same database before, you've potentially created a saga spanning multiple microservices now.
Even if you do adopt microservices, you'll probably end up consolidating all the data somewhere else for reporting, analytics, data warehousing, etc purposes anyway (so you end up with a giant OLAP DB instead of a giant OLTP one)
Shameless plug here: https://www.reactivated.io/documentation/why-nix/
And reactivated is awesome, by the way. I just worry about how brittle it will be as both Django and React evolve.
So I adopted Nix "in anger" and now new machines always build successfully on the first try.
For me it was easy to set up. 80% done on the first day. It helped that I understood the conceptual model and benefits first. There's a lot of directions you can take it, I'd recommend getting something really simple working then iterating. Don't try to build the Sistine Chapel with all the different things integrated on day one.
It's hard to overstate how well that decision has worked out for me. Computers used to stress me out a lot more because of dependencies and broken builds. Now I have a much healthier relationship with computers.
So I would say curl|sh is probably the way to go, which I guess means I'd only want to install it into a container. But I do really like the idea of Nix.
Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible?
Is the weak spot here any OS dependencies being installed as part of the continuous build?
I think that means containers alone are insufficient for creating reproducible builds, not that containers make reproducible builds impossible.
Containers are to ops like C is to programming languages. Man is it useful, but boy are there footguns everywhere.
Usually still better than the ops equivalent of assembly, which is what came before though - all the custom manual processes.
Kubernetes is maybe like early C++. A bit weird, still has footguns, and is it really helping? Mostly it is in most cases, but we’re also discovering many more new failure modes and having to learn some new complex things.
As it matures, new classes of problems will be way easier to solve with less work, and most of the early footguns will be found and fixed.
There will be new variants that hide the details and de-footgun things too, and that will have various trade offs.
No idea what will be the equivalent of Rust or Java yet.
Can you explain this?
npm doesn't allow you to delete any published versions (you can only deprecate them). You aren't allowed to publish a version that's already been published.
Even when there have been malicious packages published the solution has been to publish newer versions of the package with the old code. There's no way to delete the malicious package (maybe npm internally can do it?).
They only need to "npm ci" (based on package-lock.json) instead of "npm install" (based on package.json) within the Docker container to get a fully reproducible build.
DevOps 101 more or less..
This is the container community’s response to this ambiguity I think.
The RUN command is Turing complete, therefore cannot be idempotent. Nearly every usable docker image and base image include the RUN command, often explicitly to do things that are not repeatable, like "fetch the latest packages from the repository".
This is all before you get to application code which may be doing the same non-repeatable actions with gems or crates or maven or node modules, or calling a service and storing the result in your build. Getting repeatable docker images usually is a matter of first getting your build system to be repeatable, and then expanding it to include the docker images. And often times standing up a private repository so you have something like a reliable snapshot of the public repository, one you can go back to if someone plays games with the published versions of things.
The main reason the RUN statement isn't idempotent, in my opinion, is that it can reach out to the network and e.g. fetch the latest packages from a repository, like you mention. (Other things like the clock, race conditions in multi-threaded programs, etc. can also cause RUN statements to do different things on multiple runs, but I'd argue those are relatively uncommon and unimportant.)
Maybe someday someone will invent a sort of EBPF for containers, but usually the first batch of RUN commands and the last are calling package managers, and in between you're doing things like creating users and setting permissions using common unix shell commands, which have the same problems.
With the possible exception of the unix package managers, we have no hope of those ever being rewritten inside of a proof system, because the package manager is often a form of dogfooding for the language community. The Node package manager is going to be written in Node and use common networking and archive libraries. Same for Ruby, Rust, you name it.
Turing Machine model doesn’t apply to i/o so it is a meaningless statement.
Bazel and Nix achieve this by sandboxing i/o no turing incompleteness needed (tho bazel is still purposely non-tc but for totally different reason than hermeticity)
But there's no real need for repeatable build docker images. You copy the image and run it where ever you need to. The entire point of Docker is to not have to repeat the build.
They for many years close issues and reject PRs based on the assertion that Dockerfile output needs to be repeatable (specifically, that the requested feature or PR makes it not repeatable). It’s not, it can’t be without a time machine, and if it was I believe you’d find that ace icing traction would have been more difficult, even impossible.
Instead, build your artifact and publish it to an artifact repository, just like we used to.
_then_ wrap that artifact in a Docker image.
Vulnerability found in the docker image? No problem. Build a new image with the same artifact.
What you said about "wrapping" I interpret as: based on an image for instance with OS level dependencies you create another image with application level artifacts, e.g. a python application. When your app changes, you don't build the base image again, you only build the app image. This makes sense to me.
You could break the build simply by cleaning some things from the cache, without changing the Dockerfile, even thought that Dockerfile "succeeded to build"
I inherited a dozen or so docker containers a while back that I tried to maintain. Literally none of them would build— they all required going down the rabbit hole of troubleshooting some build error of some transitive dependency. So most of them never got updated and the problem got worse over time until they were abandoned.
The reason Nix is different is because it was a radically ambitious idea to penetrate deep into how all software is built, and fix issues however many layers down the stack it needed to. They set out to boil the ocean, and somehow succeeded. Containers give up, and paper over the problems by adding another layer of complexity on top. Nix is also complex, but it solves a much larger problem, and it goes much deeper to address root causes of issues.
Assuming that the container's Dockerfile is persisted somewhere in source control, the base image used by that Dockerfile is tagged with a version whose upstream hasn't changed, and that the container isn't modified from the image that Dockerfile produced, you get extremely reproducable builds with extremely explicit dependencies therein.
That said, I definitely see the faults in all of this (the base image version is mutable, and the Dockerfile schema doesn't allow you to verify that an image is what you'd expect with a checksum or something like that, containers can be mutated after startup, containers running as root is still a huge problem, etc), but this is definitely a step up from running apps in VMs. Now that I'm typing this out, I'm surprised that buildpacks or Chef's Habitat didn't take off; they solve a lot of these problems while providing similar reproducability and isolation guarantees.
To make it much more reproducible, you need to do it in two stages: first, pinning to a specific upstream version, installing all of the packages you want, and then tagging the output image. Then you pin to to that tag and use that to prep your app. That's... probably... going to be pretty repeatable. Only downside is that if there is, say, a security update released by Ubuntu that's relevant, you've now got to rebuild both your custom base and your app, and hope that everything still works.
This along with the supply chain issues you mentioned is why some maintainers are moving towards using distroless base images instead, though these can be challenging to debug when things go wrong due to them being extremely minimal, down to not having shells.
(that sounds like a energy-level-transition in developer productivity and debugging capability, if so)
OK, there's one concession, there's an env var that indicates if it's a dev and prod environment. We try to use it sparingly. Useful for stuff like not reporting exceptions that originate in a dev environment.
Basically, there's a default.nix file in the repo, and you run nix-shell and it builds and launches you into the environment. We don't depend on anything outside of the environment. There's also a dev.nix and a prod.nix, with that single env var different. There's nothing you can't run and test natively, including databases.
Oh, it also works on MacOS, but that's a different environment because some dependencies don't make sense on MacOS, so some stuff is missing.
The immutability is just at build time. So chrome and firefox aren't able to seed a unique ID in the binaries like you might be accustomed to. Funny story, we had a python dependency that would try to update itself when you imported it. I noticed because it would raise an exception when it was on a read only mount.
I know that there are quite a few people opposed to the state of containers and the technologies revolving around them. I don't think the arguments they present are bad. [Attacking the premise](https://drewdevault.com/2017/09/08/Complicated.html) isn't particularly hard. What I don't see a lot of, however, is alternatives. "Learn how to do ops" is not exactly pertinent when most documentation on the subject will point you towards containers.
In addition the whole principle, while clearly proven, does strike me as a patch on existing platforms and executable formats that weren't designed to solve the sorts of problems that we have today. While efforts could have been made to make software universally more portable it seems we opted for finding the smallest feasible packaging method with already existing technology and have been rolling with it for a decade.
So essentially I'm interesting in knowing how people find ways to reproduce the value these technologies offer, what they instead rely on, which things they leave on the table (eg "we work with petabytes of data just fine but deploying updates is a nightmare"), how much manual effort they put into it, etc. New greenfield projects attempting to replace the fundamentals rather than re-orchestrate them are also very pertinent here.
Decades of work typically. That or just standard parallel deployments, most things Docker is good at would take the average developer many unnecessary hours to reproduce.
It all boils down to “what bullshit was my management sold on”.
Promises of “easy to use, easy to replace, cloud-agnostic, doesnt require domain knowledge” etc etc etc.
Its all bullshit. At some point/scale you need experts in the field that cash in a month more than average dev per year.
It's interesting that you're not interested in what they get by not using containers...
The case that always interested me is the service that can really fit on one machine. Machines are truly gigantic in terms of memory and disk. Heck, I worked at BrickLink which ran on two beefy machines that serviced 300k active users (SQL Server is actually quite good, it turns out). (A single beefy server is a really great place to start iterating on application design! PostgreSQL + Spring Boot (+ React) is a pretty sweet stack, for example. By keeping it simple, there are so many tools you just don't need anymore, because their purpose is to recombine things you never chose to split. I can't imagine why you'd need more than 16 cores, 128G of RAM and 10T of storage to prove out an ordinary SaaS. That is a ferocious amount of resources.)
Partly forced due to internal resource constraints. However, swapping getting a working system out and tested with real users instead of waiting an undefined time to get high availability didn't lose me any sleep.
It's also the case that some systems can withstand a degree of downtime that others can't, or it's not worth paying the cost for the perceived benefit gained.
Depends on how many nines of uptime you need, of course, and other such things.
(Obviously, there are reams of cases where this isn't sufficient)
Thankfully I had both onsite and offsite backups; but it took a couple of weeks to order the new equipment, install it and set everything up again.
Now I "devops" all the things and always have the ability to spin up all the services and restore from backups within minutes. This is the new standard in my opinion.
1. How would power conditioners and lightning arresters handle these sorts of situations?
2. Was this using VMWare fault tolerant VMs, just out of curiosity? (It continually live-migrates one VM to a hot spare over a 10G link so either can disappear and the VM continues to run.) Or was this a bespoke application architecture implementing effectively the same thing?
In my case because I was working for a checkout solution for a Cash & Carry chain, with 1000 stores in 25 countries, each with tens of tills, self service points, queue busting tills who also had to serve the web shop. And that solution had to work also when there was no Internet access.
The linked Drew Devault seems to have a glib response for quite a few things. His devs shouldn't build software for users take is just as infuriating. You just want to say "Yeah, I'd rather not, but unfortunately it kinda goes along with writing software. Send 1000 happy users to me to build my software on every platform, because I'd be glad to let them. Otherwise, what is your alternative?"
It's easier to incrementally switch to Nix first then NixOS later.
We don't use systemd services for our code. We only use NixOS as an operating system, not for application layer. Our code works just the same on any linux distribution, or even macos minus linux-only stuff. That way we don't need to insist that everyone uses the same OS for development, and no one is forced into VMs. Everything can be run and tested locally.
Not using NixOps or Morph. I recall considering a few others as well. In each case, I wasn't able to understand what they were doing that I couldn't do with a simple script. Instead, there's a python program that does an rsync, installs some configuration files, and runs the nix build. It deploys to any linux OS, but it does some stuff differently for NixOS, and the differences account for about 100 lines while increasing scope (managing filesystems, kernel versions, kernel modules, os users, etc). A deploy to all hosts takes about a minute because it runs in parallel. Deploys are zero downtime, including restarting a bunch of web apps.
The NixOS configuration itself is another ~200 lines, plus a bit per host for constants like hostname and ip addresses. It's really neat being able to define an entire operating system install in a single configuration file that gets type checked.
It is very simple and works great. Deploying from macOS to NixOS is possible as well.
https://github.com/krebs/krops
https://tech.ingolf-wagner.de/nixos/krops/
I feel similarly about many cloud services in e.g. AWS. I get where it's sometimes handy, but the management overhead can get pretty insane. I used to hear people say "just do X in AWS and ta-da you're done!" only to find out that in many cases it isn't actually a net improvement until you are doing a lot of the same thing. And then it costs $$$$$$.
Elastic resources are pretty magical when you need them, but the majority of software projects just don't end up requiring as many resources as devs/management often seem to assume. Granted, migrating to something like AWS, if you do end up needing it, is a pain, but so is adding unnecessary layers of AWS complexity for your CRUD app/API backend.
Also wonder how many devs came up on AWS and prefer it for familiarity and not having to know and worry too much about aspects of deployment outside of the code itself (i.e not having to know too much about managing postgres, nginx, etc).
In recent years I've done work for companies where it feels like the end result of a lot of very expensive AWS infra is... more expensive, and not really superior to things we had ten years ago when hosting on-prem.
There's a level of scale for which distributed AWS/Azure is necessary, but I'm not sure how many shops need that.
The devops guy at a previous gig thought I was some kind of naive simp for insisting we didn't need a cluster of Amazon's version of Redis. We were storing like, a megabyte of data in there. That's not a typo.
It's not that he was an idiot. Smart guy actually. Not sure if he was just padding his resume with experience ("sure, I've admin'd Redis clusters!") or if he was locked into some kind of default thinking that wouldn't even allow him to consider some kind of minimal setup.
> Containers only give reproducible deployments, not builds, so they would be a step down.
This is not true, if you use a docker image A with specific version to build then the result is reproducible.
That's what most people should be doing, you pin versions in your build image that's it.
> Kubernetes has mostly focused on stateless applications so far. That's the easy part
Kubernetes can run database, and stateless application is not a solved problem, I mean how does your services get restarted on your system, is it something that you had to re-create as well?
Does that matter if the current stack works just fine?
Imagine someone who built a calculator app with jQuery javascript 10 years ago, and all it does is add and subtract numbers. You could spend time porting it to Ember, and then migrating to Angular, and then porting it to React, and then porting it to React with SSR and hooks.
If the calculator app worked with 15 year old code, you can either leave it be, or try to keep up with latest tech trends and continuously refactor and port the code.
I think there are a lot of "calculator app" type components of many systems that, at a certain point, can be considered "done, complete" without ever needing a rebuild or rewrite. Even if they were built on technologies that might now be considered antiquated.
Which they don't, because nothing has convinced me of the crappiness of some of my code more thoroughly than watching other people try and fail to use it. They are a mirror and people don't always like what they see in it. That guy who keeps talking trash about how stupid everyone else is for not understanding his beautiful code is deflecting. If they're right then my code does not have value/isn't smart which means I don't have value/am not smart, so clearly the problem is that I'm surrounded by assholes.
I don't think we would be using as much 3rd party software without StackOverflow, and almost nobody creates a StackOverflow for internal tools. Nobody writes books for internal tools. Nobody writes competitors for internal tools which either prove or disprove the utility of the original tool. All of those options are how some people learn, and in some cases how they debug. You're cutting off your own nose when you reinvent something and don't invest the time to do all of these other things. If I lose someone important or get a big contract I didn't think I could win, I can't go out and hire anyone with 3 years experience in your internal tool, trading money for time. I have to hire noobs and sweat it out while we find out if they will ever learn the internal tools or not.
The older I get the more I see NIH people as working a protection racket, because it amplifies the 'value' of tenure at the company. The externalities created really only affect new team members, delaying the date when you can no longer dismiss their ideas out of hand. You have in effect created an oligarchy, whereas most of the rest of us prefer a representative republic (actual democracy is too many meetings).
Some of what I said is a performance piece. If I insist that I am human and thus make mistakes, then it makes space both for others to make mistakes and for us to have a frank conversation of how, in our more lucid moments, we can do things that fight/counteract our basic nature.
It's morally equivalent to putting a plastic cover over the Big Red Button so that pushing it becomes a conscious and considered act. Deciding to push the button is step 1, moving the cover is step 2, and step 1.5 or 2.5 is realizing this is in fact not what you want to do after all, instead of a momentary bit of inattention, bad judgement, or the Imp of the Perverse.. It's not that I won't make mistakes, but that (hopefully) I a make them less often. But making them less often, I'm entrusted with bigger potential for mistakes, and so my mistakes tend to cause more harm. In a Precautionary Principle sense, we end up roughly within the same order of magnitude. When I make 5x fewer mistakes I'm allowed to be responsible for problems that are twice as risky.
If you want candor about mistakes you have to start by giving people permission to make them (or at least, to make new ones).
People have been building this infrastructure since long before containers, thousands of servers in single operational clusters. Highly automated data infrastructure without containers is pretty simple and straightforward if this is your business.
https://www.redhat.com/en/resources/crunchy-data-and-openshi...
The great majority of Kubernetes experts I have met use a managed service and have not been exposed to the internals or the control plane back end. An abstracted version of Kubernetes that works well for most use cases but makes it more difficult to solve some of the problems listed in this thread.
Most folks dont really know Kubernetes. They know how to deploy X application using X abstraction (Helm, Operators, CD tooling) to some worker node in a public cloud using public sources and thats basically it. It's no wonder they cannot solve some of the common problems because they didnt really deploy the apps. They filled out a form and pushed a button.
The original Linux container design was never intended with this type of software in mind, and tacitly reduces the control a process has over its resources. Invariants required for performance are violated, and therefore performance suffers relative to bare metal (or conditionally VMs). The v2 container implementation, which isn't widely available yet, recognizes these problems and attempts to remedy some of these poor behaviors of the v1 container implementation.
If your database does not assume strict control of its underlying hardware resources then this does not affect you. However, this kind of strict resource control is idiomatic in all the highest performing databases architectures, to great effect, so it is the most performant databases that are most adversely affected by containerization.
FWIW, virtual machines used to have similar terrible performance for high-performance databases for similar reasons. Now there are mechanisms for the database to effectively bypass the VM similar to the kernel, and the loss of performance is minimal. Container environments typically don't offer similar bypass mechanisms.
If I gave you a Dockerfile for a node.js service that installed its dependencies using npm, and the service used non-trivial dependencies, and I gave you this file 10 years after it was first written, chances are you would not be able to build a new image using that Dockerfile.
This would be a problem for you if you had to make some small change to that node.js service (without changing any dependencies).
For example Crunchy Postgres offers an enterprise supported Kubernetes Operator that leverages stateful sets. Yet I am reading in these comments that it is an unsolved issue.
It's very easy to do in Go.
You pin the version used to build your app ( the runtime version ) In your git repo you have dependcies
That's it.
He ponders why it is that big websites inevitably have kernel developers. Way out of their domain of expertise, right? If you adopt a technology, you're responsible for it.
When Kubernetes inevitably has an issue that is a blocker for us, I don't have confidence in my ability to fix it. When an internal python or shell program has an issue that is a blocker for us, I change it.
PostgreSQL is used by probably millions of people, but we've had to patch it on multiple occasions and run our fork in production. Nobody wants to do that, but sometimes you have to.
The point is, you can't just say "oh, we use kubernetes so we don't have to think about it". No. You added it to your stack, and now you're responsible for it. Pick technologies that you're able to support if they're abandoned, unresponsive to your feature requests and bug reports, or not interested in your use case. Pick open source, obviously.
This is another reason I like Nix. It's a one-line change to add a patch file to a build to fix an issue. So I can contribute a fix upstream to some project, and then I don't have to wait for their release process, I can use the patch immediately and let go of it whenever the upstream eventually integrates it. It lowers the cost of being responsible for third party software that we depend on.
I doubt that this happen in reality because Kubernetes cover uses cases for pretty much everyone, now I would doubt the knowledge of a regular dev to try to mimic a solution that k8s already does.
The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go.
When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two had developed much better BS-detectors than 20-something me.
Overall though, I'm finding the more unique sensory info you give your brain, the better it gets at solving problems in general. Even if that info isn't very related to your goal.
It's a bit like trying not to be a hammer that sees everything as a nail.
"Everything old is new again" does not BEGIN to do tech justice. Compared to nearly every other profession on the planet, tech knowledge churn is like 10x as fast.
Spot on! Moreover, every time the wheel is unnecessarily reinvented previous lessons are lost.
What is even worse is that people really want to reinvent the wheel and get defensive if you point that out.
Comments like "Or is cope for a loss of neuroplasticity" are a good example.
I have been studying a different trade, completely unrelated to software engineering or tech, and by far the weirdest thing is reading books or watching seminars from 20 or 30 years ago that are still relevant. How much of technology writing has that honor? Very little.
Like I have been obsessed with tech and computers my whole life, I studied computer stuff when all the other kids were outside playing football or chasing girls or whatever, I self-taught to a very high level of expertise and that knowledge has just been getting systematically ripped away. Which is fine, to an extent, gotta embrace change and yada yada yada. But I no longer have the patience to grind to expertise-level on something over a period of years only to find out the industry has moved on to something completely different to do the exact same thing! All because some starry-eyed shit somewhere in FAANG doesn't like something!
It's amazing how some people are familiar with every framework of the month and don't understand how a computer works...
[1] https://en.wikipedia.org/wiki/List_of_important_publications...
I did not mean my comment to say that seminal works in CS theory are useless, just that most of the books published in this field cover more practical matters and they tend to age very quickly. Like I have an e-bookshelf full of Packt freebies back when they did that, and those books aren't really that useful now unless I want to start on an old version of something.
By definition the "old things" you use would also be "just a big bunch of unnecessary complexity" since they were once new.
For me, the choice is a trade off between the journey and the destination.
Destination is the final objective of your project, business value or whatever.
The journey is tinkering with "stuff".
Depending on the project there's value and enjoyment in both.
[0]: https://mcfunley.com/choose-boring-technology
Doesn't fit in this case, I'd say Nix is still cutting edge.
It may also be simple, but it's not easy (in the Rich Hickey sense).
However, learning Nix / NixOS is quite difficult - it requires a specific set of skills (functional programming, shell, Nix building blocks which you can learn from existing Nix code), the documentation is lacking, error messages are cryptic, debugging support is bad.
The truth is tautologically unhelpful: choose good technologies.
Most people have trouble separating snake oil and fads from good things, so old/boring is a safe heuristic.
Nix is not boring, but it is honest. It's hard because it doesn't take half measures. It's not an easy learning curve gaslighting to you to a dead end.
But taste in choosing good technologies is hard to teach, especially when people's minds are warped by a shitty baseline.
This is a really good way of describing a certain type of technology. I'm stealing this line.
I was really pro Nix at the time I took on a rails project and it caused many problems for my team due to Nix just not having good support for my use case. If I could go back in time I would definitely choose something else.
The rails support was more of just to handle packaging existing software as opposed to being good for a development environment.
I've not used Nix so I legitimately don't know (but I am quite interested in it)
I absolutely agree language-specific package managers need better support. Nix RFC 109 is a simple-stupid way to get some "important from derivation" in Nixpkgs, so we can collectively dogfood them "lang2nix" tools and finally make them good.
It is really sad to me that many users understandly think "hmm, C is crazy, surely this Nix should be even better with more uniform language ecosystems packages!", only to die on the hell of the lang2nixs all being just not quite good enough. This is huge stumbling block for developement shops that try to use Nix that I want to see removed!
Typically, technologies I see described as boring have poor features for managing complexity.
Usually with poor reasonong justify meeting some silly definition of "simple“ and "boring" that ultimately makes your life harder.
So as you get older and you get more competent, the relative benefits of boring tech grow.
Whereas if you’re a noob at everything, being a noob with cool tech is better.
If the problems are hard enough, and you are competent, you iterate on boring tech for a while to solve the hard problems, and then it becomes cool tech.
Get the simple to understand, basic thing working and push the harder refactor / abstraction until later. Leave the fancy stuff for a future engineer who better understands the problem.
That future engineer is you with more experience with the actual problem.
In simple systems, you often have the option of simply throwing away entire components and remaking them because the system is simple and the dependencies between components are still clear.
Like, there's no point in scratching away at your own web CMS just to prove to yourself that you really do need Rails or Django. Especially when part of the purpose of a framework is to guide your design toward patterns which will fit with the common, preexisting building blocks that you are likely to end up needing. If you do the design separately, you risk ending up in a place where what you've built isn't really suited to being adapted to other stuff later.
For a humble example of this from myself, I built an installer system for Linux computers— initially very simple, just a USB boot, perform partitioning, unpack a rootfs tarball, and set up the bootloader. Then we added the ability to do the install remotely using kexec. Then we added an A/B partition scheme so there would always be fallback. It got more and more complicated and fragile, and the market around us matured so that various commercial and OSS solutions existed that hadn't when we first embarked on this effort. But the bigger and less-maintainable the homegrown system became, the more features and capabilities it had which it made it difficult to imagine ditching it in favour of something off the shelf. There isn't really a moral to this story other than it would be even worse if we'd built the in-house thing without doing a thorough evaluation of what was indeed already out there at the time.
With the average tenure of a software developer at a company being about 2 years, that future engineer is for sure some other poor soul.
Sure I can write a server in 5 lines of code but adding all that stuff is many many many person years of work and plus I'd have to go through all the learning pains of doing it wrong, crashing things, losing data, before I'd add it correctly.
I want some existing solution that domain experts already created that do things right.
It has nothing to do with "sexy/unsexy". It's unsexy to use UnrealEngine. It's sexy to write your own. But unsexy ships while sexy is spending time recreating stuff that already exists, unsexy is moving on to the actual content and shipping.
With new software one does not know if it is the one that will survive challenges from even newer software.
I'm in my late 50s and been developing for 35 of them. Honestly, containers are not rocket science and solve a mountain of ops-type problems simply.
If you really are only developing a single instance to be deployed in-house, then you may find you don't need them. But as soon as your instance might possibly get exposed to more than one environment, they're a no-brainer for me.
I have a semi-docker setup where I stay with the base OS + PostgreSQL and the rest is docker. But that means that upgrade the OS/DB is not that simple.
I tried before dockerize PG, but still the OS need management anyway and was harder to use diagnostics/psql/backups with docker....
We've been using it in prod for a couple years. There's a couple dozen production deployments outside the company as well. It's open source, MIT licensed. It doesn't have documentation yet, so it's currently only for people that don't mind reading the source or have talked to us directly about it.
It uses Nix because postgresql has C extensions, which can depend on anything in the software universe. Schema depends on extensions, so it's not technically possible to separate schema migrations from Nix without duct tape abd glue. So schematic is a sort of "distribution" of PostgreSQL that has a package manager (for extensions, schema, content, etc), and manages revisions.
If this is interesting to others here, I can do a "Show HN" post after getting the docs in order.
Kubernetes can absolutely host stateful applications. A StatefulSet is the app-level construct pretty much intended for exactly that use case. My company runs a distributed database on top of Kubernetes.
(We have another app that uses StatefulSets to maintain a cache across restarts, so that it can come up quickly, as it otherwise needs to sync a lot of data. But, it is technically stateless: we could just sync the entire dataset, which is only ~20 GiB, each time it starts, but that is wasteful, and it makes startup, and thus deployments of new code, quite slow. It would also push the limits of what an emptyDir can accommodate in our clusters' setup.)
(The biggest issue we've had, actually, with that, is that Azure supports NVMe, and Azure has AKS, but Azure — for God only knows what reason — apparently doesn't support combining the two. We'd love to do that, & have managed k8s & nice disks, but Azure forces us to choose. This is one of my chief gripes about Azure's services in general: they do not compose, and trying to compose them is just fraught with problems. But, that's an Azure issue, not a Kubernetes one.)
Reach out to dev@mixrank.com. Hiring globally for pretty much all roles, including junior roles.
Or take that problem to another level and use golang.
Instead, you use python and try to fix the problems with another immature solution: Nix, which has uniform builds... but not really in certain cases... and it is the same on every Linux... except not really so you kind of have to use NixOS.
That said, I would prefer managing that stuff through SDNs rather than trying to make user space overly complex software replication of commodity capabilities. Or, just use a cloud provider to start managing the isolation for the parts that benefit for it. What really baffles me is the use of k8s in aws. Why?
BTW - congrats on effective use of Nix. I use it extensively at home for my home services. It’s got some weirdness to it for sure but it’s a heck of a lot better than almost anything I’ve ever seen to date. I can’t wait to see what evolves out of it.
I remember it being pretty irritating to use, though, since it wasn't particularly easy to get Apollo to deploy to a desktop machine in the same way it would in production, and of course you couldn't isolate yourself from the desktop's installed dependencies in the same way. I'm using Docker nowadays and it definitely feels a lot smoother.
This is a nice writeup: https://www.allthingsdistributed.com/2014/11/apollo-amazon-d...
Really really smart idea that IMO helped Amazon in the 2010s immensely. While everyone else was figuring out k8s and whatnot, Amazon had a good system with CI in place for years.
I wonder how it's fared over time. Amazon was never known for internal tooling in many other places. I hope Apollo is still running strong today.
And yeah, the other thing that made it work, I guess, was having the machine image be very minimal, very tightly controlled, and very infrequently changed - so you didn't have to worry about things changing all the time due to the upstream distro.
The Guardian has hundreds of servers running, pretty much all EC2 instances. EC2 images are baked and derived from official images, similarly to the way you bake a docker image.
We built tools before docker became the de facto standard, so we could easily keep the EC2 images up to date. We integrated pretty well with AWS so that the basic constructs of autoscaling and load balancer were well understood by everyone.
The stack is mostly JVM based so the benefits of running docker locally weren't really significant. We've evaluated moving to a docker solution a few times and always reached the conclusion that the cost of doing so wouldn't be worth the benefits.
Now for a company that starts today I don't think I'd recommend that, it just so happen that The Guardian invested early on the right tooling so that's pretty much an exception.
This is an underappreciated point I think sometimes. Once you have a team which is familiar with your current, working setup, the benefits of moving away have to be pretty huge for it to be worthwhile.
Any reason why? It sounds pretty good.
For anyone who doesn't know about it: https://github.com/guardian/riff-raff
I think for a company starting out that AMIs (Amazon Machine Images), which from your description is probably what you're using, is actually a much better way to go than docker containers, because you get a large part of the orchestration for free with the AWS EC2 auto-scaling and health detection without most of docker complexity.
(I would suggest using something like Terraform to set it up in a reproducible way though)
I wouldn't necessarily recommend building this from scratch today, it was largely put in place around 8 years ago, and there are few compelling reasons for us to switch.
If you statically link all your binaries then your deployment system can be rsync a directory.
The only dependency is that the Linux kernel is new enough. Other than that the packages we deploy will run on literally any setup.
I don’t use python much but I don’t know why it wouldn’t be possible build it statically and package your entire dependency set into a single directory.
Stack looked like:
FreeBSD on bare metal servers (host service provided a base image, our shell script would fetch source, apply patches, install a small handful of dependencies, make world, manage system users, etc)
OTP/BEAM (Erlang) installed via rsync etc from build machine
Application code rsynced and started via Makefile scripts
Not a whole lot else. Lighttpd and php for www. Jails for stud (a tls terminator, popular fork is called hitch) and ffmpeg (until end to end encrypted media made server transcoding unpossible).
No virtualized servers (I ran a freebsd vm on my laptop for dev work, though).
When WA moved to Facebook infra, it made sense to use their deployment methodology for the base system (Linux containers), for organizational reasons. There was no consideration for which methodology was technically superior; both are sufficient, but running a very different methodology inside a system that was designed for everyone to use one methodology is a recipie for operational headaches and difficulty getting things diagnosed and fixed as it's so tempting to jump to the conclusion that any problem found on a different setup is because of the difference and not a latent problem. We had enough differences without requiring a different OS.
And now I feel self-conscious about my pile of AWS and Docker!
Of course, today, my fleet is down to less than ten hosts, depending on exactly how you count, and I'm usually the only user, so I can do whatever.
But this assumes the source of rsync is already static.
Handling dependencies is done in a build process, upstream.
Fetching dependencies at deployment time is a direct path to deployment failures (especially if the dependency management relies on external resources or non-locked versionning).
You can actually very simply run those tar.gz files without docker involved, just cgroups. But then you’ll have to write some daemon scripts to start, stop, restart, etc
Follow this path and soon you’ll have a (worst) custom docker. Try to create a network out of those containers and soon a (worst) SDN network appears.
Try to expand that to optimal node usage and soon a (worst) Kubernetes appears.
My point here is: it’s just software packaged with their dependencies. The rest seems inconsequential, but it’s actually the hard part.
K8s may be the "worst" orchestrator for you, and you may not actually need all or any of that complexity/functionality.
But starting with containers viewed as an RPM/tarball with some extra sauce (union filesystem and cgroups/jail) is a way better mental model than the whole "immutable infrastructure" meme to me.
Many services I write use only network resources, no disk.
https://fly.io/blog/docker-without-docker/
A container is a set of processes running from that image.