Pretty accurate. Everything I've seen that doesn't scale only doesn't because it was written by idiots who read lots of blog posts about scalability.
I'm still convinced we could run our entire software platform off 5 EC2 instances and an S3 bucket. But the last 10 years of technical mismanagement turned this into a microservices shit show. Splunk gets more traffic than our clients do.
The last 5 years I found myself developing for companies having a bunch of nonsense "microservices" with totally ridiculous separation, which added more to their problems than solving them.
The common denominator of those companies is they all had no significant traffic, had their initial codebase developed by (unmanaged) amateurs, and were later managed by people inexperienced in software.
Due to the heavy resistance in all cases, I eventually became tired of proposing simpler solutions and the endless discussions around it, and accepted premature microservice architectures to be part of my infinite source of income.
My favorite microservices pattern is having lots of separate little services, so you get that sweet overhead and complicated debugging, with a shared database, so you can still enjoy systemwide single points of failure and obscure side effects on data.
Have those microservices use different patterns (language, framework, infra, etc), and outnumber your devs by a magnitude so everybody gets a piece of that sweet context switching, multiple times a day.
I was complaining recently to a manager at another company about a half-assed "microservice breakout" project in a previous job where some teams moved from one big service + one big DB to many big services + single DB and introduced fun new DB connection + perf issues... and was dismayed to learn that this other manager didn't even think "a bunch of services all talking to the same giant DB" seemed like best practices.
Have also seen this coupled with the microservices pattern of having all the microservices share a large common library, with all microservices built together and deployed together all the time. All the pain of shared code across teams plus the pain of shared DB plus the pain of debugging across completely arbitrary process boundaries.
This can actually make sense when you have loosely coupled services that mostly talk to the database and external systems, and not directly to each other, that have a common domain model (due to the shared database) and common helper libraries, and that you want to be restartable separately, and/or be distributed over several hosts. Of course, the process boundaries should be well-chosen and not arbitrary.
One time, I had to put together an architecture diagram for a big investor presentation (diagram of an unusual bespoke many-servers system).
Basically, get boxes on the diagram for each aspect the exec might want to talk about or answer questions about. So that they'd have something to point to, and investors would have some level of understanding, as well as confidence that we thought of and were doing something about each thing. (This is a bit different than the diagrams we were using to discuss and build it internally.)
So I mention to a colleague that I "just know", if a room of tech investors see the Web facet of this (even though that's the easiest technical part of the system), and this one part is all going through only one server (which is all it would ever need), someone is gonna be concerned that we might not know what we're doing about scalability, and I bet the exec doesn't want to get derailed debating that with them.
Colleague advised we go full "System Design Interview!", so I spewed all the unnecessary vapid things we could do, that some semi-technical person might be tuned to look for. Trying to modify the spew to be sensible enough (if encumbering overkill) that, if investors happened to have a hardcore veteran techie vetting (like one of the people who built AWS or Google from nuts and bolts), that they wouldn't become enraged with disgust, and kill me where I stand. The investor meeting seemed to go well.
In a different (real, internal) diagram, for networking architecture, I also broke out out some logical functions into distinct physical servers (some fixed, some variable) in an information/control flow compartment. Then I added a comment on the diagram, something about that these might be combined into a single server for efficiency, for smaller deployments, like customer on-prem. And, I didn't add, if our own data center suddenly needed to host a hugely-popular service for others, and we suddenly had enough hardware for that, the software would be ready (but we'd still only need one server in that one spot that a System Design Interview book-prepped person would think needed a load balancer and workers).
But you still need the thing that runs docker. What is that for you? And if you have to set that up anyway, why not install stuff at that level instead?
I think that even when you plan to just manually deploy dockers on raw hardware, it’s still worth it. There’s plenty of stuff you can ignore if you do so. Ex: do I test with exactly the same versions of libs/binaries that I have on production? You can also easily do rolling deployments even if you have a single machine. There’s plenty of value while staying reasonably simple. Not something you can say about k8s. It’s nowhere near simple.
I have an old PHP+C app which only runs on ubuntu 1804 and below
And another which requires 2004 plus
They both require physical access to the same hardware. I want them to coexist on the same physical machine (I can't add a second as I'm space and power constrained), and switch between them trivially and quickly.
Now I haven't actually done this yet, but my feeling is I can dockerise it, push the dev files through, and run one image or another depending on what I need in a given hour.
By default something running inside Docker is immutable and doesn't run standard system utilities. You have to make arrangements for logging, data persistence, daemon auto-restart, and so forth. It can all be done, but the necessity to do all of this differently is what I mean by paradigm shift.
I use it for some projects because it makes moving between servers really easy. If you use docker-compose and put volumes inside one directory, it involves only a few simple steps: install docker on a new machine, rsync the directory there, ssh in, and run docker-compose up. That's it. With a fast network (or small amounts of data) it can be done in 2-5 minutes. Doing it without docker requires you to maintain either 1k+ lines of Ansible playbooks (and test it regularly!), or spending a day configuring everything manually.
Full backups are also trivial, if you can afford some downtime: docker-compose down && restic … && docker-compose up. If you cannot afford downtime, the docker volumes go onto a ZFS or LVM partition.
Because it's nearly impossible to screw up a docker container in a matter the OS stops working.
I think you'll find that most devs don't want to have anything to do with operating system management and that docker images extract that away pretty quickly and easily. Just request a VM of a particular size and setup your containers on it. No jacking with terraform or whatever.
And also makes it easy to test with different database versions, gives you 5 minute onboarding for the new devs on team (run this docker compose and you have everything in place). Lets you test your software in exactly the same environment as in production. Switch between stack versions in no time. Etc. It really solves real problems. Not only enterprise level bullsht.
I've never seen a dev stack in docker actually work. What I see is the poor new guy spending several hours trying to get docker working and configuring private repositories that don't work, then nothing works when they finally get it up which results in the dude spending at least 2 more days trying to reverse engineer the shit show or consult little pockets of knowledge hiding in out of date confluence documents and other people's heads who are on holiday.
I’ve been in mostly dis functional software company. And … that’s not my experience.
I least not lately. 10 years ago : absolutely. But nowadays it’s a docker command or a git pull then yarn/gradle/whatever … and if that does not work something is wrong and a ticket is raised.
How so? Is it really that inconceivable that someone has never experienced those before?
From personal experience, I used to use TFS for version control for a few years before using git at my newer job. I also never once had to deal with yarn or gradle.
And I also know from experience, that you can make a unmaintainable mess in every build system.
Also, who is gonna teach the juniors how to use the tools, if everyone sets that knowledge as mandatory? Or do you expect everyone to learn it in their free time before starting any tech job?
> The step in those builds should works on developer machines. Is that too much to ask?
Quite often, yes. (Secrets that are only on the build server/shared database you cna't connect to without a barely working VPN and such)
> If the step in the build do not work? How did that get merged in the first place ?
Because there is no CI/CD pipeline to stop you?
There are tons of reasons why it can be extremely hard to get into a legacy project as a new person. And that's just setting it up. Ideally, this wouldn't be the case. But sadly, the real world sucks
Man, you’re bringing me back a long long while ago.
Last time I worked without reproducible builds was 2005-ish ?
My experience : once someone in the C-suite vicinity enforce a lock-down, things fall in place rather quickly.
Ex :
nothing get deploy manually. I don’t care if we miss the next prod push, we will catch the next one.
> the real world sucks
My work is as grounded in the real world as well. And I spend year contracting for bottom feeders company. Hard to do more real crap.
At some point is valuable to define priorities.
> Because there is no CI/CD pipeline to stop you?
There are tons of reasons why it can be extremely hard to get into a legacy project as a new person
Fine. But lack of validation should not be a reason. If no pipeline checks and no humans checks as well. Then you are deploying un-vetted software.
> And also makes it easy to test with different database versions, gives you 5 minute onboarding for the new devs on team (run this docker compose and you have everything in place).
It's eerie that I had this experience 5 minutes ago, in a stand-up onboarding meeting for a new colleague: they ran into issues when trying to launch the project through the IDE run configuration (IDE version, plugins, project configuration etc.) but the container based approach that I introduced just in case about a year ago just worked, the first time.
Provided, I actually went the extra mile and documented how to authenticate with the necessary private repos etc., so one could follow a short list of steps bit by bit.
> Lets you test your software in exactly the same environment as in production.
I'd say containers are good for this, but not perfect. Depending on what you're developing and how, you might still run into stupid issues: mostly in setups where you use Windows for your development box and bind mount files, the permissions will get screwed up, as will line endings.
I'd say the best possible developer experience nowadays is along the lines of:
- have GNU/Linux distro X as your developer workstation
- have the same GNU/Linux distro X as the base for your containers (a more cut down version, but with same package repos)
- still ship software in containers, ideally run those on nodes that have some GNU/Linux distro
That's why I primarily base all of my container images on Ubuntu nowadays, same OS as my workstation. It's not efficient or optimized, but it's good enough, predictable and mostly free of headaches: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo...
I can just install some software on the workstation and have it run basically the same as it will in the container, same install instructions and everything (though maybe less focus on clearing up package caches not to bloat container layers, which doesn't matter outside of those). Sometimes that is useful for local debugging or launching things directly with IDE integrations, as opposed to trying to get remote debugging working (though things like DBs, Keycloak, Redis etc. are still better in containers, I'm talking more about back end/front end tooling/runtimes).
At this point I'm basically unironically shipping my dev box, sans dotfiles. It doesn't scale, but it's the least painful dev and ops experience I've had in years. One can probably take elements from this for more sane and enterprise setups, too (such as focusing on GNU/Linux instead of Windows for development, or at least something with a similar file system).
> if you have a technology stack that's full of problems
This is every company older than 7 years whose primary business is not creating beautifully engineered backend software. Unless quality is somehow forced on them by regulation or contracts, then it's just 90% of companies.
There's so many promises with the idea of containers that simply are not filled and when they are it's the same complexity and caveats of a DIY solution.
It looks tidier, it's presented with the word "docker" everywhere but you're solving the same problems in almost identical ways. Take reverts; you need commits with tags, volumes with backups, etc.
Am I talking about doing it with or without docker there? Well, Both!
This is what you keep discovering - you're doing the same thing in a new way; it's a presentation technology, not a simplification or task reduction one.
A good article, but it also makes me wonder what happened to Ted Dziuba. He was everywhere 15 years ago and then suddenly he wasn't and I'm not sure why.
“My top issues are stopping illegal immigration, taxes and government waste — and the Second Amendment to the Constitution,” said the candidate during a recent interview.
I wish I could scream this at the top of my lungs at some of the places I worked. In one place, we spent more time adjusting and refactoring Terraform files than actually writing application code despite having about a thousand users, most of whom used the app once for a few minutes in the morning.
Cool take, come back to me on how well this thinking worked when you start landing Enterprise customers and you suddenly need to 10x your talent pool. You use bare metal servers? It will be a lot harder to find devops talent with k8s being a standard. You built your web app on a low code tool? It will be impossible to meet that edge case your big Enterprise client needs. Being scalable can be over thinking, or it can save you from losing customers, context matters and each startup is different.
I'm reminded of the system that ran on one bare metal application server and one database server (not including backups) that was replaced with 80 services and a 10x ops spend for production. It was also slower than the original system.
Build your monolith with modules and APIs and split your applications, not your services. If you need to split a module out, your API just has another transmission protocol instead.
Don't forget the geniuses who setup everything to run on GCP/AWS through K8S because hurr durr autoscaling when their needs are mostly static and the servers have to be up 24/7 anyways so they pay out the ass for mediocre servers. Go on, pay for your xlarge while I get 4 top of the line servers for the same price at hetzner. And with the savings, I can even pay a full time sysadmin.
I see it as a long term investment. In a few years many companies will be taking the opposite approach and transforming their 300 microservices into a couple of monoliths, because suddenly "simple systems are more scalable" or some similar claim. And from an engineer's POV it will be a great time to be in the industry.
I've more often been surprised how well applications scale in practice that don't follow the scalability guru's advice. Having worked at a couple of places that actually needed scaling, I've learned that monoliths written in Java, even using Spring boot (which still sucks imnsho, but for other reasons entirely), can actually handle quite a bit of load, if written even half-decently. (I've seen apps with upwards of 1000 req/s written this way, having p99 response times below 30ms)
I've also seen a lot of different setups: Minecraft's backend does absolutely everything right IMO, and their sub-10ms p99.9 response times showed as much as well, and I don't have to tell you Minecraft has quite a few users.
Or another client I won't name, that used microservices, Kafka message queues everywhere and the most complex architecture I've ever seen, all in the name of scalability, couldn't even calculate their p99 response times because they didn't even answer 99% of their requests of their requests (that is, >1% of requests were dropped), at loads of about 5 requests per minute.
Moral of the story: the big picture tech decisions aren't always as important as you might think, implementation details/quality can make or brake any architecture.
HA! I wanted to see "break" (as in fracture) any architecture, but I think "brake" (as in device for arresting or preventing the motion) works even better here — the details of the code increase 'friction' in the machine and prevent it's efficient motion...
Overall, I quite agree; the devil is in the details, and those details can swamp the 'architectural' decisions. Best to get both mostly right vs one 'perfect' and be ruined by overlooking the other.
Funny, the general sentiment about Minecraft's backend was the exact opposite back when I was doing server hosting a number of years ago. Maybe that's changed though.
At the time at least, it was very clear that Minecraft was written for single player in mind. Nearly everything ran in a single main loop, causing virtually every multiplayer server to only be able to utilized 1 core. This meant most servers couldn't scale beyond ~50 players with vanilla Minecraft. Community optimizations could take this to 200-300, but usually sacrificed something for those performance gains.
Then a ton of work went into workarounds like Bungee, which enabled the next-gen Minecraft servers to host thousands of players on a "single" logical server. Of course, the main game loop was actually still single threaded and those thousands of players couldn't truly interact with each other as you would expect with vanilla Minecraft.
I assumed they meant the auth servers or other services (or maybe realms), considering Minecraft's game servers are typically player hosted, total Minecraft players is irrelevant in that context.
You are correct! I was talking about Auth and related services which are hosted by mojang. I never worked with the realms team so I don't have any insight into their codebase, and my insight into the Auth services is dated a couple of years as well, but I don't imagine they've broken it.
Scaling monoliths by running multiple webapp instances is the easiest way to scale and works up to the point your shared database instance can't keep up. Your schema and queries matter more than anything else. I wouldn't call this "scale" which is technically correct but not what's usually meant in these types of discussions.
Cursing sometimes helps get the point across on subjects that are often cargo-culted. It's often done tongue-in-cheek to express some deep frustration. And it's fun. :)
Also see the motherfuckingwebsite series, The Best Page in the Universe, etc.
Scaling is just one type of optimization, right? So, one should really focus on getting a system working first and then use data to surgically scale the real bottlenecks (back-of-envelope calculation to identify scaling bottlenecks early on falls into the general category of using data to drive optimization).
96 comments
[ 3.8 ms ] story [ 171 ms ] threadI'm still convinced we could run our entire software platform off 5 EC2 instances and an S3 bucket. But the last 10 years of technical mismanagement turned this into a microservices shit show. Splunk gets more traffic than our clients do.
of unzipped data
:v
Do you really need those 5 nines?
If my bank can go down every week for patching, you probably can too.
What matters if they will pay someone else for that reliability they don't need.
The common denominator of those companies is they all had no significant traffic, had their initial codebase developed by (unmanaged) amateurs, and were later managed by people inexperienced in software.
Due to the heavy resistance in all cases, I eventually became tired of proposing simpler solutions and the endless discussions around it, and accepted premature microservice architectures to be part of my infinite source of income.
We ended up with 3 service. Exact same API is provided. It does not matter : nobody is using it anyway !
Have also seen this coupled with the microservices pattern of having all the microservices share a large common library, with all microservices built together and deployed together all the time. All the pain of shared code across teams plus the pain of shared DB plus the pain of debugging across completely arbitrary process boundaries.
Basically, get boxes on the diagram for each aspect the exec might want to talk about or answer questions about. So that they'd have something to point to, and investors would have some level of understanding, as well as confidence that we thought of and were doing something about each thing. (This is a bit different than the diagrams we were using to discuss and build it internally.)
So I mention to a colleague that I "just know", if a room of tech investors see the Web facet of this (even though that's the easiest technical part of the system), and this one part is all going through only one server (which is all it would ever need), someone is gonna be concerned that we might not know what we're doing about scalability, and I bet the exec doesn't want to get derailed debating that with them.
Colleague advised we go full "System Design Interview!", so I spewed all the unnecessary vapid things we could do, that some semi-technical person might be tuned to look for. Trying to modify the spew to be sensible enough (if encumbering overkill) that, if investors happened to have a hardcore veteran techie vetting (like one of the people who built AWS or Google from nuts and bolts), that they wouldn't become enraged with disgust, and kill me where I stand. The investor meeting seemed to go well.
In a different (real, internal) diagram, for networking architecture, I also broke out out some logical functions into distinct physical servers (some fixed, some variable) in an information/control flow compartment. Then I added a comment on the diagram, something about that these might be combined into a single server for efficiency, for smaller deployments, like customer on-prem. And, I didn't add, if our own data center suddenly needed to host a hugely-popular service for others, and we suddenly had enough hardware for that, the software would be ready (but we'd still only need one server in that one spot that a System Design Interview book-prepped person would think needed a load balancer and workers).
And another which requires 2004 plus
They both require physical access to the same hardware. I want them to coexist on the same physical machine (I can't add a second as I'm space and power constrained), and switch between them trivially and quickly.
Now I haven't actually done this yet, but my feeling is I can dockerise it, push the dev files through, and run one image or another depending on what I need in a given hour.
Full backups are also trivial, if you can afford some downtime: docker-compose down && restic … && docker-compose up. If you cannot afford downtime, the docker volumes go onto a ZFS or LVM partition.
If people want to use Docker without k8s you need to do at least some of what k8s does some other way.
For a single server compose sounds good.
I also imagine there are a few cloud native docker services that do the kubernetying for you, hiding the complexity.
I think you'll find that most devs don't want to have anything to do with operating system management and that docker images extract that away pretty quickly and easily. Just request a VM of a particular size and setup your containers on it. No jacking with terraform or whatever.
I least not lately. 10 years ago : absolutely. But nowadays it’s a docker command or a git pull then yarn/gradle/whatever … and if that does not work something is wrong and a ticket is raised.
Theres already a lotnof complexity in those "simple" parts
From personal experience, I used to use TFS for version control for a few years before using git at my newer job. I also never once had to deal with yarn or gradle.
And I also know from experience, that you can make a unmaintainable mess in every build system.
Also, who is gonna teach the juniors how to use the tools, if everyone sets that knowledge as mandatory? Or do you expect everyone to learn it in their free time before starting any tech job?
The step in those builds should works on developer machines. Is that too much to ask?
If the step in the build do not work? How did that get merged in the first place ?
Ideally, you would
> The step in those builds should works on developer machines. Is that too much to ask?
Quite often, yes. (Secrets that are only on the build server/shared database you cna't connect to without a barely working VPN and such)
> If the step in the build do not work? How did that get merged in the first place ?
Because there is no CI/CD pipeline to stop you?
There are tons of reasons why it can be extremely hard to get into a legacy project as a new person. And that's just setting it up. Ideally, this wouldn't be the case. But sadly, the real world sucks
Last time I worked without reproducible builds was 2005-ish ?
My experience : once someone in the C-suite vicinity enforce a lock-down, things fall in place rather quickly.
Ex : nothing get deploy manually. I don’t care if we miss the next prod push, we will catch the next one.
> the real world sucks
My work is as grounded in the real world as well. And I spend year contracting for bottom feeders company. Hard to do more real crap. At some point is valuable to define priorities.
> Because there is no CI/CD pipeline to stop you? There are tons of reasons why it can be extremely hard to get into a legacy project as a new person
Fine. But lack of validation should not be a reason. If no pipeline checks and no humans checks as well. Then you are deploying un-vetted software.
I assume familiarity with standards tools of the tech stack.
famous words.
It's eerie that I had this experience 5 minutes ago, in a stand-up onboarding meeting for a new colleague: they ran into issues when trying to launch the project through the IDE run configuration (IDE version, plugins, project configuration etc.) but the container based approach that I introduced just in case about a year ago just worked, the first time.
Provided, I actually went the extra mile and documented how to authenticate with the necessary private repos etc., so one could follow a short list of steps bit by bit.
> Lets you test your software in exactly the same environment as in production.
I'd say containers are good for this, but not perfect. Depending on what you're developing and how, you might still run into stupid issues: mostly in setups where you use Windows for your development box and bind mount files, the permissions will get screwed up, as will line endings.
Here's an example: https://blog.kronis.dev/everything%20is%20broken/containers-...
I'd say the best possible developer experience nowadays is along the lines of:
That's why I primarily base all of my container images on Ubuntu nowadays, same OS as my workstation. It's not efficient or optimized, but it's good enough, predictable and mostly free of headaches: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo...I can just install some software on the workstation and have it run basically the same as it will in the container, same install instructions and everything (though maybe less focus on clearing up package caches not to bloat container layers, which doesn't matter outside of those). Sometimes that is useful for local debugging or launching things directly with IDE integrations, as opposed to trying to get remote debugging working (though things like DBs, Keycloak, Redis etc. are still better in containers, I'm talking more about back end/front end tooling/runtimes).
If you go to this other post of mine, it actually has an example of this: https://blog.kronis.dev/articles/a-week-of-linux-instead-of-... (search for "Being able to run the same thing inside of containers and on the host system.")
At this point I'm basically unironically shipping my dev box, sans dotfiles. It doesn't scale, but it's the least painful dev and ops experience I've had in years. One can probably take elements from this for more sane and enterprise setups, too (such as focusing on GNU/Linux instead of Windows for development, or at least something with a similar file system).
This is every company older than 7 years whose primary business is not creating beautifully engineered backend software. Unless quality is somehow forced on them by regulation or contracts, then it's just 90% of companies.
It looks tidier, it's presented with the word "docker" everywhere but you're solving the same problems in almost identical ways. Take reverts; you need commits with tags, volumes with backups, etc.
Am I talking about doing it with or without docker there? Well, Both!
This is what you keep discovering - you're doing the same thing in a new way; it's a presentation technology, not a simplification or task reduction one.
I can relate. I had a similar epiphany, but for different reasons.
Build your monolith with modules and APIs and split your applications, not your services. If you need to split a module out, your API just has another transmission protocol instead.
As Paul Graham said, do things that don't scale.
-Emily
Angry dad leaned heavily on the “I’m gonna [X] my foot up your ass!” construction.
Getting people to give a shit is totally the harder problem.
I've also seen a lot of different setups: Minecraft's backend does absolutely everything right IMO, and their sub-10ms p99.9 response times showed as much as well, and I don't have to tell you Minecraft has quite a few users.
Or another client I won't name, that used microservices, Kafka message queues everywhere and the most complex architecture I've ever seen, all in the name of scalability, couldn't even calculate their p99 response times because they didn't even answer 99% of their requests of their requests (that is, >1% of requests were dropped), at loads of about 5 requests per minute.
Moral of the story: the big picture tech decisions aren't always as important as you might think, implementation details/quality can make or brake any architecture.
HA! I wanted to see "break" (as in fracture) any architecture, but I think "brake" (as in device for arresting or preventing the motion) works even better here — the details of the code increase 'friction' in the machine and prevent it's efficient motion...
Overall, I quite agree; the devil is in the details, and those details can swamp the 'architectural' decisions. Best to get both mostly right vs one 'perfect' and be ruined by overlooking the other.
At the time at least, it was very clear that Minecraft was written for single player in mind. Nearly everything ran in a single main loop, causing virtually every multiplayer server to only be able to utilized 1 core. This meant most servers couldn't scale beyond ~50 players with vanilla Minecraft. Community optimizations could take this to 200-300, but usually sacrificed something for those performance gains.
Then a ton of work went into workarounds like Bungee, which enabled the next-gen Minecraft servers to host thousands of players on a "single" logical server. Of course, the main game loop was actually still single threaded and those thousands of players couldn't truly interact with each other as you would expect with vanilla Minecraft.
Also see the motherfuckingwebsite series, The Best Page in the Universe, etc.