Seems like all the issues described here are with stateful application and running HA mongo. I don’t think these performance issues are really a result of K8s itself. I do agree backing stuff up in this situation isn’t easy. I think that’s why a lot of folks punt their stateful services to managed services.
People are getting confused, alot of open source software released by major players (Kubernetes, Next.JS (which is really just FBs take on react SSR), Microservice architectures), have no place in a company smaller than 100 people. But they use them anyways, and it ends up costing a ton of money for almost no business benefit.
These posts come up so frequently, people keep falling for buzzwords they don't understand and then start blaming the tech. Distributed systems are all cool and fun but use them as a last option when you have exhausted all other choices.
> I took a moment to calculate the cost of hosting my app on the K8s offerings of a Hyperscaler like AWS, Google or Azure just to figure out that this won’t be an option for me.
Clear signal that you can't justify the overheads. Cloud services are costly but in most cases you'll pay similar cost in terms of time and in some cases lot more if you pick something complex that you don't know about.
> Cloud services are costly but in most cases you'll pay similar cost in terms of time and in some cases lot more if you pick something complex that you don't know about.
Even just something like EC2 instances is costly - and a surprising amount of stuff in the cloud is just containers or EC2 instances, before redundancy even. Adding in specialized services makes things even more costly.
Problem is that nowadays it's hard to find a team that knows how to properly handle hardware - if you manage to get a handful of people that know how to do that cloud is only sensible for very few edge cases.
And as the article states - in many cases you probably don't even need all that fancy from K8s. I can for the cost of using K8s cloud stuff just put a second copy of our hardware in a different DC (or maybe even a third one), and just fail between DCs if needed. Way simpler setup, and excludes a lot of failure cases I'd catch with a K8s setup.
"Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications."
Do you see "not recommended for smaller-scale applications" anywhere? Including on the entire home page? Looking for "small", "big" and "large" also yields nothing.
> These posts come up so frequently, people keep falling for buzzwords they don't understand and then start blaming the tech. Distributed systems are all cool and fun but use them as a last option when you have exhausted all other choices.
I've been in the SaaS application business since the early 2000's. When I first started, the big infra question was running on in-house infra or data-center infra. If it was the latter, then the follow-up question was which hosting provider. Regardless, as an Engineer, you were likely touching something that looked and felt like a variation of the box sitting under your desk.
Eventually we got cloud providers, serverless, containers, etc. Growing through the progression of technologies, they all seemed like a natural evolutiion of things to solve certain problems with other approaches.
For newer engineers starting out, they may only have seen these latter things, and at the scales and circumstances that make them a good choice. As such, it may not be that they're the new and shiny thing, but that they're the _only_ thing people have seen.
K8s requires either a (small) ops team to run it or using a hosted cloud solution. It's powerful and pays off when you have this, but if you can't afford to pay for the cloud hosting or hiring a team to do it, you're not going to get the benefits, you're not the target market.
Many products have a minimum size at which they are effective, no one is buying an Oxide rack for their homelab or side project, no startup going through YC is buying an IBM mainframe, because they are both obviously nowhere near the minimum effective size/complexity/spend. K8s is the same, and yet it seems to trip up so many people.
Running a self-hosted Mongo cluster is tedious and complicated task by itself. Mongo Inc isn’t incentivized to make it easy and user friendly because they have a cloud offering for you.
In addition to that Kubernetes isn’t really tailored to stateful workloads, so there are all sorts of gotchas and surprises there.
Depends on what you're using your cluster for. When you're big enough it's worth just treating the database like any other job and running it on the cluster, but this can be tricky to get right. For small K8s deployments many people choose to outsource the database to their cloud provider and have it separate from the cluster as there are fewer things that can go wrong.
For all your data: yes. It means persistent filesystems and databases.
Kubernetes will still be perfect to manage your running apps, but data is "the rest of the owl" of most devops / cd proponents and tools. "I should be able to rollback to a previous working state", well I hope your last deployment did not change your database schema.
I use k8s in a very small setup (3 worker nodes, 3 ctrl planes) in my startup. But i'm working professionally with k8s for 6 years.
My setup is relative big but also relative simple and most of it is also IaC. I don't need to backup my setup and i don't need to document how i setup everything.
I have blue/green deployment, multiply enviorrnments (dev, test, prod) configured 100% equal, a good monitoring stack + tracing, s3 replication, postgresql ha and offside backup.
i can shut down a node and nothing happens.
But you know, if k8s doesn't fit your time and experience and benefit estimation, don't use it.
Why on earth did the author even believe he needed K8s to serve 500 concurrent users? You could do that on a couple basic web servers behind a load balancer 15+ years ago. Hell, you could probably run the whole app, including the database on a single server if you don't care about availability.
Building my app, deploying a container image and building a helm chart is not a lot of work if you have experience and the initial additional work is only happening once.
After that, you have your whole app definition in code, seting up a second or third env takes 5 minutes.
I really should write a blog article about my stack and the advantges of it (git, kubernetes, argocd, helm, ...)
I’ve read such insanely doctrinaire opinions here on HN like you must NEVER!!! ssh into a system, and instead have some rube-goldberg distributed contrivance for deployment and debugging and anything else. I presume this attitude is how we got to people running their phpbb blog in a 5 node kubernetes cluster.
A dedicated server from OVH running some processes under lxc and systemd can achieve 100% yearly uptime. Your hobby project isn’t Google.
If my backend home server dies. I will rebuild it from spare parts right away.
If one of frontend servers dies, nothing really happened. Ill spawn new one somewhere else (VPS, cheap dedibox, whatever).
Sidenote: I used to take pride with these kinds of uptime. I now see it as a significant risk. If I take over a machine with an uptime of more than a month, I assume that the machine cannot survive a reboot. Which means I spend a lot of time trying to figure out how to deploy that server with all the data it needs. These days there’s likely deployment systems in place that prevent some of the horrors I’ve seen, but there was a time in the not to distant past when you had to assume someone ssh’d into a box at least once a quarter to muck with something. If they didn’t reboot, there was no guarantee the changes they made would survive.
Well, it depends on model how you manage servers. Cattle vs Pets. And of course true for takeover, thats nightmare whatever it have uptime or not ;)
I have just several servers here and there so their are all pets to me. Im 99% sure, server would boot cleanly. The remaining 1% is HW failure, with, can happen.
Example: I have a software FW running on some older HW here w/ 2 disks in Soft RAID 1. I had to power it down for 2 days due to electric maintenance in building. What a suprise was booting it. One disk failed right away making creepy noise. Other was ok, but server was booting long, so I hooked up monitor to check things out. Well, second disk was NOT in good condition either, it started to spit I/O errors here and there. Anyway. I managed to boot FW to working state, but immediatly got spare HW and started preparing for replacement. Now it runs 3 disks in Soft RAID 1 :)
Oh I really like those comments. People think that keeping autoupgrades on without control keeps them secure. Right... And then you keep hearing stories about server wont go up because of update, or stuff breaks horribly. Not to mention brining new bugs to the table.
Its not that simple. I care what I install on my servers. Everything is carefully selected. Additionally, I follow KISS concept, so I run simple things that are managable.
Blue–green deployment is very interesting, where with deployment you swap between production and staging. This makes mistakes easy to revert, with downtime into the minutes. Downtime of minutes is very acceptable for startups/hobby projects.
Distributed systems are cool, but people I think got carried away with complexity. Hardest part here is active-active replication, but this is needed for very few who need true 100% availability and big scaling. For majority of people active-backup setups are perfectly fine. You can also run frontend-backend mode where frontend takes care of caching, passing write to backend nodes. I run couple of systems like this and lives good. Failure of frontend node is absolutly no brainer. Failure of backend node with active-backup is also easy. You just lose last not commited and not synced transactions.
> All I’m saying is: Choose your weapons carefully.
That's the right moral of the story to take away.
> Today, I’m back with a single machine that handles the peak 500 concurrent users
And this is what tells me k8s is probably not the right tool for the job. Something that runs on a single machine should absolutely run on a single machine... Unless if there's a pre-existing, funded and operated cluster you can tap into.
No abstraction comes for free, as the author mentioned you need to figure routing, backup, etc. You are paying a premium for the fact that you want to make that somewhat generic enough to fit with multiple use cases. The abstraction doesn't materialize for the operator of the cluster. It's there for the customers of that cluster - the devs who can now avoid to configure whatever the cluster is already handling.
We ran our project on Google's K8s for $76 a month for quite a while. They provide one free Master. There is no way, EVER, that I would host K8s myself for the size of the projects I usually work on. But having a panic in one of my services at 1am, and K8s auto-restarting the pod is amazing. Namespaces even let us run three separate environments for that same $76. We have a couple thousand users per day who are in the app all day, so not a large project. We use other services like GCP Cloud Storage, Pub/Sub, PostgreSQL, etc. that require next to no setup. We are up to 3 nodes in our small cluster and still costing us less than $350 a month.
K8s is repetitive. You create one YAML for a service and the rest are similar. Rolling releases are a simple script. As small as our company and product is, I think K8s has been one of our best decisions.
Some things like MongoDB can quickly triple your costs though. We were able to pick technologies that kept the cost really low and stack maintainable.
I run kubernetes as... a standalone node. I run it on a dedicated server with two disks on btrfs RAID1, with a subvolume for each pod that needs it (hostPath).
Not the minikube version, not the kubespray version: the kubeadm version, installed by hand and minimally tweaked (NodePorts 0-65535), with dual-stack networking support.
I have ~3 tenants, 5-15 pods each, and managing this configuration with docker was a steadily increasing pain for the last few years:
- I had docker-compose split-files for each service for each tenant to keep a consistent state, so I used a simple bash script to generate a really long docker-compose line to bring everything together, networking was a real pain
- Dual-stack support in docker does not exist/work in compose v3 to this day, but v2 has it
- Querying the configuration and state of a container was an exercise in debugging the output
- For certificates, I had to run letsencrypt/certbot externally (2 tenants with separate accounts for wildcard certs), but use traefik in docker-compose (the syntax pain was on top of the rest, of course)
Kubernetes simplified all the pain points above:
- I still have the service split-files, but I don't care about network addresses and links between pods anymore, just their names
- better isolation between all the architecture points, and clearer definition and integration
- IPv4&IPv6 native just working
- "describe" gets me all the information I need in a pretty format, "get type name -o yaml" if I need the configuration
- standard nginx ingress + cert-manager, and I can have multiple tenants without resorting to hacks
DBs are regular containers in the pod that requires them, just like any other container. Daily subvolume snapshots are small and effective to make sure the data is safe.
Kubernetes is a much better orchestrator than docker compose, availability is a plus if you really need it. You don't need to build an HA control plane plus 3 workers if you have no use for it.
42 comments
[ 2.6 ms ] story [ 79.0 ms ] thread> I took a moment to calculate the cost of hosting my app on the K8s offerings of a Hyperscaler like AWS, Google or Azure just to figure out that this won’t be an option for me.
Clear signal that you can't justify the overheads. Cloud services are costly but in most cases you'll pay similar cost in terms of time and in some cases lot more if you pick something complex that you don't know about.
Even just something like EC2 instances is costly - and a surprising amount of stuff in the cloud is just containers or EC2 instances, before redundancy even. Adding in specialized services makes things even more costly.
Problem is that nowadays it's hard to find a team that knows how to properly handle hardware - if you manage to get a handful of people that know how to do that cloud is only sensible for very few edge cases.
And as the article states - in many cases you probably don't even need all that fancy from K8s. I can for the cost of using K8s cloud stuff just put a second copy of our hardware in a different DC (or maybe even a third one), and just fail between DCs if needed. Way simpler setup, and excludes a lot of failure cases I'd catch with a K8s setup.
From the homepage https://kubernetes.io/:
"Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications."
Do you see "not recommended for smaller-scale applications" anywhere? Including on the entire home page? Looking for "small", "big" and "large" also yields nothing.
if one wants to run service with high availability (no downtime if server died), what are other choices then?..
I've been in the SaaS application business since the early 2000's. When I first started, the big infra question was running on in-house infra or data-center infra. If it was the latter, then the follow-up question was which hosting provider. Regardless, as an Engineer, you were likely touching something that looked and felt like a variation of the box sitting under your desk.
Eventually we got cloud providers, serverless, containers, etc. Growing through the progression of technologies, they all seemed like a natural evolutiion of things to solve certain problems with other approaches.
For newer engineers starting out, they may only have seen these latter things, and at the scales and circumstances that make them a good choice. As such, it may not be that they're the new and shiny thing, but that they're the _only_ thing people have seen.
There's a difference between an EC2 instance and EKS pricing.
Bare minimum, one control plane, zero usage, no workload, EKS starts at $873 per year.
EC2 starts at $36/year. t4g.small (2 vCPU, 2GB) is $145/year.
Many products have a minimum size at which they are effective, no one is buying an Oxide rack for their homelab or side project, no startup going through YC is buying an IBM mainframe, because they are both obviously nowhere near the minimum effective size/complexity/spend. K8s is the same, and yet it seems to trip up so many people.
In addition to that Kubernetes isn’t really tailored to stateful workloads, so there are all sorts of gotchas and surprises there.
With an operator, you get HA setup with switch over etc. basically for free.
I would prefer relativly cheap managed db over hosting it myself but its absolutly fine.
For all your data: yes. It means persistent filesystems and databases.
Kubernetes will still be perfect to manage your running apps, but data is "the rest of the owl" of most devops / cd proponents and tools. "I should be able to rollback to a previous working state", well I hope your last deployment did not change your database schema.
MongoDB is shitsoftware anyway.
I use k8s in a very small setup (3 worker nodes, 3 ctrl planes) in my startup. But i'm working professionally with k8s for 6 years.
My setup is relative big but also relative simple and most of it is also IaC. I don't need to backup my setup and i don't need to document how i setup everything.
I have blue/green deployment, multiply enviorrnments (dev, test, prod) configured 100% equal, a good monitoring stack + tracing, s3 replication, postgresql ha and offside backup.
i can shut down a node and nothing happens.
But you know, if k8s doesn't fit your time and experience and benefit estimation, don't use it.
The upfront cost of k8s is bigger though.
Building my app, deploying a container image and building a helm chart is not a lot of work if you have experience and the initial additional work is only happening once.
After that, you have your whole app definition in code, seting up a second or third env takes 5 minutes.
I really should write a blog article about my stack and the advantges of it (git, kubernetes, argocd, helm, ...)
A dedicated server from OVH running some processes under lxc and systemd can achieve 100% yearly uptime. Your hobby project isn’t Google.
Just go away and burn ;)
> A dedicated server from OVH running some processes under lxc and systemd can achieve 100% yearly uptime.
Sidenote: I used to take pride with these kinds of uptime. I now see it as a significant risk. If I take over a machine with an uptime of more than a month, I assume that the machine cannot survive a reboot. Which means I spend a lot of time trying to figure out how to deploy that server with all the data it needs. These days there’s likely deployment systems in place that prevent some of the horrors I’ve seen, but there was a time in the not to distant past when you had to assume someone ssh’d into a box at least once a quarter to muck with something. If they didn’t reboot, there was no guarantee the changes they made would survive.
I have just several servers here and there so their are all pets to me. Im 99% sure, server would boot cleanly. The remaining 1% is HW failure, with, can happen.
Example: I have a software FW running on some older HW here w/ 2 disks in Soft RAID 1. I had to power it down for 2 days due to electric maintenance in building. What a suprise was booting it. One disk failed right away making creepy noise. Other was ok, but server was booting long, so I hooked up monitor to check things out. Well, second disk was NOT in good condition either, it started to spit I/O errors here and there. Anyway. I managed to boot FW to working state, but immediatly got spare HW and started preparing for replacement. Now it runs 3 disks in Soft RAID 1 :)
Its not that simple. I care what I install on my servers. Everything is carefully selected. Additionally, I follow KISS concept, so I run simple things that are managable.
I have 2x 5$ VPSes on OVH for small side projects
>up 828 days
>up 1025 days
That's the right moral of the story to take away.
> Today, I’m back with a single machine that handles the peak 500 concurrent users
And this is what tells me k8s is probably not the right tool for the job. Something that runs on a single machine should absolutely run on a single machine... Unless if there's a pre-existing, funded and operated cluster you can tap into.
No abstraction comes for free, as the author mentioned you need to figure routing, backup, etc. You are paying a premium for the fact that you want to make that somewhat generic enough to fit with multiple use cases. The abstraction doesn't materialize for the operator of the cluster. It's there for the customers of that cluster - the devs who can now avoid to configure whatever the cluster is already handling.
K8s is repetitive. You create one YAML for a service and the rest are similar. Rolling releases are a simple script. As small as our company and product is, I think K8s has been one of our best decisions.
Some things like MongoDB can quickly triple your costs though. We were able to pick technologies that kept the cost really low and stack maintainable.
Not the minikube version, not the kubespray version: the kubeadm version, installed by hand and minimally tweaked (NodePorts 0-65535), with dual-stack networking support.
I have ~3 tenants, 5-15 pods each, and managing this configuration with docker was a steadily increasing pain for the last few years:
- I had docker-compose split-files for each service for each tenant to keep a consistent state, so I used a simple bash script to generate a really long docker-compose line to bring everything together, networking was a real pain
- Dual-stack support in docker does not exist/work in compose v3 to this day, but v2 has it
- Querying the configuration and state of a container was an exercise in debugging the output
- For certificates, I had to run letsencrypt/certbot externally (2 tenants with separate accounts for wildcard certs), but use traefik in docker-compose (the syntax pain was on top of the rest, of course)
Kubernetes simplified all the pain points above:
- I still have the service split-files, but I don't care about network addresses and links between pods anymore, just their names
- better isolation between all the architecture points, and clearer definition and integration
- IPv4&IPv6 native just working
- "describe" gets me all the information I need in a pretty format, "get type name -o yaml" if I need the configuration
- standard nginx ingress + cert-manager, and I can have multiple tenants without resorting to hacks
DBs are regular containers in the pod that requires them, just like any other container. Daily subvolume snapshots are small and effective to make sure the data is safe.
Kubernetes is a much better orchestrator than docker compose, availability is a plus if you really need it. You don't need to build an HA control plane plus 3 workers if you have no use for it.