I think I misunderstand. You seem to say you're using cordon+drain to prevent a cluster from scaling down, and it saves money?
How that, it seems counterintuitive to me?
Imagine that kubernetes isn’t moving workloads to nodes that have spare space that would fit.
Or, think of it as defragging.
You basically have enough work for 3 nodes but it’s spread over 5 nodes with some spare capacity on each.
Kubernetes is not killing the extra workloads to consolidate it all on the 3 nodes it would have fit on; kubernetes only does bin-packing on new workloads.
This might be true for "stock" Kubernetes, but it seems likely that the grandparent post is using cluster autoscaling (since they mention "auto-scaled up"), for which this post is not accurate.
The cluster-autoscaler component does remove nodes with low utilization and Pods fitting onto other nodes, effectively defragging the cluster. There's just plenty of reasons for it to not drain a node (as per its FAQ[1]).
But it's highly configurable, so I wonder if the grandparent post just needed some configuration changes to suit their scenario better.
Most autoscaling functionality is implemented on top of the upstream cluster-autoscaler component, and GKE documentation claims to remove nodes when pods fit onto other nodes as well[1], so it does have the same "defragging" functionality. It might not be configured aggressive enough for their liking though (but even then, GKE has a more aggressive autoscaling profile that might do the trick).
My point is that while the original post is reporting problems with autoscaling in Kubernetes without enough details to go by, cluster-autoscaler is fairly good at its job of downsizing clusters if you take the time to a) optimize its configuration and b) make sure your workloads are configured in a way that allows downsizing clusters.
In particular an ephemeral directory mounted on e.g. /tmp counts as "local storage", so in common configurations nothing will get evicted until you work around it by adding "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" to all deployments.
You can really get upvotes on HN for linking to common Certified Kubernetes Administrator concepts here? Y'all need to take the practice exam at https://killer.sh if this is considered noteworthy. It'll blow your mind, if not make my job easier with you all being able to help.
These posts give folks the opportunity to complain about how complicated kubernetes is. When else will people get to describe their ansible + docker compose production environment managing 10,000+ nodes "flawlessly" ?
Yeah I really don't trust most devs outside the space to make good infra decisions at this point if these extremely common concepts like cordon and drain to your average SRE is considered news worthy or worthy of being mocked to them. Just hire a SRE like you were supposed to in the first place or god help them when they actually decide they need one at their definition of "scale."
The article lays it out into two steps, with "Step 1" being to `cordon` and "Step 2" being to `drain`. My understanding was that `cordon` used to be required to (safely) drain a node, hence why so many tutorials still show it, but that on newer versions of K8s, draining a node will automatically cordon it as well.
Of course, if you want to keep the node up and running, but evict the pods, then absolutely that is when `cordon` _should_ be used.
If you want to drain 3 nodes as fast as possible it's best to start by cordoning all 3 of them and after that run drain on each respectively. This will cause minimal unnecessary interruption of pods. What we want to avoid is that new pods start on nodes that we soon want to remove/drain.
Is `drain` still a kubectl only construct? I was looking for an API to do this once upon a time and it seemed all the logic for this command lived in the CLI only.
Yup, as of 1.21 drain was still basically a `kubectl` thing – seems like they provided an API for Evictions and for some reason decided not extend that to offer a full 'drain this node' API. That said, Metal3 controllers provide some of this functionality in case you need to manage a bare-metal worker cluster.
22 comments
[ 3.1 ms ] story [ 25.6 ms ] threadPretty nuts to think I’ve saved six figures in VM costs just doing that once or so a month.
Or, think of it as defragging.
You basically have enough work for 3 nodes but it’s spread over 5 nodes with some spare capacity on each.
Kubernetes is not killing the extra workloads to consolidate it all on the 3 nodes it would have fit on; kubernetes only does bin-packing on new workloads.
The cluster-autoscaler component does remove nodes with low utilization and Pods fitting onto other nodes, effectively defragging the cluster. There's just plenty of reasons for it to not drain a node (as per its FAQ[1]).
But it's highly configurable, so I wonder if the grandparent post just needed some configuration changes to suit their scenario better.
[1] https://github.com/kubernetes/autoscaler/blob/master/cluster...
I can’t be sure which one the GP is referring to, but GKE definitely operates the way they described.
My point is that while the original post is reporting problems with autoscaling in Kubernetes without enough details to go by, cluster-autoscaler is fairly good at its job of downsizing clusters if you take the time to a) optimize its configuration and b) make sure your workloads are configured in a way that allows downsizing clusters.
[1] https://cloud.google.com/kubernetes-engine/docs/concepts/clu...
Silly default IMO but apparently "intentional".
https://github.com/kubernetes/autoscaler/issues/2048
Draining a node will allow it to shut down.
Of course, if you want to keep the node up and running, but evict the pods, then absolutely that is when `cordon` _should_ be used.
If you want to drain 3 nodes as fast as possible it's best to start by cordoning all 3 of them and after that run drain on each respectively. This will cause minimal unnecessary interruption of pods. What we want to avoid is that new pods start on nodes that we soon want to remove/drain.