I have completely given up on anything related to stateful applications in Kubernetes. Most of the time it can work okay but the amount of headache's we've experienced with it we've completely abandoned using Kubernetes for anything isn't stateless and requiring dynamic scaling.
Not OP, but I've been doing it for a few years. Can't say I've ever seen an issue - with Prometheus Operator, which the article mentions, as few times as well
Online resizing of pv has been possible for quite a while.
The complaint from the author is that you can do this for any existing pv that was created by the sts pvc definition, but scaling up the sts will create a new pod and pvc using the original spec. Altering that spec in the sts manifest is unpleasant when it really shouldn't be.
If you need stateful files in a microservice, just mount a network filesystem. Yes, it's slow. Yes, it's buggy. Yes, it's not very portable. Yes, there are locking issues. But if you really need stateful files in a microservice, something is already fucked up. Network filesystems are a simple and cheap way to add that functionality and more. You get 'infinite' storage, redundancy, persistence, shared volumes, intra-cluster data availability, centralized data lifecycle management, etc, and no need for more custom Kubernetes logic just to persist some files. It's a shitty solution that is actually good enough.
I actually think k8s is (or can be) a decent solve for non-critical, low cost databases if you're confident about navigating the toolchain. The one big gap is storage resizing, which is part of why we're pointing it out.
I'd double down even more on your latter comment of "just dont use k8s" for stateful needs as its a minefield. But mounting a network fs is just gonna create a whole host of headaches when they decide to use the mount with anything that doesn't have native io fencing etc.
NFS doesn't have to be slow and buggy, its just that people try to set it up on a buggy cloud provider, and get burned. Tons of massive, mission-critical bare metal VMWare infra is backed by NFS and something like a NetApp. The one I used to admin years ago was connected via redundant 40Gb ethernet and it could easily push many GB/s at very consistent and low latency, and would fail over instantly without interruption.
This is more a criticism of operators than StatefulSets. The promise of operators is easier automated management of your resources. If the operator can't do basic things like resizing disks, it's flawed and needs to be fixed.
However, I've always been suspicious of operators, so maybe my bias is leaking here. Kubernetes is already a level of abstraction and adding another seems risky and unnecessary.
The flaw is definitely most acute when it is combined with operators, but it would make storage resizes vastly more accessible for less capable devs if it were able to be done declaratively via the spec instead of via a painful workaround.
I agree with this. I manage a monorepo of 10s of terraform/terragrunt modules most of which are deployed to a k8s cluster.
the whole point of code as infrastructure is that your git repo is an audit log of whats changed in your infra. Once you leave it up to operators or persistent volume claim templates (or whatever else makes side effectful changes with being told) you're throwing all that niceness of control out the window.
Infra as code can't actively manage things like database clusters, handling promotion of a secondary, automatic deployment of new replicas, scaling. Using it to deploy RDS Aurora doesn't count. My impression is that operators are designed to handle those "active" decisions in real-time, and that's outside the realm of terraform, GitOps etc.
Indeed, and if the k8s api was so precisely laid out that you could look at a log of side effect changes that came as the result of scalers, operator, templates etc that would be one thing, but we aren't there yet.
Right now what i have is "oh the cluster is in a different state (there is drift) than the code, lets be a detective to see why this changed".
To be fair there are tools that use operators (like Flux) to apply desired state onto Kubernetes clusters using Git, so the GitOps is still there. You can even combine flux with ClusterAPI to create clusters with CRDs through Git, which can be quite handy
As long as the operator is really well designed but it is rarely the case. I switched from helm to bare k8s manifest for the same reason, helm/operator can be helpful, but not when the maintainer introduces breaking change (that does not relate to the service by itself) that needs a delete/deploy of the k8s resources
To be fair Helm is just a template layer for manifests, so it would take quite a bit of effort to break things.. especially if you're using a projects official Helm charts.
Now that you've hit cloud scale, engineering leadership can make the case for budget for an ops hire to keep the thing working. Bigger headcount is always a plus.
Also make sure that Dev and Ops are separate departments that are not allowed to work together. Devops? We don't do that here. BTW you need to fix that 15-second query ASAP.
Listening to the empire builders among engineering management, they'll happily tell you that a few more hires will get the current baby back on schedule plus future babies can be done in 8, maybe even 7, months.
An operator could trivially create the pvc ahead of any sts scaling op. The sts will simply look for an existing pvc by a predictable name and use it rather than create a new one, regardless of whether the pre-created pvc matches the spec of what the sts defines.
I like the idea of operator pattern (Operators aren't a built-in resource yet; they REALLY should be) but absolutely HATE how opaque they are. You really don't have a window into seeing what they are doing; often times (in my experience), you only know things are off much later on (because most operators rely on Kubernetes events to notify human operators that things have gone wrong).
I cannot understand why people in this thread are saying people who want state are holding it wrong. If you are saying that k8s falls down on state — literally the only hard problem — then why are you bothering take on the complexity of k8s? What are you running on k8s that is stateless and can’t be served by ec2 instances in auto scaling groups? State management is the killer feature for k8s; it manages all the complexity and presents you a fiction where stateful apps can be written like they’re actually stateless.
People are definitely sleeping on k8s' capability in managing stateful workloads. We run various datastores on it all the time, and it's quite reliable for a lot of use cases (largely because the underlying cloud block stores it's orchestrating are incredibly robust).
That said there are warts that should be removed, but that's not surprising.
The vast majority of developers don't understand what "stateless" means, since if you want to get technical is no such thing as a stateless system.
You might have drained requests off of a pod, but it still has a "state", it still has things cached in memory, it might still have connections open to some outside entity (Database, whatever), and the developer, not K8s, is responsible for catching those signals, cleaning up things in memory, gracefully terminating connections, handing off in-flight workloads to another pod, etc. Even in the upper echelons of the tech I see a very small minority of developers actually aware of all the things that can make stateless workloads stateful. Which is OK for something where the stakes are low, but if you're a DBA or a Systems oriented person you'll see people make these (very wrong) assumptions all the time and recoil in horror.
And that's why the title DBA is going the way of the dodo; state in this instance does not include cache and connections, because those are theoretically idempotent.
Theoretically is not actually, and a ton of the assumptions in k8s break down when you look at them super closely, but they break down when things have gone wrong. If your cache is properly designed it doesn't matter if it's a little stale; you're not using it for things where correct and up to date is the most important thing. Connections, too; it shouldn't matter if your connection pool needs to spin up another connection, because if any of the conditions exist that the connection fails (or takes too long and stalls requests for any appreciable time), you already should have been paged and be on the way to resolving the incident.
This totally misunderstands the motivation for statefull sets. The only thing they offer is the stable per-task names. They are a reimplantation of jobs in borg where you always have stable task numbers. It's really more for shared services then ones with persistent state.
When do you not need automated rolling updates? Or stable network identifiers? I'm sure there are cases, but it seems like supporting them should be the default -- are statefulsets somehow expensive?
Consider a job of N identical API servers. Your request can go to any of them. To update we could just add new tasks and take away old tasks. It's a gradual update, but not "rolling" is the sens that each task name has a single love instance at a time.
A basic stateless web app behind a load balancer can tolerate unstable network identifiers as long as you automate registering/deregistering the new names, which kubernetes services do very well.
Stateful apps can be a bit more exceptional because they often have client libraries that require you to hardcode network addresses via connection strings and other tooling that makes that glue automation more tricky and worth the added guarantee around consistent naming.
>"When do you not need automated rolling updates? Or stable network identifiers?"
Most of the time with stateless apps.
In regards to automated rolling updates, the key word there is "ordered." It will always select the same ordinal index first(reverse order starting with highest ordinal.)[1] This is different than the regular deployment strategy of "rolling update" which is also one by one but will select any pod to whack first.
Regular pods don't have stable network identifiers as they are inherently unstable. In regular pods stable network identifiers are provided by either the cluterIP or a load balancer service.
> Persistent storage is its own API, you don't need to use it with a satefull set. All of the other properties are implied by stable task names.
I believe that's only partially true, since even if one has a PVC in something like a Deployment (the antipattern I see *a lot* in folks new to k8s trying to run mysql or whatever), then "k scale --replicas=10 deploy/my-deploy" will not allocate new PVCs for the new pods
That's why I think StatefulSets are valuable as their own resource type: they represent _future_ storage needs
Persistent storage tied to a specific member of the deployment is a StatefulSet feature. That's likely what they mean by "stable", although I really wish they were more clear about it
Your comment makes no sense. Persistent storage is one component of a stateful set. You are not even using correct Kubernetes terminology, I have never once seen the term "stable task name."
Sometimes operators interfere with what you're trying to do, as the article points out the Prometheus operator. Pro tip: Scale down the operator, make your changes, put everything back the way the operator expects it, scale operator back up.
I had to do a MongoDB migration and I was glad we had originally deployed it using the standard k8s yaml manifests instead of operators. It was a layer of complexity removed.
Sparse block storage is one of the cloud features I miss most. What I'd expect:
* You can create a volume, but pay only for in-use blocks.
* You create a volume bigger than what you'll ever need => no resizing required (At least 16 TiB, but possibly even something crazy like an exabyte)
* There is some kind of "trim" operation that marks a block as free
* Ideally it's possible to choose the maximum block index (size of block-volume) independently from the maximum number of in-use blocks (for cost control)
Great feature, but cloud providers are incentivised to make it difficult for people to save money. This would instantly vaporise a big chunk of revenue, because people would not have to pay for pre-provisioned storage.
Amazon historically has gone after 'low margin' businesses. The cloud, at 25% margins (for AWS) is is 'low margin' compared to traditional software sales.
> Great feature, but cloud providers are incentivised to make it difficult for people to save money.
Every business is incentivized this way. Competition is what pushes prices down, of which there is plenty in the cloud industry (though maybe still not as much as we would like).
AWS EBS volumes used by a PVC can be expanded without downtime by increasing the request on the PVC. The only prerequisite is that the StorageClass has allowVolumeExpansion=true
Edit: Now that I actually read TFA I see it mentions most CSI drivers already provide this functionality. They provide a workaround similar to what I’m sure most people use, and I agree this functionality seems like it could be handled by StatefulSet.
Yep, the cloud long term is a terrible investment. Due to its sticky nature innovation in technology is seldom passed down to consumers and instead is pocketed by the provider.
I guarantee you internally cloud providers over provision their storage.
i read a story on HN recently about someone having left a `nohup yes > /some/efs/mount/file.txt` type command running, and not noticing until the bill came due...
since then, I'm totally on board with pre-provisioned storage. at the very least, some way to put an upper bound on volume size (maybe efs has that? not sure.)
For production software, you should always have an upper limit on everything.
There is always a point where you can tell in advance that it doesn't make sense to keep going. It is never sensible to literally go on forever with anything -- it can only break stuff in annoying ways when it runs into real world limitations (like financial ones in that case.)
I sound aggressive about this because it's such a common mistake. It always goes something like,
"Why does this list have to be unbounded?"
"Well, we don't want to give the user an error because it's full."
"Okay, but does it really need to support 35,246,953 instances?"
"Sure, why not?"
"How long would the main interaction with the system take if you stress it to that level?"
"Oh, I don't know, at that level it might well take 20 minutes."
"And the clients usually timeout after?..."
"5 seconds."
"Would the user rather wait for 20 minutes and then get a response that might be outdated by that time, or get an error right away?"
"They may well prefer the error at that point."
"So let's go backwards from that. Will it ever make sense to support more than 250,000 instances?"
"That corresponds to the five second timeout and then some. I guess that's fine in practise..."
Often limits need to be addressed at the product level. At some point at my $work we started pushing hard enough back at product to say that we're placing a hard limit on every entity. What the limit is can be negotiated, re-evaluated, and changed -- but changes to it need to be intentional and done with consideration to operational impact.
My personal pet peeve is when i'm dealing with a client library that doesn't expose some sort of timeout.
I'd also add unbounded queues to the list above as a subtle place where the lack of a limit can really cause production issues. Everything may look like it's working fine until you realize that you've got a 4gb process with a giant buffer that'll take forever to drain.
"Where is such block storage available? Seems fantastical...?"
SAN vendors have been selling devices that deliver thin provisioned network block devices you can snapshot, clone, resize dynamically and replicate in a cluster for about 20 years now. Third parties can do all of this in cloud environments as well. See Netapp Cloud Volumes. You could roll your own using Ceph, or even a ghetto version with a subset of these capabilities with thin provisioned LVM+iSCSI, DRBD etc.
You should be able to point and click a thin provisioned, dynamically expandable, snapshot-able, clone-able, replicate-able pay as you go network block device with whatever degree of performance, availability and capacity you wish to pay for. The fact that you can't is a function of the business model and mentality of IaaS cloud operators; they don't want your old fashioned 'stateful' workloads. Neither does kubernetes. So you're swimming against the current.
Sparse block storage is achievable with s3backer and similar tools which treat bucket/object storage as arbitrarily large volumes.
Practically, sizing thin-provisioned filesystems too large wastes a lot of space on filesystem metadata.
All operating systems and clouds that I've worked with support online volume and filesystem resizing which is as close to the semantics of thin-provisioning as I've ever needed. I wasn't actually aware that StatefulSets don't resize volumes as smoothly as desired but I haven't had a problem manually deleting pods to get the filesystem on a PVC to grow (the way it works for Deployments).
k8s is made for stateless servers. But "stateful sets" has always seemed like a bolt on.
Also, for important "big data" stores with replication, I don't want to trust all-in-one third party operators, which is what all K8S operators seem to aspire to be. Your data is important, and you should evaluate all your data operations use cases yourself.
As someone who's been eyeing Kubernetes as a learning experience, that actually is really disheartening to hear. If you're being forced to manage your database outside of Kubernetes, what are you using Kubernetes for? Running your web/worker servers?
My mid-sized shop uses managed offerings for MySQL and Postgres but we run several stateful workloads, including ElasticSearch and MongoDB, in GKE.
I personally haven't experienced the issues the OP is describing. It might be that StatefulSets and PVCs are more stable now then when OP tried them. Of course using managed/hosted K8s makes a big difference.
We use managed K8s. And we dropped StatefulSet maybe 4 months ago.
There might have been a way to support our use case better. But this was managed service and we couldn't easily tweak every settings. But also, we didn't want to.
Our goal of using Kubernetes is to make difficult stuff easy. Not to make easy stuff difficult. And in our skillset and capacity moving stateful services outside K8s was easier, cheaper and more reliable.
if you're interested in it as a learning experience, this is a great setup. start by putting the stateless parts of your application into k8s, to get the normal flow. then try and move your database in . stateful workloads are tricky and maybe impossible to really do well in k8s, but there are a lot of options that almost get you there, and maybe get you to "good enough." and the learning you will get is fantastic for understanding lots of core k8s capabilities.
I don't have a lot of experience with production k8s so I haven't seen too many issues. I have seen PVCs get stuck though. Recently, I was unable to do a rolling restart because I messed up and used zonal disks on a regional cluster. The scheduler would assign pods to nodes in the "wrong" zone and then the disks could never be attached properly.
The fix was to add scheduler hints that moved the pods to the correct zone as I should have done in the beginning. (On first deployment, the disk is created on the correct node and thus zone, so it all seemed to work).
9/10 times stuckness is a bug in particular csi or node image that you use. Those are fixable. The zonal scheduling thing I believe was fixed a while ago
Ah, I've been on 1.19 for a while so I'm definitely behind. I'm using GKE regional clusters and as such cannot do zero downtime upgrades to the control plane (they break DNS) so it's risky upgrading.
Hm the topology-aware provisioning should be fixed since at least 1.12/13 and is working well on our self-managed regional clusters. I'd bug GKE support for this...
Just curious - how does GKE control plane upgrade break DNS?
I don't have access to GKE support, unfortunately, so the best I can do is submit product feedback.
The cluster's DNS server refuses connections briefly during control plane upgrades or updates. I haven't tried their node local DNS yet (enabling it will cause downtime, natch), maybe it'd help.
PVCs getting stuck in Pending is THE WORST. Sometimes Kubernetes events can tell you why (i.e. no PV matches the claim, PV out of capacity on Kubernetes end), but sometimes it doesn't and you need to find the Pod running the StorageClass to see what went wrong...which can be painful (thank you, NetApp Trident, for making me spend 30 minutes going all over tarnation to figure out why PVCs were stuck only to tell me subtly through a billion lines of logs that the trident-csi couldn't log into Azure)
Either way, this is a good example (for me) of how Kubernetes needs some work to be friendlier to operators
I just looked at the scripts for the last StatefulSet thing I did, and it has the exact workaround in the apply script. I.e. recreate the ss, edit the pvcs, and restart the rollout when there is a volume size change.
K8s has really been an experience of encountering 2-4 year old bugs (closed by the stale bot of course) and missing features. We accumulate increasingly more workarounds and additional pieces of complexity to deal with this. Everything is always changing, but somehow still staying the same.
This fundamentally untrue:
“But, Kubernetes was originally intended to act as a container orchestration platform for stateless workloads, not stateful applications.”
Even from the early days, there were constructs to allow stateful applications to be built on top of it.
“Stateful applications such as MySQL, Kafka, Cassandra, and Couchbase for a while, the introduction of Pet Sets has significantly improved this support.”
Generally, from teams we talk to at Coherence (withcoherence.com), it seems to be a mistake to try to run stateful workloads in Kubernetes. As much as the cost savings compared to managed services like AWS RDS or GCP Cloud SQL are attractive, the configuration time and maintenance are just not worth it. Our perspective is that container orchestration tools should be used to manage stateless workloads, and that stateful workloads should delegate to managed services where volume reliability, scalability, and backup can be handled with purpose-built tools.
These are valid points. I’ve personally needed to do things like delete the StatefulSet, change volume size, update/change pvc, update StatefulSet, reapply. I just took it as a given and hadn’t thought much about the ergonomics.
It’s really not that big of a deal and probably a ten minute process, tops. That said, OP makes a good point that mechanisms to handle this kind of stuff exist in CSI, so why not leverage / take advantage of that versus pissing in the wind (especially when you run into CRD that wants to undo the above work while you’re in the process)?
At the time we designed the StatefulSet, the resize behavior hadn’t been designed / finalized.
Now that it has, it’s really a matter of someone having the time and patience to drive it though. I don’t know whether someone has sorted out all the design implications (like what happens if someone adds new resource requests, or what happens if a new resource request is rejected, and how that gets reported to the user), but it’s a small change that needs a fair amount of design thought, which tends to be hard to drive through.
StatefulSets are introduced as "when you need persistent storage" and not a lot more than that. A key idea that is often skipped is that k8s doesn't want to presume how a cluster of "database" nodes ("database" in the wide sense, including RMQ, Redis etc) coordinate their HA.
So when a k8s node dies with a sts on it, unlike other pods, the sts pod is NOT recreated (not wanting to interfere with whatever mechanism the db cluster has to deal with it). I've found most people working with k8s don't know about this behaviour.
Managing state is hard and always has been regardless of the underlying technology.
I have been using Kubernetes for a few years now and whenever I see people putting manual steps they should instead be using an operator. What the author is describing as a problem has a solution (they even added it to their own operator).
Kubernetes has a steep learning curve but if you want to do nontrivial thing like have state than you need to learn nontrivial things like operators.
The default controllers that come with Kubernetes do not cover everything under the sun and they do require extensions via the operator pattern for some tasks.
The Kubernetes authors were explicit in stating that stateless workloads were their design target, and it is somewhat obvious that StatefulSets were a concession, so this isn't terribly surprising to me.
I have a feeling that OpenShift handles stateful workloads more gracefully due to their enterprise-oriented customer base, but I have nothing to back that.
100 comments
[ 0.25 ms ] story [ 158 ms ] threadIf the PVC size changes Kubernetes automatically does an online resize of the filesystem: https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/
This has been possible for 2-3 years if you had the flag enabled.
However, I've always been suspicious of operators, so maybe my bias is leaking here. Kubernetes is already a level of abstraction and adding another seems risky and unnecessary.
the whole point of code as infrastructure is that your git repo is an audit log of whats changed in your infra. Once you leave it up to operators or persistent volume claim templates (or whatever else makes side effectful changes with being told) you're throwing all that niceness of control out the window.
Right now what i have is "oh the cluster is in a different state (there is drift) than the code, lets be a detective to see why this changed".
So you start debugging the helm chart, and kustomize, and the operator.
Not fun.
You can't make a baby in 1 month with 9.5 women.
That said there are warts that should be removed, but that's not surprising.
You might have drained requests off of a pod, but it still has a "state", it still has things cached in memory, it might still have connections open to some outside entity (Database, whatever), and the developer, not K8s, is responsible for catching those signals, cleaning up things in memory, gracefully terminating connections, handing off in-flight workloads to another pod, etc. Even in the upper echelons of the tech I see a very small minority of developers actually aware of all the things that can make stateless workloads stateful. Which is OK for something where the stakes are low, but if you're a DBA or a Systems oriented person you'll see people make these (very wrong) assumptions all the time and recoil in horror.
Theoretically is not actually, and a ton of the assumptions in k8s break down when you look at them super closely, but they break down when things have gone wrong. If your cache is properly designed it doesn't matter if it's a little stale; you're not using it for things where correct and up to date is the most important thing. Connections, too; it shouldn't matter if your connection pool needs to spin up another connection, because if any of the conditions exist that the connection fails (or takes too long and stalls requests for any appreciable time), you already should have been paged and be on the way to resolving the incident.
StatefulSets are valuable for applications that require one or more of the following.
Stable, unique network identifiers.
Stable, persistent storage.
Ordered, graceful deployment and scaling.
Ordered, automated rolling updates.
[1] https://kubernetes.io/docs/concepts/workloads/controllers/st...
When do you not need automated rolling updates? Or stable network identifiers? I'm sure there are cases, but it seems like supporting them should be the default -- are statefulsets somehow expensive?
Stateful apps can be a bit more exceptional because they often have client libraries that require you to hardcode network addresses via connection strings and other tooling that makes that glue automation more tricky and worth the added guarantee around consistent naming.
Most of the time with stateless apps.
In regards to automated rolling updates, the key word there is "ordered." It will always select the same ordinal index first(reverse order starting with highest ordinal.)[1] This is different than the regular deployment strategy of "rolling update" which is also one by one but will select any pod to whack first.
Regular pods don't have stable network identifiers as they are inherently unstable. In regular pods stable network identifiers are provided by either the cluterIP or a load balancer service.
[1] https://kubernetes.io/docs/tutorials/stateful-application/ba...
And I would guess the doc is a bit of after the fact justification for a design choice already made.
I believe that's only partially true, since even if one has a PVC in something like a Deployment (the antipattern I see *a lot* in folks new to k8s trying to run mysql or whatever), then "k scale --replicas=10 deploy/my-deploy" will not allocate new PVCs for the new pods
That's why I think StatefulSets are valuable as their own resource type: they represent _future_ storage needs
* You can create a volume, but pay only for in-use blocks.
* You create a volume bigger than what you'll ever need => no resizing required (At least 16 TiB, but possibly even something crazy like an exabyte)
* There is some kind of "trim" operation that marks a block as free
* Ideally it's possible to choose the maximum block index (size of block-volume) independently from the maximum number of in-use blocks (for cost control)
Every business is incentivized this way. Competition is what pushes prices down, of which there is plenty in the cloud industry (though maybe still not as much as we would like).
Edit: Now that I actually read TFA I see it mentions most CSI drivers already provide this functionality. They provide a workaround similar to what I’m sure most people use, and I agree this functionality seems like it could be handled by StatefulSet.
I guarantee you internally cloud providers over provision their storage.
since then, I'm totally on board with pre-provisioned storage. at the very least, some way to put an upper bound on volume size (maybe efs has that? not sure.)
Unsurprisingly, AWS EFS does not support NFS quotas.
There is always a point where you can tell in advance that it doesn't make sense to keep going. It is never sensible to literally go on forever with anything -- it can only break stuff in annoying ways when it runs into real world limitations (like financial ones in that case.)
I sound aggressive about this because it's such a common mistake. It always goes something like,
"Why does this list have to be unbounded?"
"Well, we don't want to give the user an error because it's full."
"Okay, but does it really need to support 35,246,953 instances?"
"Sure, why not?"
"How long would the main interaction with the system take if you stress it to that level?"
"Oh, I don't know, at that level it might well take 20 minutes."
"And the clients usually timeout after?..."
"5 seconds."
"Would the user rather wait for 20 minutes and then get a response that might be outdated by that time, or get an error right away?"
"They may well prefer the error at that point."
"So let's go backwards from that. Will it ever make sense to support more than 250,000 instances?"
"That corresponds to the five second timeout and then some. I guess that's fine in practise..."
It's not that hard!
Often limits need to be addressed at the product level. At some point at my $work we started pushing hard enough back at product to say that we're placing a hard limit on every entity. What the limit is can be negotiated, re-evaluated, and changed -- but changes to it need to be intentional and done with consideration to operational impact.
My personal pet peeve is when i'm dealing with a client library that doesn't expose some sort of timeout.
I'd also add unbounded queues to the list above as a subtle place where the lack of a limit can really cause production issues. Everything may look like it's working fine until you realize that you've got a 4gb process with a giant buffer that'll take forever to drain.
here it is: https://news.ycombinator.com/item?id=32175328
No snapshots. No support for ACLs. Poor performance. High cost.
SAN vendors have been selling devices that deliver thin provisioned network block devices you can snapshot, clone, resize dynamically and replicate in a cluster for about 20 years now. Third parties can do all of this in cloud environments as well. See Netapp Cloud Volumes. You could roll your own using Ceph, or even a ghetto version with a subset of these capabilities with thin provisioned LVM+iSCSI, DRBD etc.
You should be able to point and click a thin provisioned, dynamically expandable, snapshot-able, clone-able, replicate-able pay as you go network block device with whatever degree of performance, availability and capacity you wish to pay for. The fact that you can't is a function of the business model and mentality of IaaS cloud operators; they don't want your old fashioned 'stateful' workloads. Neither does kubernetes. So you're swimming against the current.
I haven't used it, so your mileage is a total unknown, but our "big data" folks like it.
AWS won't change to your desired model, they mint money on underutilized resources (and hidden I/O charges/network)
Practically, sizing thin-provisioned filesystems too large wastes a lot of space on filesystem metadata.
All operating systems and clouds that I've worked with support online volume and filesystem resizing which is as close to the semantics of thin-provisioning as I've ever needed. I wasn't actually aware that StatefulSets don't resize volumes as smoothly as desired but I haven't had a problem manually deleting pods to get the filesystem on a PVC to grow (the way it works for Deployments).
Also, for important "big data" stores with replication, I don't want to trust all-in-one third party operators, which is what all K8S operators seem to aspire to be. Your data is important, and you should evaluate all your data operations use cases yourself.
It was painful. We quickly realized that rolling upgrades often got blocked. The reason being peristent volume claims sometimes get stuck.
And this is just one of few problems.
Since we moved databases and stateful services outside Kubernetes, everything got faster and more reliable.
I personally haven't experienced the issues the OP is describing. It might be that StatefulSets and PVCs are more stable now then when OP tried them. Of course using managed/hosted K8s makes a big difference.
There might have been a way to support our use case better. But this was managed service and we couldn't easily tweak every settings. But also, we didn't want to.
Our goal of using Kubernetes is to make difficult stuff easy. Not to make easy stuff difficult. And in our skillset and capacity moving stateful services outside K8s was easier, cheaper and more reliable.
K8s is great way to easily deploy and scale services. As we leverage Kafka, a lot of applications fallback on Topics and KTables for storage.
Kubernetes with jobs, cron jobs, operators, secrets etc is effectively an Operating System for the cloud.
Definitely worth the investment to learn and practice.
The fix was to add scheduler hints that moved the pods to the correct zone as I should have done in the beginning. (On first deployment, the disk is created on the correct node and thus zone, so it all seemed to work).
Just curious - how does GKE control plane upgrade break DNS?
The cluster's DNS server refuses connections briefly during control plane upgrades or updates. I haven't tried their node local DNS yet (enabling it will cause downtime, natch), maybe it'd help.
Either way, this is a good example (for me) of how Kubernetes needs some work to be friendlier to operators
K8s has really been an experience of encountering 2-4 year old bugs (closed by the stale bot of course) and missing features. We accumulate increasingly more workarounds and additional pieces of complexity to deal with this. Everything is always changing, but somehow still staying the same.
Even from the early days, there were constructs to allow stateful applications to be built on top of it.
They have made it easier to run stateful applications, but to say it wasn’t designed to support stateful applications is incorrect: https://kubernetes.io/blog/2016/08/stateful-applications-usi...
“Stateful applications such as MySQL, Kafka, Cassandra, and Couchbase for a while, the introduction of Pet Sets has significantly improved this support.”
It’s really not that big of a deal and probably a ten minute process, tops. That said, OP makes a good point that mechanisms to handle this kind of stuff exist in CSI, so why not leverage / take advantage of that versus pissing in the wind (especially when you run into CRD that wants to undo the above work while you’re in the process)?
Now that it has, it’s really a matter of someone having the time and patience to drive it though. I don’t know whether someone has sorted out all the design implications (like what happens if someone adds new resource requests, or what happens if a new resource request is rejected, and how that gets reported to the user), but it’s a small change that needs a fair amount of design thought, which tends to be hard to drive through.
https://github.com/kubernetes/enhancements/pull/3412
So when a k8s node dies with a sts on it, unlike other pods, the sts pod is NOT recreated (not wanting to interfere with whatever mechanism the db cluster has to deal with it). I've found most people working with k8s don't know about this behaviour.
I have been using Kubernetes for a few years now and whenever I see people putting manual steps they should instead be using an operator. What the author is describing as a problem has a solution (they even added it to their own operator).
Kubernetes has a steep learning curve but if you want to do nontrivial thing like have state than you need to learn nontrivial things like operators.
The default controllers that come with Kubernetes do not cover everything under the sun and they do require extensions via the operator pattern for some tasks.
Just my two cents.
I have a feeling that OpenShift handles stateful workloads more gracefully due to their enterprise-oriented customer base, but I have nothing to back that.