HN Explorer

Ask HN: What is your Kubernetes nightmare?

220 points by wg0 ↗ HN
Everything self-hosted has its maintenance tax but why Kubernetes (especially self hosted) is so hard? What aspect is that makes Kubernetes operationally so hard?

- Is it the networking model that is simple from the consumption standpoint but has too many moving parts for it to be implemented?

- Is it the storage model, CSI and friends?

- Is it the bunch of controller loops doing their own things with nothing that gives a "wholesome" picture to identify the root cause?

For me personally, first and foremost thing on my mind is the networking details. They are "automatically generated" by each CNI solution in slightly different ways and constructs (iptables, virtual bridges, routing daemons, eBPF etc etc) and because they are generated, it is not uncommon to find hundreds of iptable rules and chains on a single node and/or similar configuration.

Being automated, these solutions generate tons of components/configurations which in case of trouble, even if one has mastery on them, would take some time to hoop through all the components (virtual interfaces, virtual bridges, iptable chains and rules, ipvs entries etc) to identify what's causing the trouble. Essentially, one pretty much has to be a network engineer because besides the underlying/physical (or the virtual, I mean cloud VPCs) network, k8s pulls its very own network (pod network, cluster network) implemented on the software/configuration layer which has to be fully understood to be able to maintained.

God forbid, if the CNI solution has some edge case or for some other misconfiguration, it keeps generating inadequate or misconfigured rules/routes etc resulting in a broken "software defined network" that I cannot identify in time on a production system is my nightmare and I don't know how to reduce that risk.

What's your Kubernetes nightmare?

EDIT: formating

262 comments

[ 2.7 ms ] story [ 351 ms ] thread
ciguy
What are you fishing for exactly here? K8s has it's issues, but the way you have phrased the question is only going to get you biased answers.
hedgedoops2
I guess OP is looking for real life stories, of 'nightmares', caused by excessively complex or opaque kubernetes configurations.

AKA "Tell us about how your boss/team has shot themselves in the foot by using kubernetes [needlessly or wrong]".

robertlagrant
It's only biased if you think of these technologies as teams to cheer or boo. If you just want to be pragmatic and get things done then knowing the pitfalls is valuable.
wg0
I am a k8s user myself and I believe it is one of the most innovative projects out there. My intent here is to reach out to the wider community to share my fears/pain points and see if:

- Is there a knowledge gap that I'm having and I can work on myself to be better? - Are there any practices, techniques, tricks, SOPs, solutions, mechanisms that wider community has developed or still under evolution that I can put at use.

I know k8s has issues but knowing those issues up close (including why they are) can make one a better operator of the machinery at hand.

So mainly, I'm seeking advice, opinions, views that come from people's experiences and aren't necessarily in official docs or books/courses/workshops/seminars etc.

rglullis
> only going to get you biased answers.

Unlike, say, every opinion from engineers who are fond of "resume-oriented software development"?

Or every blog post from the dev team in a company that is working with kubernetes, even when there is absolutely no need to, but they do it anyway, because "it helps with hiring"?

Of course opinions, testimonials and even reports will be biased. They are still important anyway.

lbriner
Ironically, I think one of the biggest issues is around packaging, specifically Helm charts (but if there are others, it is probably the same). In many frameworks, packaging is to help people by hiding complexity. Need an ingress? Use a Helm chart!

But then upgrading can be very risky because if you have any problem at all, unless you understand the helm chart you can rarely simply downgrade/uninstall, you could have caused a fatal problem and for a cluster, the resilience is meaningless if you make a change that blocks access to all service.

Other issues relate to dependencies and breaking changes which might be subtle and which might not be easy to discover like the fact some old resource uses a v1beta type which becomes deprecated.

I think once it is working, Kubernetes is very reliable for me but it is when making infrastructure changes that things can go south very quickly. Updating deployments etc. is fine.

aliswe
I am not going to contest this but one upside is that you can install several ingresses at once and you don't have to uninstall the old one until the new one works
JensRantil
Honestly, creating my first K8s deployment of a service; typing out at least 150 lines of YAML to define my Deployment, figuring out how my ConfigMaps, Secrets, and Volumes, Services are defined and connected together. Vanilla K8s YAML is extremely low-level.
speedgoose
Yes it’s a bit much. When I was beginning with kubernetes I was writing Docker compose files first and then converting them to kubernetes using https://kompose.io/
bovermyer
I'm a platform engineer and I still think that Kubernetes and its tooling are unnecessarily complex.

As an aside, when I think "low-level" with regards to computer programming, I think machine byte code - closer to the hardware - so this statement read a little funny to me.

OJFord
Yeah, it may be complex and highly configurable, but it's hard to imagine getting much higher level than defining networking, storage, secrets, etc. through YAML.
e12e
I feel it's a lot like the Java enterprise world of FactoryFactoryFactory-classes - a col league coined the term "horizontal abstraction" for this type of trend; you never build a pyramid/hierarchy that composes truly higher level/abstract reasoning - you just complect in various constructs - that all keep hold of most of the complexity from the level "below".

So you get to write 30-40 lines of yaml for each of your ten slightly different services...

robertlagrant
Having experienced many Word documents full of deployment instructions and screenshots on how to deploy software, a few lines of YAML has been amazing for me :)

But no doubt there are other tools that are even better.

GordonS
Compare with Docker Compose files, which still use YAML, but a lot less of it, and less verbose too.
KptMarchewa
And does a lot less too, since it describes single node deployment, without any complexities behind networking and communication of services on different nodes.
tpetry
A docker compose file is also used for swarm mich is a multi-server deployment.
robertlagrant
But is that a more complex file than the single node one?
GordonS
While the file specification is for Docker Compose, Docker Swarm uses the same files, and supports multi-node deployments.

There are a few differences in supported features; for example, IIRC, only Docker Swarm supports `secrets`.

p_l
The most freeing moment with k8s for me was integrating that no, you don't need to write all that YAML (or JSON), and going into generating manifests from other formats - in my case, usually Jsonnet with mix of libraries both public and private which quickly embedded knowledge specific to our setup.

I think one of the best things I ever heard from another colleague was "I took the example jsonnet and had working version for completely new deployment within hour" :)

JensRantil
Agree! That said, it's important to remember that someone needs to maintain those manifests, too. Copy-paste might work initially, but you will need to modify them at some point. Which means you need in-house knowledge of that K8s YAML, anyway. For a smaller shop this might be non-trivial. For a larger shop you likely have something close to a Platform team that can maintain the manifests to make them easy to use.
secondcoming
I'm buried up to my balls in this right now. My favourite part is nested 'spec' objects where I've no clear view into what each spec is.
Bayart
Try k9s[1], the xray view (:xray [ressource]) shows you nested resources as a tree. I find it very useful (and k9s in general is a fantastic administration tool).

[1]: https://k9scli.io

bavell
There's a vscode k8s language plugin that gives you autocomplete and tooltips when hovering over yaml keys, saves me a ton of time in situations like these.
e12e
I haven't really used it in anger, but I sort of think kpt might be helpful in managing k8s: https://kpt.dev/
Bayart
> Vanilla K8s YAML is extremely low-level.

I find trying to untangle something deployed with Kubespray/Helm/anything automatized far more headache inducing than flat YAML files.

JensRantil
Yeah, been there and it makes things one notch even worse. :)
dhagz
I've found that creating the k8s resources manually in a local KinD/microk8s cluster and then spewing the resulting YAML to be much easier than typing the YAML directly.
anothernewdude
I hate that kubectl wants all the images to be already built. Instead I'm forced to keep docker-compose yaml around to actually build the damn things first. Which introduces more yaml that kubectl will insist on reading.

The documentation at the main kubernetes site is poor, and is being deprecated, but not in favour of anything new.

bovermyer
Interesting. You build images at deployment time? Why?
anothernewdude
Quick iteration. More like I deploy at build time. CI/CD takes minutes to deploy to a shared environment. Rather than contending for a bottleneck, we do local development on a single node K8S environment.

Maintaining docker-compose as well is a pain, and it's repeating ourselves.

skinnyarms
Give Skaffold a shot, takes a lot of the pain out of running locally. I think of it as a simple wrapper around Docker, Kubernetes, Helm and Port-Forwarding (plus other options I don't use) that makes it easy to use the same build/deploy definition to work locally or build/deploy in CI/CD.

Couple examples:

- Build the images that need it with "skaffold build"

- Can watch for changes are rebuild automatically when you "skaffold dev"

- Can automatically detect your services (or arbitrary) ports and forward them when working locally

- Advanced features like profiles and modules for supporting multiple environments

https://skaffold.dev/

MPSimmons
If you have a fully-functioning "best practices" Kubernetes environment, each of the following topics ends up with its own full-depth tech:

- Compute

- Deployment

- CI

- Networking

- Storage

- Policies

Imagine running a microservices solution without an orchestration solution - how many people would it take to administer the servers, the storage, the network, the policies, etc. And with Kubernetes, you get maybe a couple of teams if you're lucky. This is the power and the leverage of the platform.

But also, imagine in that environment, how many things can go wrong, and the amount of expertise that you need to properly debug them. You still need that amount of expertise, because all of that complexity is still in place (or at least most of it is) - if your physical disks are throwing errors, you need someone who knows how to debug and replace that. Not hard. But then you have Ceph above that, and Rook above that (or whatever storage solution you use). And then you've got the deployment that has to make the PVC successfully. And it's like that for everything. Every problem has the potential to be a full stack problem for any one of half a dozen stacks.

It's a lot.

iso1631
My team looks after 150 servers from Singapore to LA, about half physical, half virtual, some which just sit on shelves between jobs. Being pesamistic takes about 100 hours a year of feeding, upgrading, etc, about $6k a year.
throwaway894345
That's comparing steady state with bootstrapping a Kubernetes environment. I suspect steady state Kubernetes is comparable.
daneel_w
Kubernetes itself. Maybe I just need some more "hands-on therapy".
dangerface
> What aspect is that makes Kubernetes operationally so hard?

Inherited a web site and hosting from another studio. They setup a php site in a docker inside a vps. They don't use micro services its one monolith container. They didn't setup any way to get logs out of the thing. They don't use docker compose to build an image, they get a console for the container and use it like a vps.

They literally just use it to add another layer of containerisation on their vps.

You already need to understand linux to use docker or kubernetes, If you don't use micro services or need horizontal scaling its just more to learn, an extra layer of complexity thats super fragile and a nightmare to debug.

It has such a niche use case but every one use it where its not useful because its trendy. They want to put on their cv that they have used docker / kubernetes they don't have to write that it wasn't necessary and caused issues.

melezhik
This is that I have to use yaml to configure k8s. Every k8sish tooling has it's own yaml API, including helm, gitops, argocd and friends, so you end up having a bunch of brittle and very hard to understand and maintain yaml files ... Sigh
sureglymop
I think that it's just the "kubectl" tool which accepts yaml input. This tool then talks to the actual API of your cluster and uses json (and maybe other formats).
melezhik
It is not only k8s tool unfortunately. Helm and argocd also exposes YAML api. I am not against YAML in general btw, it’s just a variety of tools’ specific yaml formats one mixes and have to learn every time that makes one’s life difficult
k8sToGo
I hate helm. Writing helm charts sucks. We use kustomize, but still YAML at the end.
tyingq
Extra layers, debugging things that are ephemeral, and overlay networks.
CSDude
I have 2:

1. We build our own custom build system, because there is no CI that can do actual DAGs (maybe a few). A custom Kubernetes operator that parses Jsonnet files to create 100s of CRDs and pods to achieve extreme parallelization. EKS was 144$/mo (now 72$) but no info on master node types. Using watch endpoints with hundreds of pods did not scale well. They had to bump up the master node instances to c5.18xlarge, but same price for managed. But figuring out it was needed to do just scale-up took days. One c5.18xlarge is 2k$ month, and EKS runs at least 3 for HA. So it's a horror story for them. But we also had 100s of worker nodes so it might offset some of them.

2. Similar to CI, we allowed devs to deploy all microservices (~80) from any branch so that they can port-forward and use them. All of them had Ingress endpoints. Days after headaches and frustrations, it turns out nginx ingress generates megabytes of configuration whenever a new deployment occurs, forks a new subprocess with new cfg, kills the other connections. When it's done often, it takes 30GB of memory when 50 developers use it (~4000 pods) and it often dies and restarts. Similar story for Prometheus, kube-state-metrics; they do not like short-lived containers and hug on memory.

KronisLV
> We build our own custom build system, because there is no CI that can do actual DAGs (maybe a few).

Have you had a look at GitLab CI? They have a bit of documentation here: https://docs.gitlab.com/ee/ci/directed_acyclic_graph/

Now, I don't work on any projects that are too complicated, but I recall that piece of functionality working as one would expect: https://docs.gitlab.com/ee/ci/yaml/index.html#needs

Also there's Drone CI, which also supports setting up dependencies in your pipelines, if you'd prefer something that's not connected to GitLab CI so closely: https://docs.drone.io/pipeline/docker/syntax/parallelism/

ghusto
TLDR: Complexity.

The deprecation lifecycle, and running ingress controllers in an automatic scaling group.

The first isn't as much of an issue if you have a (partially) dedicated team for managing your clusters, but can be prohibitively expensive (effort / time-wise) for smaller organisations.

The second highlights a bigger problem in K8s in general. I'll have to give a little background first:

If you run an Nginx ingress controller on a node that's part of an ASG — i.e. a group where nodes can disappear, or increase in number — you will experience service disruption to a small percentage of your requests, every time a scaling event occurs. This is caused by a misalignment between timeout values for your load balancer and Nginx, which can not be fixed:

* https://github.com/kubernetes/ingress-nginx/issues/6281 * https://github.com/kubernetes/ingress-nginx/issues/6791 * https://github.com/kubernetes/ingress-nginx/issues/7175

The fix is to only run the controllers on nodes that reside in a separate statically sized group, and perform updates to them out of hours when necessary :|

I'll leave you to decide on whether that's a fix or not, but the larger point it highlights is how _theoretically_ everything's great in K8s, but the headaches introduced by the complexity often make it not worth it.

Another example is pod disruption budgets. These are needed because the behaviour of K8s when instructed to shutdown a node is, well, to shutdown the node. Seems reasonable, until you realise that it doesn't handle moving the workloads off that node _first_. No, at some point later, the scheduler realises the pods aren't running and schedules them somewhere else. So you use a combination of PDBs to tell K8s that it must keep n pods of this deployment running at all times, and distribution rules to tell it pods must run on different nodes. This solution falls apart when you have pods that should only have a single instance running.

lazide
This is always a problem with ingress/load balancer unless you buy some REALLY fancy (auto failover) enterprise hardware loadbalancers unfortunately, regardless of K8S. It does make it harder to see what is going on though or work around it.
bajsejohannes
The decoupling of ingress and deployments always bothered me, although it might not be a _nightmare_ exactly.

In short, the ingress may route traffic to a pod after it is killed. The solution is that when a pod gets a SIGTERM signal, it should mark itself not ready, wait for some amount of time and then shut down (see e.g. https://deepsource.io/blog/zero-downtime-deployment/). I've heard arguments for this behavior, but it's not the same trade-offs I would make.

ygouzerh
More distributed system than Kubernetes, but quite fun : We deployed a MongoDB cluster on our Kubernetes Clusters. Our application was a having a chat feature that stored the messages into the MongoDB cluster. After some months, we realized that we got some weird issues, some messages was arriving in the wrong orders, like : 1. A : Hi ! 2. B : Bye ! See you next time ! 3. A : Great and you ? 4. B : Hello ! How are you ?

We thought it was an application issue, but it was that actually on the database side : the timestamp of each message was using the local time of the mongodb instance. And between different instances, the time was different. We realized that the Kubernetes Nodes had issues to connect to the NTP server, due to a rule in an random firewall.

When we fixed it, every other messages where in the good order

lampshades
1. A : Hi !

2. B : Bye ! See you next time !

3. A : Great and you ?

4. B : Hello ! How are you ?

marcosdumay
> due to a rule in an random firewall

The eternal practice of middleboxing your network. This didn't work well at the time LANs were completely isolated, break much better nowadays when LANs are just a convention over WANs, and fails for virtual LANs on a single host too.

Yet people just do it, every single time. Because setting the security in a single place is expected to be easier than setting at the endpoints (I blame Windows for that culture). What is kinda understandable, but here we are, talking about Kubernets, and having that same culture.

jokethrowaway
Honestly I don't get the hate k8s get.

I run my own clusters and it just works.

Sure I have to ignore a lot of crap in the setup phase, there are so many products out there I don't want to pay for. The nightmare may come from some devop installing a bunch of helm charts without configuring things properly.

Scaling down to a minimal cluster is a real concern: I would like to run k8s for some micro project that literally run on 5$ vps but it's too heavy for that.

roflyear
Never in my experience when someone says it just works does it just work in reality. Usually they are hiding some things that they are keeping up with but for some reason don't include those issues as the thing not working.
cyberpunk
I look after a lot of k8s professionally (hundreds of clusters), and the only issues I've had in the last few years were a istio/pilotd bug and a bit of pain moving from e.g v1beta1->v1 during normal upgrades.

Everything else that's gone wrong with these environments was cloud/hw related, or app stuff.

Guess it depends on where your competencies are.

roflyear
> Guess it depends on where your competencies are.

That's true of almost everything, no?

I use k8s myself (nowhere near that scale, basically the smallest scale you can think of... one cluster...) and don't think it is a bad product.

k8sToGo
What about using k3s as a minimal cluster?
Kab1r
I find that even k3s' low overhead is often too heavy to run in tiny VMs and SBCs.
sureglymop
For me it is actually getting services exposed. Say i buy a domain and then i (trivially) containerize my applications and set up services in kubernetes. Now comes the networking part which is just a pain. How do I make my service accessible? It's easy with docker and an nginx reverse proxy but with kubernetes it's always seemed to be a real pain.
7sidedmarble
Just making your pod accessible is not as complicated as it seems. All you need are: 1. A kubernetes service resource. This just contains a selector that points at your pod. 2. An ingress. You point this at the service you just made. You will get a static IP. Point your DNS at that.

And you're good to go. This assumes that you are using a provider that comes with an 'ingress controller' out of the box (which is what actually makes the ingress function. It's usually just nginx). If not, install the nginx ingress controller with helm. Then install cert-manager with helm for tls cert provisioning.

sureglymop
So if I'm not using a cloud provider but just have a k8s cluster on a small vps with a public ip i just need the nginx ingress controller? What if i want https? Is there a way to automatically enable let's encrypt for different services and domains/subdomains?
linuxftw
There's a wealth of material online to explain these things. let's encrypt can integrate with a number of ingress controllers trivially. Like much of anything else, you need to actually experiment with it to understand how it all fits together.
Bayart
Nginx ingress controller + cert-manager is the most common, best documented way of doing this. If you don't have a domain already pointing to your public IP, you can use nip.io.
buzer
It's not quite that simple if you are not with provider that provides LoadBalancer service integration with Kubernetes. Normally the input to Kubernetes cluster is essentially NodePort. That's normally more or less random high port (like 31453) that is exposed on all nodes/nodes that run the service that matches the selector. Unless you want your visitors to add that to the URL (and keep DNS up to date with active nodes), using it to provide end-user accessible HTTP/HTTPS services is not very viable.

You either need to find/create integration to provider's load balancer (or possibly CDN that allows non-1:1 port mapping) or use HostPort service. Latter has it's own share problems as well.

high_5
I'm afraid k8s would become like git - it's a great tool out of which we mostly use like 5% of it's complete capabilities, yet we all use it because everyone is using it. Yet, k8s doesn't really make all the underlying stacks go away. When the shit hits the fan, you have to troubleshoot it with knowledge of much more than just YAML syntax.
birdiesanders
k8s is just Linux, if you forget the Linux there is no hope.
captrb
I admit to being a little frustrated with systems engineers telling me I should never need shell access to a production system again, that web-based Metrics and Tracing should be enough to debug all problems. I have twenty years of muscle memory using strace, dtrace, lsof, blah blah blah to troubleshoot complex problems. Furthermore I'm only brought in when the problem is sufficiently complex. I understand that it should be a break-glass exception, but I don't want linux abstracted away completely.
muskmusk
Someone hating on kubernetes and cobbling together their own stuff that "does the same, but simpler". Usually only simpler for that person.
Normal_gaussian
Storage:

Shared FS between nodes, autoscaling volume claim sizes, autoscaling volume claim iops, and measuring storage utilisation (iops e.g.) for pod/node/pv.

How have I solved it? I haven't and I know its a key part of cost-control for us in about 12 months.

Fast deploy:

I'm trying to get a test cluster up in less than half an hour. With the DAG for building it all I'm getting a failure rate of 30% if I don't leave arbitrary timings and extra steps. I've also only automated about 25% of our stuff, so I expect it will take longer.

nijave
Are you setting up managed k8s on a cloud provider and, if so, which one?

I've had some issues with getting EKS dependency ordering correct (using Terraform)

spicyusername
- 60% of the Kubernetes ecosystem is half-baked alpha software

- Maintaining 200+ clusters for 10 small applications

- Cloud bills

- Autoscaling never working well

- Trying to untangle Terraform state without taking down Prod

garciasn
We use GKE.

1. I don't know about any of this; we don't seem to have problems.

2. This sounds like an architecture issue, not a k8s issue.

3. Our entire GKE infrastructure costs less than $50 a month.

4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases.

5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we did what we needed to do without downtime in production. While annoyingly esoteric, I thought it worked pretty fucking well compared to our alternatives.

ColdHeat
With regards to 3, isn’t the management plane alone $72 for GKE without considering the cost for the nodes? How are your costs so low?
garciasn
As noted by another HNer, we are using one cluster and up to 10 nodes as the top end, with only one running 90% of the time and only up to 3 covering the other 9%. We set 10 to the upper limit to ensure we're not going to have borked runs in case some crazy random ML model isn't going to take everything out and force killed jobs. The vast majority of workloads running on the cluster are HIGHLY variable with most running on Monday morning/afternoon and Month or Quarter starts + 3 days.

We are running many hundreds of jobs on those peak days with only a handful running on any other day. While many bring up examples where 24/7 infrastructure from a single box is more than plenty, we find that we can run micro VMs in this configuration and not have to worry about resource contention as our jobs run.

Pre-GKE, we were managing the timing manually, which was fine until we started to scale, but we found this to be a far better situation. Particularly because we simply don't have to think about it.

YMMV.

JamesSwift
> Our entire GKE infrastructure costs less than $50 a month.

Uhhh what? I mean even my personal DO based cluster runs about $40 a month. I'm skeptical a production cluster is at $50.

garciasn 

    Service                   Compute Engine Kubernetes Engine
    Cost.                     $191.78        $62.34 
    Discounts                ($13.96)       ($62.34)
    Promotions and others     $0.00          $0.00 
    Subtotal                  $177.83        $0.00
The GCE instance cost includes some of our 24/7 VMs which are the lion's share of that line item, not the micro VMs we use for the cluster.
throwaway894345
Agreed. We've had only one autoscaling issue and it was on Google's end (datacenter ran out of nodes of a particular type and thus failed to scale up). Our GKE infrastructure costs a lot more, but we do a lot of heavy compute.
garciasn
Most of our 'heavy compute' happens in BigQuery so we pay for it there, instead, but we make sure the rest of our analytics infrastructure needs are engineered to be as light as possible to keep costs low.

We are an ESOP so, as employee owners, it behooves us to be as cost-conscious as possible.

otabdeveloper4
> 3. Our entire GKE infrastructure costs less than $50 a month.

At this scale, you don't need Kubernetes, invest in a pocket calculator instead.

sleepybrett
Tell me you barely understand kubernetes without saying you barely understand kuberentes.
rglover
> 60% of the Kubernetes ecosystem is half-baked alpha software

This one is fair. Wasted a lot of time trying to find the "correct" dependencies—I remember the Nginx Ingress Controller specifically being a headache—only to find a maze of deprecations, poorly written documentation, or stuff that just flat out didn't work. That was ~18 months ago (I set up my cluster to run sites for my business and have basically left it alone) so things may have changed but at the time I remember being surprised after hearing so much hype.

throwaway894345
Pretty sure nginx ingress controller is one of the more solid and widely-used pieces. I've had a lot more trouble with cert-manager, but it seems to be in a stable state on my cluster now and anyway similar solutions in the bare-VM world are just as painful (IIRC I gave up trying to get terraform to do the handshake for AWS ACM).
rglover
What I can say definitively is that having gone from not doing any infra work to using k8s and then over the past few months trying my hand at a bare-metal setup, just spinning up a Linux box and hand installing deps via apt or snap was far more enjoyable/easy to follow.

Primarily because there was very little obscurity (i.e., config files that automate away a lot of thinking or Dockerfiles/containers doing the same). It also left me feeling more confident about stability because if something isn't working, it's pretty clear what I broke/forgot. Worst "bug" I ran into was a snap server hanging when installing a dependency.

nunez
literally heaps of resources used in Kubernetes cluster are using API versions with alpha in the name!
INTPenis
I honestly kinda like kubernetes and I have no problem tying together a bunch of distributed resources in my head.

The biggest nightmare for me is networking, simply because I'm not trained in networking. I know the basics to become a senior sysadmin but it's not natural to me. So mix in kubernetes and it becomes even more abstract.

mcv
I'm probably the wrong person to ask. My Kubernetes nightmare is having to configure anything from scratch. Or having to fix something that doesn't work.

My comfort zone is where Kubernetes works fine and I don't have to touch it, or only update trivial stuff.

Deletionk
For me the primary thing is OOM killed. But swap support is being worked on.

And then perhaps the proper handling of persistent disk.

oneplane
The only nightmare is the same one as with npm and pip etc.: dependencies hiding behind other dependencies and having badly documented charts being used that don't share basic lifecycle information.

Kubernetes itself has always been fine, just like a bare network or bare OS has been, but when you start stacking stuff built by other people (especially when the stuff isn't of the best quality) it just goes downhill from there.

Perhaps the actual nightmare is inadequate quality control... but that's not really specific to packaging or shared components in Kubernetes.

pritambarhate
Slow performance.

So we have a few Spring Boot based webapps which were running (along with PgSQL) on a shared AWS t2.medium instance, we migrated these to a GKE cluster with a node pool of e2-standard-2 instances. The nodes are on a private network and don't have public IPs. The services are exposed via Load Balancer based Ingress (with SSL). Even after allocating one core to PgSQL and 2GB RAM, the API calls from the GKE applications are perceptively slower than that of the shared AWS t2.medium instance based deployment. Tried giving generous CPU and RAM to the applications however, it still didn't improve the response time. Since these are the very fist applications being moved to this cluster, there isn't much else running on this cluster.

Now sure what's causing the slowness. Have any of you experienced something like this in GKE?

k8sToGo
Slower in what sense? Latency, throughput, computation time?
uberduper
Are you setting cpu requests in your pod spec? This influences the cgroup cpu.shares for the containers and (unless this has been fixed) leaving cpu requests unset results in cpu.shares=2, which jvm interprets badly.
pritambarhate
Thanks for replying. Yes we are setting the CPU requests in our pod spec. I will experiment with this further and see if solves the issue.