FWIW kops gave my team a better kube experience on AWS than EKS (their managed version)
guessing EKS isn't as bad now as it was 12 months ago, but 'managed kube' varies tremendously across clouds, especially re ingress & overlay networks
in my experience of AWS EKS, GCE's GKE and digitalocean's one, GCE has the tightest integration between load balancers and ingress, and even that can be clunky
kops is overlay agnostic and lets you choose, which is a con but also a pro
my guess is anyone that needs serious networking performance is still not sold on kube
We use EKS. It's been buggy especially in the networking plugin (CNI). For example, version 1.5.4 released at the end of September caused ip rules to randomly disappear and break things in mysterious ways. Anyone who launched a cluster during the 4 days it took them to change the default had to manually downgrade their version of CNI. Of course, EKS doesn't provide a way to control the version of components they add to new clusters. Or, as I remember, the master account that is given initial full access to the cluster.
More broadly it just has so many rough edges and unfinished things. You can now do audit logging but you can't set an audit configuration so it logs everything (I think we had 200k events a day on a tiny cluster). You can tie service accounts to IAM roles but only if you cluster was created or upgraded after a certain date. You can't change or add subnets to the cluster without recreating it. Can't back a cluster or restore a cluster. Private load balancers require a non-standard configuration. EKS is still on 1.14 even though 1.17 is already out. The list goes on. Here's the current list of big and small issues people have with EKS: https://github.com/aws/containers-roadmap/issues?q=is%3Aopen...
My enterprise and teams new to k8s have been very happy with Rancher on k3s with Postgres for racher's data plane, then just cloud providers configured (AWS in our case). It's very easy to deploy new clusters on AWS (we use terraform to deploy rancher and the rancher managed clusters) managed by rancher, have had zero issues with cloud provider resources integrations.
Almost none of the engineers or devops team have kubernetes experience and we've been very happy with Rancher/RKE versus our previous experience with EKS.
Heads up, AWS has a public GitHub repo (https://github.com/aws/containers-roadmap) where users can submit bugs and feature request for managed Kubernetes (EKS). It is quite robust and actively maintained by AWS engineers.
Recently managed Kubernetes on AWS (EKS) rolled out the ability to make your master public or private. Previously if you set it to private, you were only able to lookup the master endpoint DNS entry within the VPC. After much backlash[1], they now advertise the private ip publically which allows much greater flexibility. A highly anticipated and nice change.
making an open source contribution to benefit a big 5 company with deep networking expertise in-house and which also has forked secret versions of mysql and postgres should at least earn me some kind of rebate
Kops is very convenient. It's been the go-to solution for standing up clusters on Kubernetes in my last two companies. In fact, I'm using it tomorrow to set up a demo cluster for a talk in San Diego.
If you're looking to standup a kubernetes server and not on AWS, take a look at kubeadm[0][1]. kubeadm does just enough to build a cluster and not much more -- it's a really light-feeling (even though what it sets up is complicated) bit of kit.
I use it to run on baremetal, and it's a wonderfully simple and robust tool (as well as officially supported) -- it's almost too good, because you should really know everything it's doing and if you use it from the start it's easy to get lost when something goes wrong (most of the time nothing will, but eventually something will).
The difference between kubeadm and 'The Hard Way' is setting up a ton of certificates you then have to manage and manifests/config for etcd, kubelet, kube-proxy, kube-apiserver, kube-controller-manager, and core-dns. If you diff the default manifests of kubeadm and 'The Hard Way' you'll find they're not especially different.
You should still take the time to read the docs on the components and decide if you want to make changes. 'The Hard Way' doesn't explain what's going on any more than kubeadm does. It's just a set of instructions to follow (that are GCP specific).
> The difference between kubeadm and 'The Hard Way' is setting up a ton of certificates you then have to manage and manifests/config for etcd, kubelet, kube-proxy, kube-apiserver, kube-controller-manager, and core-dns. If you diff the default manifests of kubeadm and 'The Hard Way' you'll find they're not especially different.
The end results are similar but the process is vastly different. kubeadm is almost a one-shot tool, doing it the hard way is a learning process. I was trying to say that obviously when you want to just get it done (or script your cluster) then use kubeadm, but if you're learning k8s it's better to do it the hard way at least once. Building an intuition for for the system is made up of (and where to go/which logs to look at when something goes wrong) is important.
> 'The Hard Way' doesn't explain what's going on any more than kubeadm does
Even just following a guide, running the ancillary tools yourself, downloading and running the binaries, etc is way more explanation than kubeadm or other tools give you -- 90% on a fresh machine you just run kubeadm and you're "done" (for some sense of the word).
Before kubernetes the hard way[0] (which is a fantastic resource, though it's gcp specific) existed, I used the CoreOS guides[1] (this was before they started just pushing "use tectonic" and obviously before they were bought out and essentially merged into atomic/fedora/whatever) -- they were fantastic and did more of a walk through of the what and why of the pieces of k8s. Reading through these guides was essential to building my intuition about k8s (I've even written about the process[2]), and I think it's important for people to do this experimentation as well.
One of the best things about k8s is it's ability to unchain you from cloud providers -- using only managed k8s (without something like crossplane[3]) would be effectively squandering that advantage IMO. Yes most companies don't actually need to run multi-cloud, but I expect that the ability to offer k8s as a managed service interface will make it easy for anyone to become/run a cloud provider, and that's a future I want to see -- more cloud providers means more competition which means a better world for startups and those who consume hosting services.
I once created a Kops cluster and decided to update masters from m4 to m5 family and then noticed the AMI did not have the actual drivers to mount NVMe disks but the blue green did not work and until I could revert it the old master instances were terminated and I could not get them back. But it was ~2 years ago and I believe it would be much more useful than the current state EKS is in.
20 comments
[ 4.4 ms ] story [ 48.5 ms ] threadguessing EKS isn't as bad now as it was 12 months ago, but 'managed kube' varies tremendously across clouds, especially re ingress & overlay networks
in my experience of AWS EKS, GCE's GKE and digitalocean's one, GCE has the tightest integration between load balancers and ingress, and even that can be clunky
kops is overlay agnostic and lets you choose, which is a con but also a pro
my guess is anyone that needs serious networking performance is still not sold on kube
seriously - why would you use k8s?
it's basically a MLM scam ran by google
Almost none of the engineers or devops team have kubernetes experience and we've been very happy with Rancher/RKE versus our previous experience with EKS.
Recently managed Kubernetes on AWS (EKS) rolled out the ability to make your master public or private. Previously if you set it to private, you were only able to lookup the master endpoint DNS entry within the VPC. After much backlash[1], they now advertise the private ip publically which allows much greater flexibility. A highly anticipated and nice change.
[1] https://github.com/aws/containers-roadmap/issues/221
I use it to run on baremetal, and it's a wonderfully simple and robust tool (as well as officially supported) -- it's almost too good, because you should really know everything it's doing and if you use it from the start it's easy to get lost when something goes wrong (most of the time nothing will, but eventually something will).
[0]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kub...
[1]: https://github.com/kubernetes/kubeadm/
You should still take the time to read the docs on the components and decide if you want to make changes. 'The Hard Way' doesn't explain what's going on any more than kubeadm does. It's just a set of instructions to follow (that are GCP specific).
The end results are similar but the process is vastly different. kubeadm is almost a one-shot tool, doing it the hard way is a learning process. I was trying to say that obviously when you want to just get it done (or script your cluster) then use kubeadm, but if you're learning k8s it's better to do it the hard way at least once. Building an intuition for for the system is made up of (and where to go/which logs to look at when something goes wrong) is important.
> 'The Hard Way' doesn't explain what's going on any more than kubeadm does
Even just following a guide, running the ancillary tools yourself, downloading and running the binaries, etc is way more explanation than kubeadm or other tools give you -- 90% on a fresh machine you just run kubeadm and you're "done" (for some sense of the word).
Before kubernetes the hard way[0] (which is a fantastic resource, though it's gcp specific) existed, I used the CoreOS guides[1] (this was before they started just pushing "use tectonic" and obviously before they were bought out and essentially merged into atomic/fedora/whatever) -- they were fantastic and did more of a walk through of the what and why of the pieces of k8s. Reading through these guides was essential to building my intuition about k8s (I've even written about the process[2]), and I think it's important for people to do this experimentation as well.
One of the best things about k8s is it's ability to unchain you from cloud providers -- using only managed k8s (without something like crossplane[3]) would be effectively squandering that advantage IMO. Yes most companies don't actually need to run multi-cloud, but I expect that the ability to offer k8s as a managed service interface will make it easy for anyone to become/run a cloud provider, and that's a future I want to see -- more cloud providers means more competition which means a better world for startups and those who consume hosting services.
[0]: https://github.com/kelseyhightower/kubernetes-the-hard-way
[1]: https://coreos.com/kubernetes/docs/1.2.2
[2]: https://vadosware.io/post/fresh-dedicated-server-to-single-n...
[3]: https://crossplane.io/