There should be more of a boilerplate very stable version for smaller companies but I think in general Kubernetes makes it really easy to automate away a lot of the ops and security stuff and get app pipelines running pretty quickly which is really nice.
Openshift is big in on-prem enterprise setups because as flawed as the install process is or as ridiculous licensing is, you get a remarkably complete system pretty easily out of a bunch of servers that integrates with things like your corporate LDAP.
There are companies trying to provide this - I don't know if anyone is doing it very successfully yet, but Microk8s, k3s, etc are all basically "distros" of kubernetes for different targets.
I think part of the issue is that many of the bigger players want you to use their managed k8s solution and not just give you their setup - I think that's a temporary thing. A few of the big boons of k8s is the ability to be multi-cloud, and vendors are losing out on a revenue stream by not embracing that (though, AWS has a new multi-cloud k8s thing, so maybe they get it).
I kind of don't understand why people don't use managed K8's and avoid needing to pay ops people to manage their setup.
One can learn the developer side of K8's in a week or so enough to be deploying apps and setting up services and pods without the overhead of managing etcd and distributed clusters.
> You need k8s-savvy ops people, or really good devs who like ops stuff
They are called DevOps Engineers and it's a standard role these days.
And it doesn't matter whether they like it or not. In many organisations these days developers own their piece of work all the way through to production and are expected to provide an ops-like support for it.
I use managed k8s, but have sympathy for the bare metal folks. Wiring servers together using kubeadm is so simple and powerful that the missing pieces of the puzzle (networking and storage, for example) feel frustratingly close.
In an ideal world, you could purchase or rent commoditized resources (compute, storage, bandwidth) and have those resources deploy your application in a standardized way. K8s is so close to that out of the gate, that having to use cloud provider's expensive and proprietary offerings feels sub-optimal.
I agree that cloud offerings are the way to go for anyone without good reasons not to (eg compliance, lack of connectivity, or full control over upgrades). But a lot of the "I prefer on prem" attitude comes from realizing that having a black box of an ingress controller on my cluster isn't the end state of how apps get deployed
It really depends on what your needs are and whether your provider's k8s service addresses them. AKS for example does work but, I'm told, is missing some fundamental stuff in regards to security etc. (Caveat: I only know this through others.)
OTOH if a service is available and would work for someone's use case, by all means, use the service. I've set up native k8s clusters just as exercises, and it's no picnic–and I didn't need to manage those clusters, later. Just doing build exercises alone was enough to scare me straight.
Managed k8s doesn't mean you don't need ops people. You still need ops people - you are still running a complex and customized multi-node kubernetes cluster. You just need fewer ops people by following a vendors optimistic path.
One thing I've found about k8s is running a cluster can be very costly, and using a managed cluster on one of the big three clouds is even more so. Also, when rolling your own cluster it requires a lot of elbow grease to get things in a working state, and even more to keep things working.
It feels awesome from a developer standpoint once you work out all the kinks. It's very natural describing your service with a bit of yaml, and then being able to then spawn it into your infrastructure and have it automatically scale out.
If there was a cost effective k8s cluster you could rent then that would be the thing that finally kills traditional infrastructure for me.
That's the exact same feeling I get too. If I'm a mom/pop shop online, why can't I get a k8s cluster w/just 1 machine for $5/month? I mean the big three (4 w DO) can all get you a bare machine for $5/mo that can run Docker. Why not give the k8s management side for free for those people?
Well, in this case the pop is me, and I know k8s but I don't want to invest 15-35 / month on a side project but would still rather have a decently managed site rather. I don't want to go back to FTP crap at this point.
A car is a hugely complicated machine and I'm sure in the 1910s people said only some people will ever need them. Bad analogy considering cars kill lots of people but take that part out :)
A ton of the value from k8s is in the programmatic and replicable configuration. 1 computer pets are still bad in a mom and pop shop. 1 computer livestock is still better.
Not to mention everything you have to do to get a machine to parity with Kubernetes--set up SSH, process management, monitoring, logging, etc. I don't think I'd go so far as to say Kubernetes is the right tool for this use case, but I certainly empathize with the sentiment.
But being able to blow away a k8s node and bring it back up in 2 minutes (or have k8s auto-spawn a replacement node) is WAY faster than having to reinstall wordpress, reinstall nginx, set IP addresses, fiddle with your DNS, restore a database from a backup, etc, etc, etc.
In this case the state can come from a centrally managed k8s controller. Something that most cloud providers provide and have good SLAs on.
Kubernetes just makes that story so much simpler. Heck a good cloud provider can just handle any machines fucking up automatically without any human intervention on your side.
From my experience, deploying to a k8s cluster managed by someone else is extremely simple and straightforward. Tell it which container to run and off it goes. You get zero-downtime rolling deploys, automatic restarts on crashes, easy rollbacks, service discovery (if you have multiple services) and more for free.
Managing a cluster is the tricky part of k8s but just running your stuff on an existing cluster is extremely straightforward. I could definitely see the attraction of a cheap managed k8s cluster with a single node. Maybe hosted k3s or something.
IaaS model is more 'everything is chargeable in small units', a free control plane doesn't really gel with that IMO, and also they have some huge users that it'd cost a lot to run the control plane for, pushing up the price of what you connect to it (or something must give) and making more PaaS-style like DO & Scaleway anyway.
You can get a small cluster from Digital Ocean pretty cheaply. It's more than $5 (I forget the exact price) but they'll provide a free master node and then you just pay for one additional kubelet (and you can add more as you need them). I have used it for one hobby project and have been pretty happy with it.
Same with Linode, the management machines are not massive scale HA control planes like you get with the expensive hosts (It seems like Linode may run a single 2GB droplet), but for most of my clusters I don't need it.
What is your utilization? IMHO I would rather do one EC2 instance then have EC2 Batch spot and lambda workers take the other load. In a good design most media assets should never hit the full time provisioned server.
And, not everything should be in a container. Just because you can does not necessarily mean you should. Containers are awesome but they are not a cure-all silver bullet.
I found this to be actually quite a well balanced take. There are quite a lot of useful considerations in here which I don't often see in HN discussions about k8s, as it's usually from people well invested in that ecosystem now.
Some of our teams have been drifting naturally towards ECS Fargate and found it to be a great fit. Almost invariably the k8s question comes up, and it's usually a reaction to the popularity of k8s and EKS. It's quite easy to start doubting your own well thought-out decisions in the face of a large popular elephant sitting in your rooms which wants to trample on everything.
My friend and colleague, Paul Stack: https://twitter.com/stack72 are at very opposite ends of the spectrum when it comes to Kubernetes. We put our heads together and tried to create a balanced take on Kubernetes where our opinions didn't cloud the article. It's obviously very polarizing, but I think we managed to ask some good questions before you started the adoption process.
Thx for writing that. To me it seems mind-boggling that in the 21st century you still have to point out to people that you shouldn't just adopt a technology because it seems cool. Of course you should ask yourself what problem you're trying to solve. Apparently people still don't get this.
I'm not sure I agree about templating YAML files–why is templating JSON any better, really?–but YMMV.
I agree that technology shouldn't be adopted just because it is new unfortunately the job market makes it so many early engineers (and already established engineers that will want to move at some point) end up practicing RDD as a matter of self interest.
there's basically no other options for running your own cloud & there's no particular major weak points for Kubernetes that other containerizing automation engines don't also have.
people think they're being smart by telling you to check whether such and such fits your needs. usually it's not bad advice. but the deck is pretty stacked here. nothing else has shown itself to be as flexible, as extensible, as consistent. well demonstrated but it's capability at handling many different add-ons (controllers, operators). rarely is cloud architecture do readily deployable to other non-core concerns, especially via the core tools & constructs.
if you want to go march off in to tech that does less, that will only do a little bit, that less people will be familiar with, that has less community, sure. spend a couple hours justifying to yourself why you are not going to use the really amazing all encompassing tech that's quickly supplanted everything else. pick some other combination of technologies which you will assemble, make your own stack, yeah, you're an independent thinker, you don't need convention or all this extra stuff! it's just gonna slow you down. go for it.
Kubernetes is not easy. but I find arguing against it to be difficult. there have been tools upon tools upon tools for so long. but little out there has brought the different concerns together, found overarching paradigms to create a basic, trusted quality of life that Kubernetes has clarified. again, it's not easy, but learning it ought to keep paying forward again & again, as you pick up new interests, want new services. even if those are Amazon services, ACK now will let you build Kubernetes deployment units that include those AWS services. the paradigm works.
Just because you have to trust your roommates a little doesn't mean that the idea of roommates is broken by design. It just means that you should be careful who you sign a lease with.
> You understand how VPS/EC2 etc work. They are also multi-tenant architectures
EC2 basic instances are multitenant, but there are such a thing as EC2 dedicated instances (and dedicated hosts), which run on single-customer reserved hardware.
Unfortunately, the hoopla is loud enough that developers could reasonably be expected to push for it just so they could have real experience to put on their resumes.
I think the answer is Linux on Linux. Fault Tolerant Linux on Distributed Machine Linux on traditional multicore Linux. Distributed machine Linux just sees the multicore Linux boxes as devices. Fault Tolerant Linux does reruns when things fail.
62 comments
[ 3.4 ms ] story [ 129 ms ] threadOpenshift is big in on-prem enterprise setups because as flawed as the install process is or as ridiculous licensing is, you get a remarkably complete system pretty easily out of a bunch of servers that integrates with things like your corporate LDAP.
I think part of the issue is that many of the bigger players want you to use their managed k8s solution and not just give you their setup - I think that's a temporary thing. A few of the big boons of k8s is the ability to be multi-cloud, and vendors are losing out on a revenue stream by not embracing that (though, AWS has a new multi-cloud k8s thing, so maybe they get it).
One can learn the developer side of K8's in a week or so enough to be deploying apps and setting up services and pods without the overhead of managing etcd and distributed clusters.
Setup = rep groups, internal services, etc? You need k8s-savvy ops people, or really good devs who like ops stuff.
They are called DevOps Engineers and it's a standard role these days.
And it doesn't matter whether they like it or not. In many organisations these days developers own their piece of work all the way through to production and are expected to provide an ops-like support for it.
In an ideal world, you could purchase or rent commoditized resources (compute, storage, bandwidth) and have those resources deploy your application in a standardized way. K8s is so close to that out of the gate, that having to use cloud provider's expensive and proprietary offerings feels sub-optimal.
I agree that cloud offerings are the way to go for anyone without good reasons not to (eg compliance, lack of connectivity, or full control over upgrades). But a lot of the "I prefer on prem" attitude comes from realizing that having a black box of an ingress controller on my cluster isn't the end state of how apps get deployed
But people still use the cloud.
OTOH if a service is available and would work for someone's use case, by all means, use the service. I've set up native k8s clusters just as exercises, and it's no picnic–and I didn't need to manage those clusters, later. Just doing build exercises alone was enough to scare me straight.
It feels awesome from a developer standpoint once you work out all the kinks. It's very natural describing your service with a bit of yaml, and then being able to then spawn it into your infrastructure and have it automatically scale out.
If there was a cost effective k8s cluster you could rent then that would be the thing that finally kills traditional infrastructure for me.
A car is a hugely complicated machine and I'm sure in the 1910s people said only some people will ever need them. Bad analogy considering cars kill lots of people but take that part out :)
After your single cattle is murdered, the next one has to be raised right?
State has to come from somewhere. Downtime has a cost for most businesses.
But being able to blow away a k8s node and bring it back up in 2 minutes (or have k8s auto-spawn a replacement node) is WAY faster than having to reinstall wordpress, reinstall nginx, set IP addresses, fiddle with your DNS, restore a database from a backup, etc, etc, etc.
Kubernetes just makes that story so much simpler. Heck a good cloud provider can just handle any machines fucking up automatically without any human intervention on your side.
The same can't be said for a single machine.
Managing a cluster is the tricky part of k8s but just running your stuff on an existing cluster is extremely straightforward. I could definitely see the attraction of a cheap managed k8s cluster with a single node. Maybe hosted k3s or something.
https://www.digitalocean.com/products/kubernetes/
https://www.scaleway.com/en/kubernetes-kapsule/
IaaS model is more 'everything is chargeable in small units', a free control plane doesn't really gel with that IMO, and also they have some huge users that it'd cost a lot to run the control plane for, pushing up the price of what you connect to it (or something must give) and making more PaaS-style like DO & Scaleway anyway.
Make sure to use metal LB or all your ingresses will be served with Google LBs that are hecka expensive.
I forgot to tear down a small EKS cluster last month. 4 days, 60€.
I run a 3-node Kubernetes cluster on Google cloud. It costs me around $100 per month (with 1-year reserved instances). Not terribly expensive.
Some of our teams have been drifting naturally towards ECS Fargate and found it to be a great fit. Almost invariably the k8s question comes up, and it's usually a reaction to the popularity of k8s and EKS. It's quite easy to start doubting your own well thought-out decisions in the face of a large popular elephant sitting in your rooms which wants to trample on everything.
ECS is definitely more suited for small use cases however.
https://github.com/rebataur/fskube
My friend and colleague, Paul Stack: https://twitter.com/stack72 are at very opposite ends of the spectrum when it comes to Kubernetes. We put our heads together and tried to create a balanced take on Kubernetes where our opinions didn't cloud the article. It's obviously very polarizing, but I think we managed to ask some good questions before you started the adoption process.
Would love to hear more via twitter too! https://twitter.com/briggsl
I'm not sure I agree about templating YAML files–why is templating JSON any better, really?–but YMMV.
people think they're being smart by telling you to check whether such and such fits your needs. usually it's not bad advice. but the deck is pretty stacked here. nothing else has shown itself to be as flexible, as extensible, as consistent. well demonstrated but it's capability at handling many different add-ons (controllers, operators). rarely is cloud architecture do readily deployable to other non-core concerns, especially via the core tools & constructs.
if you want to go march off in to tech that does less, that will only do a little bit, that less people will be familiar with, that has less community, sure. spend a couple hours justifying to yourself why you are not going to use the really amazing all encompassing tech that's quickly supplanted everything else. pick some other combination of technologies which you will assemble, make your own stack, yeah, you're an independent thinker, you don't need convention or all this extra stuff! it's just gonna slow you down. go for it.
Kubernetes is not easy. but I find arguing against it to be difficult. there have been tools upon tools upon tools for so long. but little out there has brought the different concerns together, found overarching paradigms to create a basic, trusted quality of life that Kubernetes has clarified. again, it's not easy, but learning it ought to keep paying forward again & again, as you pick up new interests, want new services. even if those are Amazon services, ACK now will let you build Kubernetes deployment units that include those AWS services. the paradigm works.
https://thenewstack.io/unfixable-kubernetes-security-hole-me...
You can't trust the cloud provider:
https://unit42.paloaltonetworks.com/windows-server-container...
https://media-exp1.licdn.com/dms/image/C5622AQH70okwLkMlNg/f...
What about the concept of not having roommates? Plenty of infrastructure alternatives exist besides "containers".
So in your world you have to only use bare metal ? Sounds fun.
EC2 basic instances are multitenant, but there are such a thing as EC2 dedicated instances (and dedicated hosts), which run on single-customer reserved hardware.
It's a bug but a very minor one. Especially if you are securing traffic inside the cluster.
* Easy simulation of networks between cgroups.
* A mechanism for splitting a ulimit across kernels.
* Hot swappable kernels and kernel tuning parameters so machines can be quickly re-configured.