You say that like it's a bad thing. A declarative model is infinitely better for representing complex systems than scripts and mind space. The challenge is actually being able to get to that point.
An sh script is a series of things that almost any developer who uses Linux can understand: command line statements. We use them all the time. Suppose the complicated declarative model you've made doesn't work one day and the person who originally wrote it is gone? Even if you have someone to debug it who knows the k8s languages: usually you can't just use the yml files alone, you need terraform or something, plus maybe some other services and "sidecar" containers that do other things for you. With a sh script, you just have the script with a bunch of commands that you can understand and look up easily, in a linear order, to figure out the problem. You might not understand every command, but you can run each one until you get to the error, then focus in on that area. With k8s, you need to figure out a huge series of intermixed deps and networks and services just to start, then find the one that is failing (if that is the one failing and it's not just being masked by another failed service that you didn't know about).
It's a developer tool made originally by google. Of course it's popular. Which isn't to say it's bad, it's just not much of a question as to why it's popular.
-------
Kubernetes - kubernetes.io
Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. It was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation.
Kubernetes is very complex and took a long time to learn properly. And there have been fires among the way. I plan to write extensively on my blog about it.
But at the end of the day: having my entire application stack as YAML files, fully reproducible [1] is invaluable. Even cron jobs.
Note: I don't use micro services, service meshes, or any fancy stuff. Just a plain ol' Django monolith.
Maybe there's room for a simpler IAC solution out there. Swarm looked promising then fizzled. But right now the leader is k8s[2] and for that alone it's worth it.
[1] Combined with Terraform
[2] There are other proprietary solutions. But k8s is vendor agnostic. I can and have repointed my entire infrastructure with minimal fuss.
I've just started to look into it, but it seems like the project has been focusing on improving the onboarding experience since it has a reputation for being a huge pain to set up. Do you think it has gotten easier lately?
No. Not easier in my opinion. And some of the fires you only learn after getting burnt badly. [1]
Note: my experience was all with cloud-provided Kubernetes, never running my own. So it was already an order of magnitude easier. Can't even imagine rolling my own. [2]
Consistency and standardized interfaces for AppOps regardless of the hyper-cloud I use. Kubernetes basically has an equivalent learning curve, but you only have to do it once
They operate at different layers. K8s sits on top of the infrastructure which terraform provisions. It's far more dynamic and operates at runtime, compared to terraform which you execute ad-hoc from an imperative tool (and so only makes sense for the low level things that don't change often).
Nothing. We use Terraform to provision a simple auto-scaling cluster with loadbalancers and certs, does exactly the same thing but there is no Docker and k8s. Few million lines less Go code turning yaml filed into seggfaults.
EKS, GKE and the like have a number of limitations. For example: they can be pretty far behind in the version of K8S they support (GKE is at 1.15 currently, EKS at 1.16; K8S 1.18 was released in at the end of March this year.
can it handle networking (including load balancing and reverse proxies with automatic TLS) or virtualized persistent storage? Make it easy to integrate common logging system?
Cause those are the parts that I miss probably the most when dealing with non-k8s deployment, and I haven't had the occasion to use Nomad.
For load balancing you can just run one of the common LB solutions (nginx, haproxy, Traefik) and pick up the services from the Consul service catalog. Traefik makes it quite nice since it integrates with LetsEncrypt and you can setup the routing with tags in your Nomad jobs: https://learn.hashicorp.com/nomad/load-balancing/traefik
What Nomad doesn’t do is setup a cloud provider load balancer for you.
Those are the advantage and the problem of nomad. We're using it a lot by now.
Nomad, or rather, a Nomad/Consul/Vault stack doesn't have these things included. You need to go and pick a consul-aware loadbalancer like traefik, figure out a CSI volume provider or a consul-aware database clustering like postgres with patroni, think about logging sidecars or logging instances on container hosts. Lots of fiddly, fiddly things to figure out from an operative perspective until you have a platform your development can just use. Certainly less of an out-of-the-box experience than K8.
However, I would like to mention that K8 can be an evil half-truth. "Just self-hosting a K8 cluster" basically means doing all of the shit above, except its "just self-hosting k8". Nomad allows you to delay certain choices and implementations, or glue together existing infrastructure.
I count the "glue it with existing infrastructure" to be higher cost than doing it from scratch. It was one feature that I definitely knew regarding Nomad, as one or two people who used it did chime in years ago in discussion, but for various reasons that might not be applicable to everyone I consider those unnecessary complication :)
Depending on how big the infrastructure is and how long you want to migrate over... usually there's not enough resources to "redo all from scratch", millions of LoC are already in production, people who owned key services are no longer in the company, other priorities for business exists other than have what you have already working in k8s..
The context was rather different (home setup), but all that you mention can be used as arguments Noth for and against redo, basing on situation in company, future needs, etc.
I have actually done a "lift and shift" where we moved code that had no support or directly antagonistic one to k8s because various problems reached situation where CEO said "replace the old vendor completely" - we ended up using k8s to wrestle with the amount of code to redeploy.
nomad is strictly a job scheduler. If you want networking you add consul to it and they integrate nicely. Logging is handled similarly to Kubernetes.
Cool thing about Nomad that it's less prescriptive
- Yes, just added CSI plugin support. Previously had ephemeral_disk and host_volume configuration options, as well as the ability to use docker storage plugins (portworx)
- I haven’t personally played with it, but apparently nomad does export some metrics, and they’re working on making it better
I've been using Nomad for my "toy" network, and I like it. It runs many services, and a few periodic jobs. Lightweight, easy to set up, and has enough depth to handle some of the weirder stuff.
Nomad, in its free offering, cannot compete with k8s for organization-wide usage:
- no RBAC
- no quotas
- no preemption
- no namespacing
This means: everyone is root on the cluster, including any CI/CD system that wants to test/update code. And there's no way to contain runaway processes with quotas/preemption.
> I can and have repointed my entire infrastructure with minimal fuss.
When you get to that blog post please consider going in depth on this. Would love to see actual battletested information vs. the usual handwavy "it works everywhere".
I sure will. 99% of the work was ingress handling and SSL cert generation. Everything else was fairly seamless.
Even ingress is trivial if you use a cloud balancer per ingress. But I wanted to save money so use a single cloud balancer for multiple ingresses. So you need something like ingress-nginx, which has a few vendor-specific subtleties.
According to the article you are wrong about "infrastructure as code". Kubernetes is infrastructure as data, specifically YAML files. Puppet and Chef are infrastructure as code.
Edit: not sure why the down votes, I was just trying to point out what seems like a big distinction that the article is trying to make.
They don't do loops or recursion. They don't even do iterative steps in the way that Ansible YAML has plays/tasks.
Yes, higher-level tools like Kustomize or Jsonnet or whatever else you use for templating the files are Turing-complete - but that's at the level of you on your machine generating input to Kubernetes, not at the level of Kubernetes itself. That's a valuable distinction - it means you can't have a Kubernetes manifest get halfway through and fail the way that you can have an Ansible playbook get halfway through and fail; there's no "halfway." If something fails halfway through your Jsonnet, it fails in template expansion without actually doing anything to your infrastructure.
(You can, of course, have it run out of resources or hit quota issues partway through deploying some manifest, but there's no ordering constraint - it won't refuse to run the "rest" of the "steps" because an "earlier step" failed, there's no such thing. You can address the issue, and Kubernetes will resume trying to shape reality to match your manifest just as if some hardware failed at runtime and you were recovering, or whatever.)
I think you could think of "infrastructure as code" as he described it as a superset of "infrastructure as data". Both have the benefit of being able to be reproducibly checked into a repo. Declarative systems like Kubernetes/"infrastructure as data" just go even further in de-emphasizing the state of the servers and make it harder to get yourself into unreproducible situations.
With "Swarm", do you mean Docker Swarm? Why has it "fizzled"?
The way I learned it in Bret Fisher's Udemy course, Swarm is very much relevant, and will be supported indefinitely. It seems to be a much simpler version of Kubernetes. It has both composition in YAML files (i.e. all your containers together) and the distribution over nodes. What else do you need before you hit corporation-scale requirements?
> What else do you need before you hit corporation-scale requirements?
Cronjobs, configmaps, and dynamically allocated persistent volumes have been big ones for our small corporation. Access control also, but I'm less aware of the details here, other than that our ops is happier to hand out credentials with limited access, which was somehow much more difficult with swarm
Swarm has frankly also been buggy. "Dead" but still running containers - sometimes visible to swarm, sometimes only the local Docker daemon - happen every 1-2 months, and it takes forever to figure out what's going on each time.
I use Swarm in production and am learning k8s as fast as possible because of how bad Swarm is:
1. Swarm is dead in the water. No big releases/development afaik recently
2. Swarm for me has been a disaster because after a couple of days some of my nodes slowly start failing (although they’re perfectly normal) and I have to manually remove each node from the swarm, join them, and start everything up again. I think this might be because of some WireGuard incompatibility, but the strange thing is that it works for a week sometimes and other times just a few hours
To add another side, I use Swarm in production and continue to do so because of how good it is.
I've had clusters running for years without issue. I've even used it for packaging B2B software, where customers use it both in cloud and on-prem - no issues whatsoever.
I've looked at k8s a few times, but it's vastly more complex than Swarm (which is basically Docker Compose with cluster support), and would add nothing for my use case.
I'm sure a lot of people need the functionality that k8s brings, but I'm also sure that many would be better suited to Swarm.
This happened to me too when I was using swarm in 2017. Had to debug swarm networks where nodes could send packets one way but not the other. Similar problems as #2 where stuff just breaks and resetting the node is the quickest way I found to fix it.
Switched to k8s in late 2017 and it’s been much more solid. And that’s where the world has moved, so I’m not sure why you’d choose swarm anymore.
A little off topic but why do these orchestration tools always prefer to use YAML as I really feel it a harder to understand format than JSON or better TOML and it's the only thing that I don't like about Ansible.
I see ruby kind of uses YAML often but are people comfortable editing YAML files? I always have to look up how to do arrays and such when I edit them once in a while.
Anything new needs a certain amount of sustained practice before you get the hang of it. I think I had to learn regex like four times before it stuck. I haven’t hit that point with TOML yet so I avoid it.
I’d suggest using the deeper indentation style where hyphens for arrays are also indented two spaces under the parent element. Like anything use a linter that enforces unambiguous indentation.
I prefer YAML for human-writeable config because JSON is just more typing and more finicky. The auto-typing of numbers and booleans in YAML is a pretty damn sharp edge though and I wish they’d solved that some other way.
Out of curiosity why do you feel swarm fizzled out?
I’ve deployed swarm in a home lab and found it really simple to work with, and enjoyable to use. I haven’t tried k8, but I often see view points like yours stating that k8 is vastly superior.
So are you saying that, no matter what, if you want to reply your whole infrastructure as code (networks, dmz, hosts, services, apps, backups etc ) that you are going to have to reproduce that somehow (whatever the combo of AWS services are, OR just learn K8S
Effectively, "every infrastructure as code project will reimplement Kubernetes in Bash"
Kubernetes is outstanding because proving ol bash scripts were bad and re-writing all Bash glue in Go glue, slapping 150 APIs on top (60 "new" and about hundred "old versions" https://kubernetes.io/docs/reference/generated/kubernetes-ap...), adding few dozen opensource must have projects - so finally it can be called "cloud native" - boom, a new cloud native bash for the cloud is born!
Bash glue has rarely failed me. Never used k8s, but the horror stories I hear ensure I won’t feel dirty for writing a little bash to solve problems anytime soon.
Instead of a varied interface tools set, I can have one with consistent interfaces and experiences. Kubernetes is where everything is going, hence why TF and Ansible have both recently released Kubernetes related products / features. It's their last attempt at remaining relevant (which is more than likely wasted effort in the long run). They have too much baggage (existing users in another paradigm) to make a successful pivot.
Ironically for me, those two tools are part of the blessed triad that we use for all of our infrastructure as code and end-user virtual machine initial setup.
If we only got to keep two tools it would be kubernetes and terraform.
This isn’t a competition, they are tools. Ansible is widely used and will continue to be so for a long long time. Its foundations - ssh, python and yaml are also in for the long run to manage infrastructure...
Yaml is on the way out, Cuelang will replace it where it's used for infra. It's quite easy to start by validating yaml and then you quickly realize how awesome having your config in a well thought-out language is!
well, then perhaps the growing frustration with a configuration language where meaning depends on invisible chars is an argument. And then what helm and others are doing text interpolating and add helpers for managing indentation.
There are many experiments into alternatives happening right now, so I do believe yaml's days are numbered. I'm actively replacing it where ever I encounter it with a far superior alternative. Cue is far more than a configuration language however, worth the time to learn and adopt at this point.
I worked in networking for the longest time. When I started there network guys and server guys (at least where I was). They were different people who did different things who kinda worked together.
Then there were storage area networks and similar, networks really FOR the server and storage guys.... that kind of extended the server world over some of the network.
Then comes VMware and such things and now there was a network in a box somewhere that was entirely the server guy's deal (well except when we had to help them... always).
Then we also had load balances who in their own way were a sort of code for networks ... depending on how you looked at it (open ticket #11111 of 'please stop hard coding ip addresses').
You also had a lot of software defined networking type things and so forth brewing up in dozens of different ways.
Granted these descriptions are not exact, there were ebs and flows and some tech that sort of did this (or tired) all along. It all starts to evolve slowly into one entity.
Swarm is still supported and works. I have it running on my home server and love it.
Kubernetes is fine, but setting it up kind of feels like I'm trying to earn a PhD thesis. Swarm is dog-simple to get working and I've really had no issues in the three years that I've been running it.
The configs aren't as elaborate or as modular as Kubernetes, and that's a blessing as well as a curse; it's easy to set up and administer, but you have less control. Still, for small-to-mid-sized systems, I would still recommend Swarm.
> setting it up kind of feels like I'm trying to earn a PhD thesis.
The kind of people who has to both set the cluster up and keep it up and also has to develop the application and deploy it and keep it up etc is not the target audience.
K8s shines when the roles of managing the cluster and running workloads on it are separated. It defines a good contract between infrastructure and workload. It lets different people focus on different aspects.
Yes it still has rough edges, things that are either not there yet, or vestigial complexity of wrong turns that happened through it's history. But if you look at it through the lense of this corporate scenario it starts making more sense than when you just think of what a full-stack dev in a two person startup would rather use and fully own/understand.
There are elements of our company that want to move to Kubernetes for no real reason other than it's Kubernetes. I can't wait to see the look on their faces when they realise we'll have to employ someone full-time to manage our stack.
Do you have a recommended tutorial for engineer with backend background to setup a simple k8 infra in ec2, I am interested in understanding devops role better
One of the things nobody liked to talk about in public when test automation was slowly "replacing" testers is that if you had the testers write automation, they brought none of the engineering discipline we tend to take as a given to the problem.
It's hard to make tests maintainable. Doubly so if you aren't already versed in techniques to make code maintainable.
I wonder sometimes if we aren't repeating the same experiment with ops right now.
I'm not sure "a plain ol' Django monolith" with none of the "fancy stuff" is either what people are referring to when they say "kubernetes", or a great choice for that. I could run hello world on a Cray but that doesn't mean I can say I do supercomputing. Our team does use it for all the fancy stuff, and spends all day everyday for years now yamling, terraforming, salting, etc so theoretically our setup is "entire application stack as YAML files, fully reproducible", but if it fell apart tomorrow, I'd run for the hills. Basically, I think you're selling it from a position which doesn't use it to any degree which gives sufficient experience required to give in-depth assessment of it. You're selling me a Cray based on your helloworld.
Reading this charitably: I guess I agree. k8s is definitely overpowered for my needs. And I'm almost certain my blog or my business will never need that full power. Fully aware of that.
But I'm not sure one can find something of "the right power" that has the same support from cloud providers, the open source community, the critical mass, etc. [1]
Eventually, a standard "simplified" abstraction over k8s will emerge. Many already exist, but they're all over the place. And some are vendor specific (Google Cloud Run is basically just running k8s for you). Then if you need the power, you can eject. Something like Create React App, but by Kubernetes. Create Kubernetes App.
Curious why run it at all? The cost must be 10 times more this way. It is mostly for the fun of learning.
I come from the opposite approach. I have 4 servers two digital ocean $5 and two vulr $2.50 instances. One holds the db. One server as the frontend/code. One server to do heavy work and another to server a heavy site and holds backups. For $15 I'm hosting hundreds of sites, running so many background processes. I couldn't imagine hitting that point where k8s would make sense just for myself unless for fun.
How do you do automated deployments, though? I don't like using K8s for small stuff, but I am also extremely allergic to having to log on to a server to do anything. Dokku hits the sweet spot for me, but at work I would probably use Nomad instead.
What's wrong with logging in to a server? I love logging in to a server and tinkering with it. Sure, for those who operate fleets of hundreds it's not scalable, but for a few servers that's a pleasure.
The problem is when you need to duplicate that server or restore it due to some error, you have no idea what all the changes you made are.
Besides, it's additional hassle and a chance for things to go wrong, the way I have it set up now is that production gets a new deployment whenever something gets pushed to master and I don't have to do anything else.
> The problem is when you need to duplicate that server or restore it due to some error, you have no idea what all the changes you made are.
A text file with some setup notes is enough for simple needs, or something like Ansible if its more complex. A lot of web apps aren't much more than some files, a database, and maybe a config file or three (all of which should be versioned and backed up).
But this is a solved problem since... well, at least since the beginning of internet. I managed 1000s of Linux & BSD systems over the past 25 years and I have scripts that do all that since the mid 90s that automate everything. I never install anything manual; if I have to do something new, I first write + test a script to do that remotely. Also, all this 'containerization' is not new; I have been using debootstrap/chroot since around that time as well. I run cleanly separated multiple versions of legacy (it is a bit scary how much time it takes to move something written early 2000s to a modern Linux version) + modern applications without any (hosting/reproducibility) issues since forever (in internet years anyway).
True; I learned many years ago that that is not a good plan. Although, I too, love it. But I use my X220 and Openpandora at home to satisfy that need. Those setups I could not reproduce if you paid me.
Make backup of /etc and package list. Usually that's enough to quickly replicate a configuration. It's not like servers are crashing every day. I'm managing few servers for a last 5 years and I don't remember a single crash, they just work.
I'm logging into a server because i need to, not because its 'pleasure'.
I don't hate it but if you need to login to a server regularly because you need to do an apt upgrade, you should have enabled automatic security updates and not login every few days.
If your server runs full because of some logfiles or stuff, you should fix the underlying issue and not needing to login to a server.
You should trust your machines, independently if it is only one machine, 2, 3 or 100. You wanna be able to go on holiday and know your systems are stable, secure and doing their job.
And logging in also implies a snow flake. Doesn't matter as long as that machine runs and as long as you have not that many changes but k8s actually makes it very simple to finally have an abstraction layer for infrastructure.
This is what I like to do. In my case, even the CI/CD host is just a systemd service I wrote.
The service just runs a script that uses netcat to listen on a special port that I also configured GitHub to send webhooks to, and processes the hook/deploys if appropriate.
Then when it's done, systemd restarts the script (it is set to always restart) and we're locked and loaded again. It's about 15 lines of shell script in total.
Set your pod to pull image always and have entrypoint shell script that clones the repo, kill the pod so on restart you could get your code deployed.
You could run init container with Kaniko that pushes image to repo and then main container that pulls that back but for that you need to do kubectl rollout restart deploy <name>
If you are looking for pure CI/CD gitlab has awesome support or you could do Tekton or Argo. They can run on the same cluster.
Flux or Argo can help with this. The operator lives on your cluster, and ensures your cluster state matches a Git repo with all your configuration in it.
If you do, the recipe is to reduce the number of components, get the most reliable components you can find, and make the single points of failure redundant.
Saying you can use Kubernetes to turn whatever stupid crap people tend to deploy with it highly available, is like saying you can make an airliner reliable by installing some sort of super fancy electronic box inside. You don't get more reliability by adding more components.
Are you ok with your application going down for each upgrade? With Kubernetes, it's very simple to configure a deployment so that downtime doesn't happen.
If and only if your application supports it. Database schema upgrades can be tricky for instance, if you care about correctness.
On the other hand, atomic upgrades by stopping the old service and then starting the new service on a Linux command line (/Gitlab runner) can be done in 10 seconds (depending on the service of course – dynamic languages/frameworks sometimes are disadvantaged here). I doubt many customers will notice 10 second downtimes.
And that downtime can even be avoided without resorting to k8s. A simple blue-green deployment (supported by DNS or load balancer) is often all that's needed.
K8s only makes sense at near Google-scale, where you have a team dedicated to managing that infrastructure layer (on top of the folks managing the rest of the infrastructure). For almost everyone else, it's damaging to use it and introduces so much risk. Either your team learns k8s inside out (so a big chunk of their work becomes about managing k8s) or they cross their fingers and trust the black box (and when it fails, panic).
The most effective teams I've worked on have been the ones where the software engineers understand each layer of the stack (even if they have specialist areas of focus). That's not possible at FAANG scale, which is why the k8s abstraction makes sense there.
Takes a couple of minutes at most for an average application upgrade / deployment, a lot of places can deal with that. Reddit is less reliable than what I used to manage as a one man team.
> Saying you can use Kubernetes to turn whatever stupid crap people tend to deploy with it highly available, is like saying you can make an airliner reliable by installing some sort of super fancy electronic box inside. You don't get more reliability by adding more components.
This is a bit funny, considering Airbus jets use triple-redundancy and a voting system for some of their critical components. [1]
Are you using a framework (cPanel etc) for this or just individual servers talking to each other? Need to move my hosting to something more reliable and cheaper...
Do you manage to aggregate all logs on a single place? Do you have the same environment as staging? How do you upgrade your servers? Do you have multiple teams deploying on their own components? Do you have a monitoring/metric service? How do you query the results of a Cron? Can you rollback to the correct version when you detect an error at production?
remote syslog has been a thing for how many years?! As has using a standard distribution for your app with a non-root-user for each app, easily wiped and set up every deploy (hint: that's good for security too!). Monitoring was also a solved problem and I guess Cron logs to syslog. Rollback works just like a regular deploy? (I wonder how good k8s helps you with db-schema rollbacks?)
Setting all of that from scratch is not really that easy, and I wouldn't consider "monitoring" to have been a "solved problem". syslog over TCP/UDP had many issues which is why log file shippers happened, and you still need to aggregate and analyze it. Getting application to reliably log remotely is IMO easier with k8s than remote-writing syslog, as I can just whack the developer again and again till it logs to stdout/stderr then easily wrap it however I want.
Deploying as distribution package tends to not work well when you want to deploy it more than once on a specific server (which quickly leads us to classic end-result of that approach, which is VM per deployment minimum - been there, done that, still have scars).
Management of cron jobs was a shitshow, is a shitshow, and probably will be a shitshow except for those that run their crons using non-cron tools (which includes k8s).
Yes, k8s makes it easier and more consistent. But it's not like all the stuff from the past suddenly stopped working or was not possible like GP made it sound ;)
I'm learning Elixir now, and it's quite confusing to me how one would go about deploying Elixir with K8s. How much you should just let the runtime handle.
How much of K8s is just an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang.
I've been very pleased with Nomad. It strikes a good balance between complexity and the feature set. We use it in production for a medium sized cluster and the migration has been relatively painless. The nomad agent itself is a single binary that bootstraps a cluster using Raft consensus.
> But I'm not sure one can find something of "the right power" that has the same support from cloud providers, the open source community, the critical mass, etc.
I totally agree. I would dearly like something simpler than Kubernetes. But there isnt a managed Nomad service, and apparently nothing in between Dokku and managed Kubernetes either.
That’d be a great first step if the purpose is to learn Kubernetes. If, however, you want to set up a cluster for real use then you will need much more than bare bones Kubernetes (something that solves networking, monitoring, logging, security, backups and more) so consider using a distribution or a managed cloud service instead.
Setting up your own k8s from scratch is kind of like writing your own string class in C++: it’s a good exercise (if it’s valuable for your learning path) but you probably don’t want to use it for actual work.
Maintaining a cluster set up like that is a ton of work. And if you don’t perform an upgrade perfectly, you’ll have downtime. Tools like kops help a lot but you’ll still spend far more time than the $70/month it costs for a managed cluster.
I find that K3s is great for getting started. It has traefik included and its less of a learning curve to actually be productive, vs diving in with K8s and having to figure out way more pieces.
Conversely I think the trend of writing infrastructure as yaml may be the worst part of modern ops. It’s really hard to think of a worse language for this.
Its typing is too weak. It’s block scoping is dangerous. References, classes and attachments are all pretty bad for reuse. Schemas are bolt on and there is no standard query language for it.
Fortunately YAML is not actually necessary in k8s, and only provided as a convenience because writing JSON by hand (or Proto3, lol) is annoying verging on insane.
We can build higher level abstractions easily having a schema to target and we can build them in whatever we want. That's a big boon for me :)
For kubernetes thats absolutely true and I think more people should do that (disclaimer I work with this https://github.com/stripe/skycfg daily). I think it actually would be easier for people to understand the k8s system if they did.
But, yaml is now everywhere in the ops space. Config management systems use it, metrics systems use it, its the defacto configuration format right now and that is unfortunate cause its bad.
Do you have a good tutorial for doing Django or a standard 3 tier web app on Kubernetes? We are using kubernetes at my workspace, but it seems way too complicated to consider for something like that. Maybe if I can bridge the gap between architectures it will help.
Yeah, I think people are overthinking it. The real reason is that if you do a superficial investigation you will quickly come back with the impression that k8s is near universally supported across cloud vendors and gives an appearance of providing a portable solution where otherwise the only alternative would be vendor lock-in. It makes it a no-brainer for anybody starting out with a new cloud deployment.
Because everyone chases the newest, shiniest thing in tech, and it's not cool nor fun to make boring old stuff in C then copy one binary and maybe a config to the server.
Even if one does have a single binary and config file that one can just copy to a server and run, there's more to non-trivial deployments than that. For example, how do you do a zero-downtime deployment where you copy over a new binary, start it up, switch new requests over to the new version, but let the old one keep running until either it finishes handling all requests that it already received or a timeout is reached? One reason why Kubernetes is popular is that it provides a standard, cross-vendor solution to this and other problems.
Most web applications don't need any of that. Also, I didn't say k8s was useless, just that it's the new thing everyone wants (that they probably don't need).
Then you need to add management of storage for it, management of logs, integration of monitoring, healthchecks, maybe some multiple environment case because UAT is good thing to have, etc. etc.
That's more "basic farm vehicle / lorry" than Ferrari.
You always have those concerns, it's just implemented differently. Customer hurling abuse at you over the phone (or worse - in person) is a form of healthchecks and monitoring, if worse than often common "have someone log in to the server every day and check if it's alive".
So is frantically logging into server to manually truncate log files that filled your one and only disk volume and caused the above abuse.
So is "we're losing customers because of how slow it is" yet not having a single idea why it is slow, because it runs fast when dev checks on their laptop.
All of the above are based on actual real world events, sometimes involving large corporations. In fact, the large corps seem to have most issues with manual work, because they can afford throwing cannon fodder ^W^W "experienced engineers" at the problems.
At some point it becomes a question of what is good use of your time. I disagree heavily with people claiming that running kubernetes is somehow orders of magnitude more complex than anything else (especially with k3s and using non-etcd backing stores). The complexity is necessary complexity, which you can tackle in various ways including YOLO.
Sometimes the YOLO approach however bites in the worst moment, and spending time on bespoke scripts, or figuring out configuration drift, are all costs that show up as you tackle said complexity.
Personally, the reason I went with kubernetes in the first production deployment I did with it, after being vocal anti-docker person at work, was because of... cost efficiency. Both in terms of my time (even though we had to spend significant amount of time migrating, as it was lift&shift of existing software), and in terms of compute costs - thanks to heavily loaded nodes our worst compute bill never reached above 20% of previous "condition normal". I don't think we ever really had more than 10 servers on purpose. Using k8s paid for itself.
I've yet to meet anyone who can easily explain how the CNI, services, ingresses and pod network spaces all work together.
Everything is so interlinked and complicated that you need to understand vast swathes of kubernetes before you can attach any sort of complexity to the networking side.
I contrast that to it's scheduling and resourcing components which are relatively easy to explain and obvious.
Even storage is starting to move to overcomplication with CSI.
I half jokingly think K8s adoption is driven by consultants and cloud providers hoping to ensure a lock-in with the mechanics of actually deploying workloads on K8s.
An ingress object creates an nginx/nginx.conf. That nginx server has an IP address which has a round robin IPVS rule. When it gets the request it proxy's to a service ip which then round robins to the 10.0.0.0/8 container IP.
Ingress -> service -> pod
It is all very confusing but once you look behind the curtain it's straight forward if you know Linux networking and web servers. The cloud providers remove the requirement of needing Linux knowledge.
It's confusing because a lot of people being exposed to K8s don't necessarily know how Linux networking and web servers work. So there is a mix of terminology (services, ingress, ipvs, iptables, etc) and context that may not be understood if you didn't come from running/deploying Linux servers.
I don't think this is accurate which plays into the parents point, I guess.
Looking at the docs, ingress-nginx configures an upstream using endpoints, which are essentially Pod IPs, with skips kubernetes service based round-robin networking altogether.
Assuming you use an ingress that does configure services instead, and assuming you're using a service proxy that uses ipvs (i.e. kube-proxy in default settings) then your explanation would have been correct.
For the most part, kubernetes networking is as hard as networking with loads of automation. Often, depth in both those skills are pretty exclusive, but if you're using the popular and/or supported CNI not doing things like changing in-flight, your average dev just needs to learn basic k8s debugging such as kubectl get endpoints to check whether his service selectors are setup correctly, and curl them to check whether the pods are actually listening on those ports.
Assuming that like us, you spend the last 10 - 12 years deploying IPv6 and currently running servers on IPv6 only networks, the Kubernetes/Docker network stack is just plain broken. It can be done, but you need to start thinking about stuff like BGP.
Kubernetes should have been IPv6 only, with optional IPv4 ingress controllers.
It really feels like Kubernetes was developed by some enterprise Java developers. Nothing seems well defined, everything is done in the name of abstraction, but the rules of the abstraction are never clearly stated, only the purpose is.
I really hope someone takes the mantle of Leslie Lamport (creator of the language TLA - "the quixotic attempt to overcome engineers' antipathy towards mathematics") and replaces Kubernetes with some software with a first principles approach.
It helps going from the bottom up, IMO. It's a multi-agent blackboard system with elements of control theory, which is a mouthful, but it essentially builds from smaller blocks up.
Also, after OpenStack, the bar for "consulting-driven software" is far from reached :)
Services create a private internal DNS name that points to one or more pods (which are generally managed by a Deployment unless you're doing something advanced) and may be accessed from within your cluster. Services with Type=NodePort do the same and also allocate one or more ports on each of the hosts which proxies connections to the service inside the cluster. Services with Type=LoadBalancer do the same as Type=NodePort services and also configure a cloud load balancer with a fixed IP address to point to the exposed ports on the hosts.
A single Service with Type=LoadBalancer and one Deployment may be all you need on Kubernetes if you just want all connections from the load balancer immediately forwarded directly to the service.
But if you have multiple different services/deployments that you want as accessible under different URLs on a single IP/domain, then you'll want to use Ingresses. Ingresses let you do things like map specific URL paths to different services. Then you have an IngressController which runs a webserver in your cluster and it automatically uses your Ingresses to figure out where connections for different paths should be forwarded to. An IngressController also lets you configure the webserver to do certain pre-processing on incoming connections, like applying HTTPS, before proxying to your service. (The IngressController itself will usually use a Type=LoadBalancer service so that a load balancer connects to it, and then all of the Ingresses will point to regular Services.)
When you are coming from old sysadmin world and you mastered unix systems architecture and software - what k8s does is very straightforward because its the same you already know.
K8S is extremely complicated for huge swarm of webdevs and java developers that really reqlly dont understand how the stuff they use/code really works.
K8S was supposed to decrease the need for real sysadmins but in my view it actually increased the demand because of all the obscure issues one can face in production if they dont really understand what they are doing with K8S and how it works under the hoods.
I think you're right for small clusters, you end up needing more sysadmins. But to manage 1000 node Kubernetes cluster, I suspect it can be done with less administration
Im managing 20000 vcpus infra both k8s and plain vms. IMHO there is no big difference there. It all depends on the tools you are using around orchestration.
In my experience having good sysadmins is still the key to best infrastructure management no matter the size of the company.
It makes a bit more sense if you see Kubernetes as the new Linux: a common foundation that the industry agrees on, and that you can build other abstractions on top of. In particular Kubernetes is the Linux Kernel, while we are in the early days of discovering what the "Linux distro" equivalent is, which will make it much more friendly / usable to a wider audience
Likewise, Linux is also a confusing mess of different parts and nonsensical abstractions when you first approach it. It does take some time to understand how to use it, and in particular how to do effective troubleshooting when things aren't working the way you expect.
But I 100% agree--I think it's the new Linux. In 5-10 years, it'll be the "go to", if not sooner.
I'd say it's down to two things. First is the sheer amount of work they're putting into standardization. They just ripped out some pretty deep internal dependencies to create a new storage interface. They have an actual standards body overseen by the Linux Foundation. So I agree with the blog post there.
The second reason is also about standards, but using them more assertively. Docker had way more attention and activity until 2016 when Kubernetes published the Container Runtime Interface. By limiting the Docker features they would use, they leveled the playing field between Docker and other runtimes, making Docker much less exciting. Now, new isolation features are implemented down at the runc level and new management features tend to target Kubernetes because it works just as well with any CRI-compliant runtime. Developing for Docker feels like being locked in.
Yes. But I think k8s took a lot of attention away from Docker in terms of headspace and developer interest. RedHat for example is pushing CRI-O, which is a minimalist CRI-compliant runtime which lets admins focus even more on k8s and less on the whole runtime level.
Yes. The big difference, however, is that k8s removed docker from consideration when actually running the system. Yes, you have docker underneath, and are probably going to use docker to build the containers.
But deploy to k8s? There's no docker outside of few bits involving "how to get to the image", and the actual docker features that are used are also minimized. The result is that many warts of docker are completely bypassed and you don't have to deal with impact of legacy decisions, or try to wrangle system designed for easy use by developer at local machine into complex server deployment. And, IMHO, interfaces used by k8s for the advanced features are much, much better than interfaces used or exported by docker.
It's confusing, but Docker images (and image registries) are also an open standard that Docker implements [1].
A lot of the Kubernetes "cool kids" just run containerd instead of Docker. Docker itself also runs containerd, so when you're using Kubernetes with Docker, Kubernetes has to basically instruct Docker to set up the containers the same way it would if it were just talking to containerd directly. From a technical perspective, you're adding moving parts for no benefit.
If you use containerd in your cluster, you can then use Docker to build and push your images (from your own or a build machine), but pull and run them on your Kubernetes clusters without Docker.
as a sys-admin, I like k8s because it solves sys-admin problems in a standardized way. Things like, safe rolling deploys, consolidated logging, liveness and readiness probes, etc. And yes, also because it's repeatable. It takes all the boring tasks of my job and let's me focus on more meaningful work, like dashboards and monitoring.
I host about a dozen rails apps of different vintage and started switching from Dokku to Digital Ocean Kubernetes. I had a basic app deployed with load balancer and hosted DB in about 6 hours. Services like the nginx ingress are very powerful and it all feels really solid. I never understood Dokku internals either so them being vastly simpler is no help for me. I figured for better or worse kubernetes is here to stay and on DO it is easier than doing anything on AWS really. I have used AWS for about 5 years and have inherited things like terraformed ECS clusters and Beanstalk apps. I know way more about AWS but I feel you need to know so much that unless you only do ops you cannot really keep up.
I found deploying databases with Dokku to be really intuitive. CockroachDB is great, but still a lot more steps than dokku postgres:create <db>. The whole certificates thing is quite confusing. Otherwise, k3s on-prem is great
They are not comparable. You might use ansible, salt, puppet or chef to deploy kubelet, apiserver, etc. You could, barring those with self-love, even deploy Ansible tower on Kubernetes to manage your kubernetes infrastructure.
I'm not intimately familiar with those, but I did a lot of similar things with scripts.
As far as I can tell: those are imperative. At least in some areas.
Kubernetes is declarative. You mention the end state and it just "figures it out". Mind you, with issues sometimes.
All abstractions leak. Note that k8s's adamance about declarative configuration can make you bend over backwards. Example: running a migration script post deploys. Or waiting for other services to start before starting your own. Etc.
I think in many ways, those compete with Terraform which is "declarative"-ish. There's very much a state file.
I've only ever used cfengine and Ansible, but they are both declarative. Hell, Ansible uses yaml files too.
I would be somewhat surprised to find out Puppet and Chef weren't declarative either. Because setting up a system in an imperative fashion is ripe for trouble. You may as well use bash scripts at that point.
I've used Ansible for close to 10 years for hobby projects. And setting up my development environment. Give me a freshly installed Ubuntu laptop, and I can have my development environment 100% setup with a single command.
Check out nix for actual development environments. Huge fan of that as well.
I can buy a new laptop and be back to 100% in a few minutes. Though the amount of time I spent learning how to get there far exceeds any time savings. Ever.
I've been testing out nix and I haven't found out how to install packages in a declarative way yet. Using "nix-env -iA <whatever>` seems really imperative.
How are you doing that? Do you use something like home-manager, or do you just define a default.nix and then nix-shell it whenever you need something?
Yes `nix-env -iA` is installing packages in an imperative way. I think it is there to be some kind of tool that people from other OS can relate to. Purist say you should avoid using it for installing packages and instead list global packages in `/etc/nixos/configuration.nix` for globally installed packages and home-manager for user specific ones, and if you need temporarily just to try something out use `nix-shell -p <whatever>`.
Back to your second question, you can configure the system through `/etc/nixos/configuration.nix` it is enough to configure system as a service. Pretty much everything you could do through Chef/Puppet/Saltstack/Ansible/CFEngine etc.
home-manager is taking it a step further and do this kind of configuration per user. It is actually written in a way that can be added to NixOS (or nix-darwin for OS X users) to integrate with the main configuration so then when you're declaring users you can also provide a configuration for each of them.
So it all depends what you want to do, the main configuration.nix is good enough if your machine to run specific service, that's pretty much all you need, you don't care about each user configuration in that scenario, you just create users and start services using them.
If you have a workstation, home-manager while not essential can be used to take care of setting up your individual user settings, stuff like dot-files (although it goes beyond that). The benefit of using home-manager is that most of what you configure in it should be reusable on OS X as well.
If you care about local development, you can use Nix to declare what is needed, for example[1]. This is especially awesome if you have direnv + lorri installed
When you do that you magically will get your CDE (that includes all needed tools, in this case proper python version, you also enter equivalent of virtualenv with all dependencies installed and extra tools) by just entering the directory, if you don't have them installed all you have to do is just call `nix-shell`.
I also can't wait when Flakes[2] get merged. This will standardize setup like this and enable other possibilities.
I use Ansible for managing my infra, and the only time my playbooks look imperative is when I execute a shell script or similar, which is about 5% of total commands in my playbooks.
One way to test if your playbook is declarative is try to rearrange the states and have them in different order. If the playbook breaks with different order it is imperative.
Ansible is YAML, but it's definitely imperative YAML - each YAML file is a list of steps to execute. It uses YAML kind of like how Lisp uses S-expressions, as a nice data structure for people to write code in, but it's still code.
Sure, the steps are things like "if X hasn't been done yet, do it." That means it's idempotent imperative code. It doesn't mean it's declarative.
CFEngine is slightly less imperative, but when I was doing heavy CFEngine work I had a printout on my cubicle wall of the "normal ordering" because it was extremely relevant that CFEngine ran each step in a specific order and looped over that order until it converged, and I cared about things like whether a files promise or packages promise executed first so I could depend on one in the other.
Kubernetes - largely because it insists you use containers - doesn't have any concept of "steps". You tell it what you want your deployment to look like and it makes it happen. You simply do not have the ability to say, install this package, then edit this config file, then start this service, then start these five clients. It does make it harder to lift an existing design onto Kubernetes, but it means the result is much more robust. (For some of these things, you can use Dockerfiles, which are in fact imperative steps - but once a build has happened you use the image as an artifact. For other things, you're expected to write your systems so that the order between steps doesn't matter, which is quite a big thing to ask, but it is the only manageable way to automate large-scale deployments. On the flip side, it's overkill for scale-of-one tasks like setting up your development environment on a new laptop.)
You can save your Kubernetes manifests in any order. Stuff that depends on other stuff just won't come up until the other stuff exists.
For example, I can declare a Pod that mounts a Secret. If the Secret does not exist, the Pod won't start -- but once I create the Secret the pod will start without requiring further manual intervention.
What Kubernetes really is, under the hood, is a bunch of controllers that are constantly comparing the desired state of the world with the actual state, and taking action if the actual state does not match.
The configuration model exposed to users is declarative. The eventual consistency model means you don't need to tell it what order things need to be done.
A combination of things, mostly related to Kubernetes' scope and use case being different from Ansible/CFEngine/etc. Kubernetes actually runs your environment. Ansible/CFEngine/etc. set up an environment that runs somewhere else.
This is basically the benefit of "containerization" - it's not the containers themselves, it's the constraints they place on the problem space.
Kubernetes gives you limited tools for doing things to container images beyond running a single command - you can run initContainers and health checks, but the model is generally that you start a container from an image, run a command, and exit the container when the command exits. If you want the service to respawn, the whole container respawns. If you want to upgrade it, you delete the container and make a new one, you don't upgrade it in place.
If you want to, say, run a three-node database cluster, an Ansible playbook is likely to go to each machine, configure some apt sources, install a package, copy some auth keys around, create some firewall rules, start up the first database in initialization mode if it's a new deployment, connect the rest of the databases, etc. You can't take this approach in Kubernetes. Your software comes in via a Docker image, which is generated from an imperative Dockerfile (or whatever tool you like), but that happens ahead of time, outside of your running infrastructure. You can't (or shouldn't, at least) download and install software when the container starts up.
You also can't control the order when the containers start up - each DB process must be capable of syncing up with whichever DB instances happen to be running when it starts up. You can have a "controller" (https://kubernetes.io/docs/concepts/architecture/controller/) if you want loops, but a controller isn't really set up to be fully imperative, either. It gets to say, I want to go from here to point B, but it doesn't get much control of the steps to get there. And it has to be able to account for things like one database server disappearing at a random time. It can tell Kubernetes how point B looks different from point A, but that's it.
And since Kubernetes only runs containers, and containers abstract over machines (physical or virtual), it gets to insist that every time it runs some command, it runs in a fresh container. You don't have to have any logic for, how do I handle running the database if a previous version of the database was installed. It's not - you build a new fresh Docker image, and you run the database command in a container from that image. If the command exits, the container goes away, and Kubernetes starts a new container with another attempt to run that command. It can do that because it's not managing systems you provide it, it's managing containers that it creates. If you need to incrementally migrate your data from DB version 1 to 1.1, you can start up some fresh containers running version 1.1, wait for the data to sync, and then shut down version 1 - no in-place upgrades like you'd be tempted to do on full machines.
And yeah, for databases, you need to keep track of persistent storage, but that's explicitly specified in your config. You don't have any problems with configuration drift (a serious problem with large-scale Ansible/CFEngine/etc.) because there's nothing that's unexpectedly stateful. Everything is fully determined by what's specified in the latest version of your manifest because there's no other input to the system beyond that.
Again, the tradeoff is this makes quite a few constraints on your system design. They're all constraints that are long-term better if you're running at a large enough scale, but it's not clear the benefits are worth...
I agree. It is impressive how much it can orchestrate. It is also very useless in the real cloud because developers there are dealing with higher-level abstractions to solve problems for the business.
The most simplistic task - execute some code in response to even in a bucket - makes kubernetes with all its sophisticated convergence capabilities completely useless. And even if somebody figures this out and puts the opensource project on github to do this on kubernetes - it just going to break at slightest load.
Not to mention all the work to run kubernetes at any acceptable level of security, or keep the cost down, do all patching, scaling, logging, upgrades... Oh, the configuration management itself for kubernetes? Ah sorry, I forgot, there are 17 great open-source projects exists :)
> The most simplistic task - execute some code in response to even in a bucket - makes kubernetes with all its sophisticated convergence capabilities completely useless.
That's because you're not thinking web^Wcloud scale. To execute some code in response to event you need:
- several workers that will poll the source bucket for changes (of course you could've used existing notification mechanism like aws eventBridge, but that will couple you k8s to vendor-specific infra, so it kinda deminishes the point of k8s)
- distributed message bus with persistanse layer. Kafka will work nicely because they say so on Medium, even though it's not designed for this use case
- a bunch of stateless consumers for the events
- don't forget that you'll need to write processing code with concurrency in mind because you're actually executing it in truly destributed system at this point and you've made a poor choice for your messaging system
Wait, I can do all these with s3 and lambda at any scale - for pennies :) Will probably take few hours to set everything up with tools like stackery.io
So once again, why developers need kubernetes for? If the most simple problem becomes a habitholy mess :)
It is common to have significant logic and complexity in the configuration management manifests, but I'd argue that it's possible to move most of that to packaging and have your configuration management just be "package state latest, service state restarted."
Chef and from what I heard, since I didn't use it, Puppet are declarative, but since their DSL is really Ruby, it is really easy to introduce imperative code.
Ansible uses YAML, but when I used it few times it felt that you still use it in imperative way.
The saltstack (which also uses YAML) was the closest from that group (never used CFengine, but the author wrote research paper and shown that declarative is the way to go, so I would imagine he would also implement it that way).
If you truly want a declarative approach designated from a ground up, then you should try Nix or NixOS.
Having extensively used Chef and K8s, the difference is that they try to deal with chaos in unmanaged way (Puppet is the closest to "managed"), but when dealing with wild chaos you lack many ways of enforcing the order. Plus they don't really do multi-server computation of resources.
What k8s brings to the table is a level of standardization. It's the difference between bringing some level of robotics to manual loading and unloading of classic cargo ships, vs. the fully automated containerized ports.
With k8s, you get structure where you can wrap individual program's idiosyncracies into a container that exposes standard interface. This standard interface allows you to then easily drop it into server, with various topologies, resources, networking etc. handled through common interfaces.
I said that for a long time before, but recently I got to understand just how much work k8s can "take away" when I foolishly said "eh, it's only one server, I will run this the classic way. Then I spent 5 days on something that could be handled within an hour on k8s, because k8s virtualized away HTTP reverse proxies, persistent storage, and load balancing in general.
Now I'm thinking of deploying k8s at home, not to learn, but because I know it's easier for me to deploy nextcloud, or an ebook catalog, or whatever, using k8s than by setting up more classical configuration management system and deal with inevitable drift over time.
But what do you use to manage those containers and surrounding infra (networking, proxies, etc)?
I've been down the route of using Puppet for managing Docker containers on existing systems, Ansible, Terraform, Nomad/Consul. But in the end it all is just tying different solutions together to make it work.
Kubernetes (in the form of K3s or a other lightweight implementation) just works for me, even in a single server setup. I barely have to worry about the OS layer, I just flash K3s to a disk and only have to talk to the Kubernetes API to apply declarative configurations.
Only things I'm sometimes still need the OS layer for is networking, firewall or hardening of the base OS. But that configuration is mostly static anyways and I'm sure I will fine some operators for that to manage then through the Kubernetes API as IaC if I really need to.
I used to have a bunch of bash scripts for bootstrapping my docker containers. At one point I even made init scripts, but that was never fully successful.
And then one day I decided to set up kubernetes as a learning experiment. There is definitely some learning curve about making sure I understood what deployment, or replicaset or service or pod or ingress was, and how to properly set them up for my environment. But now that I have that, adding a new app to my cluster, and making it accessible is super low effort. i have previous yaml files to base my new app's config on.
It feels like the only reason not to use it would be learning curve and initial setup... but after I overcame the curve, it's been a much better experience than trying to orchestrate containers by hand.
Perhaps this is all doable without kubernetes, and there is a learning curve, but it's far from the complicated nightmare beast everyone makes it out to be (from the user side, maybe from the implementation details side)
It would mean I removed ~20% of the things that were annoying me and left 80% still to solve, while kubernetes goes 80% for me with the remaining 20% being mostly "assembly these blocks".
Plus, a huge plus of k8s for me was that it abstracted away horrible interfaces and behaviours of docker daemon and docker cli.
K3s on few devices. Thinking of grabbing a HP microserver or something with similar case for ITX ryzen (embedded EPYC would be probably too expensive), some storage space, maybe connect few extra bits of compute power into a heterogenous cluster. Put everything except maybe PiHole on it, with ZFS pool exported over bunch of protocols as backing store for persistent volume claim support.
What a container lets you do is move a bunch of imperative logic and decisions to build time, so that there are very few decisions made at deploy time. I'm not trying to have a bunch of decisions made that are worded as statements. I've watched a long succession of 'declarative' tools that make a bunch of decisions under the hood alienate a lot of people who can't or won't think that way, and nobody really should have to, even if they can. There are so many things I'd rather being doing with my day than dealing with these sorts of systems because otherwise it won't get done, and I'm heavily invested in the outcome.
I think the build, deploy, start and run-time split is an important aspect that gets overlooked quite a bit, and is critical to evaluating tools at this point. That is why we aren't still doing everything with Chef or Puppet. Whether we continue doing it with Kubernetes or Pulumi or something else matters a bit less.
Repeatability is not the goal, as others in this thread have implied. The goal is trusting that the button will work when you push it. That if it doesn't work, you can fix it, or find someone who can. Doing that without repeatability is pretty damned hard, certainly, but there are ways to chase repeatability without ever arriving at the actual goal.
For certain things like layer 4 and layer 7 routing or firewall policies, health checking and failover, network-attached volumes, etc you have to choose software and configure it on top of getting that configuration in that tooling. So you are doing kernel or iptables or nginx or monit/supervisord configurations and so on.
But basic versions of these things are provided by Kubernetes natively and can be declared in a way that is divorced from configuring the underlying software. So you just learn how to configure these broader concepts as services or ingresses or network policies, etc, and don't worry about the underlying implementations. It's pretty nice actually.
Kubernetes is one way to deploy containers. Configuration systems like Ansible/Salt/Puppet/Chef/etc are another way to deploy containers.
Kubernetes also makes it possible to dynamically scale your workload. But so does Auto Scaling Groups (AWS terminology) and GCP/Azure equivalents.
The reality is that 99% of users don't actually need Kubernetes. It introduces a huge amount of complexity, overhead, and instability for no benefit in most cases. The tech industry is highly trend driven. There is a lot of cargo culting. People want to build their resumes. They like novelty. Many people incorrectly believe that Kubernetes is the way to deploy containers.
And they (and their employers) suffer for it. Most users would be far better off using boring statically deployed containers from a configuration management system. Auto-scaled when required. This can also be entirely infrastructure-as-code compliant.
Containers are the real magic. But somehow people confused Kubernetes as a replacement for Docker containers, when it was actually a replacement for Docker's orchestration framework: Docker Swarm.
In fact, Kubernetes is a very dangerous chainsaw that most people are using to whittle in their laps.
Expectations have been set unrealistically high and online communities like this one make matters worse. Big players with dedicated devops teams use Kubernetes all the time so why shouldn't I? It's only a matter of time before "Hello world" tutorials include a chapter on container orchestration.
So we end up with a plethora of full stack developers who can barely keep up with their current development stacks willfully deploying their software on systems that they're just barely competent with.
I know this because I almost deployed a side project with Kubernetes because it was expected of me despite the fact that being mediocre at it was the best that I could hope to become and that's an easy way to chop off a leg or three.
Hm. Systemd already runs all your services in cgroups, so the same resource limit handles are available. It doesn't do filesystem isolation by default, but when we're talking about Go / Java / Python / Ruby software does that even matter? You statically link or package all your dependencies anyway.
Not only systemd runs your code it also does file system isolation built-in, runs containers, both privileged and non-privileged and sets up virtual networking for free.
systemd-nspawn / machined makes the other systems look like very complicated solutions in search of a problem
Name may not be pretty but it's an official feature of systemd which is used to debug the systemd development and it is far easier to take backups incrementally because the container files are just plain files in /var/lib/machines/ and apparently you already have it if systemd is on your system. (May need an additional package to be installed from OS package repo.)
I run nspawn instances as development environments for developers and I can also run docker inside it.
I've been using Kubernetes exclusively for the past two years after coming from a fairly large Saltstack shop. I think traditional configuration management is flawed. Configuration drift _will_ happen because something, somewhere, will do something you or the formula/module/playbook didn't account for. A Dockerfile builds the world from (almost) scratch and forces the runtime environment to be stateless. A CM tool constantly tries to shape the world to its image.
Kubernetes isn't a silver bullet of course, there will be applications where running it in containers adds unnecessary complexity, and those are best run in a VM managed by a CM tool. I'd argue using k8s is safe default for deploying new applications going forward.
It's not just the configuration format. There is a whole 'Kubernetes runtime' (what they call the 'control loops' aka 'controllers') that runs 24/7 watching the configuration live and making appropriate changes.
Unlike Ansible (and I suspect the others) where it's really only more of a 'run once' type of thing... And sometimes if you try running it a second time it won't even succeed.
Ansible is a little special in how imperative it is, a better comparison is Puppet which is intended to do periodic "convergence" runs, although these are more typically hourly or daily than continuous.
if you're on microservices, it's no brainer. You'll need an army of DevOps with semi-custom scripts to maintain the same. It's really automating a lot of stuff.
Helm + Kubernetes let our company's ability to launch microservices with no DevOps involved. You just provide the name of the project, push to git and GitLab CI will pick it up and do the stuff by the template. Even junior developers in our team are doing that from day one.
Isn't that a future we dream about? If you have too much load it will autoscale pod, if node is overloaded it will autoscale node pool, if you have a memory leak it will restart the app so you can sleep at night. I can provide a million examples that make our 100+ microservices management so much simpler. No Linux kungfu, 0 bash scrips, no SSH, and interaction with OS, not a single devops role for 15+ developers team.
Our management of cluster is just a simple "add more CPU or memory to this nodepool", sometimes change a nodepool name for deployment for certain service. All done simple cloud management UI.
For those who call microservices fancy stuff. No, we are a startup with fast delivery, deploy cycle. We have tons of subproject , integrations, and our main languages are nodejs, golang and python. Some of these are not good at multi-thread so no way to run it as a monolith. The other one is used only when it's needed for high performance.
So All together Microservices + Kubernetes + Helm + good CI + proper pubsub gives our backend extremely simple fast cycle of development, delivery, and what's important flexibility in terms of language/framework/version.
What is also good is the installation of services. With helm I can install high availability redis setup for free in 5 minutes. The same level of setup will cost you several thousand dollars for devops work and further maintenance and update. With k8s it's simple helm install stable/redis-ha
So yeah, I can totally understand some simple projects don't need k8s. I can understand you can build something is Scala and Java slowly but with high quality as a monolith. You don't need k8s for 3 services. I can understand some old DevOps don't want to learn new things and they complain about a tool that reduces the need of these guys. Otherwise, you really need k8s.
Because soon from one program on a dev server, there is a need to run databases, log gathering, multiply the previous to do parallel testing in clean environment, etc. etc.
Just running supporting tools for a small project where there was insistence on self-hosting open source tools instead of throwing money at slack and the like? K3s would have saved me weeks of work :|
K8s is great - if you are solving infrastructure at a certain scale. That scale being a Bank, Insurance Company or mature digital company. If you're not in that class then it's largely overkill/overcomplex IMO when you can simply use Terraform plus managed Docker host like ECS and attach cloud-native managed services.
Again the cross cloud portability is a non starter, unless you're really at scale.
k8s as a bunch of other benefits beside just scaling and you can run a single node cluster with the same uptime characteristics as your proposed setup and get all these benefits.
And, we only have to learn one complex system and avoid learning each cloud, one of which decided product names which have little relation to what they do was a good idear
What k8s really scales is the developer/operator power. Yes, it is complex, but pretty much all of it is necessary complexity. At small enough scale with enough time, you can dig a hole with your fingers - but a proper tool will do wonders to how much digging you can do. And a lot of that complexity is present even when you do everything the "old" way, it's just invisible toil.
And a lot of the calculus changes when 'managed services' stop being cost effective or aren't an option at all, or you just want to be able to migrate elsewhere (that can be at low scale too, because of being price conscious).
We have a mature TF module library and can roll out complex, well configured infra in a matter of hours, reliably. That said it's platform specific.
Sure, managed service costs are certainly a thing, but to my point that only really start to become an issue at significant scale, assuming you're well configured.
The cost metrics that make "it's cheaper to use managed service than pay the cost of extra engineer to specialize in infrastructure" aren't universal. In fact, I usually have to work from the opposite direction, where hiring a senior Ops specialist who can wrangle everything from shelving the physical hw to network booting k8s cluster on-premises can be cheaper that Heroku/AWS/etc.
> you can simply use Terraform plus managed Docker host like ECS and attach cloud-native managed services
That's not actually simple at all, and you would need to build a lot of the other stuff that Kubernetes gives you for free.
Kubernetes gives you an industry standard platform with first-class cloud vendor support. If you roll your own solution with ECS, what you are really doing is making a crappy in-house Kubernetes.
I'd disagree - my team migrated from running containers on VMs (managed via Ansible) to ECS + Fargate (managed by Terraform and a simple bash script).
It wasn't a simple transition by any means, but one person wrapped it up in 4 weeks - now we have 0 downtime deployments, scaling up/down in matter of seconds, and ECS babysits the containers.
Previously we had to deploy a lot of monitoring on each VM to ensure that containers are running, we get alerted when one of the application crashed and didn't restart because Docker daemon didn't handle it etc etc.
Now, we only run stateless services, in a private VPC subnet, Load balancing is delegated to ALB, we don't need service discovery, meshes etc. Configuration is declarative, but written in much friendlier HCL (I'm ok with YAML, but to a degree).
ECS just works for us.
Just like K8S might work for a bigger team, but I wouldn't adopt it at our shop, simply because of all of the complexity and huge surface area.
My question is: Why is only k8s so popular when there are better alternatives for a large swath of users? I believe the answer is "Manufactured Hype". k8s is from a purely architectural standpoint the way to go, even for smaller setups, but the concrete project is still complex enough that it requires dozens of different setup tools and will keep hordes of consultants as well as many hosted solutions from Google/AWS/etc in business for some time to come, so there's a vested interest in continuing to push it. Everyone wins, users get a solid tool (even if it's not the best for the job) and cloud providers retain their unique selling point over people setting up their own servers.
I still believe 90% of users would be better served by Nomad. And if someone says "developers want to use the most widely used tech", then I'm here to call bullshit, because the concepts between workload schedulers and orchestrators like k8s and nomad are easy enough to carry over from one side to the other. Learning either even if you end up using the other one is not a waste of time. Heck, I started out using CoreOS with fleetctl and even that taught me many valuable lessons.
I didn't say more users, I said appropriate for more users. The alternative I mentioned is Nomad and I wish more people would give it a try and decide for themselves. The momentum behind it is Hashicorp, makers of Vault, Consul, Terraform, Vagrant, all battle-proven tools. The fact that there's one big player behind it really shows in how polished the tool, UI and documentation is.
The issue that I have with managed k8s is that these products will decrease the pressure to improve k8s documentation, tooling and setup itself. And then there's folks (like me) who want or need to run something like k8s on bare metal hardware outside of a cloud where the cloud-managed solution isn't available.
> I still believe 90% of users would be better served by Nomad.
Well sure, but if the story just ended with "everyone use the least exciting tool", then there'd be few articles for tech journals to write.
But Kubernetes promises so much, and deep down everyone subtly thinks "what if I have to scale my project?" Why settle for good enough when you could settle for "awesome"? It's just human nature to choose the most exciting thing. And given that I do agree that there's some manufactured hype around Kubernetes, it isn't surprising to me why few are talking about Nomad.
I'm resisting kubernetes and might go with nomad (currently I'm "just using systemd" and I get HA from the BEAM VM)... But I do also get the argument that the difference between kubernetes and nomad is that increasingly kubernetes is supported by the cloud vendors, and nomad supports the cloud vendors.
I got a bit disillusioned with k8s and looked at Nomad as an alternative.
As a relatively noob sysadmin, I liked it a lot. Easy to deploy and easy to maintain. We've got a lot of mixed rented hardware + cloud VPS, and having one layer to unify them all seemed great.
Unfortunately I had a hard convincing the org to give it a serious shot. At the crux of it, it wasn't clear what 'production ready' Nomad should look like. It seemed like Nomad is useless without Consul, and you really should use Vault to do the PKI for all of it.
It's a bit frustrating how so many of the HashiCorp products are 'in for penny, in for a pound' type deals. I know there's _technically_ ways for you use Nomad without Consul, but it didn't seem like the happy path, and the community support was non-existent.
Please tell me why I'm wrong lol, I really wanted to love Nomad. We are running a mix of everything and its a nightmare
I'm sympathetic toward the idea of a system made of interchangeable parts, but I also kinda feel like it's a bit unrealistic, maybe? Even with well-defined interfaces, there will always be interop problems due to bugs or just people interpreting the interface specs differently. Every new piece to the puzzle adds another line (or several) to a testing matrix, and most projects just don't have the time and resources to do that kind of testing. It's unfortunate, but IMO understandable that there's often a well-tested happy path that everyone should use, even when theoretically things are modular and replaceable.
Nomad + Consul is the happy path. Adding Vault into the mix is nice, but not required.
Consul by itself is the game-changer. Even in k8s it's a game-changer. It solves so many questions in an elegant way.
"How do I find and reach the things running in (orchestrator) with (unknown ip/random port) from (legacy)?" being the most important. You run 5 servers, and a relatively lightweight client on everything (which isn't even outright required, but it sure is useful!), and you get a _lot_ with that.
Consul provides multiple interfaces and ingress points to find everything. It also is super easy to operate, and has a pretty big community.
If you absolutely cannot have Consul, Nomad is still a really good batch job engine, and makes a very great "distributed cron," which is more extensible, scalable, and easy to use than something like Jenkins for the same task.
My team is pretty small (was 4 people, now 6) and we manage one of the worlds largest nomad and consul clusters (there are some truly staggeringly large users of Vault so I won't make that claim). Even when shit really hits the fan, everything is designed in a way that stuff mostly works; and there's enough operator friendly entry points that we can always figure out the problem.
Vault is great for just a PKI, even if you aren't using it for anything else. There are some tools that just do PKI, but Vault works a real treat at it. Any Terraform backend that supports encryption + Terraform + Vault gives you such an amazing workflow. We use a mix of short and long certs, with different roles based on what's getting a cert.
For now, we have CRLs disabled on all short-lived backends, enabled on long-lived backends and we're actually looking at disabling storing short-lived certs in the storage system at all, and just cranking the TTL down to really truly short. We've tested it as low as 30m, but a more real-world max-ttl is 1 week, with individual apps setting it as low as they can handle. For reference we run more than 10 PKI backends, and adding one (or a bunch) more is just a little terraform snippet for us.
The way it works via hashicorp template land, is that you just plop
{{ with secret "name-of-pki/issue/name-of-role" "common_name=my.allowed.fqdn" "ttl=24h" }} {{ .Data.certificate }} {{ end }}
into your Nomad template stanza, or use consul-template directly as a binary, or use vault agent with it's template capability. You can get the CA chain if required the same way, just hitting a different PKI endpoint.
Also, as of Vault 1.4, Vault's internal raft backend is now production ready, making it a snap to run.
Try running through a few of the Vault quick-start guides, and replicating them in Terraform as much as possible. There's a few things TF does not handle gracefully last I checked (initial bootstrap), but you can get around that by using a null_resource or just handling that outside Terraform.
Edit: just noticed an actual Nomad user replied as well, and I like their answer better. Consider mine an addendum. :)
Batch workloads rarely require Consul, but for deploying your standard network services on Nomad: Consul is basically required. You could likely use any number of service mesh systems instead (either as sidecars, Docker network plugins, or soon CNI), but you'll be doing a lot of research and development on your own I'm afraid.
The Nomad team is by no means opposed to becoming more flexible in the future (and indeed better CNI support is landing soon as a first step), but we wanted to focus on getting one platform right and a pleasure to use before trying to genericize and modularize it.
Thanks for reaching out! Since I have the chance I'll add - Nomad is pretty awesome, and I love the work your team is doing.
My org looked at Nomad at a time when there was a lot of pressure from above to deliver something as soon as possible. Two weeks just weren't enough to full lay of the land ¯\_(ツ)_/¯
Funny thing is even if I could plug in my own service discovery into Nomad, I would probably chuck it away and replace it with Consul after a few weeks anyway haha
Nomad isn't really feature mature or user friendly enough, you still eventually need 100 bolt-ons.
I think a Distributed OS is the only sane solution. Build the features we need into the kernel and stop futzing around with 15 abstractions to just run an isolated process on multiple hosts.
As the Nomad Team Lead I sympathize with your first statement, but I hope our continued efforts will dissuade you from the second.
Linux (and the BSDs) are remarkably stable, festureful, and resilient operating systems. I would hate to give up such a strong foundation. Nomad can crash without affecting your running services. Nomad can be upgraded or reconfigured without affecting your running services. Nomad can be observed, developed, and debugged as a unit often without having to consider the abstractions that sit above or below it. The right number of abstractions is a beautiful thing. Just no more and no less. :)
TIL about kudo, The Kubernetes Universal Declarative Operator. We've been doing the exact same things in a custom go CLI for 2 years.
The kubernetes ecosystem is really amazing and full of invaluable resources. It's vast, complex, but well-thought. Getting to know all ins and outs of the project is time consuming. So much things to learn and so little time to practice...
I work on KUDO team. Would love to hear what you think about it. All devs hang out in #kudo channel on Kubernetes community slack, please don’t hesitate to join and say hi.
My opinion is that Kubernetes is the common integration point. Tons of stuff works with Kubernetes without having to know about each other, making deployments much much easier.
Lets use Bazel, and Bazel's rules_k8s to build\containerize\test\deploy only the microservices of my monorepo that changed.
Lets use Istio's "istioctl manifest apply" to deploy a service mesh to my cluster that allows me to pull auth logic / service discovery / load balancing / tracing out of my code and let Istio handle this.
Lets configure my app's infrastructure (Kafka (Strimzi), Yugabyte/Cockroach, etc) as yaml files. Being able to describe my kafka config (foo topic has 3 partitions, etc) in yaml is priceless.
Lets move my entire application and its infrastructure to another cloud provider by running a single bazel command.
k8s is the common denominator that makes all this possible.
Terraform explicitly doesn't want to deal with deployment of stuff that is inside VMs etc. and tries to tell you to use managed services or cloud-config yamls as the solution.
You can write your own providers, you can use the provisioned support, but TF doesn't like that and it shows.
I'll add my use case: we use hosted kubernetes to deploy all of our branches of all of our projects as fully functional application _stacks_, extremely similarly to how they will eventually run in production. Want to try something and show it to someone in the product owner level? Ok there will be a kube nginx-ingress backed environment up in build-time + a few minutes.
The environments advertise themselves via that same modified ingress's default backend. We stick a tiny bit of deploy yaml in our projects, the deployments kube tagging gives us all the details we need to provide diffs, last build time, links to git repos, web sites etc for the particular environment. The yaml demonstrates conclusively how an app could or should be run, regardless of os or software choice, so when we hand it to ops folks there is a basis for them to run from.
Because Google is an advertising company, their search engine controls what people believe in and they also have some good engineers but they are probably not well known. There is very little they couldn't advertise into popularity. Whenever you see overcomplicated software or infrastructure its always a way to waste executive function, create frustration and create unnecessary mental overhead. If the technology you're using isn't making it easier for you to run your infrastructure from memory, reduce the use of executive function and decrease frustration then you should ignore it. Don't fall for the fashion trends.
I'm not criticizing. I've actually used kubernetes and read the source code. It's a good tool, I just think its too much mental overhead for most companies since they won't use most of what it provides. If you're working on a large team with responsible parties who have a clearly defined roles it is a great tool but I've seen two person projects with startup infrastructure waste obscene amounts of time learning Kubernetes when they could have just stood up something basic with configuration management to get started and migrate to Kubernetes when it was reasonable to do so. People need to start with a goal and then ask what tool meets the objectives of their goal. In a lot of cases people complain about the tool they are using because they start with Kubernetes and then try to figure out how they can use it on the job.
Your point is do something simple because k8s is hard? 1) even small scale dev teams an business still need non simple.software processes. 2) learning Kubernetes is easier now than learning the underlying cloud. It's really about all the other things k8s provides. Maybe you haven't seen it used in enough contexts yet to appreciate those other benefits?
Call me biased [1] but K8s will take over the world! Yes you get containers and micro-services and all that good stuff, but now with Anthos [2] its also the best way to achieve multi-cloud and hybrid architectures. What's not to like!
Is there any benefit of Anthos over deploying straight to GKE if you're already bought into GCP? We've had this debate several times recently and can't come up with a good answer.
If you are bought in to GCP and plan to stay there, then maybe not much. OTOH, Anthos would allow you to do easier migrations from on-prem, support hybrid workloads, or consolidate multi-cloud clusters including those running on say, AWS [1] if you like.
Digital ocean has a managed kubernetes service that does not cost anything except the resources you use. The master node and management is free, you only pay the node pools and stuff like block storages (their version of EBS) or load balancers.
At this point, most big cloud providers cost almost the same but in terms of maturity, google’s offering is still ahead. I have not tried out digital ocean’s hosted solution, but the might be the cheapest.
Do you guys think k8s is doing a job which previously the jvm did in enterprise? i.e. if everything is on the jvm, building distributed systems doesn't require a network of containers.
Can k8s success be explained partly due to the need for a more polyglot stack?
680 comments
[ 3.3 ms ] story [ 354 ms ] threadThe system becomes so complex that most people screw up simple things like redundancy, perimeter security and zero downtime updates.
I've seen all of the above from very bright and capable people.
-------
Kubernetes - kubernetes.io
Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. It was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation.
Original author(s): Google
Kubernetes is very complex and took a long time to learn properly. And there have been fires among the way. I plan to write extensively on my blog about it.
But at the end of the day: having my entire application stack as YAML files, fully reproducible [1] is invaluable. Even cron jobs.
Note: I don't use micro services, service meshes, or any fancy stuff. Just a plain ol' Django monolith.
Maybe there's room for a simpler IAC solution out there. Swarm looked promising then fizzled. But right now the leader is k8s[2] and for that alone it's worth it.
[1] Combined with Terraform
[2] There are other proprietary solutions. But k8s is vendor agnostic. I can and have repointed my entire infrastructure with minimal fuss.
Open cron.yaml and see. With schedule. Self documented.
Amazing. Every time. Even when some as my k8s battle wounds are still healing (or permanently scarred). See other replies for more info.
Note: my experience was all with cloud-provided Kubernetes, never running my own. So it was already an order of magnitude easier. Can't even imagine rolling my own. [2]
[1] My personal favorite. Truly egregious, despite how amazing k8s is. https://github.com/kubernetes/kubernetes/issues/63371#issuec...
[2] https://github.com/kelseyhightower/kubernetes-the-hard-way
We used K8S on a large project and I felt like it really, really wasn't necessary.
So terraform creates the cluster, DNS and VPC. Then k8s runs pretty much everything.
Mainly just kustomize piped into kube apply.
But, but, but. Having to a create a one-off database migration script imperatively.
In retrospect though, maybe it's exactly what I needed. Great suggestion.
Cause those are the parts that I miss probably the most when dealing with non-k8s deployment, and I haven't had the occasion to use Nomad.
What Nomad doesn’t do is setup a cloud provider load balancer for you.
For persistent storage, Nomad uses CSI which is the same technology K8s does: https://learn.hashicorp.com/nomad/stateful-workloads/csi-vol...
Logging should be very similar to K8S. Both Nomad and K8S log to a file and a logging agent tails and ships the logs.
Disclosure, I am a HashiCorp employee.
Kinda feels bad that I don't have anything to use it on right now.
Thinking about completing my Hashicorp Bingo card.
Nomad, or rather, a Nomad/Consul/Vault stack doesn't have these things included. You need to go and pick a consul-aware loadbalancer like traefik, figure out a CSI volume provider or a consul-aware database clustering like postgres with patroni, think about logging sidecars or logging instances on container hosts. Lots of fiddly, fiddly things to figure out from an operative perspective until you have a platform your development can just use. Certainly less of an out-of-the-box experience than K8.
However, I would like to mention that K8 can be an evil half-truth. "Just self-hosting a K8 cluster" basically means doing all of the shit above, except its "just self-hosting k8". Nomad allows you to delay certain choices and implementations, or glue together existing infrastructure.
K8 requires you do redo everything, pretty much.
I have actually done a "lift and shift" where we moved code that had no support or directly antagonistic one to k8s because various problems reached situation where CEO said "replace the old vendor completely" - we ended up using k8s to wrestle with the amount of code to redeploy.
- Yes, just added CSI plugin support. Previously had ephemeral_disk and host_volume configuration options, as well as the ability to use docker storage plugins (portworx)
- I haven’t personally played with it, but apparently nomad does export some metrics, and they’re working on making it better
- no RBAC
- no quotas
- no preemption
- no namespacing
This means: everyone is root on the cluster, including any CI/CD system that wants to test/update code. And there's no way to contain runaway processes with quotas/preemption.
When you get to that blog post please consider going in depth on this. Would love to see actual battletested information vs. the usual handwavy "it works everywhere".
Even ingress is trivial if you use a cloud balancer per ingress. But I wanted to save money so use a single cloud balancer for multiple ingresses. So you need something like ingress-nginx, which has a few vendor-specific subtleties.
Edit: not sure why the down votes, I was just trying to point out what seems like a big distinction that the article is trying to make.
How about "infrastructure-as-some-sort-of-text-file-versioned-in-my-repository". It's a mouthful, but maybe it'll catch on.
Yes, higher-level tools like Kustomize or Jsonnet or whatever else you use for templating the files are Turing-complete - but that's at the level of you on your machine generating input to Kubernetes, not at the level of Kubernetes itself. That's a valuable distinction - it means you can't have a Kubernetes manifest get halfway through and fail the way that you can have an Ansible playbook get halfway through and fail; there's no "halfway." If something fails halfway through your Jsonnet, it fails in template expansion without actually doing anything to your infrastructure.
(You can, of course, have it run out of resources or hit quota issues partway through deploying some manifest, but there's no ordering constraint - it won't refuse to run the "rest" of the "steps" because an "earlier step" failed, there's no such thing. You can address the issue, and Kubernetes will resume trying to shape reality to match your manifest just as if some hardware failed at runtime and you were recovering, or whatever.)
The difference between code and data is pretty big.
One implies an expectation that the user is going to write some kind of algorithm whereas the other is basically a config file.
The way I learned it in Bret Fisher's Udemy course, Swarm is very much relevant, and will be supported indefinitely. It seems to be a much simpler version of Kubernetes. It has both composition in YAML files (i.e. all your containers together) and the distribution over nodes. What else do you need before you hit corporation-scale requirements?
Cronjobs, configmaps, and dynamically allocated persistent volumes have been big ones for our small corporation. Access control also, but I'm less aware of the details here, other than that our ops is happier to hand out credentials with limited access, which was somehow much more difficult with swarm
Swarm has frankly also been buggy. "Dead" but still running containers - sometimes visible to swarm, sometimes only the local Docker daemon - happen every 1-2 months, and it takes forever to figure out what's going on each time.
1. Swarm is dead in the water. No big releases/development afaik recently
2. Swarm for me has been a disaster because after a couple of days some of my nodes slowly start failing (although they’re perfectly normal) and I have to manually remove each node from the swarm, join them, and start everything up again. I think this might be because of some WireGuard incompatibility, but the strange thing is that it works for a week sometimes and other times just a few hours
3. Lack of GPU support
I've had clusters running for years without issue. I've even used it for packaging B2B software, where customers use it both in cloud and on-prem - no issues whatsoever.
I've looked at k8s a few times, but it's vastly more complex than Swarm (which is basically Docker Compose with cluster support), and would add nothing for my use case.
I'm sure a lot of people need the functionality that k8s brings, but I'm also sure that many would be better suited to Swarm.
If K8s supported compose scripts out of the box (not Kompose) that'd basically make Swarm unnecessary (at least for me)
Switched to k8s in late 2017 and it’s been much more solid. And that’s where the world has moved, so I’m not sure why you’d choose swarm anymore.
I see ruby kind of uses YAML often but are people comfortable editing YAML files? I always have to look up how to do arrays and such when I edit them once in a while.
Anything new needs a certain amount of sustained practice before you get the hang of it. I think I had to learn regex like four times before it stuck. I haven’t hit that point with TOML yet so I avoid it.
I’d suggest using the deeper indentation style where hyphens for arrays are also indented two spaces under the parent element. Like anything use a linter that enforces unambiguous indentation.
I prefer YAML for human-writeable config because JSON is just more typing and more finicky. The auto-typing of numbers and booleans in YAML is a pretty damn sharp edge though and I wish they’d solved that some other way.
I’ve deployed swarm in a home lab and found it really simple to work with, and enjoyable to use. I haven’t tried k8, but I often see view points like yours stating that k8 is vastly superior.
Effectively, "every infrastructure as code project will reimplement Kubernetes in Bash"
If we only got to keep two tools it would be kubernetes and terraform.
This isn’t a competition, they are tools. Ansible is widely used and will continue to be so for a long long time. Its foundations - ssh, python and yaml are also in for the long run to manage infrastructure...
Yaml will still be used in 100 years, k8s is yaml based...
Which is step 2 of my Enterprise adoption strategy. Step 1 is starting with validation, step 3 and beyond is where the real fun starts!
There are many experiments into alternatives happening right now, so I do believe yaml's days are numbered. I'm actively replacing it where ever I encounter it with a far superior alternative. Cue is far more than a configuration language however, worth the time to learn and adopt at this point.
Once everything is "infrastructure as code", the app team becomes less dependent on other teams in the org.
People like to own their own destiny. Of course, that also removes a lot of potential scapegoats, so you now mostly own all outages, tech debt, etc.
I worked in networking for the longest time. When I started there network guys and server guys (at least where I was). They were different people who did different things who kinda worked together.
Then there were storage area networks and similar, networks really FOR the server and storage guys.... that kind of extended the server world over some of the network.
Then comes VMware and such things and now there was a network in a box somewhere that was entirely the server guy's deal (well except when we had to help them... always).
Then we also had load balances who in their own way were a sort of code for networks ... depending on how you looked at it (open ticket #11111 of 'please stop hard coding ip addresses').
You also had a lot of software defined networking type things and so forth brewing up in dozens of different ways.
Granted these descriptions are not exact, there were ebs and flows and some tech that sort of did this (or tired) all along. It all starts to evolve slowly into one entity.
Kubernetes is fine, but setting it up kind of feels like I'm trying to earn a PhD thesis. Swarm is dog-simple to get working and I've really had no issues in the three years that I've been running it.
The configs aren't as elaborate or as modular as Kubernetes, and that's a blessing as well as a curse; it's easy to set up and administer, but you have less control. Still, for small-to-mid-sized systems, I would still recommend Swarm.
Are you following "k8s the hard way"? I've never had this problem; either:
`gcloud container clusters create`
Or
`install docker-for-mac`
And you have a k8s cluster up and running. Maybe it's more work on AWS?
The kind of people who has to both set the cluster up and keep it up and also has to develop the application and deploy it and keep it up etc is not the target audience.
K8s shines when the roles of managing the cluster and running workloads on it are separated. It defines a good contract between infrastructure and workload. It lets different people focus on different aspects.
Yes it still has rough edges, things that are either not there yet, or vestigial complexity of wrong turns that happened through it's history. But if you look at it through the lense of this corporate scenario it starts making more sense than when you just think of what a full-stack dev in a two person startup would rather use and fully own/understand.
It's hard to make tests maintainable. Doubly so if you aren't already versed in techniques to make code maintainable.
I wonder sometimes if we aren't repeating the same experiment with ops right now.
But I'm not sure one can find something of "the right power" that has the same support from cloud providers, the open source community, the critical mass, etc. [1]
Eventually, a standard "simplified" abstraction over k8s will emerge. Many already exist, but they're all over the place. And some are vendor specific (Google Cloud Run is basically just running k8s for you). Then if you need the power, you can eject. Something like Create React App, but by Kubernetes. Create Kubernetes App.
[1] Though Nomad looks promising.
I come from the opposite approach. I have 4 servers two digital ocean $5 and two vulr $2.50 instances. One holds the db. One server as the frontend/code. One server to do heavy work and another to server a heavy site and holds backups. For $15 I'm hosting hundreds of sites, running so many background processes. I couldn't imagine hitting that point where k8s would make sense just for myself unless for fun.
Besides, it's additional hassle and a chance for things to go wrong, the way I have it set up now is that production gets a new deployment whenever something gets pushed to master and I don't have to do anything else.
A text file with some setup notes is enough for simple needs, or something like Ansible if its more complex. A lot of web apps aren't much more than some files, a database, and maybe a config file or three (all of which should be versioned and backed up).
I don't hate it but if you need to login to a server regularly because you need to do an apt upgrade, you should have enabled automatic security updates and not login every few days.
If your server runs full because of some logfiles or stuff, you should fix the underlying issue and not needing to login to a server.
You should trust your machines, independently if it is only one machine, 2, 3 or 100. You wanna be able to go on holiday and know your systems are stable, secure and doing their job.
And logging in also implies a snow flake. Doesn't matter as long as that machine runs and as long as you have not that many changes but k8s actually makes it very simple to finally have an abstraction layer for infrastructure.
The service just runs a script that uses netcat to listen on a special port that I also configured GitHub to send webhooks to, and processes the hook/deploys if appropriate.
Then when it's done, systemd restarts the script (it is set to always restart) and we're locked and loaded again. It's about 15 lines of shell script in total.
You could run init container with Kaniko that pushes image to repo and then main container that pulls that back but for that you need to do kubectl rollout restart deploy <name>
If you are looking for pure CI/CD gitlab has awesome support or you could do Tekton or Argo. They can run on the same cluster.
Flux - https://github.com/fluxcd/flux
ArgoCD - https://argoproj.github.io/argo-cd/
If you do, the recipe is to reduce the number of components, get the most reliable components you can find, and make the single points of failure redundant.
Saying you can use Kubernetes to turn whatever stupid crap people tend to deploy with it highly available, is like saying you can make an airliner reliable by installing some sort of super fancy electronic box inside. You don't get more reliability by adding more components.
Are you ok with your application going down for each upgrade? With Kubernetes, it's very simple to configure a deployment so that downtime doesn't happen.
On the other hand, atomic upgrades by stopping the old service and then starting the new service on a Linux command line (/Gitlab runner) can be done in 10 seconds (depending on the service of course – dynamic languages/frameworks sometimes are disadvantaged here). I doubt many customers will notice 10 second downtimes.
K8s only makes sense at near Google-scale, where you have a team dedicated to managing that infrastructure layer (on top of the folks managing the rest of the infrastructure). For almost everyone else, it's damaging to use it and introduces so much risk. Either your team learns k8s inside out (so a big chunk of their work becomes about managing k8s) or they cross their fingers and trust the black box (and when it fails, panic).
The most effective teams I've worked on have been the ones where the software engineers understand each layer of the stack (even if they have specialist areas of focus). That's not possible at FAANG scale, which is why the k8s abstraction makes sense there.
This is a bit funny, considering Airbus jets use triple-redundancy and a voting system for some of their critical components. [1]
[1] https://criticaluncertainties.com/2009/06/20/airbus-voting-l...
Deploying as distribution package tends to not work well when you want to deploy it more than once on a specific server (which quickly leads us to classic end-result of that approach, which is VM per deployment minimum - been there, done that, still have scars).
Management of cron jobs was a shitshow, is a shitshow, and probably will be a shitshow except for those that run their crons using non-cron tools (which includes k8s).
How much of K8s is just an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang.
I totally agree. I would dearly like something simpler than Kubernetes. But there isnt a managed Nomad service, and apparently nothing in between Dokku and managed Kubernetes either.
The test is going from zero to production traffic in a new cloud region.
That’d be a great first step if the purpose is to learn Kubernetes. If, however, you want to set up a cluster for real use then you will need much more than bare bones Kubernetes (something that solves networking, monitoring, logging, security, backups and more) so consider using a distribution or a managed cloud service instead.
Maintaining a cluster set up like that is a ton of work. And if you don’t perform an upgrade perfectly, you’ll have downtime. Tools like kops help a lot but you’ll still spend far more time than the $70/month it costs for a managed cluster.
We have plenty of yaml 'code' which is simple and does exactly what it needs to do.
For all other usecases, there are plenty of alternatives, including libs for your preferred language.
Most people use the yaml way because its easy and does exactly what it needs to do.
Everyone else has plenty of well supported and well working alternatives.
We can build higher level abstractions easily having a schema to target and we can build them in whatever we want. That's a big boon for me :)
But, yaml is now everywhere in the ops space. Config management systems use it, metrics systems use it, its the defacto configuration format right now and that is unfortunate cause its bad.
1. It's simple to get started with, but complex enough to tweak to your needs in respect to simplicity of deployment, scaling and resource definition.
2. It's appealingly cloud-agnostic just at the time where multiple cloud providers are all becoming viable and competitive.
I think it's more #2 and #1; as always, timing is everything.
You always have those concerns, it's just implemented differently. Customer hurling abuse at you over the phone (or worse - in person) is a form of healthchecks and monitoring, if worse than often common "have someone log in to the server every day and check if it's alive".
So is frantically logging into server to manually truncate log files that filled your one and only disk volume and caused the above abuse.
So is "we're losing customers because of how slow it is" yet not having a single idea why it is slow, because it runs fast when dev checks on their laptop.
All of the above are based on actual real world events, sometimes involving large corporations. In fact, the large corps seem to have most issues with manual work, because they can afford throwing cannon fodder ^W^W "experienced engineers" at the problems.
At some point it becomes a question of what is good use of your time. I disagree heavily with people claiming that running kubernetes is somehow orders of magnitude more complex than anything else (especially with k3s and using non-etcd backing stores). The complexity is necessary complexity, which you can tackle in various ways including YOLO.
Sometimes the YOLO approach however bites in the worst moment, and spending time on bespoke scripts, or figuring out configuration drift, are all costs that show up as you tackle said complexity.
Personally, the reason I went with kubernetes in the first production deployment I did with it, after being vocal anti-docker person at work, was because of... cost efficiency. Both in terms of my time (even though we had to spend significant amount of time migrating, as it was lift&shift of existing software), and in terms of compute costs - thanks to heavily loaded nodes our worst compute bill never reached above 20% of previous "condition normal". I don't think we ever really had more than 10 servers on purpose. Using k8s paid for itself.
I've yet to meet anyone who can easily explain how the CNI, services, ingresses and pod network spaces all work together.
Everything is so interlinked and complicated that you need to understand vast swathes of kubernetes before you can attach any sort of complexity to the networking side.
I contrast that to it's scheduling and resourcing components which are relatively easy to explain and obvious.
Even storage is starting to move to overcomplication with CSI.
I half jokingly think K8s adoption is driven by consultants and cloud providers hoping to ensure a lock-in with the mechanics of actually deploying workloads on K8s.
An ingress object creates an nginx/nginx.conf. That nginx server has an IP address which has a round robin IPVS rule. When it gets the request it proxy's to a service ip which then round robins to the 10.0.0.0/8 container IP.
Ingress -> service -> pod
It is all very confusing but once you look behind the curtain it's straight forward if you know Linux networking and web servers. The cloud providers remove the requirement of needing Linux knowledge.
Is there an easier + simpler alternative?
It's confusing because a lot of people being exposed to K8s don't necessarily know how Linux networking and web servers work. So there is a mix of terminology (services, ingress, ipvs, iptables, etc) and context that may not be understood if you didn't come from running/deploying Linux servers.
Looking at the docs, ingress-nginx configures an upstream using endpoints, which are essentially Pod IPs, with skips kubernetes service based round-robin networking altogether.
Assuming you use an ingress that does configure services instead, and assuming you're using a service proxy that uses ipvs (i.e. kube-proxy in default settings) then your explanation would have been correct.
For the most part, kubernetes networking is as hard as networking with loads of automation. Often, depth in both those skills are pretty exclusive, but if you're using the popular and/or supported CNI not doing things like changing in-flight, your average dev just needs to learn basic k8s debugging such as kubectl get endpoints to check whether his service selectors are setup correctly, and curl them to check whether the pods are actually listening on those ports.
Kubernetes should have been IPv6 only, with optional IPv4 ingress controllers.
I really hope someone takes the mantle of Leslie Lamport (creator of the language TLA - "the quixotic attempt to overcome engineers' antipathy towards mathematics") and replaces Kubernetes with some software with a first principles approach.
Badly! That'll be $500, thanks for your business.
On a serious note, the whole stack is keeping ok-ish coherence considering the number of very different parties putting a ton of work into it.
In a few years' time it'll be the source of many war stories nobody cares about.
Also, after OpenStack, the bar for "consulting-driven software" is far from reached :)
A single Service with Type=LoadBalancer and one Deployment may be all you need on Kubernetes if you just want all connections from the load balancer immediately forwarded directly to the service.
But if you have multiple different services/deployments that you want as accessible under different URLs on a single IP/domain, then you'll want to use Ingresses. Ingresses let you do things like map specific URL paths to different services. Then you have an IngressController which runs a webserver in your cluster and it automatically uses your Ingresses to figure out where connections for different paths should be forwarded to. An IngressController also lets you configure the webserver to do certain pre-processing on incoming connections, like applying HTTPS, before proxying to your service. (The IngressController itself will usually use a Type=LoadBalancer service so that a load balancer connects to it, and then all of the Ingresses will point to regular Services.)
K8S is extremely complicated for huge swarm of webdevs and java developers that really reqlly dont understand how the stuff they use/code really works.
K8S was supposed to decrease the need for real sysadmins but in my view it actually increased the demand because of all the obscure issues one can face in production if they dont really understand what they are doing with K8S and how it works under the hoods.
Which I find hilarious.
Likewise, Linux is also a confusing mess of different parts and nonsensical abstractions when you first approach it. It does take some time to understand how to use it, and in particular how to do effective troubleshooting when things aren't working the way you expect.
But I 100% agree--I think it's the new Linux. In 5-10 years, it'll be the "go to", if not sooner.
The second reason is also about standards, but using them more assertively. Docker had way more attention and activity until 2016 when Kubernetes published the Container Runtime Interface. By limiting the Docker features they would use, they leveled the playing field between Docker and other runtimes, making Docker much less exciting. Now, new isolation features are implemented down at the runc level and new management features tend to target Kubernetes because it works just as well with any CRI-compliant runtime. Developing for Docker feels like being locked in.
Isn't the most popular k8s case to deploy Docker images still though?
But deploy to k8s? There's no docker outside of few bits involving "how to get to the image", and the actual docker features that are used are also minimized. The result is that many warts of docker are completely bypassed and you don't have to deal with impact of legacy decisions, or try to wrangle system designed for easy use by developer at local machine into complex server deployment. And, IMHO, interfaces used by k8s for the advanced features are much, much better than interfaces used or exported by docker.
A lot of the Kubernetes "cool kids" just run containerd instead of Docker. Docker itself also runs containerd, so when you're using Kubernetes with Docker, Kubernetes has to basically instruct Docker to set up the containers the same way it would if it were just talking to containerd directly. From a technical perspective, you're adding moving parts for no benefit.
If you use containerd in your cluster, you can then use Docker to build and push your images (from your own or a build machine), but pull and run them on your Kubernetes clusters without Docker.
[1] https://en.wikipedia.org/wiki/Open_Container_Initiative
As far as I can tell: those are imperative. At least in some areas.
Kubernetes is declarative. You mention the end state and it just "figures it out". Mind you, with issues sometimes.
All abstractions leak. Note that k8s's adamance about declarative configuration can make you bend over backwards. Example: running a migration script post deploys. Or waiting for other services to start before starting your own. Etc.
I think in many ways, those compete with Terraform which is "declarative"-ish. There's very much a state file.
I would be somewhat surprised to find out Puppet and Chef weren't declarative either. Because setting up a system in an imperative fashion is ripe for trouble. You may as well use bash scripts at that point.
I've used Ansible for close to 10 years for hobby projects. And setting up my development environment. Give me a freshly installed Ubuntu laptop, and I can have my development environment 100% setup with a single command.
I can buy a new laptop and be back to 100% in a few minutes. Though the amount of time I spent learning how to get there far exceeds any time savings. Ever.
A few years ago I even bothered to have two EFI loaders: one for amd an one intel, in case I want to change architecture as well.
Instead, every project I work on has a shell.nix in the root (and if it's not a project I control, I have a shell.nix mapping elsewhere).
Check it out, run nix-shell. Profit.
Once you're really ready for the big leagues, run it with --pure.
Back to your second question, you can configure the system through `/etc/nixos/configuration.nix` it is enough to configure system as a service. Pretty much everything you could do through Chef/Puppet/Saltstack/Ansible/CFEngine etc.
home-manager is taking it a step further and do this kind of configuration per user. It is actually written in a way that can be added to NixOS (or nix-darwin for OS X users) to integrate with the main configuration so then when you're declaring users you can also provide a configuration for each of them.
So it all depends what you want to do, the main configuration.nix is good enough if your machine to run specific service, that's pretty much all you need, you don't care about each user configuration in that scenario, you just create users and start services using them.
If you have a workstation, home-manager while not essential can be used to take care of setting up your individual user settings, stuff like dot-files (although it goes beyond that). The benefit of using home-manager is that most of what you configure in it should be reusable on OS X as well.
If you care about local development, you can use Nix to declare what is needed, for example[1]. This is especially awesome if you have direnv + lorri installed
you can add these to home-manager configuration:
When you do that you magically will get your CDE (that includes all needed tools, in this case proper python version, you also enter equivalent of virtualenv with all dependencies installed and extra tools) by just entering the directory, if you don't have them installed all you have to do is just call `nix-shell`.I also can't wait when Flakes[2] get merged. This will standardize setup like this and enable other possibilities.
[1] https://github.com/takeda/example_python_project
[2] https://www.tweag.io/blog/2020-05-25-flakes/
Sure, the steps are things like "if X hasn't been done yet, do it." That means it's idempotent imperative code. It doesn't mean it's declarative.
CFEngine is slightly less imperative, but when I was doing heavy CFEngine work I had a printout on my cubicle wall of the "normal ordering" because it was extremely relevant that CFEngine ran each step in a specific order and looped over that order until it converged, and I cared about things like whether a files promise or packages promise executed first so I could depend on one in the other.
Kubernetes - largely because it insists you use containers - doesn't have any concept of "steps". You tell it what you want your deployment to look like and it makes it happen. You simply do not have the ability to say, install this package, then edit this config file, then start this service, then start these five clients. It does make it harder to lift an existing design onto Kubernetes, but it means the result is much more robust. (For some of these things, you can use Dockerfiles, which are in fact imperative steps - but once a build has happened you use the image as an artifact. For other things, you're expected to write your systems so that the order between steps doesn't matter, which is quite a big thing to ask, but it is the only manageable way to automate large-scale deployments. On the flip side, it's overkill for scale-of-one tasks like setting up your development environment on a new laptop.)
For example, I can declare a Pod that mounts a Secret. If the Secret does not exist, the Pod won't start -- but once I create the Secret the pod will start without requiring further manual intervention.
What Kubernetes really is, under the hood, is a bunch of controllers that are constantly comparing the desired state of the world with the actual state, and taking action if the actual state does not match.
The configuration model exposed to users is declarative. The eventual consistency model means you don't need to tell it what order things need to be done.
This is basically the benefit of "containerization" - it's not the containers themselves, it's the constraints they place on the problem space.
Kubernetes gives you limited tools for doing things to container images beyond running a single command - you can run initContainers and health checks, but the model is generally that you start a container from an image, run a command, and exit the container when the command exits. If you want the service to respawn, the whole container respawns. If you want to upgrade it, you delete the container and make a new one, you don't upgrade it in place.
If you want to, say, run a three-node database cluster, an Ansible playbook is likely to go to each machine, configure some apt sources, install a package, copy some auth keys around, create some firewall rules, start up the first database in initialization mode if it's a new deployment, connect the rest of the databases, etc. You can't take this approach in Kubernetes. Your software comes in via a Docker image, which is generated from an imperative Dockerfile (or whatever tool you like), but that happens ahead of time, outside of your running infrastructure. You can't (or shouldn't, at least) download and install software when the container starts up.
You also can't control the order when the containers start up - each DB process must be capable of syncing up with whichever DB instances happen to be running when it starts up. You can have a "controller" (https://kubernetes.io/docs/concepts/architecture/controller/) if you want loops, but a controller isn't really set up to be fully imperative, either. It gets to say, I want to go from here to point B, but it doesn't get much control of the steps to get there. And it has to be able to account for things like one database server disappearing at a random time. It can tell Kubernetes how point B looks different from point A, but that's it.
And since Kubernetes only runs containers, and containers abstract over machines (physical or virtual), it gets to insist that every time it runs some command, it runs in a fresh container. You don't have to have any logic for, how do I handle running the database if a previous version of the database was installed. It's not - you build a new fresh Docker image, and you run the database command in a container from that image. If the command exits, the container goes away, and Kubernetes starts a new container with another attempt to run that command. It can do that because it's not managing systems you provide it, it's managing containers that it creates. If you need to incrementally migrate your data from DB version 1 to 1.1, you can start up some fresh containers running version 1.1, wait for the data to sync, and then shut down version 1 - no in-place upgrades like you'd be tempted to do on full machines.
And yeah, for databases, you need to keep track of persistent storage, but that's explicitly specified in your config. You don't have any problems with configuration drift (a serious problem with large-scale Ansible/CFEngine/etc.) because there's nothing that's unexpectedly stateful. Everything is fully determined by what's specified in the latest version of your manifest because there's no other input to the system beyond that.
Again, the tradeoff is this makes quite a few constraints on your system design. They're all constraints that are long-term better if you're running at a large enough scale, but it's not clear the benefits are worth...
The most simplistic task - execute some code in response to even in a bucket - makes kubernetes with all its sophisticated convergence capabilities completely useless. And even if somebody figures this out and puts the opensource project on github to do this on kubernetes - it just going to break at slightest load.
Not to mention all the work to run kubernetes at any acceptable level of security, or keep the cost down, do all patching, scaling, logging, upgrades... Oh, the configuration management itself for kubernetes? Ah sorry, I forgot, there are 17 great open-source projects exists :)
That's because you're not thinking web^Wcloud scale. To execute some code in response to event you need:
- several workers that will poll the source bucket for changes (of course you could've used existing notification mechanism like aws eventBridge, but that will couple you k8s to vendor-specific infra, so it kinda deminishes the point of k8s)
- distributed message bus with persistanse layer. Kafka will work nicely because they say so on Medium, even though it's not designed for this use case
- a bunch of stateless consumers for the events
- don't forget that you'll need to write processing code with concurrency in mind because you're actually executing it in truly destributed system at this point and you've made a poor choice for your messaging system
So once again, why developers need kubernetes for? If the most simple problem becomes a habitholy mess :)
Ansible uses YAML, but when I used it few times it felt that you still use it in imperative way.
The saltstack (which also uses YAML) was the closest from that group (never used CFengine, but the author wrote research paper and shown that declarative is the way to go, so I would imagine he would also implement it that way).
If you truly want a declarative approach designated from a ground up, then you should try Nix or NixOS.
But it has full power of ruby at your disposal (both at load/compile time and run time). So it usually turns imperative quickly.
What k8s brings to the table is a level of standardization. It's the difference between bringing some level of robotics to manual loading and unloading of classic cargo ships, vs. the fully automated containerized ports.
With k8s, you get structure where you can wrap individual program's idiosyncracies into a container that exposes standard interface. This standard interface allows you to then easily drop it into server, with various topologies, resources, networking etc. handled through common interfaces.
I said that for a long time before, but recently I got to understand just how much work k8s can "take away" when I foolishly said "eh, it's only one server, I will run this the classic way. Then I spent 5 days on something that could be handled within an hour on k8s, because k8s virtualized away HTTP reverse proxies, persistent storage, and load balancing in general.
Now I'm thinking of deploying k8s at home, not to learn, but because I know it's easier for me to deploy nextcloud, or an ebook catalog, or whatever, using k8s than by setting up more classical configuration management system and deal with inevitable drift over time.
can't you do that just with containers?
And then one day I decided to set up kubernetes as a learning experiment. There is definitely some learning curve about making sure I understood what deployment, or replicaset or service or pod or ingress was, and how to properly set them up for my environment. But now that I have that, adding a new app to my cluster, and making it accessible is super low effort. i have previous yaml files to base my new app's config on.
It feels like the only reason not to use it would be learning curve and initial setup... but after I overcame the curve, it's been a much better experience than trying to orchestrate containers by hand.
Perhaps this is all doable without kubernetes, and there is a learning curve, but it's far from the complicated nightmare beast everyone makes it out to be (from the user side, maybe from the implementation details side)
It would mean I removed ~20% of the things that were annoying me and left 80% still to solve, while kubernetes goes 80% for me with the remaining 20% being mostly "assembly these blocks".
Plus, a huge plus of k8s for me was that it abstracted away horrible interfaces and behaviours of docker daemon and docker cli.
Are we talking about k8 base on your own server rack at your house?
I think the build, deploy, start and run-time split is an important aspect that gets overlooked quite a bit, and is critical to evaluating tools at this point. That is why we aren't still doing everything with Chef or Puppet. Whether we continue doing it with Kubernetes or Pulumi or something else matters a bit less.
Repeatability is not the goal, as others in this thread have implied. The goal is trusting that the button will work when you push it. That if it doesn't work, you can fix it, or find someone who can. Doing that without repeatability is pretty damned hard, certainly, but there are ways to chase repeatability without ever arriving at the actual goal.
But basic versions of these things are provided by Kubernetes natively and can be declared in a way that is divorced from configuring the underlying software. So you just learn how to configure these broader concepts as services or ingresses or network policies, etc, and don't worry about the underlying implementations. It's pretty nice actually.
Kubernetes is one way to deploy containers. Configuration systems like Ansible/Salt/Puppet/Chef/etc are another way to deploy containers.
Kubernetes also makes it possible to dynamically scale your workload. But so does Auto Scaling Groups (AWS terminology) and GCP/Azure equivalents.
The reality is that 99% of users don't actually need Kubernetes. It introduces a huge amount of complexity, overhead, and instability for no benefit in most cases. The tech industry is highly trend driven. There is a lot of cargo culting. People want to build their resumes. They like novelty. Many people incorrectly believe that Kubernetes is the way to deploy containers.
And they (and their employers) suffer for it. Most users would be far better off using boring statically deployed containers from a configuration management system. Auto-scaled when required. This can also be entirely infrastructure-as-code compliant.
Containers are the real magic. But somehow people confused Kubernetes as a replacement for Docker containers, when it was actually a replacement for Docker's orchestration framework: Docker Swarm.
In fact, Kubernetes is a very dangerous chainsaw that most people are using to whittle in their laps.
So many people miss this. k8s is a very complex system and the talent it takes to manage it well, rare.
Extremely rare.
1st Question : Define k8s network, in detail, with all of the services and a set of services
IF you make it out of that one and the follow ups we can move on to the rest
So we end up with a plethora of full stack developers who can barely keep up with their current development stacks willfully deploying their software on systems that they're just barely competent with.
I know this because I almost deployed a side project with Kubernetes because it was expected of me despite the fact that being mediocre at it was the best that I could hope to become and that's an easy way to chop off a leg or three.
systemd-nspawn / machined makes the other systems look like very complicated solutions in search of a problem
Name may not be pretty but it's an official feature of systemd which is used to debug the systemd development and it is far easier to take backups incrementally because the container files are just plain files in /var/lib/machines/ and apparently you already have it if systemd is on your system. (May need an additional package to be installed from OS package repo.)
I run nspawn instances as development environments for developers and I can also run docker inside it.
I've never used it but might find myself using it in the future.. can you elaborate on these a bit? I'm curious what the pitfalls might be
Kubernetes isn't a silver bullet of course, there will be applications where running it in containers adds unnecessary complexity, and those are best run in a VM managed by a CM tool. I'd argue using k8s is safe default for deploying new applications going forward.
Unlike Ansible (and I suspect the others) where it's really only more of a 'run once' type of thing... And sometimes if you try running it a second time it won't even succeed.
Our management of cluster is just a simple "add more CPU or memory to this nodepool", sometimes change a nodepool name for deployment for certain service. All done simple cloud management UI. For those who call microservices fancy stuff. No, we are a startup with fast delivery, deploy cycle. We have tons of subproject , integrations, and our main languages are nodejs, golang and python. Some of these are not good at multi-thread so no way to run it as a monolith. The other one is used only when it's needed for high performance. So All together Microservices + Kubernetes + Helm + good CI + proper pubsub gives our backend extremely simple fast cycle of development, delivery, and what's important flexibility in terms of language/framework/version.
What is also good is the installation of services. With helm I can install high availability redis setup for free in 5 minutes. The same level of setup will cost you several thousand dollars for devops work and further maintenance and update. With k8s it's simple helm install stable/redis-ha
So yeah, I can totally understand some simple projects don't need k8s. I can understand you can build something is Scala and Java slowly but with high quality as a monolith. You don't need k8s for 3 services. I can understand some old DevOps don't want to learn new things and they complain about a tool that reduces the need of these guys. Otherwise, you really need k8s.
Because soon from one program on a dev server, there is a need to run databases, log gathering, multiply the previous to do parallel testing in clean environment, etc. etc.
Just running supporting tools for a small project where there was insistence on self-hosting open source tools instead of throwing money at slack and the like? K3s would have saved me weeks of work :|
Again the cross cloud portability is a non starter, unless you're really at scale.
And, we only have to learn one complex system and avoid learning each cloud, one of which decided product names which have little relation to what they do was a good idear
What k8s really scales is the developer/operator power. Yes, it is complex, but pretty much all of it is necessary complexity. At small enough scale with enough time, you can dig a hole with your fingers - but a proper tool will do wonders to how much digging you can do. And a lot of that complexity is present even when you do everything the "old" way, it's just invisible toil.
And a lot of the calculus changes when 'managed services' stop being cost effective or aren't an option at all, or you just want to be able to migrate elsewhere (that can be at low scale too, because of being price conscious).
Sure, managed service costs are certainly a thing, but to my point that only really start to become an issue at significant scale, assuming you're well configured.
The cost metrics that make "it's cheaper to use managed service than pay the cost of extra engineer to specialize in infrastructure" aren't universal. In fact, I usually have to work from the opposite direction, where hiring a senior Ops specialist who can wrangle everything from shelving the physical hw to network booting k8s cluster on-premises can be cheaper that Heroku/AWS/etc.
Then the question becomes "Is K8 more of a shovel or an excavator". I think its fair to say that its more the latter.
That's not actually simple at all, and you would need to build a lot of the other stuff that Kubernetes gives you for free.
Kubernetes gives you an industry standard platform with first-class cloud vendor support. If you roll your own solution with ECS, what you are really doing is making a crappy in-house Kubernetes.
Previously we had to deploy a lot of monitoring on each VM to ensure that containers are running, we get alerted when one of the application crashed and didn't restart because Docker daemon didn't handle it etc etc.
Now, we only run stateless services, in a private VPC subnet, Load balancing is delegated to ALB, we don't need service discovery, meshes etc. Configuration is declarative, but written in much friendlier HCL (I'm ok with YAML, but to a degree). ECS just works for us.
Just like K8S might work for a bigger team, but I wouldn't adopt it at our shop, simply because of all of the complexity and huge surface area.
I still believe 90% of users would be better served by Nomad. And if someone says "developers want to use the most widely used tech", then I'm here to call bullshit, because the concepts between workload schedulers and orchestrators like k8s and nomad are easy enough to carry over from one side to the other. Learning either even if you end up using the other one is not a waste of time. Heck, I started out using CoreOS with fleetctl and even that taught me many valuable lessons.
Where is the momentum?
Hosted GKE costs the same per month as an hour of DevOps time, what's wrong with paid management for k8s?
The issue that I have with managed k8s is that these products will decrease the pressure to improve k8s documentation, tooling and setup itself. And then there's folks (like me) who want or need to run something like k8s on bare metal hardware outside of a cloud where the cloud-managed solution isn't available.
Well sure, but if the story just ended with "everyone use the least exciting tool", then there'd be few articles for tech journals to write.
But Kubernetes promises so much, and deep down everyone subtly thinks "what if I have to scale my project?" Why settle for good enough when you could settle for "awesome"? It's just human nature to choose the most exciting thing. And given that I do agree that there's some manufactured hype around Kubernetes, it isn't surprising to me why few are talking about Nomad.
As a relatively noob sysadmin, I liked it a lot. Easy to deploy and easy to maintain. We've got a lot of mixed rented hardware + cloud VPS, and having one layer to unify them all seemed great.
Unfortunately I had a hard convincing the org to give it a serious shot. At the crux of it, it wasn't clear what 'production ready' Nomad should look like. It seemed like Nomad is useless without Consul, and you really should use Vault to do the PKI for all of it.
It's a bit frustrating how so many of the HashiCorp products are 'in for penny, in for a pound' type deals. I know there's _technically_ ways for you use Nomad without Consul, but it didn't seem like the happy path, and the community support was non-existent.
Please tell me why I'm wrong lol, I really wanted to love Nomad. We are running a mix of everything and its a nightmare
Consul by itself is the game-changer. Even in k8s it's a game-changer. It solves so many questions in an elegant way.
"How do I find and reach the things running in (orchestrator) with (unknown ip/random port) from (legacy)?" being the most important. You run 5 servers, and a relatively lightweight client on everything (which isn't even outright required, but it sure is useful!), and you get a _lot_ with that.
Consul provides multiple interfaces and ingress points to find everything. It also is super easy to operate, and has a pretty big community.
If you absolutely cannot have Consul, Nomad is still a really good batch job engine, and makes a very great "distributed cron," which is more extensible, scalable, and easy to use than something like Jenkins for the same task.
My team is pretty small (was 4 people, now 6) and we manage one of the worlds largest nomad and consul clusters (there are some truly staggeringly large users of Vault so I won't make that claim). Even when shit really hits the fan, everything is designed in a way that stuff mostly works; and there's enough operator friendly entry points that we can always figure out the problem.
I'm assuming your team is using vault for PKI, but is there a similarly happy path for issuing certs without Vault.
I started off just using `openssl` but it all felt very janky, and I didn't really have any idea how CRLs should be setup
For now, we have CRLs disabled on all short-lived backends, enabled on long-lived backends and we're actually looking at disabling storing short-lived certs in the storage system at all, and just cranking the TTL down to really truly short. We've tested it as low as 30m, but a more real-world max-ttl is 1 week, with individual apps setting it as low as they can handle. For reference we run more than 10 PKI backends, and adding one (or a bunch) more is just a little terraform snippet for us.
The way it works via hashicorp template land, is that you just plop
into your Nomad template stanza, or use consul-template directly as a binary, or use vault agent with it's template capability. You can get the CA chain if required the same way, just hitting a different PKI endpoint.Also, as of Vault 1.4, Vault's internal raft backend is now production ready, making it a snap to run.
Try running through a few of the Vault quick-start guides, and replicating them in Terraform as much as possible. There's a few things TF does not handle gracefully last I checked (initial bootstrap), but you can get around that by using a null_resource or just handling that outside Terraform.
Edit: just noticed an actual Nomad user replied as well, and I like their answer better. Consider mine an addendum. :)
Batch workloads rarely require Consul, but for deploying your standard network services on Nomad: Consul is basically required. You could likely use any number of service mesh systems instead (either as sidecars, Docker network plugins, or soon CNI), but you'll be doing a lot of research and development on your own I'm afraid.
The Nomad team is by no means opposed to becoming more flexible in the future (and indeed better CNI support is landing soon as a first step), but we wanted to focus on getting one platform right and a pleasure to use before trying to genericize and modularize it.
My org looked at Nomad at a time when there was a lot of pressure from above to deliver something as soon as possible. Two weeks just weren't enough to full lay of the land ¯\_(ツ)_/¯
Funny thing is even if I could plug in my own service discovery into Nomad, I would probably chuck it away and replace it with Consul after a few weeks anyway haha
I think a Distributed OS is the only sane solution. Build the features we need into the kernel and stop futzing around with 15 abstractions to just run an isolated process on multiple hosts.
Linux (and the BSDs) are remarkably stable, festureful, and resilient operating systems. I would hate to give up such a strong foundation. Nomad can crash without affecting your running services. Nomad can be upgraded or reconfigured without affecting your running services. Nomad can be observed, developed, and debugged as a unit often without having to consider the abstractions that sit above or below it. The right number of abstractions is a beautiful thing. Just no more and no less. :)
The kubernetes ecosystem is really amazing and full of invaluable resources. It's vast, complex, but well-thought. Getting to know all ins and outs of the project is time consuming. So much things to learn and so little time to practice...
Lets use Istio's "istioctl manifest apply" to deploy a service mesh to my cluster that allows me to pull auth logic / service discovery / load balancing / tracing out of my code and let Istio handle this.
Lets configure my app's infrastructure (Kafka (Strimzi), Yugabyte/Cockroach, etc) as yaml files. Being able to describe my kafka config (foo topic has 3 partitions, etc) in yaml is priceless.
Lets move my entire application and its infrastructure to another cloud provider by running a single bazel command.
k8s is the common denominator that makes all this possible.
can't... terraform make all of that possible?
You can write your own providers, you can use the provisioned support, but TF doesn't like that and it shows.
operations details are hidden from developers and development details (the details of the workload) are hidden from the operations engineers.
The environments advertise themselves via that same modified ingress's default backend. We stick a tiny bit of deploy yaml in our projects, the deployments kube tagging gives us all the details we need to provide diffs, last build time, links to git repos, web sites etc for the particular environment. The yaml demonstrates conclusively how an app could or should be run, regardless of os or software choice, so when we hand it to ops folks there is a basis for them to run from.
1. I work for GCP 2. https://cloud.google.com/anthos/gke
1. https://cloud.google.com/blog/topics/anthos/multi-cloud-feat...
The live VMWare migration to Anthos is also quite impressive too
What is the cheapest way to setup a production kubernetes on a cloud provider?
Can k8s success be explained partly due to the need for a more polyglot stack?
Is it as easy as two simple commands?
I've never really thought it was that useful for (for example) nodejs, where you can just npm install your whole environment and deps, and off you go.
- Automatic scaling of pods and cluster VMs to meet demand.
- Flexible automated process monitoring via liveness/readiness probes.
- Simple log streaming across horizontally scaled pods running the same app/serving the same function using stern.
- Easy and low cost metrics aggregation with Prometheus and Grafana.
- Injecting secrets into services.
I'd imagine there are other things can offer the same, but I find it convenient to have them all in the same place.