not in definitive, it was more a "see you later". The company didn't have enough resources (people) to maintain the cluster and we decided to use ECS while we were still maturing as a team.
For my new projects nowadays, I'm pushing mainly serverless approaches using AWS Lambdas (behind API Gateways for stuff that needs to be reachable by HTTP).
I think this shifts the complexity from managing Kubernetes and its accompanying ten-thousand-yaml-files to infrastructure-as-code and the complexities of dealing with AWS. And I happen to prefer the latter, even though it's not infinitely better by any margin.
For the few things that needs to be always-online, or 3rd party self-hosted apps, I'm still on Kubernetes, or pure Docker if possible.
Not the OP, but I have had success with CDK. The main advantages for me have been discoverability with respect to resource properties, along with proper, higher-level abstractions pertaining to AWS infrastructure. https://aws.amazon.com/blogs/devops/leverage-l2-constructs-t...
Vendor lock-in is always omni-present, even if you are using Kubernetes. The only difference is that the vendor lock-in you get when you run Kubernetes is mainly with the infrastructure layer hosting the code, instead of the code entrypoints.
Not to mention any managed services you want to use, which also will lock you in. So I don't do anything extreme to avoid vendor lock-in, other than making my code general enough to only have a small surface area for the lambda entry point. As a practical example, all of my APIs hosted on Lambdas are ordinary ASP.Net apps that would work identically if hosted in Docker containers.
Pricing so far is one of the biggest benefits of doing serverless approaches. I'm down to paying a couple of dollars per month for something that I'd pay tenfold for if doing Kubernetes. Both the monetary sum of only paying for what you're using, and also not having to worry about cluster maintenance, scaling and management is a godsend.
We did. Our use case is spinning up containers on demand to user actions, giving them ephemeral, internet-routable hostnames, and shutting them down when all inbound connections have dropped. Because users are waiting to interact with these containers, we found the start times with Kubernetes too slow and its architecture to be a bad fit.
> we found the start times with Kubernetes too slow
Just curious if you could elaborate here? I work with k8s on docker, and we're also going to be spinning up ephemeral containers (and most of the other things you say) with jupyter notebooks. We're all in on k8s, but since you might be ahead of me, just wondering what hurdles you have faced?
Our big problem was fetching containers took too long since we have kitchen sink containers that are like 10 GB (!) each. They seem to spin up pretty fast though if the image is already pulled. I've worked on a service that lives in the k8s cluster to pull images to make sure they are fresh (https://github.com/lsst-sqre/cachemachine) but curious if you are talking about that or the networking?
From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser?
Jupyter notebooks are actually a use case we think about a lot, you can try a live demo with a Jupyter notebook here: https://jamsocket.com/tmpenv/
It wasn't really one thing with Kubernetes that was slow, but that the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying. The image pulling you mention is a good example of that; having pre-pulled images is a big factor, but we have too many images to push every image to every node, instead we'd like the scheduler to be aware of which node has which image. We could do that with node affinity, but what we'd end up building would be more work than if we wrote our own scheduler to support it from day one.
> From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser?
Our goal is subsecond container starts. We're not there yet, and might not get there with Docker, but we have a POC that is there with WebAssembly-based workloads. Too bad those are rare :)
(By the way, I'm always happy to chat about this stuff, my email is in my profile)
It turned out to be somewhat tricky, because it increased the size of the Node object, and colocating node heartbeats onto the same object meant that a bigger object was changing relatively often. But that was addressed by moving heartbeats to a different object: https://github.com/kubernetes/enhancements/issues/589
Very cool, I didn't know about this either. I feel like so many of these features are coming in which is great, but also part of the drag of k8s is the kind of constant upgrade churn and having to keep your yaml fresh.
AWS has put work into fast-starting containers [1] using tricks like lazy loading container storage, profiling container startup, non-lazily priming critical blocks, and caching shared blocks. IIRC parts of it are open source. I don't know if enough of it is open source to be helpful, but it's cool stuff!
> the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying
Since we were headed down that path, we took a step back and asked what we were really getting out of Kubernetes, and most of it was things that were orthogonal to our intended use case. The way Kubernetes is architected around control loops works great for its intended use case, but we wanted a more event-driven system.
Event driven ... like a streaming data pipeline? Given your comment about Jupyter notebooks, that makes sense. It might be the Mesos project is better architected for your use-case. Then again, I think Mesos ported some of their schedulers to Kubernetes.
If you're pulling big images you could try kube-fledged (it's the simplest option, a CRD that works like a pre-puller for your images), or if you have a big cluster you can try a p2p distributor, like kraken or dragonfly2.
Also there's that project called Nydus that allows starting up big containers way faster. IIRC, starts the container before pulling the whole image, and begins to pull data as needed from the registry.
Ah thanks! "Lazy pulling" is what I was looking for. I was trying to find estargz (didn't remember the name) and I couldn't find a proper keyword to do it :P :D
Yeah I think we considered this, but we want the container to actually run as the user and have all the permissions set up so they can have all the right access on the cluster (kind of like a PaaS), although I think we are doing some of the stuff with the starting the container while the data is still streaming down. Black magic.
This is really awesome. Thank you for sharing this.
One of my ideas lately has been to upgrade FaaS to a full on server after a set amount of traffic. Or said differently, a dedicated server spin up that serves the same app as callable functions ala scalable RPC and upgrade to a dedicated instance composed of said functions. The best of both worlds.
Combine the scale to zero of Serverless combined with the scalability and capacity of a dedicated server.
Wow, this is excellent! At a previous job, we had been using k8s + knative to spin up containers on demand, and likewise were unhappy with the delays. Spawner seems excellent.
One question: have you had to do any custom container builds on demand, and if so, have you had to deal with large "kitchen sink" containers (e.g. a Python base image with a few larger packages installed from PyPI, plus some system packages like Postgres client)? We would run up against extremely long build image times using tools like kaniko, and caching would typically have only a limited benefit.
I was experimenting using Nix to maybe solve some of these problems, but never got far enough to run a speed test, and then left the job before finishing. But it seems to me some sort of algorithm like Nixery uses (https://nixery.dev) to generate cacheable layers with completely repeatable builds and nothing extraneous would help.
Maybe that's not a problem you had to solve, but if it is, I'd love your thoughts.
Kind of curious what made it too slow for your use case? I'm guessing you did not users to wait for something like kube-dns to update or the workload scheduler? Of course things like spinning up a Pod can be slow. Or non-Kubernetes things like doing DNS ACME challenges could affect things.
But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes.
Our goal is sub-second container starts (admittedly, we're not there yet), and with Kubernetes we'd have to create Pods, create Services, wait for the scheduler to update, etc. We didn't go down the rabbit hole of profiling where the slowness was, but it was clear that Kubernetes just wasn't built with the type of speed we wanted. We realized we'd have to contend with a lot of design decisions that were the right choice for the things Kubernetes optimizes for (replication, resiliency), but not the right choice for us (fast launches of ephemeral containers).
> But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes.
I'm not sure I understand this part, I guess we could use Kubernetes operators to scale up the underlying compute resources and manage the containers ourselves? This adds a lot of complexity for our use case.
I just wrote a controller that does pretty much that – spawn containers on demand and report back status changes. While this solution does require some knowledge, it so far has been perfectly reliable and reasonably fast. I can fathom the need for processes to spawn and tear down faster in specific use cases than the Kubernetes scheduler would allow for, but for us a few seconds of wait time has been perfectly reasonable.
It's always been my understanding that with things like k8s and other orchestration stuff, you're supposed to spin up before you need the capacity? You set a threshold, like 75% capacity, and if you're over that for a bit, you spin up a new container(s) to get you back to under effectively 75% capacity.
Yes, that's the scaling model that works best for Kubernetes if the use case supports it. Our use case precludes it, because we are focused on uses where containers need to be spun up on a per-user (or per-group-of-users) basis as they use an application.
Precludes feels like the word word here. Nothing prevents you from satisfying your use case and spinning up vms before they are needed.
I wrote the student vm system for udacity, and I spun up student vms before they needed them, with some last mile loading to finalize the files they need. The student VMs were not using k8s, although a small piece of the infrastructure did.
I worried most about untrusted users working in a complex environment with the ability to harm the experience of other users, and just used GCE.
For me, boot time was < 5 minutes, so if you can predict the next five minutes of demand, you can boot those machines early. If you are wrong you will pay extra or someone will wait extra time, but still less than the full boot time. Generally it takes less than 10 seconds to access a vm with your coursework on in.
Keep a pool of eg 5 pods that are warm and good to go. When a user needs one, the pod pulls in necessary config data which should be quick as it's already initialized.
There isnt. K8 does start pods and services fast. Im able to launch an entire stateful WordPress pod (multi-container) in just ~10 seconds. Including the provisioning and attaching times of PVs from scratch. This is at Digitalocean. You can easily run stateful things like WP if you build your pods well and use PVCs - even without needing to make them stateful sets. It ends up being a neatly constructed, integral VM living on virtualization. Everything is taken care by K8.
When using K8, if you use the most basic K8 features and concepts, things generally work out pretty ok.
10s isn’t terribly long but, honestly I think 1s should be achievable and would open up more use cases for K8s in general (cloud provider physical machine latency not withstanding of course).
1s is very achievable iff you have spare capacity and images already pulled. There's good reasons the latter is hard to achieved, and the former is a tradeoff few are willing to make as it increases costs.
Never started. My work infrastructure is Elastic Beanstalk and my personal infra is either hand-managed containers or Dokku. Previous gig maintained an internal abstraction on top of Kubernetes but it wasn't something I ever had to mess with.
I used it to host several small projects in cheap virtual machines. The setup is very straightforward.
I guess we just need better editing support of YAMLs.
I went from on-metal K8s clusters, which were a complete PITA and required a full team to manage, to using EKS which has been everything K8s should be... easy peasy.
We have had a few teams try, but as soon as you go beyond "I want to run some code for a bit", nobody really has anything for you. Instead of trying to re-invent the wheel (service discovery, mutual TLS, cross-provider capabilities) successfully, it went downhill quite fast and they moved back. (this was mostly due to cost as other services can get expensive really quickly, and because of the lack of broadly available knowledge for the custom stuff they had to build)
If a team were to start with no legacy and no complexity and there isn't going to be multi-team/multi-owner/shared-services I could see them using something else. But that applies to anything.
1. I used to use it at an old company in the early days of K8s. We ran our own setup, as EKS and AKS didn't exist. GCP did, but we were on AWS. It really is very complicated, however, with EKS, GCP, and AKS, it makes it a lot easier. Note, for users it's a lot simpler than the alternatives. Sure it's no heroku, but it vastly makes things easier compared to running on AWS, GCP, Azure, or worse, on bare metal.
2. Essential. K8s solves a problem that's quite complex. You can't really solve it in a simple manner.
3. Probably, but that 10% will require the additional abstractions and complications anyway, and it'll be easier to manage one system rather than 2.
k8s is actually very simple as a user. It's complicated to operate it yourself without EKS< GKE, etc. But from an end user perspective you write some delcarative manifests, they get put into an event bus, and then the state of the world is reconciled with your manifests. Easy peasy.
Depends. Are you a >500 Developer org with many services? Then it's easy compared to what's out there. Anything less than that I'd say it's complex and you'd be better off using a PaaS
> 2. Is that complexity incidental or essential?
Depends. If you're going to do simple things forever then it's an overkill. But if you expect to grow in unknown ways in the future and don't want to waste your time doing bunch of migrations in the future then it's essential.
> 3. Could we get away with a simpler set of abstractions for 90% of applications?
Maybe? Heroku, AppEngine, CloudFoundry tried, but didn't go too far. Let's see what new crop of PaaS offerings are able to do
> Anything less than that I'd say it's complex and you'd be better off using a PaaS
Kubernetes is available as a managed service in AWS, Azure, Google etc and this is likely to be the most popular deployment model.
By any definition this is a PaaS and if you add in custom monitoring, logging, security, ingress etc. is going to be just as simple and significantly cheaper than using a managed solution.
If you're just building a basic website then sure it's an overkill but fewer people are building those these days.
it really comes down to what you're building. many web app startups would be better served paying for PaaS that manage this for them. as an example: Netlify/Vercel. if you need a database add FaunaDB to that. if that sounds risky or expensive, consider the cost of building a DevOps team.
In my experience, Kubernetes can be straightforward if:
1. You read the O'Reilly book first (or another good book). There are a few unexpected abstractions (replicas, services, deployments, etc) which the book explains nicely.
2. You pay for a hosted Kubernetes. Google's is great. EKS is workable, but you may need to spend more time configuring it.
3. You don't mess with the networking system, and nothing goes horribly wrong.
Our clusters peak out at close to 400 CPUs, and Kubernetes generally does what it says it will do.
One caveat: If your app can be deployed using a "platform as a service" (Heroku, Render, etc), that's usually a better idea than Kubernetes. Kubernetes makes sense when a PaaS starts feeling too limited.
On the PaaS front, I've also found that for smaller applications / new startups the pricing between starting with k8s or starting with a PaaS are pretty similar.
Definitely. Works for FaaS too up to some extent (especially if you need hot-standby/zero-latency startup). Problem is mostly that once you scale beyond the "look it says hello world" levels the price goes up so fast you can essentially pay someone to "make it cheaper by running it elsewhere" and still be better off.
Most questions seem to revolve around a tiny part of the puzzle, or a small "just starting out" phase and completely forgets about the lifecycle of the business process that it is built for, and the existing systems it needs to interact with. Even a startup will have that problem considering most are trying to get bought which essentially means being absorbed into a legacy company. So even starting out with no legacy to worry about is just a stay of execution.
Do you write code in python? Is it really complicated to write a script to fetch some data, extract info, upload it someplace else? It’s not.
But then, someone is trying to fetch 50GB files and now you need to play with buffers. The script misbehaves so the API rate limits you and now you need to handle credentials, back-off, etc. The script hangs in some strange state and you need to add structured logs to figure out what is happening. Now we need to upload multiple files in parallel, are we going multi-process or multi-thread? Is python the right language? Are we going to use one pod or many?
See how it quickly gets complicated? Add to all that the fact that it’s easy to spin-up rabbitMQ with some defaults with helm locally. So you do that in production as well and when it goes down you don’t know what’s happening.
As another commentator said, there is a level of knowledge that is required to things in production reliably.
I feel like you might be listing a bunch of edge cases that won't affect most people, or at least are as likely as a bunch of other edge cases with completely different optional solutions
It’s not any more complicated than doing it other ways if you want control of the full infra stack. Advantage of k8s is you get a hardened unified API that works everywhere.
> 3. Could we get away with a simpler set of abstractions for 90% of applications?
Yeah but you can do that in k8s too, check out knative serving for example. K8s encourages the creation of higher level abstractions, with the advantage that you always have the break glass to dig into the primitives, which you don’t get with a lot of other systems.
K8s does not have to be complex. If all your doing is, hosting a bunch of various web services, it is really simple. Actually, doesn't have to be just web: you can host services that only can communicate within kubernetes, or services that monitor and manage some XYZ resource, etc.. and that will all be really simple.
Even hosting redis etc.. is really straight forward.
It is funny, but the complexity starts to happen where you want kubernetes to handle other stuff: like hosting databases, or other storage resources, and if you want to for some reason I will never understand have your external services essentially communicate directly with kubernetes rather than have some middleware service you pay for do that for you (like a load balancer, etc.)
One thing I did have an issue with was setting up SSL... that was surprisingly stupid. Should have been much easier to do that with LetsEncrypt.
It's too dynamic to have universal answers.. but I'll give it a shot:
1. It's only as complicated as you make it. Kubernetes is essentially PKI (which is a must in any case), a REST API, and a scheduler. It stores some stuff somewhere, and you can add more stuff for it do have more features. I wouldn't call that complicated and it's essentially what Swarm and Mesos do as well (minus the PKI part).
2. PKI is essential. If you think that's complicated that's a whole different problem. Everything else is incidental. If adding more OpenAPIV3 schemas or REST API seem complex, again, not really a Kubernetes thing, mostly a general software development thing.
3. Yes, as 90% of applications really only exist as mediocre CRUD viewers you could run on a potato. Also, 90% of applications don't need to be as highly available or scalable as people might think. Then again, ecosystem complexity in software development combined with the lack of general knowledge (i.e. how to use an RDBMS properly) makes that while the software is simple and could be run as a single statically compiled binary, it generally is a mess, requiring more messes to make it run. But since that is cheaper (less developer time spent, more cheaper developers available to do that type of work), that is where we end up.
1 - No. People try to implement old, complex, stateful apps on K8 by just slapping on some stuff. That creates problems.
2- See 1.
3 - If you can containerize your app in a simple way, then yes.
Note that a stateful app that would require attention in a bare metal server or a singular VM would still require that kind of attention on K8 as well. K8 just removes the need to manage the VM infra. And makes running your infra as code much easier.
If you need to run a stateful app in a highly available manner, you can do it in K8 and it would be good - however you will spend a similar effort for maintaining the high available services like you do in other venues. Ie, if your stateful app requires a Percona cluster and a NFS cluster off of K8, you will still need to launch and maintain those services. K8 operators make these a lot easier to launch and maintain. But its still maintenance nonetheless.
Using managed, hosted databases can work for the database part. But they are expensive. So launching a database cluster via a K8 operator would be cheaper to maintain. NFS is a problematic thing across all platforms. So if you need it, you either launch a rook-ceph cluster to provide a shared filesystem or use a hosted service like Google File Store.
Came to this conclusion for teams of our current size based on years of experience and experimentation (never at the expense of the business).
They are a better fit because they are much easier to manage and it's much easier to debug issues when something goes wrong. We are a small team and we hope to stay that way. But our operational responsibilities are growing significantly. The extra cognitive overhead of working with technologies like kubernetes would prevent us from scaling up our effort the way that we want to.
It's hard to answer your question in detail outside of a very long essay.
Kubernetes is overkill, but one will have to pry containers from my cold, dead hands. I will not deal with installing dependencies from the OS package manager and editing /etc files any more.
Yeah I am not hardline anti-Docker. And anti-k8s only for small teams/companies.
With larger teams, I have written and maintained custom k8s operators in production. It was a great fit for the problems we had at that scale of developers.
A terrible fit for the problems my current team has.
What, zero? (GP didn't say anything about using Hashicorp-managed products, they're open source and beer-free to use. Another comment says Hashicorp's platform doesn't even offer Nomad (yet?) anyway.)
If you need features offered by the self-managed Enterprise version of a Hashicorp product, I've heard the price tag is something like low six figures per product.
Hashicorp is very inflexible about support plans -- either you go all in on their Enterprise product, or you're self supported. By the time you've licensed Nomad, Consul and Vault -- because they interact and you will find Nomad support ends where Consul support begins, and so on -- it is a LOT of money.
It depends on your size. For a fairly minimal close-to-best-practices you'll need for each DC, each on a separate physical host (I may be missing something):
3 x Consul server
3 x Nomad server
2/3 x Vault server
It's long since I operated k8s but IIRC I think you can get similar capabilities and redundancy with 3-5 machines?
That's before you start looking at actual runner nodes, load balancers, proxies, logging and monitoring infra, etc...
Unless you cheat (which I think many do) or you're big enough, that overhead can be meaningful.
FWIW we recognized this was too much overhead for many users. Nomad 1.3 supports service discovery so you can start without Consul, and 1.4 will support secure variables to get folks farther along without requiring Vault.
So 3 Nomad servers should give you a pretty featureful and highly available cluster these days.
Yeah or, like, spin up three medium servers in different zones and have each server run all three services. We did that for a production setup for years and it worked fantastically. There's no need to have nomad/consul/vault all on different servers unless they are significantly underpowered or the workloads are crazy.
If best practices say otherwise, then maybe they should be reconsidered.
Sure, but at this point there's so much else we get from Consul that, like, what's the point...
I guess the path is set but I'd personally much prefer having a recognized deployment scenario be hosting Consul server and Nomad server on the same physical machines, and accommodating (be it through code or just docs) for making that play well with security, certs, and resource usage without becoming a confounding mess.
Even Vault, if the operator accepts and/or mitigates the sidechannel aspects - from a security perspective that still shouldn't be a step down from anything Nomad-specific?
Seeing as HC already provides solutions for all of these supposed to be serving for Nomad, doesn't it make more sense to make them play together smoother and nice on the same machine rather than reinventing a lesser wheel for each of them?
Entirely true, but I also think that neither k8 nor Nomad are that useful if you're not at a scale where the above is negligeable? It costs roughly 500 usd a month on aws for those 9 servers.
OT: I really do not like Hashicorp. Terraform has a terrible DSL, and terrible documentation. Also, I paid $70 for their VMWare Vagrant plugin ages ago, it was so buggy to be unusable, they were unresponsive on the Github repo, and they completely ignored my request for a refund under their own 30-day guarantee. Not very professional.
I really don't get why people love that company so much.
Managing k8s requires a strong DevOps team. A strong DevOps engineer is a solid software engineer plus a specialty. These people are rare and expensive and extremely hard to attract to startups.
Therefore you either can’t find anyone or more likely you hire less good DevOps engineers.
The solution is to not use k8s as a startup. The less a DevOps engineer can shoot themselves in the foot the better.
Nope, I like k8s. What I don't like is people trying to be overly smart with it and leaving a configuration hell of templates, weird network configurations and broken certs behind them. For my personal workloads it's all basic containers with a reverse proxy, though.
I really wish Rancher didn't abandon Rancher 1.6 and moved to k8s.
This was a perfect solution for a small business and bare metal.
I am trying to move on k3s but it is just too complex to run anything and there is still not solved problem of exposing services to internet.
What I want is to declare I want this service to be under this domain and this IP - so for that you still need to configure your load balancer (bare metal) manually, setup certificates etc. I am writing a tool to automate this, but it's been a pain.
> What I want is to declare I want this service to be under this domain and this IP - so for that you still need to configure your load balancer (bare metal) manually, setup certificates etc. I am writing a tool to automate this, but it's been a pain.
So this is not going to work for several reasons.
One being that on bare metal you don't have a cloud provider, so there is no load balancer it can talk to.
Second - it will setup a hostname and a certificate on the ingress, but there is no way to contact it from outside world. The domain still needs A record pointing at the server and in the cluster that may be a local IP or a set of IPs.
What I have in mind is an external server that is not being a part of the cluster that bears the role of load balancer. It will contact the cluster and look for services and then setup up a reverse proxy based on their declared hostname, then setup certificates and update DNS records at DNS provider.
As far as I know something like this does not exist.
Maybe Traefik has such a capability, but their documentation is so complex I have no idea.
Actually I'm using it on bare metal and it works. Initial setup wasn't very hard but I think it could be more intuitive. Overall I think documentation for self-hosting kubernetes sometimes a bit incomplete.
Yes, I need to add A records with IPs for each domain, but that's one time setup. I did it manually, but you can automate it [1] (depends on what you use for DNS provider but you can extend it to support your provider or maybe there is another existing solution).
I'm not sure that one server in front of the cluster is more reliable than using all cluster nodes for load balancing. I guess that in automated solutions like [1] cluster's node could be automatically deleted from DNS if it went down.
My setup is not so big so I don't have real need for load balancing, but it seems possible with existing solutions.
Sure it does, I ran kube-vip[1](but there are many others, e.g. metallb) as my cloud controller, all it needs are valid static IPs/range/dhcp and it will assign these to LoadBalancer services(which you usually only need one of for your ingress) and it will either ARP or use BGP to route external traffic.
As for DNS records, external-dns[2] works perfectly as long as your DNS as some way to doing automatic updates.
The problem with kube-vip is that it has poor documentation. I have read it many times and still don't know how I could use it.
Last time I was running something assigning IP addresses to the dedicated server interface I got it null routed and provider threatened to terminate the service because it was interfering with other clients network.
So if I see things like ARP, BGP, DHCP it is not clear what exactly it does on the network and how that would work in the real world.
I am missing an example where I have a server with a static IP from which I want to access the exposed services that are on a private network.
All I really want is an automatically configured reverse proxy that will direct traffic to appropriate services and take care of certificates and DNS.
Before the Kubernetes I used Rancher 1.6 and that was super simple. For instance I would start a wordpress container and then all I needed to do was to add a reverse proxy entry with its hostname as a backend and point where the certificates are (that was before lets encrypt).
Closest I could get was exposing a NodePort and having nginx to reverse proxy to the nodes at given port, but that seems more complex / fragile, as I need o keep track which service uses which port and it is still manual, so I might as well just use containers without Kubernetes.
Yes! My startup of 5 people did. We started out with a managed Kubernetes cluster on DigitalOcean, but there were a number of reasons that caused us to not be very comfortable with that setup.
- Taking random .yml configs from The InternetTM to install an Nginx Ingress with automatic LetsEncrypt certs felt not-exactly-great. It's no better than piping curl to bash, except the potential impact is not that your computer is dead, but the entirety of prod goes down.
- Because of this, upgrades of Kubernetes are a pain. The DigitalOcean admin panel will complain about problems in 'our' configs, that aren't actually OUR configs. We don't know how to fix that, or if ignoring the warnings and upgrading will break our production apps.
- Upgrades of Kubernetes itself aren't actually zero downtime, and we couldn't figure out how to do that (even after investing a significant amount of research time).
- We were using only a tiny subset of the functionality in Kubernetes. Specifically we wanted high-availability application servers (2+ pods in parallel) with zero-downtime deployments, connecting to a DO managed PostgreSQL instance, with a webserver that does SSL-termination in front of it.
- Setting up deployments from a GitLab CI/CD pipeline was pretty hard, and it turned out the functionality for managing a Kubernetes cluster from GitLab was not really done with our use case in mind (I think?).
- It would be bad enough if DigitalOcean shit the bed, but the biggest problem was that we couldn't reliably recognize if something was a problem caused by us, or by DO. Try explaining that one to your customers.
Summarizing: it was just too complex and fragile, even once you wrap your head around what the hell a Pod, a Deployment, an Ingress and Ingress Controller, and all of the other Kubernetes lingo actually means. I suspect you need a dedicated infra person who knows their stuff to make this work, so it could very well make sense for larger companies, but for our situation it was overkill.
We were not intellectually in control of this setup, and I do not feel comfortable running production workloads (systems used by 20k high-school students, mission-critical applications used by logistical companies) on something we couldn't quite grasp.
We went to a much simpler setup on Fly.io, and have been happy since. It's a shame they seem to be too young of a company to really be super reliable, but I suspect this is only a matter of time. In terms of feature set, it's all we need.
Yeah, definitely! This is why I am not that harsh of Kubernetes as a tool at all, I'm just saying that it's not suitable for us for these reasons. In our context of <4 FTE of dev power it just isn't worth the manpower we have to throw at it to make it work, I'd much rather invest that time into moving our core business forward. I might see ourselves moving back to it in the future, but in the meantime we really just need a Heroku / Fly.io / DO apps / AWS ELB or so.
At my previous employer (~50 FTE of devs, 2-ish FTE dedicated to infra) Kubernetes worked perfectly fine, and I think in that context it made a lot more sense.
Yep! And we had to make a decision whether we would focus on our core business of developing great applications for end-users, or spend more time running infra and try to wrap our heads around the mountain of complexity that is k8s. That choice at our size is a no-brainer, although that trade-off might be very different for larger teams.
For context, I ran a DevOps team for the last 4 years that managed two products on AWS - one on EKS and one on ECS. I also just finished building out more or less that exact infrastructure on DO.
I can pretty confidently say, that's not K8s, that's Digital Ocean. On AWS, we ran the EKS infrastructure (which was not simple) with basically half a dev's time for years. It was only when it started to scale to millions of users that we needed to build a team to support it. It was still a much smaller team than the one that supported the ECS product (two devops).
I was mostly managing and not coding by the time Kubernetes was in our stack, so while I'm very familiar with infrastructure in general (and I know ECS inside and out unfortunately), I hadn't used Kubernetes directly much before I build this DO infrastructure. But I got it up in a week and though DO is a nightmare, k8s is an absolute joy as a DevOps. Holy shit it's perfect. It does exactly what it needs to, with exactly the right abstractions, with perfectly reasonable defaults.
The reality is that infrastructure work is just that complicated.
You wouldn't try to have a team of front end engineers build your rest backend. It's not reasonable to expect javascript engineers to know how to build and operate an infrastructure - at least not with out dedicating themselves to learning the tooling and space full time for a while. Think of it from the perspective of a frontend engineer learning Python and Django to build out a rest backend, and then multiply the complexity by 4. That's just infrastructure regardless of what you're using.
That said, if something like Fly.io can fit your needs, that's great! I haven't used them so I can't speak to them directly, but I know that with Heroku, the trade off was cost and, eventually, being limited in what you could build. Eventually you would need to build something that just couldn't be built with Heroku. A quick glance at Fly, the pricing looks reasonable, but I'm guessing the build limits will still apply.
That's fair enough. We took a look at 'native' AWS, but there are a multitude of reasons why just dealing with AWS at all is a huge upfront time investment too if you don't hire somebody already skilled at this (complicated billing, just figuring out the product names for their various services, to name a few).
> The reality is that infrastructure work is just that complicated.
Yes, if you need the flexibility of running anything in any setup. What we really wanted was 'yeet a docker image with a web server in it + env vars at some magic beast that'll run it for me, slap an SSL-cert on it, and make sure it's always online'. We tried to replicate this with Kubernetes, so we got the full complexity of k8s unloaded upon us.
Heroku was what we really wanted, but it was always too expensive. Fly.io strikes a good balance here, the defaults are sane, it's still flexible enough for other services, and it's relatively cheap (spend is similar to DO K8s).
> You wouldn't try to have a team of front end engineers build your rest backend.
Well, yes and no. I wouldn't expect frontend engineers to know the ins and outs of everything backend, but to build on your metaphore a bit further: Setting up a basic Node backend with express serving static files shouldn't take multiple weeks, even for a frontend engineer. I feel like I was trying to do the infra equivalent of that, and it did take me forever.
> A quick glance at Fly, the pricing looks reasonable, but I'm guessing the build limits will still apply
The build limits could be an issue but really isn't for us right now. It's fairly easy to build locally though (in our case: in our GitLab CI/CD runners)
> Well, yes and no. I wouldn't expect frontend engineers to know the ins and outs of everything backend, but to build on your metaphore a bit further: Setting up a basic Node backend with express serving static files shouldn't take multiple weeks, even for a frontend engineer. I feel like I was trying to do the infra equivalent of that, and it did take me forever.
Yeah, that's just what infrastructure work is. Like I said, take that analogy, multiply the complexity by 4 (at least... really maybe multiply it by an order of magnitude).
Let me put it in perspective. I've been coding since I was 12, I taught myself C to build a MUD in middle school and high school. I had about a decade of full stack professional experience in Java, PHP, javascript and I'd done infrastructure work with EC2 and chef before. When I moved into DevOps it was overwhelming.
I've been in DevOps for 4 years. I built that equivalent DO infrastructure with Kubernetes just last week (and in a week). I started the week going "Fuck, I don't know what I'm doing." The first 3 days were just spent reading documentation. Day four was spent writing the terraform and kubernetes manifests - with a distinct feeling that none of this was going to work because I was missing several key pieces. Day 5 was spent putting a few of those pieces in place and debugging. I finally got it working late Friday night. I took on a ton of tech debt and made a bunch of compromises just to get something working. I'm not the least bit happy with what I have working and intend to totally rebuild it on AWS when it comes time to build production.
And that's with 4 solid years of doing infrastructure work full time under my belt. For someone with no infrastructure experience? I would estimate 1 - 3 months. There's just way too much to learn to think you could do it quickly and simply.
With an express backend, if you have javascript experience, you really don't have much to learn. You need to learn how http interacts with the backend, how the backend interacts with the database, and databases (SQL). That's it. Learning database is not nothing, there's a lot that comes with it, but that's still only 2 new tools really.
With infrastructure, you need to learn networking, databases, securty, container orchestration (how does high availability work? Scaling?), bash, linux, provisioning, terraform, Docker, Kubernetes manifests, monitoring, secrets handling, and more. And for a lot of these things, the solutions are far from simple or perfect. Even when done as well as can be with modern tech it feels shakey and cobbled together at the end. You're tying a dozen different tool types together to solve a dozen different problems and you have dozens of choices for each tool type.
Like I said, infrastructure is just like that. And it's important to have the right expectations going in to it.
If you can't tell, I've had this conversation with my peers who stayed in full stack a lot.
Nomad + Consul(with Consul Connect) + Vault. With Terraform obv.
We don't really have a use-case for Boundary but it looks pretty neat as well if you do.
Was on k8s for years and I don't miss it one bit.
While there definitely is some complexity once you get serious and set everything up properly with raft, federation, Connect, CAs, proxies, ACLs, proper secrets lifecycles... I find it's worth it. With the current assumptions that HC will keep improving and existing bugs and edge-cases will be ironed out.
I really like the look of nomad and want to give it a go. The two things holding me back are:
1) I don't really want to manage the installation but there aren't any(?) cloud hosts for nomad that I can see.
2) It doesn't seem as widely used so community support seems thin. There aren't many blog posts about good patterns with it etc, and I'd worry that we'd get stuck and end up reverting back to k8s.
There is no installation needed with Nomad, it's a standalone binary.
Just fire it in a small Debian (or Alma) instance on EC2 or GCE and you're done. That should solve point 1.
Point 2 is debatable. Lots of people nowadays put Kubernetes on their resume but that doesn't mean they are great architects or technicians, yet a good part of running production on Kubernetes is doing it right.
You'll see much fewer people with Nomad on their resume, but on the other hand you know they're not here for the buzz, they're usually more experienced and know what they're talking about.
Koyeb also moved off Kubernetes and went with Nomad. We started with Kubernetes, thinking it was the right abstraction layer for us to build our platform, but then quickly ran into major limitations. The big ones: as others have mentioned in this thread, its complexity; security (we wanted to explore using Firecracker on Kubernetes, but it was very experimental at that time); we were not interested in keeping up with its release cycles; global and multi-zone deployments was not as straightforward as we needed; and the overhead (10-25% of RAM) was a cost we were not willing to take (we are around 100MB with our new architecture).
Nomad replaces parts of K8. It is not a drop-in replacement. If one only want the container orchestration that is fine but then you need Consul for service discovery and so on.
I've described this as "Nomad is a container orchestrator, while Kubernetes has a container orchestrator".
Nomad, Consul and Vault interoperate extremely well and are mostly pleasant to use, but I found myself missing the rest of the ecosystem pretty quickly, especially around ingresses, and I think they made the wrong decision on the networking model compared to Kubernetes.
That said, I haven't played with Consul Connect, the Consul+Envoy service mesh, yet. That might address a lot of the problems. But fundamentally I can't help but think that Nomad and Kubernetes both made a run of it and Kubernetes came out the winner of mindshare and ecosystem.
They're also adding in a basic secrets k/v store in the next version. Their intention (I believe) is to target small use-cases and IOT use-cases; while allowing folks to scale to gigantic levels when mixing in Consul/Vault.
320 comments
[ 4.3 ms ] story [ 280 ms ] threadhttps://en.wikipedia.org/wiki/Gartner_hype_cycle
For my new projects nowadays, I'm pushing mainly serverless approaches using AWS Lambdas (behind API Gateways for stuff that needs to be reachable by HTTP).
I think this shifts the complexity from managing Kubernetes and its accompanying ten-thousand-yaml-files to infrastructure-as-code and the complexities of dealing with AWS. And I happen to prefer the latter, even though it's not infinitely better by any margin.
For the few things that needs to be always-online, or 3rd party self-hosted apps, I'm still on Kubernetes, or pure Docker if possible.
So cloudformation YAML? or CDK?
Also how is the cost of Lambda? I know for AWS the logic apps have a really high cost (but function apps seem to be reasonable)
Not to mention any managed services you want to use, which also will lock you in. So I don't do anything extreme to avoid vendor lock-in, other than making my code general enough to only have a small surface area for the lambda entry point. As a practical example, all of my APIs hosted on Lambdas are ordinary ASP.Net apps that would work identically if hosted in Docker containers.
Pricing so far is one of the biggest benefits of doing serverless approaches. I'm down to paying a couple of dollars per month for something that I'd pay tenfold for if doing Kubernetes. Both the monetary sum of only paying for what you're using, and also not having to worry about cluster maintenance, scaling and management is a godsend.
We ended up writing our own control plane that uses NATS as a message bus. We are in the process of open sourcing it here: https://github.com/drifting-in-space/spawner
Just curious if you could elaborate here? I work with k8s on docker, and we're also going to be spinning up ephemeral containers (and most of the other things you say) with jupyter notebooks. We're all in on k8s, but since you might be ahead of me, just wondering what hurdles you have faced?
Our big problem was fetching containers took too long since we have kitchen sink containers that are like 10 GB (!) each. They seem to spin up pretty fast though if the image is already pulled. I've worked on a service that lives in the k8s cluster to pull images to make sure they are fresh (https://github.com/lsst-sqre/cachemachine) but curious if you are talking about that or the networking?
From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser?
It wasn't really one thing with Kubernetes that was slow, but that the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying. The image pulling you mention is a good example of that; having pre-pulled images is a big factor, but we have too many images to push every image to every node, instead we'd like the scheduler to be aware of which node has which image. We could do that with node affinity, but what we'd end up building would be more work than if we wrote our own scheduler to support it from day one.
> From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser?
Our goal is subsecond container starts. We're not there yet, and might not get there with Docker, but we have a POC that is there with WebAssembly-based workloads. Too bad those are rare :)
(By the way, I'm always happy to chat about this stuff, my email is in my profile)
The kubernetes scheduler should be aware of which node has which image, that is why the Node object has the status.images field: https://kubernetes.io/docs/reference/generated/kubernetes-ap....
It turned out to be somewhat tricky, because it increased the size of the Node object, and colocating node heartbeats onto the same object meant that a bigger object was changing relatively often. But that was addressed by moving heartbeats to a different object: https://github.com/kubernetes/enhancements/issues/589
It doesn't get all the way to what we want, but it could be used to build a piece of it.
[1] Gigabytes in milliseconds: Bringing container support to AWS Lambda without adding latency. https://www.youtube.com/watch?v=A-7j0QlGwFk
https://cloud.google.com/kubernetes-engine/docs/how-to/image...
> the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying
Since we were headed down that path, we took a step back and asked what we were really getting out of Kubernetes, and most of it was things that were orthogonal to our intended use case. The way Kubernetes is architected around control loops works great for its intended use case, but we wanted a more event-driven system.
Also there's that project called Nydus that allows starting up big containers way faster. IIRC, starts the container before pulling the whole image, and begins to pull data as needed from the registry.
https://github.com/senthilrch/kube-fledged
https://github.com/dragonflyoss/Dragonfly2
https://github.com/uber/kraken
https://nydus.dev/
https://github.com/containerd/stargz-snapshotter/blob/main/d...
One of my ideas lately has been to upgrade FaaS to a full on server after a set amount of traffic. Or said differently, a dedicated server spin up that serves the same app as callable functions ala scalable RPC and upgrade to a dedicated instance composed of said functions. The best of both worlds.
Combine the scale to zero of Serverless combined with the scalability and capacity of a dedicated server.
One question: have you had to do any custom container builds on demand, and if so, have you had to deal with large "kitchen sink" containers (e.g. a Python base image with a few larger packages installed from PyPI, plus some system packages like Postgres client)? We would run up against extremely long build image times using tools like kaniko, and caching would typically have only a limited benefit.
I was experimenting using Nix to maybe solve some of these problems, but never got far enough to run a speed test, and then left the job before finishing. But it seems to me some sort of algorithm like Nixery uses (https://nixery.dev) to generate cacheable layers with completely repeatable builds and nothing extraneous would help.
Maybe that's not a problem you had to solve, but if it is, I'd love your thoughts.
But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes.
> But on other hand, I can't quite figure out why something would prevent, you, yourself, from running the service that hosts the VMs that hosts the containers on demand on Kubernetes.
I'm not sure I understand this part, I guess we could use Kubernetes operators to scale up the underlying compute resources and manage the containers ourselves? This adds a lot of complexity for our use case.
Is that not how this works?
I wrote the student vm system for udacity, and I spun up student vms before they needed them, with some last mile loading to finalize the files they need. The student VMs were not using k8s, although a small piece of the infrastructure did.
I worried most about untrusted users working in a complex environment with the ability to harm the experience of other users, and just used GCE.
For me, boot time was < 5 minutes, so if you can predict the next five minutes of demand, you can boot those machines early. If you are wrong you will pay extra or someone will wait extra time, but still less than the full boot time. Generally it takes less than 10 seconds to access a vm with your coursework on in.
When using K8, if you use the most basic K8 features and concepts, things generally work out pretty ok.
For personal projects I roll with just docker.
I'm bothered by the minimal requirements of k8s, I want to deploy on 5$ machines
[0] https://rancher.com/docs/k3s/latest/en/
I used it to host several small projects in cheap virtual machines. The setup is very straightforward. I guess we just need better editing support of YAMLs.
Thanks for the recommendation!
or just containers on the virtual machine?
I would love to deploy with docker and no other orchestration tools.
I went from on-metal K8s clusters, which were a complete PITA and required a full team to manage, to using EKS which has been everything K8s should be... easy peasy.
If a team were to start with no legacy and no complexity and there isn't going to be multi-team/multi-owner/shared-services I could see them using something else. But that applies to anything.
1. Is it really so complicated?
2. Is that complexity incidental or essential?
3. Could we get away with a simpler set of abstractions for 90% of applications?
2. Essential. K8s solves a problem that's quite complex. You can't really solve it in a simple manner.
3. Probably, but that 10% will require the additional abstractions and complications anyway, and it'll be easier to manage one system rather than 2.
... and we're now fully back to the mainframe era with the people in white coats who "run" the computer.
The cloud truly is mainframe 2.0.
b) Kubernetes clusters can span multiple accounts, clouds etc.
Depends. Are you a >500 Developer org with many services? Then it's easy compared to what's out there. Anything less than that I'd say it's complex and you'd be better off using a PaaS
> 2. Is that complexity incidental or essential?
Depends. If you're going to do simple things forever then it's an overkill. But if you expect to grow in unknown ways in the future and don't want to waste your time doing bunch of migrations in the future then it's essential.
> 3. Could we get away with a simpler set of abstractions for 90% of applications?
Maybe? Heroku, AppEngine, CloudFoundry tried, but didn't go too far. Let's see what new crop of PaaS offerings are able to do
Kubernetes is available as a managed service in AWS, Azure, Google etc and this is likely to be the most popular deployment model.
By any definition this is a PaaS and if you add in custom monitoring, logging, security, ingress etc. is going to be just as simple and significantly cheaper than using a managed solution.
If you're just building a basic website then sure it's an overkill but fewer people are building those these days.
1. You read the O'Reilly book first (or another good book). There are a few unexpected abstractions (replicas, services, deployments, etc) which the book explains nicely.
2. You pay for a hosted Kubernetes. Google's is great. EKS is workable, but you may need to spend more time configuring it.
3. You don't mess with the networking system, and nothing goes horribly wrong.
Our clusters peak out at close to 400 CPUs, and Kubernetes generally does what it says it will do.
One caveat: If your app can be deployed using a "platform as a service" (Heroku, Render, etc), that's usually a better idea than Kubernetes. Kubernetes makes sense when a PaaS starts feeling too limited.
Most questions seem to revolve around a tiny part of the puzzle, or a small "just starting out" phase and completely forgets about the lifecycle of the business process that it is built for, and the existing systems it needs to interact with. Even a startup will have that problem considering most are trying to get bought which essentially means being absorbed into a legacy company. So even starting out with no legacy to worry about is just a stay of execution.
But then, someone is trying to fetch 50GB files and now you need to play with buffers. The script misbehaves so the API rate limits you and now you need to handle credentials, back-off, etc. The script hangs in some strange state and you need to add structured logs to figure out what is happening. Now we need to upload multiple files in parallel, are we going multi-process or multi-thread? Is python the right language? Are we going to use one pod or many?
See how it quickly gets complicated? Add to all that the fact that it’s easy to spin-up rabbitMQ with some defaults with helm locally. So you do that in production as well and when it goes down you don’t know what’s happening.
As another commentator said, there is a level of knowledge that is required to things in production reliably.
> 3. Could we get away with a simpler set of abstractions for 90% of applications?
Yeah but you can do that in k8s too, check out knative serving for example. K8s encourages the creation of higher level abstractions, with the advantage that you always have the break glass to dig into the primitives, which you don’t get with a lot of other systems.
2) necessary at scale, incidental before then.
3) yes.
Even hosting redis etc.. is really straight forward.
It is funny, but the complexity starts to happen where you want kubernetes to handle other stuff: like hosting databases, or other storage resources, and if you want to for some reason I will never understand have your external services essentially communicate directly with kubernetes rather than have some middleware service you pay for do that for you (like a load balancer, etc.)
One thing I did have an issue with was setting up SSL... that was surprisingly stupid. Should have been much easier to do that with LetsEncrypt.
Then you run into a litany of issues with networking (like you mentioned SSL termination) and stateful apps or databases.
Even in this thread, someone mentioned how Redis defaults lead to a lot of issues in containers.
1. It's only as complicated as you make it. Kubernetes is essentially PKI (which is a must in any case), a REST API, and a scheduler. It stores some stuff somewhere, and you can add more stuff for it do have more features. I wouldn't call that complicated and it's essentially what Swarm and Mesos do as well (minus the PKI part).
2. PKI is essential. If you think that's complicated that's a whole different problem. Everything else is incidental. If adding more OpenAPIV3 schemas or REST API seem complex, again, not really a Kubernetes thing, mostly a general software development thing.
3. Yes, as 90% of applications really only exist as mediocre CRUD viewers you could run on a potato. Also, 90% of applications don't need to be as highly available or scalable as people might think. Then again, ecosystem complexity in software development combined with the lack of general knowledge (i.e. how to use an RDBMS properly) makes that while the software is simple and could be run as a single statically compiled binary, it generally is a mess, requiring more messes to make it run. But since that is cheaper (less developer time spent, more cheaper developers available to do that type of work), that is where we end up.
2- See 1.
3 - If you can containerize your app in a simple way, then yes.
Note that a stateful app that would require attention in a bare metal server or a singular VM would still require that kind of attention on K8 as well. K8 just removes the need to manage the VM infra. And makes running your infra as code much easier.
If you need to run a stateful app in a highly available manner, you can do it in K8 and it would be good - however you will spend a similar effort for maintaining the high available services like you do in other venues. Ie, if your stateful app requires a Percona cluster and a NFS cluster off of K8, you will still need to launch and maintain those services. K8 operators make these a lot easier to launch and maintain. But its still maintenance nonetheless.
Using managed, hosted databases can work for the database part. But they are expensive. So launching a database cluster via a K8 operator would be cheaper to maintain. NFS is a problematic thing across all platforms. So if you need it, you either launch a rook-ceph cluster to provide a shared filesystem or use a hosted service like Google File Store.
Replaced with Linux servers and SSH.
Have done a lot of work with k8s in the past. Not the right tool for my startup.
They are a better fit because they are much easier to manage and it's much easier to debug issues when something goes wrong. We are a small team and we hope to stay that way. But our operational responsibilities are growing significantly. The extra cognitive overhead of working with technologies like kubernetes would prevent us from scaling up our effort the way that we want to.
It's hard to answer your question in detail outside of a very long essay.
We use systemd to manage services.
We use Ansible to set up servers.
Our infrastructure spans AWS, Google Cloud, and servers in a datacenter.
Linux namespaces are a brilliant idea.
With larger teams, I have written and maintained custom k8s operators in production. It was a great fit for the problems we had at that scale of developers.
A terrible fit for the problems my current team has.
That's before you start looking at actual runner nodes, load balancers, proxies, logging and monitoring infra, etc...
Unless you cheat (which I think many do) or you're big enough, that overhead can be meaningful.
FWIW we recognized this was too much overhead for many users. Nomad 1.3 supports service discovery so you can start without Consul, and 1.4 will support secure variables to get folks farther along without requiring Vault.
So 3 Nomad servers should give you a pretty featureful and highly available cluster these days.
If best practices say otherwise, then maybe they should be reconsidered.
I guess the path is set but I'd personally much prefer having a recognized deployment scenario be hosting Consul server and Nomad server on the same physical machines, and accommodating (be it through code or just docs) for making that play well with security, certs, and resource usage without becoming a confounding mess.
Even Vault, if the operator accepts and/or mitigates the sidechannel aspects - from a security perspective that still shouldn't be a step down from anything Nomad-specific?
Seeing as HC already provides solutions for all of these supposed to be serving for Nomad, doesn't it make more sense to make them play together smoother and nice on the same machine rather than reinventing a lesser wheel for each of them?
I really don't get why people love that company so much.
Therefore you either can’t find anyone or more likely you hire less good DevOps engineers.
The solution is to not use k8s as a startup. The less a DevOps engineer can shoot themselves in the foot the better.
I am trying to move on k3s but it is just too complex to run anything and there is still not solved problem of exposing services to internet.
What I want is to declare I want this service to be under this domain and this IP - so for that you still need to configure your load balancer (bare metal) manually, setup certificates etc. I am writing a tool to automate this, but it's been a pain.
After initial setup you can do it quite easily.
Exposing a service on selected domain is several lines in Ingress and adding certificates is several more. Example: https://cert-manager.io/docs/tutorials/acme/nginx-ingress/#s...
What I have in mind is an external server that is not being a part of the cluster that bears the role of load balancer. It will contact the cluster and look for services and then setup up a reverse proxy based on their declared hostname, then setup certificates and update DNS records at DNS provider.
As far as I know something like this does not exist.
Maybe Traefik has such a capability, but their documentation is so complex I have no idea.
Yes, I need to add A records with IPs for each domain, but that's one time setup. I did it manually, but you can automate it [1] (depends on what you use for DNS provider but you can extend it to support your provider or maybe there is another existing solution).
I'm not sure that one server in front of the cluster is more reliable than using all cluster nodes for load balancing. I guess that in automated solutions like [1] cluster's node could be automatically deleted from DNS if it went down.
My setup is not so big so I don't have real need for load balancing, but it seems possible with existing solutions.
[1] https://github.com/kubernetes-sigs/external-dns
As for DNS records, external-dns[2] works perfectly as long as your DNS as some way to doing automatic updates.
1. https://kube-vip.io/
2. https://github.com/kubernetes-sigs/external-dns
Before the Kubernetes I used Rancher 1.6 and that was super simple. For instance I would start a wordpress container and then all I needed to do was to add a reverse proxy entry with its hostname as a backend and point where the certificates are (that was before lets encrypt).
Closest I could get was exposing a NodePort and having nginx to reverse proxy to the nodes at given port, but that seems more complex / fragile, as I need o keep track which service uses which port and it is still manual, so I might as well just use containers without Kubernetes.
https://www.haproxy.com/documentation/kubernetes/latest/inst...
We were not intellectually in control of this setup, and I do not feel comfortable running production workloads (systems used by 20k high-school students, mission-critical applications used by logistical companies) on something we couldn't quite grasp.
We went to a much simpler setup on Fly.io, and have been happy since. It's a shame they seem to be too young of a company to really be super reliable, but I suspect this is only a matter of time. In terms of feature set, it's all we need.
At my previous employer (~50 FTE of devs, 2-ish FTE dedicated to infra) Kubernetes worked perfectly fine, and I think in that context it made a lot more sense.
Meanwhile, going fly.io sounds sensible to me.
I can pretty confidently say, that's not K8s, that's Digital Ocean. On AWS, we ran the EKS infrastructure (which was not simple) with basically half a dev's time for years. It was only when it started to scale to millions of users that we needed to build a team to support it. It was still a much smaller team than the one that supported the ECS product (two devops).
I was mostly managing and not coding by the time Kubernetes was in our stack, so while I'm very familiar with infrastructure in general (and I know ECS inside and out unfortunately), I hadn't used Kubernetes directly much before I build this DO infrastructure. But I got it up in a week and though DO is a nightmare, k8s is an absolute joy as a DevOps. Holy shit it's perfect. It does exactly what it needs to, with exactly the right abstractions, with perfectly reasonable defaults.
The reality is that infrastructure work is just that complicated.
You wouldn't try to have a team of front end engineers build your rest backend. It's not reasonable to expect javascript engineers to know how to build and operate an infrastructure - at least not with out dedicating themselves to learning the tooling and space full time for a while. Think of it from the perspective of a frontend engineer learning Python and Django to build out a rest backend, and then multiply the complexity by 4. That's just infrastructure regardless of what you're using.
That said, if something like Fly.io can fit your needs, that's great! I haven't used them so I can't speak to them directly, but I know that with Heroku, the trade off was cost and, eventually, being limited in what you could build. Eventually you would need to build something that just couldn't be built with Heroku. A quick glance at Fly, the pricing looks reasonable, but I'm guessing the build limits will still apply.
> The reality is that infrastructure work is just that complicated.
Yes, if you need the flexibility of running anything in any setup. What we really wanted was 'yeet a docker image with a web server in it + env vars at some magic beast that'll run it for me, slap an SSL-cert on it, and make sure it's always online'. We tried to replicate this with Kubernetes, so we got the full complexity of k8s unloaded upon us.
Heroku was what we really wanted, but it was always too expensive. Fly.io strikes a good balance here, the defaults are sane, it's still flexible enough for other services, and it's relatively cheap (spend is similar to DO K8s).
> You wouldn't try to have a team of front end engineers build your rest backend.
Well, yes and no. I wouldn't expect frontend engineers to know the ins and outs of everything backend, but to build on your metaphore a bit further: Setting up a basic Node backend with express serving static files shouldn't take multiple weeks, even for a frontend engineer. I feel like I was trying to do the infra equivalent of that, and it did take me forever.
> A quick glance at Fly, the pricing looks reasonable, but I'm guessing the build limits will still apply
The build limits could be an issue but really isn't for us right now. It's fairly easy to build locally though (in our case: in our GitLab CI/CD runners)
Yeah, that's just what infrastructure work is. Like I said, take that analogy, multiply the complexity by 4 (at least... really maybe multiply it by an order of magnitude).
Let me put it in perspective. I've been coding since I was 12, I taught myself C to build a MUD in middle school and high school. I had about a decade of full stack professional experience in Java, PHP, javascript and I'd done infrastructure work with EC2 and chef before. When I moved into DevOps it was overwhelming.
I've been in DevOps for 4 years. I built that equivalent DO infrastructure with Kubernetes just last week (and in a week). I started the week going "Fuck, I don't know what I'm doing." The first 3 days were just spent reading documentation. Day four was spent writing the terraform and kubernetes manifests - with a distinct feeling that none of this was going to work because I was missing several key pieces. Day 5 was spent putting a few of those pieces in place and debugging. I finally got it working late Friday night. I took on a ton of tech debt and made a bunch of compromises just to get something working. I'm not the least bit happy with what I have working and intend to totally rebuild it on AWS when it comes time to build production.
And that's with 4 solid years of doing infrastructure work full time under my belt. For someone with no infrastructure experience? I would estimate 1 - 3 months. There's just way too much to learn to think you could do it quickly and simply.
With an express backend, if you have javascript experience, you really don't have much to learn. You need to learn how http interacts with the backend, how the backend interacts with the database, and databases (SQL). That's it. Learning database is not nothing, there's a lot that comes with it, but that's still only 2 new tools really.
With infrastructure, you need to learn networking, databases, securty, container orchestration (how does high availability work? Scaling?), bash, linux, provisioning, terraform, Docker, Kubernetes manifests, monitoring, secrets handling, and more. And for a lot of these things, the solutions are far from simple or perfect. Even when done as well as can be with modern tech it feels shakey and cobbled together at the end. You're tying a dozen different tool types together to solve a dozen different problems and you have dozens of choices for each tool type.
Like I said, infrastructure is just like that. And it's important to have the right expectations going in to it.
If you can't tell, I've had this conversation with my peers who stayed in full stack a lot.
I have spent the past couple weeks working with kustomize since I do not like helm and while it gets the job done I think Tanka would be better.
We are on GKE which makes things a lot easier and I personally would not choose to run my own cluster.
(disclaimer: i worked at hashi for 4 months in 2020 but not related to nomad)
Personal; I use docker-compose on VMs
We don't really have a use-case for Boundary but it looks pretty neat as well if you do.
Was on k8s for years and I don't miss it one bit.
While there definitely is some complexity once you get serious and set everything up properly with raft, federation, Connect, CAs, proxies, ACLs, proper secrets lifecycles... I find it's worth it. With the current assumptions that HC will keep improving and existing bugs and edge-cases will be ironed out.
creating a scalable system is complicated within aws account limits
all we really want is to shove docker containers behind a load balancer and not worry about having to manage yet another system
There's still use-cases where k8s wins; but nomad handles state a bit better and is easier to reason about from scratch.
1) I don't really want to manage the installation but there aren't any(?) cloud hosts for nomad that I can see. 2) It doesn't seem as widely used so community support seems thin. There aren't many blog posts about good patterns with it etc, and I'd worry that we'd get stuck and end up reverting back to k8s.
Point 2 is debatable. Lots of people nowadays put Kubernetes on their resume but that doesn't mean they are great architects or technicians, yet a good part of running production on Kubernetes is doing it right.
You'll see much fewer people with Nomad on their resume, but on the other hand you know they're not here for the buzz, they're usually more experienced and know what they're talking about.
We wrote about our decision to switch here: https://www.koyeb.com/blog/the-koyeb-serverless-engine-from-...
Nomad, Consul and Vault interoperate extremely well and are mostly pleasant to use, but I found myself missing the rest of the ecosystem pretty quickly, especially around ingresses, and I think they made the wrong decision on the networking model compared to Kubernetes.
That said, I haven't played with Consul Connect, the Consul+Envoy service mesh, yet. That might address a lot of the problems. But fundamentally I can't help but think that Nomad and Kubernetes both made a run of it and Kubernetes came out the winner of mindshare and ecosystem.
Can you elaborate on how Nomad handles state differently than K8S and what makes it better?