> We have built Cuber for scaling Pushpad. Cuber has been used in production for over a year and it is stable and reliable. We had 100% uptime and we saved 80% on cloud costs.
Pretty sure the 100% and 80% refer to using Cuber for Pushpad. So its probably more about using Kubernetes than about Cuber.
to be fair cloud is (on the whole) about 5x the price compared to bare metal. Depending on several factors which make it really hard to compare (apples to oranges and all that).
For my use-case, which does not require HA, is relatively B/W intensive etc; K8S on bare metal can be 11x cheaper (with the same amount of hands-off, click to deploy, no messing around) as GCP.
So, fundamentally, there can be enormous gains w.r.t price vs cloud, 80% cost reduction leaves an infinitesimally small margin though; if they really are providing all the same benefits as a public cloud.
This is not a bare-metal vs Cloud comparison, the Cuber PaaS is supposed to run on anything including cloud based on their docs. So when they say "save 80% on cloud costs" I am assuming they mean for eg. save 80% on using Cuber on GCP compared to deploying same workload on GKE.
It would be very absurd to claim something like - I was using i9-13900KS but realized I could run the same workload on my raspberry pi, but hey I also used this packaging tool in the process therefore I saved 80% costs because of the packaging tool.
If the generated configurations are suitable and the SRE hire is (as they should be) able to trace through how they're generated and audit the results, I don't see why said SRE would want to remove it unless/until it actively gets in the way.
When I first start working on pre-existing infrastructure, finding out what my users' current pain points are and figuring out how to work on them is always the priority, and removing a tool that's currently working, probably creating additional hopefully-transitory pain points in the process, seems like the opposite of a good place to start.
> Kubernetes is up to 80% cheaper compared to PaaS like Heroku and you can choose between different cloud providers (no lock-in).
I'm not a fan of these business case buzzword sentences. There is lock-in, although less, and "up to" is meaningless. It could be 81% cheaper, or it could be more expensive.
I can deploy an application on a machine running a specific Debian version. Go to some provider of bare metal or VMs, they usually have a "Get a Debian version soandso machine" button. Then I copy some files to a specific directory, then I run a setup.sh script that turns this specific version of Debian into a server running my application.
But how do I deploy "on Kubernetes"? Where do I get a button "Rent a Kubernetes"?
Kubernetes is an orchestrator that you control via an API, there is no file system to put things on and you don't run anything directly. You tell Kubernetes where to get your software from (as a container image) and how it should run it, and Kubernetes will then make sure the specified number of instances of your software is running with the settings you specify, on a suitable machine, with any storage volumes and other resources you request, or tell you it can't.
> But how do I deploy "on Kubernetes"?
Depends on the way it's set up. With vanilla Kubernetes you put container images in a registry that Kubernetes can access, write a manifest YAML and send that to the Kubernetes API (e.g. using the kubectl utility).
> Where do I get a button "Rent a Kubernetes"?
GCP, AWS, Azure, DigitalOcean, ... though they're labelled slightly differently.
Well, you could think about "deploying on Debian" as a combination of these things (but it's not limited to these things):
* Comply with general Linux rules for creating applications, i.e. either provide a file containing ELF executable with the appropriate metadata s.a. execution bit set etc. or similarly, a script file with a properly formatted shebang line.
* Comply with Debian package manager rules, i.e. create a properly formatted DEB file that can, in principle, be installed by "deb" package manager (i.e. doesn't create dependency problems, pre/post-install scripts execute etc.)
It's similar for Kubernetes deployment: except on top of creating a DEB package, you now also need to provide a "chart", that is a document (or a collection of documents) that describe in a formal way how the application is to be installed.
In a way very similar to how you can side-step creation of DEB package by, say, writing a Makefile and using that to install, you could also side-step creating a chart by, eg. writing a script that creates all the necessary Kubernetes entities. It's also not necessary that you create a chart in addition to DEB: it's possible that you use a Makefile-like approach for the part that would be better managed by DEB package, but still provide a chart for Kubernetes (it also has a package manager, however poorly designed and implemented: Helm).
---
In a way similar to how you could rent a bare-metal box or a VM, you can rent a Kubernetes cluster. Prices very a lot, and similar to the VM prices will greatly depend on the components you choose to put in your cluster.
This being said, running applications on Kubernetes is anything but cheap: Kubernetes adds a significant management layer overhead, but that pails on comparison to the typical way to deploy applications where there's rarely any form of cohesion between components (i.e. Linux container images that support individual components rarely have much of an overlap that can be automatically deduplicated, which means the application deployment ends up having tens or hundreds of copies of the same popular libraries, like libc or pthreads etc.) But it doesn't end there: Etcd, the Kuberenetes' database backend is hugely overloaded by all kinds of traffic sent to it. Most of that traffic is of no interest to the operator. So, your network also takes a noticeable performance hit from running Kubernetes in it (a typical solution is to dedicate an entirely separate physical network for Kubernetes' "control plane").
Of course, it's possible to come up with an equally bad if not worse solution in-house, if you want the same features provided by Kubernetes, but it's also possible and relatively easy to come up with a solution that will be less resource-hungry, while performing the same function.
Bottom line: claims about 80% cheaper are pulled out of... well, we'll not discuss where they pulled them out of.
> Kubernetes is up to 80% cheaper compared to PaaS like Heroku
What do the authors ever mean? How so? The smallest k8s cluster that makes sense has 3 nodes. This means you need to run at least 3 VMs and pay for everything else you need (storage, secrets, egress). And on some cloud providers like AWS, you pay an extra hourly fee just for the privilege of using k8s (that amounts to around $70 last time I checked). So the above statement makes no any sense.
I would guess "up to" means once these costs you mention are amortized. Sure k8s isn't going to beat a $10 dyno or however much they cost now, but I bet a cluster of 10 large VMs each running 50 containers blows the equivalent Heroku setup out of the water on costs.
K8s basically gets cheaper the more you run (as cloud costs, setup costs, engineering, bin-packing, volume discounts stack up), whereas Heroku basically scales linearly.
For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more. Monoliths get too huge and unwieldy if that many developers need to push changes on different release cadences.
The key thing is that a container needs about 10-20% of the memory of a virtual machine. Generally, a large cluster allows for very efficient "bin packing" so that workloads with different requirements all get distributed so that every VM is loaded to about 80-90%, whereas you'd be lucky to have 10% across your fleet of VMs. (Don't forget to include all the non-prod instances!)
At the tiniest scales, Kubernetes stops making sense because it's too complex to manage.
However, it does scale down to a single node. E.g.: Azure Kubernetes Service (AKS) will cheerfully run with just one Linux box and has no extra hidden costs.
> For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more
A few months back it was a headline statement on the kubernetes.io web site. Something like 'XX% of organisations with more than 1,000 developers use Kubernetes".
Its origins started in Google, which has thousands of developers, and it is roughly speaking a clone of their Borg system, which, again, was designed for that scale.
"XX% of organisations with more than 1,000 developers use Kubernetes" does not mean "most Kubernetes users work for organisations with more than 1,000 developers".
In my recent experience a ton of tiny startups which should be using Heroku etc are running Kubernetes for no other reason than engineer ego and "that's what the big boys are doing".
> For reference, the primary market for Kubernetes is orgs with over 1,000 developers that just can't make deployments to virtual machines work cost effectively any more.
Kubernetes is a surprisingly reasonable option for an organization with 5-10 developers that has outgrown Heroku-like services.
I don't think I've ever budgeted more than $150/month for a managed Kubernetes controller that Just Works, and that was a relatively expensive one. Everything was preconfigured and we could get started in 20 minutes. The actual servers in the cluster were just ordinary cloud VMs with locked-down images, and they were managed automatically.
Actually running an app on Kubernetes requires two things: (1) a Dockerfile, and (2) a YAML file listing the Kubernetes resources for the app. The YAML file might be anywhere from half a page to 2 pages for a simple one. There's an O'Reilly book that explains what to put in them.
I've also deployed large applications using Chef and Terraform. These often require far more configuration than slapping something on Kubernetes, and Chef offers many more ways for things to go wrong.
However, I do have several rules for Kubernetes happiness:
1. If your app is simple enough to run on a Heroku-like service, do that instead.
2. If you do choose Kubernetes, buy and skim the O'Reilly book, or a similar book.
3. Pay a cloud vendor for managed Kubernetes.
4. Do not mess with the networking config. Almost every Kubernetes horror story I've heard involves network overlays.
I'm not saying anyone ought to use Kubernetes, but it's surprisingly reasonable and pleasant for smallish full-time teams if you follow the rules above.
As a solopreneur using tiny k8s clusters, I agree wholeheartedly with #3 and #4 of your list. I don't think #2 is necessary and #1 is probably a good choice if you haven't already jumped onto the k8s bandwagon.
> At the tiniest scales, Kubernetes stops making sense because it's too complex to manage.
Managing k8s at the smallest scales is actually easier IMO. If you're using a managed k8s (you should be) then there's not much to actually manage. I've been using single-node clusters to great benefit for my business for years. Let's me focus all my time on billable hours and product updates. Huge force multiplier for me as a solopreneur.
This is interesting. I have only used k8s and k3s at the larger scales.
For small scale hosting who do you recommend? I have a few projects I want to get going and thinking it might help to be able to leverage my existing kubernetes knowledge. I always figured it would be too expensive to go with kubernetes for a small project.
KVM automatically overprovisions memory. Probably other hypervisors do it too.
The utilization numbers only show that a particular administrator doesn't know how to use technology X, but knows better how to use technology Y or that the defaults in technology Y happen to work better for a particular benchmark (and the administrator doesn't know how to use either X or Y).
I believe that the later is usually the case. Compounded by laziness and lack of real financial incentive to optimize every last bit of the existing infrastructure.
The situation will, however, flip on its head if you want to build a reliable high-performance system. In this case Kubernetes will be an impendiment. Nothing it does it does well or can be optimized / configured to do well. When push comes to shove, you'll have to either get rid of Kubernetes, or lose the race.
I’ve been using Civo for a while to get round some of the GCP costs (which are as bad as AWS for indie devs). So far no complaints and I like that it feels faster than GCP.
Idk what the author is referring to specifically but I can speak to my experience. I've been running a few single-node k8s clusters for a few years now. On GCP, the control plane fee is waived if you're running in a single region (no $70/mo fee). So basically you only pay for the VM itself.
As a solopreneur, I've saved so much time and money letting k8s run my workloads while I rack up billable hours and build products.
I'd argue ingress is even easier with Docker Compose. As for operators, it depends, but e.g. database operators are not really necessary on a single node cluster as you don't have discrepancies between nodes.
I like the fact you qualified ingress as "reasonably" painless.
Frankly it is not painless at all. Ingress is complicated and there is a ton of way to configure it, with all the forced tie in with the cloud provider's offering, be it l3 or l4 load balancing, various proxy services, etc.
It's not easy at all either on docker-compose. You have to roll your own proxy container and take care of TLS termination, cert renewals, manage specific headers and config, etc.
I am not sure what exactly I want to add to the conversation, only that either way, it's not "easy". I'd be glad to be proven wrong if you have a specific solution that makes ingress really painless.
You can run a single-node cluster for $5 on DigitalOcean, where a single Heroku prod node would cost $25. This price estimate doesn’t seem outlandish to me, but you do have to push it a bit to get to 80%.
If everyone else runs Kubernetes and no one uses docker-compose you might be looking at a ton of overhead, e.g. ITSEC may still expect you to run all of the stuff they offer a helm chart for. There may be no easy way to get the level of access on a VM to install docker-compose in the first place. Might well be worth running a single-node cluster just to be able to use an existing platform.
So my take is that if you're concerned with the expense of 3+ VMs, then Kubernetes isn't for you. We simply run the worker nodes on physical hardware, anything below that is a little weird for production environments, in my opinion.
Something like k3s makes a lot of sense for development environments though. It is much easier to create a production like environment with something like k3s than it ever was using VMs.
I do feel like there's room in the middle for something more flexible than Ansible and Puppet, but less complex than Kubernetes. If you're just running a few VMs, "upgrading" to Kubernetes doesn't seem like it's worth the hassle. On the other hand, deploying the same application, or applications to say 4 or 8 VMs, that starting to get i little annoying. I'd argue for a tool that augments Docker-Compose, one that understands distribution of applications, but doesn't deal with networking (the network team can handle that). I suppose a little like Nomad?
Argo is kind of antithetical to what this project is trying to solve. This project wants to be a Dockerfile like approach to Kubernetes. It focuses on simplicity. ArgoCD is decidedly not that, even if longer term something like that is the more correct way to manage production deployments.
I could see this project being picked up by some dev who just wants it on the cluster then an SRE ports it to something like Argo. Ideally the dev should learn Argo in the first place but a world where every dev understands what Argo is doing feels pretty unrealistic.
Side note (kind of off topic but I’ve been thinking about Argo a lot lately) I’ve been a bit more bearish on Argo for non production environments because it encourages pets and not cattle. I think nowadays the correct way to do environments is a gitops production cluster and dynamic pushed based ephemeral environments that are spawned from that production configuration and templates accordingly. Actually maybe it isn’t off topic: that’s what I’d like to see to offer to developers: some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo
that's a fair point, but Argocd + Packer is just that right ?
>some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo
Totally with you there, but you can build this on top of Argo by having your opinionated manifest language (like Packer or Cuberfile or something else). But there is no joy to maintaining the Kubernetes api interfaces.
you could achieve this with packer and some custom tooling yeah. But just saying “and some custom tooling” is like saying “you could do what Dropbox does with a custom rsync script”. There’s no off the shelf tooling to do it. One does not simply “build this on top of Argo by having your opinionated manifest language”.
There’s probably 10 tools I could customize to do this, packer being one of them, but no real patterns I’m aware of where people are actually doing it with argocd or fluxcd config as a base. Usually you’re either all in or all out on gitops, so I would be interested in seeing blog posts or libraries of people who have done hybrid approaches (push and pull deployments) and the trade offs they had to make. There is a decent amount of decisions that have to be made to build an abstraction like that, so anyone who has done it before I’m happy to stand on the shoulders of. It’s been on my list of unsolved devops/infra problems to unwrap for a couple of years now
In the end there are really three steps that need to happen for true “copy prod” ephemeral environments. You need to provision some stack of infra and then deploy some stuff on top of it and then pull some arbitrary, extremely customizable state into that deployment. I’ve been quite interested in Layerform recently for the first part of that, I think they are getting close to the right abstraction on the infra side. But then you still have to deploy (Argo, helm, flux, something) into that infra stack. Then you have to get the state into that deployment somehow. there is no answer for the latter two from Layerform (terraform is not the right tool for deploys or state backups once you do anything beyond a hello world app). Getting the state in has its own whole Pandora’s box (can I copy customer data from prod into another environment? How do I do so on a way that doesn’t violate SOC2/hipaa/whatever? How will my system’s data and configurations affect the third party systems it connects to?) that I think is simply just not generalizable. But the first two steps should be able to be generalized, and the last step is something that tools can be made available to make it easier with the knowledge that it’s basically business logic at that point
well my intention was not to trivialize the effort. I think i was trying to ask an architecture question:
if you were to build something like Cuber...would you build it over Argocd ? or build it entirely from the ground up.
It would still take equal amount of effort probably, but you get the benefit of a larger ecosystem and more work on the underlying infra, while you can focus entirely on the abstractions.
How so? From the rest of your comment it sounds like you may be saying that target environments are the pets, but ArgoCD makes no assumption about that. It will deploy to any cluster you configure it to deploy to, and spinning up a cluster is a logically (very!) separate operation from deploying an app. Argo only deals with the latter.
If you want something to actually provision an ephemeral environment for you, ArgoCD is not the tool. It's more focused than that, and that's its strength. All it does is deploy k8s manifests out of git to whatever cluster you tell it to deploy them to. That cluster could be ephemeral or not, it doesn't matter to Argo.
Of course you could use argo to do what I'm describing, but the overhead of checking in code is 1. clunky for dynamic environments (it would require constantly committing to some repo) and 2. not well suited for provisioning dynamically from a local machine.
So by virtue of the way argo works it encourages pets, because its setup is not very dynamic. Like you said, in environments when you want to be able to roll back, that is its strength. But my point I was making above is that really you want that for only one environment: prod when it all boils down to it. Every other environment not being a dynamic branch off of prod just creates maintenance and state issues down the line
It's not just prod, it's any shared environment that needs to live for some time. An example is an integration environment, especially if it's shared by multiple teams.
A "pet" is a machine or environment that you can't easily delete and replace. Nothing we're discussing can't easily be deleted and replaced. What you're describing relates to some other property.
As for "dynamic", with Argo you can just edit the namespaces of Applications in a branch and deploy a new environment.
> it would require constantly committing to some repo
If you have an automated CI/CD pipeline, you already do this anyway.
It's not clear to me what kind of operation you're thinking of, but in the same sort of way that k8s is overkill for many scenarios, so is ArgoCD. But the issue is not that it encourages pets or can't easily handle dynamic environments.
Just chiming in here to say how great of a product ArgoCD is. We've been using it for a few years now and it makes for a highly configurable deployment management layer on top of our clusters. Highly recommend it for managing multiple application deployments where things like RBAC, observability, and configurability matter. Also doesn't hurt that it has a decent UX (although we primarily interface via automatons these days). The project maintainers are also extremely responsive and professional (thanks @alexmt!).
All that being said, Cuber definitely feels like a different sort of market fit.
>All that being said, Cuber definitely feels like a different sort of market fit.
but it could have been built on top of Argo right ?
I totally get an opinionated out-of-the-box workflow and manifest file. But i would rather see that on top of the Argo infra (with its millions of person hours of kuberentes fighting).
ArgoCD is meh. Its a UI over your k8s clusters. What happened to us is sre/devops types just use k9s or the console. The devs could not be bothered to log into argo. They just merge, and code gets deployed. They don't hang out on argo. With 5k+ pods, the UI doesn't even work. So it isn't designed for large systems which makes it further weird.
Not really. It's a deployment tool, the UI is there to help see what's happening with deployments. It's not intended to be a primary interface to cluster management the way k9s or Lens etc. is.
> They just merge, and code gets deployed.
How does "code get deployed" though? There has to be something that's actually (1) monitoring branches and (2) running the equivalent of `kubectl apply` to deploy manifests. That's what ArgoCD does.
If you're saying devs are using k9s or kubectl to manually trigger the apply from a branch, that has all sorts of drawbacks, which is exactly what Argo is designed to address.
k8s is, and I wrote it countless times, a huge waste of resources.
The whole event driven distributed design is.
There aren't many use cases that require more than 5k concurrent clients and the number is even higher without the HTTP overhead (a single bare metal server can handle with the proper language).
For most companies it's a waste of money.
k8s is hype tech designed to drive business to public cloud providers, because of its complexity and high hardware demands.
9 bare metal servers minimum.
And then you have those pods, which are collections of services (e.g. postgres + api server + some cache).
But in the case of postgres, why wouldn't you have a dedicated cluster instead of multiple, isolated instances as it is the case with k8s?
I know, now the EDA faction chimes in with their dogma. Again most cases don't require even 5k concurrent clients. And most projects never will grow to the point of that many concurrent clients.
Addendum: And then I read the who is hiring thread and it's all k8s k8s k8s... frustrating
k8s is a complex solution for complex problems. You don't need k8s to host landing or personal blog type of service.
Once you move to a type of project with complex demands around deployments, log management, failover management, secrets, environments, resource isolation, etc., then building something to solve this set of problems over and over again is very expensive when off-the-shelf battle-tested solution is available.
The thing is, k8s has a lot of mindshare. It's entirely easier to go and find some pertinent issue with your k8s set-up through a simple search, whereas e.g. nomad and swarm have much less mindshare.
You don't have to go full complexity with k8s. You can just use a managed k8s provider and ship a simple deployment with an anti-affinity policy to distribute across VMs on creation. Sure, there are other solutions [and this one is likely to not be the cheapest], but they're all vendor locked--k8s is about as close to a cloud agnostic compute as we're likely to get.
I enjoy docker swarm for small hobby projects. It's simple and easy to maintain if you want to benefit from container apps and configuration.
Once your service grows, you'll notice some rough edges around the docker swarm. When you start having dedicated SRE/DevOps roles, moving from heroku/swarm to k8s will make sense.
> And then you have those pods, which are collections of services (e.g. postgres + api server + some cache).
Pods are not intended to bundle those 3 services. Typically one would use higher level abstractions (Deployment for the API, StatefulSet for the database), each composed of multiple Pods that should be scheduled on different physical machines.
It’s no wonder you’ve had a bad experience trying to use it the way you describe. Following the intended patterns provides a standard way to achieve rolling updates, horizontal autoscaling, and a ton of other functionality that used to be bespoke code at each organization. The old way was often buggy, and onboarding new engineers took much longer.
Maybe their terminology was wrong, but what about this point:
> But in the case of postgres, why wouldn't you have a dedicated cluster
Does k8s not push you to mix postgres pods in the same host as your API pods? Unless you go out of your way to handicap it.
I understand the appeal of applying the k8s (or swarm, or nomad) model to stateless API containers, but mixing in stateful db containers and pretending they're all equivalent makes less sense to me.
Many applications don't need multiple physical machines worth of database.
Persistent storage has been a solved problem, especially when using a managed Kubernetes service. What's the problem with running stateful workloads on a cluster?
My intent wasn’t to criticize you, but apparently I stuck a nerve…
Of course you can have all that without k8s. I’ve been operating cloud infra for over a decade, and I used to roll my own solutions to implement that functionality, so I’m quite familiar with that approach. Embracing kubernetes has made life much easier. I do read a lot, but my opinion is based on operating production systems. Whether you like it or not, k8s has become the defacto standard interface for distributed systems.
High level programming languages and libraries provide useful layers of abstraction in their own domains. Do you also reject those, and use assembly language for everything?
This kind of defeats the purpose of k8s which is to run arbitrary software on a flexible stack. If you need a separate packaging tool you might as well compile a native package.
Just seems like another abstraction layer to me. With its own DSL, which will have bugs and limitations. And of course buildpacks - yet another abstraction thats simple until you need to build and maintain your own because none fit quite right. I don't know, not sure this solves it.
K3s is fine, but it was started on an older version of kubernetes, so there's a few places where K8s has solved an issue (like bootstrapping) that k3s had to solve already in a different way.
K0s is a new project I like that is similar to k3s in spirit. Microk8s is a snap version from canonical with some neat out of the box stuff (like the simplest GPU support); but again, only available as a snap.
Honestly though just use kubespray for production, it's fine
Kubespray is fine if you can spare hours of Dev time. Where I used to work I wasn't involved directly with maintaining the k8s cluster using kubespray, but from the daily meetings it was clear that it is no easy task.
Sure it helps with deploying a serviceable production cluster, but when you start throwing at it things like disaster recovery, the automation scripts can grow quite large and complex, even leveraging kubespray.
I was going to say. Using a managed service is specialized enough. I can't imagine building one up with everything from scratch. Feels like a full time role "Kubernetes Sys Admin"
kubespray is not good advice unless it's changed massively since I last used it in 2020 or so.
Imagine a 3 hour Ansible-playbook execution, which randomly failed on different tasks, most of which got fixed by hitting <up><enter> until it applied...
We switched to RKE, which runs even on our bigger clusters (nothing mad, maybe 40 nodes) in a few minutes and works perfectly..
k0s is better but also has a lot of bugs. it's the closest to vanilla kubernetes among all the distributions.
> like the simplest GPU support
linux users should be ready to install the nvidia device plugin. if they can't do that, they're never going to succeed in running a gpu accelerated application on their cluster anyway.
> like bootstrapping
in my experience, writing all the bootstrap scripts is painful. but now that there's chatgpt, so much of the drudgery as gone away.
K3s is the best out of any of these "lite" k8 frameworks. As someone who has never worked in tech, it took me about five hours to setup k3s with cert manager with a few PHP and nodejs deployments. This is using my own helm setup making it trivial to upgrade.
If I understand correctly, Cuber is meant to fit between "I run a simple Digital Ocean droplet with my code" and "I have people managing my kubernetes cluster and deployments". I am a freelance software developer running sizeable client applications (low millions requests per month, 300 ~ 500 domains, fully custom made) in my own kubernetes cluster along with my own (side)projects so I feel that I fit that bill somewhat.
My first impression is that it looks quite awesome! If this is able to remove a big build process of containers then that would be awesome. I don't want to build my docker containers locally, I don't want to 'manage' a docker registry, I don't want to build my containers in my cluster (e.g. Kaniko). However, everything says it will push to the Docker registry and I haven't been able to find any private registry or non-registry option. That would be quite a dealbreaker because I don't want to 'store' my images long term, I will only use them for the few moments where the pods are deployed with a new image and the few days/weeks where they might restart. But retention is not a huge requirement.
i stopped reading the spec, after i have seen that the only label they attach is the lb. anyone who really uses k8s will not at this product longer than it takes, reading the readme or the first bit of code.
92 comments
[ 0.21 ms ] story [ 73.6 ms ] threadIs there a source on such bold claims?
Pretty sure the 100% and 80% refer to using Cuber for Pushpad. So its probably more about using Kubernetes than about Cuber.
For my use-case, which does not require HA, is relatively B/W intensive etc; K8S on bare metal can be 11x cheaper (with the same amount of hands-off, click to deploy, no messing around) as GCP.
So, fundamentally, there can be enormous gains w.r.t price vs cloud, 80% cost reduction leaves an infinitesimally small margin though; if they really are providing all the same benefits as a public cloud.
It would be very absurd to claim something like - I was using i9-13900KS but realized I could run the same workload on my raspberry pi, but hey I also used this packaging tool in the process therefore I saved 80% costs because of the packaging tool.
If the generated configurations are suitable and the SRE hire is (as they should be) able to trace through how they're generated and audit the results, I don't see why said SRE would want to remove it unless/until it actively gets in the way.
When I first start working on pre-existing infrastructure, finding out what my users' current pain points are and figuring out how to work on them is always the priority, and removing a tool that's currently working, probably creating additional hopefully-transitory pain points in the process, seems like the opposite of a good place to start.
I'm not a fan of these business case buzzword sentences. There is lock-in, although less, and "up to" is meaningless. It could be 81% cheaper, or it could be more expensive.
I think "up to 80% cheaper" implies "No more than 80% cheaper".
I can deploy an application on a machine running a specific Debian version. Go to some provider of bare metal or VMs, they usually have a "Get a Debian version soandso machine" button. Then I copy some files to a specific directory, then I run a setup.sh script that turns this specific version of Debian into a server running my application.
But how do I deploy "on Kubernetes"? Where do I get a button "Rent a Kubernetes"?
What kind of a question is this?
It means to deploy "on Kubernetes".
It doesn't mean to run a script on a server.
It means you need a Kubernetes cluster where you have deploy permissions. Then you can deploy on Kubernetes.
> Where do I get a button "Rent a Kubernetes"?
https://cloud.google.com/kubernetes-engine/ https://www.digitalocean.com/products/kubernetes https://azure.microsoft.com/en-us/products/kubernetes-servic... https://aws.amazon.com/eks/
(In no particular order)
Kubernetes is an orchestrator that you control via an API, there is no file system to put things on and you don't run anything directly. You tell Kubernetes where to get your software from (as a container image) and how it should run it, and Kubernetes will then make sure the specified number of instances of your software is running with the settings you specify, on a suitable machine, with any storage volumes and other resources you request, or tell you it can't.
> But how do I deploy "on Kubernetes"?
Depends on the way it's set up. With vanilla Kubernetes you put container images in a registry that Kubernetes can access, write a manifest YAML and send that to the Kubernetes API (e.g. using the kubectl utility).
> Where do I get a button "Rent a Kubernetes"?
GCP, AWS, Azure, DigitalOcean, ... though they're labelled slightly differently.
* Comply with general Linux rules for creating applications, i.e. either provide a file containing ELF executable with the appropriate metadata s.a. execution bit set etc. or similarly, a script file with a properly formatted shebang line.
* Comply with Debian package manager rules, i.e. create a properly formatted DEB file that can, in principle, be installed by "deb" package manager (i.e. doesn't create dependency problems, pre/post-install scripts execute etc.)
It's similar for Kubernetes deployment: except on top of creating a DEB package, you now also need to provide a "chart", that is a document (or a collection of documents) that describe in a formal way how the application is to be installed.
In a way very similar to how you can side-step creation of DEB package by, say, writing a Makefile and using that to install, you could also side-step creating a chart by, eg. writing a script that creates all the necessary Kubernetes entities. It's also not necessary that you create a chart in addition to DEB: it's possible that you use a Makefile-like approach for the part that would be better managed by DEB package, but still provide a chart for Kubernetes (it also has a package manager, however poorly designed and implemented: Helm).
---
In a way similar to how you could rent a bare-metal box or a VM, you can rent a Kubernetes cluster. Prices very a lot, and similar to the VM prices will greatly depend on the components you choose to put in your cluster.
This being said, running applications on Kubernetes is anything but cheap: Kubernetes adds a significant management layer overhead, but that pails on comparison to the typical way to deploy applications where there's rarely any form of cohesion between components (i.e. Linux container images that support individual components rarely have much of an overlap that can be automatically deduplicated, which means the application deployment ends up having tens or hundreds of copies of the same popular libraries, like libc or pthreads etc.) But it doesn't end there: Etcd, the Kuberenetes' database backend is hugely overloaded by all kinds of traffic sent to it. Most of that traffic is of no interest to the operator. So, your network also takes a noticeable performance hit from running Kubernetes in it (a typical solution is to dedicate an entirely separate physical network for Kubernetes' "control plane").
Of course, it's possible to come up with an equally bad if not worse solution in-house, if you want the same features provided by Kubernetes, but it's also possible and relatively easy to come up with a solution that will be less resource-hungry, while performing the same function.
Bottom line: claims about 80% cheaper are pulled out of... well, we'll not discuss where they pulled them out of.
It is one line of code.
kubectl apply -f k8s/deployment.yaml
Think of Kubernetes as an operating system for a cluster of computers.
You can rent managed Kubernetes from any cloud provider.
What do the authors ever mean? How so? The smallest k8s cluster that makes sense has 3 nodes. This means you need to run at least 3 VMs and pay for everything else you need (storage, secrets, egress). And on some cloud providers like AWS, you pay an extra hourly fee just for the privilege of using k8s (that amounts to around $70 last time I checked). So the above statement makes no any sense.
K8s basically gets cheaper the more you run (as cloud costs, setup costs, engineering, bin-packing, volume discounts stack up), whereas Heroku basically scales linearly.
The key thing is that a container needs about 10-20% of the memory of a virtual machine. Generally, a large cluster allows for very efficient "bin packing" so that workloads with different requirements all get distributed so that every VM is loaded to about 80-90%, whereas you'd be lucky to have 10% across your fleet of VMs. (Don't forget to include all the non-prod instances!)
At the tiniest scales, Kubernetes stops making sense because it's too complex to manage.
However, it does scale down to a single node. E.g.: Azure Kubernetes Service (AKS) will cheerfully run with just one Linux box and has no extra hidden costs.
Is there a source for this statement?
Its origins started in Google, which has thousands of developers, and it is roughly speaking a clone of their Borg system, which, again, was designed for that scale.
In my recent experience a ton of tiny startups which should be using Heroku etc are running Kubernetes for no other reason than engineer ego and "that's what the big boys are doing".
And you say Heroku, "etc", but the thing is that there is very little heroku-like alternative. What "etc" is there really?
The real alternative is a Linux VM with docker-compose.
Once you outgrow that, managed kubernetes is the logical next step, even for a small team.
Kubernetes is a surprisingly reasonable option for an organization with 5-10 developers that has outgrown Heroku-like services.
I don't think I've ever budgeted more than $150/month for a managed Kubernetes controller that Just Works, and that was a relatively expensive one. Everything was preconfigured and we could get started in 20 minutes. The actual servers in the cluster were just ordinary cloud VMs with locked-down images, and they were managed automatically.
Actually running an app on Kubernetes requires two things: (1) a Dockerfile, and (2) a YAML file listing the Kubernetes resources for the app. The YAML file might be anywhere from half a page to 2 pages for a simple one. There's an O'Reilly book that explains what to put in them.
I've also deployed large applications using Chef and Terraform. These often require far more configuration than slapping something on Kubernetes, and Chef offers many more ways for things to go wrong.
However, I do have several rules for Kubernetes happiness:
1. If your app is simple enough to run on a Heroku-like service, do that instead.
2. If you do choose Kubernetes, buy and skim the O'Reilly book, or a similar book.
3. Pay a cloud vendor for managed Kubernetes.
4. Do not mess with the networking config. Almost every Kubernetes horror story I've heard involves network overlays.
I'm not saying anyone ought to use Kubernetes, but it's surprisingly reasonable and pleasant for smallish full-time teams if you follow the rules above.
Managing k8s at the smallest scales is actually easier IMO. If you're using a managed k8s (you should be) then there's not much to actually manage. I've been using single-node clusters to great benefit for my business for years. Let's me focus all my time on billable hours and product updates. Huge force multiplier for me as a solopreneur.
For small scale hosting who do you recommend? I have a few projects I want to get going and thinking it might help to be able to leverage my existing kubernetes knowledge. I always figured it would be too expensive to go with kubernetes for a small project.
The utilization numbers only show that a particular administrator doesn't know how to use technology X, but knows better how to use technology Y or that the defaults in technology Y happen to work better for a particular benchmark (and the administrator doesn't know how to use either X or Y).
I believe that the later is usually the case. Compounded by laziness and lack of real financial incentive to optimize every last bit of the existing infrastructure.
The situation will, however, flip on its head if you want to build a reliable high-performance system. In this case Kubernetes will be an impendiment. Nothing it does it does well or can be optimized / configured to do well. When push comes to shove, you'll have to either get rid of Kubernetes, or lose the race.
As a solopreneur, I've saved so much time and money letting k8s run my workloads while I rack up billable hours and build products.
Frankly it is not painless at all. Ingress is complicated and there is a ton of way to configure it, with all the forced tie in with the cloud provider's offering, be it l3 or l4 load balancing, various proxy services, etc.
It's not easy at all either on docker-compose. You have to roll your own proxy container and take care of TLS termination, cert renewals, manage specific headers and config, etc.
I am not sure what exactly I want to add to the conversation, only that either way, it's not "easy". I'd be glad to be proven wrong if you have a specific solution that makes ingress really painless.
Something like k3s makes a lot of sense for development environments though. It is much easier to create a production like environment with something like k3s than it ever was using VMs.
I do feel like there's room in the middle for something more flexible than Ansible and Puppet, but less complex than Kubernetes. If you're just running a few VMs, "upgrading" to Kubernetes doesn't seem like it's worth the hassle. On the other hand, deploying the same application, or applications to say 4 or 8 VMs, that starting to get i little annoying. I'd argue for a tool that augments Docker-Compose, one that understands distribution of applications, but doesn't deal with networking (the network team can handle that). I suppose a little like Nomad?
Plus has clean github integration for incremental deploys, etc
I could see this project being picked up by some dev who just wants it on the cluster then an SRE ports it to something like Argo. Ideally the dev should learn Argo in the first place but a world where every dev understands what Argo is doing feels pretty unrealistic.
Side note (kind of off topic but I’ve been thinking about Argo a lot lately) I’ve been a bit more bearish on Argo for non production environments because it encourages pets and not cattle. I think nowadays the correct way to do environments is a gitops production cluster and dynamic pushed based ephemeral environments that are spawned from that production configuration and templates accordingly. Actually maybe it isn’t off topic: that’s what I’d like to see to offer to developers: some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo
>some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo
Totally with you there, but you can build this on top of Argo by having your opinionated manifest language (like Packer or Cuberfile or something else). But there is no joy to maintaining the Kubernetes api interfaces.
There’s probably 10 tools I could customize to do this, packer being one of them, but no real patterns I’m aware of where people are actually doing it with argocd or fluxcd config as a base. Usually you’re either all in or all out on gitops, so I would be interested in seeing blog posts or libraries of people who have done hybrid approaches (push and pull deployments) and the trade offs they had to make. There is a decent amount of decisions that have to be made to build an abstraction like that, so anyone who has done it before I’m happy to stand on the shoulders of. It’s been on my list of unsolved devops/infra problems to unwrap for a couple of years now
In the end there are really three steps that need to happen for true “copy prod” ephemeral environments. You need to provision some stack of infra and then deploy some stuff on top of it and then pull some arbitrary, extremely customizable state into that deployment. I’ve been quite interested in Layerform recently for the first part of that, I think they are getting close to the right abstraction on the infra side. But then you still have to deploy (Argo, helm, flux, something) into that infra stack. Then you have to get the state into that deployment somehow. there is no answer for the latter two from Layerform (terraform is not the right tool for deploys or state backups once you do anything beyond a hello world app). Getting the state in has its own whole Pandora’s box (can I copy customer data from prod into another environment? How do I do so on a way that doesn’t violate SOC2/hipaa/whatever? How will my system’s data and configurations affect the third party systems it connects to?) that I think is simply just not generalizable. But the first two steps should be able to be generalized, and the last step is something that tools can be made available to make it easier with the knowledge that it’s basically business logic at that point
It would still take equal amount of effort probably, but you get the benefit of a larger ecosystem and more work on the underlying infra, while you can focus entirely on the abstractions.
How so? From the rest of your comment it sounds like you may be saying that target environments are the pets, but ArgoCD makes no assumption about that. It will deploy to any cluster you configure it to deploy to, and spinning up a cluster is a logically (very!) separate operation from deploying an app. Argo only deals with the latter.
If you want something to actually provision an ephemeral environment for you, ArgoCD is not the tool. It's more focused than that, and that's its strength. All it does is deploy k8s manifests out of git to whatever cluster you tell it to deploy them to. That cluster could be ephemeral or not, it doesn't matter to Argo.
So by virtue of the way argo works it encourages pets, because its setup is not very dynamic. Like you said, in environments when you want to be able to roll back, that is its strength. But my point I was making above is that really you want that for only one environment: prod when it all boils down to it. Every other environment not being a dynamic branch off of prod just creates maintenance and state issues down the line
A "pet" is a machine or environment that you can't easily delete and replace. Nothing we're discussing can't easily be deleted and replaced. What you're describing relates to some other property.
As for "dynamic", with Argo you can just edit the namespaces of Applications in a branch and deploy a new environment.
> it would require constantly committing to some repo
If you have an automated CI/CD pipeline, you already do this anyway.
It's not clear to me what kind of operation you're thinking of, but in the same sort of way that k8s is overkill for many scenarios, so is ArgoCD. But the issue is not that it encourages pets or can't easily handle dynamic environments.
All that being said, Cuber definitely feels like a different sort of market fit.
but it could have been built on top of Argo right ?
I totally get an opinionated out-of-the-box workflow and manifest file. But i would rather see that on top of the Argo infra (with its millions of person hours of kuberentes fighting).
Not really. It's a deployment tool, the UI is there to help see what's happening with deployments. It's not intended to be a primary interface to cluster management the way k9s or Lens etc. is.
> They just merge, and code gets deployed.
How does "code get deployed" though? There has to be something that's actually (1) monitoring branches and (2) running the equivalent of `kubectl apply` to deploy manifests. That's what ArgoCD does.
If you're saying devs are using k9s or kubectl to manually trigger the apply from a branch, that has all sorts of drawbacks, which is exactly what Argo is designed to address.
The whole event driven distributed design is.
There aren't many use cases that require more than 5k concurrent clients and the number is even higher without the HTTP overhead (a single bare metal server can handle with the proper language).
For most companies it's a waste of money.
k8s is hype tech designed to drive business to public cloud providers, because of its complexity and high hardware demands. 9 bare metal servers minimum.
And then you have those pods, which are collections of services (e.g. postgres + api server + some cache). But in the case of postgres, why wouldn't you have a dedicated cluster instead of multiple, isolated instances as it is the case with k8s?
I know, now the EDA faction chimes in with their dogma. Again most cases don't require even 5k concurrent clients. And most projects never will grow to the point of that many concurrent clients.
Addendum: And then I read the who is hiring thread and it's all k8s k8s k8s... frustrating
Once you move to a type of project with complex demands around deployments, log management, failover management, secrets, environments, resource isolation, etc., then building something to solve this set of problems over and over again is very expensive when off-the-shelf battle-tested solution is available.
You don't have to go full complexity with k8s. You can just use a managed k8s provider and ship a simple deployment with an anti-affinity policy to distribute across VMs on creation. Sure, there are other solutions [and this one is likely to not be the cheapest], but they're all vendor locked--k8s is about as close to a cloud agnostic compute as we're likely to get.
Once your service grows, you'll notice some rough edges around the docker swarm. When you start having dedicated SRE/DevOps roles, moving from heroku/swarm to k8s will make sense.
You can do more on a single server than hosting "landing pages" or "personal blog".
I host seemingly complex projects with simple methods, without needing any complex k8s crap.
That's what you people aren't getting.
You're so full into the complexity that you don't see the simple solution as something worthwhile.
WHAT about k8s is more battle tested than decades of doing the simple and efficient on a 1-3 servers?
You're brainwashed. Possibly even a shill of one of those expensive cloud providers.
Pods are not intended to bundle those 3 services. Typically one would use higher level abstractions (Deployment for the API, StatefulSet for the database), each composed of multiple Pods that should be scheduled on different physical machines.
It’s no wonder you’ve had a bad experience trying to use it the way you describe. Following the intended patterns provides a standard way to achieve rolling updates, horizontal autoscaling, and a ton of other functionality that used to be bespoke code at each organization. The old way was often buggy, and onboarding new engineers took much longer.
> But in the case of postgres, why wouldn't you have a dedicated cluster
Does k8s not push you to mix postgres pods in the same host as your API pods? Unless you go out of your way to handicap it.
I understand the appeal of applying the k8s (or swarm, or nomad) model to stateless API containers, but mixing in stateful db containers and pretending they're all equivalent makes less sense to me.
Persistent storage has been a solved problem, especially when using a managed Kubernetes service. What's the problem with running stateful workloads on a cluster?
You can have all that without k8s. k8s doesn't equal auto scaling, "rolling updates".
You are a booksmart idiot.
This guy, seriously.
Completely ignores everything I wrote and starts bullshitting.
Of course you can have all that without k8s. I’ve been operating cloud infra for over a decade, and I used to roll my own solutions to implement that functionality, so I’m quite familiar with that approach. Embracing kubernetes has made life much easier. I do read a lot, but my opinion is based on operating production systems. Whether you like it or not, k8s has become the defacto standard interface for distributed systems.
High level programming languages and libraries provide useful layers of abstraction in their own domains. Do you also reject those, and use assembly language for everything?
Seems more complex than a yaml manifest - Is there a Kubernetes operator available for these Cuberfiles
K0s is a new project I like that is similar to k3s in spirit. Microk8s is a snap version from canonical with some neat out of the box stuff (like the simplest GPU support); but again, only available as a snap.
Honestly though just use kubespray for production, it's fine
Sure it helps with deploying a serviceable production cluster, but when you start throwing at it things like disaster recovery, the automation scripts can grow quite large and complex, even leveraging kubespray.
If you can, use a managed service like gks.
Imagine a 3 hour Ansible-playbook execution, which randomly failed on different tasks, most of which got fixed by hitting <up><enter> until it applied...
We switched to RKE, which runs even on our bigger clusters (nothing mad, maybe 40 nodes) in a few minutes and works perfectly..
We had much better luck just scripting kubeadm over ssh orchestration tooling.
k0s is better but also has a lot of bugs. it's the closest to vanilla kubernetes among all the distributions.
> like the simplest GPU support
linux users should be ready to install the nvidia device plugin. if they can't do that, they're never going to succeed in running a gpu accelerated application on their cluster anyway.
> like bootstrapping
in my experience, writing all the bootstrap scripts is painful. but now that there's chatgpt, so much of the drudgery as gone away.
My first impression is that it looks quite awesome! If this is able to remove a big build process of containers then that would be awesome. I don't want to build my docker containers locally, I don't want to 'manage' a docker registry, I don't want to build my containers in my cluster (e.g. Kaniko). However, everything says it will push to the Docker registry and I haven't been able to find any private registry or non-registry option. That would be quite a dealbreaker because I don't want to 'store' my images long term, I will only use them for the few moments where the pods are deployed with a new image and the few days/weeks where they might restart. But retention is not a huge requirement.
Just my 2c.
i stopped reading the spec, after i have seen that the only label they attach is the lb. anyone who really uses k8s will not at this product longer than it takes, reading the readme or the first bit of code.