Show HN: Glasskube – Open Source Kubernetes Package Manager, alternative to Helm (github.com)
Most developers working with Kubernetes use Helm, an open-source tool created during a hackathon nine years ago. However, with the rapid growth of Kubernetes packages to over 800 packages on the CNCF landscape today, the prerequisites have changed, and we believe it’s time for a new package manager. Every engineer we talked to has a love-hate relationship with Helm, and we also found ourselves defaulting to Helm despite its shortcomings due to a lack of alternatives.
We have spent enough time trying to get Helm to do what we need. From looking for the correct chart, trying to learn how each value affects the components and hand-crafting a schemaless values.yaml file, to debugging the final release if it inevitably fails to install, the experience of using Helm is, for the most part, time consuming and cumbersome.
Charts often become more complex, requiring the use of sub-charts. These umbrella charts tend to be even harder to maintain and upgrade, because so many different components are bundled into a single release.
We talked to over 100 developers and found that everyone developed their own little workarounds, with some working better than others. We collected the feedback poured everything we learned from that into a new package manager. We want to build something that is as easy to use as Homebrew or npm and make package management on Kubernetes as easy as on every other platform.
Some of the features Glasskube already supports are
Typesafe package configuration via UI or interactive CLI to inject values from other packages, ConfigMaps, and Secrets.
Browse our central package repository so there is no need to look for a Helm repository to find a specific package.
All packages are dependency-aware so they can be used and referenced by multiple other packages even across namespaces. We validate the complete dependency tree - So packages get installed in the correct namespace.
Preview and perform pending updates to your desired version with a single click of a button. All updates have been tested in the Glasskube test suite before being available in the public repository.
Use multiple repositories and publish your own private packages (e.g., your company's internal services packages, so all developers will have the up-to-date and easily configured internal services).
All features are available via UI or interactive CLI. You can also manage all packages via GitOps.
Currently, we are focused on enhancing the user experience, aiming to save engineers as much time as possible. We are still using Helm and Manifests under the hood. However, together with the community, we plan to develop an entirely new packaging and bundling format for all cloud-native packages. This will provide package developers with a straightforward way to define how to install and configure packages, offer simple upgrade paths, and enable us to provide feedback, crash reports, and analytics to every developer working on Kubernetes packages.
We also started working on a cloud version. You can pre-signup here in case you are interested: https://glasskube.cloud
We'd greatly appreciate any feedback you have and hope you get the chance to try out Glasskube.
126 comments
[ 5.5 ms ] story [ 194 ms ] threadYes at the moment Glasskube packages wrap around Helm charts and manifests. We don't have a dedicated packaging format as of now.
We are actively looking into OCI images and other possibilities to bundle Kubernetes packages, but focus on features like multi namespace dependencies and simplicity at the moment.
How would you like to package Kubernetes packages or what should be avoided from your perspective?
Helm, as an abstraction layer, is a real pain in the ass. Having another abstraction layer here is pure madness.
I wish you luck, but I do not wish to board your boat.
As a power Kubernetes user providing Kubernetes based paas to internal customers, we are not looking for more GUI or, abstractions over helm.
There are already a lot of solutions out there in this area like helmsman/argocd/helmfile/cross plane helm-provider. And we like kubernetes resources + gitops based automations over any other fancy tools.
Most of the time problems around helm is it's string based templating and lack of type safety. That's why timoni looks a more promising solution in this space. It's lack of packages is the limiting factor.
Another interesting approach is kapp-controller and carvel tooling. Packaging helm charts, OCI images, etc as OCI artifacts to use as a combined source is really interesting. We were considering using kapp-kontroller however our current dependence helm, and some architectural concerns caused us to pass on kapp-controller for now.
As to the question of what could be selling points towards a new "Package Manager" would be,
* Timoni like packages that has templating with type safety. * A large package pool, or * Abstraction over helm packages that could add type safety, or better yet automatic or at least semi-automatic conversion of helm charts (One can dream ;)) * Full management through kubernetes API / CRDs * Multicluster management, or fleet management.
We already looked into pkl, but this would require every package author to either have Java (and pkl) running on their system or we would need to package the jre (and pkl) in order to make it work probably. But Kubernetes examples are already out there (https://github.com/apple/pkl-k8s-examples) and we are keeping an eye on it.
I haven't tried to integrate pkl-go into something like glasskube so I am open to that part being painful because software gonna software, but I believe the general statement of pkl being Java-only is incorrect
However things like helmfile with renovate paired with a pipeline is my personal preference even if just for ensuring things remain consistent in a repo.
The `update all` button for instance seems terrifying on a cluster that means anything at all. None the less it's still cool for personal projects and the like!
The package controller reminds me a lot of Helm tiller with older versions of helm, and it became a big security issue for a lot of companies, so much so that helm3 removed it and did everything clientside via configmaps. Curious how this project plans on overcoming that.
> However things like helmfile with renovate paired with a pipeline is my personal preference even if just for ensuring things remain consistent in a repo.
Glasskube packages can also be put inside a GitOps repository as every package is a CR (custom resource). (They can even be configured via the CLI using the `--dry-run` and `--output yaml` flags and than put into git. In addition we are working on pull request to support package updates via Renovate: https://github.com/renovatebot/renovate/issues/29322
> The package controller reminds me a lot of Helm tiller with older versions of helm, and it became a big security issue for a lot of companies, so much so that helm3 removed it and did everything clientside via configmaps. Curious how this project plans on overcoming that.
As helm3 is now a client side tool only, that means that it can't enforce any RBAC by itself. OLM introduced Operator Groups (https://olm.operatorframework.io/docs/advanced-tasks/operato...) which introduces a permissions on an operator level. We might introduce something similar for Glasskube packages. Glasskube itself will still require be quite powerful, but we can than scope packages and introduce granular permissions.
The main pains remain unaddressed
- authoring helm charts sucks
- managing different values per environment
- connecting values across charts so I don't have to
Platform teams try to create internal developer platforms to further standardize Kubernetes configurations across teams and clusters, where developers can only do minor modifications. From my experience we want to reduce snow flake configurations. This is also a reason why we created Glasskube in the first place.
> - authoring helm charts sucks
Yes, 100% and we are on a mission to change this in future.
> - managing different values per environment
Glasskube packages are still configurable, but come with meaningful default values.
> - connecting values across charts so I don't have to
This is already possible, you can reference package configuration values from other packages easily via Glasskube, not needing to provide the same values multiple times.
This misses the point, what we need is something more like Terraform, which has a way to get dynamic information from resource values that are assigned by the system. One such example would be the secret that the postgres operator generates for connecting in the api server that needs access.
> > - managing different values per environment
Most charts already come with meaningful defaults. The issue is that you need simpler defaults for multiple environments that the user doesn't have to think about. There ought to be some higher level information coming into the pipeline that tells the tool what environment I'm working with and assign certain values automatically.
> Yes, 100% and we are on a mission to change this in future.
Configuration needs a proper language. Please avoid Yaml and something bespoke. There are a few configuration languages emerging, CUE is my personal pick in the horse race.
> This misses the point, what we need is something more like Terraform, which has a way to get dynamic information from resource values that are assigned by the system. One such example would be the secret that the postgres operator generates for connecting in the api server that needs access.
It is also already possible to inject values from secrets during runtime. You can for example create a Glasskube package that has a dependency on cnpg and add a `cluster.yaml` to your package and then dynamically patch the connection string (or credentials) from it to your deployment.
See the "ValueFrom" section of our configuration documentation for the exact inner workings: https://glasskube.dev/docs/design/package-config/
... this is what we have today, how do I know what value to patch in from, like what is the name of the secret?
Looking at that link makes me think this is like another layer of helm on helm, especially with the same go template values in yaml that are going to be fed into helm templates under the hood.
Putting more yaml on top of templated yaml is not the way to create the next package manager for k8s.
Fundamentally there’s no such thing as a k8s “package”. OLM is great for packaging operators, but I don’t see why we need yet another Helm.
It was a mistake that shouldn’t be repeated.
Adding black boxes on top of black boxes is not a good way to abstract complexity. Helm does nothing more than any template engine does, yet requires me to trust not only the competency of some random chart author but also that they will correctly account for how my k8s environment is configured.
When I inevitably have to debug some deployment, now I'm digging through not only the raw k8s config, but also whatever complexity Helm has added on to obfuscate that k8s config complexity.
Helm is an illusion. All it does is hide important details from you.
I found Kustomize extremely annoying to work with. Changing simple configuration options required way too much work.
Take a simple example - changing the URL value of an ingress. This is something every deploy is going to have to set, since it will be different for every cluster.
In Kustomize, I first have to find the ingress resource, then recreate the nesting properly in my kustomize file, and then repeat that for every deployment.
In Helm, I just change one entry in a values file, that is clearly named so I know what I am setting.
In addition, it is REALLY hard to refactor resources once there are a lot of Kustomization files. If I want to redo how the deployment works, I have to change every Kustomize file that any other repo using my project uses. If I have a lot of other people who are pulling in my project and then using kustomize on top of it, we have to coordinate any changes, because changing the structure breaks all Kustomizations.
With Helm, as long as I keep the same values file structure, I am free to move things around however I want. I can use values in completely new locations without having to change anything about the values files themselves.
I just don't see how it is easier. I find it a lot easier to read a default values file and figure out what every setting does rather than read 20 k8s yaml files trying to figure out what does what.
In some ways, I kind of feel like the Kustomize enthusiasts LIKE the things I find annoying about it; they think you SHOULD have to read every resource and fully understand it, and they don't want anyone to be able to change anything without every kustomizer also changing things. I get the theory that everyone should know the underlying resources, but in practicality I find Kustomize to be the wrong level of abstraction for what I want to do.
Pros:
- comes from someone with deep k8s experience
- has features for secrets and dynamic information based on k8s version and CRDs
- thinks about the full life-cycle and e2e process
Cons: (at the time)
- holds CUE weird, there are places where they overwrite values (helm style) which is antithetical to CUE philosophy. This was rationalized to me as keeping with the Helm mindset rather than using the CUE mindset, because it is what people are used to. I think this misses the big opportunity for CUE in k8s config.
- has its own module system that probably won't integrate with CUE's (as it stands today), granted CUE's module system wasn't released at the time, but it seems the intention is to have a separate system because the goal is to align with k8s more than CUE
- Didn't allow for Helm modules to be dependencies. This seems to have since changed, but requires you to use FluxCD (?)
I didn't ever adopt it because of the philosophical differences (me from CUE, stefan from k8s). I have seen others speak highly of it, definitely worth checking out to see if it is something you'd like. I have plans for something similar once an internal Go package in CUE is made publicly available (https://github.com/cue-lang/cue/commits/master/internal/core...). The plan is to combine the CUE dep package with OpenTofu's graph solver to power a config system that can span across the spaces.
Feel free to disregard, but it would help me understand if you briefly explain how this fits in with / compares to existing tools like argocd.
I watched your video and I saw that argo was one of the tools you were installing, so clearly this is occupying a different niche - But I'm not sure what that is yet :)
It is often used by developers to get a glimpse of the state of core application of a company without cluster access.
Glasskube focuses on the packages your core application depends on. Managing the life cycle of these infrastructure components, testing updates and providing upgrading paths. You can still put Glasskube packages into your GitOps repo and sync them via ArgoCD into the cluster. Our PackageController will do the rest.
That being said, I'm confused as to where Glasskube is positioned in solving this problem. In the title of this post, you are claiming Glasskube is an "alternative to Helm"; although in your documentation you have a "Glasskube vs Helm" guide that explicitly states that "Glasskube is not a full replacement of Helm". I'm trying to understand how these two statements can be true. To make things more confusing, under the hood Glasskube repositories appear to be a repackaging of a Helm repository, albiet with a nicer UI.
From what I've gathered after reading the docs, Glasskube is being positioned as an easier way to interact with Helm charts - offering some easy-to-use tooling for upgrades and dependency management. To me, that doesn't exactly feel like it replaces Helm, but simply supplements my use of it, because it doesn't actually combat the real problems of using Helm.
My biggest pain points, some of which I don't think Glasskube is addressing, that I think are at the crux of switching off Helm:
- The arbitrary nature of how value files are laid out - every chart appears to have its own standards for which fields should be exposed and the nomenclature for exposing them
- Helm releases frequently get stuck when updating or rolling back, from which they can't be fixed without needing to be uninstalled and reinstalled
- I need to reference the Helm chart values file to know what is exposed and what values and types are accepted (Glasskube schema'd values files does address this! Yay!)
Apart from the Helm chart values schema, I don't think Glasskube solves these fundamental problems. So I'm not sure why I would spend the large amount of effort to migrate to this new paradigm if the same problems could still cause headaches.
Lastly, I would also concur with @llama052's comment, that an "update all" button will always be forbidden in my, and probably most other, companies. Considering the serious lack of standardisation that comes with Helm chart versioning (whether the app version changes between charts, whether roles or role bindings need to be updated, whether values have been deprecated or their defaults have changed, etc.), it's incredibly risky to update a Helm chart without understanding the implications that come with it. Typically our engineers have to review the release notes for the application between the two Helm chart versions, at least test in dev and staging for a few days, and only then can we feel comfortable releasing the changes - one chart at a time. Not to mention that if you are in charge of running a system with multiple applications, you probably want to use GitOps, and in that case a version upgrade would require a commit to the Git repository and not just a push of a button on the infra IDP.
If I need a better GUI anywhere, it’s probably in EKS, or something that makes working with EKS a bit less painful.
Glasskube on the other hand is a package manger where you can find look up, install and configure packages via a cli and UI and overcome some of the shortcomings of Helm.
From Wikipedia: "A package manager or package-management system is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer in a consistent manner." https://en.wikipedia.org/wiki/Package_manager
Helm is a package manager as it consistently,
* Can pull and deploy applications via packages * Can manage (upgrade/reconfigure/delete) deployed applications * Can search and find helm charts.
So the difference is it lacks a GUI? Afaik GUI was never a requirement for a package manager.
And another perspective is, as GlassKube does not provide a packaging mechanism, and uses helm in the backend (established in another question, which I'll also reply) it's not really a package manager but a frontend to another one. (examples: dpkg - package manager - apt-get/apt/aptitude frontend)
Also IMHO, Considering CNCF landscape, Glasskube is more positioned as a Continues Delivery tool than a package manager. But this is my take.
But more importantly, what I'm getting at is that with Terraform I get infrastructure as code.
[0] "The package manager for Kubernetes" https://helm.sh/
[1] "Get up to speed with Helm, the preeminent package manager for the Kubernetes container orchestration system." https://www.oreilly.com/library/view/learning-helm/978149208...
[2] "Helm is a package manager for Kubernetes." https://en.wikipedia.org/wiki/Helm_(package_manager)
- package manager
- templating engine
- deployment tool
You’ll hear various opinions on how good it is at each of these roles. In my personal experience it is a decent package manager, a poor but serviceable templating engine, and a horrifically bad deployment tool.
Normally you pair Helm with something like Flux or Argo if you want IaC
IMO terraform is probably not the right tool for the job for managing deployments. It can do it, but like helm itself, it’s also not super great at doing cluster deployments. If you’re looking for a good GUI like experience Argo is a good option.
I like Terraform for managing infra, and it’s good at a lot of things, but managing deployments on a cluster with IaC is not one of them. Why? Mainly because deployments are much more dynamic than infrastructure and the amount of throat clearing required for terraform to perform a state diff is much much much higher than other tech. Much better to look at the tech I mentioned (Argo and Flux) for that, because they do state diff for these things in milliseconds. I’ll leave it to the reader to figure out how long it takes terraform to do this.
It’s possible to go entirely in the “everything is a kube manifest” direction using technologies like Crossplane and Cluster API and (for AWS) technologies like ACK. But I don’t think we are entirely there yet for these technologies in terms of maturity, so in my recent designs I usually settle for provisioning of cluster and initial bootstrapping with terraform before mostly handing off to Argo for deployment, but then doing this weird counterbalance for having to go back to terraform when infra stuff is necessary. I can see a future world, however, where you bootstrap management clusters with something like terraform but then basically everything else, both infra (clusters, buckets, IAM, etc) and deployments (Helm) is declarative through tech like Argo and Crossplane provisioning.
The tough part right now is when you have application devs that need to provision infrastructure and then deploy on top of it. Right now that looks like asking your developer to write some app specific terraform like S3/IAM/KMS/Redis/whatever and then deploying their app on top of it with Argo or flux or what not. The ideal maybe looks like using the same tech for both eventually, as well as even provisioning the cluster that the stack runs on with the same tech.
Think of an ArgoCD ApplicationSet that generates a bunch of Applications. Those Applications render a bunch of Helm charts, and inside those charts there are CRDs used by some random operator like Strimzi, Grafana or Vector.
Given YAML's lack of syntax and the absense of any sort of standard for rendering templates, it's practically impossible to know what are the actual YAML being injected in the Kubernetes API when you make a top-level change. It's trial and error, expensive blue-green deployments and hundreds of debugging minutes all the way, every month.
Think of it as codecov analysis, but just for resource changes.
https://akuity.io/blog/the-rendered-manifests-pattern/
I think proper programming language support is the way to go.
Ideally a static type system that isn't turing complete and guaranteed to terminate. So something like starlark with types.
Personally, I would prefer a SQLite database. Ok I'll show myself out.
CUE is also pragmatic in that it has integrations with yaml, json, jsonschema, openapi, protobuf
See also: KCL, which is very similar and might _actually_ be "the winner". Time will tell.
Having used CUE, mainly outside of Kubernetes, I cannot see myself switching to KCL. I really like having a configuration language that isn't so tied to a specific system and which I can use with the Go SDK
this exists for k8s[0]. there have been other users based on the same library[1], I heard reddit did something similar internally
[0] - https://github.com/cruise-automation/isopod [1] - https://github.com/stripe/skycfg
Programs take inputs and can output arbitrary data such as resources. However they can do so with type safety, and everything else a programming ecosystem can achieve.
For asset distribution it uses wasm, and that's it!
If you want to check it out its here: github: (https://github.com/davidmdm/yoke) docs: (https://davidmdm.github.io/yoke-website)
I like that you said: > I think proper programming language support is the way to go.
I think we need to stop writing new ways of generating yaml since we already have the perfect way of doing so. Typed languages!
This means general purpose languages do not qualify, and more generally, no general recursion
To protect against somebody writing a non-terminating program?
General programming languages come with a lot of general purpose benefits from their ecosystems like package managers npm, cargo, go modules, etc.
They have test runners, and control flow.
Lots of them already have type definitions for kubernetes and if you are working in Go you have access to almost the entire kubernetes ecosystem.
Maybe we are throwing the baby out with the bath water when we disqualify general purpose languages?
You speak like turning incomplete languages cannot have the control structures, tooling, and ecosystems we enjoy elsewhere, which would be the wrong assessment. I recommend you take a look at CUE to see how this can be true
The OpenAPI specs are probably better than the Go language types for k8s. They have more of the validation information and you can get at the CRDs / versions actually running in the cluster.
However there’s no reason we should rule out general purpose languages.
We have a lot of configuration based IaC and configuration tooling a la jsonnette and cue and yet these are riddled with their own problems and DX issues.
Anyways we don’t need to see eye to to eye on this but I respect your position.
We’ve learned the hard way general purpose languages are poor for configuration at scale. I know first hand having worked on some of the larger prod infrastructures out there.
At scale, the best SRE’s out there still have trouble reasoning about the system and end up pushing bad config that takes down prod.
Languages like CUE really are different and better. CUE in particular hits the right balance for configuration of millions of lines of k8s yaml.
CUE was created by the same person who wrote the Borg precursor and also worked on BCL & GCL.
However there's something about a full blown general purpose language that is so much more flexible.
I don't think that the fact that people can and do write bad programs disqualifies general purpose languages from being great tools to build packages.
I am sure there is just as equally bad CUE, Jsonette, PKL, etc out there.
Other than CDK8s I don't know of other tools that have tried in this space to use general purpose languages to define their packages, and I think CDk8s uses are generally happy. Much more so than helm users at least.
I am not sure I can agree with this statement > We’ve learned the hard way general purpose languages are poor for configuration at scale
I think we've just assumed this, or seen a pulumi project we didn't like working in.
I believe and hope there will be plenty of room to experiment and innovate in this space!
If your config is the source of truth of what your infra should be, then you can use source control tools to roll back to a known good state, or to binary search for when a problem was introduced.
If you use programmatic config, then you can't find out the intended state of your system without executing a program. You can't grep through program executions in a meaningful way, especially at scale. So you can't do even simple things like search for a string.
Guaranteeing termination is helpful, but it doesn't solve the main problem that programmatic config puts a large complexity barrier between you and the ability to understand your infrastructure.
Tools like Helm give up a fair amount of this declarative benefit. And IMO that's one of the reasons why it's almost always a worse experience to use a helm chart than to just render the chart once and for all and forget Helm ever existed.
This is how Pants (build system) works. You have declarative Starlark which supports basic programming semantics and this generates a state the engine reads and tries to produce.
I've been meaning to dive into jsonnet for a while but it'd be good to have a higher level representation that didn't rely on sophisticated templating and substitution engines like current k8s.
Compare k8s to Terraform where you have modules, composability, variables. These can be achieved in k8s but you need to layer more tooling on (kustomize, helm, etc). There could be a richer config system than "shove it in YAML"
Things like explicit ordering and dependencies are hard to represent in pure yaml since they're ",just text fields" without additional tools
Languages can be declarative or imperative. For instance, Pulumi and CDK are declarative.
> If you use programmatic config, then you can't find out the intended state of your system without executing a program. You can't grep through program executions in a meaningful way, especially at scale. So you can't do even simple things like search for a string.
I don’t understand — nothing stops a language from having an intermediate compilation step that shows the intended state and is searchable. Beyond that, programmatic config means you can add in hooks to make plans or runs more interrogatable.
> Guaranteeing termination is helpful, but it doesn't solve the main problem that programmatic config puts a large complexity barrier between you and the ability to understand your infrastructure.
It seems like this is untrue — having seen templated IaC that is hundreds of thousands of lines and cdk that defers that complexity to an abstraction that I have to understand once, I’d always take the latter.
Agreed that helm use is a giant mistake and liability.
[0]: https://nickel-lang.org/
For example: You have a problem that is suitable for some message queue -> Apache Kafka. Now you have 7 new problems and the complexity warrants perhaps 3 other services, and on, and on.
Nowadays you can also start with a light weight MQ like Rabbit MQ and decouple your service just into a hand full components. This will set you up for scalability without introducing massive overheads.
In end it is also always a knowledge game. How experienced are you or how much time are you willing to invest into learning and understanding a technology.
Here's how I use it: https://github.com/shepherdjerred/homelab/tree/main/cdk8s
https://github.com/cdk8s-team/cdk8s
I was unaware there was a name for this pattern, thank you.
I can't quite put my finger on it, but having multiple, orphaned commit histories inside a single repository sounds off, even if technically feasible.
With directories you need to resort to diffing to spot any changes between files in folders.
That said there are some merge conflict scenarios that make it a little annoying to do in practice. The author doesn’t seem to mention this one, but if you have a workflow where hotfixes can get promoted from older versions (eg prod runs 1.0.0, staging is running 1.1.0, and you need to cut 1.0.1) then you can hit merge conflicts and the dream of a simple “click to release” workflow evaporates.
That isn't quite my understanding - but I am happy to be corrected.
There wouldn't be be a staging->main flow. Rather CI would be pushing main->dev|staging|prod, as disconnected branches.
My understanding of the problem being solved, is how to see what is actually changing when moving between module versions by explicitly outputting the dynamic manifest results. I.e. instead of the commmit diff showing 4.3 -> 5.0, it shows the actual Ingress / Service / etc being updated.
> With directories you need to resort to diffing to spot any changes between files in folders.
Couldn't you just review the Commit that instigated that change to that file? If the CI is authoring the change, the commit would still be atomic and contain all the other changes.
> but if you have a workflow where hot-fixes can get promoted from older versions
Yeah 100%.
In either case, I'm not saying it's wrong by any stretch.
It just feels 'weird' to use branches to represent codebases which will never interact or be merged into each other.
Probably just a matter of taste, but I think having the files for different environments "side by side" makes it actually easier to compare them if needed, and you still have the full commit history for tracking changes to each environment.
When I played with this we had auto-push to dev, then click-button to merge to staging, then trigger some soak tests and optionally promote to prod if it looks good. The dream is you can just click CI actions to promote (asserting tests passed).
> Couldn't you just review the Commit that instigated that change to that file?
In general though a release will have tens or hundreds of commits; you also want a way to say “show me all the commits included in this release” and “show me the full diff of all commits in this release for this file(s)”.
> In either case, I'm not saying it's wrong by any stretch.
Yeah, I like some conceptual aspects of this but ultimately couldn’t get the tooling and workflow to fit together when I last tried this (probably 5 years ago at this point to be fair).
I might be misunderstanding what you mean by staging in this case. If so, my bad!
I don't think staging ever actually gets merged into prod via git history, but is rather maintained as separate commit trees.
The way that I visualised the steps in this flow was something like:
In that model, there isn't actually ever a merge conflict that can occur between staging and prod, because you're not dealing with merging at all.The way you then deal with a delta (like ver 1.0.1 in your earlier example) is to create a PR directly against the Prod branch, and then next time you do a full release, it just carries out the usual process, 'ignoring' what was there previously.
It's basically re-invented the terraform delta flow, but instead of the changes being shown via Terraform by comparing state and template, it's comparing template and template in git.
> ultimately couldn’t get the tooling and workflow to fit together when I last tried this
I genuinely feel like this is the bane of most tooling in this space. Getting stuff from 'I can run this job execution on my desktop', to 'this process can scale across multiple teams, integrated across many toolchains and deployment environments, with sane default' still feels like a mess today.
edit: HN Formatting
One immediate question: Your docs say "Upgrading CRDs will be taken care of by Glasskube to ensure CRs and its operators don't get out-of-sync." but searching for "CRD" in your docs doesn't lead to any concrete results.
This is one of our biggest pain ponts with Helm right now. Can you share your plans?
[1] <https://stackable.tech/en/>
[2] <https://www.youtube.com/watch?v=Q8OSYOgBdCc>
We will update the docs though.
Even with all that said, operating system packages require a vast amount of testing, development, and patching, constantly, even within those small parameters. Packages feel easy because potentially hundreds of hours of development and testing have gone into that package you're installing now, on that platform you're on now, with the components and versions you have now.
Kubernetes "packages" aren't really packages. They are a set of instructions of components to install and configure, which often involves multiple distinct sets of applications. This is different in a couple ways: 1) K8s "packages" are often extremely "loose" in their definition, leading to a lot of variability, and 2) they are built by all kinds of people, in all kinds of ways, making all kinds of assumptions about the state of the system they're being installed into.
There's actually multiple layers of dependencies and configuration that have to come together correctly for a Kubernetes "package" to work. The K8s API version has to be right, the way the K8s components are installed and running have to be right, the ACLs have to be right, there has to be no other installed component which could conflict, the version of the components and containers being installed by the package need to be pinned (and compatible with everything else in the cluster), and the user has to configure everything properly. Upgrades are similarly chaotic, as there's no sense of a stable release tree, or rolling releases. It's like installing random .deb or .rpm or .dmg files into your OS and hoping for the best.
Nothing exists that does all of this today. To make Kubernetes packaging as seamless as binary platform-specific packaging, you need an entire community of maintainers, and either a rolling-release style (ala Homebrew) or stable versioned release branches. You basically need a project like ArtifactHub or Homebrew to manage all of the packages in one way. That's a big undertaking, and in no way profitable.
Building them for different Kubernetes version or environments is something we also already thought about and need to happen at some time to bake more configuration into to the build step.
One can throw Flux and Git* actions or what not in the mix. The outcome is a boring CI/CD implementation. Boring is good. Boring and powerful because of how Jsonnet works.
It's pretty neat in the right hands.
By analogy, this is the same issue as frontend programming faces with the DOM. Introducing a VDOM / reducer paradigm (like react) would go a long way towards solving these problems.
This is basically how control theory works in general though. You have a state, a goal, and a perturbation toward the goal. I think this is the right level of abstraction if you want a powerful and flexible tool.
> it's not really possible to know how the various components will interact....Introducing a VDOM / reducer paradigm (like react) would go a long way towards solving these problems.
I think the problem here is that the physical characteristics and layouts of the machines makes such a huge difference that it would be prohibitively costly to virtualize or simulate this in a meaningful way. So instead, people use subsets of the physical structure to verify that configuration states work. You do this by having `dev`, `staging`, `prod` environments, using colored deployments, canary analysis, partial rollouts etc.
This says nothing about ease of use. And for software development, ease of use matters. Otherwise we would all use assembler, or at most C++. They're very powerful and flexible.
If anything, too much power and flexibility is a problem.
It sounds like your pitch is focused on users, but I think you might want to think about how to attract a package ecosystem. Some things that would make my life easier as a packager:
* Statically typed config language. It's insane that we're generating YAML files with Go templates. I'm a big fan of jsonnet, but whatever it is it should not be possible to have errors at runtime.
* A better way to document to users the various options and how they play together. For any moderately complex package it become very challenging to communicate how to configure the chart for your environment
* Better testing infrastructure; am I creating valid k8s resources given all of the possible configuration options?
What was really annoying was the constant moving and changing the yaml that kubernetes wanted. After you update your cluster and things stop working, it's really not about the layers on top, but it's about kubernetes and having to keep up with the new beta versions and also sunsetting other things.
That to me sounds like you're angry at helm and argocd, but don't realize it.
The kubernetes apiserver publishes all the resources it supports, including custom-resource-definitions, including typed specifications that can be used to validate what you're submitting client-side.
If helm weren't a dumb layer of yaml templating, it could tell you locally, like compiling a typed programming language, "this helm chart won't work on your cluster because you have the beta version of this CRD and need the alpha version", or it could even transform things into the correct version.
The kubernetes API provides everything that's needed to statically verify what Groups/Kinds/Versions exist, and tooling like helm is just too dumb to work with it.