Interesting post and illustrative of some of the problems with supply chain security.
The more layers of abstraction and packaging that are added the harder it becomes for end-users to see what's going on.
One of the big problems is stale packages not keeping all the layers under them updated. This problem is likely to get worse as time goes by, as people move on from projects and things become unmaintained.
One option is to fork the packaging at one level and then take on maintenance yourself (e.g. creating/maintaining docker images) but then ofc you have all the work of doing that yourself.
Another option (in theory) could be someone selling a service of maintained, updated, scanned images/helm charts, but I've not seen previous efforts in this area be hugely successful...
hopefully the OSSF will come up with some good solutions in this space, but it's fundamentally a time consuming thing to fix and maintain.
We still have a lot of work to do but you can do things like providing your own base image that you customize and maintain to your specs and we build and test the apps on top of it
That's really interesting and I hope it'll be successful, it'd be great to see more solutions tackle this area. I guess starting with an entperise price point, means you can hopefully get a revenue level to help widen out the product set.
There is a difference between unnecessary work and necessary work. When you take in a dependency, you become dependent on it. That is necessary work.
It's great to highlight that dependencies always mean work. There's probably some deeper truth to that, at least in what I do. But that doesn't mean all work is bad. After all, work is what we put in to make things, well, work.
I "maintain" a bunch of fixes (that is, forks) of upstream stuff that I use all around the place. I used to maintain a few packages just for recreation. I have selinux dependencies, ansible playbooks, system accounts, certificates, and a lot of other stuff that I need to maintain in order to make stuff work.
I guess I wanted to say don't be afraid to maintain your dependencies. Sure, it's work, but not unnecessary work.
“But each layer here adds an element of required trust”, how often we simply glance over that and assume. In the same time, building everything from source is also neither reasonable nor 100% secure. Glad smart people like Dan are looking into this.
What's not reasonable about building from source? It really seems like a small cost to snapshot a version of third-party code into your own source control system. There really aren't (or shouldn't be) that many different things in your environment.
The amount of dependancies, build environment setup steps, across all of the components, and sheer complexity of putting the entire stack together from scratch, would exclude a substantial portion of the target users... hence, while possible, is not reasonable, certainly not on angling basis.
Building from source seems me the only way to get to a reasonable state when it comes to security patching.
However these modern stacks contain a lot of small parts. The article gives the kiwigrid/k8s-sidecar as an example. Also, well shown in the article, it is not as easy as copying some commands to your own Dockerfile. Look at the busybox image using glibc from another debian image.
It's not reasonable in the sense that it is a hell lot of work and would require more time and effort the average devops/sre/whatever (team) has.
It would also go against the promise that Kubernetes would make things easy because you could just do helm install stable/prometheus
I worked on buildpacks for a while. Those had to track "only" a few hundred upstream dependencies and build them from source. It took a fulltime team to build and maintain the infrastructure.
The problem here is economic. Because there is no body of information about software assets acting as public good, each individual is forced to internalise the costs. Either you internalise the cost of building everything yourself, or you internalise the risk of accepting incoming assets on faith. But you bear a cost.
I look upon this and despair. How are average folks supposed to trust and use these things? It feels like NPM all over again. I keep seeing these kinds of things across modern tech stacks/languages and wonder why some brainiac has not looked at this in a more general sense of a trusted supply chain and audit/update process approach to fix this or come up with some transformational approach that at a conceptual level can be applied across languages and tech stacks.
I think that kube-prometheus specifically tries to do too much, turning the simple problem of running a few services into its own brand-new problem. It attacks the problem of "it's hard to write a manifest to run a docker container", which isn't that hard, and it replaces it with its own configuration language, integrations that don't exist upstream, etc.
The general answer is to avoid helm charts that run more than one piece of software. If you want Prometheus, install Prometheus. If you want Grafana, install Grafana (and accept that you have twice as much work, because you are running twice as much software). Helm has a culture of bundling all components into one chart, which makes it easy to make a mess. You should reject this culture and install the components that you need, and accept that it's going to be extra effort -- running other people's software is hard!
(My full-time job is running our company's open source project in the cloud for people that can't get it installed themselves. It is a multi-person full-time job. It is not yet trivial to run other people's software, even if you talk to those people all day and have commit access to the repository.)
Here is a slightly off-topic rant. When I first started using Kubernetes, I installed the Prometheus Helm chart. It contains a config file that searches for objects in Kubernetes that have a "prometheus.io/scrape" annotation to control scraping. I didn't know that at the time, and searched in vain for a way to scrape multiple containers in the same pod. Everything on the Internet said that was impossible, and indeed... it is impossible with the random configuration that the Helm chart supplied. I wrote my own Prometheus configuration and got exactly what I wanted. The abstraction layer that showed up without any documentation in the Helm chart caused me infinite trouble, whereas if I had to write the scrape configurations from the start by myself, this would have never been a problem. I wouldn't have gotten started in 1 minute, but I would have understood the software I was responsible for operating.
It's dangerous to explode random configuration and code into your Kubernetes cluster. Helm is a tool that explodes random configuration and code into your Kubernetes cluster. Tread carefully!
To answer your original question, "How are average folks supposed to trust and use these things?", when I first came to the Kubernetes world I saw Helm and immediately thought "this doesn't seem like a very good idea to me", and haven't used it much. I suspect that most reasonable people will come to the same conclusion, but the allure of "get this tedious task done in 15 minutes" will snare a few more victims along the way.
The gist is that tags are mutable and deletable. Local caching means that every node can disagree on what the tag resolves to, and so can the registry. What version corresponds to which tag at which location at a given time is not deterministic.
The only way out is to use fully-digested references. The digest means that there can be no mutation of what's being pointed to. It will mean the same in every configuration and on every node, no matter when the reference is resolved and no matter when the reference was created.
It also means every layer has to be kept indefinitely, forever. Tags at the very least are pointers that stop indefinite and exponential storage growth.
Tags don't help exponential growth, because you cannot reliably reason from a tag-only reference back to the exact assets it represents. That's feasible with fully-digested references.
Without complete asset transparency, these issues will only grow worse.
Off and on I have pitched folks the idea that we need some concept of "universal asset accounting" or a "universal asset graph".
A federatable body of data about assets at various levels of detail, abstraction and age.
For example: I should be able to have the digest for a file and be able to see anywhere it has ever showed up. I should be able to zoom in and out from that pivot point: to debian repositories, to container layers, to dependency graphs in NPM, to Helm charts. I should be able to see where it lives in my own systems, and exactly how it came to be there, from the network operations in my own systems all the way back to tickets and commits. For any given asset I should be able to discover everything that was assembled, transformed, transported and inspected in its scope.
The best description of this I've seen is "chain of custody": Like with evidence in criminal proceedings. Docker and Git's use of content-addressable hashes helps in this regard, but there is still considerable work to be done when tracking third-party dependencies.
We used "chain of custody" as our analogy for buildpacks when I was working on them.
It's not completely accurate, though, for the same reason as "supply chain". There isn't a single linear sequence of agents, actions and assets for a given asset. It's a graph in which things can appear many times in many permutations.
From this viewpoint, Helm is really kind of broken-by-design. It takes a perfectly good system (Kubernetes Manifests) which allows you to exactly specify the exact versions of all the containers you want on your cluster, and it wraps it in an inscrutable "helm install" invocation which can potentially pull containers from anywhere at deploy time.
I've used helm template which generates manifests but unfortunately this isn't really the common use case, and it doesn't give you that much compared to your favorite text templating language.
I think that an accounting system which encodes uncertainty ("it could be any one of these two dozen images") will help create pressure on the upstream, as well as downstream visibility into who is affected.
I’d agree with this, particularly Helm v1/2 which installed an agent (Tiller) to inject imperative statefulness back into your deployments, this obviating some of the benefits of the elegant declarative design of k8s.
As of Helm 3 Tiller is gone and it’s much better, but you’re also right that the templating is horrible. I think you have clearly gone wrong when you have a template operator to indent your yaml the correct amount when composing reusable chunks into your manifests, white space debugging in yaml is not what we should be spending our time on in 2021.
I think the kustomize approach of layering JSON patches is also not generalizable enough, though it’s better than Helm for simple cases. It does come with some good dev tooling in skaffold and so that’s what I’m using for now.
I have long thought that simple code objects in your language of choice are likely to be the long-term winner, operator frameworks, etc. - the OpenAPI generated code isn’t that good in Python (I suspect it’s better in Go) but Pulumi or Pykube are examples of this approach done well (though neither are suitable for every use case at this time).
Sadly Helm is the current default and it’s where the community has standardized so it will take a lot of work to map all the accumulated wisdom from the Helm charts into some other approach.
This is an epic mess in the making to be honest and no one takes it seriously. It’s really not important at all for anyone I have spoken to. Most of our stacks are magical fucking unicorns to the marketing and decision makers and the majority of the developers these days. The level of response I get is akin to “but it’s got electrolytes” while blindly downloading god knows what into production kubernetes clusters. Hell even kubernetes does it. I’m waiting for something to waltz in and do some damage. Even worse is some of the scanning software which itself ships with CVE ridden dependencies. I mentioned this to one of the larger supply chain scanning vendors and they didn’t even know.
99% of our tech is like smoking to look cool and being in with the crowd. But someone’s going to get cancer one day.
I take it seriously, and I'm considered a crusty old lunatic sitting in the corner ranting about imaginary security problems. I built my own version of WebPack because I couldn't trust the huge number of maintainers in the dependency chain for that project. Docker/Kube gives me the screaming heebie-jeebies.
The only time anyone will take it seriously is when they're hacked. And then their defence will be "but this is what everyone does! It's industry-standard".
I've come to accept it's a fact of life in the industry at the moment. Hopefully it'll change, but the only way I can see it changing is if the level of hacking increases to become intolerable. Yet there appears to be a high tolerance for hacking incidencts - it's accepted folklore that hackers are very smart people that can't be stopped no matter how good your system is.
At the core of this issue is reproducibility. If you can fully reproduce the software you use, you no longer need to trust that X binary/container was built properly (although you still need to trust the source code itself, as well as the source of the source code). You can also always rebuild your software with newer dependencies (e.g. if they have received security updates).
I don't agree (or rather, I don't think it's the only option). Reproducibility is necessary when you are unable to trust the accuracy and integrity graph leading up to a particular asset. But if you have a trusted, exhaustive, tamper-resistant history of how an asset came to be, you can skip a rebuild altogether.
In terms of an asset being software that you're deploying, I don't get this attitute.
If the process for producing the software is unreproducible, then you're running the risk that the behaviour of the software will be unreproducible. As in you run the generation process once, and then again with the same inputs, and you'll get an output that works differently.
Now of course you can store the generated asset, which gives some certainty in the roll back case. But roll backs are hopefully rare, what's more common is to make small changes to inputs (like source code changes), and hope that the generate asset has only been affected by the intended change.
To give an example of how this works where no care has been taken that the process is reproducible, I've seen people try to make code changes to Docker images, and end up with changes that they didn't intend (like a different version of Debian in the image, or a different Ruby/Python version).
I think I misunderstood your emphasis. I've seen folks position reproducible builds as the best and only solution to trust and integrity problems; I buttonholed you as the same.
I do appreciate the general benefits of controlling the factors which go into an asset. Build reproducibility is a standard that requires a high capability to track and control your assets. In that respect it's an excellent goal.
Recapping: I say yes to reproducibility as a practice, but I don't see it as a substitute for an accounting mechanism. They're complementary.
The problem with operating at the level of the containerization abstraction is fundamentally the number of dependencies you are calling upon, and the emergent properties of all of these things interacting together (i.e. the mutually-shared state). Furthermore, you have to contend with the explosion in the dependency graph, because of the mindset that seems to be pervasive throughout: Everything should be vendored and never replicated in-house.
If you really want to get to the bottom of it, minimizing your interactions with external dependencies (i.e. binary images compiled outside of your control) is the only way to get closer to a provably-secure solution. Some might take offense with the prospect of a system that is "provably" secure, but all things are relative if you want to get really pedantic about it. I will concede that in many cases that security is not paramount, but it should always be a consideration when other humans are involved with your software. For instance, authentication protocols that a junior developer can read and understand in a few hours are going to be more secure than something that takes a team full of enterprise architecture and containerization specialists many weeks and meetings to figure out. Something that someone can play with in their head without much difficulty is going to be easier to probe for potential flaws. All of my project managers could explain in-depth the hows & whys of authentication/session protocols throughout our codebase because we went with something very simple.
I don't understand how so many seem to get buried under these mountains of tools/vendors either. Our entire list of 3rd party code-area dependencies is as follows: AWS, Windows, O365 (+Teams), SQLite, C# 9.0, VS2019, Git[Hub], .NET Core, AspNetCore (+Blazor), Apache PDFBox, Jenkins, iOS, XCode, MacOS. This is it. 100% of what we require for operating our humble software factory, including on-going delivery & support. There is no containerization and certainly no orchestration of such things. Our entire stack can be stood up on a t3a.nano without any performance problems. Developers enjoy being able to walk up to any blank windows PC and arrive at an operational development environment within 20 minutes (assuming you have to install VS2019; <60 seconds if you do not). I cannot comprehend sacrificing one ounce of this agility for even the most tantalizing of shiny new things. Yes, we had to write a lot of stuff in house. No, I would never go back and do it differently. This is better in almost every way. I have operated in environments where 100% of the things are outsourced. It is a hellish dystopia by comparison. There is probably some happy medium, but my security objectives are too intensive to explore such grey areas.
The vast majority of services exposed by pods created by off-the-shelf helm charts have a very small attack surface. If they were offering a full bash shell or something, it would be a different matter.
The chances of being able to exploit a given libc bug via something like node-exporter are near zero.
Still wouldn't give me confidence for handling valuable data though...
42 comments
[ 4.0 ms ] story [ 98.1 ms ] threadThe more layers of abstraction and packaging that are added the harder it becomes for end-users to see what's going on.
One of the big problems is stale packages not keeping all the layers under them updated. This problem is likely to get worse as time goes by, as people move on from projects and things become unmaintained.
One option is to fork the packaging at one level and then take on maintenance yourself (e.g. creating/maintaining docker images) but then ofc you have all the work of doing that yourself.
Another option (in theory) could be someone selling a service of maintained, updated, scanned images/helm charts, but I've not seen previous efforts in this area be hugely successful...
hopefully the OSSF will come up with some good solutions in this space, but it's fundamentally a time consuming thing to fix and maintain.
This is what we aim for with Tanzu Application Catalog https://tanzu.vmware.com/application-catalog
We still have a lot of work to do but you can do things like providing your own base image that you customize and maintain to your specs and we build and test the apps on top of it
Disclosure: I work for VMware, but not on this.
> The Team Edition is priced at $150,000 per year. The Org Edition is priced at $995,000 per year
[0]: https://docs.bitnami.com/tanzu-application-catalog/faq/get-s...
There is a difference between unnecessary work and necessary work. When you take in a dependency, you become dependent on it. That is necessary work.
It's great to highlight that dependencies always mean work. There's probably some deeper truth to that, at least in what I do. But that doesn't mean all work is bad. After all, work is what we put in to make things, well, work.
I "maintain" a bunch of fixes (that is, forks) of upstream stuff that I use all around the place. I used to maintain a few packages just for recreation. I have selinux dependencies, ansible playbooks, system accounts, certificates, and a lot of other stuff that I need to maintain in order to make stuff work.
I guess I wanted to say don't be afraid to maintain your dependencies. Sure, it's work, but not unnecessary work.
However these modern stacks contain a lot of small parts. The article gives the kiwigrid/k8s-sidecar as an example. Also, well shown in the article, it is not as easy as copying some commands to your own Dockerfile. Look at the busybox image using glibc from another debian image.
It's not reasonable in the sense that it is a hell lot of work and would require more time and effort the average devops/sre/whatever (team) has.
It would also go against the promise that Kubernetes would make things easy because you could just do helm install stable/prometheus
The problem here is economic. Because there is no body of information about software assets acting as public good, each individual is forced to internalise the costs. Either you internalise the cost of building everything yourself, or you internalise the risk of accepting incoming assets on faith. But you bear a cost.
Nuclear Protocol Magisty?
The general answer is to avoid helm charts that run more than one piece of software. If you want Prometheus, install Prometheus. If you want Grafana, install Grafana (and accept that you have twice as much work, because you are running twice as much software). Helm has a culture of bundling all components into one chart, which makes it easy to make a mess. You should reject this culture and install the components that you need, and accept that it's going to be extra effort -- running other people's software is hard!
(My full-time job is running our company's open source project in the cloud for people that can't get it installed themselves. It is a multi-person full-time job. It is not yet trivial to run other people's software, even if you talk to those people all day and have commit access to the repository.)
Here is a slightly off-topic rant. When I first started using Kubernetes, I installed the Prometheus Helm chart. It contains a config file that searches for objects in Kubernetes that have a "prometheus.io/scrape" annotation to control scraping. I didn't know that at the time, and searched in vain for a way to scrape multiple containers in the same pod. Everything on the Internet said that was impossible, and indeed... it is impossible with the random configuration that the Helm chart supplied. I wrote my own Prometheus configuration and got exactly what I wanted. The abstraction layer that showed up without any documentation in the Helm chart caused me infinite trouble, whereas if I had to write the scrape configurations from the start by myself, this would have never been a problem. I wouldn't have gotten started in 1 minute, but I would have understood the software I was responsible for operating.
It's dangerous to explode random configuration and code into your Kubernetes cluster. Helm is a tool that explodes random configuration and code into your Kubernetes cluster. Tread carefully!
To answer your original question, "How are average folks supposed to trust and use these things?", when I first came to the Kubernetes world I saw Helm and immediately thought "this doesn't seem like a very good idea to me", and haven't used it much. I suspect that most reasonable people will come to the same conclusion, but the allure of "get this tedious task done in 15 minutes" will snare a few more victims along the way.
Tags can be mutated in most (not all) container registries. A tag that you have vetted today could change without your knowledge.
Helm chart versions can also be overwritten.
Ideally, everything should be pulled from repositories you control.
Which reminds me: never don't do that. Always use fully-digested references. Don't fail to use fully-digested references.
The relevant section is here: https://livebook.manning.com/book/knative-in-action/chapter-...
The gist is that tags are mutable and deletable. Local caching means that every node can disagree on what the tag resolves to, and so can the registry. What version corresponds to which tag at which location at a given time is not deterministic.
The only way out is to use fully-digested references. The digest means that there can be no mutation of what's being pointed to. It will mean the same in every configuration and on every node, no matter when the reference is resolved and no matter when the reference was created.
Off and on I have pitched folks the idea that we need some concept of "universal asset accounting" or a "universal asset graph".
A federatable body of data about assets at various levels of detail, abstraction and age.
For example: I should be able to have the digest for a file and be able to see anywhere it has ever showed up. I should be able to zoom in and out from that pivot point: to debian repositories, to container layers, to dependency graphs in NPM, to Helm charts. I should be able to see where it lives in my own systems, and exactly how it came to be there, from the network operations in my own systems all the way back to tickets and commits. For any given asset I should be able to discover everything that was assembled, transformed, transported and inspected in its scope.
It's not completely accurate, though, for the same reason as "supply chain". There isn't a single linear sequence of agents, actions and assets for a given asset. It's a graph in which things can appear many times in many permutations.
I've used helm template which generates manifests but unfortunately this isn't really the common use case, and it doesn't give you that much compared to your favorite text templating language.
As of Helm 3 Tiller is gone and it’s much better, but you’re also right that the templating is horrible. I think you have clearly gone wrong when you have a template operator to indent your yaml the correct amount when composing reusable chunks into your manifests, white space debugging in yaml is not what we should be spending our time on in 2021.
I think the kustomize approach of layering JSON patches is also not generalizable enough, though it’s better than Helm for simple cases. It does come with some good dev tooling in skaffold and so that’s what I’m using for now.
I have long thought that simple code objects in your language of choice are likely to be the long-term winner, operator frameworks, etc. - the OpenAPI generated code isn’t that good in Python (I suspect it’s better in Go) but Pulumi or Pykube are examples of this approach done well (though neither are suitable for every use case at this time).
Sadly Helm is the current default and it’s where the community has standardized so it will take a lot of work to map all the accumulated wisdom from the Helm charts into some other approach.
99% of our tech is like smoking to look cool and being in with the crowd. But someone’s going to get cancer one day.
The only time anyone will take it seriously is when they're hacked. And then their defence will be "but this is what everyone does! It's industry-standard".
I've come to accept it's a fact of life in the industry at the moment. Hopefully it'll change, but the only way I can see it changing is if the level of hacking increases to become intolerable. Yet there appears to be a high tolerance for hacking incidencts - it's accepted folklore that hackers are very smart people that can't be stopped no matter how good your system is.
It is about time we embraced Guix/Nix.
If the process for producing the software is unreproducible, then you're running the risk that the behaviour of the software will be unreproducible. As in you run the generation process once, and then again with the same inputs, and you'll get an output that works differently.
Now of course you can store the generated asset, which gives some certainty in the roll back case. But roll backs are hopefully rare, what's more common is to make small changes to inputs (like source code changes), and hope that the generate asset has only been affected by the intended change.
To give an example of how this works where no care has been taken that the process is reproducible, I've seen people try to make code changes to Docker images, and end up with changes that they didn't intend (like a different version of Debian in the image, or a different Ruby/Python version).
I do appreciate the general benefits of controlling the factors which go into an asset. Build reproducibility is a standard that requires a high capability to track and control your assets. In that respect it's an excellent goal.
Recapping: I say yes to reproducibility as a practice, but I don't see it as a substitute for an accounting mechanism. They're complementary.
If you really want to get to the bottom of it, minimizing your interactions with external dependencies (i.e. binary images compiled outside of your control) is the only way to get closer to a provably-secure solution. Some might take offense with the prospect of a system that is "provably" secure, but all things are relative if you want to get really pedantic about it. I will concede that in many cases that security is not paramount, but it should always be a consideration when other humans are involved with your software. For instance, authentication protocols that a junior developer can read and understand in a few hours are going to be more secure than something that takes a team full of enterprise architecture and containerization specialists many weeks and meetings to figure out. Something that someone can play with in their head without much difficulty is going to be easier to probe for potential flaws. All of my project managers could explain in-depth the hows & whys of authentication/session protocols throughout our codebase because we went with something very simple.
I don't understand how so many seem to get buried under these mountains of tools/vendors either. Our entire list of 3rd party code-area dependencies is as follows: AWS, Windows, O365 (+Teams), SQLite, C# 9.0, VS2019, Git[Hub], .NET Core, AspNetCore (+Blazor), Apache PDFBox, Jenkins, iOS, XCode, MacOS. This is it. 100% of what we require for operating our humble software factory, including on-going delivery & support. There is no containerization and certainly no orchestration of such things. Our entire stack can be stood up on a t3a.nano without any performance problems. Developers enjoy being able to walk up to any blank windows PC and arrive at an operational development environment within 20 minutes (assuming you have to install VS2019; <60 seconds if you do not). I cannot comprehend sacrificing one ounce of this agility for even the most tantalizing of shiny new things. Yes, we had to write a lot of stuff in house. No, I would never go back and do it differently. This is better in almost every way. I have operated in environments where 100% of the things are outsourced. It is a hellish dystopia by comparison. There is probably some happy medium, but my security objectives are too intensive to explore such grey areas.
The chances of being able to exploit a given libc bug via something like node-exporter are near zero.
Still wouldn't give me confidence for handling valuable data though...