My tldr; konfd writes out k8s configmaps based on other k8s resources like secrets, configmaps, etc. Really useful for writing out complete config files into a pod namespace without relying on external config backends.
Question: While looping on a syncInterval is certainly clean and understandable, it feels suboptimal when all the templates sources are themselves watchable with a k8s client. Benefits of switching to a watch model:
1. Speed: It'd be nice to have the template rendering fire immediately after a source secret changed versus waiting for syncInterval.
2. Resource Utilization: Switching from syncInterval to a watch should save significant cycles by avoiding reprocessing templates when config hasn't changed.
This is great feedback. I'm still on the fence about a watch mode for the following reasons:
1. The template configmap will not change as often as the secrets and configmap key/value pairs they reference. This means you'll always need a sync loop to regenerate the template and compare it with the current generated version. This is how most configuration management systems work. Maybe we can watch the data and uses changes as triggers?
2. Watch is a bit racy when a template references multiple secrets and configmaps. It's really hard to prevent a partial config. One idea is to scale the konfd replicaset to 0 before making changing a bunch of secrets and configmaps, then scaling konfd back to 1 to trigger the processing of the templates to pick up the new values. The other option is to wait the "sync-interval", but you'll run the risk of partial configs or needing multiple runs to generate the complete config.
One possible solution to prevent 1 and 2 is to only reference one external configmap or secret for each template resource. I'm still testing these patterns in real life and will update the docs on which patterns work best for each situation. With that said, I think I'll still add a watch mode so others can test and provide feedback.
The second one has plagued me with a couple kubernetes clients I've written, but something about your restatement of the problem struck me with (IMO, of course) a nice idiomatic solution: a "debounce" fan-in (ish) channel between the ResultChan() and consumer that bundles up watch notifications after some configurable wait period. And actually, it's a common enough use case (I'd definitely use it too) that I'll bet it would be useful to have in client-go itself.
And secondly, I would love to see this kind of concept upgraded to a first-class citizen in kubernetes. Essentially, I think an equivalent of `/etc/environment` for your cluster would be a powerful way of generalizing app components. It would be great if just placing resources in your cluster allows them to pick up at least some of their required configuration, much like customizing `/etc/environment` does for *nix machines.
Kelsey, I (and others) would kindly appreciate it if you'd stop calling anything in Kubernetes as it exists today a "secret" and writing new code to further encourage its use. People should be aware in no uncertain terms that the "secrets" store in etcd is totally unencrypted and insecure.
If you'd like to find a place to help, I'd suggest focusing your efforts on connecting Kubernetes to Hashicorp Vault, which is truly secure, and deprecating the old unencrypted etcd-backed implementation.
A prototype is a great way to get the conversation started. Most people don't even know why Vault should be considered, let alone why it's better than K8s secrets (which in their world works fine).
At its core, I'm having some difficulty wrapping my brain around the idea that any organization that's large and complex enough to demand to run their own compute platform (e.g. K8S) isn't also large and complex enough to demand a secure configuration system by default.
The API refers to these objects as secrets. How does it improve understanding to pick some other name because you feel the current one is undeserved? I'm not disagreeing with that assessment, but it seems odd to just refuse to use it and chide others for doing so. It that supposed to be a protest or something?
It's more a criticism of the implementation. Kubernetes deserves a secrets store, but the one it's got isn't worthy of the name. I'm advocating for engineering efforts to be prioritized over other projects on making it a secure implementation, because people are likely to be misled into thinking the existing implementation is already secure.
First, thanks for taking the time to offer feedback regarding what we call "secrets" in Kubernetes. As you've rightfully pointed out the current implementation of "secrets" are stored unencrypted in etcd; something like Hashicorp's Vault [1] would be a much better choice for the given use-case.
I've already began exploring how an integration between Vault and Kubernetes would work [2]. There are some things to work out, but those discussions are under way [3]. The current prototype/example demonstrates one way of leveraging Vault from applications deployed on Kubernetes, without changes to the Kubernetes core.
While the vault-controller [2] works, we can do better. One idea is to consider what a deeper Vault integration looks like. Ideally we can modify parts of Kubernetes, mainly the kubelet agent that runs on every node, to handled the secure generation and renewal of unique Vault tokens for each Pod during the Pod creation phase.
Another idea would be to rethink "secrets" altogether and rebuild the entire feature on top of the Vault API. We can "hide" Vault under the current "secrets" API, which would let us remain backwards compatible. That would be phase one. Phase two could introduce new Secret extensions, which would enable users to declaratively manage Vault tokens, backends, and secret renewals through the Kubernetes API.
I appreciate your efforts here, Kelsey. What I'm looking for (and I realize I'm not your manager; this is just my opinion borne of a year of experience with K8S) is a focus on strengthening the core first, then going back to the veneer. Great edifices start with a strong foundation.
Also, the engineering teams need to do a better job of integrating community PRs that address implementor demands. For example, Consul K/V support has been asked for since 2014 [1], yet nobody seems to be in a hurry to integrate the (generously-provided) functionality [2].
Finally, I'd like to see deadlines and decision-makers appointed for making decisions such as the one you discuss, so we can avoid endless debates and make forward progress quickly.
We all want a strong foundation and will continue to work to create one. In parallel, many of us like building and sharing things, and given that most things in Kubernetes are just abstractions, the things we are building today will continue to work tomorrow, even if the "secrets" backend is reimplemented on top of something like Vault.
I'm hopeful people find things like konfd useful, even if you only use configmaps and avoid secrets entirely -- a use case supported by konfd.
If someone can access your etcd, they can write new pods to run into it which will then be distributed to nodes to run.
Those new pods could reference existing secrets, be in any namespace (e.g. kube-system), and are exactly the same as root on all k8s nodes.
Because of that, having access to etcd is equivalent to having admin cluster access. The secrets are no more secret if they're encrypted in etcd or not since anyone with access to etcd can launch pods that will decrypt arbitrary secrets.
This is true iwth vault as well. Once secrets are in vault, access to etcd will still result in all secrets being available to the attacker. The threat model is no better.
There are plans to encrypt them in etcd for the sake of them not appearing in backups and because people like you constantly bitch about it, but it doesn't mater and saying they shouldn't be called secrets because of this is FUD.
First, I'm offended by your characterization of my argument as "bitching," as though our concerns are unjustified.
At any rate, you bring up two points: First, that etcd is insecure, which is a problem in itself that needs to be addressed. Communications between K8S and etcd and between applications and etcd need to be secured, and K8S itself needs role-based ACLs in the core.
Second, for the Vault use case, the best practice is not to place Vault tokens in etcd that grant direct access to any secrets. Instead, the best practice (as Kelsey's vault-controller project does) is to pass a "wrapper token," which is a single-use token. The application consumes the wrapper token (hopefully quickly) and exchanges it for a longer-lived token, which can then be used to access the secret data.
Once the wrapper token has been used, it is subsequently worthless to someone who has direct etcd access. And Vault's auditing capabilities can help you detect misuses of wrapper tokens.
The comment you're replying to doesn't say that etcd is insecure; just that access to etcd is equivalent in power to Kubernetes API access.
It's hard to have a productive discussion in such a confrontational tone, especially when you don't really have all the facts. If you have questions about how to secure Kubernetes communication with etcd, feel free to ask!
What facts do you think I don't possess? Does etcd have an ACL capability yet? Does it possess auditing capability?
AFAIK etcd only supports TLS client authentication, which is arguably a fraction of what it needs in order to be considered a secure store in any meaningful sense. Consul is miles ahead in this regard.
What etcd does and doesn't support doesn't matter here.
Pods do not access etcd. Nothing that is not fully trusted has access. You don't need ACLs when the only users of etcd are 100% trusted (aka just kubernetes core components).
The kubernetes API provides secrets to pods and can do its own validation (and there's experimental support for that). It can provide its own auditing. That's where it actually matters, not etcd.
The fact that you don't allow anything to connect to etcd besides the K8S components doesn't really make etcd itself inherently secure. It also implies that access to etcd is an all-or-nothing proposition, which doesn't work well in complex environments. People will need to get into etcd in order to perform debugging, maintenance, etc., and at big organizations, they will need differing levels of access depending on their respective roles.
And your assumption that only K8S will access etcd isn't necessarily shared by others. In the model implementation, perhaps that is so; but in the real world, implementors may desire to share etcd with other applications. This too necessitates additional security options for etcd itself.
If someone was actually interested in security, they would not allow tenant (or user / etc) defined components access the base infrastructure's etcd, even if it had ACLs.
Basic separation of concerns would call for a separate instance of etcd.
In a complex environment, spinning up a new etcd should not be a big deal.
Now, having ACLs, is not a bad thing, but they should only be used to lock down the base infrastructure access even more, and still have a separate instance for other applications.
Also, "authentication" is not identical to "authorization" or "access control" -- and it's not obvious to the reader that discussion of one includes the other. The words "ACL" and/or "access control" really need to be specifically used in the title and body of the pertinent documentation to be easily locatable by the researcher.
Finally, it doesn't appear that the K8S API server can authenticate to etcd yet. (That's not etcd's fault, but it does contribute to my impression of the integration's overall security posture.) Again, please correct me if I'm mistaken.
Open Source also does not mean free products and support.
As a case in point, you were giving out about 2 different pieces of open source software in this thread.
Neither of which you have to pay for. Both of which are complex system that have taken many many people hours to build. As a return, is suggesting that you may submit a PR (or even a bug, with suggestions so that the people CoreOS hires to write docs know there is an issue)
I have no doubt. After spending 1 year with K8S, I've become extremely frustrated with it and its components. I honestly expected more (and better) out of Google and its partners and contributors.
They don't owe me anything in particular, but I do wish more people would let their fruit ripen on the vine instead of in the grocery store (or worse, at the purchaser's house).
And judging by the comment scores, I'm not the only one who feels this way.
Conversely, you may note that I do get very excited and happy about products that are well-designed, well-documented, and aren't dumped onto an unsuspecting public before they are mature.
Thanks. I've no doubt that the teams involved are well intentioned, and I don't want to discourage them from doing quality work. And I know it can be frustrating to receive negative feedback.
So if anyone's feelings have been hurt, please know that was not my intent.
Hi kelsey, this is pretty nice !
Is there any way to bribe you into slowing down on the awesome sauce delivery pipeline so people can catch up !? asking for a friend.
If you find the time can you elaborate on why you elected to use ConfigMaps for the templates too instead of building a 3rd party ressource ?
This way you wouldn't have to use annotations and the definition might be a little more terse (or not)
Is there something about configmaps that make implementation easier ? or provides additional behavior wrt to pod lifecycle or something like that ?
ConfigMaps offer everything I need for a tool like konfd, and ConfigMaps work with the majority of clusters deployed today. While I could have used a ThirdPartyResource, that would require me to design a custom scheme and write a little more code up front.
Also, there are a few issues [1] with ThirdPartyResource objects including the lack of validation and inconsistencies when interacting with ThirdPartyResource objects through the Kubernetes API [2]. That's not to say ThirdPartyResource's should not be used, but it's not a decision take lightly.
Now, if konfd were to grow or add more features, then I think a ThirdPartyResource would be the way to go -- mainly because ConfigMaps require the use of "reserved" keys and annotations to configure behavior.
40 comments
[ 3.0 ms ] story [ 94.7 ms ] threadMy tldr; konfd writes out k8s configmaps based on other k8s resources like secrets, configmaps, etc. Really useful for writing out complete config files into a pod namespace without relying on external config backends.
Question: While looping on a syncInterval is certainly clean and understandable, it feels suboptimal when all the templates sources are themselves watchable with a k8s client. Benefits of switching to a watch model:
1. Speed: It'd be nice to have the template rendering fire immediately after a source secret changed versus waiting for syncInterval.
2. Resource Utilization: Switching from syncInterval to a watch should save significant cycles by avoiding reprocessing templates when config hasn't changed.
1. The template configmap will not change as often as the secrets and configmap key/value pairs they reference. This means you'll always need a sync loop to regenerate the template and compare it with the current generated version. This is how most configuration management systems work. Maybe we can watch the data and uses changes as triggers?
2. Watch is a bit racy when a template references multiple secrets and configmaps. It's really hard to prevent a partial config. One idea is to scale the konfd replicaset to 0 before making changing a bunch of secrets and configmaps, then scaling konfd back to 1 to trigger the processing of the templates to pick up the new values. The other option is to wait the "sync-interval", but you'll run the risk of partial configs or needing multiple runs to generate the complete config.
One possible solution to prevent 1 and 2 is to only reference one external configmap or secret for each template resource. I'm still testing these patterns in real life and will update the docs on which patterns work best for each situation. With that said, I think I'll still add a watch mode so others can test and provide feedback.
And secondly, I would love to see this kind of concept upgraded to a first-class citizen in kubernetes. Essentially, I think an equivalent of `/etc/environment` for your cluster would be a powerful way of generalizing app components. It would be great if just placing resources in your cluster allows them to pick up at least some of their required configuration, much like customizing `/etc/environment` does for *nix machines.
If you'd like to find a place to help, I'd suggest focusing your efforts on connecting Kubernetes to Hashicorp Vault, which is truly secure, and deprecating the old unencrypted etcd-backed implementation.
Also, it's just a start; the existing "secrets" implementation needs to be completely scrapped in the name of security.
If you'd like to participate in these discussions, here are the details for the sig-auth's meetings & agenda items: https://github.com/kubernetes/community/tree/master/sig-auth
At its core, I'm having some difficulty wrapping my brain around the idea that any organization that's large and complex enough to demand to run their own compute platform (e.g. K8S) isn't also large and complex enough to demand a secure configuration system by default.
There's your answer. I don't like it any more than you do, but that's the world we live in.
I've already began exploring how an integration between Vault and Kubernetes would work [2]. There are some things to work out, but those discussions are under way [3]. The current prototype/example demonstrates one way of leveraging Vault from applications deployed on Kubernetes, without changes to the Kubernetes core.
While the vault-controller [2] works, we can do better. One idea is to consider what a deeper Vault integration looks like. Ideally we can modify parts of Kubernetes, mainly the kubelet agent that runs on every node, to handled the secure generation and renewal of unique Vault tokens for each Pod during the Pod creation phase.
Another idea would be to rethink "secrets" altogether and rebuild the entire feature on top of the Vault API. We can "hide" Vault under the current "secrets" API, which would let us remain backwards compatible. That would be phase one. Phase two could introduce new Secret extensions, which would enable users to declaratively manage Vault tokens, backends, and secret renewals through the Kubernetes API.
[1] https://www.vaultproject.io
[2] https://github.com/kelseyhightower/vault-controller
[3] https://github.com/kubernetes/kubernetes/issues/10439#issuec...
Also, the engineering teams need to do a better job of integrating community PRs that address implementor demands. For example, Consul K/V support has been asked for since 2014 [1], yet nobody seems to be in a hurry to integrate the (generously-provided) functionality [2].
Finally, I'd like to see deadlines and decision-makers appointed for making decisions such as the one you discuss, so we can avoid endless debates and make forward progress quickly.
[1] https://github.com/kubernetes/kubernetes/issues/1957
[2] https://github.com/kubernetes/kubernetes/pull/31622
I'm hopeful people find things like konfd useful, even if you only use configmaps and avoid secrets entirely -- a use case supported by konfd.
// PS. To help avoid confusion, I wish K8s (not really) secrets examples didn't use the word "vault" unless using, you know, "vault".
Those new pods could reference existing secrets, be in any namespace (e.g. kube-system), and are exactly the same as root on all k8s nodes.
Because of that, having access to etcd is equivalent to having admin cluster access. The secrets are no more secret if they're encrypted in etcd or not since anyone with access to etcd can launch pods that will decrypt arbitrary secrets.
This is true iwth vault as well. Once secrets are in vault, access to etcd will still result in all secrets being available to the attacker. The threat model is no better.
There are plans to encrypt them in etcd for the sake of them not appearing in backups and because people like you constantly bitch about it, but it doesn't mater and saying they shouldn't be called secrets because of this is FUD.
At any rate, you bring up two points: First, that etcd is insecure, which is a problem in itself that needs to be addressed. Communications between K8S and etcd and between applications and etcd need to be secured, and K8S itself needs role-based ACLs in the core.
Second, for the Vault use case, the best practice is not to place Vault tokens in etcd that grant direct access to any secrets. Instead, the best practice (as Kelsey's vault-controller project does) is to pass a "wrapper token," which is a single-use token. The application consumes the wrapper token (hopefully quickly) and exchanges it for a longer-lived token, which can then be used to access the secret data.
Once the wrapper token has been used, it is subsequently worthless to someone who has direct etcd access. And Vault's auditing capabilities can help you detect misuses of wrapper tokens.
It's hard to have a productive discussion in such a confrontational tone, especially when you don't really have all the facts. If you have questions about how to secure Kubernetes communication with etcd, feel free to ask!
AFAIK etcd only supports TLS client authentication, which is arguably a fraction of what it needs in order to be considered a secure store in any meaningful sense. Consul is miles ahead in this regard.
By all means, correct me if I'm mistaken.
Pods do not access etcd. Nothing that is not fully trusted has access. You don't need ACLs when the only users of etcd are 100% trusted (aka just kubernetes core components).
The kubernetes API provides secrets to pods and can do its own validation (and there's experimental support for that). It can provide its own auditing. That's where it actually matters, not etcd.
And your assumption that only K8S will access etcd isn't necessarily shared by others. In the model implementation, perhaps that is so; but in the real world, implementors may desire to share etcd with other applications. This too necessitates additional security options for etcd itself.
Basic separation of concerns would call for a separate instance of etcd.
In a complex environment, spinning up a new etcd should not be a big deal.
Now, having ACLs, is not a bad thing, but they should only be used to lock down the base infrastructure access even more, and still have a separate instance for other applications.
Can you please ensure it's in the Security section of the documentation? It's not mentioned there at all! https://coreos.com/etcd/docs/latest/op-guide/security.html
Also, "authentication" is not identical to "authorization" or "access control" -- and it's not obvious to the reader that discussion of one includes the other. The words "ACL" and/or "access control" really need to be specifically used in the title and body of the pertinent documentation to be easily locatable by the researcher.
Finally, it doesn't appear that the K8S API server can authenticate to etcd yet. (That's not etcd's fault, but it does contribute to my impression of the integration's overall security posture.) Again, please correct me if I'm mistaken.
https://github.com/coreos/etcd/blob/master/Documentation/v2/... and https://github.com/coreos/etcd/blob/master/Documentation/op-... are the two pages.
As a case in point, you were giving out about 2 different pieces of open source software in this thread.
Neither of which you have to pay for. Both of which are complex system that have taken many many people hours to build. As a return, is suggesting that you may submit a PR (or even a bug, with suggestions so that the people CoreOS hires to write docs know there is an issue)
"I (and others) would kindly appreciate it if you'd stop"
"If you'd like to find a place to help, I'd suggest"
"engineering teams need to do a better job of integrating community PRs "
"Finally, I'd like to see deadlines and decision-makers appointed"
"What facts do you think I don't possess?"
They don't owe me anything in particular, but I do wish more people would let their fruit ripen on the vine instead of in the grocery store (or worse, at the purchaser's house).
And judging by the comment scores, I'm not the only one who feels this way.
Conversely, you may note that I do get very excited and happy about products that are well-designed, well-documented, and aren't dumped onto an unsuspecting public before they are mature.
I think this article is pertinent: https://medium.com/@thejameskyle/dear-javascript-7e14ffcae36...
So if anyone's feelings have been hurt, please know that was not my intent.
If you find the time can you elaborate on why you elected to use ConfigMaps for the templates too instead of building a 3rd party ressource ? This way you wouldn't have to use annotations and the definition might be a little more terse (or not)
Is there something about configmaps that make implementation easier ? or provides additional behavior wrt to pod lifecycle or something like that ?
Also, there are a few issues [1] with ThirdPartyResource objects including the lack of validation and inconsistencies when interacting with ThirdPartyResource objects through the Kubernetes API [2]. That's not to say ThirdPartyResource's should not be used, but it's not a decision take lightly.
Now, if konfd were to grow or add more features, then I think a ThirdPartyResource would be the way to go -- mainly because ConfigMaps require the use of "reserved" keys and annotations to configure behavior.
[1] https://github.com/kubernetes/kubernetes/issues/22768
[2] https://github.com/kubernetes/kubernetes/issues/29542