If you're thinking of HashiCorp Vault as a platform that is solely there for secure storage of static secrets (i.e. basically a secure key/value store), then you probably would think it is over-hyped.
The power of Vault is in the integrations and the unification of authorization policies across those different integrations.
"I need a way to allow my different AWS accounts to issue TLS certificates within particular namespaces of a common CA" - Vault can do that.
"I am managing service identity using mTLS at the moment, but now I need to get short-lived SSH keys for those service identities" - Vault can do that too.
"I need to issue credentials that allow access to AWS and GCP IAM roles to my on-premise application which has a JWT" - you guessed it, Vault can do that.
Agreed. Vault is awesome, dynamically creating TLS certificates for VPN-usage, creating new MySQL usernames/passwords, with limited lifecycles, etc. Most of the love I have for it is from things above and beyond mere static-secrets.
It also has to be said that KMS will get expensive quickly too. Though it is probably fair to say that a three node vault installation with dynodb backend won't be so cheap it'll be rock-solid and I think it could be documented within an hour - for a develop to use it at least.
Edit: I'd never heard of SOPS, apart from the many open issues and pull-requests making it look a little swamped, it seems like a nice tool. Especially given that it can work with GPG-keys.
I'm one of the creators of Vault. I read this back when it was posted and I'd be happy to share my thoughts. I'll note its worth reading through to the last paragraph and into the comments, the title is a bit bait-y and the article does a better job than the title gives itself credit for.
Broadly speaking, if you're looking at Vault to solve a specific problem X for a specific consumption type Y on a specific platform Z, then it _probably is overkill_ (I wouldn't say "overhyped" :)). i.e. "encrypted key/value via env vars on AWS Lambda". "X via Y on Z." The power of Vault is: multiple use cases, multiple consumption modes, and multiple platforms supported with a single consistent way to do access control, audit logging, operations, etc.
I can't stress that "single consistent way to do access control, audit logging, operations, etc." enough. Multiple security use cases dangling off that consistency is really important as soon as you hit N=2 or N=3 security use cases.
If you need say... encrypted KV and encryption-as-a-service and dynamic just-in-time credentials and a PKI system (certificate), and you need this as files and as env vars, and you need this on Kubernetes and maybe also on EC2, then Vault is -- in my totally biased opinion -- going to blow any other option out of the water.
That's a somewhat complex use case but its something Vault excels at. For simpler use cases, Vault is making more and more sense as we continue to make Vault easier to use. For example, we now provide a Helm chart and official K8S integration so you can run Vault on K8S very easily. And in this mode, developers don't even need to know Vault is there cause their secrets show up as env vars and files just like normal K8S secrets would.
Also, this article is from June 2019 and in 10 short months we've made a ton of progress on simplifying Vault so it gets closer to that "X via Y on Z" use case. Here are some highlights I can think of off the top of my head but there are definitely more, this is just from memory:
* We have integrated storage as an option now, so you don't need separate storage mechanisms.
* Our learn guides went from basically zero to lots of content which makes it much easier to learn how to use Vault: https://learn.hashicorp.com/vault
> We have integrated storage as an option now, so you don't need separate storage mechanisms.
Off topic, but seeing as you mentioned it - do you see the integrated Raft storage becoming the preferred solution? Without it, there's (IMHO) a bit of a chicken-and-egg bootstrapping problem between Vault and Consul.
Yes it will become the preferred solution. We just went GA with it (after a year of testing) in our last release. We’re now recommending this to customers. We’ll continue to provide support for Consul indefinitely and aren’t forcing customers to switch over.
(Note “support” above is literal human support. “Support” in terms of enabling that use case will be around indefinitely but planned forever, we’ll always support different storage backends. But for our paying customers, we only support Consul and integrated storage for the sake of providing good service.)
Vault is awesome. Thank you for creating it. We use it at Klarrio for issuing Kafka client certificates on demand in a multi-tenat DC/OS cluster. It sits right in the core of the platform and it just never fails us.
keycloak is pretty nice for that. You can just drop some docker-compose somewhere (or k8s) or plain wildfly/tomcat and you have saml idp + oauth2 (openid connect) out of the box including ui and self-service for users - it's also completly themeable (that's what I'm doing at the moment for a project)...
you don't even need a ldap/ad in the background but you can connect to one.
however it's strictly for authentication and authorization - basically users + roles
vault is different usecase but can probably be connected.
> I need an equivalent to Windows' Active Directory in Linux world that ideally can also federate with/masquerade as AD. Can Vault be such thing?
Vault is not such a thing.
AD is basically LDAP + Kerberos. Neither Vault nor Keycloak can fill that role. You would need Samba 4 setup as a domain controller [1] for that.
If you need AD but for Linux desktops you could take a look at FreeIPA [2].
If your feeling brave you can combine an LDAP server (like Wren:DS, 389 or OpenLDAP) with a Kerberos implementation like MIT Kerberos and create an AD like experience yourself. Though I would not recommend that.
Only for bootstrapping the TLS key of the vault servers we needed another tool.
Sure there is sops.
We opted for google/tink
This gives us asymmetric encryption with KMS for bootstrapping the vault AMIs on AWS.
We create Vault AMIs with Packer and put encrypted TLS keys inside the AMIs.
This way we have immutable machine images, which can crash/restart using an Autoscaling group and we can attach all decryption processes to the instance role of the EC2 machines
The article is thorough and informative and it seems to me that it is entirely incorrect to characterize it as "Hit Piece" -- there is simply too much, high-quality technical information in the article to reduce it to a "hit piece."
To call it "trashy" makes me suspect you did not read all or much of the content.
I skimmed it this time, and read it one of the 2 previous times it was on HN.
There is some thoughtful technical information in the article.
I admit it is hard for me to recover from click baity titles, and that probably lead to my choice of words.
I guess I would rephrase my salty comment as - “this article would be better if it simply focused on the value of SOPS + KMS+ git for some use cases instead of characterizing Vault as ‘overhyped’.”
Anyone here used Vault to secure a Phoenix application? In particular, the idea that a short lived database credential can get auto-refreshed and then notify the Repo's supervisor to restart the repo with new credentials?
That's how I'd probably do it but I'd rather not reinvent the wheel if someone's already done it.
This would amount to potentially "restarting" the app every hour? That would cause potential errors every hour.
I think I saw somewhere that it's actually quite easy to do this with OTP. You can just tell the supervisor to kill the children and when they respawn, they'll pick up the new credentials.
I cannot comment on Vault as I have not tried it yet, but I am a happy user of sops for some deployments.
In my case the secrets are stored in a sops-encrypted JSON file in the repository. Sops encrypts the secrets but keeps the JSON structure so changes to a specific secret can be tracked in git with a oneline change instead of a binary blob diff of the full file which is very nice.
This is a bad analysis. Sure, Vault is infrastructure you have to maintain. For a small team, it's probably not a good tradeoff. But even if you dismiss Vault's ability to do dynamic secrets (which is actually one of its primary benefits), the security model for Vault is much stronger than KMS-encrypted secrets in Git. The author appears to be claiming that protecting your Vault server from a local root privilege escalation attack is so difficult that it's better to ... instead rely on the security of your team members' IAM credentials which are floating around on their workstations, and which talk to KMS over a public API.
No matter how you manage your secrets, one of the most important factors is how often you rotate your secrets. If you can't transition to fully dynamic just-in-time secret provisioning that Vault can provide for certain backends (including AD, OpenLDAP, PKI certs, various database engines, Consul, and more), then you still ought to be rotating your secrets on a regular basis. And in that case, a tool like Vault (or AWS Secrets Manager or GCP Secret Manager) is going to be a lot easier to build a rotation framework on top of than a git repo with secrets encrypted in place.
Exampme, it can authenticate using K8S JWT, or AWS IAM. Using K8S JWT, the process of generation token is gone, you use the token from k8s. Send it to Vault.
If you just look for something like a key-value store for env var and authorized using simple username/password pair then KMS or any key-value store with a HTTP on top of it will work.
Sops creator and co-maintainer here. We love Sops. We love Vault. We use both of them at Mozilla in various places. Beyond the "tool war", which is silly, what really matters is using tools that integrate well with your workflow, provide real value and strong security.
Until you dig into deeper use cases, it's easy to get hung up on Vault as JUST a KV store. Honestly, if that's all you want, and you're on a cloud provider, using a combo of their storage + KMS or other services (i.e. SSM Parameter Store) is probably the more practical approach.
That said, the proposed drawbacks aren't necessarily ... drawbacks? Granted my perspective is from deploying Vault to the Cloud vs. on-prem.
1. Yeah, Vault needs a separate place to do its storage. However, that's a strength if you're following an immutable infrastructure pattern. You store the data in something like DynamoDB and still have the freedom of tearing down and re-creating the Vault servers themselves.
2. Vault may be more expensive right out of the gate, but if you're trying to cover ALL of its functionality with cloud services, it'll start saving you money eventually. Furthermore, many of the cloud alternatives have service limits and quotas. I mean geez, if you want an internal CA through AWS, you're paying a flat $400 a month + costs per certificate.
3. Vault has a learning curve, but it's not worse than having to memorize the buffet of CLI commands through your cloud provider. Yes, getting it set up for the first time can be a jigsaw puzzle, but when everything is up and running, it's smooth sailing. (Plug - I have a project that automates setting up Vault on AWS: https://github.com/jcolemorrison/vault-on-aws)
4. As for vulnerabilities of the "default implementation" - Yes, the public cloud presents more opportunities for exposure, but that's not limited to just Vault. Furthermore, if someone gets root access to your vault servers...that's not a vault thing. 80% of the 2019 massive cloud breaches are the result of misconfigurations and account compromises (source: https://www.paloaltonetworks.com/resources/research/unit42-c...).
If you have a handful (say less than 10) repos, the article is right – Vault is an overkill. You can and should just check-in encrypted secrets with git-crypt or SOPS. Whenever master enc/dec keys are rotated, you can afford to make a dozen PRs and commits. Note here - I'm not talking about rotating revoked secrets. I'm taking about the master key rotation.
If you have hundreds of repos, the mere fact of "rotating" all secrets when the master key changes is a laborious task. Opening, tracking and waiting for hundreds of PRs to be merged, with stuff failing for unrelated reasons to secrets (random unit test or PR check failures, etc) will take months.
https://12factor.net/config – this exists for a reason. Your PR checks have nothing to do with production secrets. Your code is compiled with Java/Go/Node/whatever and is published as an artifact, and this also has nothing correlated to secrets nor configuration. You should be able to run the same artifact (docker image or jar or whatever) – with different configurations and secrets for different environments.
While the article is good and technical, it skips over these parts. If you're just doing a toy service with no regard for anything of the above, then yeah, by all means just use SOPS and don't bother with vault. But if you're an organization, then don't read this assume that this works for 100s of services & engineers with differing CI/CD practices and schedules, and no luxury time to wait hours for a key=value change or waiting months for a full rotation.
35 comments
[ 2.8 ms ] story [ 86.7 ms ] threadNote- I dont work for Hashicorp but have used many of their excellent products from time to time.
This is the first that I have seen the article and it directly addresses a knowledge gap for me that I have been eager to fill.
The power of Vault is in the integrations and the unification of authorization policies across those different integrations.
"I need a way to allow my different AWS accounts to issue TLS certificates within particular namespaces of a common CA" - Vault can do that.
"I am managing service identity using mTLS at the moment, but now I need to get short-lived SSH keys for those service identities" - Vault can do that too.
"I need to issue credentials that allow access to AWS and GCP IAM roles to my on-premise application which has a JWT" - you guessed it, Vault can do that.
It also has to be said that KMS will get expensive quickly too. Though it is probably fair to say that a three node vault installation with dynodb backend won't be so cheap it'll be rock-solid and I think it could be documented within an hour - for a develop to use it at least.
Edit: I'd never heard of SOPS, apart from the many open issues and pull-requests making it look a little swamped, it seems like a nice tool. Especially given that it can work with GPG-keys.
Broadly speaking, if you're looking at Vault to solve a specific problem X for a specific consumption type Y on a specific platform Z, then it _probably is overkill_ (I wouldn't say "overhyped" :)). i.e. "encrypted key/value via env vars on AWS Lambda". "X via Y on Z." The power of Vault is: multiple use cases, multiple consumption modes, and multiple platforms supported with a single consistent way to do access control, audit logging, operations, etc.
I can't stress that "single consistent way to do access control, audit logging, operations, etc." enough. Multiple security use cases dangling off that consistency is really important as soon as you hit N=2 or N=3 security use cases.
If you need say... encrypted KV and encryption-as-a-service and dynamic just-in-time credentials and a PKI system (certificate), and you need this as files and as env vars, and you need this on Kubernetes and maybe also on EC2, then Vault is -- in my totally biased opinion -- going to blow any other option out of the water.
That's a somewhat complex use case but its something Vault excels at. For simpler use cases, Vault is making more and more sense as we continue to make Vault easier to use. For example, we now provide a Helm chart and official K8S integration so you can run Vault on K8S very easily. And in this mode, developers don't even need to know Vault is there cause their secrets show up as env vars and files just like normal K8S secrets would.
Also, this article is from June 2019 and in 10 short months we've made a ton of progress on simplifying Vault so it gets closer to that "X via Y on Z" use case. Here are some highlights I can think of off the top of my head but there are definitely more, this is just from memory:
* We have integrated storage as an option now, so you don't need separate storage mechanisms.
* Our learn guides went from basically zero to lots of content which makes it much easier to learn how to use Vault: https://learn.hashicorp.com/vault
* We have an official, feature-packed Kubernetes integration to do stuff like secret injection and rotation automatically. We also publish a Helm chart to run Vault on Kubernetes. https://learn.hashicorp.com/vault?track=getting-started-k8s#...
We're looking at ways to make running Vault much, much easier. More on that later this year. :)
Off topic, but seeing as you mentioned it - do you see the integrated Raft storage becoming the preferred solution? Without it, there's (IMHO) a bit of a chicken-and-egg bootstrapping problem between Vault and Consul.
(Note “support” above is literal human support. “Support” in terms of enabling that use case will be around indefinitely but planned forever, we’ll always support different storage backends. But for our paying customers, we only support Consul and integrated storage for the sake of providing good service.)
https://www.keycloak.org/
I need an equivalent to Windows' Active Directory in Linux world that ideally can also federate with/masquerade as AD. Can Vault be such thing?
If you can use public cloud, have you considered Azure Active Directory? We're a Linux shop using it very successfully for on-prem and cloud systems.
you don't even need a ldap/ad in the background but you can connect to one.
however it's strictly for authentication and authorization - basically users + roles
vault is different usecase but can probably be connected.
Vault is not such a thing.
AD is basically LDAP + Kerberos. Neither Vault nor Keycloak can fill that role. You would need Samba 4 setup as a domain controller [1] for that.
If you need AD but for Linux desktops you could take a look at FreeIPA [2].
If your feeling brave you can combine an LDAP server (like Wren:DS, 389 or OpenLDAP) with a Kerberos implementation like MIT Kerberos and create an AD like experience yourself. Though I would not recommend that.
[1] https://wiki.samba.org/index.php/Setting_up_Samba_as_an_Acti...
[2] https://www.freeipa.org/
You can even do cross-forest trust to AD, log into your Linux systems as users who live in AD...
Only for bootstrapping the TLS key of the vault servers we needed another tool.
Sure there is sops.
We opted for google/tink
This gives us asymmetric encryption with KMS for bootstrapping the vault AMIs on AWS.
We create Vault AMIs with Packer and put encrypted TLS keys inside the AMIs. This way we have immutable machine images, which can crash/restart using an Autoscaling group and we can attach all decryption processes to the instance role of the EC2 machines
Vault checks all the boxes for the authors own “Ideal Secrets Management Solution Requirements“.
If Vault’s only feature was its PKI, it would be underhyped in my opinion.
To call it "trashy" makes me suspect you did not read all or much of the content.
There is some thoughtful technical information in the article.
I admit it is hard for me to recover from click baity titles, and that probably lead to my choice of words.
I guess I would rephrase my salty comment as - “this article would be better if it simply focused on the value of SOPS + KMS+ git for some use cases instead of characterizing Vault as ‘overhyped’.”
That's how I'd probably do it but I'd rather not reinvent the wheel if someone's already done it.
In both cases you might have short service interruptions. So why not simply restart the app which is definitely more technology agnostic.
I think I saw somewhere that it's actually quite easy to do this with OTP. You can just tell the supervisor to kill the children and when they respawn, they'll pick up the new credentials.
In my case the secrets are stored in a sops-encrypted JSON file in the repository. Sops encrypts the secrets but keeps the JSON structure so changes to a specific secret can be tracked in git with a oneline change instead of a binary blob diff of the full file which is very nice.
No matter how you manage your secrets, one of the most important factors is how often you rotate your secrets. If you can't transition to fully dynamic just-in-time secret provisioning that Vault can provide for certain backends (including AD, OpenLDAP, PKI certs, various database engines, Consul, and more), then you still ought to be rotating your secrets on a regular basis. And in that case, a tool like Vault (or AWS Secrets Manager or GCP Secret Manager) is going to be a lot easier to build a rotation framework on top of than a git repo with secrets encrypted in place.
Exampme, it can authenticate using K8S JWT, or AWS IAM. Using K8S JWT, the process of generation token is gone, you use the token from k8s. Send it to Vault.
If you just look for something like a key-value store for env var and authorized using simple username/password pair then KMS or any key-value store with a HTTP on top of it will work.
That said, the proposed drawbacks aren't necessarily ... drawbacks? Granted my perspective is from deploying Vault to the Cloud vs. on-prem.
1. Yeah, Vault needs a separate place to do its storage. However, that's a strength if you're following an immutable infrastructure pattern. You store the data in something like DynamoDB and still have the freedom of tearing down and re-creating the Vault servers themselves.
2. Vault may be more expensive right out of the gate, but if you're trying to cover ALL of its functionality with cloud services, it'll start saving you money eventually. Furthermore, many of the cloud alternatives have service limits and quotas. I mean geez, if you want an internal CA through AWS, you're paying a flat $400 a month + costs per certificate.
3. Vault has a learning curve, but it's not worse than having to memorize the buffet of CLI commands through your cloud provider. Yes, getting it set up for the first time can be a jigsaw puzzle, but when everything is up and running, it's smooth sailing. (Plug - I have a project that automates setting up Vault on AWS: https://github.com/jcolemorrison/vault-on-aws)
4. As for vulnerabilities of the "default implementation" - Yes, the public cloud presents more opportunities for exposure, but that's not limited to just Vault. Furthermore, if someone gets root access to your vault servers...that's not a vault thing. 80% of the 2019 massive cloud breaches are the result of misconfigurations and account compromises (source: https://www.paloaltonetworks.com/resources/research/unit42-c...).
If you have hundreds of repos, the mere fact of "rotating" all secrets when the master key changes is a laborious task. Opening, tracking and waiting for hundreds of PRs to be merged, with stuff failing for unrelated reasons to secrets (random unit test or PR check failures, etc) will take months.
https://12factor.net/config – this exists for a reason. Your PR checks have nothing to do with production secrets. Your code is compiled with Java/Go/Node/whatever and is published as an artifact, and this also has nothing correlated to secrets nor configuration. You should be able to run the same artifact (docker image or jar or whatever) – with different configurations and secrets for different environments.
While the article is good and technical, it skips over these parts. If you're just doing a toy service with no regard for anything of the above, then yeah, by all means just use SOPS and don't bother with vault. But if you're an organization, then don't read this assume that this works for 100s of services & engineers with differing CI/CD practices and schedules, and no luxury time to wait hours for a key=value change or waiting months for a full rotation.