Ask HN: How does your company manage its encryption keys?
We just had an interesting data loss at work, that was due to data being encrypted at rest. We somehow managed to delete the encryption keys (still figuring out how), which became an obvious problem once our main database instance was rebooted.
Luckily we were able to restore the data, but now I (we) really want to learn what a proper setup would look like.
If you have any clear overview reading on the topic I'd be very interested to to know about it.
In particular I'm wondering: how do you back up your encryption keys, or even put them in escrow somewhere? Assuming we don't rotate the keys constantly I would love to just save them in somewhing like a passsword manager that's secured with 2FA/FIDO.
Would love to hear your thoughts!
247 comments
[ 2.7 ms ] story [ 303 ms ] threadWe use Hashicorp's Vault product to manage SSH credentials, TLS certificates, as well as application secrets across thousands of users, tens of thousands of virtual machines, and hundreds of applications.
We pay for the enterprise version, but the free version is more than capable for most needs. Avoid a password manager if you can, it leads to poor security practices and availability issues depending on how and where the data is stored (IMHO, YMMV). Use single sign on where ever possible.
Disclaimer: No affiliation other than a satisfied customer and frequent end user of the product.
The only thing protecting the unseal key is access to your KMS. So one rogue SRE can unseal the vault rather than requiring collusion of 1+ SRE members.
Again, this comes down to your risk tolerances and what you are protecting. I think for most workloads, the value KMS autounseal brings is worth the risk, but if you want to have tightest control, then the Shamir Split (M of N) is the best option.
I find it hard to build initial trust in the system, without involving the trust of an administrator + subsequent automation.
Not only it is opt-out, but the mentioned "off" toggle doesn't even exist.
The privacy policy is here: https://www.hashicorp.com/privacy
1) Secrets checked into code means when the code gets stolen, this is an unimaginably major breach. Code tends to get stolen eventually and most tech shops will never know / only know years later because they don't have access to the channels who will sell your code.
2) You can track secrets you have in storage, who has access and who is using them
3) When a secret does get stolen, they can be rotated with ease. You do not want to be spending on the order of man months manually auditing a huge code base and manually indexing and rotating every secret. With good setup you can do this without taking any systems down.
4) Most support HSM / KMS systems which encrypt the data using a key in a hardware security module, which, in theory the key cannot physically leave – the HSM will re-encrypt all your other keys. The HSM, if used properly can be a continuous bottleneck to decrypting any information, meaning, if your encrypted objects are stolen, the attacker needs active access to your (e.g. AWS) KMS system to decrypt them.
Password managers are good for individual users, but provide bad security characteristics for development and deployment.
AWS Secrets manager, AWS's competitor to Vault used a similar system I worked on that was developed within Twitch for this purpose.
At scale, you can very granularly define policies for each secret. When a secret is accessed, it is done so through a user or application identity. Each access is also logged.
you should only let the instance access the secret it requires.
As OP wrote, you did not solve it, just moved it to a different level.
Secrets may still need to go to config files for third party stuff, but you can write your in-house applications to fetch secrets at runtime and hold them only in memory. That's less opportunity for compromise vs. both memory and disk. Also have heard that Linux's process address space isolation is less prone to vulnerabilities than user account separation or filesystem permissions. Not really sure how true that is.
A typical deployment might involve placing the manager on a secure host that has access to generate and rotate keys, for example.
The manager can then configured to re-generate keys and vend them on-demand to instances that need require them. You can configure these keys with very limited access, and also make them expiring.
The manager then becomes effectively a keystore that can never export master keys, but only vends out some limited-scope keys to other instances.
Other instances would have to authenticate using some pre-configured host keys or even be authenticated directly though the cloud provider.
If your instances are compromised, the worst someone can do is to get access to a limited-scope key that will expire.
Hopefully you have other measures in place that would prevent and detect someone from just sitting on an instance and sucking up all your data and exporting them.
That means that each application got a ServiceAccount (SA) and each user got a username/password. Based on your identity, you get access to specific secrets from Vault.
It does admit the possibility that an operator's share could be copied. To work around that you can get a proper HSM that needs a quorum of smart cards presented to unlock. (The offline, low QPS, root of trust-oriented ones are not exactly cheap, but much cheaper than the network-attached ones targeting high QPS transactions). Vault Enterprise has PKCS#11 integration.
With the Thales nShield stuff, you can replicate key material from one to another for redundancy while allegedly still preserving the "can't ever get keys out in plaintext" property. Not sure about others.
https://github.com/sethvargo/vault-on-gke/blob/master/terraf...
The KMS ring itself is only accessible by Vault. People with high enough privileges for our Vault GCP project could technically grant themselves access to it, but on day-to-day business, nobody can view the project.
At some other place, where we were using AWS, I wrote a script that would store encrypt unseal keys (need multiple due to shamir) via pgp using their keybase public key. IIRC you can store encrypted keys in Vault that can be accessed only for unsealing purposes (please correct me on this). When needing an unseal, the script on the user side would then decrypt the key and submit it to Vault for unsealing. It worked well enough, and it felt like being in the movie/game GoldenEye, but nowhere as slick as auto unseal.
Regardless of the setup, yes, at the end of the day any solution is really just pushing the problem further down the stack.
* You can split the key between different persons, and you can even implement "n of k" schemes, like you specify (at key creation time) that you need any 4 out of 9 shards to unseal the vault. You can then keep those shards on separate operator's laptops, in separate backup systems etc.
* You can use a hardware security module to unseal the vault (support for that is not included in the free version, IIRC).
But even if the vault wasn't stored encrypted, it'd still be a huge improvement over "keys on NFS", because only machine administrators get access to the whole DB, and you can limit and audit the access of everybody else in a sane manner.
Smart cards are held by two separate people, each card set is different. Both have to be present to restart the HSM.
Our HSM can require up to five different cards be required for certain operations.
For us, we only require two for normal operations.
For HSM management (key generation, card authentication, etc..) we require a third card set member to be present. This protects against accidental key erasure, or fraud.
Automatic password rotation for AD credentials[1] is especially useful to us.
Meanwhile we use the project level variables Gitlab API[2]
Secrets are kept in Gitlab and requested by applications through a token that can easily be revoked
[1] https://www.vaultproject.io/docs/secrets/ad#password-rotatio...
[2] https://docs.gitlab.com/ee/api/project_level_variables.html
I will be very interested in hearing how others do it.
For context: We run a centralised salt-master, salt master unencrypts content using gpg filters as part of variable generation (salt "pillars"). So it's encrypted at rest and encrypted in our git repositories.
What we do/did, is:
* grab a pair of differently branded USB sticks.
* LUKS encrypt the USB sticks; we used a keyfile which is encrypted on our machines with our GPG key.
* encrypt salt's GPG private key with all of our keys.
* encrypt some of the "irreplacable" private keys (IE: CA roots) with all of our GPG keys.
* store it all on the pair of USB drives
* put the USB drives in a real-life vault, give the keys to the office team.
We haven't needed to recover, but it's clearly documented how to recover if anything went wrong.
It'd be better to take a page out of the cryptocurrency handbook and print the keys out as QR codes (maybe with a simple password so it can only be restored if you know the pw).
- All "hot" keys were stored in an offline credential manager in specific vaults depending on who needed access to them. Only staff with actual clearance could request temporary access to a vault (fully background checked, 1 year employment, etc).
- Copies of each vaulth and our master CA cert were written to 4 encrypted USB sticks. Two stored on-site in the fire-safe and two off-site at our safety deposit box that only c-level staff could access. (We had the same process with our tokens and master logins for AWS).
- Any work using those keys was on a pair-up basis, so at least two people, one doing the work and the other observing.
- We had a detailed policy around this that covered each step in the process and who needs to approve them; everyone who could feasibly need to access the keys was briefed annually as part of our security awareness training.
We handled a LOT of sensitive financial data, so this was the most appropriate way that we could find that maintained both sensible availability and key control.
So in order to get to the keys you needed:
- Access to the fire safe (Senior Ops, Senior Security and C-Level only).
- The LUKS passphrase for the USB sticks (Senior Security and some C-Level only).
- The passphrase for the specific vault (Senior Security and some C-Level only).
I don't know how the passphrases were managed by our sec team, but I know that the C-Level staff had physical envelopes in their home safes.
Seems like we hear more frequently about the actual secrets being stored encrypted (potentially with hardware protection) in a central place, and only the keys to unlock them being distributed like this.
AWS ACM for SSL certs.
AWS SSM for ssh, eliminates the needs for ssh keys.
Not everyone loves AWS (me included), but this stack works nicely in removing the need to ever touch raw encryption key files locally.
If they're dev-keys, I think this is pretty common.
Since it's of interest to HN, I am working on educating our very small team on how keys should be protected and used. I am the youngest developer by about 15 years. It's a very rural company and it often feels like all learning and passion for development stalled around 2005. It's a company that gave me a chance to grow into a development role with no previous experience so I feel indebted to try my best to keep the lights on.
I don't judge too harshly- anyone who has black and white principles on these matters has never worked in any other industry most likely... all you can do is your best to steer the ship and convey the downsides.
I think it's important too because it helps us understand how much friction people will tolerate.
In many cases, even a small amount of friction will cause people to stop functioning completely; I recently tried setting up vault and it was a nightmare, I understand why people avoid picking it up.
That doesn't mean we should not try; we have to become the advocates, arbiters and helpers for those systems.
Good luck, you're not alone.
It should be no surprise that people do insecure stuff under deadline pressure.
This is definitely true, but not actually surprising. It's much easier to notice that, say, a violin performance or plumbing repair is done very badly, than it is to actually do it correctly yourself.
Which also leads to a great deal of exasperation when people (either apparently or actually) don't even notice that what they're doing is insecure. There's a big difference between "yeah, it's broken, but it'd be a huge pain to fix and we'd probably get it wrong anyway, so we'd rather take our chances" versus "there is no problem".
There are multiple automated systems that scan public repos for credentials. 5 minutes later you are mining bitcoins for them.
> When you push to a public repository, GitHub scans the content of the commits for secrets. If you switch a private repository to public, GitHub scans the entire repository for secrets.
> When secret scanning detects a set of credentials, we notify the service provider who issued the secret. The service provider validates the credential and then decides whether they should revoke the secret, issue a new secret, or reach out to you directly, which will depend on the associated risks to you or the service provider.
https://help.github.com/en/github/administering-a-repository...
Not particularly germane to the discussion, but really disappointed in how SendGrid handled things. I notified them immediately, rotated all API tokens, and tey could not turn it off, so the spammer sent messages for days and eventually my SG account got suspended.
IP Rules are super helpful in this case, still need to rotate when exposed but can limit the exposure.
The CEO was not pleased with the 30K (or maybe it was 60K) bill... and I just pointed at the CTO and was like "I fought this battle and was overruled"
It seems convenient until the credentials change (which they ought to now and then). Then when you check out an old revision of the project, it is broken. You end up having to copy and paste the new creds back in time and it's finicky as hell.
I wish you the energy to keep trying to change this, don't let it get to you too much, I know this stuff can be frustrating as hell.
The company solved this by giving me a root username and password that worked on every single important database in the company, at least every customer database.
I had to beg them to create a somewhat restricted account.
The same company was however deeply sceptical to all kinds of remote work. The security equivalent of penny wise pound foolish I guess :-]
If they slack off, at least they do it in the office and not freely at home (imagine the possibilities!)
To keep the charade up I sent one of every 20 requests to them to do for me.
But we aren't allowed internet access on our workstations because "security"
in the last startup I worked, all jwt tokens were created from a 10 letter long shared "secret" stored in json config files all over the place :p
even dev environments had same key lol
Poorly seems to be the answer industry wide. Both encryption and disaster recovery are hard tasks. Combine them and you have recipe for a total mess.
Accessible over USB or HTTP, it supports every major crypto algorithm [1], and keys can be backed up onto another HSM via a wrap key (if they are marked as exportable -- you can also control what can and cannot be exported -- in fact, every operation may be allowed or disallowed per key).
Every operation is logged for audit, of course, and the device may be setup to require logs to be read before they are overwritten. In combination with configuring a special authentication key to access the logs, you can ensure that every operation on the HSM is logged to a remote store before additional operations may be completed.
It does depend on your existing physical security, so that has to be taken into account when designing architectures including it. The micro form factor at least makes it trivial to put into an internal USB port.
And of course, if you require a more enterprise grade tool, you may want to use an HSM in combination with a tool like Hashicorp Vault to manage your keys throughout your orgnaization.
[1] https://developers.yubico.com/YubiHSM2/Product_Overview/
Worth noting accessible using api via REST https://docs.microsoft.com/en-us/rest/api/keyvault/
and powershell https://docs.microsoft.com/en-us/azure/key-vault/secrets/qui...
In general I would suggest using a key vault. AWS, GCP, and Azure all have cloud versions that are backed by virtual HSM's built on top of actual physical HSM's. For the vast majority of usages they're good enough. Use admin account management to enforce 2FA/FIDO for all AWS/Azure/GCP logins. (You should be enforcing 2FA with phone/FIDO auth anyway.)
If you need truly paranoid backups, you can back the key up onto a portable hard drive that you lock in a safe in the closet, with a few key people who know the code.
I recommend against using a (cloud-synced) password manager. Cloud key vaults do the same thing but offer specific features relevant to server stuff. And if you want more paranoia, a physical safe is probably safer than extending your attack surface to a cloud-synced password manager.
Also: make sure that you set up a ~quarterly ritual of opening and verifying the backup. For crucial backup fallback systems, you want to make sure you actually use the system so that you know if it fails.
None of our secret material is particularly hard to revoke or replace; we don't run an internal CA or anything like that.
Usage includes SSH authentication, file encryption (backups and exchanges), git commit signatures and password/secret storage using `pass`.
Copies of the offline master keys keys are stored on flash in safes onsite and offsite in bank vaults, and sub-keys are valid for one year.
We use Hashicorp's Vault for secrets that require automated access.
It's been quite a few years since I interacted with them, but for some keys there is a server somewhere with an HSM installed, and two people have credentials for it. If you need something signed you send it to them, with a justification for why it needs to be signed with the real keys, and they will send you a path to get the signed file, and remind you to delete the signed file when you no longer need it.
This is overkill for some things, and probably would be considered sloppy for others.
There are different CAs for different purposes. There's an intermediate for device management, and another for user or service auth purposes.
All of the key management is "built-in" and managed, so there really isn't much overhead. All software-based, with FIPS certified key management. It's very easy to encrypt data this way. It is expensive though.
Disclaimer: while I used to work for Vormetric / Thales, I no longer do.