Show HN: EnvKey 2.0 – End-To-End Encrypted Environments (now open source) (v2.envkey.com)
I'm so happy to finally show you all this release after years of hard work. I posted the first version of EnvKey to HN back in 2017 (https://news.ycombinator.com/item?id=15330757), then went through YC in W18 (https://news.ycombinator.com/item?id=16569534).
EnvKey is an end-to-end encrypted configuration and secrets manager. It protects your organization's API keys, encryption keys, credentials, and other secrets, and makes it easy to run servers, scripts, tests, and everything else with the latest config. It also helps you avoid duplication in your configuration, react to environment updates in real-time, resolve conflicts smoothly, and a lot more.
You get an intuitive, spreadsheet-like UI for managing environments, along with a developer-friendly CLI that does almost anything the UI can. Running any program in any language with the latest environment variables is as simple as:
envkey-source -- any-shell-command
es -- any-shell-command
es -w -- ./start-server
es -w --rolling -- ./start-server
es -r ./reload-env.sh -- ./start-server
es -r ./env-change-hook.sh
es 'curl https://$HOST_URL'
eval "$(es)"
echo $'\n\neval "$(es --hook bash)"\n' >> ~/.bash_profile
EnvKey's client-side end-to-end encryption is built with the NaCl crypto library. Whether you use EnvKey Cloud or host EnvKey yourself, no configuration or secrets are ever sent to the host running EnvKey in plaintext. Public keys are verified by a web of trust. Invitations are verified out-of-band. Secrets are never accessed through a web browser. More details on security and encryption can be found here: https://docs-v2.envkey.com/docs/security
Let me know what you think! Thanks!
66 comments
[ 3.2 ms ] story [ 117 ms ] threadWould this tool minimize the risk of an employee snapshots the disk and RAM of their machine?
https://en.m.wikipedia.org/wiki/Cold_boot_attack
I’m assuming the client side code isn’t keeping the secrets in some kind of TPM right? ie anyone with root access to the server or the ability to dump the memory could pull them out.
EnvKey's API server has no access to secrets data. But you're right that a server pulling secrets via an ENVKEY access key would still have to expose those secrets in RAM to the process that needs them. We don't claim to protect user devices or connected servers from endpoint compromise.
We do offer some features to help mitigate this though:
- You can limit access to an ENVKEY to specific IPs or CIDR ranges.
- On a user device, secrets are periodically removed from RAM if they haven't been accessed recently.
- On a user device, only encrypted data is stored on disk, and the OS keychain is used on Mac and Windows.
- If an ENVKEY's access it cut off, envkey-source can immediately kill the process, making the secrets harder to access from RAM.
- Audit logs can help you track exactly what was exposed and when in the event of an incident.
- Access to secrets is 'pushed' as far down the stack as possible... so your secrets at least wouldn't be accessible in some cloud provider database--they would need to get access to the running process itself or the machine's RAM.
I looked at the ruby sdk code and it doesn't periodically remove them from RAM. Storing the ENVs in the `ENV` object makes me nervous, because its an easy target for supply attacks to dump the `ENV` to logs / remote servers.
When loading a single environment via an ENVKEY access key, you're correct that there is no RAM eviction (in many cases this could break an app, unless it was designed to handle it).
If you don't want to use ENV/environment variables, you could also output your environment to a file and then pull it into your app and parse it (you can easily export to json, yaml, dotenv, or pam format):
$ es --dot-env > .env
$ es --json > .env.json
$ es --yaml > .env.yaml
$ es --pam > /etc/environment
Looking at the ruby sdk, I don't see any decryption libraries being used. I'd imagine a more secure solution would be to give the sdk the decryption key when the variables need to be decrypted, they are only decrypted by the process in RAM.
Since its ruby, and everything has access to everything in the process, it might make more sense to key management to exist in another process (thus limiting what access simple supply chain attacks have access to).
The language SDKs (including ruby) wrap the envkey-source binary, which is where all the decryption and verification logic lives:
https://docs-v2.envkey.com/docs/envkey-source
There is an -m/--mem-cache flag that you can pass to envkey-source that makes it work just as you're describing. It keeps the variables in RAM and listens for updates, keeping them all up-to-date.
Having to trigger a network call each time the key needs to be used (like to sign a transaction) isn't great though :-/.
Is there a way to handle the configuration in Terraform?
[1] https://www.doppler.com/
That said, I'd say the key difference is that EnvKey places much more emphasis on security and privacy. Doppler is cloud-hosted and doesn't use end-to-end encryption, so it requires that you trust them to keep your data safe and not make any mistakes in their data security, network security, who they hire, what sub-processors they trust, etc. In my (very biased) opinion, that is not sufficient protection for the most sensitive data.
Doppler also uses a web interface, which is great for convenience and UX, but leaves you vulnerable to compromised browser plugins, XSS, etc. I don't believe browsers are sufficiently secure currently for secrets management (if this changes, we'll add a web interface to EnvKey ASAP!).
There are real costs in terms of UX to end-to-end encryption. EnvKey tries to be as user-friendly as possible, but doesn't compromise on security or privacy. Doppler's design, as I understand it, leans more toward the other side of this tradeoff.
On Terraform--yes, it's easy to use EnvKey with Terraform. The general pattern for integrating EnvKey with any tool or platform is to set your ENVKEYs as secrets in whatever way the platform handles them, then expose each ENVKEY as an environment variable to your VMs, containers, or whatever else you're running. We'll write up a Terraform-specific guide soon to make it as easy as possible.
This is a super rare view, but one I truly appreciate. The risk of compromised browser plugins especially is vastly understated and underappreciated by... way too many "security" "experts".
(And yes I know you probably know this better than most as a Sandstorm dev :)
We do recognize that the current security tradeoffs of Doppler aren't going to satisfy everyone. For Enterprises, we offer Enterprise Key Management (EKM), which allows orgs to encrypt their secrets using a cloud KMS. Of course, this still doesn't satisfy every concern. And so, for customers requiring additional security guarantees, stay tuned!
[0] https://docs.doppler.com/docs/security-fact-sheet
Does that answer your question?
In other words, PID 1 is special, because it may have child processes that it never created, and needs to be aware and handle them properly. Otherwise, you can end up leaking zombie processes.
It sounds like envkey-source isn't aware that it may adopt orphaned child processes. Killing them isn't the main issue. Checking their exit statuses is the main issue.
Basically, on unix systems, the command you pass in to envkey-source is run via:
exec.Command("sh", "-c", c)
(c is the command you passed as a string.)
Stdout/stderr is piped through, and .Wait() is called on the command. If envkey-source is in watch mode, it will send a SIGTERM when the environment is updated, then re-run the process once the initial process has died. I can verify that, for example, if a server listening on ports is restarted in this way, the process will die and the ports will be cleared before the new process is started (this has been well-tested).
Do you see a problem with this approach? We will prioritize making all this bulletproof.
Looking at your code, what's missing is a SIGCHLD handler. Basically, your code doesn't know when one of its children dies. You're making an assumption that you know how many children you currently have, based on how many you spawned; but this is misleading due to PID1 semantics re: orphaned processes.
SIGCHLD lets you know that a child process has died. For each SIGCHLD received, your program should (must!) call 'waitpid' (or one of its related functions) to wait on the dead child process. You don't need to waitpid inside the signal handler; you just need to make sure that the counts of signals and waitpid calls eventually match up.
This is in a different language, but here's a nicely writen article about implementing PID1 in Rust:
https://www.fpcomplete.com/rust/pid1/
Someone in Go-land must have written a similar module. Your solution might be an 'import' away.
Is there somewhere I can ping you once we make the changes? Would be great to be sure we haven't missed anything in your estimation.
Edit: made an issue to track this: https://github.com/envkey/envkey/issues/3
I'm not an expert on the topic. Like you said, we're on HN: there's probably five people here who have written PID1 for an actual Unix. :) But I'm happy to take a look.
https://github.com/Yelp/dumb-init
https://github.com/krallin/tini
https://v2.envkey.com/pricing
https://docs-v2.envkey.com/docs/language-sdks
Also, if I'm using "Enterprise Self Hosted", which means I am doing the hosting myself - am I not responsible for all the protection features you outlined there? If it's on my hardware, I'm doing the backups, patches, DDOS protection and so on?
I'm sorry if I sound blunt, but the pricing page looks like you tried to enlist as many buzzwords as possible to sound relevant. Auto-cluster mode sounds like PM2 cluster mode (PM2 being the supervisor tool).
There's many oddities in the pricing page, I won't list them all but I think you can see why I'm skeptic about your software. Vault does a great job dealing with secrets and is less ambiguous about what or how it does it.
The goal is to make running self-hosted, production-ready EnvKey as easy and hands-off as possible. All the details on setup, deployment, and management are here:
https://docs-v2.envkey.com/docs/enterprise-self-hosted
The main differences with Vault are EnvKey's client-side end-to-end encryption, which doesn't trust the host API server under any circumstances, much simpler integration, much easier setup/maintenance for self-hosting (takes about an hour for a scalable, production-ready cluster), and many developer productivity features around managing environments, avoiding duplication, handling environment updates, access control, and a lot more.
All that said, I take your points and we will try to improve our pricing page.
Vault attempts to mitigate this with its 'seal' functionality, but in practice if an attacker gains access to a running Vault server, and/or the cloud provider account it's running in, there are many ways that secrets could be exposed.
EnvKey allows you to avoid trusting the host server, full stop.
Here's a good example: https://github.com/slingamn/vault-exfiltrate
The Vault docs include a list of 'hardening' steps for secure production usage. These are great steps to take, but each one represents a mistake that could be made. And because the Vault process is trusted with plaintext secrets, the stakes are high. Making a mistake could lead to a compromise.
With EnvKey, the host server is never sent secrets in plaintext. For defense in depth, we also follow best practices for hardening our networks. But I think we've seen with Okta and other incidents that despite best intentions, best efforts, and strong engineering, trusting the host server whatsoever just isn't good enough anymore.
For a human user, there are multiple layers of security: email/SSO authentication, a per-device encryption key (stored in the OS credential store on Mac/Windows), and, optionally: a passphrase that locks the per-device encryption key and a lockout that clears RAM and locks EnvKey on the device after a specified time period until the user inputs their passphrase. EnvKey organization owners can require that passphrases/lockouts be set for all org members.
That's because EnvKey doesn't (and can't) completely eliminate the secrets you have to manage, but it does minimize them to a single secret (the ENVKEY) for each environment.
So now instead of setting a bunch of variables in ansible-vault, in Kubernetes secrets, in AWS secrets manager, etc. etc. for every environment that you run, you can just set a single ENVKEY in each of those tools, and then access/update/manage everything in a single place with all the productivity features and additional security that EnvKey offers.
That being said, when looking at your actual website initially with no context, I couldn't really figure out what it did. My first instinct was to hit the "Docs" button.
The "Don't README" header and the just download my code and run it and figure out how it works later mantra with regard to an end-to-end encryption software (or software in general) was very off-putting to me. I actually wasn't very interested in figuring out what your software did at that point, as it just was flashing red-flag in my head. Was it malware? Was it really end-to-end encrypted?
Just a suggestion, you might want to reword some of that front landing page of the Docs page. I understand your wanting to convey that it "just works", but you might want to convey a bit better somehow what exactly what is just working in the first place, and do so in a way that isn't screaming "don't try to figure out how it's working". Maybe also advertise a bit more clearly that it's open source.
The whole "unless you really want to" is a little weird, too. Every end-to-end encryption software you use you should understand fully what encryption is being used, especially when targeting developers handling core secrets for major organizations or what have you.
Edit: what do you think now that I've removed the (probably misguided) playful language in the headings? https://docs-v2.envkey.com/docs Better?
I totally understood what you were trying to do, I just think if I interpreted it that way at first, others might as well. I at least had your other stuff to go off of to eventually get me to figure it out, whereas people encountering your page on the internet wouldn't. Was just trying to help :)
V1 is straightforward to setup, straighforward to use, and the best of it: it just works. Adapting all services to use it is very easy, even if you don't want to change the application itself, as long as uses environment variables, you can just launch it through envkey-source and it just works. I must say it has worked flawlessly for me for the last 3.5 years. So I'm very happy with it. In case anybody is considering to use it, I would encourage to give it a try. Honestly.
I cannot speak too much about the latest version (v2) yet, because we have not "migrated", although for what I've read so far, it looks even better than V1 as a product.
I think the reload functionality might be quite handy when running some services (although not a clue how well it will work in practice), and I always missed the option to have an API or command line to update secrets without opening the GUI. I mean, the GUI is great, but sometimes if you have to update a bunch of stuff (like SSL certificates) on a bunch of services, command line is just more convenient because you can script it or do it all at once.
I also appreciate the fact that it is moving to open-source because that means there is no "lock-in" to the company anymore, and even if the company fails, I can still use the product.
So, having said this, let me add a little of criticism on V2.
I don't like the fact that, as a customer, we have to migrate from V1 to V2. I mean, maybe I've got it wrong, but as far as I understood V1 is going end of life by the end of the year and if I want to continue using the product I have to migrate to V2. I guess that there is probably no way for EnvKey to migrate secrets from V1 to V2 transparently, so clients need to do this one-time migration, but that means changing all EnvKeys on all our services because the current ones will stop working, plus updating all libraries in all our services to V2. It is not like a difficult change to make, but we manage a lot of services and this looks like some sort of "bureaucracy" to go through to change to a major version.
I'm sure this is not a problem for new customers of EnvKey, but as 3.5 year-customer it is something unpleasant to do. Again, it might probably not take us long to do it, but I foresee spending some time with it, and I don't want to focus on a 3rdparty tool maintenance, I want to focus on my customer's problems. Some sort of backwards compatibility, or automatic migration would have been awesome. Dane, if you tell me the same "EnvKeys" will keep working on both V1 and V2, then I'll be amazed and happy, but so far I haven't read any communication about that, so I'm just guessing it won't.
The other thing that changed is pricing. As I mentioned before, I like the fact that it is now open source. Here is the thing, though, I'll be more than happy to continue paying for the service, because hosting it myself means having to spend time on it, which again, I don't want. I want to focus on the problems of my customers, so here comes the thing for me, now there are 4 tiers: OpenSource, Community Cloud (Free), Business Cloud (499/mo) and Enterprise Self-Hosted (1499/mo).
I've been paying 20/mo for the last 3.5 years, which was a nice price for our small team (even cheap). Going from paying 20/mo to 499/mo will make me think on self-hosting or looking for another solution. It exceeds the money I would like to spend on this. I miss something in the middle. I would happily continue paying 20/mo or...
First I'll address the pricing concerns, then the migration from v1 to v2.
The intention with the v2 is that all customers currently paying $20/mo should fit very comfortably on the free Community Cloud tier and never have to worry about hitting usage limits. Given the limits, I believe this will be the case for typical usage patterns encompassing 90% of organizations on this tier, but if it turns out not to be, the limits will be adjusted upward. Does that help to address your concerns?
We'll also add some clarification on user devices and active connections to the pricing page. Those definitely do need to be explained more clearly. I'll write a quick summary here for now.
User devices: unlike v1 which has user-based auth and allows a user to sign into their account from any device, v2 uses device-based authorization. Now when you accept an invitation, just the computer you accept it on will be authorized. To sign in from a different computer, that computer needs to be authorized with a device invitation (these work just like user invitations). Pricing in the v2 is based on the number of authorized devices rather than the number of user accounts.
Active connections: yes, you got this right. Using the new watch/reload functionality of envkey-source maintains an open socket connection to be notified of changes. Signed in user devices that have EnvKey running also use a connection in order to receive organization updates immediately. So active connections = [number of signed in devices with EnvKey running] + [number of active envkey-source watchers].
Now onto the v1 > v2 migration. This was a tough decision. Due to major improvements to the underlying end-to-end encryption libraries and algorithms, v1 and v2 accounts are unfortunately not compatible with each other. I really wish the upgrade could be seamless, but I ended up deciding that faster, more scalable, and more secure encryption was worth the tradeoff in the long run. This process is automated to the extent that is possible given the need to generate new encryption keys in v2. Sadly that does still leave v1 customers with some work to do in order to move over, as you pointed out. I totally understand the frustration here, but am hopeful that the many improvements in v2 will outweigh the one-time cost of switching for the majority of customers.
Having put the time and energy into v1 and recommended it far and wide it's disappointing, but I guess we're not your target market any more.
It is still not clear to me what "40 server ENVKEYs" means. Is this different projects, or each ENVKEY on each of the projects? What counts towards this quota?
I've read a comment from you today (on another thread) about migrating from OpenPGP (RSA) in v1 to NaCl (EC) in v2. So I guess V2 encryption/decryption works faster on the gui/cli and security is stronger. I still would have loved as a customer to have EnvKey done this transparently to me. No idea on the internals, but something in the line of: whenever a customer updates any of its secrets, re-encrypt everything to use V2... but probably given existing architecture/design this is probably either too complex or unfeasible. Which makes me wonder... what would happen if an attack was found on curve25519 or certain type of attack was found? Just wondering, out of curiosity, if the current V2 design would support re-encrypting using a different algorithm (or even another key) in the client-side without other major changes (even if client has to re-encrypt messages from the CLI/GUI). Just wondering.
I've decided I'm going to give it a try to re-import all keys in order to see how a migration would look like and see if I'm hitting any limits beyond the free tier, but if I am, even though I would pay 2-3x what I'm paying now, I think either I'll move to the open-source version or look for something else. In any case I'll drop you an e-mail with my experience.
Coming back to the pricing discussion, as a customer I still like V1 pricing for its simplicity/clarity. You pay per users and that's the end of it. I believe a combination of nº of projects and nº of users might be the way to go for your product, because as a customer is easy to understand and easy to predict, and even if there is a fixed price per user/project then the more projects you add, the more you pay, incrementally... but this is just a thought. Same with the limits... I mean, it would be nice to say, here are the limits, if you surpass them regularly, they would be charged by X amounts.. which is also incremental.
Anyway, thanks for mentioning in another comment you are considering some adjustments. I mean, as drcongo said, maybe we are not your target anymore, maybe we are just a vocal minority, you are the one with all the info anyway. The new pricing might be the right thing for your company, not a clue, although I honestly think there can be something in the middle that even if it gets you marginarlly more money/users at the beginning, might allow your customers to stay and grow as their company grows, which will help you grow as they grow. Final though, the current jump in princing from the free tier to the business tier makes me hesitant to even use the free tier.
And again, the product itself is amazing and I am very happy with it, no complaints at all with it.
- minor edits for clarity -
This sounds more like a traditional PKI than a web of trust. Could you comment further on how this is a web of trust, and who is allowed to sign keys?
Importantly, this is also restricted by the API server in the authorization layer. To invite a new user to an organization, authorize a device, or generate a new ENVKEY for an environment, a user must have appropriate permissions in the organization.
The trusted root is passed along, signed, through the invitation flow or ENVKEY generation, so that all keypairs in the organization agree on the trusted root of the organization, and can verify signature chains back to that root in order to trust a public key. Initially, the trusted root will be the keypair associated with the device that created the organization, but it can be revoked and replaced by another trusted keypair (again, given appropriate API permissions).