Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
- On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
- IMO, this has become an even more pronounced problem with the popularity of running non-containerized LLM agents in the same user space as the developer's primary OS user. It's a secret exfiltration exploiter's dream.
- Environment variables are usually passed down to other spawned processes instead of being contained to the primary process which is often the only one that needs it.
- Systemd exposes unit environment variables to all system clients through DBUS and warns against using environment variables for secrets[1]. I believe this means non-root users have access to environment variables set for root-only units/services. I could be wrong, I haven't tested it yet. But if this is the case, I bet that's a HUGE surprise to many system admins.
I think ephemeral file sharing between a secret managing process (e.g. 1Password's op cli tool) and the process that needs the secrets (flask, terraform, etc.) is likely the only solution that keeps secrets out of files and also out of environment variables. This is how Systemd's credentials system works. But it's far from widely supported.
Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Edit: I think 1Password's op client has a good start in that each new "session" requires my authorization. So I can enable that tool for a cli sessions where I need some secrets but a rogue process that just tries to use the `op` binary isn't going to get to piggyback on that authorization. I'd get a new popup. But this is only step 1. Step 2 is...how to share that secret with the process that needs it and we are now back to the discussion above.
> On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
This is a very good point I'd never realised! I'm not sure how you get around it, though, as if that program can even find a credential and decrypt a file, if it runs as the user then everything else can go find that credential as well.
--passphrase-fd was always the correct thing to do (gpg has had it for ages), but support for that is very meager. I mean, you can’t even kubectl login using environment variables, you pretty much have to pass tokens through command-line arguments and we’ve known that’s terrible for around forty-five years.
Linux security model is pretty broken without namespaces. systemd has some bells and whistles that help but if you want something better than environment variables you're naturally going to reach for cgroups.
I haven't seen much language support for it, though. On one part maybe because it's Linux only.
People that write in Rust (and maybe Go, depends how easy FFI is) should give it a try.
I wanted for a time to get some support for it in PHP, since wrapping a C function should be easy, but the thought of having to also modify php-fpm put a dent in that. I can't and don't want to hack on C code.
In practice it'd be great if the process manager spawn children after opening a secret file descriptor, and pass those on. Not in visible memory, not in /proc/*/environ
Since at least 2012, environment variables have been at least as secure as ordinary memory:
commit b409e578d9a4ec95913e06d8fea2a33f1754ea69
Author: Cong Wang <xiyou.wangcong@gmail.com>
Date: Thu May 31 16:26:17 2012 -0700
proc: clean up /proc/<pid>/environ handling
You can't read another process's environment unless you can ptrace-read the process, and if you can ptrace-read the process you know all its secrets anyway.
> Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
I think environment variables are recommended to pass configuration parameters, and also secrets, in containerized applications managed by container orchestration systems.
By design, other processes cannot inspect what environment variables are running in a container.
Also, environment variables are passed to child processes because, by design, the goal is to run child processes in the same environment (i.e., same values) as the parent process, or with minor tweaks. Also, the process that spawns child processes is the one responsible for set it's environment variables, which means it already has at least read access to those secrets.
All in all I think all your concerns are based on specious reasoning, but I'll gladly discuss them in finer detail.
> - On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
Here's the thing though, the security model of most operating systems means that running a process as a user is acting as that user. There are some caveats (FreeBSD has capsicum, Linux has landlock, SELinux, AppArmor, Windows has integrity labels), but in the general case, if you can convince someone to run something, the program has delegated authority to act on behalf of that user. (And some user accounts on many systems may have authority to impersonate others.)
While it's by no means the only security model (there exists fully capability based operating systems out there), it's the security model that is used for most forms of computing, for better or worse. One of the consequences of this is that you can control anything within your domain. I can kill my own processes, put them to sleep, and most importantly for this, debug them. Anything I own that has secrets can grab them my ptrace/process_vm_readv/ReadProcessMemory/etc.
> Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Honestly, my answer is still systemd-creds. It's easy to use and avoids the problem that plain environment variables have. It's a few years old by now, should be available on popular distros. Although credential support for user-level systemd services was added just a few weeks ago.
A TL;DR example of systemd-creds for anyone reading this:
# Run the initial setup
systemd-creds setup
# This dir should have permissions set to 700 (rwx------).
credstore_dir=/etc/credstore.encrypted
# For user-level services:
# credstore_dir="$HOME/.config/credstore.encrypted"
# Set the secret.
secret=$(systemd-ask-password -n)
# Encrypt the secret.
# For user-level services, add `--user --uid uidhere`.
# A TPM2 chip is used for encryption by default if available.
echo "$secret" | systemd-creds encrypt \
--name mypw - "$credstore_dir/mypw.cred"
chmod 600 "$credstore_dir/mypw.cred"
The process you start in the service will then be able to read the decrypted credential from the ephemeral file `$CREDENTIALS_DIR/mypw`. The environment variable is set automatically by systemd. You can also use the command `systemd-creds cat mypw` to get the value in a shell script.
At least systemd v250 is required for this. v258 for user-level service support.
You've described the classic Unix security model, with minor improvements. While decent for the time it is showing its age. Specifically the difficulty in adapting to cheap, ubiquitous computing which it wasn't designed for.
If you need to keep secrets from other processes—don't run them under the same user account. Or access them remotely, although that brings other tradeoffs and difficulties.
> Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Plain text files are fine, it's the permissions on that file that is a problem.
The best way is to be in control of the source to the program you want to run then you can make a change to always protect against leaking secrets by starting the program as a user that can read the secrets field. After startup the program reads the whole file and immediately drops privileges by switching to a user that cannot read the secrets file.
Interesting read. Another interesting fact is that `setenv()` is fundamentally broken on POSIX, and should essentially never be called in library code. In application code, it should be called only in the absence of any alternative, and certainly before any threads have started.
The reason is that `getenv()` hands out raw pointers to the variables, so overwriting a variable using `setenv()` is impossible to guard against. Treat with extreme caution.
> POSIX-specified utilities use uppercase envvars, but that’s not prescriptive for your programs. Quite the contrary: you’re encouraged to use lowercase for your envvars so they don’t collide with the standard tools.
> But in reality, not many applications use lowercase. The proper etiquette in software development is to use ALL_UPPERCASE
I always prefer lower case for env variables in scripts. Thanks for pointing out that it can help reduce clashes with standard tools.
I get anxiety whenever I need to set an environment var on Linux. There are (somewhat distro-specific) ways that work properly, but the usual procedures you find online stops working once you reboot (or close the terminal I think?). They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific. Windows has the annoyance of needing to restart the the terminal (or open a new one) for changes to take effect, but works well other than that.
This is much simpler than having to remember where the dang environment variables button is in "advanced system settings" or whatever awful command prompt syntax sets it permanently.
We're almost rid of dotfiles in $HOME. Can we please also get rid of the abuse of environment variables for configuration? There is no reason your program needs to muck up the environment table of every process by defining MY_PROGRAM_API_KEY instead of taking a command line argument or reading a configuration file. It's not "secure" just because it's not on the command line. And it will mess up for users because it's nigh impossible to ensure you have the same environment variables over all login types. The variable might be there when you run locally, but not in an ssh session or cron. Some ephemeral configuration such as LD_LIBRARY_PATH and PWD is difficult to handle without environment variables, but those are rare exceptions and not the norm.
Contribute to a clean environment! Don't use environment variables for configuration! </rant>
...and people have to find special ways to set them. I didn't even know that pam_env existed until a recent security vulnerability announcement. It's never come to my attention before, I don't think I've ever seen it utilized. I've now made a runbook item to disable it. It's a shame that "configuration" includes undoing the fetishes and helpfulness [0][1] of others.
pam-config -d --env
[0] Crowded elevator atrium. Multiple elevators running. Elevator wants to close, another one is coming (oh! I heard it "ding"!). Somebody is holding the elevator which wants to depart and trying to wave me in. Why doesn't somebody push them out?
[1] I'm at a stop sign. Some complete idiot is trying to turn left onto the street I'm leaving and waving me to turn left in front of them. Fuck no! I turn in front of you, somebody rearends you, you fly forward into me: my fault! You should be able to make this turn, if you can't go around the block! [2]
[2] I could go out of my way and turn right. Or I can just wait and see what happens.
SRE/Sysadmin/DevOps/Whatever here, while blog didn't talk about doing anything difficult but setting ENVVAR standards, I will point out all replacements are just as frustrating especially when talking about secrets.
Anything involving vaults where Application reaches out to specific secret vault like Hashicorp Vault/OpenBao/Secrets Manager quickly becomes massive vendor lock in where replacement is very difficult due to library replacement and makes vault uptime extremely important. This puts Ops in extremely difficult place when it becomes time to upgrade or do maintenance.
Config files have problem of you have secrets, how do you get them into config file since config files are generally kept in public systems? Most of the time it's some form of either "Template replacement by privileged system before handing it to application" or "Entire Config File gets loaded into secret vault and passed into application". Templating can be error prone and loading entire config files into secret manager is frustrating as well since someone could screw up the load.
Speaking of config files, since most systems are running containers, and unless you are at Ops discipline company, these config files are never in the right place, it becomes error prone for Ops to screw up the mounting. Also, whatever format you use, JSON/YAML/TOML is ripe for some weird config file bug to emerge like Norway problem in YAML.
Getting secrets from Kubernetes Secrets API I've seen done but lock in again. I'd strongly not recommend this approach unless you are designing a Kubernetes operator or other such type system.
I will say I've seen Subprocess thing bite people but I've seen less and less subprocess generation these days. Most teams go with message bus type system instead of sub processing since it's more robust and allows independent scaling.
Eh, they are just more command line arguments, ones that go on the left side of the command instead of the right. I guess an alternative is something the Windows registry, but I'm not seeing that as a great improvement since it's less direct.
I can't recommend Varlock [1] enough. A great way to manage env vars in a project. It lets you define which ones are necessary or optional, their types, and where they should be fetched from.
>"Wow, I really enjoyed writing this…
…and I hope it wasn’t a boring read."
No, it was very interesting actually!
An excellent deep-dive into the murky area of Unix/Linux environment variables, how they are set, how they are passed, what's really going on behind the scenes.
Basically a must-read for any present or future OS designer...
Observation (if I might!) -- environment variables are to programs (especially chains of programs, parent processes, sub processes and sub-sub processes, etc.) what parameters are to functions -- and/or what command-line parameters are... they're sort of like global variables that get passed around a lot...
They also can influence program behavior -- and thus the determinism of a program -- and thus the determinism of a chain of programs...
Phrased another way -- software that works on one developer's machine might not work on another developer's machine and/or in a production environment because one crucial environment variable was either set or not set, or set to the wrong value...
(NixOS seems to understand this pretty well... that "hermetically sealing" or closing or "performing a closure around" (that's probably slightly the wrong language/terminology in "Nix-speak" but bear with me!) a software environment, including the environment variables is the way to create deterministic software builds that run on any machine... but here I'm digressing...)
But my main point: I complete agree with the article's author -- environment variables are a legacy mess!
Although, if we think about it, environment variables (if we broaden the definition) are a sub-pattern of anything that affects the state of an individual machine or runtime environment -- in other words, things such as the Windows Registry, at least the global aspects of it -- are also in the same category.
Future OS's, when they offer environments for programs or chains of programs to run -- should be completely containerized -- that is, the view of the system -- what data/settings/environment variables/registry variables/files/syscalls/global variables it has access to -- should be completely determinable by the user, completely logabble, completely transparent, and completely able to be compared, one environment to another.
In this way, software that either a) fails to work at all b) works in a non-deterministic way -- can be more easily debugged/diagnosed/fixed (I'm looking at you, future AI's that will assist humans in doing this!) -- then if all of that information is in various different places, broken, and/or opaque...
To reiterate:
>"Wow, I really enjoyed writing this…
…and I hope it wasn’t a boring read."
No, I really enjoyed reading it(!), it's a brilliant article, and thank you for writing it! :-)
At a past firm, I was trying to debug how a particular ENV var was getting set. I started out thinking it was something simple like the user's .bashrc or equivalent.
I quickly realized that there were roughly 10 "layers" of env var loadings happening where the first couple layers were:
- firm wide
- region
- business unit
- department
- team etc etc
I ended up having to turn on a bash debug flag so that I could see exactly where the var was getting set in the layer stack.
static secrets that you can even extract and set in a config file (or store in a secrets manager) need to go. TPM secrets that you can't extract, and public keys that you can pin without considering them "secret", along with Oauth / IAM roles, are the way to go.
Environment variables also suck due to having zero typo existence. Write FOO_BAR instead of FOO_BARS? Silent failure. V1 of your package recognizes FOO_BAR and V2 changes it to FOO_BARS (because now we can have more than one)? Silent failure.
One of the worst things about Environment variables among others discussed here is the implicit and opaque nature of them. Majority of applications rely on them in the *nix world. Even if more explicit and obvious ways of configuration files or remote services (consul/etcd, et al.) and command line arguments are supported env vars are traditionally supported as well.
But as mentioned in the article it is just a global hashmap that can be cloned and extended for child processes. Maybe in 1979 it was a good design decision. Today it sometimes hurts.
For example, kubernetes by default literally pollutes the container’s environment with so-called service links. And you will have fun time debugging a broken application if any of those “default” env vars conflict with the env vars that your app might expect to see.
They are ubiquitous and we are living in the world of neo-conservatism in IT where legacy corner cuts are treated as a standard and never challenged (hello /bin, /usr/bin, /lib, /usr/lib)[0]
Really, really interesting read. I use env vars daily, and it never occurred to me how they actually work. It shows how many things we take for granted have such interesting implementations. :)
My favorite environment variable trivium (that's the singular of trivia) is that PS1 and a few others that "everyone" thinks of as environment variables are not environment variables but shell variables. PS1 doesn't even show up in "env" output!
45 comments
[ 2.5 ms ] story [ 64.3 ms ] thread- On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
- IMO, this has become an even more pronounced problem with the popularity of running non-containerized LLM agents in the same user space as the developer's primary OS user. It's a secret exfiltration exploiter's dream.
- Environment variables are usually passed down to other spawned processes instead of being contained to the primary process which is often the only one that needs it.
- Systemd exposes unit environment variables to all system clients through DBUS and warns against using environment variables for secrets[1]. I believe this means non-root users have access to environment variables set for root-only units/services. I could be wrong, I haven't tested it yet. But if this is the case, I bet that's a HUGE surprise to many system admins.
I think ephemeral file sharing between a secret managing process (e.g. 1Password's op cli tool) and the process that needs the secrets (flask, terraform, etc.) is likely the only solution that keeps secrets out of files and also out of environment variables. This is how Systemd's credentials system works. But it's far from widely supported.
Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Edit: I think 1Password's op client has a good start in that each new "session" requires my authorization. So I can enable that tool for a cli sessions where I need some secrets but a rogue process that just tries to use the `op` binary isn't going to get to piggyback on that authorization. I'd get a new popup. But this is only step 1. Step 2 is...how to share that secret with the process that needs it and we are now back to the discussion above.
1: https://www.freedesktop.org/software/systemd/man/latest/syst...
This is a very good point I'd never realised! I'm not sure how you get around it, though, as if that program can even find a credential and decrypt a file, if it runs as the user then everything else can go find that credential as well.
https://pypi.org/project/keyring/
memfd_secret comes to mind https://man7.org/linux/man-pages/man2/memfd_secret.2.html
I haven't seen much language support for it, though. On one part maybe because it's Linux only.
People that write in Rust (and maybe Go, depends how easy FFI is) should give it a try.
I wanted for a time to get some support for it in PHP, since wrapping a C function should be easy, but the thought of having to also modify php-fpm put a dent in that. I can't and don't want to hack on C code.
In practice it'd be great if the process manager spawn children after opening a secret file descriptor, and pass those on. Not in visible memory, not in /proc/*/environ
cmdline is a different story.
Something like: my_secret = create_secret(value)
Then ideally it's an opaque value from that point on
I think environment variables are recommended to pass configuration parameters, and also secrets, in containerized applications managed by container orchestration systems.
By design, other processes cannot inspect what environment variables are running in a container.
Also, environment variables are passed to child processes because, by design, the goal is to run child processes in the same environment (i.e., same values) as the parent process, or with minor tweaks. Also, the process that spawns child processes is the one responsible for set it's environment variables, which means it already has at least read access to those secrets.
All in all I think all your concerns are based on specious reasoning, but I'll gladly discuss them in finer detail.
Here's the thing though, the security model of most operating systems means that running a process as a user is acting as that user. There are some caveats (FreeBSD has capsicum, Linux has landlock, SELinux, AppArmor, Windows has integrity labels), but in the general case, if you can convince someone to run something, the program has delegated authority to act on behalf of that user. (And some user accounts on many systems may have authority to impersonate others.)
While it's by no means the only security model (there exists fully capability based operating systems out there), it's the security model that is used for most forms of computing, for better or worse. One of the consequences of this is that you can control anything within your domain. I can kill my own processes, put them to sleep, and most importantly for this, debug them. Anything I own that has secrets can grab them my ptrace/process_vm_readv/ReadProcessMemory/etc.
Honestly, my answer is still systemd-creds. It's easy to use and avoids the problem that plain environment variables have. It's a few years old by now, should be available on popular distros. Although credential support for user-level systemd services was added just a few weeks ago.
A TL;DR example of systemd-creds for anyone reading this:
You can now configure your unit file, e.g.: The process you start in the service will then be able to read the decrypted credential from the ephemeral file `$CREDENTIALS_DIR/mypw`. The environment variable is set automatically by systemd. You can also use the command `systemd-creds cat mypw` to get the value in a shell script.At least systemd v250 is required for this. v258 for user-level service support.
If you need to keep secrets from other processes—don't run them under the same user account. Or access them remotely, although that brings other tradeoffs and difficulties.
Plain text files are fine, it's the permissions on that file that is a problem.
The best way is to be in control of the source to the program you want to run then you can make a change to always protect against leaking secrets by starting the program as a user that can read the secrets field. After startup the program reads the whole file and immediately drops privileges by switching to a user that cannot read the secrets file.
Works for more than just secrets too.
The reason is that `getenv()` hands out raw pointers to the variables, so overwriting a variable using `setenv()` is impossible to guard against. Treat with extreme caution.
> But in reality, not many applications use lowercase. The proper etiquette in software development is to use ALL_UPPERCASE
I always prefer lower case for env variables in scripts. Thanks for pointing out that it can help reduce clashes with standard tools.
This is much simpler than having to remember where the dang environment variables button is in "advanced system settings" or whatever awful command prompt syntax sets it permanently.
Contribute to a clean environment! Don't use environment variables for configuration! </rant>
[1] I'm at a stop sign. Some complete idiot is trying to turn left onto the street I'm leaving and waving me to turn left in front of them. Fuck no! I turn in front of you, somebody rearends you, you fly forward into me: my fault! You should be able to make this turn, if you can't go around the block! [2]
[2] I could go out of my way and turn right. Or I can just wait and see what happens.
Anything involving vaults where Application reaches out to specific secret vault like Hashicorp Vault/OpenBao/Secrets Manager quickly becomes massive vendor lock in where replacement is very difficult due to library replacement and makes vault uptime extremely important. This puts Ops in extremely difficult place when it becomes time to upgrade or do maintenance.
Config files have problem of you have secrets, how do you get them into config file since config files are generally kept in public systems? Most of the time it's some form of either "Template replacement by privileged system before handing it to application" or "Entire Config File gets loaded into secret vault and passed into application". Templating can be error prone and loading entire config files into secret manager is frustrating as well since someone could screw up the load.
Speaking of config files, since most systems are running containers, and unless you are at Ops discipline company, these config files are never in the right place, it becomes error prone for Ops to screw up the mounting. Also, whatever format you use, JSON/YAML/TOML is ripe for some weird config file bug to emerge like Norway problem in YAML.
Getting secrets from Kubernetes Secrets API I've seen done but lock in again. I'd strongly not recommend this approach unless you are designing a Kubernetes operator or other such type system.
I will say I've seen Subprocess thing bite people but I've seen less and less subprocess generation these days. Most teams go with message bus type system instead of sub processing since it's more robust and allows independent scaling.
[1] https://varlock.dev/
No, it was very interesting actually!
An excellent deep-dive into the murky area of Unix/Linux environment variables, how they are set, how they are passed, what's really going on behind the scenes.
Basically a must-read for any present or future OS designer...
Observation (if I might!) -- environment variables are to programs (especially chains of programs, parent processes, sub processes and sub-sub processes, etc.) what parameters are to functions -- and/or what command-line parameters are... they're sort of like global variables that get passed around a lot...
They also can influence program behavior -- and thus the determinism of a program -- and thus the determinism of a chain of programs...
Phrased another way -- software that works on one developer's machine might not work on another developer's machine and/or in a production environment because one crucial environment variable was either set or not set, or set to the wrong value...
(NixOS seems to understand this pretty well... that "hermetically sealing" or closing or "performing a closure around" (that's probably slightly the wrong language/terminology in "Nix-speak" but bear with me!) a software environment, including the environment variables is the way to create deterministic software builds that run on any machine... but here I'm digressing...)
But my main point: I complete agree with the article's author -- environment variables are a legacy mess!
Although, if we think about it, environment variables (if we broaden the definition) are a sub-pattern of anything that affects the state of an individual machine or runtime environment -- in other words, things such as the Windows Registry, at least the global aspects of it -- are also in the same category.
Future OS's, when they offer environments for programs or chains of programs to run -- should be completely containerized -- that is, the view of the system -- what data/settings/environment variables/registry variables/files/syscalls/global variables it has access to -- should be completely determinable by the user, completely logabble, completely transparent, and completely able to be compared, one environment to another.
In this way, software that either a) fails to work at all b) works in a non-deterministic way -- can be more easily debugged/diagnosed/fixed (I'm looking at you, future AI's that will assist humans in doing this!) -- then if all of that information is in various different places, broken, and/or opaque...
To reiterate:
>"Wow, I really enjoyed writing this… …and I hope it wasn’t a boring read."
No, I really enjoyed reading it(!), it's a brilliant article, and thank you for writing it! :-)
Upvoted and favorited!
At a past firm, I was trying to debug how a particular ENV var was getting set. I started out thinking it was something simple like the user's .bashrc or equivalent.
I quickly realized that there were roughly 10 "layers" of env var loadings happening where the first couple layers were:
- firm wide
- region
- business unit
- department
- team etc etc
I ended up having to turn on a bash debug flag so that I could see exactly where the var was getting set in the layer stack.
My guess would be either Jekyll or hand rolled, due to the url structure.
Posted from a productive Windows workstation.
But as mentioned in the article it is just a global hashmap that can be cloned and extended for child processes. Maybe in 1979 it was a good design decision. Today it sometimes hurts.
For example, kubernetes by default literally pollutes the container’s environment with so-called service links. And you will have fun time debugging a broken application if any of those “default” env vars conflict with the env vars that your app might expect to see.
https://kubernetes.io/docs/tutorials/services/connect-applic...
They are ubiquitous and we are living in the world of neo-conservatism in IT where legacy corner cuts are treated as a standard and never challenged (hello /bin, /usr/bin, /lib, /usr/lib)[0]
[0] - https://askubuntu.com/a/135679