I recently ran into a few roadblocks as well when trying to get my python applications set up in a true 12factor way, and I was surprised by how much I had begun to take for granted the ease of configuration that the Go library Viper provided. I went ahead and wrote a library [Gila](https://gitlab.com/dashwav/gila) that implements a similar set of features that I think does a very good job of allowing for 12factor app building in python.
Specific to the article, my way of handling the issue of "turn everything into an environment variable" is to rely on the cascading feature of Gila/Viper and store sane defaults in code that only are overridden when need be. This allows for default values to be set, overridden by config files provided, and further overridden either by code or by ENV variables.
This would allow us to instead of having config written in python (something I usually try to stay away from) we could have it written in something like YAML or TOML instead and loaded in on the fly in each environment, while still allowing for overrides in either the code or by ENV variables.
I recently (last few months) have switched all of my python projects over to using Gila and I have been super impressed by how well the language itself lends to building dockerized 12factor apps when using tools made for that purpose - which is a sentiment I would not have had last year at this time.
This looks great, I'm going to consider this for future projects!
I do a lot of Django, and have found the `django-configurations` package to be pretty good. The thing it's missing that Gila provides is the config file layer, but that's also not _typically_ how Django projects are configured, config is generally done in Python rather than config files.
Yeah I had used django in the past and had run-ins with `django-configurations` and while it is an extremely powerful configuration library, I set out to make Gila a much more portable and simpler API for more generalized use - with the goal that this library can be nearly drop in with 0 configuration in most projects.
I would love to hear any feedback should you decide to use Gila in one of your projects!
Just lightly skimmed the docs, so I am probably missing something. Do all the consumers of gila config use the singleton class instance and just call gila.get() everywhere?
If I'm just focusing on how to make my code easier to read by my fellow developers, I would want to see something like this be based off of a dictionary-style API, that could be imported and used like so:
import gila as config
bucket_name = config["bucket_name"]
Since .get() is already so close to this API, I wonder if you considered this and rejected it for a specific reason?
First off, thank you for looking at the project and giving feedback - I really appreciate it!
- Do all the consumers of gila config use the singleton class instance and just call gila.get() everywhere?
You can either use the singleton pattern (which is the recommended way of utilizing the library) or you can assign the Gila object to a local variable and ship it around your code (this would allow you to have two separate configurations at the same time). You could see that here: https://gitlab.com/dashwav/gila/-/blob/develop/examples/mult...
I also think striving for readability is the way to go when building code in general, and that is actually one of the reasons I opted to keep the `.get()` syntax! This may come down to personal opinions but I think that when the behavior of a library is fundamentally different than that of a base type like dict it should be explicit that a library is being used.
While dictionaries also have a `.get()` syntax I think that seeing `config['value']` would lead a developer to make the assumption that config is simply a dictionary, and this might lead to erroneous code whereas `gila.get("value")` (or even `config.get('value')`) will make it more obvious that this is not a dictionary but actually a library that is doing more than a dictionary under the hood.
As a final point this syntax keeps it closer to the Viper go library that I was inspired by and therefore eases the transition between the two should anyone have familiarity with either library.
I'm curious what the argument is for overriding config files with ENV vars, rather than vice versa. I'm not saying it's wrong, it's just not something I've considered before in terms of prioritisation if you have competing settings in both.
It's a common pattern with container and cloud platforms. Configurations are passed declaratively through whatever yaml interface they expose as environment variables. This way you can have a higher level config that glues all the components together.
Lately, I've switched from doing mostly Python to mostly Go, and I've fallen in love with the Flags-First https://github.com/peterbourgon/ff approach. Basically, you have three levels of config: a conf file, ENV vars, and CLI flags. As you do local dev, you switch stuff on and off with CLI flags, and then when it's time to promote to prod, you switch to having everything as an ENV var or a secrets file or whatever is easy to do in your hosting system. It's very convenient, and Go's standard flags package gives you a simple way to get strongly typed values out of strings.
The python environment story has led me to the conclusion that it's not really suited for serious production software.
I was recently onboarded into a project which is using Python in a server-less setting, and when my colleague walked me through the concept of setting up a virtual environment for each one of my python projects I could not really believe it. It seems like you practically need to containerize a python code-base to make it repeatable.
> led me to the conclusion that it's not really suited for serious production software.
While I get that you may have been frustrated with a Python setup, concluding that it's not suitable for "serious" production software isn't a proportionate response, and the evidence of many companies using it for "serious" production systems suggests that it is in fact fine. Examples include Instagram and Dropbox.
> It seems like you practically need to containerize a python code-base to make it repeatable.
This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary. Arguably even for anything that doesn't produce completely static binaries.
Yes Python has a learning curve for production use, all languages and ecosystems do. In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages. Python is not out of the ordinary, it's not the worst. It's not the best either, it's fairly reasonable for its age, complexity, and typical uses.
> While I get that you may have been frustrated with a Python setup
It's not so much that I was frustrated by this, it's more the case that given the other ecosystems I have worked with, I was kind of shocked that people actually manage to ship projects with this type of tooling.
> This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary
I don't agree. I think npm does an excellent job of making js packages repeatable for example.
> In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages.
To be honest, that's a big part of the reason I'm not a fan of interpreted languages in general in production environments. I think something like Python is fine for a setting like a notebook where you only have to worry about your own workstation, and you just want to iterate quickly without a ton of constraints on performance or robustness, but for running code in an un-supervised environment? I would much rather have a compiler checking every possible issue ahead of time rather than relying on testing and monitoring at runtime.
And Python seems particularly bad, because if I have to use a virtual environment to get the same code to run consistently on my workstation and my colleagues, that just gives me no confidence that I know how it's going to behave in some cloud execution environment. I'm not saying Python is "the worst" but basically all the properties of Python as a language ecosystem seem to run counter to what you want in order to produce robust, reliable production software.
I haven't worked with npm in quite a while but as far as I remember, it does pretty much the same as pip with a virtual env and a requirements file -- it installs all dependencies into a directory for that specific project.
The ergonomics might be better with npm but they both solve the problem of consistent execution across different machines pretty similarly.
The big difference I have seen is that with npm I have never run into issues because the code is being executed with the incorrect version of the interpreter, or something weird like python being installed at an unexpected path.
Also npm is just one tool which behaves consistently and takes full responsibility of making a project portable. With Python it seems as if it's at the whim of the developer to choose from a variety of tools to complete this task.
In other words, if I see `pacakage.json` in my project directory, I know it will tell me everything I need to know to run the project. With Python, if the previous developer hasn't documented what tools they were using, I might have to do some archeology to understand how they have encapsulated the project's dependencies.
Or worse, if they never encapsulated their project's dependencies, I might just have to guess how the environment was set up on their workstation and try to recreate it through trial and error.
As below node has the luxuries of both having been able to learn lessons from python and ruby ~and~ not being a victim of its own success of being already integrated with and a dependency of most major operating systems. (That said javascript the language definitely ~is~ a victim of its own success in browsers - there's probably some metaphorical parallel here..)
But the 'python version' problem (and really just any notion of a 'system python' in general) is still absolutely one of the bigger pain points of the ecosystem. I can say that for nix OS's pyenv ( https://github.com/pyenv/pyenv/ ) is a real godsend for this though. Pure-bash, no bootstrap problem, doesn't need to be 'installed' / can just be cloned and used, and has been able to reliably install any version of python I've needed into any of my relevant platforms in a well isolated ~/.pyenv/version/v3.x.y from which I can '~/.pyenv/versions/v3.x.y/bin/python -m venv .venv' in my project's dev bootstrap/Makefile. It's not part of my production-bound Dockerfiles which just base off of the official python images but it's invaluable for development and testing.
If you are going to use it I definitely recommend cloning it directly to ~/.pyenv rather than installing it via brew or somesuch as those tend to lag further behind the most up-to-date python versions than I'd like. And this is of course in no way a general solution to all of python's many packaging woes, and it'll probably never really be as simple as packages.json for many very valid if unfortunate reasons, but the 'python version' problem is at least pretty well handled by pyenv.
I'm guessing npm also has a nice luxury that nobody writes core parts of the operating system in JavaScript.
Versus, one of Python's bigger challenges with environment management is that you need to come up with an environment isolation mechanism that won't also interfere with the operation of the OS itself, or cut you off from being able to use userland tools that rely on it.
Which. . . Python being an essential part of virtually every production system out there being one of the root causes behind a situation that makes people feel skittish about its appropriateness for use in production is a fun thought. It's a particularly delightfully Unixy brand of ouroboros.
Yes this seems like a maassive problem with a band-aid solution. Like the fact that I might try to change my system Python version to get my Python project working, and that could cause side-effects in completely un-related areas - like maybe my C++ build system is now broken - just seems like an absolutely absurd state of affairs.
It seems crazy to me that the precise Python version isn't defined in the Python file itself, and the interpreter is responsible for sorting out compatibility issues.
With the pip-tools package and pip-compile, having a stable requirements system is pretty straightforward. We have no problems with pyenv and pip locally, and dh-virtualenv in production. We're considering moving to poetry as it brings a load of nice features and more automation, but really it's not been a problem for us.
Node I'd say is about as complicated. You still need to care about the Node version, about as much as the Python version. a node_modules directory is pretty similar to a virtualenv, it's not portable, maybe a little more reproducible by default, but no better than a pip-tools managed requirements file in my experience. Package upgrades are a little trickier than with pip-tools. You still need to care about things like what version of OpenSSL is on your machine, or do you have libpq installed, etc. One problem with JS is whether to compile and if so, how, what do you run, etc. You probably don't want to ship a node_modules to production with all the random package build scripts that it includes, so there's some additional complexity there introducing a build process.
I probably would not choose C++ either unless I really needed its USP's like extreme memory/performance optimization or required compatibility with exotic hardware. But at least with C++ you can compile a binary which won't behave differently on a different system most of the time.
But there are a lot of other languages out there besides C++ and Python.
> But at least with C++ you can compile a binary which won't behave differently on a different system most of the time.
Not really, no. Only if the compiler, compiler version, compilation flags, kernel version, platform, platform version, cpu version, directory structure, shared object versions, locale, environment (LD_PRELOAD, LD_LIBRARY_PATH, TZ), config in pwd, .local, and /etc or maybe $XDG_*, "~/Library/Application Support/" are all the same.
The GP specifically was talking about the binary behaving differently, so the compiler version and compilation flags would be irrelevant. The kernel, system libraries, environment variables, etc. would all still be relevant, though.
I think they're referring to unpredictable behavior under future versions of language VMs and garbage collectors vs. statically compiled binaries with either no GC or a statically linked GC library.
Most of the time, all of that machinery going on behind the scenes is very convenient, but coding strictly to POSIX and the C/C++ standard libraries / STL and keeping your compilation artifacts gives you a lot more stability than coding to some version of the Java standard library (or other language VM) and keeping around your JVM installer with hopes that the installer will agree to install and run on future systems.
I remember being bitten by testing our product against the earliest JRE we supported and also the latest JRE sun had released at the time. I was aware that reading environment variables was deprecated in the earliest JRE I tested with, and I knew it had been un-deprecated in the latest JRE I tested, and the methods I was using were present in all of the JRE versions in between. What I had missed was that there was at least one JRE version in the middle where the deprecated environment variable reading methods actually threw runtime errors.
I've also run into issues in the past with the version of MSVC I was required to use having an installer that depended upon some ancient version of the CLR which would no longer install on my machine. There were some workarounds, but I needed to talk to several of our helpdesk people before I found someone who knew the required magic incantations. Had the MSVC installer been coded strictly to C++ stdlib, STL, Win32 and MFC, it would have run just fine on my modern desktop.
Tbh Ive debugged too many multithreaded c++ programs with memory leaks, corrupt stacks, corrupt heaps, double frees, string algorithms flying off the end of a buffer, etc to take the idea that a C++ program running twice in a row will do the same thing.
I used to think that, end then I just realized that this is really just Python showing its roots as a relatively old programming language with Unix roots.
C and C++ have a a fairly similar problem with dependency hell due to shared libraries being installed in a single system-wide location. They have their own hacks for getting dealing with it. I don't know which way of doing it is technically superior, all I know is none of them offer a developer experience that I'd choose for myself if I were designing a new programming language in this day and age.
FWIW, I give Python some credit for coming up with a solution that doesn't rely on containerization, and therefore will work consistently on many OSes, and not just Linux. Containerization is an option, too. It's really down to whether you need a system that works for one language on many OSes, or one that works for many languages on one OS.
Or, if you'd rather keep it really simple, there are zipapps, which are Python's answer to fat binaries. Similar to uberjars in Java. Or to good ol' static compilation in most the languages I like to use even when someone isn't paying me to do it.
(That said, there is one thing about environment management in Python that absolutely drives me up the wall: There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way. Which sucks for everyone, and makes learning the ropes without a good mentor way more of a chore than it needs to be. That's not too far off from C, either, though.)
> There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way
Yeah I guess this is really my main issue. Like it seems like every Python project needs to come with a README to explain which tools are required, and maybe even which order they have to be applied to get the desired result. It just feels like an almost absurd step back when compared to other language ecosystems where these are well-solved problems.
C and C++ dependency hell comes from prioritizing user experience over dev experience. Imagine if every little tool you used shipped a full fledged system running in a docker container. It would be a nightmare.
As a user it's possible to put up with this, even enjoy it, when there are a handful of tools using it. A container running your webserver and its dependencies like database and caching for instance, that's reasonable.
It makes sense to me that I might want to containerize a project when deploying it to production to guarantee a consistent environment. It seems strange to me that I effectively have to containerize all projects of a certain language on my own workstation because they might have side-effects on each other.
Don't get me wrong, I hate that too. I simply disagree with your assertion that Python isn't production-ready on this basis, because of what a "production" environment typically looks like. Very different from your workstation and mine.
Yeah I mean that's just part of the story. I am also considering the fact that Python's type system and the fact that it's an interpreted language mean that you are going to have to put a lot of effort into testing for issues which can be caught by other languages at compile time, and the fact that you're leaving a lot of performance on the table.
It's not that Python is a "bad language", but all these things taken together just basically work against everything I am trying to achieve in a solid production environment.
The practice is pretty common and a good reflex to have. In your version there would be a risk of a race condition if it was possible for the environment to change between the calls.
Now, it’s not really an issue here with environment variables. But in general, doing assignment and then walking through the cases is safer.
I actually always use the second one but with just `if ENV`, to catch the case where the variable is set but empty, which happens a lot in practice with Docker&co.
It has various options for cascading configuration sources, such that it makes it relatively straightforward to build a hybrid approach in the event that you're attempting to migrate from a file-based config system to a more modern 12-factor approach.
At one point I needed to write a custom config source for Amazon Secrets Manager, and it proved quite easy to implement. Perhaps I should open source that…
> One of my favorite Python features is the way that the files and directories your application is made of map one-to-one with how you import and use them in code.
Funny to see this stated explicitly in this way. In my opinion, this is one of Python's biggest flaws (I'm a big fan of everything but the module system). Paths to files on disk should be treated as string literals, not magic unquoted strings that look like they're language keywords.
You can easily end up in situations where a directory in your project's tree has a name conflict with some library, and this causes issues, which is mind-bogglingly bad design (incidentally: also not a one-to-one map). If you don't live and breathe Python, and accidentally put a hyphen in a filename, God help you.
There's also the thing where you used to need to have a magic __init__.py file for a folder to be a considered as a module, but now it's optional, although I still have no idea what the formula is for when Python decides something is a module…
It's not really optional. There's just this new thing called namespace packages which is not quite the same. I don't fully understand what it's for yet.
Namespace packages let you ship `foo.bar` and `foo.baz` separately. `bar` and `baz` are really two separate packages that are both part of the `foo` namespace. `foo` is not a real package.
They are not new but implicit namespace packages are new-ish. Before 3.3 it was more complicated to set up, now you just omit __init__.py from `foo`.
I don't see any reason to use this if `foo` is shipped as a single package.
I think anything that treats paths on disks/library names as string literals would be workable. So, for instance, the stuff from the importlib standard library would probably be good enough if it was a top level/global construct that worked with quoted strings.
Name conflicts were basically fixed in 2003 with absolute/relative imports ( https://www.python.org/dev/peps/pep-0328/ ) - you don't ever have to include the name of your library in imports within your library using relative imports and via absolute imports they will never conflict with a base-level (site/system) package. Regarding hyphens and such in package names pretty much every language has restrictions on identifier names.
If I create a file in my project named requests.py, then a sibling file's `import requests` starts importing that file, instead of the library. Maybe name conflicts are "basically fixed" in the sense that there are ways to avoid them, but there is still surprising and magical (in the worse possible sense) behavior to trip over.
> Regarding hyphens and such in package names pretty much every language has restrictions on identifier names.
This particular restriction is an artifact of the bad design of treating paths on disk as special language tokens instead of string literals.
If your python files are not in a python package (do not have __init__.py in their directory) then they will be importing non-relatively and resolution depends on sys.path. If you have a multi-file python project in which your source files have inter-dependencies you should be structuring it as [nested] packages not as a directory of scripts that happen to try to talk to each other (and thus be doing 'from . import requests'). It's not that this is a 'way to avoid' a problem, it's just that this is the correct way to do this. I fully agree though that package management remains one of python's biggest pain points, made even worse by just how much bad and long-outdated information is out there, and by how 'easy' it is to just dump code into a .py and run it, not knowing just how wrong that is for anything but a temporary scratch file or a deliberately engineered self-contained single-file portable script like black.
Python doesn't treat 'paths on disk' as special language tokens, python chooses to handle certain tokens by possibly accessing files on disk. Classes in java (lacking 'modules') resolve to equally named files during compilation, as it goes in go, node, haskell, and pretty much everything I can think of with the sole exception of C/C++ preprocessor driven multi-file development (which is even more filesystem-coupled than the former as you literally type filenames).
If it's one thing I find about working efficiently with Python development and execution env, - it's that you should plan for its configuration from day 1 to accomodate items like multiple python versions and to have a well defined idea of what you need (e.g. is python + pip installation enough or do you want to utilize anaconda).
In a cloud native world, code should not be aware of the environment. I don't see a reason to have multiple files per env and not just different configuration. This kinda of creates a mental model against true configurability.
The solution with multiple files actually goes against the 12fapp
"Sometimes apps batch config into named groups (often called “environments”) named after specific deploys, such as the development, test, and production environments in Rails. This method does not scale cleanly"
Nothing is black and white. There is a multi-dimensional scale that your application will land on. The dimensions might be static (yaml) versus dynamic (py) or all data stored outside the app (ENV vars only or similar, devout adherence to the 12 factor religion) versus storing config 100% in your app.
If you plotted those axis' on a four quadrant grid you'd be able to draw a little dot somewhere on it for your particular app. Your app is not my app. My apps are not always the same, either. My personal blog for instance has a far simpler configuration than the tools I build for my clients.
"In a cloud native world, code should not be aware of the environment."
This is a dogmatic fallacy. There are tons of reasons code should be aware of its environment. The problem is that most people are unable to agree on what "aware of its environment" actually means. I do not agree that my code should not know what the staging or production environment is. This is literal devops! Developing and building your application/system with the logistics of shipping it in mind.
When the dust has settled from configuration arguments the most important thing you can do when designing your application is to first replace the word "application" with the word "system" in your mind and begin to think about it as a system. The system contains subsystems: config, persistence, presentation, blob storage, logging, etc... and all of these are self-contained and isolated units that start and stop for their own unique reasons during the app/system lifecycle. If you design things to allow for this decoupling, the right approach for configuration will manifest itself.
Author claims frustration with Django's settings, only to re-implement it almost exactly. How is ENV=production any different from DJANGO_SETTINGS_MODULE=myproject.settings.production?
Sure, by default Django may create a single settings.py, but common practice is to split that into a settings package, containing a base settings module for common settings, and other files for different scenarios, say "development" and "production". Either / all of them can load secrets etc. from the environment, so the "production" settings file is probably better thought of as "deployment" settings, when supporting multiple deployed environments e.g. staging and production.
I don't like to build applications that are 100% coupled to a framework. In my ideal world, Django (if it is being used) is just one puzzle-piece of my application.
Then again it will depend greatly on the surface area of your application. A small app will do fine with a single configuration file or the DJANGO_ approach, but a larger application deserves its own config.
It's been a while since I've been relying on environment variables per-configuration key. I haven't defined profiles such as dev or production for a really long time, with Django projects in particular. I have collected a handful of little secrets that I can share with you.
I do have if DEBUG: based switch all along my settings file, and enforce DEBUG=True in manage.py at the same time: in manage.py set os.environ['DEBUG'] = '1', and in settings.py DEBUG = bool(os.getenv('DEBUG', False)).
My objective is that when a new developer clones the repo, and runs ./manage.py runserver or shell, it will load with all development settings by default.
In theory this could cause the problem that running manage.py commands in production would run them with DEBUG=True. I tried that, and in practice this turned out to never be a problem. But this could certainly bite you if you're not careful.
In particular, I also spread env vars in two config files: docker-compose.persistent.yml for persistent deployments, and docker-compose.ephemeral.yml for ephemeral deployments.
To complete that, because I don't want to store secrets in the git repository, I do have things like that to load multi-line environment variables in CI:
So for example, in PROD_EMAIL_SETTINGS I have a a multiline value with EMAIL_SOME_SETTING=something for every setting that settings.py will fetch from the environment to provision the Python email settings.
And if I want the production email server settings in staging and production, I can load the set of variables that configures for the production email server with that single line.
Anyway, for me the best setting profile management, is no profile management at all: local clone should have development settings by default and just work with 0 config, configuration for deployments in compose files for non-secrets, in CI vars for secrets, and I should be able to easily decide if I need any of the settings that are in production to develop something in which case I end up building a custom profile with environment variables (ie. to debug a blackbox such as a partner API) just "for the time being", for example I have dumped the blackbox request/responses and then build up my unit tests based on that.
I'm extremely happy with that trivial setup that I have in a bunch of projects now for the last years, I thought that's what everybody who's not using a profile-based system was doing, I thought that was the essence of 12factor/config, at least that's what I understood.
Mitigating human errors is certainly important. The brittle nature of your onboarding story is exactly why I wrote this post and encourage the practice in it.
Larger apps that I deploy have a development.py.example file that people can copy to development.py (or fredsmith.py, madonna.py, it doesn't matter. These would not be checked-in to Git)
Then all I need to do before running anything in my project is prefix with ENV=fredsmith.
As a team we could settle on a different name for this variable. We could all have our own environments. We could even decide, as a team, to check them into Git so that we can all see each others config in case there is an issue.
As someone who has been on call for applications with such complicated configuration logic, this looks like a treasure chest of future frustration.
One reason that 12 factor apps use environment variables to store configuration is because then you have a single source of truth on how your application is configured.
Using the principles of the 12 factor app, someone who had no hand in writing the code doesn't have to wrap their heads around whatever convoluted logical path your application takes to figuring out which Redis instance it will connect to, especially during an outage in which their priority is to fix your service as quickly as possible. They can in an emergency SSH into whatever machine is hosting your application or exec into the docker container and inspect the environment to have an almost complete picture of how your application is configured.
The "globals().update(vars(module))" line actually increased my blood pressure. Please show some consideration to your teammates that help you keep your applications running.
Have had nothing but bad experiences keeping apps running that used dotenv in node land and viper in go land. The 12 factor app people did not speak lightly.
I agree with you 100%. There is nothing stopping you from using environment variables in these configuration files. I do it all the time, in fact. If you scroll down to real world examples I share a few examples, one is for connecting to Redis. (grep page for os.environ)
This approach is not mutually exclusive with 12-factor. It is complementary to 12-factor.
> They can in an emergency SSH into whatever machine is hosting your application or exec into the docker container and inspect the environment to have an almost complete picture of how your application is configured.
Totally agree with this too. I build apps with ops in mind and this specific realtime debug ability.
> The "globals().update(vars(module))" line actually increased my blood pressure.
Being a little dramatic here, aren't we? Please elaborate on what is wrong here. I have been writing Python professionally for approaching 12 years, so I am very interested. It is certainly not something you would see every day, but this is not an area of the codebase you visit on a daily basis. Also, as a sidenote, I would strongly encourage you to look under the hood at some of your favorite/popular Python libraries. You'll vomit out your nose.
> Please show some consideration to your teammates that help you keep your applications running.
I forklifted FarmLogs from Heroku to a homebrew Kube cluster in 2016, led operations and support with that tool (40 microservices around the time I left) all while continuing to be an IC and ultimately became the CTO. I have an extreme amount of consideration for others when building software (almost to a fault). Clearly I have some work to do on my writing.
> Have had nothing but bad experiences keeping apps running that used dotenv in node land and viper in go land.
Yes. IMHO viper and dotenv strictly hobby tools that should be used for projects that involve a single developer and a single environment. I agree whole heartedly that they encourage bad practices and make things tough to debug and find.
I cannot tell you how many times I had to matrix my way into a running container and try to figure out what the fuck was going wrong. Let me tell you ... when you have a microservice that depends on 50 environment variables to boot you are not going to be happy. If you can shove as much of that INTO the application as possible (without handicapping yourself or your fellow team mates in the production environments, losing flexibility in deployment target, etc..) why would you not want to do it? That is ultimately what I want people to gain from this.
Sorry if I was abrasive in my response. Let me address two of the points here:
>> The "globals().update(vars(module))" line actually increased my blood pressure.
> Being a little dramatic here, aren't we? Please elaborate on what is wrong here.
and
>> Please show some consideration to your teammates that help you keep your applications running.
The reason I don't like this is that it first dynamically imports the right module and then implicitly sets global variables using the variables from that module using vars. This means that someone couldn't use simple greps or such searches to trace the logic through which a parameter was set (a go-to tool when debugging code someone else put into a docker container).
- - -
> Clearly I have some work to do on my writing.
Your writing is good. I disagree with your approach to solving the problem. Willing to accept that my position is wrong, but am reluctant to do so since it's built on bitter experience.
Applications that load their configuration different depending on "dev/staging/prod" environment are one of my greatest enemies. It's a bizarrely common pattern that creates a whole level of confusion, miscommunication, and errors in figuring out the configuration of applications running on developer computers, in testing, and in production. Even better, since one of the sets of configuration is usually obtained from file or env vars, there's no clear advantage over not having this switch at all. So you sometimes end up with one of the environments used in all cases, often "dev" because that's the one the developers started using.
Your configuration paradigm should be: simple. Environment variables are probably preferable because they're widely supported by different tools (on account of being a "12 factor app" thing for whatever that's worth), but I'd say a file is also just fine, so long as it's easy to figure out where it's being loaded from (I don't want to see any "configuration is merged from files found at the following five paths and then overridden with environment variables..."). The chief concern in operations is being able to quickly and easily determine the set of configuration in use, and strangely enough, developers appreciate this too.
The overhead saved by having some "dev/prod" switch or override scheme is really not overhead at all, if you set one environment variable in your dev/test/prod infrastructure you can set ten.
> It's a bizarrely common pattern that creates a whole level of confusion, miscommunication, and errors in figuring out the configuration of applications running on developer computers, in testing, and in production.
This stems from not having one person or one team design the entire configuration system that will support the entire application.
All the mayhem you are referring to is certainly possible. I have experienced it too. But I would posit that this is not due to having 'named environments' but rather just a poor implementation with no leadership.
I would disagree with you, in that I encounter this exact problem with first-party software on a team of three. Yes, anyone with total knowledge of the configuration system can figure out the configuration state, but my point is that it should be as easy as possible without any complications - because in practice, even if "everyone knows everything" many mistakes and inconsistencies are introduced.
This is one of those things that is tough to discuss wihtout really seeing code. We might be talking about two different things right now. I can certainly understand the issue you are discussing, but I feel like I have been able to avoid it with this approach.
Another thing that I did not put in the post but that I tend to add-in to this concept is a 'dump my env' function. I will occassionally bundle this up and write it out when certain relevant issues or crashes happen.
def get_snapshot():
"""
Provides a snapshot of the current system configuration.
"""
return filter_dict(vars(module), exclude_predicate)
Very handy. From my ipython shell I can just hit config.get_snapshot() and understand the state of the world.
Aside from every other problem, this kind of system is impossible to test, because the infinite configuration combination space.
many people are pretty good at getting test coverage for their code nowadays. But the equally important test coverage for config variations remains a common blind spot.
The whole "env is figured out behind the scenes and work is done to autofetch the correct values to load into the CONSTANT_CASE variables you import all in one line" thing seems to be a pretty clear violation of "Explicit is better than implicit."
In my experience the productivity you gain from such abstractions is negligible compared to the complexity they add to your code.
74 comments
[ 6.0 ms ] story [ 132 ms ] threadSpecific to the article, my way of handling the issue of "turn everything into an environment variable" is to rely on the cascading feature of Gila/Viper and store sane defaults in code that only are overridden when need be. This allows for default values to be set, overridden by config files provided, and further overridden either by code or by ENV variables.
This would allow us to instead of having config written in python (something I usually try to stay away from) we could have it written in something like YAML or TOML instead and loaded in on the fly in each environment, while still allowing for overrides in either the code or by ENV variables.
I recently (last few months) have switched all of my python projects over to using Gila and I have been super impressed by how well the language itself lends to building dockerized 12factor apps when using tools made for that purpose - which is a sentiment I would not have had last year at this time.
https://gitlab.com/dashwav/gila
https://gila.readthedocs.io/en/latest/index.html
I appreciate any feedback either the author or HN in general have as well.
I do a lot of Django, and have found the `django-configurations` package to be pretty good. The thing it's missing that Gila provides is the config file layer, but that's also not _typically_ how Django projects are configured, config is generally done in Python rather than config files.
I would love to hear any feedback should you decide to use Gila in one of your projects!
If I'm just focusing on how to make my code easier to read by my fellow developers, I would want to see something like this be based off of a dictionary-style API, that could be imported and used like so:
Since .get() is already so close to this API, I wonder if you considered this and rejected it for a specific reason?- Do all the consumers of gila config use the singleton class instance and just call gila.get() everywhere?
You can either use the singleton pattern (which is the recommended way of utilizing the library) or you can assign the Gila object to a local variable and ship it around your code (this would allow you to have two separate configurations at the same time). You could see that here: https://gitlab.com/dashwav/gila/-/blob/develop/examples/mult...
I also think striving for readability is the way to go when building code in general, and that is actually one of the reasons I opted to keep the `.get()` syntax! This may come down to personal opinions but I think that when the behavior of a library is fundamentally different than that of a base type like dict it should be explicit that a library is being used.
While dictionaries also have a `.get()` syntax I think that seeing `config['value']` would lead a developer to make the assumption that config is simply a dictionary, and this might lead to erroneous code whereas `gila.get("value")` (or even `config.get('value')`) will make it more obvious that this is not a dictionary but actually a library that is doing more than a dictionary under the hood.
As a final point this syntax keeps it closer to the Viper go library that I was inspired by and therefore eases the transition between the two should anyone have familiarity with either library.
I was recently onboarded into a project which is using Python in a server-less setting, and when my colleague walked me through the concept of setting up a virtual environment for each one of my python projects I could not really believe it. It seems like you practically need to containerize a python code-base to make it repeatable.
While I get that you may have been frustrated with a Python setup, concluding that it's not suitable for "serious" production software isn't a proportionate response, and the evidence of many companies using it for "serious" production systems suggests that it is in fact fine. Examples include Instagram and Dropbox.
> It seems like you practically need to containerize a python code-base to make it repeatable.
This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary. Arguably even for anything that doesn't produce completely static binaries.
Yes Python has a learning curve for production use, all languages and ecosystems do. In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages. Python is not out of the ordinary, it's not the worst. It's not the best either, it's fairly reasonable for its age, complexity, and typical uses.
It's not so much that I was frustrated by this, it's more the case that given the other ecosystems I have worked with, I was kind of shocked that people actually manage to ship projects with this type of tooling.
> This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary
I don't agree. I think npm does an excellent job of making js packages repeatable for example.
> In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages.
To be honest, that's a big part of the reason I'm not a fan of interpreted languages in general in production environments. I think something like Python is fine for a setting like a notebook where you only have to worry about your own workstation, and you just want to iterate quickly without a ton of constraints on performance or robustness, but for running code in an un-supervised environment? I would much rather have a compiler checking every possible issue ahead of time rather than relying on testing and monitoring at runtime.
And Python seems particularly bad, because if I have to use a virtual environment to get the same code to run consistently on my workstation and my colleagues, that just gives me no confidence that I know how it's going to behave in some cloud execution environment. I'm not saying Python is "the worst" but basically all the properties of Python as a language ecosystem seem to run counter to what you want in order to produce robust, reliable production software.
The ergonomics might be better with npm but they both solve the problem of consistent execution across different machines pretty similarly.
Also npm is just one tool which behaves consistently and takes full responsibility of making a project portable. With Python it seems as if it's at the whim of the developer to choose from a variety of tools to complete this task.
In other words, if I see `pacakage.json` in my project directory, I know it will tell me everything I need to know to run the project. With Python, if the previous developer hasn't documented what tools they were using, I might have to do some archeology to understand how they have encapsulated the project's dependencies.
Or worse, if they never encapsulated their project's dependencies, I might just have to guess how the environment was set up on their workstation and try to recreate it through trial and error.
But the 'python version' problem (and really just any notion of a 'system python' in general) is still absolutely one of the bigger pain points of the ecosystem. I can say that for nix OS's pyenv ( https://github.com/pyenv/pyenv/ ) is a real godsend for this though. Pure-bash, no bootstrap problem, doesn't need to be 'installed' / can just be cloned and used, and has been able to reliably install any version of python I've needed into any of my relevant platforms in a well isolated ~/.pyenv/version/v3.x.y from which I can '~/.pyenv/versions/v3.x.y/bin/python -m venv .venv' in my project's dev bootstrap/Makefile. It's not part of my production-bound Dockerfiles which just base off of the official python images but it's invaluable for development and testing.
If you are going to use it I definitely recommend cloning it directly to ~/.pyenv rather than installing it via brew or somesuch as those tend to lag further behind the most up-to-date python versions than I'd like. And this is of course in no way a general solution to all of python's many packaging woes, and it'll probably never really be as simple as packages.json for many very valid if unfortunate reasons, but the 'python version' problem is at least pretty well handled by pyenv.
Versus, one of Python's bigger challenges with environment management is that you need to come up with an environment isolation mechanism that won't also interfere with the operation of the OS itself, or cut you off from being able to use userland tools that rely on it.
Which. . . Python being an essential part of virtually every production system out there being one of the root causes behind a situation that makes people feel skittish about its appropriateness for use in production is a fun thought. It's a particularly delightfully Unixy brand of ouroboros.
It seems crazy to me that the precise Python version isn't defined in the Python file itself, and the interpreter is responsible for sorting out compatibility issues.
Node I'd say is about as complicated. You still need to care about the Node version, about as much as the Python version. a node_modules directory is pretty similar to a virtualenv, it's not portable, maybe a little more reproducible by default, but no better than a pip-tools managed requirements file in my experience. Package upgrades are a little trickier than with pip-tools. You still need to care about things like what version of OpenSSL is on your machine, or do you have libpq installed, etc. One problem with JS is whether to compile and if so, how, what do you run, etc. You probably don't want to ship a node_modules to production with all the random package build scripts that it includes, so there's some additional complexity there introducing a build process.
But there are a lot of other languages out there besides C++ and Python.
Not really, no. Only if the compiler, compiler version, compilation flags, kernel version, platform, platform version, cpu version, directory structure, shared object versions, locale, environment (LD_PRELOAD, LD_LIBRARY_PATH, TZ), config in pwd, .local, and /etc or maybe $XDG_*, "~/Library/Application Support/" are all the same.
I think they're referring to unpredictable behavior under future versions of language VMs and garbage collectors vs. statically compiled binaries with either no GC or a statically linked GC library.
Most of the time, all of that machinery going on behind the scenes is very convenient, but coding strictly to POSIX and the C/C++ standard libraries / STL and keeping your compilation artifacts gives you a lot more stability than coding to some version of the Java standard library (or other language VM) and keeping around your JVM installer with hopes that the installer will agree to install and run on future systems.
I remember being bitten by testing our product against the earliest JRE we supported and also the latest JRE sun had released at the time. I was aware that reading environment variables was deprecated in the earliest JRE I tested with, and I knew it had been un-deprecated in the latest JRE I tested, and the methods I was using were present in all of the JRE versions in between. What I had missed was that there was at least one JRE version in the middle where the deprecated environment variable reading methods actually threw runtime errors.
I've also run into issues in the past with the version of MSVC I was required to use having an installer that depended upon some ancient version of the CLR which would no longer install on my machine. There were some workarounds, but I needed to talk to several of our helpdesk people before I found someone who knew the required magic incantations. Had the MSVC installer been coded strictly to C++ stdlib, STL, Win32 and MFC, it would have run just fine on my modern desktop.
C and C++ have a a fairly similar problem with dependency hell due to shared libraries being installed in a single system-wide location. They have their own hacks for getting dealing with it. I don't know which way of doing it is technically superior, all I know is none of them offer a developer experience that I'd choose for myself if I were designing a new programming language in this day and age.
FWIW, I give Python some credit for coming up with a solution that doesn't rely on containerization, and therefore will work consistently on many OSes, and not just Linux. Containerization is an option, too. It's really down to whether you need a system that works for one language on many OSes, or one that works for many languages on one OS.
Or, if you'd rather keep it really simple, there are zipapps, which are Python's answer to fat binaries. Similar to uberjars in Java. Or to good ol' static compilation in most the languages I like to use even when someone isn't paying me to do it.
(That said, there is one thing about environment management in Python that absolutely drives me up the wall: There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way. Which sucks for everyone, and makes learning the ropes without a good mentor way more of a chore than it needs to be. That's not too far off from C, either, though.)
Yeah I guess this is really my main issue. Like it seems like every Python project needs to come with a README to explain which tools are required, and maybe even which order they have to be applied to get the desired result. It just feels like an almost absurd step back when compared to other language ecosystems where these are well-solved problems.
Generally, newer projects are settling around Poetry.
As a user it's possible to put up with this, even enjoy it, when there are a handful of tools using it. A container running your webserver and its dependencies like database and caching for instance, that's reasonable.
Most serious production workflows are containerized these days -- are you sure that this supports your conclusion?
It's not that Python is a "bad language", but all these things taken together just basically work against everything I am trying to achieve in a solid production environment.
Also checking for environments is done weirdly, why not use
if 'ENV' not in os.environ:
instead of
ENV = os.environ.get("ENV", None)
if ENV is None:
Also it's a bit better from the performance perspective (but not that it also matters in this case).
Then why mention it?
Now, it’s not really an issue here with environment variables. But in general, doing assignment and then walking through the cases is safer.
Why do you say that? It looks like it can handle any number of environments to me.
It has various options for cascading configuration sources, such that it makes it relatively straightforward to build a hybrid approach in the event that you're attempting to migrate from a file-based config system to a more modern 12-factor approach.
At one point I needed to write a custom config source for Amazon Secrets Manager, and it proved quite easy to implement. Perhaps I should open source that…
Funny to see this stated explicitly in this way. In my opinion, this is one of Python's biggest flaws (I'm a big fan of everything but the module system). Paths to files on disk should be treated as string literals, not magic unquoted strings that look like they're language keywords.
You can easily end up in situations where a directory in your project's tree has a name conflict with some library, and this causes issues, which is mind-bogglingly bad design (incidentally: also not a one-to-one map). If you don't live and breathe Python, and accidentally put a hyphen in a filename, God help you.
They are not new but implicit namespace packages are new-ish. Before 3.3 it was more complicated to set up, now you just omit __init__.py from `foo`.
I don't see any reason to use this if `foo` is shipped as a single package.
If the implementation is easy to explain, it may be a good idea.
What would you do instead?
> from './app/utils.py' import numbers
> Regarding hyphens and such in package names pretty much every language has restrictions on identifier names.
This particular restriction is an artifact of the bad design of treating paths on disk as special language tokens instead of string literals.
Python doesn't treat 'paths on disk' as special language tokens, python chooses to handle certain tokens by possibly accessing files on disk. Classes in java (lacking 'modules') resolve to equally named files during compilation, as it goes in go, node, haskell, and pretty much everything I can think of with the sole exception of C/C++ preprocessor driven multi-file development (which is even more filesystem-coupled than the former as you literally type filenames).
The solution with multiple files actually goes against the 12fapp
"Sometimes apps batch config into named groups (often called “environments”) named after specific deploys, such as the development, test, and production environments in Rails. This method does not scale cleanly"
If you plotted those axis' on a four quadrant grid you'd be able to draw a little dot somewhere on it for your particular app. Your app is not my app. My apps are not always the same, either. My personal blog for instance has a far simpler configuration than the tools I build for my clients.
"In a cloud native world, code should not be aware of the environment."
This is a dogmatic fallacy. There are tons of reasons code should be aware of its environment. The problem is that most people are unable to agree on what "aware of its environment" actually means. I do not agree that my code should not know what the staging or production environment is. This is literal devops! Developing and building your application/system with the logistics of shipping it in mind.
When the dust has settled from configuration arguments the most important thing you can do when designing your application is to first replace the word "application" with the word "system" in your mind and begin to think about it as a system. The system contains subsystems: config, persistence, presentation, blob storage, logging, etc... and all of these are self-contained and isolated units that start and stop for their own unique reasons during the app/system lifecycle. If you design things to allow for this decoupling, the right approach for configuration will manifest itself.
Sure, by default Django may create a single settings.py, but common practice is to split that into a settings package, containing a base settings module for common settings, and other files for different scenarios, say "development" and "production". Either / all of them can load secrets etc. from the environment, so the "production" settings file is probably better thought of as "deployment" settings, when supporting multiple deployed environments e.g. staging and production.
Then again it will depend greatly on the surface area of your application. A small app will do fine with a single configuration file or the DJANGO_ approach, but a larger application deserves its own config.
I do have if DEBUG: based switch all along my settings file, and enforce DEBUG=True in manage.py at the same time: in manage.py set os.environ['DEBUG'] = '1', and in settings.py DEBUG = bool(os.getenv('DEBUG', False)).
My objective is that when a new developer clones the repo, and runs ./manage.py runserver or shell, it will load with all development settings by default.
In theory this could cause the problem that running manage.py commands in production would run them with DEBUG=True. I tried that, and in practice this turned out to never be a problem. But this could certainly bite you if you're not careful.
In particular, I also spread env vars in two config files: docker-compose.persistent.yml for persistent deployments, and docker-compose.ephemeral.yml for ephemeral deployments.
To complete that, because I don't want to store secrets in the git repository, I do have things like that to load multi-line environment variables in CI:
So for example, in PROD_EMAIL_SETTINGS I have a a multiline value with EMAIL_SOME_SETTING=something for every setting that settings.py will fetch from the environment to provision the Python email settings.And if I want the production email server settings in staging and production, I can load the set of variables that configures for the production email server with that single line.
Anyway, for me the best setting profile management, is no profile management at all: local clone should have development settings by default and just work with 0 config, configuration for deployments in compose files for non-secrets, in CI vars for secrets, and I should be able to easily decide if I need any of the settings that are in production to develop something in which case I end up building a custom profile with environment variables (ie. to debug a blackbox such as a partner API) just "for the time being", for example I have dumped the blackbox request/responses and then build up my unit tests based on that.
I'm extremely happy with that trivial setup that I have in a bunch of projects now for the last years, I thought that's what everybody who's not using a profile-based system was doing, I thought that was the essence of 12factor/config, at least that's what I understood.
Larger apps that I deploy have a development.py.example file that people can copy to development.py (or fredsmith.py, madonna.py, it doesn't matter. These would not be checked-in to Git)
Then all I need to do before running anything in my project is prefix with ENV=fredsmith.
As a team we could settle on a different name for this variable. We could all have our own environments. We could even decide, as a team, to check them into Git so that we can all see each others config in case there is an issue.
Using different names for the same thing across different teams also looks like something I'd typically strive against.
One reason that 12 factor apps use environment variables to store configuration is because then you have a single source of truth on how your application is configured.
Using the principles of the 12 factor app, someone who had no hand in writing the code doesn't have to wrap their heads around whatever convoluted logical path your application takes to figuring out which Redis instance it will connect to, especially during an outage in which their priority is to fix your service as quickly as possible. They can in an emergency SSH into whatever machine is hosting your application or exec into the docker container and inspect the environment to have an almost complete picture of how your application is configured.
The "globals().update(vars(module))" line actually increased my blood pressure. Please show some consideration to your teammates that help you keep your applications running.
Have had nothing but bad experiences keeping apps running that used dotenv in node land and viper in go land. The 12 factor app people did not speak lightly.
This approach is not mutually exclusive with 12-factor. It is complementary to 12-factor.
> They can in an emergency SSH into whatever machine is hosting your application or exec into the docker container and inspect the environment to have an almost complete picture of how your application is configured.
Totally agree with this too. I build apps with ops in mind and this specific realtime debug ability.
> The "globals().update(vars(module))" line actually increased my blood pressure.
Being a little dramatic here, aren't we? Please elaborate on what is wrong here. I have been writing Python professionally for approaching 12 years, so I am very interested. It is certainly not something you would see every day, but this is not an area of the codebase you visit on a daily basis. Also, as a sidenote, I would strongly encourage you to look under the hood at some of your favorite/popular Python libraries. You'll vomit out your nose.
> Please show some consideration to your teammates that help you keep your applications running.
I forklifted FarmLogs from Heroku to a homebrew Kube cluster in 2016, led operations and support with that tool (40 microservices around the time I left) all while continuing to be an IC and ultimately became the CTO. I have an extreme amount of consideration for others when building software (almost to a fault). Clearly I have some work to do on my writing.
> Have had nothing but bad experiences keeping apps running that used dotenv in node land and viper in go land.
Yes. IMHO viper and dotenv strictly hobby tools that should be used for projects that involve a single developer and a single environment. I agree whole heartedly that they encourage bad practices and make things tough to debug and find.
I cannot tell you how many times I had to matrix my way into a running container and try to figure out what the fuck was going wrong. Let me tell you ... when you have a microservice that depends on 50 environment variables to boot you are not going to be happy. If you can shove as much of that INTO the application as possible (without handicapping yourself or your fellow team mates in the production environments, losing flexibility in deployment target, etc..) why would you not want to do it? That is ultimately what I want people to gain from this.
>> The "globals().update(vars(module))" line actually increased my blood pressure.
> Being a little dramatic here, aren't we? Please elaborate on what is wrong here.
and
>> Please show some consideration to your teammates that help you keep your applications running.
The reason I don't like this is that it first dynamically imports the right module and then implicitly sets global variables using the variables from that module using vars. This means that someone couldn't use simple greps or such searches to trace the logic through which a parameter was set (a go-to tool when debugging code someone else put into a docker container).
- - -
> Clearly I have some work to do on my writing.
Your writing is good. I disagree with your approach to solving the problem. Willing to accept that my position is wrong, but am reluctant to do so since it's built on bitter experience.
Your configuration paradigm should be: simple. Environment variables are probably preferable because they're widely supported by different tools (on account of being a "12 factor app" thing for whatever that's worth), but I'd say a file is also just fine, so long as it's easy to figure out where it's being loaded from (I don't want to see any "configuration is merged from files found at the following five paths and then overridden with environment variables..."). The chief concern in operations is being able to quickly and easily determine the set of configuration in use, and strangely enough, developers appreciate this too.
The overhead saved by having some "dev/prod" switch or override scheme is really not overhead at all, if you set one environment variable in your dev/test/prod infrastructure you can set ten.
This stems from not having one person or one team design the entire configuration system that will support the entire application.
All the mayhem you are referring to is certainly possible. I have experienced it too. But I would posit that this is not due to having 'named environments' but rather just a poor implementation with no leadership.
Another thing that I did not put in the post but that I tend to add-in to this concept is a 'dump my env' function. I will occassionally bundle this up and write it out when certain relevant issues or crashes happen.
Very handy. From my ipython shell I can just hit config.get_snapshot() and understand the state of the world.many people are pretty good at getting test coverage for their code nowadays. But the equally important test coverage for config variations remains a common blind spot.
In my experience the productivity you gain from such abstractions is negligible compared to the complexity they add to your code.