I'd hazard a guess that it is because Python is so old. It was literally created in the 80s. Package management, explosion of dependencies, multiple implementations of the interpreter, being added to the base operating system, and all that fancy stuff came after. I remember doing versioning and packaging in C++ in early 2000s, and it being just as annoying but for different reasons.
People are trying to fix it, and their are some great packages out there that make it easier. Pyenv, Poetry, this asdf tool the fine article mentions, all seem like good solutions.
>I'd hazard a guess that it is because Python is so old. It was literally created in the 80s. Package management, explosion of dependencies, multiple implementations of the interpreter, being added to the base operating system, and all that fancy stuff came after.
So? You can still create nice tools for it, even decades after a language has been created...
It's not. People arbitrarily complicate their personal workflows, especially having years of habit predating the existence of modern tools, but this is a voluntary decision to make local development complicated and you can choose not to work that way.
As an example, any time Homebrew ships a new update I run “pipx reinstall-all” and, for local projects, I let pipenv / poetry rebuild the virtualenv the next time I use it. This is automated and takes far less of my time than, say, managing dependency updates.
I think part of the challenge (from my perspective, as someone who only rarely needs to mess with python) is not that the workflow you describe is particularly complicated in isolation, it's the shear number of different tools that ostensibly have the same goal: to manage python virtual environments and packages. Because there are so many different tools, there is a near-limitless supply of different articles and SO posts in which people describe the "best" way to accomplish the most basic setup and maintenance of a virtual environment. It's quite difficult for new and/or occasional users to choose a good way to get started. Even if there is an "official" way documented on python.org for each Python version, when you run into a problem with it, inevitably you're stuck filtering through lots of advice to "just use <third-party tool xyz> instead".
Ah, my mistake! I had thought it was a huge problem and a constant pain in the ass, but it turns out that actually everyone is just stupid. Whew, what a relief.
>As an example, any time Homebrew ships a new update I run “pipx reinstall-all” and, for local projects, I let pipenv / poetry rebuild the virtualenv the next time I use it. This is automated and takes far less of my time than, say, managing dependency updates.
So, giving up (and re-installing everything from scratch or rebuilding virtualenvs, using several third party tools (pipx on top of pip, pipenv and/or poetry in lieu/in place of virtualenv) is your argument to how "it's not complicated" and people merely "arbitrarily complicate their personal workflows"?
The key part which you missed is that everything is automated, always installed in an isolated virtual environment, the package are automatically tracked and updated, and the tools which make this happen are installed from Homebrew and update at the same time.
As to why there are two tools, it's because they have separate goals: pipx provides a robust way to manage global installs of Python utilities in per-tool virtualenvs while tools like poetry or pipenv are aimed at projects you're developing and handle things like automating the process of installing updates & updating the corresponding metadata files so you can ship clean updates. All of them use virtualenvs and pip underneath — it's just a modern workflow.
The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber. Learning to use a tool every decade or so is a LOT less time consuming than doing this by hand. The average poster in this thread has already spent more time on this topic than my projects have needed for the mechanics of installing Python packages in any given year.
>The key part which you missed is that everything is automated, always installed in an isolated virtual environment, the package are automatically tracked and updated, and the tools which make this happen are installed from Homebrew and update at the same time.
That we can automate our way out of a mess doesn't mean there is a mess.
Even more so if the automation involves extra inconvenience (like, from writing the automation to waiting to rebuild), which with a different design in the Python packaging wouldn't be needed.
>The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber.
I can find and install updates just fine in other package managers for other languages, without messing my setup.
I can also move my whole environment to any path I like (with just copying).
Your position seems to be that Python packaging is a mess for non-specific reasons but also that nobody is allowed to write tools to improve usability. Think about pip – that wasn't the first tool to do that job but it (gasp) automated several things which didn't have great solutions. Virtualenv has the same story. This will be no different.
> I can find and install updates just fine in other package managers for other languages, without messing my setup.
Just like Python, in other words? (remember, this thread is not about installing package updates)
> I can also move my whole environment to any path I like (with just copying).
The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime. A compiled language which produces a static binary won't need that, of course, but that doesn't avoid the need to periodically recompile to stay current with dependencies and language changes. In all of these cases, automation makes the process fast and easy.
>Your position seems to be that Python packaging is a mess for non-specific reasons
No, for very specific reasons. Historical cruft, too many competing tools (from pip to conda, and from venv to poetry, not to mention setuptools), conceptual garbage like wheels, pths, etc., non-movable virtual environments, and so on.
>The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime.
And yet node handles this much more elegantly (with C++/gyp modules). And minor releases should absolutely no need this.
Python was created in the 90s when most unix users were on huge minicomputer-like shared systems (mostly at universities). In those days you had entire IT staff dedicated to curating and maintaining all of the software on the system, making sure it ran well, keeping it updated, etc. Guido explicitly did not want to bog the language development down with time spent on building packaging, deployment, etc.--those were matters for the system admins. We've all kind of been dealing with the consequences of that decision for many years.
The general trend I've noticed over the years is that they've been reducing edge-cases at the cost of making normal behavior worse. The change noted in TFA is one of these: always upgrading a package's dependencies when upgrading a package greatly reduces the numbers of combinations of versions that users can possibly have and so probably reduces the number of bug reports that come down to local environment things, but it comes at the cost of effectively removing the ability to upgrade a specific tool without fucking with the rest of your system.
Similarly I'm sure that making `brew install X` implicitly perform `brew upgrade` simplifies things a lot for them, but it's been a major source of hassle for me and it's making me consider switching away from homebrew.
This was one of the pain points that made me move back to Linux on my devices, and I'm glad I did it. Homebrew is infuriating when you're used to real package managers.
When you switch from Linux to Mac/Windows, you think "wow, this OS was actually made for GUIs. Everything just works. I don't have to restart ALSA every 3 days."
When you switch from Mac/Windows to Linux, you think "wow, this OS was actually made for software development. Everything just works. I don't have to resolve a library version conflict every 3 days."
I have never needed to restart ALSA, but I agree that it can be a bit rougher around the edges. Bluetooth is flakey everywhere, but I do seem to encounter more "edge cases" in Linux than my Mac.
If you can live with the rough edges, I find the general experience with applications or my printer suddenly being incompatible under MacOS for "reasons" is much less of an issue under Linux.
Agree that Mac OS backwards compatibility is atrocious!
I think another big challenge for Linux is battery life and sleep/wake on laptops. Last time I considered replacing Mac OS with Ubuntu on my laptop, I got scared away by reports that battery life would decrease by half.
My Lenovo laptop gets better battery life under Linux than it did under Windows, and I've generally found the battery life estimations (for my use cases anyway) to be complete bullshit even under MacOS.
I'm excited to be able to upgrade to the M1 series Mac's as it sounds like I might actually get battery life, though I have the 2019 16" Pro and can't justify the upgrade just yet, even if that thing overheats with just a Slack video.
Ran into this myself. Starting using `pyenv` instead. Never looked back. Allows you to maintain multiple versions of python outside brew's control and the system's.
The asdf Python plugin uses python-build, the same back-end that pyenv uses. So the tools are very similar. I prefer asdf to the collection of programs you mentioned because with asdf you get a unified CLI syntax for adding/updating plugins and installing/removing versions. I like being able to use the same toolchain and UI to manage different versions of Python, Node.js, Ruby, etc. Check out the list of asdf plugins [1] — it's quite impressive.
Not trying to spam, but have you looked at `anyenv`? It installs and updates all of these installers (and more), and keeps plugins up to date. I've been very happy with it.
I believe I have noticed this as well with Ruby as well. All Gems work great, then, after a brew operation, I need to reinstall/rebuild against an updated Ruby. It's certainly frustrating and can bite you, especially since it is unexpected.
I was worried that even miniconda seemed kinda bloaty, but it was the only way I could consistently use numpy on apple silicon. Really good option for local dev environments
If you haven't yet, try out mamba. It's more or less a drop-in conda replacement but written in C++ using a super fast SAT solver. The speed difference is incredible--I rarely even have time to finish reading the output before it's done installing a conda environment.
Use Homebrew to install pyenv and pyenv-virtualenv. Then use those to install pythons and collections of packages. You'll have stable development environments.
What? I did "pyenv install 3.9.1" on M1 and it worked absolutely fine, it built a arm64 build and everything. Maybe it's because I had Xcode installed?
Many of the problems people appear to be having with pyenv on M1 are related to it pulling include paths for homebrew that include homebrew libffi, but not passing the library path. For cases like cffi this results in header/library mismatch and you get runtime dynamic linker errors.
Other replies here appear to assume basic compilation issues, but the issues I've seen on various GitHub projects are mostly related to these more insidious C toolchain challenges. They're not M1-specific, but they are manifesting on M1 machines.
`brew install openssl` or `brew upgrade openssl` will still break pyenv virtualenvs, though they can then be fixed with `pyenv rehash`. Maybe this makes sense? I didn't sort through why it was happening.
I know this is frustrating, and I expect it affects more than just Python developers, but I think the author sums it up pretty well here:
> Homebrew’s Python is not for you. It exists to serve Homebrew, or more accurately, Homebrew’s other formulae. The primary purpose of Homebrew’s Python formula is to enable other Python-dependent Homebrew packages to work.
If you want to develop Python (likely Ruby/ PHP/ whatever), you are best keeping your dev environment separate from the interpreters installed for the purpose of running apps managed by Home-brew (or apt/ rpm/ etc).
Speaking as a Homebrew maintainer (and separately as someone who does both Ruby and Python development locally): this is the correct answer.
Homebrew's guarantees are no different from those of apt or any other system package manager; the only difference is that we don't have a stable distribution version to lock our Python or Ruby versions against. Individual developers will always be best served to isolate their development environments from the overarching runtime environment that's been distributed to them.
Would it make sense to enforce this by making the Homebrew python keg-only? Would other Homebrew packages that need Python still work? This way I can keep my own separate installation of Python that does not interfere with Homebrew.
> Would it make sense to enforce this by making the Homebrew python keg-only?
Maybe. I wasn't privy to any discussions around making Python keg-only, but my experience with other formulae that are keg-only is that users will force-link keg-only packages into `/usr/local` anyways. What's more, there are probably enough people out there who do just want a barebones Python 3 installation and don't really care if it gets upgraded underneath them, and we'd like to provide that by default.
The Python ecosystem already contains mature tooling on both Linux and macOS for managing Python installations that are isolated from Homebrew and other system-level package managers, like pyenv. In general, I'd recommend using pyenv in parallel with Homebrew; that will get you the best of both worlds.
Article author here. I think my biggest complaint is that there’s no disclaimer on the box [1]. Homebrew folks know about these problems, but they don’t warn users that their virtual environments will be obliterated. If Homebrew explained that these problems can happen, and that developers might want to use a different source for installing Python interpreters, my ire would be lessened considerably.
The decision to remove build options was informed (in part) by user experience: for every one user who was using them correctly to configure a package slightly differently for their uses, there were fifteen users following outdated SO answers and reporting their own breakages to us. Like every other package manager that distributes binary builds, we made a decision to improve the 99% use case.
Hi there! I'm one of the Homebrew maintainers, albeit not a very active one for the past year or so.
I agree that we could definitely improve our documentation on what people can and should expect from a Homebrewed version of Python; I'll look into opening a PR that improves these docs tonight.
That would be really helpful. In another thread you mentioned that "individual developers will always be best served to isolate their development environments from the overarching runtime environment that's been distributed to them". I agree, but it took me way longer to figure that out than I think was necessary, precisely because the "Homebrew and Python" docs don't mention that. I imagine a lot of other developers have been similarly affected, so many thanks in advance for looking into opening a pull request to enhance the documentation.
I gave up and just started using the OSX system copy (along w/ Google Colab), the main reason being that nearly every time there was a system update something would break and I would have to pin/unpin python@3.x
Is this somehow a bad thing? Your comment certainly seems to imply that it is.
At the end of the day, they're people volunteering their time for little to no benefit. Of course they should put themselves above their users; their users aren't even paying them for said time.
It certainly is a bad thing, it has led to them making a lot of user hostile changes that have made Homebrew a much worse project than it used to be. This article describes just one example of the kind of things that are considered ok by the Homebrew maintainers.
Once again, they're volunteers doing unpaid work. They're not beholden to anyone's interests except their own.
If you don't like their decisions, you're free to criticize them, but to imply that (rightly) putting themselves above their users is a bad thing is an entirely different beast when they don't owe anything to their users.
At the end of the day, if you don't like their decisions, you're free to work towards becoming a maintainer and make your voice heard, or fork it and do whatever you want and think is right. But nobody owes you or any other user anything unless you explicitly are paying them for the time and effort.
To be absolutely clear: what Homebrew is doing here is consistent with what every other system package manager does. If you've ever done `do-release-upgrade` to bump your Ubuntu LTS, you'll have run into the exact same problem.
The only difference is that Homebrew doesn't have a stable system distribution to pin to (we're not Apple, and we have no say over macOS's development), so we make upgrade decisions for packages like Python based on their stability with our packaged ecosystem. It's a poor practice to use your system Python distribution for any sort of virtualized development environment, and Homebrew behaves identically to every other system package manager in that regard.
Please keep in mind that we're a community of volunteers answering to millions of users. Your idea of "user hostile" is our idea of "simple for the sake of preserving our sanity."
No other package manager I've used automatically performs destructive actions on a package installation without asking me, homebrew does. But aside from that, homebrew used to be much nicer, it had options on formulae, it worked with the system ruby instead of installing its own periodically, it allowed patches that improved system integration even if they made the resulting binary less command line focused (for example emacs has macOS specific patches locked behind building the gui app).
And yeah, it's the homebrew maintainer's project, but when you're in a position to help people or make their life worse, it's immoral to do the latter.
MacPorts allows you to install multiple Python versions at the same time and I personally have several venvs with Python 2.7, 3.6, 3.7, 3.8, 3.9... all running without a glitch.
Homebrew and its excellent community was once indispensible on the Mac, but various pain points including this flakiness made me switch to (mini)conda for nearly everything, despite initial apprehension. There are some wrinkles and the resolver is slow, but in my hands conda is just so much more reliable than Homebrew ever was.
MacPorts is installing Python just fine here, fwiw. There's a package for each Python 3.x version so you can have whatever random versions you need installed. They can be used in virtualenvs easily and they don't break when you have to install an update for a 3.x.y release.
Not only that, MacPorts also kept all versions installed even after upgrade unless it's explicitly uninstalled (port echo inactive / port uninstall inactive); suffix every binaries with versions by default (e.g. pip-3.8, ansible-3.8) and only linked them as unversioned when explicitly set (port select --set python3 python38) which should be more familiar to people coming from Linux/BSD world.
How well does macports and home brew play together on the same system these days?
I used to use ports back in the days and appreciated it coming from FreeBSD previously but at some point most things I wanted were simply only available in brew, at that time brew and ports didn’t play well together on the same system so sometime around SL/Lion I switched to brew completely.
Default prefix of Homebrew is now /opt/homebrew on macOS on ARM, which should not conflict with MacPorts anymore.
For macOS on Intel, MacPorts does not play well with Homebrew because it took /usr/local, so ports built from source might use Homebrew files and break.
I see so as an Intel user I'd better stay away from mac ports if I want to use software only published on brew. I probably won't buy a new computer in the coming 5 years, so arm isn't relevant for me at this moment. But good to know for the future.
I was using MacPorts when Homebrew came out. I was amazed how quickly Homebrew spread, given that it did everything worse.
I honestly put its popularity down to it being written in Ruby, which made it the default choice for the large population of Ruby programmers on the Mac.
MacPorts binary archives were added back in 2011 across all three architectures at the time (ppc, i386, x86_64). Some packages are still maintained for ppc even today.
As mentioned by the other poster they were both source distributions at first, followed by binaries later.
The pain is a bit lost to memory, but even as a long-time MacPorts user I understand why people wanted to switch. If everything has to be built from source you also have to build all your build-time dependencies from source, even weird ones for building docs or whatever. The behavior of MacPorts then (and still unchanged today) is to upgrade to the latest versions of all dependencies when installing or upgrading a requested package. So even if you had zlib sitting around if an update was pushed to zlib you're going to be building a new one. A lot of packages were forced to autoreconf because the Mac was still gaining popularity and a lot of source distributions didn't have ./configure scripts that supported this or that Mac-specific or MacPorts-specific functionality. So now you have to download and install whatever autotools you need to do autoreconf. And of course MacPorts ships a separate package for autoconf 1.3, 1.4, 1.5, and individual packages can depend on specific major versions of autoconf, so even if you had an up-to-date autoconf 1.4 you're still getting an upgrade to autoconf 1.3 if your package happened to ask for it.
It was ugly. Homebrew's initial insistence on requiring the re-use of system packages cut down on flexibility (you can't get a new zlib, you have to wait for Apple to ship one) but greased a lot of wheels people did care about. But as time has gone by both systems have improved on their faults and are in a better place than before. This particular bug sucks for the people affected but I think that the Homebrew people will eventually figure this one out, even if it means making their system more MacPorts-like. Like they have been for the past ten years. ;)
Some years ago I tried submitting some fixes for MacPorts. It usually took a long time before they were picked up.
When I did the same for some homebrew formula I got a response a few hours later.
For me homebrew has a much more responsive community and due to their use of Github also a lower bar for contributions. Both might add to homebrew popularity.
MacPorts uses GitHub too now. We maintain much more ports than Homebrew, and few of our committers have time to regularly go through the PR queue, so our community seems inactive. But actually, we just need more hands on PR management.
One important distinction is that most active ports in MacPorts have specific maintainers, and we usually put off merging PRs until the port maintainer approves. With more than 20,000 ports, it hard for the committers to test every change thoroughly, so we rely on our port maintainers to resolve issues and review complex PRs whenever possible.
In Homebrew, any maintainer could review and merge a PR instantly, combined with fewer ports to maintain, average time to merge would be much shorter.
MacPorts has definitely always been the more correct and better way to do the job Brew claims to do.
But "brew" & "homebrew" is just a more charming and hipster name for a newbie to write in his blog for other newbies to follow, repeat eleventy-seven-hundred times.
Long-term homebrew user here, I have no blog and find the homebrew naming annoying. But being developed in Ruby and on GitHub made it a lot more welcoming than MacPorts ever felt when I used it.
I also never understood the concerns about correctness. Very few people want homebrew to be an apt/rpm replacement for servers that will provide stability for decades. Most Macs are effectively single-user systems, and if something goes wrong then nuking the whole homebrew directory and starting over is not a big deal.
Nuking the whole directory and starting over is a big deal for many people, myself included. I don't have time to waste, I want my software to just work, and that is why I use MacPorts.
A postgres update seriously fucked up my work Ruby install as well. We were using an old version and homebrew updated openssl on us (from 1.0.1 to 1.1 I think?) with no way of going back. (The Ruby install I was running was linked to 1.0.1, so it was not happy with having its library deleted.) I spent a month trying to figure out how to get that shit working again.
I really like Homebrew but its dependency management is horrible. The moment I saw that dyld "image not found" error message I didn't even have to finish reading the article to know what happened.
Python's insistence that it be a system-level utility in one global location, with one global set of libraries, etc. is kind of why we're in this mess of many different ways to install and use python (and the myriad of workarounds like virtual envs, pyenv, pipx, etc.). The official CPython installer is fine for some simple scripting or learning the language, but as soon as you need to tightly control and define multiple different python installs, different versions of libraries, etc. to satisfy different constraints it becomes a huge headache.
I'd be perfectly happy with homebrew if it didn't auto-update all packages when installing something new. E.g. I do `brew install ffmpeg`, and as part of that it upgrades python from 3.8 to 3.9 (???), breaking imports in all my scripts. I know how to work around this, but still, surprising choice for the default behavior.
I totally get that package management is Hard. However, I feel like Homebrew should be able to track what's there because it's a dependency vs what's there because the user wants it to be and act accordingly.
Granted, I've only lived in these tools just long enough to install what I need, so I'm sure I'm naively overlooking a lot of nuance.
Ah interesting. It appears that the reason for this difference is that Python on Homebrew is delivered as a single package for development and execution. The true intention should have been to have had Python-Homebrew to create the environment for Homebrew and Python-Dev which you can use for your development.
Sort of analogous to the way the JVM is distributed usually as separate packages with the JRE and the JDK.
I don't write very much prod-grade Python, so I just recreate the venv with my requirements.txt instead.
I'm a little surprised more people aren't using `anyenv`. That lets you install & update pyenv, nodenv, rbenv, etc. (though some support for certain languages is spotty) and thus get everything built/installed the way you want it.
Also, when using pyenv, I almost always set `env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install -v ${pylatest}` (macOS) or `env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install -v ${pylatest}` (Linux). (I also set some CFLAGS, but these are personal preference.) I've had plenty of situations where not having the shared/framework build has screwed something up (and in fact that was my original reason for switching from Homebrew or package manager-installed Python years ago), and I've never had issues where the shared build screwed something up.
asdf[1] mentioned in the article is similar to anyenv but instead of being a wrapper around *env, it implements its own simple framework for creating a version manager (e.g. [2]). Usually asdf plugin uses the same underlying installer as its *env equivalent (e.g. asdf-python uses python-build, asdf-ruby uses ruby-build, asdf-erlang uses kerl).
I will almost resist the urge to truncate the title to "Python is Not for You" -- but I'd like to second the benefits of using a python version manager such as pyenv. Yes, all the versions waste an ungodly amount of space, but you probably can't/won't use Clojure instead I guess.
Article author here. I think folks like asdf and pyenv because of the ease with which new versions can be installed and removed. For example, I can run asdf install python 3.9.1 and have a new version installed in minutes, without leaving the terminal console. I don't think that's possible via the official python.org installers.
My experience on MacOS was that it doesn't update and cant be uninstalled without using 3rd party scripts.
I could be mistaken, but I didn't find any documentation on how to do it and the fact that other people had projects to try and do it mean that I wasnt alone on finding this a bit of a mess.
As someone who doesn't often use Python what is the current "best" solution for installing and managing your environment?
Last time I used Python on macOS and Windows I manually installed the official version from python.org and used pip and virtualenv. It wasn't too bad but it seems things have changed in the last 6 years or so? Or is this still the most flexible way to go?
For totally greenfield python projects check out poetry, it is an opinionated python project workflow that's very close to the nodejs/npm ecosystem's package.json. It does a virtualenv behind the scenes, pins your python dependencies with a lock file, and even has niceties like dev scripts as first class citizens.
Otherwise if you're trying to use some python project with particular dependencies pyenv is a good way to get a specific python version. But if the project has a lot of native dependencies (think scientific computing with lots of fortran, C++, etc. code to compile) then you're almost certainly better off with conda as a higher-level entire binary system manager.
108 comments
[ 2.7 ms ] story [ 160 ms ] threadPeople are trying to fix it, and their are some great packages out there that make it easier. Pyenv, Poetry, this asdf tool the fine article mentions, all seem like good solutions.
So? You can still create nice tools for it, even decades after a language has been created...
As an example, any time Homebrew ships a new update I run “pipx reinstall-all” and, for local projects, I let pipenv / poetry rebuild the virtualenv the next time I use it. This is automated and takes far less of my time than, say, managing dependency updates.
So, giving up (and re-installing everything from scratch or rebuilding virtualenvs, using several third party tools (pipx on top of pip, pipenv and/or poetry in lieu/in place of virtualenv) is your argument to how "it's not complicated" and people merely "arbitrarily complicate their personal workflows"?
Oh, the irony!
As to why there are two tools, it's because they have separate goals: pipx provides a robust way to manage global installs of Python utilities in per-tool virtualenvs while tools like poetry or pipenv are aimed at projects you're developing and handle things like automating the process of installing updates & updating the corresponding metadata files so you can ship clean updates. All of them use virtualenvs and pip underneath — it's just a modern workflow.
The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber. Learning to use a tool every decade or so is a LOT less time consuming than doing this by hand. The average poster in this thread has already spent more time on this topic than my projects have needed for the mechanics of installing Python packages in any given year.
That we can automate our way out of a mess doesn't mean there is a mess.
Even more so if the automation involves extra inconvenience (like, from writing the automation to waiting to rebuild), which with a different design in the Python packaging wouldn't be needed.
>The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber.
I can find and install updates just fine in other package managers for other languages, without messing my setup.
I can also move my whole environment to any path I like (with just copying).
> I can find and install updates just fine in other package managers for other languages, without messing my setup.
Just like Python, in other words? (remember, this thread is not about installing package updates)
> I can also move my whole environment to any path I like (with just copying).
The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime. A compiled language which produces a static binary won't need that, of course, but that doesn't avoid the need to periodically recompile to stay current with dependencies and language changes. In all of these cases, automation makes the process fast and easy.
No, for very specific reasons. Historical cruft, too many competing tools (from pip to conda, and from venv to poetry, not to mention setuptools), conceptual garbage like wheels, pths, etc., non-movable virtual environments, and so on.
>The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime.
And yet node handles this much more elegantly (with C++/gyp modules). And minor releases should absolutely no need this.
Similarly I'm sure that making `brew install X` implicitly perform `brew upgrade` simplifies things a lot for them, but it's been a major source of hassle for me and it's making me consider switching away from homebrew.
I can't tell if that would influence the behavior you're experiencing, but it sounds like it makes the brew command more deterministic
When you switch from Mac/Windows to Linux, you think "wow, this OS was actually made for software development. Everything just works. I don't have to resolve a library version conflict every 3 days."
If you can live with the rough edges, I find the general experience with applications or my printer suddenly being incompatible under MacOS for "reasons" is much less of an issue under Linux.
I think another big challenge for Linux is battery life and sleep/wake on laptops. Last time I considered replacing Mac OS with Ubuntu on my laptop, I got scared away by reports that battery life would decrease by half.
I'm excited to be able to upgrade to the M1 series Mac's as it sounds like I might actually get battery life, though I have the 2019 16" Pro and can't justify the upgrade just yet, even if that thing overheats with just a Slack video.
https://github.com/pyenv/pyenv
What's nice is you can find analog programs for a lot of different languages and platforms including
[1] https://asdf-vm.com/#/plugins-all?id=plugin-list
If you're having trouble installing something with brew on M1, run arch x86_64 brew instead of brew to do a Rosetta installation.
https://github.com/pyenv/pyenv/issues/1643#issuecomment-7293...
For installing Python on the M1 Apple Silicon. This will install x86_64 versions, but those will work just fine.
Other replies here appear to assume basic compilation issues, but the issues I've seen on various GitHub projects are mostly related to these more insidious C toolchain challenges. They're not M1-specific, but they are manifesting on M1 machines.
My current Mac is going to be my last. Linux on Thinkpad is next
> Homebrew’s Python is not for you. It exists to serve Homebrew, or more accurately, Homebrew’s other formulae. The primary purpose of Homebrew’s Python formula is to enable other Python-dependent Homebrew packages to work.
If you want to develop Python (likely Ruby/ PHP/ whatever), you are best keeping your dev environment separate from the interpreters installed for the purpose of running apps managed by Home-brew (or apt/ rpm/ etc).
Homebrew's guarantees are no different from those of apt or any other system package manager; the only difference is that we don't have a stable distribution version to lock our Python or Ruby versions against. Individual developers will always be best served to isolate their development environments from the overarching runtime environment that's been distributed to them.
[1] https://docs.brew.sh/Homebrew-and-Python.html
Thanks again for bringing it to our attention!
Maybe. I wasn't privy to any discussions around making Python keg-only, but my experience with other formulae that are keg-only is that users will force-link keg-only packages into `/usr/local` anyways. What's more, there are probably enough people out there who do just want a barebones Python 3 installation and don't really care if it gets upgraded underneath them, and we'd like to provide that by default.
The Python ecosystem already contains mature tooling on both Linux and macOS for managing Python installations that are isolated from Homebrew and other system-level package managers, like pyenv. In general, I'd recommend using pyenv in parallel with Homebrew; that will get you the best of both worlds.
[1] https://docs.brew.sh/Homebrew-and-Python.html
The decision to remove build options was informed (in part) by user experience: for every one user who was using them correctly to configure a package slightly differently for their uses, there were fifteen users following outdated SO answers and reporting their own breakages to us. Like every other package manager that distributes binary builds, we made a decision to improve the 99% use case.
I agree that we could definitely improve our documentation on what people can and should expect from a Homebrewed version of Python; I'll look into opening a PR that improves these docs tonight.
At the end of the day, they're people volunteering their time for little to no benefit. Of course they should put themselves above their users; their users aren't even paying them for said time.
If you don't like their decisions, you're free to criticize them, but to imply that (rightly) putting themselves above their users is a bad thing is an entirely different beast when they don't owe anything to their users.
At the end of the day, if you don't like their decisions, you're free to work towards becoming a maintainer and make your voice heard, or fork it and do whatever you want and think is right. But nobody owes you or any other user anything unless you explicitly are paying them for the time and effort.
The only drawback is that the Homebrew organization won’t support, update, or build bottles for, custom taps.
The only difference is that Homebrew doesn't have a stable system distribution to pin to (we're not Apple, and we have no say over macOS's development), so we make upgrade decisions for packages like Python based on their stability with our packaged ecosystem. It's a poor practice to use your system Python distribution for any sort of virtualized development environment, and Homebrew behaves identically to every other system package manager in that regard.
Please keep in mind that we're a community of volunteers answering to millions of users. Your idea of "user hostile" is our idea of "simple for the sake of preserving our sanity."
And yeah, it's the homebrew maintainer's project, but when you're in a position to help people or make their life worse, it's immoral to do the latter.
MacPorts allows you to install multiple Python versions at the same time and I personally have several venvs with Python 2.7, 3.6, 3.7, 3.8, 3.9... all running without a glitch.
I wish more people try out MacPorts.
I used to use ports back in the days and appreciated it coming from FreeBSD previously but at some point most things I wanted were simply only available in brew, at that time brew and ports didn’t play well together on the same system so sometime around SL/Lion I switched to brew completely.
For macOS on Intel, MacPorts does not play well with Homebrew because it took /usr/local, so ports built from source might use Homebrew files and break.
I honestly put its popularity down to it being written in Ruby, which made it the default choice for the large population of Ruby programmers on the Mac.
The pain is a bit lost to memory, but even as a long-time MacPorts user I understand why people wanted to switch. If everything has to be built from source you also have to build all your build-time dependencies from source, even weird ones for building docs or whatever. The behavior of MacPorts then (and still unchanged today) is to upgrade to the latest versions of all dependencies when installing or upgrading a requested package. So even if you had zlib sitting around if an update was pushed to zlib you're going to be building a new one. A lot of packages were forced to autoreconf because the Mac was still gaining popularity and a lot of source distributions didn't have ./configure scripts that supported this or that Mac-specific or MacPorts-specific functionality. So now you have to download and install whatever autotools you need to do autoreconf. And of course MacPorts ships a separate package for autoconf 1.3, 1.4, 1.5, and individual packages can depend on specific major versions of autoconf, so even if you had an up-to-date autoconf 1.4 you're still getting an upgrade to autoconf 1.3 if your package happened to ask for it.
It was ugly. Homebrew's initial insistence on requiring the re-use of system packages cut down on flexibility (you can't get a new zlib, you have to wait for Apple to ship one) but greased a lot of wheels people did care about. But as time has gone by both systems have improved on their faults and are in a better place than before. This particular bug sucks for the people affected but I think that the Homebrew people will eventually figure this one out, even if it means making their system more MacPorts-like. Like they have been for the past ten years. ;)
When I did the same for some homebrew formula I got a response a few hours later.
For me homebrew has a much more responsive community and due to their use of Github also a lower bar for contributions. Both might add to homebrew popularity.
MacPorts uses GitHub too now. We maintain much more ports than Homebrew, and few of our committers have time to regularly go through the PR queue, so our community seems inactive. But actually, we just need more hands on PR management.
One important distinction is that most active ports in MacPorts have specific maintainers, and we usually put off merging PRs until the port maintainer approves. With more than 20,000 ports, it hard for the committers to test every change thoroughly, so we rely on our port maintainers to resolve issues and review complex PRs whenever possible.
In Homebrew, any maintainer could review and merge a PR instantly, combined with fewer ports to maintain, average time to merge would be much shorter.
But "brew" & "homebrew" is just a more charming and hipster name for a newbie to write in his blog for other newbies to follow, repeat eleventy-seven-hundred times.
I also never understood the concerns about correctness. Very few people want homebrew to be an apt/rpm replacement for servers that will provide stability for decades. Most Macs are effectively single-user systems, and if something goes wrong then nuking the whole homebrew directory and starting over is not a big deal.
I really like Homebrew but its dependency management is horrible. The moment I saw that dyld "image not found" error message I didn't even have to finish reading the article to know what happened.
[1] https://docs.conda.io/en/latest/
For others similarly curious, it appears setting [`$HOMEBREW_NO_AUTO_UPDATE=1`](https://github.com/Homebrew/brew/blob/2.7.7/docs/Manpage.md#...) is the magic sauce (or the perhaps more accurate link: https://github.com/Homebrew/brew/blob/2.7.7/Library/Homebrew... )
Granted, I've only lived in these tools just long enough to install what I need, so I'm sure I'm naively overlooking a lot of nuance.
https://xkcd.com/1987/
Sort of analogous to the way the JVM is distributed usually as separate packages with the JRE and the JDK.
I don't write very much prod-grade Python, so I just recreate the venv with my requirements.txt instead.
Also, when using pyenv, I almost always set `env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install -v ${pylatest}` (macOS) or `env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install -v ${pylatest}` (Linux). (I also set some CFLAGS, but these are personal preference.) I've had plenty of situations where not having the shared/framework build has screwed something up (and in fact that was my original reason for switching from Homebrew or package manager-installed Python years ago), and I've never had issues where the shared build screwed something up.
[1]: https://asdf-vm.com/
[1]: https://asdf-vm.com/#/plugins-all
In my experience they work great, you can install multiple versions, and even pip3 seems to just work with them.
I'm not a heavy Python user, but ever since I started using the official installers it just works.
I could be mistaken, but I didn't find any documentation on how to do it and the fact that other people had projects to try and do it mean that I wasnt alone on finding this a bit of a mess.
Last time I used Python on macOS and Windows I manually installed the official version from python.org and used pip and virtualenv. It wasn't too bad but it seems things have changed in the last 6 years or so? Or is this still the most flexible way to go?
I would be grateful for any guidance.
Otherwise if you're trying to use some python project with particular dependencies pyenv is a good way to get a specific python version. But if the project has a lot of native dependencies (think scientific computing with lots of fortran, C++, etc. code to compile) then you're almost certainly better off with conda as a higher-level entire binary system manager.
For managing virtual environments on my preferred shell (Fish), I use (and maintain) VirtualFish [2].
For managing project dependencies, I activate environments via VirtualFish and then use Poetry [3] to update the dependencies within the environments.
[1]: https://github.com/danhper/asdf-python
[2]: http://github.com/justinmayer/virtualfish
[3]: http://python-poetry.org