>Avoid using Poetry for new projects. Poetry predates many standards for Python tooling. This means that it uses non-standard implementations of key features, such as the dependency resolver and configuration formats in pyproject.toml files.
Was going to comment the same thing. Would love to hear the author expand further on why not use Poetry. I've found it to be pretty solid and continue to use Poetry + Pyenv for all my projects, but open to hearing the case for PDM or Hatch.
One thing I saw which was weird as hell was poetry has serious issues with changing windows requirements (specifically removing them) and has a tendency to throw all sorts of fun errors (its method of removing files is not file system friendly.)
As far as I can tell the issue is still open and has been for awhile.
I've never worked on a team that uses Poetry, but in my current company another team uses it, and I haven't found it really as slick as I would have imagined, primarily because you need to create a venv and install poetry into that before you even get started, which by that point why not just pip install the rest anyway? For standalone applications it just seems like an unnecessary extra step. It doesn't even mandate a build lifecycle like Maven so what are you getting?
But that's not what soured me on Poetry. What soured me was recently I needed to create a release of one of their libraries with a Git commit in the local version identifier and... Poetry doesn't do that. There's an issue that was open on GitHub for years before they finally agreed to implement it, and since February the change is now merged to master, but despite several point releases since then, that change has not landed in any of them. When will we be able to get a local version part? Who knows!
This experience has really made me skeptical of Poetry being the One True Packaging Tool that fixes everything. As usual, it just fixes the things the devs want fixed and everything else is still janky or half implemented. From my perspective, if you're gonna deal with jank anyway, might as well just deal with the standard jank that comes as part of Python itself.
Actually poetry comes with an optional self-installer, though I prefer to manage it with pipx. And it's recommended not to install it into your project env, as there's the potential for conflicting dependency versions.
After struggling with the complexity of pyenv and slowness of poetry I'm really happy with rye.
It manages both python versions (which it downloads instead of compiling) and package versions. It's written in rust so it's faster and can replace pipx as well for installing global tools. (Some people will recommend uv which rye is slowly merging with buy uv is still missing some rye features, probably some time in the future you might want to switch).
Yeah to me Poetry was a game changer in dependency tooling. Not that better tooling can come about, but it’s recently worked great on multiple production grade projects.
No complaints, and has shoot-in-the-foot safety features
Aside from all the obvious issues of having no distinction between transitive and direct dependencies, it completely breaks cross-platform support if any of your dependencies have platform-specific sub-dependencies (which is not uncommon in python).
pip freeze was an overly simplified hyperbolic jest...
fair point, so when this packaging mess happens, can one not strike a balance, and define the dependencies that you need, then, and let the resolver handle the rest?
Agree, I'm a poetry person too. I feel like TFA might be getting ahead of itself on this point. Just because something was ratified as a "standard" doesn't immediately make everything prior irrelevant.
Agreed. Most of these are fairly uncontroversial. A few will probably be new to some, particularly because users of Python are heterogenous and people find themselves with years of experience on projects that never upgraded their language version, e.g. from 3.6, and don't know that they're missing.
A final few, even with the explanation given, feels like preference, such as the Poetry opinion. I suppose if I wrote a similar article, it would end up with a similar mix of obvious defaults and a few of my preferences.
It could also be because of different use cases? For example, writing a library for packaging, vs a deployed application, vs a personal projects' workbench. Or perhaps if you are collaborating with people who use Windows without WSL. I have heard tell that Poetry can sometimes trip over its own shoelaces on Windows. I have never experienced it myself, and don't personally care to support any projects involving Windows, but if I did I might have different preferences.
I've not had any issues using mamba environments and vanilla pip. Not sure why I would need to change that? I have packages that thousands of people use, and others that help manage billions of dollars. I've managed to keep things simple without upgrading tooling. This article feels like it's suggesting flavor of the month technologies without explaining why the status quo is bad. Also, you can't really use virtualenvs easily when you have packages that depend on C libraries.
Overall, generally reasonable. Nothing is a hard and fast rule, if you know what you're doing and have a good reason for going down an alternative path, that's great. But if on a new project you generally follow these you'll be setting yourself up for success.
It's interesting to me that dataclasses seemed to be a slimmed down attrs but in practice I find it replaces namedtuple not attrs
For my use the key is how easily you can add simple conversion of values to attrs. IIRC this was intentionally omitted from dataclasses. For a 1-off using a factory with a dataclass is easy but repeated uses send me back to attrs
Yeah, I think namedtuple's popularity (recommended in this link) was a mistake. It's sole modern use case should be to turn tuples used as an unnamed struct into a named struct while preserving tuple indexing for backward compatibility.
Most people don't need, say, a[-3] as an alias for a.field_name.
Otherwise, for those who want the standard library, use a dataclass with frozen=True for immutability.
attrs looks interesting, can it be used in workflows for processing GBs of data? I don't see any Cython compatibility or other Python to compiled code options. I typically just get by with numpy recarrays, but I'm old fashioned and looking for something more modern.
Attrs would not replace your record array. IIRC you can supply slots so it might be useful for converting raw data into your numpy types, but I suspect you'd find the overhead in function calls problematic
If you want easier than recarray and almost as fast I think you're in pandas-ville
I'm surprised it doesn't mention rye. I now that technically the projects are combining and uv is eventually meant to eat rye but for now I think it offers higher level functionality that I'm unsure if uv provides. Like pinning /downloading different Python interpreters. Rye and uv both allow installing tools in custom venvs removing the need for pipx.
For many sections I'd ask myself "Why?" Sure, I know you can click on the links and do deeper research, but I'd love a quick "why" blurb for each section. Just a short "Why this is nice? Well, because..."
As far as I know you're right, also using pyproject.toml generates platform independent dependency trees (I don't have experience with the tools he mentions but that's what poetry does), requirements.txt does not and must be generated on a server or carefully picking the crossplatform version of the packages.
Also kinda weird he says to not use pip, but requirements.txt are pip commands... And should be running with pip... Hmm I don't know about this article anymore.
uv and rye (which depends on uv) use pyproject.toml and spit out requirements.txt like files as lock files. They aren't platform independent but are easier to install in a docket container (just use pip inside the container). Not sure about hatch/pdm which they recommended. Some people use pip-compile which also converts toml files in requirements.txt files
Has anyone tried orchestrating Python code using Elixir/Erlang? I feel like the true concurrency can only be achieved in the actor model of BEAM, and async still just does concurrent code, not parallel.
Is there something wrong with pipenv[0] that I'm not aware of? It really feels like a full solution, I've been using it for years, and I can't understand why it isn't more widely used.
what are the pros and cons with pydantic models vs data classes? i like the runtime validation in pydantic models.
the pattern i landed on is that 1) only use primitive data types such as str, int, pydantic models and pandas dataframes to present data. 2) no classes with methods, no polymorphism, only use module functions.
Pydantic models are significantly more heavyweight than dataclasses -- They serve as validators as opposed to dataclasses, which are primarily just tools for managing structured data. Both are important problems, but they are different.
The power of Pydantic models comes with benefits in terms of what they're able to do out of the box. But this comes at a significant cost of speed, which does matter for some applications. There are plenty of applications where it simply doesn't make sense to validate types every time you stand up a class.
Sure you can turn some of that off with Pydantic, etc, etc, but the fact that dataclasses don't require a third party install and are efficient and easy out of the box makes them useful and non-duplicative.
Curious why there is no mention of conda and its fast alternative mamba (micromamba). In my experience, conda is much more robust in managing. dependencies conflict (thanks to AWS and especially boto3) compared to pip.
I've been using conda+poetry as my goto combo for years now and it's served me very well. Sure it's said poetry doesn't follow certain standards, but it just abstracts away so much that I don't see myself needing anything else really. If I want to share a non-poetry artifact, it's a simple `poetry build` to create a wheel.
I did come across a weird dependency confusion issue recently when I tried using it on a server setup with piku though, but haven't gotten around to find what exactly is happening.
Also don't think I'll ever give up on YAML for it's easy read+write, and I love the simplicity of fire for CLI things. And pathlib.Path().glob() for directory listing.
55 comments
[ 3.6 ms ] story [ 122 ms ] threadWhat? This is the first I've heard of this.
As far as I can tell the issue is still open and has been for awhile.
But that's not what soured me on Poetry. What soured me was recently I needed to create a release of one of their libraries with a Git commit in the local version identifier and... Poetry doesn't do that. There's an issue that was open on GitHub for years before they finally agreed to implement it, and since February the change is now merged to master, but despite several point releases since then, that change has not landed in any of them. When will we be able to get a local version part? Who knows!
This experience has really made me skeptical of Poetry being the One True Packaging Tool that fixes everything. As usual, it just fixes the things the devs want fixed and everything else is still janky or half implemented. From my perspective, if you're gonna deal with jank anyway, might as well just deal with the standard jank that comes as part of Python itself.
It manages both python versions (which it downloads instead of compiling) and package versions. It's written in rust so it's faster and can replace pipx as well for installing global tools. (Some people will recommend uv which rye is slowly merging with buy uv is still missing some rye features, probably some time in the future you might want to switch).
No complaints, and has shoot-in-the-foot safety features
what's wrong with pip freeze? why are there so many competing tools?
it's very anti-python IMO.
I prefer my requirements.txt to include only the packages I install with pip myself (and not their dependencies).
Aside from all the obvious issues of having no distinction between transitive and direct dependencies, it completely breaks cross-platform support if any of your dependencies have platform-specific sub-dependencies (which is not uncommon in python).
fair point, so when this packaging mess happens, can one not strike a balance, and define the dependencies that you need, then, and let the resolver handle the rest?
A final few, even with the explanation given, feels like preference, such as the Poetry opinion. I suppose if I wrote a similar article, it would end up with a similar mix of obvious defaults and a few of my preferences.
It could also be because of different use cases? For example, writing a library for packaging, vs a deployed application, vs a personal projects' workbench. Or perhaps if you are collaborating with people who use Windows without WSL. I have heard tell that Poetry can sometimes trip over its own shoelaces on Windows. I have never experienced it myself, and don't personally care to support any projects involving Windows, but if I did I might have different preferences.
For my use the key is how easily you can add simple conversion of values to attrs. IIRC this was intentionally omitted from dataclasses. For a 1-off using a factory with a dataclass is easy but repeated uses send me back to attrs
Most people don't need, say, a[-3] as an alias for a.field_name.
Otherwise, for those who want the standard library, use a dataclass with frozen=True for immutability.
If you want easier than recarray and almost as fast I think you're in pandas-ville
https://github.com/astral-sh/uv
https://news.ycombinator.com/item?id=39387641
I have seen occasional hiccups, so we don't use it in prod just yet. But I use it in local all the time.
They've been responsive to issues, so give it a shot.
Isn't this obsoleted by the previous section on using pyproject.toml?
Also kinda weird he says to not use pip, but requirements.txt are pip commands... And should be running with pip... Hmm I don't know about this article anymore.
[0] pipenv.pypa.io
The power of Pydantic models comes with benefits in terms of what they're able to do out of the box. But this comes at a significant cost of speed, which does matter for some applications. There are plenty of applications where it simply doesn't make sense to validate types every time you stand up a class.
Sure you can turn some of that off with Pydantic, etc, etc, but the fact that dataclasses don't require a third party install and are efficient and easy out of the box makes them useful and non-duplicative.
I did come across a weird dependency confusion issue recently when I tried using it on a server setup with piku though, but haven't gotten around to find what exactly is happening.
Also don't think I'll ever give up on YAML for it's easy read+write, and I love the simplicity of fire for CLI things. And pathlib.Path().glob() for directory listing.