Caution: if you rely on pybind11 or a project using pybind11 (many projects do, like NumPy/SciPy/Tensorflow/PyTorch..), hold off on upgrading to Python 3.9.0 for now.
A change in Python 3.9.0 introduces undefined behavior in combination with pybind11 (rarely occurring crashes, but could be arbitrarily bad). We will work around it in an upcoming version of pybind11, and Python will separately also fix this problem in 3.9.1 slated for release in December.
I was about to ask why would they do a release if they know it's broken and blowing up their flagship libraries? It's worrying when the interpreter starts to favor shipping quickly over being functional.
But nevermind, the 3.9 release was 15 days ago and the bug was identified 8 days ago. I guess there has to be a release before people will start using it and find bugs/regressions.
There's a saying to always wait for the .1 release, this is a good illustration of why.
What makes pybind a "flagship library"? If it was very widely used this bug probably would've been caught by people testing the alpha/beta/rc versions of 3.9.
People using numpy are not in the fringe. If this wasn’t triaged during prereleases then it’s either that it doesn’t always cause problems, or there is an issue in the diversity of testers.
I personally don't like Python (I prefer Julia/Ruby/Rust), but all the new advances in machine learning are implemented in Tensorflow and PyTorch, so I came back to Python just because of these libraries and infrastructure on top of them.
I was just using the expit function from scipy yesterday by the way, because that's where it was implemented.
I don't know if I would call the SciPy ecosystem the flagship libraries of python. I know it is very popular,and probably in the top 5 use cases, and I use some of them everyday. but in my experience they've always been off in their own world in relation to the rest of the python community. Most of the people I've met that use this stuff are either scientists or finance people that are just using it as another tool along with matlab, R, and excel. Which is what they should be doing, but in their minds they're not writing python code they're using Pandas, Jupyter, and Matplotlib.
But yes, never ever jump on a new release of anything, at least in production. Always stand back and wait for someone more optimistic to find the bugs
> I don't know if I would call the SciPy ecosystem the flagship libraries of python.
I mean, sure NumPy is integral to python finance and scientific computing...but it's also right at the heart of the popular Python roguelike tutorial, too.
There's Python code that doesn't use NumPy, sure (I've got some in production right now), but I can't personally think of a Python library with nearly as deep dependencies across nearly as diverse a range of use cases.
“We did not seriously consider alternative ways to implement the new parser, but here's a brief discussion of LALR(1). [...] Other variants of LR were not considered, nor was LL (e.g. ANTLR).”
Sounds like the thought process was along the lines of:
“LL(1) makes it too hard to add complex new features to the language, we must change to something else; PEG is something else; therefore we must change to PEG”.
It’s interesting that the old parser had the same problem. Although it did the parsing using an LL(1) algorithm, it never actually verified that the grammar was LL(1) before doing so.
I'm a fellow fan of antlr but it's not really an option for the base interpreter. They're (rightfully) very stingy about deps. cpython supports a wide range of platforms and has its own very permissive license - it's a pretty self-contained codebase and they prefer to keep it that way.
The annual release cadence is a disaster for Python. I've already run into countless interpreter compatibility issues where new code has to run in a different environment. 3.6, 3.7, 3.8, and now 3.9 are all still officially supported, which just adds a huge mental burden when reading python code, because you have to remember when each feature was released.
How does it add burden when reading code? Writing code, sure, but only when you don't research the environment you're going to be deploying into. Unless you started working with Python like last year you probably remember when new useful features were released, and for stuff you haven't used before, the Python documentation always tells you when it was added.
Say your production environment runs on python 3.7. New developer on the team opens a pr and you see dataclasses, a class attribute type hint, maybe a walrus operator and some required keyword-only args. They say they've tested the code in their local virtual environment and it runs. Which of these features break the build and which are the new best practices?
Sure ci will catch most of these but maintaining 100% code coverage is a significant burden and very rare in practice.
If your new developer installs a higher version than you are running in production, the issue is with your on boarding process, not python release cadence.
The issue is that python now is effectively 4 languages at once, making all aspects of working with it more complex and prone to error, onboarding included
I love to use Python as counter example of C++'s complexity.
Have been using Python on and off since 1.6 and not only is the documentation also quite impressive, when one sums up language reference + library, just like ISO documents, its changes between versions make a Pub Quiz a very easy task.
The language is probably the new Basic, but to achieve 10 dan in Python there is plenty of stuff to chew on.
A good CI/CD, with some precommits and a containerised stack should avoid you all those troubles.
Also, nothing force you to update to the newest version, you can stay approximately 5y on the same release, but keeping the pace makes the upgrades less costly and less risky in the long run because you don't accumulate deprecated features, you're forced to cleanup the abandoned dependencies, ... It also makes the team happier and more attractive when recruiting.
Why do they keep breaking things? Don't they understand code doesn't get maintained at this rate in the real world? You write the code, and it should just keep working, like infrastructure. It's only when you have water gushing out of the ground that you worry about a water main... not because some plumbing supply company decided that certain types of plumbing suddenly don't work.
Things should last for lifetimes before they break, not 12 months.
Infrastructure doesn't "run for a lifetime" without the proper upkeep and regular maintenance... just like code. There is no set and forget in technology, just because you are doing the infrastructure work doesn't mean someone in the company isn't.
48 comments
[ 0.18 ms ] story [ 140 ms ] threadI think this is a rather innovative choice among the mainstream languages?
Mainly making it easier to do syntax extensions...
For Python devs there will be no real-world effects...
Yes, AFAICT the main reason for the parser change is to enable the pattern matching feature [0] that’s likely coming in 3.10.
[0] https://www.python.org/dev/peps/pep-0634/
A change in Python 3.9.0 introduces undefined behavior in combination with pybind11 (rarely occurring crashes, but could be arbitrarily bad). We will work around it in an upcoming version of pybind11, and Python will separately also fix this problem in 3.9.1 slated for release in December.
Details available here: https://pybind11.readthedocs.io/en/latest and https://github.com/python/cpython/pull/22670.
But nevermind, the 3.9 release was 15 days ago and the bug was identified 8 days ago. I guess there has to be a release before people will start using it and find bugs/regressions.
There's a saying to always wait for the .1 release, this is a good illustration of why.
As the top-level comment says, it rarely causes crashes.
NumPy, SciPy, and friends are, individually and collectively flagship libraries, and depend on pibind11.
> If it was very widely used this bug probably would've been caught by people testing the alpha/beta/rc versions of 3.9.
They are very widely used, but “probably” means “do enough releases and you will find exceptions”. Welcome to an exception.
I was just using the expit function from scipy yesterday by the way, because that's where it was implemented.
But yes, never ever jump on a new release of anything, at least in production. Always stand back and wait for someone more optimistic to find the bugs
I mean, sure NumPy is integral to python finance and scientific computing...but it's also right at the heart of the popular Python roguelike tutorial, too.
There's Python code that doesn't use NumPy, sure (I've got some in production right now), but I can't personally think of a Python library with nearly as deep dependencies across nearly as diverse a range of use cases.
Discussion here:
https://news.ycombinator.com/item?id=24691565
“We did not seriously consider alternative ways to implement the new parser, but here's a brief discussion of LALR(1). [...] Other variants of LR were not considered, nor was LL (e.g. ANTLR).”
[https://www.python.org/dev/peps/pep-0617/]
“LL(1) makes it too hard to add complex new features to the language, we must change to something else; PEG is something else; therefore we must change to PEG”.
[https://discuss.python.org/t/should-there-be-a-check-whether...]
Sure ci will catch most of these but maintaining 100% code coverage is a significant burden and very rare in practice.
This hardly seems like a Python specific issue.
I thought this was standard practice?
Unlike 2.7, I do not see a consensus on a 'standard' version for python 3 between 3.4-3.7. (not any of the others, that's for sure)
Have been using Python on and off since 1.6 and not only is the documentation also quite impressive, when one sums up language reference + library, just like ISO documents, its changes between versions make a Pub Quiz a very easy task.
The language is probably the new Basic, but to achieve 10 dan in Python there is plenty of stuff to chew on.
Things should last for lifetimes before they break, not 12 months.