Breaking Changes in Python-3.14
The python people don't adhere to this principle and maybe I need to give up on it, I'm just sick of this crap.
Those of us of a certain vintage will recall the tumult of python-2 to -3. Suddenly, production code needed significant re-writing. We got an arguably better python out of it, but oh! the pain.
In 3.14, (among many other things) the python developers decided to make 'forkserver' the default instead of 'fork' for the Process() method (this is for starting a child process - https://docs.python.org/3/library/multiprocessing.html). Why on earth break our code in such a wanton way? Why not leave the default alone - there was always the option to use 'forkserver' if one wanted it. Or maybe they could have created a new entrypoint with the new behaviour Process_fastserver() or some such? Oh no! Just break it and make their customers patch furiously!
When we adopt a language, we like to think that what runs today will run tomorrow - C and bash programs that I wrote 30 years ago still run. Not with python - if you use it, buckle up and make sure your regression tests are thorough, it'll be a rough ride.
Move slow and break things, perhaps?
6 comments
[ 2.6 ms ] story [ 11.6 ms ] threadfork() is very hard to get right as it keeps a lot of state around from the parent process you have to cleanly discard...
With fork, you could pass objects that couldn't be pickled (lambdas, local functions, file handles, database connections). With forkserver, everything must be pickleable. That alone breaks thousands of repos of code.
You can no longer work with global module-level objects, so it fundamentally changes how scoping rules work.
It launches a server with some extra machinery at runtime - startup cost and hidden complexity just snuck its your app without you knowing.
forkserver may be technically a better choice. But that's irrelevant. Changing the default breaks existing code.
Just a list of bad/wrong decisions IMHO:
- Reference counting instead of using a real garbage collector
- The pyproject.toml format is under-specified, comes decades too late for a problem that has been solved good enough by Apache Maven more than 2 decades ago
- The absolutely weak support for functional programming, which then was patched by list comprehensions and co later
- venv and other solutions to isolate dependencies for a project
Python is successful because of the community support and a binding to almost everything, and this sadly outweighs a lot of poor choices from the language designers and the implementation. I just always feel frustrated, that during the great breakup from 2-3, they didn't try to fix more of the known issues (which again, have been solved by other communities decades before) instead of breaking the world and still managing to half ass it.
The more I use Python in a professional environment the less I think it is suited for professional environments.
It's harder to "not break user space" when you don't have a "kernel space". No, the C API does not count.
> Those of us of a certain vintage will recall the tumult of python-2 to -3. Suddenly, production code needed significant re-writing. We got an arguably better python out of it, but oh! the pain.
I have been using Python since 2005. I recall code that required minor edits, many of which could be and were automated using CPython's own provided tools. I remember the joy of getting text files that could trivially handle a file encoding and universal newline support at the same time, and binary files that were actually binary, supplying a fundamentally distinct type of data. I remember no longer having to worry about getting `UnicodeDecodeError` from an encoding operation (https://stackoverflow.com/questions/9644099) or vice-versa (https://stackoverflow.com/questions/5096776) and no longer having to explain that nonsense to people. I remember the realization that beginners would no longer be taught to introduce arbitrary code execution exploits into their programs on the first day of lessons.
We got an inarguably, massively better Python out of it.
> C and bash programs that I wrote 30 years ago still run.
Because they are very nearly self-contained. Your life will be happier if you think of non-essential parts of the Python standard library (like `sys` and `os`) as if they were third-party.
> the python developers decided to make 'forkserver' the default instead of 'fork' for the Process() method.... Why on earth break our code in such a wanton way?
Everyone always seems to take this sort of thing extremely personally for some reason.
The entire point of default arguments is to reflect what most users are most likely to want. Sometimes that changes. The same document you link clearly explains the reason for the change:
> Changed in version 3.14: On POSIX platforms the default start method was changed from fork to forkserver to retain the performance but avoid common multithreaded process incompatibilities. See gh-84559.
And there is a link to the bug tracker where the OP explains what's wrong with `fork` on POSIX:
> By default, multiprocessing uses fork() without exec() on POSIX. For a variety of reasons this can lead to inconsistent state in subprocesses: module-level globals are copied, which can mess up logging, threads don't survive fork(), etc.
and later:
> Given people's general experience, I would not say that "fork" works on Linux either. More like "99% of the time it works, 1% it randomly breaks in mysterious way".
Note that the discussion for this goes back to 2020.
> Not with python - if you use it, buckle up and make sure your regression tests are thorough, it'll be a rough ride.
You have not even attempted to explain how this broke your program. For many other users, it demonstrably fixed issues.
In the time that you spend posting this, you could have trivially just added `multiprocessing.set_start_method("fork")` to your code; or taken more drastic action by for example vendoring the old standard library package (it's under a permissive license and is as far as I can tell pure Python, mainly layers of wrappers around `subprocess.Popen` and `os.fork` and such).
Aside from which, if you have a proper reproducer for something that works with 'fork' but not with 'forkserver', the proper place for your complaint is the issue tracker.
> buckle up and make sure your regression tests are thorough
Good advice in every programming language. Especially if you want to avoid NIH.