I think it’s just Guido’s Dutch humour not getting across. The toolchain was already recently upgraded to support 3.10, and IIUC can handle 3.999 fine.
There's nothing stopping them from going to version 4, it's just a number, but some people and organizations get really worried about major version changes in some languages.
I like how Go's approaching it, there probably will be a Go 2 but it'll be a smooth transition. I've been keeping my version of Go updated for a while now and never had to change anything.
Who thinks like this? Computer numerologists? Surely what matters is feature support and quality, not big version numbers. I for one am glad that python is stabilizing and old(ish) code tends to work well.
Not behavior, but struct layouts. Since python (as well as most other such scripting toys) never thought about proper enhancement paths, their API works by exposing headers with the structs, and not many abstract types with functional interfaces, you cannot change much. With functions only it would have been easy
The only way to remove the GIL without breaking existing multithreaded code is to replace it with multiple smaller locks. That approach is considered unacceptable because the small locks introduce substantial overhead.
Java has different semantics to Python. To make Python run fast without a GIL you'd have to change the semantics to be more like Java's, which would be a breaking change, which would mean Python 4.0, which is what they say we're unlikely to get.
You have to consider the use case in addition to the technology. The relatively simple implementation of CPython ensures you have fast startup, but trade-off other optimizations which affect long-running programs.
Lots of Python scripts only run for a short time. Code in the REPL needs to return results really quickly. The .NET JIT added significant startup costs to the point where the IronPython team had to add an interpreted mode for first runs before handing off the JIT. This is complex and arguably takes time away from CPython compatibility work.
It seems that the .NET team is coming around to the concerns around startup time, but I don't know if they have landed on an AOT solution. It probably will help executables running on .NET Core more than scripts running under IronPython. I've not kept up with JITs very much, but I don't think this pattern is limited to the .NET JIT, and is a general trade-off. I feel like JavaScript JITs may be closer to what IronPython would have wanted though.
IronPython also wasn't compatible with any python library that used the C API. Essentially we're back to the point that removing the GIL is 'easy' as long as you don't care about backwards compatibility.
But, again, IronPython does not have the same semantics as standard Python, so this would be a breaking change to adopt, and so would need to be Python 4.
Currently Python only allows certain thread interleavings, because those are the points where the GIL is released. If you no longer had the GIL, the set of possible thread interleavings would increase. If your application is not safe with a particular interleaving, it can now break. That's a breaking change, so would require Python 4.
It might be possible to retain current interleaving semantics, through research-level techniques such as transactional memory, but these do not seem likely anytime soon despite a lot of work.
It's also unclear how tractable it is to retain current garbage collection semantics without a GIL, so that could have to change as well with similar issues and similar possible solutions which seem unlikely to land anytime soon.
> Currently Python only allows certain thread interleavings, because those are the points where the GIL is released.
How is this currently avoided? My understanding is that anything that accepts a callback / lambda is potentially subject to interleavings as the code can call a C extension which releases the GIL. Moving code to C extensions is often recommended when Python performance is brought up. Am I misunderstanding something?
> It's also unclear how tractable it is to retain current garbage collection semantics without a GIL
Why is it important to retain current garbage collection semantics? Why does Python prescribe a certain garbage collection implementation and does not allow for others like Java?
The big difference is that Python doesn't allow very much static analysis.
Consider the expression "a.b.c". Where are the function calls? In other words, which of the ".{name}" return a value from some namespace and which call code? (I'm not asking about which functions are called. I'm just asking about where functions are called.)
Java semantics, including the type system and the required declarations, tell you where the function calls are. One consequence is that they're always in the same place.
That's simply not possible in Python because the answer isn't known until the expression is evaluated AND the answer can change every time the expression is evaluated.
For example, in CPython multiple threads can append to a list without a lock or insert items into a dict without a lock, and the data structure will never become corrupted. If you do the same thing in Java with an ArrayList, there's a chance that some items are lost.
Though it is worth pointing out that Java has a well-defined memory model, and Python doesn't. So the above is true for CPython, but it might not work in other implementations, or, for example, you might have a library providing data structures, and it might not hold for those, either.
> For example, in CPython multiple threads can append to a list without a lock or insert items into a dict without a lock, and the data structure will never become corrupted.
How is this achieved?
> Though it is worth pointing out that Java has a well-defined memory model, and Python doesn't.
Wouldn't you have to at least issue an mfence when a thread enters / exists the GIL?
AFAIK multithreaded Java code requires the user to explicitly acquire a lock before accessing a shared data structure. Multithreaded Python currently does not, and must remain that way to maintain compatibility.
Most Python 3 breakage was search-replace fixable (though I'd question the value of language changes falling in this category), although it is true that the text/bytes separation exposed unsound data models in some applications, which is of course nasty to deal with.
How does the the GIL prevent interleaving? My understanding is that if an "operation" takes several steps, eg. reading from a dict then IO or a C extension and finally writing to a dict the GIL will not make this atomic.
There's the performance argument as well, though that doesn't seem as fundamental to me (i.e. a sufficiently smart team of engineers might find a way around that).
I just want to be able to deploy a flask app that responds to a lot of CPU-intensive requests (making inference from ML models) that takes advantage of modern CPUs that have a stupid number of cores, without having to pay the memory overhead of duplicating the entire process' memory space N-times. Native threads would totally fix this inefficiency but the GIL breaks it. Java and .Net would be more appropriate in this production environment but they don't have the same eco-system of packages and we also don't want to pay the development cost of re-writing people's notebooks into another language and all the problems that entails. So instead, our server VMs consume a ton of unnecessary memory. /rant
Then consider writing your application in something other than a scripting language. If you're too “lazy” to write in a lower-level language don't complain when you're application is several times slower and takes several times more memory than it could.
> Who thinks like this? Computer numerologists? Surely what matters is feature support and quality, not big version numbers.
I don't think anyone's interested literally because it's a bigger number - you're responding to an imaginary argument nobody made.
People could be interested in Python 4 because they want another opportunity to make substantial breaking changes for the better, which you can only do on a major increment.
An example could be something like removing the GIL.
> I don't think anyone's interested literally because it's a bigger number - you're responding to an imaginary argument nobody made.
Yet when some browser (Chrome) switched to Very Big Version Numbers, all other browsers followed suit and matched those version numbers very closely. So clearly there is marketing behind version numbers.
I don't think the browser change was due to marketing (or not only). They needed a steady release stream that people just accepted, rather than holding off on installing 5.0, because that's a big change from 4.19. They release often now with many internal changes each time and the version number became pretty much meaningless.
Chrome now releases with "I don't care about your distro release schedule" cadence.
The article is written like there are people that want python 4 regardless of what features it has. And I know from experience that there are plenty of people out there that want new shiny versions of software, but when asked what is wrong with the current release, they can't say.
You can remove the GIL without bumping the Python version number because it isn't actually specified anywhere in the language but is an implementation detail of CPython. If you were willing to compromise CPython's single-threaded performance significantly you could remove it tomorrow. Or if you are willing to give up C extensions you could use IronPython or Jython today!
The part that was so annoying is that if unicode is great - then letting us use the nonconflicting u"" would have let us bring 2.0 libraries into 3.0 land much more easily while ALSO improving 2.x libraries. And with no pain really. If they needed to they could have a done a --compat flag.
It wasn't until 3.3 and then 3.4 that there was finally some give. We got u"" back. We also go some ability to handle the net (ASCII / Binary Transform stuff) in a quasi sensible way.
But boy - was it a screw you until then.
And I got seriously tired of having folks who don't actually maintain any code lecturing on and on about how this was the better way. Sure - if you are in a monorepo. But not if you are library author supporting endless versions of python!
It's partly that but partly completely not understanding how hopelessly broken their packaging system is and the burden that's put on popular library authors that encourages them to have extremely wide compatibility.
Depends on what you mean by substantial changes, but Delphi changed its string type from ANSI to Unicode some time ago, similar to what Python did.
For the vast majority of users it was a very easy transition, due to the substantial work done by the Delphi team to make it so. For a lot of programs, a mere recompilation was required. We had a 300kLOC program + dependencies at that time, and one of us spent a 2-3 days on migrating, fixing small issues and testing.
We've barely started (we produce embedded router/fw appliances.) We've written up our backlog items to start tackling it but when you're under a constant crunch to get features and bug fixes out the door, PM will never prioritize willingly introducing subtle bugs across the entire codebase unless there's an extremely strong reason to upgrade.
Unfortunately "no longer receives security updates" isn't strong enough. Until we can't ship a new feature without Python 3, we'll be on 2 for a long time.
It's this + they broke stuff that didn't need to be broken + made it hard / impossible to target both 2 and 3 and/or code incrementally in 3 on top of a 2 code base.
The other nightmare is if a dependency isn't actively maintained you have also port your DEPENDENCIES over. Straight waste of time in many many cases. And your PM is asking (fairly) what new cool feature do we get? And you say - the program will run more slowly after all this stupid work. And boy - did things run more slowly especially in the early version - you upgraded and then got crap as a result so PM is asking why all this work made things WORSE for users.
The main problem wasn't that 3.0 and 3.1 'sucked' or where unstable, it was that basically no libraries worked. In an alternate universe where the python core developers had coordinated with a handful of the most popular libraries and announced 3.0 with at least usable beta versions of those libraries being available then I imagine the reception being very different.
> wasn't that 3.0 and 3.1 'sucked' or where unstable, it was that basically no libraries worked.
Well, that counts to me as a version fault as well (especially since things like venv were green to say the least), but yes, a coordination would have been helpful.
There wasn't an expectation that libraries would work nor was there an expectation that the transition should be quick. That's from listening to conversations at PyCon 2008 and 2009. Back then I got the feeling that this would result in a bumpy ride later on, but accepted it because I was new to Python. It has similarities to how packaging was treated historically.
I think this precipitated a number of other problems which meant that by Python 3.3 and 3.4 inertia had set in. There appeared to be a bit of a panic that code wasn't being brought over from Python 2. This resulted in flipflopping on a recommended approach which I would argue was an unnecessary problem. There should have been a deeper assessment of the situation.
It was was only in this timeframe that differentiated features started appearing for CPython 3. My experience from the PyCons when CPython 3 was in the early versions was that attention went to possible alternative runtimes. There was a lot of energy going into PyPy, Unladen Swallow, and the like while language features for regular devs were limited.
It's only later that a coherent strategy started to form around porting libraries, providing clear and unchanging guidance on migration to Python 3, and big projects committed to migrating. I'm missing a few things, but this combination started to build real momentum.
The backend runtime stuff and the limitations of CPython still existed, but Cython and other tools helped ML usage of Python grow. I don't think people who weren't Python fans would have predicted that given the performance constraints.
But the intentionally seemed to screw folks who needed ASCII - and hey - a lot of web protocols were ASCII at the time.
Sure, do the unicode thing. But if someone needs to deal with ASCII they need to deal with it. And python 3 was pathetic here. Some folks wrote some very clear posts about this I wish I could find as they explain it a lot better than I ever could.
GvR could just do what Linus did and bump the major version number at arbitrary intervals. Remember when the kernel seemed to be stuck forever at 2.6.xx?
Insofar as semver is concerned, IMO most programming languages move too slowly. Minor version bumps often come with a bunch of backward incompatible changes and deprecations. Any one of them could justify a major version bump.
For languages you can satisfy everyone if you're willing to keep some bloat around. Rust introduced epochs where you can say "yeah, this may be a new compiler, but I wrote this code for 2015 and I like it like that". And it will compile with the language support from 2015. Batching enough changes in an epoch allows iteration in new versions (as long as you don't break compatibility between those epochs interacting with each other).
That reminds me of the original Linux versioning scheme[1] which was supposed to stay 2.x forever:
<odd>.x.x: Linus went crazy, broke absolutely _everything_, and rewrote
the kernel to be a microkernel using a special message-passing version
of Visual Basic. (timeframe: "we expect that he will be released from
the mental institution in a decade or two")
Now we're at major version 5 and still no Visual Basic rewrite. :(
It was just a humorous warning for normal people to not use odd kernel versions. Long ago odd versions were extremely experimental and this was the best example of crazy they could think of.
Python 4.0 doesn’t mean anything. Python 3.0 made breaking changes from 2.7. I think any reasonable person can agree that what was done and how it was done was a mistake. Specifically, some people decided that since X was already being broken, there’s no reason not to also break Y and Z. That happened in 2008 and we’re still dealing with it.
My go to example is the string prefix qualifier. Python 2.x defaulted to byte strings. You could use Unicode strings with the u prefix. Byte strings could be specified with the s prefix. Python 3 went all in on Unicode.
Unfortunately these prefixes weren’t legal in 3.0. This was just a nightmare for library writers who wanted to support both. And it was completely unnecessary. This was such an own goal that some later minor version added them back.
So there’s really no meaning to Python 4.0 because minor versions no longer guarantee there are no breaking changes. And that’s not a great lesson to have taken.
I liken many of these situations in Python to the core team forcing their personal idiosyncracies on people rather than doing what's best. It's like how TCO will never make it into the language, or how the majority of http request libraries (certainly the most popular ones) reinvent the wheel by ditching libcurl, or how package management is a complete dumpster fire. The Python community in general certainly likes doing things their own way, even if that way is obtuse to the extreme.
Instead of dealing with it, I moved on and delibererately extract it from my professional life at every company I go to, to great success. Luckily my job is to deploy things at very large scale and the argument to rewrite all Python as Go (or some other fast, compiled language) is extremely easy to make.
I'm doing the same thing, except it's Rust and Go that I'm pushing (and more often than not, succeeding in getting through).
Python is a liability at this point, not only because of performance but primarily due to it being badly designed. This bad design is at the core of every major Python debacle.
The parent's point seems to be that Python is too opinionated. But aren't Rust and Go even more opinionated, particularly Go? What makes you think Go et al won't continue to make highly opinionated decisions like Python did through its development?
I wouldn't exactly say that. I'm fine with strong opinions made for the greater good. Python's opinionation tends to be to satisfy personal itches people have. Often the justifications aren't good enough for the results.
I have my criticisms of Golang and Rust as well, but Go is finally getting generics. It's not that they were never open to it, but the proposal needed to be good.
Rust seriously needs to stop this rustup, curl | sh nonsense as an install method though. It's really embarrassing for a language that goes around beating everyone over the head about how secure it is. Rustup.rs is an enormous target if you want to go and sneak malicious code out into the world.
I think everyone's opinion of how to achieve the greater good could be construed as just a "personal itch". I am not sure there even could be a singular "greater good" which would be best for everyone.
Furthermore it's often not obvious what the results will be until the design is used in practice. There are lots of decisions that Go has made where I don't see them as clear and obvious wins, but I could be convinced after seeing how developers incorporate the ideas into their code.
Newer languages have an advantage here, because if their opinions turn out to be unsuccessful, then people will just forget about the language. They won't feel like a language they previously liked was made worse, like some felt with the Python 2->3 transition. But as languages like Go age, there will be more people to get upset with each change. So I worry that simply switching to a newer language doesn't really solve the problem, even if you feel like its opinions align well with your own at the current time.
You decide that you're going to write a new HTTP request library in Python.
Do you a) wrap libcurl, a blindingly fast and complete library that has had more man-hours of effort put into it than most other codebases on the planet, or b) write something from scratch?
Because out of the 8 or 9 popular Python libraries in use for solving this problem, only PycURL chose a.
I would probably write it from scratch to avoid bringing in a big dependency which requires a FFI to use it just for a simple protocol like HTTP. What languages use libcurl in their idiomatic HTTP library? As far as I can see, only Rust uses it in the standard library (EDIT: actually it doesn't). Go has its own HTTP implementation.
It's all good! We certainly rely on it in other ways, like the build system and such. libcurl is so ubiquitous I would have totally believed that we somehow developed a dependency on it.
I'm confused as to why this reflects on Python the language rather than random people writing Python code? Is it a complaint that FFI is too hard to get right?
You say "for the greater good" when really you want for YOUR greater good. The people who designed Python made decisions that they thought was best for everyone.
You can be opinionated and make good decisions. Python does the opposite. Its primary focus on user experience and low barrier to entry (by hiding upfront complexity that has a tendency to explode in your face later) made it a popular language, but also severely handicapped it, especially when it comes to pragmatism and suitability to solving hard engineering problems.
Go is very opinionated but the focus is strong on pragmatism and engineering. One could say that this strong focus makes for a very cohesive language. It doesn't support all the bell and whistles but that's not necessarily bad, if it doesn't detract from its purpose. Looking at the engineering landscape today, one can definitely say that Python missed the mark by a very wide margin and Go nailed it.
(Which is not to say that Python is going away, research and data science are its bread & butter)
> how package management is a complete dumpster fire
Poetry finally solves all my woes. You can combine it with something like pyenv so you don't get confused with multiple versions on your computer, and then it just works.
Removing the prefixes is clearly a breaking change, but how is adding them back a breaking change?
Anecdotally, I was mostly happy with the Python 3 transition and have been using Python 3 for basically all new Python code since. Not much will ever be improved if you never take any risks, so I think any kind of innovation necessarily includes "own goals". Obviously not all of them hit the mark, though.
Yeah I should clarify this. So minor versions have always been able to remove packages (as per PEP 4) but, and perhaps this is just my perception more than reality, it seems like the breaking changes of Python 3.x have decreased the psychological barrier for doing that and Python is now more willing to remove old packages (eg PEP 594). Like the genie is out of the bottle now.
It's interesting reading some of the comments on PEP 594 that say "well most of these libraries are in pure Python so you can just include them if you want them" and I'm like "uhhhh... doesn't Python have enough problems with dependency management?"
I compare all this to Java, which to me is really the gold standard for avoiding breaking changes (whatever other faults Java may have).
> the breaking changes of Python 3.x have decreased the psychological barrier for doing that
Perhaps the 2-to-3 transition caused people who dislike breaking changes to move to other languages, leaving only those who are pro-breakage in charge.
Can we please stop complaining about Python 2to3 already. Feel free to use and maintain Python 2 indefinitely or move to a different language. Most of the rest of us will move on since the conversions were not that hard and the improvements worth it and we were given 10+ years to do it.
Python is a fantastically productive language in many contexts. The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again. But whatever mistakes were made weren't malicious and they were learning as they went just like the rest of us. I greatly appreciate their efforts to fix the things that were broken, regardless of the hassle involved in upgrading.
> Feel free to use and maintain Python 2 indefinitely or move to a different language. Most of the rest of us will move on since the conversions were not that hard and the improvements worth it and we were given 10+ years to do it.
Maintaining Python 2 indefinitely is an option actively being taken away from people. More importantly, fuck anyone who wants to direct my budget! There's things written in Python 2 ages ago that don't justify additional developer time in my company. Except the community is making us do it.
If you think migrating some ancient Django app to modern Django is trivially easy, why don't you come do it for us?
Python 2.7 is not getting any security/crypto updates backported. Eventually PyPI's SSL configuration will change to a point where Python 2.7 won't handle it and then that's it. Game over, no more pip for you, even if you froze on < 21.0.
There are Linux distros out there in LTS until late into this decade that shipped with Python 2.7. Ubuntu 18 comes to mind, which EOLs in 2028. They've got a real hard problem to solve.
Thank you. Yes it is good to see some projects stepping up the fill the gaps. I only wonder as time goes on how well socialized these will be...and if people who suddenly find themselves in the position of maintaining an old Python code base will be able to find out what to do.
Tauthon is kind of interesting because in addition to guaranteeing compatibility with the latest official 2.7 release they are also trying to back-port as many python 3 feature as makes sense without breaking the backwards comparability.
You can even still get someone else to maintain it for you! Just not the Python Software Foundation. If it's an important piece of your business it may make sense to have a support contract in place regardless.
Python branches are maintained for five years. Python 2.7 was maintained for an extra five years. You were free to choose any other language with decades of long term support, and you are still free to have this one extended (yes, there are companies selling paid support for Python 2.7 or really old Django versions); not reading what was written on the box then bitching about it is what I would call “nasty”.
Btw you can still return your copy of Python for a full refund.
The lifecycle of software is a lot longer than five years and most people stewarding languages are friendlier to that notion. ANSI C & C99 still have the best support to this day.
Just because Python core can say that they support for five years and act like they're being generous when they do so does not make them less like asshats. Most software is not a living, breathing thing.
But if the libraries and extensions are not maintaining compatibility, you need to freeze your dependencies, and sometimes that means freezing the dependencies of your dependencies, or even your OS, and then you hope your machine doesn't break.
It's not a tragedy, backwards-compatibility is a burden, and I understand maintainers, especially volunteers not wanting to do it. But migration is a pain, I found the conversion was not too hard, but testing the migrated scripts for breakage was time consuming.
If my local electronics shop can still operate fine on win3.1 I'd imagine that there's a fair amount of places where no maintenance at all will be needed
Nobody forced the upgrade to COBOL and it still runs just like you can still run whatever old version of Python you're tied to, with the same downsides as if you ran an ancient version of COBOL.
No that's the point. It's something that doesn't make sense to spend any more money on. It's a tool that's being replaced but that process takes years. Unfortunately not soon enough before the end of Python 2.
The problem is, they are calling it a mistake but they were very well informed of the consequences of doing that they did. Everybody said "just fix integer division and print, finish the migration then start adding new features", and they went ahead and totally ignored that.
Whenever I work on a project that gathers interest in multiple places I get input like:
* If you do X you'll fail
* If you don't do X you'll fail
* X is neat but don't worry about it til later.
I imagine that the people in charge of python, a wildly popular programming language (at the time, and still), got all sorts of input, much of which conflicted. They chose a path and it was a bad choice, a mistake. This is clear in retrospect 13 years later, but at pycon in 2008 (a few months before python 3 officially released) the viewpoint was more mixed - a lot of people thought it was a good path and a lot of people thought it was terrible and others still thought the transition plan was annoying but would be cleared up pretty quick.
It's not so much the issue that it is clear that it was a bad decision. — they admit it, owe up to it, and I do not blame them for making the mistake and admitting it was unfortunate and that they learned their lessons.
What drives me to ire to no end are the many people that defend it on message boards with “Just upgrade; you had ten years.”, clearly not realizing the risks and money involved in such an endeavor.
It's as though the government suddenly start putting something in the water that improves it's health benefits, but will erode contemporary plumping pipes away in ten years and people say. “Just break down your walls and replace your plumbing; you have ten years.”. — it is quite expensive, and risks that my house fall over.
This is a reasonable take, but think about this from the perspective of engineers who've never made any mistakes in their lives. It's hard for them to fathom how anyone else can make a mistake like the Python devs did. The person you're answering clearly had the foresight to know what was the right thing to do, why can't everyone?
Python’s separation between ints and floats is quite weak, so it didn’t make sense for `3 / 2` to give a different result from `3.0 / 2` (1 and 1.5 respectively). Python 3 changed the division operator to return 1.5 in both cases.
This was managable in Python 2 but led to lots of float(argument) calls in every function that wanted to do float division. That lead to unnecessary calls, and unnecessary noise in the code.
Python isn't strongly typed. You got an integer floor operation with -3 / 2 = -2, but -3.2 / 2 = -1.6. This led to errors when floats were expected but integers were possible among your inputs.
In Python 3, it always gives a floating-point result. You can use a floor division operator if you want to round everything (including integers) towards negative infinity.
Python is strongly typed, and your examples show the strong (dynamic) type system in action. An example of a language that is not strongly typed would be Javascript, where you can do 1 + "1". In Python, that gets you a typeerror, because Python is strongly typed.
The issue is that `int(x) / int(y)` would yield an int result in Python 2, but a float result in Python 3. So you had to find all the spots in your code where `/` was being used as an integer division operator and replace it with `//`.
Since Python 2 does not have any form of type annotation, this made the task hard to do automatically and error-prone.
Keep in mind that // was added to python in version 2.2a3 along with `from __future__ import division`. This gave everyone almost 7 years to fix up any existing code before the first release of python 3 even happened. 2.7 only got deprecated in 2020, giving developers a total of 18 years to fix their divisions.
I don't know what the PSF should have done to placate people. Keeping a core feature like division confusing like that isn't a good option, and extending support even longer seems ludicrous to me.
I guess they mean python2 1/2 == 0 python3 1/2 == 0.5 (threw me the first time when I moved as I use this to demonstrate the differences in integer devision in C based languages).
I started reading about Python long before I began programming in it (partially due to a deeply terrible book on the topic I purchased). I ran across the integer division bit and stopped cold. "That," I remember thinking, "is a language feature they will change."
Is that because you didn’t have experience with other languages? Because integer division in python2 worked in the way I had come to expect from Fortran, etc.
No, precisely the opposite: it is because I have tons of experience with many other languages.
I get why some people think a default integer division is a good idea. I do. Having said that, it was clear that Python was aiming itself at being a kind of glue language wherein various functions were obvious, and integer division is not obvious to someone picking up programming for the first time.
If I walk up to someone and ask "What's three divided by two?" I would expect more "One and a half" answers than "Two goes into three just the one time."
Eh, the Unicode situation was irredeemably awful in Python 2 because of the era it was of. Fixing that could never have fitted into what you describe. It had to be a significant breaking change, and even if Python 3’s string handling isn’t perfect (its strings are sequences of Unicode code points, not Unicode scalar values, so they can be ill-formed which was madness to allow; I say they should have gone all in on UTF-8 instead, which trades away some sorely misleading, tempting but practically quite useless functionality in favour of performance and memory usage), it’s vastly better than the situation in Python 2, where you pretty much just couldn’t be sure of anything—on many common configurations, you couldn’t even reliably output Unicode, and might not be able to at all without setting the PYTHONIOENCODING=utf-8 environment variable before starting Python, if my memory serves me correctly (I am glad to say that I haven’t had to deal with python2 for five years now).
True, though that's one of the things that 2to3 couldn't deal with, because it's not possible to do so automatically. It can't have had any idea what was supposed to remain a string, and what was supposed to become bytes.
My problem was that there were other unspecified things that 2to3 couldn't deal with. Imagine the world where the upgrade guide was to run 2to3 and then it had explanations of how to address the remaining string issues. Instead, I found the attitude of "run 2to3, it'll probably work but if it doesn't, then you'll have to figure it out yourself."
In my opinion, the issue was that they used this feature as an excuse to have other breaking features that weren't really needed. I think besides the string upgrade, every other breaking feature should have worked with 2 to 3.
Then they could have had a clear upgrade guide that is to run 2 to 3, then clear explanations on how to fix all the remaining string issues. Instead, I have no clue what 2 to 3 fixes and what it doesn't, leading to "2 to 3 will probably work, but if it doesn't, then you'll just have to figure it out yourself."
That's unacceptable, and as a result caused many libraries to take a long time to upgrade. Personally, I had to spend 10 YEARS to wait to upgrade to Python 3 due to the wxPython library that all of my projects used that took that long to upgrade to Python 3.
yeah, I forgot to include unicode but I actually think they should have completed the print() and integer division change migration (without a major version bump, with import futures) and then done a major version bump (to 3) specifically for Unicode. And I agree- utf-8 has clearly established itself as the preferred encoding.
> Can we please stop complaining about Python 2to3 already. Feel free to use and maintain Python 2 indefinitely or move to a different language. Most of the rest of us will move on since the conversions were not that hard and the improvements worth it and we were given 10+ years to do it.
Have you done such an upgrade with a million line codebase?
No company wants to do that with such a large codebase with the potential of regressions that can cost quite a bit of money.
Reading this article makes it sound as though the experience taught him a valuable lesson and that he agrees that in retrospect Python 3 was a mistake, and it was.
There is quite a good reason why so many platforms and languages take indefinite backwards compatibility quit seriously. — it is simply a very demanding thing of large codebases to rewrite to a newer version of the language with all the potential of subtle regressions.
> The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again.
The article suggests that if they could go back in time, they would not do it and make no backwards incompatible change to Python 2.
> But whatever mistakes were made weren't malicious and they were learning as they went just like the rest of us. I greatly appreciate their efforts to fix the things that were broken, regardless of the hassle involved in upgrading.
It was certainly not malicious, but it was indeed a bad idea which is what the complaining is: that Python 3, or any such backwards incompatible changes should have never happened.
This is why Rust uses editions and many other languages a system where backwards incompatible changes must explicitly be imported and the old behavior continues to be supported indefinitely.
> Have you done such an upgrade with a million line codebase?
Yes. With a transition plan everything was ratcheted over to python3 syntax, then test compatibility. Now developers no longer have to remember the nuance between python 2 and 3 and how to write compatibility layers for both.
Now we no longer need to worry about maintaining or working around issues in dependencies that haven't been updated in a decade, and we don't need to teach python 2.7 to new developers.
> Yes. With a transition plan everything was ratcheted over to python3 syntax, then test compatibility. Now developers no longer have to remember the nuance between python 2 and 3 and how to write compatibility layers for both.
Then tell me, what were the costs of the transition when completed? and what where the projected costs when starting?
> Now we no longer need to worry about maintaining or working around issues in dependencies that haven't been updated in a decade, and we don't need to teach python 2.7 to new developers.
> It was certainly not malicious, but it was indeed a bad idea which is what the complaining is: that Python 3, or any such backwards incompatible changes should have never happened.
> You wouldn't if Python 3 didn't exist either.
This is a bit of a loony argument. There are so many useful features post py27 that may not have been possible without the breaking changes 2to3 introduced. Especially the horrific bytes/str situation in 2.7.
So your argument is that Py3 shouldn't have happened, despite all of the people actually contributing to the language wanting it so (I wonder how much does company with your multi-million line project contributes to cPython's development?) and it unlocking future improvements that have driven Python adoption and usage into the stratosphere... because your company didn't want to invest any time or resources for ongoing maintenance? Of which upgrades are an obvious and important part?
> This is a bit of a loony argument. There are so many useful features post py27 that may not have been possible without the breaking changes 2to3 introduced. Especially the horrific bytes/str situation in 2.7.
This situation could easily have been solved without breaking changes in a multitude of ways.
> So your argument is that Py3 shouldn't have happened
Yes, my argument is absolutely that none of the changes warranted breaking backwards compatibility.
> despite all of the people actually contributing to the language wanting it so (I wonder how much does company with your multi-million line project contributes to cPython's development?) and it unlocking future improvements that have driven Python adoption and usage into the stratosphere... because your company didn't want to invest any time or resources for ongoing maintenance? Of which upgrades are an obvious and important part?
That's merely an ad hominem, not an argument as to why you believe that the changes merited breaking backwards compatibility when there were far less intrusive ways to realize solutions that were close to as convenient but did not break backwards compatibility.
> Yes, my argument is absolutely that none of the changes warranted breaking backwards compatibility.
> That's merely an ad hominem, not an argument as to why you believe that the changes merited breaking backwards compatibility when there were far less intrusive ways to realize solutions that were close to as convenient but did not break backwards compatibility.
You haven't made an argument, nor given any alternatives. You've just complained that the free work of others wasn't to your specifications.
How would you propose handling the str/bytes situation without breaking backwards compatibility?
Refusing to break backward compatibility is how you end up with some of the uncomfortable anachronisms that can be found in PHP and MySQL and other things. A language that did backward compat arguably well is Java but it had to be extremely conservative, a property that has made it popular to replace wholesale with languages that are nothing like Java.
If it helps, you can convince yourself that Py3 is a different language that just happens to be very similar to Py2, and that Py2 needs new maintainers. This happens all the time.
> This situation could easily have been solved without breaking changes in a multitude of ways.
Please, do elaborate. (I'll note that if I were redesigning python3, I wouldn't settle on the design for unicode that py3 uses, something more rust-like is probably a bit better, but there's no way to transition from string-literals-are-bytes to string-literals-are-unicode without breaking backwards compatibility)
> and that he agrees that in retrospect Python 3 was a mistake, and it was.
You need to read it again. What he actually said was he should have provided more support to developers migrating code, not that he wouldn’t have done it.
I refused to get involved in 2to3 migration projects for years.
I recently took care of a 2to3 migration involving a popular web framework and a few technologies for one of my employer's clients. The codebase is large and convoluted but I still found it surprisingly easier and way more interesting than I expected.
There is some actual difficulty in dealing with packages that have deprecated for years, and other gory details. But these days 2to3 drama is mostly a meme. The Python community, with a few exceptions, has done an amazing job at keeping things working.
If you want have a bug you want attention on in an upstream project, the easiest way to get it is to do a task for them first, and doing python 2 to 3 is a usually a pretty straightforward one.
> The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again.
Saying that "probably do some things different" is quite an understatement.
Python 2 to 3 was an articulated train collision in slow motion that spans a decade. I'm sorry to say that, but it was monumentally awful.
I really, really wish that it didn't happen because I think it kept Python back at least 5 years.
People got really burnt by this upgrade process, and the answer was "we said we'd do backwards incompatible changes". "And what, are we going to have this every few years?" one might have though.
So, it isn't enough to tell people to "stop complaining about Python 2to3" and things might be done differently in the future or past.
Yeah, it's done alright for itself hasn't it? Look at the stats, it's a top #2 language on Github, StackOverflow questions, Developer surveys or any other measure. Complaining that it could have achieved this 5 years earlier, I don't see the point. We can just appreciate where it's at now, and the people who were instrumental in getting us here.
What do you think Python would've been like had it never happened? I feel like Python is a success by any measure - especially considering it could've easily been replaced by javascript in a lot of cases without a whole lot of effort. I'm sure it kept it back to some extent, especially in the short term... but now it's thriving, and without some of the legacy stuff that would've turned off some people (e.g. handling of unicode strings).
The migration could have happened more smoothly if they had let both python2 and python3 code coexist in the same project. The problem was forking the language like they did. Why couldn't they make the same interpreter run both python3 and python2 code, for a transition period, deprecating python2 features gradually? So you could have had legacy python2 code and new code written in python3. The low level implementation is not that different to require that big break. The only major difference was on string types, but that could have been handled.
Breaking stuff as they did made no sense. And the problem is that after 10+ years that python3 came out in most Linux distribution if you type python you get python2, and it will probably ever will for maintaining compatibility with scripts that started with `#!/usr/bin/python` expecting it to be python2 and not 3. That is a problem for everyone, for maintainers of the distributions, for developer and sysadmins that cannot assume that the default version of python is 3 (you don't know how many time I wated debugging programs written by people that use Arch or whatever fancy distribution and assumed that /usr/bin/python is python3, and then the script breaks on the Debian production server...)
At least since they made a new language, they could invented a new extension for the files, so the `/usr/bin/python` executable could have detected the version to use based on the extension (launch .py3 files with python3, and .py files with python2).
They should've absolutely maintained a compatibility layer in Python3 for a couple versions (say until Python 3.5?) instead of splitting python2 and python3. Something similar to Perl's "use" feature flag.
> Most of the rest of us will move on since the conversions were not that hard and the improvements worth it and we were given 10+ years to do it.
You are describing it as if it is for everyone easy to convert.
That might be true for fast-paced start-ups and web companies which rewrite their code every few years.
I work in science, in projects which have complex infrastructure and are developed in the course of many years. There is a vast amount of code which is written in the language and virtually nobody has time to rewrite and convert it. Me neither. As a consequence, Python is not really ideal for this kind of work, and I will tend to avoid it in my next job.
It might not seem to matter to you, but a good part of Python's success is actually because of things like Numpy, Scientific Python, and statistics and science libaries which were written by scientists. Some, like Konrad Hinsen (one of the authors of Numpy) are now even considering to move to another language:
At the bottom of the page, you find the note "MoinMoin powered". The Wiki software is the MoinMoin wiki software. It is the only large wiki software with multi-markup support written in Python. It has so far no Python 3 port. This is because people have written it in their spare time, and they have no time to rewrite it. Do you have the time? Would you volunteer for that?
From the point of a naïve language author or a library author, change is part of life and users of their software should just upgrade to the latest version, as you write. But this is not friendly to the users of the software. Users really have other things to do than to constantly rewrite their software according to your latest syntax change or broken dependency.
More proficient languages and infrastructure projects, like C, C++, Common Lisp, Java, and the Linux Kernel, do not have this approach - they maintain strict backwards compatibility. "WE DO NOT BREAK USER SPACE", in all caps - do you recognize that phrase? You can google for it. And this rule has a reason. You can think a few minutes how successful that project would have been without this rule. What if if you want to run a new Python on Linux, you first need to upgrade your Linux kernel, and then you need to upgrade your Linux distribution, and then you find that the new distribution release does not support HTML1, so the library you were going to use is broken, and, oh, your printer won't work with it? This is more or less what Python expects from its users.
The Python developers do not adhere to this - they break backwards compatibility. If one reads the fine print about the release cycle, it says clearly that there might be breaking changes even in minor releases.
At the same time, the Python developers did not want people to realize that Python 3 is not the same language, and used the same file extension for the new language. This has as an effect more backwards incompatibility, for example there are libraries which continue to support Python 2 for some time and then stop that support - without using a major version number increment. As a consequence, Python 2 projects which use such a library will break if using normal tools, which assume that semantic versioning is adhered to. In other words, one cannot rely on that Python library authors adhere to semantic versioning.
All this is, regardless how much Python developers fuss about it, a choice. It would have been possible to keep Python 3 backwards compatible without breaking anything. Common Lisp has done it and...
Python 4.0 won't exist. The Python crew will just ape GNOME and jump to Python 40 after we get Python 3.9.39. Then HN watches the Data Science projects across the world unite in harmony for a heralded new version, distinct from its originals, with no breaking changes. Then HN will watch again when Python 41 comes out 12 months later via a new modern release cadence and we get a 2 to 3 breakage that sends Data Scientists and others around the world into righteous indignations and they blog about it.
Personally, my wish for 4.x backward incompatible changes is simply to separate the core language from today’s huge stdlib. A small, lean stdlib with pip and basic text, numbers and io support would make for a smaller and more easily distributable package. Pip install kitchen-sink would reinstall the old stdlib so no problem for those who depend on it. Anaconda and other distros would bundle it. Debian already splits out the stdlib and makes a mess of it so would be better suited there, too.
I think the actual benefits is nor fit in little ram, but for make easier to mantaing and improve things. In Rust the sodlib is very small, but it have slowed to emerge great idioms. And instead of putting in the standard lib all the code of X popular library of today, them put the TRAITS/bare support that allow anyone to interop and implement better versions of it.
If you take that approach, where libraries must rely on third-party packages even for fundamental operations, how do you avoid the JavaScript/Rust situation where every application requires a huge number of (transitive) dependencies?
I imagine today’s stdlib converted to a pip installable package. The python.org installer would still come with it. Only software engineering types would use the stripped down version actively I’d think. Kind of like miniconda vs anaconda today.
Would the migration had been easier if you could "compile" Python? As in, interpreted/JITed languages are harder to maintain when apis change because you can't rebuild it and see if it builds or not?
I've never written any serious amount of interpreted/JITed code, unless you count C#, which i wouldn't even though it runs on the .NET CLR
For interpreted languages we can expect breaking changes in terms of missing or renamed standard library functions, but there is no reason for breaking changes in the bytecode, even over a 50 year time frame.
There isn't much development going on anyway. The old boys have clawed back their power and aren't doing much apart from protecting their positions at various corporations.
The Python 3 rewrite exploited many idealistic developers, a lot of whom have left.
People who things Python2to3 is was not a big deal are missing the fact that, there was practically no reason to upgrade to python3 until python 3.6, when the dict enhancements were released.
The inertia to upgrade can mainly be overcomed, if there are security vulnerability in existing version or you faster performance for free. Other than that, there are very few cases I have seen where upgrading at cost of breaking backwards compatibility makes sense.
199 comments
[ 4.3 ms ] story [ 325 ms ] threadBasically it‘s a hardcoded regex with two digits everywhere, I assume.
I like how Go's approaching it, there probably will be a Go 2 but it'll be a smooth transition. I've been keeping my version of Go updated for a while now and never had to change anything.
Who thinks like this? Computer numerologists? Surely what matters is feature support and quality, not big version numbers. I for one am glad that python is stabilizing and old(ish) code tends to work well.
Lots of Python scripts only run for a short time. Code in the REPL needs to return results really quickly. The .NET JIT added significant startup costs to the point where the IronPython team had to add an interpreted mode for first runs before handing off the JIT. This is complex and arguably takes time away from CPython compatibility work.
It seems that the .NET team is coming around to the concerns around startup time, but I don't know if they have landed on an AOT solution. It probably will help executables running on .NET Core more than scripts running under IronPython. I've not kept up with JITs very much, but I don't think this pattern is limited to the .NET JIT, and is a general trade-off. I feel like JavaScript JITs may be closer to what IronPython would have wanted though.
It might be possible to retain current interleaving semantics, through research-level techniques such as transactional memory, but these do not seem likely anytime soon despite a lot of work.
It's also unclear how tractable it is to retain current garbage collection semantics without a GIL, so that could have to change as well with similar issues and similar possible solutions which seem unlikely to land anytime soon.
How is this currently avoided? My understanding is that anything that accepts a callback / lambda is potentially subject to interleavings as the code can call a C extension which releases the GIL. Moving code to C extensions is often recommended when Python performance is brought up. Am I misunderstanding something?
> It's also unclear how tractable it is to retain current garbage collection semantics without a GIL
Why is it important to retain current garbage collection semantics? Why does Python prescribe a certain garbage collection implementation and does not allow for others like Java?
Consider the expression "a.b.c". Where are the function calls? In other words, which of the ".{name}" return a value from some namespace and which call code? (I'm not asking about which functions are called. I'm just asking about where functions are called.)
Java semantics, including the type system and the required declarations, tell you where the function calls are. One consequence is that they're always in the same place.
That's simply not possible in Python because the answer isn't known until the expression is evaluated AND the answer can change every time the expression is evaluated.
Though it is worth pointing out that Java has a well-defined memory model, and Python doesn't. So the above is true for CPython, but it might not work in other implementations, or, for example, you might have a library providing data structures, and it might not hold for those, either.
How is this achieved?
> Though it is worth pointing out that Java has a well-defined memory model, and Python doesn't.
Wouldn't you have to at least issue an mfence when a thread enters / exists the GIL?
How do you prevent interleaving of different threads then?
> and must remain that way to maintain compatibility.
Why? Python 3 broke a whole lot of applications and libraries.
The GIL.
> Why? Python 3 broke a whole lot of applications and libraries.
Sorry, I mean it must remain that way if it is to maintain compatibility. We’re assuming that breaking compatibility is unacceptable.
How does the the GIL prevent interleaving? My understanding is that if an "operation" takes several steps, eg. reading from a dict then IO or a C extension and finally writing to a dict the GIL will not make this atomic.
There's the performance argument as well, though that doesn't seem as fundamental to me (i.e. a sufficiently smart team of engineers might find a way around that).
More than Go, which Guido recently described as being the new language that’s most Pythonic?
https://www.youtube.com/watch?v=aYbNh3NS7jA&t=1472s
I don't think anyone's interested literally because it's a bigger number - you're responding to an imaginary argument nobody made.
People could be interested in Python 4 because they want another opportunity to make substantial breaking changes for the better, which you can only do on a major increment.
An example could be something like removing the GIL.
(Not saying that'd be a good idea or not.)
Yet when some browser (Chrome) switched to Very Big Version Numbers, all other browsers followed suit and matched those version numbers very closely. So clearly there is marketing behind version numbers.
Chrome now releases with "I don't care about your distro release schedule" cadence.
The article is written like there are people that want python 4 regardless of what features it has. And I know from experience that there are plenty of people out there that want new shiny versions of software, but when asked what is wrong with the current release, they can't say.
Anybody else remember when Solaris jumped from version 2.6 to version 7 because they didn't want to have a smaller version number than Red Hat?
What will they do now Big Sur is macOS 11?
https://www.theverge.com/2021/6/3/22466394/microsoft-windows...
They went out of their way to break things and make the 2 to 3 transition hard - burned and wore out a lot of people.
Remind me why things like u"" couldn't be used to help cover exist codebases?
It’s right there in the Zen of Python: “Purity beats practicality”. Wait, no...
It wasn't until 3.3 and then 3.4 that there was finally some give. We got u"" back. We also go some ability to handle the net (ASCII / Binary Transform stuff) in a quasi sensible way.
But boy - was it a screw you until then.
And I got seriously tired of having folks who don't actually maintain any code lecturing on and on about how this was the better way. Sure - if you are in a monorepo. But not if you are library author supporting endless versions of python!
On the other hand getting everybody from Fortran 90 to Fortran 2008 will probably take at least one more decade ;)
For the vast majority of users it was a very easy transition, due to the substantial work done by the Delphi team to make it so. For a lot of programs, a mere recompilation was required. We had a 300kLOC program + dependencies at that time, and one of us spent a 2-3 days on migrating, fixing small issues and testing.
Unfortunately "no longer receives security updates" isn't strong enough. Until we can't ship a new feature without Python 3, we'll be on 2 for a long time.
The other nightmare is if a dependency isn't actively maintained you have also port your DEPENDENCIES over. Straight waste of time in many many cases. And your PM is asking (fairly) what new cool feature do we get? And you say - the program will run more slowly after all this stupid work. And boy - did things run more slowly especially in the early version - you upgraded and then got crap as a result so PM is asking why all this work made things WORSE for users.
There was still lots of python2 package installs [1] even after 2.x officially ended.
[0] https://en.wikipedia.org/wiki/History_of_Python [1] https://dev.to/hugovk/python-version-share-over-time-6-1jb8
But then again you didn't have to switch immediately. But it was kinda "expected" that these first versions were a bit unstable.
Now people are using that as an excuse to say "Python 3 is bad?" No, sorry.
I'm happy they went from ASCII to Unicode. (hey, people didn't want to use 'setdefaultencoding' for pure BS reasons, so, yeah, you got it coming!)
Well, that counts to me as a version fault as well (especially since things like venv were green to say the least), but yes, a coordination would have been helpful.
Though it's a bit of a chicken/egg problem.
I think this precipitated a number of other problems which meant that by Python 3.3 and 3.4 inertia had set in. There appeared to be a bit of a panic that code wasn't being brought over from Python 2. This resulted in flipflopping on a recommended approach which I would argue was an unnecessary problem. There should have been a deeper assessment of the situation.
It was was only in this timeframe that differentiated features started appearing for CPython 3. My experience from the PyCons when CPython 3 was in the early versions was that attention went to possible alternative runtimes. There was a lot of energy going into PyPy, Unladen Swallow, and the like while language features for regular devs were limited.
It's only later that a coherent strategy started to form around porting libraries, providing clear and unchanging guidance on migration to Python 3, and big projects committed to migrating. I'm missing a few things, but this combination started to build real momentum.
The backend runtime stuff and the limitations of CPython still existed, but Cython and other tools helped ML usage of Python grow. I don't think people who weren't Python fans would have predicted that given the performance constraints.
Sure, do the unicode thing. But if someone needs to deal with ASCII they need to deal with it. And python 3 was pathetic here. Some folks wrote some very clear posts about this I wish I could find as they explain it a lot better than I ever could.
Of course. Also byte strings were not there yet IIRC in those earlier versions.
But you could deal with it. .decode/.encode to ASCII never stopped working
But beats annoying the rest of the world that needed to deal with non ASCII strings.
Insofar as semver is concerned, IMO most programming languages move too slowly. Minor version bumps often come with a bunch of backward incompatible changes and deprecations. Any one of them could justify a major version bump.
[1] https://lwn.net/Articles/125921/
My go to example is the string prefix qualifier. Python 2.x defaulted to byte strings. You could use Unicode strings with the u prefix. Byte strings could be specified with the s prefix. Python 3 went all in on Unicode.
Unfortunately these prefixes weren’t legal in 3.0. This was just a nightmare for library writers who wanted to support both. And it was completely unnecessary. This was such an own goal that some later minor version added them back.
So there’s really no meaning to Python 4.0 because minor versions no longer guarantee there are no breaking changes. And that’s not a great lesson to have taken.
Instead of dealing with it, I moved on and delibererately extract it from my professional life at every company I go to, to great success. Luckily my job is to deploy things at very large scale and the argument to rewrite all Python as Go (or some other fast, compiled language) is extremely easy to make.
Python is a liability at this point, not only because of performance but primarily due to it being badly designed. This bad design is at the core of every major Python debacle.
I have my criticisms of Golang and Rust as well, but Go is finally getting generics. It's not that they were never open to it, but the proposal needed to be good.
Rust seriously needs to stop this rustup, curl | sh nonsense as an install method though. It's really embarrassing for a language that goes around beating everyone over the head about how secure it is. Rustup.rs is an enormous target if you want to go and sneak malicious code out into the world.
Furthermore it's often not obvious what the results will be until the design is used in practice. There are lots of decisions that Go has made where I don't see them as clear and obvious wins, but I could be convinced after seeing how developers incorporate the ideas into their code.
Newer languages have an advantage here, because if their opinions turn out to be unsuccessful, then people will just forget about the language. They won't feel like a language they previously liked was made worse, like some felt with the Python 2->3 transition. But as languages like Go age, there will be more people to get upset with each change. So I worry that simply switching to a newer language doesn't really solve the problem, even if you feel like its opinions align well with your own at the current time.
You decide that you're going to write a new HTTP request library in Python.
Do you a) wrap libcurl, a blindingly fast and complete library that has had more man-hours of effort put into it than most other codebases on the planet, or b) write something from scratch?
Because out of the 8 or 9 popular Python libraries in use for solving this problem, only PycURL chose a.
Go is very opinionated but the focus is strong on pragmatism and engineering. One could say that this strong focus makes for a very cohesive language. It doesn't support all the bell and whistles but that's not necessarily bad, if it doesn't detract from its purpose. Looking at the engineering landscape today, one can definitely say that Python missed the mark by a very wide margin and Go nailed it.
(Which is not to say that Python is going away, research and data science are its bread & butter)
Not my favorite language, but definitely a stable one.
Poetry finally solves all my woes. You can combine it with something like pyenv so you don't get confused with multiple versions on your computer, and then it just works.
Anecdotally, I was mostly happy with the Python 3 transition and have been using Python 3 for basically all new Python code since. Not much will ever be improved if you never take any risks, so I think any kind of innovation necessarily includes "own goals". Obviously not all of them hit the mark, though.
It's interesting reading some of the comments on PEP 594 that say "well most of these libraries are in pure Python so you can just include them if you want them" and I'm like "uhhhh... doesn't Python have enough problems with dependency management?"
I compare all this to Java, which to me is really the gold standard for avoiding breaking changes (whatever other faults Java may have).
Perhaps the 2-to-3 transition caused people who dislike breaking changes to move to other languages, leaving only those who are pro-breakage in charge.
https://web.archive.org/web/20040214084001/https://www.oreil...
Perhaps it could be released as Perl 11! (https://web.archive.org/web/20190730071149/http://perl11.org... )
Python is a fantastically productive language in many contexts. The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again. But whatever mistakes were made weren't malicious and they were learning as they went just like the rest of us. I greatly appreciate their efforts to fix the things that were broken, regardless of the hassle involved in upgrading.
Maintaining Python 2 indefinitely is an option actively being taken away from people. More importantly, fuck anyone who wants to direct my budget! There's things written in Python 2 ages ago that don't justify additional developer time in my company. Except the community is making us do it.
If you think migrating some ancient Django app to modern Django is trivially easy, why don't you come do it for us?
Where do you see this happening? All I see are people and organizations deciding that they don't want to do the work of maintaining Python 2.
Python 2.7 is not getting any security/crypto updates backported. Eventually PyPI's SSL configuration will change to a point where Python 2.7 won't handle it and then that's it. Game over, no more pip for you, even if you froze on < 21.0.
There are Linux distros out there in LTS until late into this decade that shipped with Python 2.7. Ubuntu 18 comes to mind, which EOLs in 2028. They've got a real hard problem to solve.
There are several projects out there that are supporting python 2.7 with security updates. Tauthon is probably the biggest one.
You can maintain Python 2 yourself just fine, though, if you like.
It appears to only take away from people the option to use the unmaintained Python 2.
Plenty of other languages have been able to provide real long term support to their users for decades.
This is why so many of us in a position to do something about it are ditching Python. No regrets.
Btw you can still return your copy of Python for a full refund.
Just because Python core can say that they support for five years and act like they're being generous when they do so does not make them less like asshats. Most software is not a living, breathing thing.
It's not a tragedy, backwards-compatibility is a burden, and I understand maintainers, especially volunteers not wanting to do it. But migration is a pain, I found the conversion was not too hard, but testing the migrated scripts for breakage was time consuming.
FYI, the cause of my latest migration was that the latest version for the mysql-connector for python no longer works with 2.7. See the changelog entry for v8.0.24 here: https://github.com/mysql/mysql-connector-python/blob/master/...
So something to consider if you want to stay on Python 2 forever.
Is your budget going at all to maintaining Python 2 or the ancient Django you use? I'm gonna guess not.
We all know about how COBOL is alive and kicking but Python's core team decided that reality doesn't apply to them.
* If you do X you'll fail
* If you don't do X you'll fail
* X is neat but don't worry about it til later.
I imagine that the people in charge of python, a wildly popular programming language (at the time, and still), got all sorts of input, much of which conflicted. They chose a path and it was a bad choice, a mistake. This is clear in retrospect 13 years later, but at pycon in 2008 (a few months before python 3 officially released) the viewpoint was more mixed - a lot of people thought it was a good path and a lot of people thought it was terrible and others still thought the transition plan was annoying but would be cleared up pretty quick.
What drives me to ire to no end are the many people that defend it on message boards with “Just upgrade; you had ten years.”, clearly not realizing the risks and money involved in such an endeavor.
It's as though the government suddenly start putting something in the water that improves it's health benefits, but will erode contemporary plumping pipes away in ten years and people say. “Just break down your walls and replace your plumbing; you have ten years.”. — it is quite expensive, and risks that my house fall over.
In Python 2: 20 / 15 = 1
In Python 3: 20 / 15 = 1.3333333333333333
To get the old behavior a new operator was added "//"
In all fairness, the C/C++ and Java communities were and are quite substantial.
Of course the rest of us ought to expect 4/3 as the result of 20/15 ;-)
> 20 / 15 = 1
> 20.0 / 15 = 1.3333333333333333
> 20 / 15. = 1.3333333333333333
> 20. / 15. = 1.3333333333333333
In Python 3:
> X / Y is ALWAYS float division
> X // Y is ALWAYS integer division
This was managable in Python 2 but led to lots of float(argument) calls in every function that wanted to do float division. That lead to unnecessary calls, and unnecessary noise in the code.
In Python 3, it always gives a floating-point result. You can use a floor division operator if you want to round everything (including integers) towards negative infinity.
See https://www.python.org/dev/peps/pep-0238/ for details.
Since Python 2 does not have any form of type annotation, this made the task hard to do automatically and error-prone.
I don't know what the PSF should have done to placate people. Keeping a core feature like division confusing like that isn't a good option, and extending support even longer seems ludicrous to me.
I get why some people think a default integer division is a good idea. I do. Having said that, it was clear that Python was aiming itself at being a kind of glue language wherein various functions were obvious, and integer division is not obvious to someone picking up programming for the first time.
If I walk up to someone and ask "What's three divided by two?" I would expect more "One and a half" answers than "Two goes into three just the one time."
Then they could have had a clear upgrade guide that is to run 2 to 3, then clear explanations on how to fix all the remaining string issues. Instead, I have no clue what 2 to 3 fixes and what it doesn't, leading to "2 to 3 will probably work, but if it doesn't, then you'll just have to figure it out yourself."
That's unacceptable, and as a result caused many libraries to take a long time to upgrade. Personally, I had to spend 10 YEARS to wait to upgrade to Python 3 due to the wxPython library that all of my projects used that took that long to upgrade to Python 3.
No. That's just the Stockholm Syndrome talking.
Have you done such an upgrade with a million line codebase?
No company wants to do that with such a large codebase with the potential of regressions that can cost quite a bit of money.
Reading this article makes it sound as though the experience taught him a valuable lesson and that he agrees that in retrospect Python 3 was a mistake, and it was.
There is quite a good reason why so many platforms and languages take indefinite backwards compatibility quit seriously. — it is simply a very demanding thing of large codebases to rewrite to a newer version of the language with all the potential of subtle regressions.
> The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again.
The article suggests that if they could go back in time, they would not do it and make no backwards incompatible change to Python 2.
> But whatever mistakes were made weren't malicious and they were learning as they went just like the rest of us. I greatly appreciate their efforts to fix the things that were broken, regardless of the hassle involved in upgrading.
It was certainly not malicious, but it was indeed a bad idea which is what the complaining is: that Python 3, or any such backwards incompatible changes should have never happened.
This is why Rust uses editions and many other languages a system where backwards incompatible changes must explicitly be imported and the old behavior continues to be supported indefinitely.
Yes. With a transition plan everything was ratcheted over to python3 syntax, then test compatibility. Now developers no longer have to remember the nuance between python 2 and 3 and how to write compatibility layers for both.
Now we no longer need to worry about maintaining or working around issues in dependencies that haven't been updated in a decade, and we don't need to teach python 2.7 to new developers.
Then tell me, what were the costs of the transition when completed? and what where the projected costs when starting?
> Now we no longer need to worry about maintaining or working around issues in dependencies that haven't been updated in a decade, and we don't need to teach python 2.7 to new developers.
You wouldn't if Python 3 didn't exist either.
> You wouldn't if Python 3 didn't exist either.
This is a bit of a loony argument. There are so many useful features post py27 that may not have been possible without the breaking changes 2to3 introduced. Especially the horrific bytes/str situation in 2.7.
So your argument is that Py3 shouldn't have happened, despite all of the people actually contributing to the language wanting it so (I wonder how much does company with your multi-million line project contributes to cPython's development?) and it unlocking future improvements that have driven Python adoption and usage into the stratosphere... because your company didn't want to invest any time or resources for ongoing maintenance? Of which upgrades are an obvious and important part?
This situation could easily have been solved without breaking changes in a multitude of ways.
> So your argument is that Py3 shouldn't have happened
Yes, my argument is absolutely that none of the changes warranted breaking backwards compatibility.
> despite all of the people actually contributing to the language wanting it so (I wonder how much does company with your multi-million line project contributes to cPython's development?) and it unlocking future improvements that have driven Python adoption and usage into the stratosphere... because your company didn't want to invest any time or resources for ongoing maintenance? Of which upgrades are an obvious and important part?
That's merely an ad hominem, not an argument as to why you believe that the changes merited breaking backwards compatibility when there were far less intrusive ways to realize solutions that were close to as convenient but did not break backwards compatibility.
> That's merely an ad hominem, not an argument as to why you believe that the changes merited breaking backwards compatibility when there were far less intrusive ways to realize solutions that were close to as convenient but did not break backwards compatibility.
You haven't made an argument, nor given any alternatives. You've just complained that the free work of others wasn't to your specifications.
How would you propose handling the str/bytes situation without breaking backwards compatibility?
If it helps, you can convince yourself that Py3 is a different language that just happens to be very similar to Py2, and that Py2 needs new maintainers. This happens all the time.
Please, do elaborate. (I'll note that if I were redesigning python3, I wouldn't settle on the design for unicode that py3 uses, something more rust-like is probably a bit better, but there's no way to transition from string-literals-are-bytes to string-literals-are-unicode without breaking backwards compatibility)
It does exist. Python 2 is dead. The transition was more painful than it perhaps needed to be, but is IMO a better language for it than 2.7, or 2.6.
PEP-3000 was 15 years ago.
Complaining for decades about mistakes, that they admit, with hindsight, won't change anything.
You need to read it again. What he actually said was he should have provided more support to developers migrating code, not that he wouldn’t have done it.
I recently took care of a 2to3 migration involving a popular web framework and a few technologies for one of my employer's clients. The codebase is large and convoluted but I still found it surprisingly easier and way more interesting than I expected.
There is some actual difficulty in dealing with packages that have deprecated for years, and other gory details. But these days 2to3 drama is mostly a meme. The Python community, with a few exceptions, has done an amazing job at keeping things working.
If you want have a bug you want attention on in an upstream project, the easiest way to get it is to do a task for them first, and doing python 2 to 3 is a usually a pretty straightforward one.
> The maintainers would probably do some things different with the 2to3 conversion if they had to do it over again.
Saying that "probably do some things different" is quite an understatement.
Python 2 to 3 was an articulated train collision in slow motion that spans a decade. I'm sorry to say that, but it was monumentally awful.
I really, really wish that it didn't happen because I think it kept Python back at least 5 years.
People got really burnt by this upgrade process, and the answer was "we said we'd do backwards incompatible changes". "And what, are we going to have this every few years?" one might have though.
So, it isn't enough to tell people to "stop complaining about Python 2to3" and things might be done differently in the future or past.
Yeah, it's done alright for itself hasn't it? Look at the stats, it's a top #2 language on Github, StackOverflow questions, Developer surveys or any other measure. Complaining that it could have achieved this 5 years earlier, I don't see the point. We can just appreciate where it's at now, and the people who were instrumental in getting us here.
Isn't this the current situation now? Granted, not as big as Python 2 to 3, but there are backwards incompatible changes often.
Breaking stuff as they did made no sense. And the problem is that after 10+ years that python3 came out in most Linux distribution if you type python you get python2, and it will probably ever will for maintaining compatibility with scripts that started with `#!/usr/bin/python` expecting it to be python2 and not 3. That is a problem for everyone, for maintainers of the distributions, for developer and sysadmins that cannot assume that the default version of python is 3 (you don't know how many time I wated debugging programs written by people that use Arch or whatever fancy distribution and assumed that /usr/bin/python is python3, and then the script breaks on the Debian production server...)
At least since they made a new language, they could invented a new extension for the files, so the `/usr/bin/python` executable could have detected the version to use based on the extension (launch .py3 files with python3, and .py files with python2).
You are describing it as if it is for everyone easy to convert.
That might be true for fast-paced start-ups and web companies which rewrite their code every few years.
I work in science, in projects which have complex infrastructure and are developed in the course of many years. There is a vast amount of code which is written in the language and virtually nobody has time to rewrite and convert it. Me neither. As a consequence, Python is not really ideal for this kind of work, and I will tend to avoid it in my next job.
It might not seem to matter to you, but a good part of Python's success is actually because of things like Numpy, Scientific Python, and statistics and science libaries which were written by scientists. Some, like Konrad Hinsen (one of the authors of Numpy) are now even considering to move to another language:
http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stabilit...
The idea that converting is easy is also not generally true for any kind of infrastructure. Consider the Python Wiki:
https://wiki.python.org/moin/
At the bottom of the page, you find the note "MoinMoin powered". The Wiki software is the MoinMoin wiki software. It is the only large wiki software with multi-markup support written in Python. It has so far no Python 3 port. This is because people have written it in their spare time, and they have no time to rewrite it. Do you have the time? Would you volunteer for that?
From the point of a naïve language author or a library author, change is part of life and users of their software should just upgrade to the latest version, as you write. But this is not friendly to the users of the software. Users really have other things to do than to constantly rewrite their software according to your latest syntax change or broken dependency.
More proficient languages and infrastructure projects, like C, C++, Common Lisp, Java, and the Linux Kernel, do not have this approach - they maintain strict backwards compatibility. "WE DO NOT BREAK USER SPACE", in all caps - do you recognize that phrase? You can google for it. And this rule has a reason. You can think a few minutes how successful that project would have been without this rule. What if if you want to run a new Python on Linux, you first need to upgrade your Linux kernel, and then you need to upgrade your Linux distribution, and then you find that the new distribution release does not support HTML1, so the library you were going to use is broken, and, oh, your printer won't work with it? This is more or less what Python expects from its users.
The Python developers do not adhere to this - they break backwards compatibility. If one reads the fine print about the release cycle, it says clearly that there might be breaking changes even in minor releases.
At the same time, the Python developers did not want people to realize that Python 3 is not the same language, and used the same file extension for the new language. This has as an effect more backwards incompatibility, for example there are libraries which continue to support Python 2 for some time and then stop that support - without using a major version number increment. As a consequence, Python 2 projects which use such a library will break if using normal tools, which assume that semantic versioning is adhered to. In other words, one cannot rely on that Python library authors adhere to semantic versioning.
All this is, regardless how much Python developers fuss about it, a choice. It would have been possible to keep Python 3 backwards compatible without breaking anything. Common Lisp has done it and...
(Keep in mind that Guido works for (with?) Microsoft, who made Typescript)
For what environments is the size of Python's stdlib actually an issue?
Certainly in embedded with MicroPython. But for anything else (even micro-containers), I don't see how the extra MBs can be meaningful.
Is pretty clever way to do things, actually.
If you take that approach, where libraries must rely on third-party packages even for fundamental operations, how do you avoid the JavaScript/Rust situation where every application requires a huge number of (transitive) dependencies?
I've never written any serious amount of interpreted/JITed code, unless you count C#, which i wouldn't even though it runs on the .NET CLR
The Python 3 rewrite exploited many idealistic developers, a lot of whom have left.
The inertia to upgrade can mainly be overcomed, if there are security vulnerability in existing version or you faster performance for free. Other than that, there are very few cases I have seen where upgrading at cost of breaking backwards compatibility makes sense.