It's pretty bold to pin such a specific date when the release date of Debian 12 is not yet known. Debian releases are fully supported for one year after the next release launches, and then the LTS projects keep it going for another 5~7 years.
The LTS project[0] plans for a minimum of 5 year support from the date of initial release, so Aug 2026 is based on the Aug 2021 release date + 5 years. The rough-schedule on the LTS wiki also uses "June 2026" for now.
That's what we've been showing so far on https://endoflife.date/debian (With a disclaimer about what LTS means). Suggestions welcome!
I see the sort of perspective pervasively, but I don’t really understand it. I would guess that most people wouldn’t consider expunging c89 from gcc to be progress, yet this is almost universally the attitude towards python2. What am I missing?
python2 code was not necessarily forwards compatible with python3. 3 was explicitly attempting to correct some regrettable choices in python2 and wasn't syntax compatible as a result.
The effort for implementation and package maintainers to support both versions was considerable, resulting in some notable packages only supporting one or the other.
In the words of the Python maintainers:
We needed to sunset Python 2 so we can help Python users by improving Python faster.
We released Python 2.0 in 2000. We realized a few years later that we needed to make big changes to improve Python. So in 2006, we started Python 3.0. Many people did not upgrade, and we did not want to hurt them. So, for many years, we have kept improving and publishing both Python 2 and Python 3.
But this makes it hard to improve Python. There are improvements Python 2 can't handle. And we have less time to work on making Python 3 better and faster.
And if many people keep using Python 2, then that makes it hard for the volunteers who use Python to make software. They can't use the good new things in Python 3 to improve the tools they make.
Too bad for them. The rest of us get improvements. I'm way tired of being forced to use old C++ compilers because "upgrading is non revenue generating work". If you think rewriting your code in some other language is easier than upgrading to Python 3 then good luck to you.
And modern C++ is still being held back for deathly fear of breaking ABI despite clear improvements it makes impossible. People here seem to underestimate the cost of keeping backwards compatibility forever -- either you change very, very slowly (C) or your change is a clusterfuck of disparate requirements that keep blocking each other (C++).
Of course, "We want this code to run forever the way it is" is a value option that a company can choose - just not with any language it wants. And somehow, people seem a bit reluctant to switch from Python to C to be able to keep running the same programs forever, guaranteed.
You don't need to expunge c89 from gcc for progress to have happened - we've got c99 and c11 and whatever c17 is or isn't already there.
Effectively, however, python2 and python3 are different languages. Since they're interpreted, there's now a problem with '#!/usr/bin/env python' in your script: which python version is being invoked? Which language version are modules targeting? It's a mess.
But the poster child for screwing up a language version upgrade is still Perl.
And not a decade too soon. The Python 2 -> Python 3 migration will be studied for a long time as an example how not to release a new version of your language or library, right next to that Angular thingy. The number of headaches I've had to deal with because of this over the years have just about soured me on the whole Python offering, and I used to be a pretty big fan.
Great that Debian finally pulls the plug on it, that means that most if not all of the downstream distros (such as Ubuntu) will probably follow suit, that's a big fraction of the whole Linux eco-system.
Yes, it is nice to see the python 3 migration is pretty much done, it really could have happened literally a decade ago! While we're all here: which is the package manager to use?
Just fyi Poetry is still using pip under the hood. Also (unrelated to Poetry’s pip usage) if you ever build a dependency from source (the situation where pip executes arbitrary code), Poetry also executes arbitrary code.
Unless you are doing scientific computing or using Windows in which case you should use anaconda. Oh and if you are on a Mac you you use brew if you want to install things like tensorflow or imagemagick etc. You see there's only one thing to do it TM :)
I agree. Yet still most people doing scientific computing will end up with 17 installations on their system. I was just being facetious about the state of package management :)
I've sort of bypassed the whole package manager thingy by using 'Anaconda', it's served me very well so far by insulating me from those particular gotchas.
Some would consider it cheating but I haven't got enough hours in the day. If I were to write some ad copy for Anaconda it would be 'in several years of frequent use this particular user didn't have to resolve a single conflict, it all just worked'.
Which runs counter to just about all of my other experiences with Python and its package managers/distributions/toolchains, some of those stories would end with 'and as a result the whole server was hosed starting from what should have been a routine certificate update' and other such nice adventures.
If you're installing a command-line tool globally, I usually go for the system package manager (e.g. `apt` on debian), or `pip` if I want a newer version. With pip, make sure to use the `--user` flag so things aren't installed with root.
If you're writing Python code, `poetry` has been the least bad package manager in my experience. It has a lock-file, so it should help avoid supply-side attacks (like the PyTorch one from a few days ago [1]), and it means your local environment and CI/CD environment should have the same packages installed.
I'd also recommend setting the [`virtualenvs.in-project`][2] setting to `true`, to store the `.venv` in the same folder as your code, so that when you delete your code, all of the downloaded packages get deleted too. Otherwise they'll just stick around in your user directory and use up a lot of disk space.
I don't think that the Angular.js -> Angular update/rebrand is on the same level.
Migrating a codebase from Python 2 to 3 feels like a "port", doing the same for Angular feels more like a rewrite. :-)
Forgive my ignorance, but what should have been done instead of the way the Python 2 to 3 migration happened?
I know that Python 2 has stuck around for far longer than anyone wanted and that people were (are?) hesitant to migrate from 2 to 3, but I'm not sure what decisions contributed to that.
IMHO: Early versions of Python 3 were outright inferior, and IIRC couldn't handle raw bytes, because "everything should be Unicode". Except, not everything is Unicode. So people went to port, and couldn't. This compounded a problem with libraries, to create a situation where transitioning was risky. Better yet, eventually the Python devs decreed that everyone should move already... so they were going to stop making new 2.x feature releases, and only issue big fixes. But to the folks already hesitant to change, that sounded a lot like "Python 2 will be super Stable for forever".
So IMO, basically what should have happened is accounting for valid use of features that 3 wanted to kill, and actually offering a path forward, from day one, so everyone could do a migration and actually have it work.
Edit: Also (but connected), 3.0 (and IIRC a couple thereafter) should have been marked as release candidates, because they were.
I think you are making some really good points, the fact that the bulk of the initial bugs were found by users and not by the developers is what caused initial adoption of V3 to run into an immediate roadblock. If that had not happened the whole thing might have gone over a lot faster but it caused a lot of people to go from 'Yay, UTF-8 strings' to 'wait and see and let someone else handle the bugs, we'll check back in a year (or two, or five)'.
I think the biggest issue was that they named the executable same. They backported few things from python 3 to 2.7, and recommended to use `sys.version_info` and `__future__` for making code run in both python. If they said they would never rename python3 executable to python and python3 is not python, it would have been easier for everyone.
For us it was a change in string handling that stopped us being able to upgrade. Literally 10,000 things would have to be checked. We saw that red hat was supporting python 2 until 2025 so we decided to keep using python 2 until then. I think that once red hat stops supporting python 2 someone else will keep it running.
They should have made the Python 3 interpreter able to embed the Python 2 interpreter. It would’ve then been possible to mark projects as 3 while they continue to import 2 libraries.
This is similar to Rust's approach of using language "editions" separately from compiler versions (presumably because they learned from this migration).
I've been slowly porting a lot of my legacy code from 2 to 3 on an as-needed basis, there's a few bits though that are just too huge/crusty/etc that they will never get ported.
Doesn't this say something about the language if it's been so difficult for people to migrate from 2 to 3? Yes, it was a major version change, but it doesn't bode well for arguments that Python is a good language to do long-term, maintainable, and large-scale development. Sure, now things are "settled" with 3, but Python continues to get more features. It just doesn't look good for the language if even doing a language version migration was a literal brick wall for so much of the community.
Let me be blunt here: Python sucks, but that's at the level of 'programming languages people complain about and programming languages nobody uses'. However the Python libraries absolutely rock and the ease with which you can get really performant number crunching code out of what is nominally an interpreted language is amazing. Statistics, machine learning, engineering, the notebooks etc, I would not pick anything else for work like that, it's a near infinite box full of powertools.
But it would be a mistake to consider Python anything but luxury glue no matter how much you've used it for production code, it isn't and never will be a bullet proof language, there are just too many ways in which the runtime can surprise you and the long term maintainability of any Python codebase is always going to be questionable, in part because you will most likely have dependencies on stuff that will silently disappear or change out from under you while you're not watching.
I've been writing Python code pretty much since it was first released and there isn't a single piece of code that I ever put together that worked longer than a few years out of the box without some breaking change. This as opposed to just about every other language that I've used, which I find very frustrating because Python could be just about perfect. And I'm still sore about significant whitespace ;)
I have no problem running stuff from a decade ago either on the same box and installation that it was on a decade ago. But virtual environments and pip freeze are closer to mothballing an old box than displaying long term code compatibility and functionality. Their main reason to exist is as a workaround precisely because there are such issues.
The whole idea that you need a multitude of environments to support applications written in the same interpreted language is where it breaks, that should not have been necessary if backwards compatibility had not been an issue.
I started a project in 2016 and used pipenv to freeze all the dependencies. A few weeks ago, I could not get it working on a new machine. Apparently the new system python broke some of the libraries. I had also pinned the python version in pipenv, but it seems like pipenv can only install python versions that are not too old.
I really like the language, but all the tooling is seriously broken.
Pipenv has pyenv integration. It just doesn't work with python 3.6 on my machine. My guess is that it tries running pipenv itself with the old interpreter and is no longer compatible. In any case, faffing around with 25 different third-party solutions is a horrible developer experience.
Fwiw what I've settled on that seems the most stable is to use pyenv as the "outer layer", as it shims the python interpreter along with any venvs so you don't get conflicts / heisen-incompatibility. Lets you use anaconda envs as children as well. The real trick, I think, is to pick one system as the canonical master env controller and don't deviate.
If you upgrade the runtime / compiler, you'll run into changes of behaviour. This really applies to almost every implementation out there. This applies across the board with large enough codebase.
Very strange, about the whitespace. I write C++ most of the time and I love significant whitespace. I like the combination of power and (...these days, relative...) simplicity of the language for all kinds of helper tools. Probably wouldn't want to use it to write something that is large and / or needs to run fast.
Ive always found whitespace to be like variable names, restriction coming from the language always seems to make it harder to use them to convey things to the next person who comes along.
The code typesetting is for the humans not the machines.
The best thing about significant whitespace is the reduction in pernickety code review comments about it compared to other languages. If you also agree to "just run black" to format code then that kind of bullshit drops to almost 0 in a team.
But if you agree on a certain formatting tool anyway, then you don't need significant whitespace to get that. That's one of the things I like most about Go. It shipped with gofmt from day one, so literally no one ever argues about coding style.
People totally do argue about go coding style. Since gofmt doesn’t do line breaks, where to break long lines is a constant struggle. You also have stricter formatters like gofumpt addressing certain problems not handled by gofmt, but the long line problem is still up in the air AFAIK.
The bad thing about significant whitespace is it breaks copy-pasteability of code, with no way to get broken formatting back without understanding what the code does.
That is a lot worse than formatting squabbles, especially for a glue language where you are expected to be able to copy snippets for short scripts all the time.
That and even with version control you will always have to keep an eye out for accidental deletion of the last bit of whitespace on any code block. It's a pretty easy mistake to overlook without context.
Funny, for about every point I feel the opposite way :-). Python is an elegant language that hits the sweet spot where it is very expressive, and easy to do what you want, and at the same time has enough structure so you don't produce a mess all the time.
The libraries are very powerful, but a lot of them are hacks. Numpy and Pandas totally rely on magic and overloading the array indexing operator. You can't express a problem the natural way or it will be slow, you have to think the numpy way. I find it especially confusing if you create expressions with numpy arrays. Am I operating on them element wise, or creating a "cross product", or am I creating some kind of magic indexing object? Matplotlib is also annoying, there are so many ways to do things and half of it is object oriented, while half of it uses global state.
One thing I like about Python are the web frameworks (Django, Flask). Async is also good, although it took almost 10 years for vanilla Python to catch up with what Twisted already had.
About code breaking I also have the opposite experience. I wrote a little Gtk app in the late 2000s and it worked with little or no changes for many Python and Gtk versions, thanks to the dynamism of Python. That would be completely impossible with a compiled language.
Interesting, and nice to have a contrasting datapoint.
I do think that "The libraries are very powerful, but a lot of them are hacks. Numpy and Pandas totally rely on magic and overloading the array indexing operator. You can't express a problem the natural way or it will be slow, you have to think the numpy way. I find it especially confusing if you create expressions with numpy arrays. Am I operating on them element wise, or creating a "cross product", or am I creating some kind of magic indexing object? Matplotlib is also annoying, there are so many ways to do things and half of it is object oriented, while half of it uses global state." is an illustration of my point: the language itself isn't powerful enough to do the job so you rely on a lot of libraries written in different languages to glue it all together.
An 'elegant' language would provide a way to do so without all of these leaky abstractions.
To me Clojure is such a language, but it doesn't nearly have the kind of ease of use that Python has.
> is an illustration of my point: the language itself isn't powerful enough to do the job so you rely on a lot of libraries written in different languages to glue it all together.
The language is 100% capable. Matplotlib's API is horrible and confusing because one of their design goals was/is to be similar to matlab (that's where the "mat" in it's name comes from) to allow people who are comfortable with matlab to switch more easily.
Matlab's plotting facilities are a mess of global state, so matplotlib has copied it.
> An 'elegant' language would provide a way to do so without all of these leaky abstractions.
It would have to do something else, first: Exist.
In 10+ years of programming, I've yet to ever see a implementation of a complex abstraction that was comprehensive and complete enough that I didn't find a corner case requiring me to reach through it and deal with platform specific something-or-other.
You are right, you can't separate the language and the libraries, and a language would profit if it had a way to express these abstractions better. But if you'd that kind of syntax the Python language I think it would become bloated.
I think I like Python, because coming from C, a lot of things that are tedious there are effortless in Python (Strings! Lists and dicts, libraries, and so on.)
I came from C as well and even though Python does make strings, dicts and so on easier the main difference between the two (compiled vs interpreted) is exactly why you get all those libraries in the first place: it is super hard to get good performance out of an interpreted language if you want to crunch numbers.
That's why each and every number crunching problem will eventually make use of the various escape hatches to call libraries written in a language that is performant.
As for strings: BASIC also had strings. That doesn't mean that having strings (or even dicts) is what makes a language elegant.
The problem really is that there are only so many sweet spots for programming languages and usually those come with limitations with respect to the domain you want to use them for. Pick any two: expressive, fast, easy to use.
NumPy/matplotlib make perfect sense for Matlab users (it's basically a port of Matlab semantics). If Matlab were free and didn't suck so much as a general purpose language, I wonder to what extent Python would have been adopted in scientific computing (Matlab is still pretty strong in some communities, to be fair).
My biggest pain point with Python is C / Rust dependencies. So, not really Python per se, as I suspect any language relying on C ABIs will hit the same issues.
The string/bytes encoding lead to some weird runtime problems in a codebase I maintain (the quality of which is probably not that great to begin with). Also changes in non-statically compiled languages will always be a bit tricky. In general, a comprehensive unit test suite is required to step in for the compiler, otherwise there's no confident refactoring. But python is sooooo ubiquitous it's crazy, and most code does not have unit tests to make up for missing compile time checks. The codebase I mentioned earlier certainly doesn't.
I’m pretty cautious about changing my own code, but even I am not paranoid enough to write tests to verify that freakin’ strings still have the behavior I rely on.
I tried adding type hinting to a program that I had to maintain once - because it had no unit tests and I was looking for the quickest way to make it less fragile. One tended to make a small change and then find that it broke something somewhere but finding that out was a long process.
FWIW I found it nearly useless - whereas biting the bullet and refactoring it so that it could be unit tested - even just a few tests - made a world of difference.
> At least not until the day they suddenly didn’t.
That's been happening more and more recently to me. Python 2 really was lax about letting you mix types and get shit done fast, but Python 3 is a different beast entirely. Really opinionated.
It wasn’t so much the difficulty, but how little all that effort got you. It took them until about 3.7 to even reach performance parity. It was all stick no carrot.
Probably not. Java is going through the same exact thing with Java 8 right now. The issue is that humans don't plan ahead for when the software they use inevitably goes EOL, be it programming languages, libraries, or operating systems, and so only really do so when forced (sometimes only after a security incident). In the case of Python 2 and Java 8, the problem was/has been exacerbated by the organization supporting the software continuing to push the support window out further to "give time to upgrade" because in response everyone either said "I have time, I'll get to it later" or "I'll wait for the next newest version".
For Java 8 especially, the later reason is particularly amusing, as Oracle's extended support is now slated to go to 2030 (16yrs after release, for comparison Py2.7 got 10yrs), but Java 21, the next Java LTS that should release next year, will be on Oracle's standard 8 year extended support cycle, and thus extended support only goes til 2031. So why would anyone decide, much less how would one justify to a corporate overlord, to spend all the time and resources to refactor their code, upgrade libraries (assuming the libraries themselves upgrade), update build systems, perform extensive regression testing, and everything else, just for a version that would be EOL one year after what they just upgraded from?
> So why would anyone decide, much less how would one justify to a corporate overlord, to spend all the time and resources to refactor their code, upgrade libraries (assuming the libraries themselves upgrade), update build systems, perform extensive regression testing, and everything else, just for a version that would be EOL one year after what they just upgraded from?
The longer you delay upgrades, the bigger the delta, the more work it will be to update. Instead of doing two medium sized updates, one today and another one in 2030, you'll be doing a huge one in 2030. That might be a trade off you might be willing to make today, but you might regret it in 2029 when you're halfway through a multiyear migration project where all feature work is on hold.
Most Java versions are good about backcompat, almost all the pain comes with Java 9 actually enforcing boundaries and breaking things that had merely been discouraged before (in the name of JDK modularization, which we otherwise aren’t interested in).
The actual migration should you choose to do it was almost trivial. The official conversion script did most if not all the work. The changes were all cosmetic, like "print 42" changing to "print(42)".
I'd say the uproar was about the minorness itself. The syntax had these tiny but otherwise incompatible changes, and for many people, for no good reason. Fo example it was deemed that, as in the example above, print shall not be a statement anymore, but a function instead. Why? Unclear. Meanwhile, Python 3 offered few concrete improvements to many users
So I think the 2->3 moan, of which I was a party, was about that: minor, unnecessary changes that introduced little to no benefit, and yet backwards-incompatible.
> The actual migration should you choose to do it was almost trivial. The official conversion script did most if not all the work. The changes were all cosmetic, like "print 42" changing to "print(42)".
If that were it, it'd have been over a lot sooner. For some packages, bytes vs Unicode strings was bad. For many more, layers of dependencies were an impediment. The trivial stuff wasn't the problem.
But for example a big stumbling block was the scientific programming community, from which I also stem. There were no major syntactical reasons there and yet people were up in arms, for the reasons I describe above.
Most people I know migrated only when python2 literally wouldnt work for them anymore, like they needed a feature of a library only released for python3.
And yet it says a lot that people stayed, I'm not sure it says whether the python 3 move was necessary more than it says something about developers in general.
If this was a frequent event then yes that would be an issue. But Python 3 released in 2008, more than 14 years ago now, and since then there have been no sign of such a new big change. While the core developers briefly talked about making Python 3.10 into 4.0 that was scrapped, and currently the plan is just to continue with Python 3.x "forever".
Any time someone argues for a breaking change that will have oh so many benefits if we just get the whole ecosystem upgraded this one time, it should come with an explanation of how the new version will manage to make the same substantial changes without breaking any code again, or it shouldn't be taken seriously.
A lot of code I wrote over the years handles raw bytes, treating them as strings (sometimes), shuffling them over sockets, doing stupid things with said sockets, etc.
Lots of abuse of how weakly/poorly/forgivingly typed Py2 is.
Doing the same processes in Py3 is actually headache inducing because of the much stronger typing system.
I feel like there was a batch of big software from the 90s and early 2000s that took a really long time to die, and they're not necessarily dead yet: Windows XP. Java. Python 2. IPv4. X11. Free/Open/NetBSD (1). Hell, I was using (adding new data to) a MUMPS-based system on OpenVMS until 2018. But I feel like we are seeing fewer new widely-publicized death marches now, despite many more projects starting. What changed in the community?
What lesson was learned? It seems like a weird corollary of the Streisand Effect: if you announce something's EOL, you'll get more clingy users that keep the product out of the grave, creating a drag on maintenance budgets, but, maybe, if you just stop maintaining and walk away, dependent users will either notice and switch away, or die themselves, and the death will ultimately be quicker and quieter.
MUMPS... arrrrggghhhh.... instant outbreak of skin rash and stress just from hearing the name. Thank you. ;)
Seriously though, that was one interesting language, the number of weird features (the editor is a string) that you'll come across in MUMPS make it fascinating just to study and wonder what inputs into the decision making process around the design trade offs were.
Usenet activity is a very odd measure of deadness to use today.
Although maybe that is exactly what you mean: that the BSDs were already dying in the Usenet era and are still not dead? I think even if they were dying back then, over the course of the past 20 years they may have found their niches to heal and survive in.
Sorry but this is an ill informed comment. Most of the stuff you are talking about is not bad, just legacy. And what are you saying about java? How is that bad?
Also poeple dont like to change shit that runs well, thats why we have so much legacy stuff around.
In the real world, it doesnt matter if your whole damn business is still running their warehouse system on a DOS program running in a god damn DOS emulator on a Win xp VM running on some linux box. What matters is that it works.
I have a friend that I have been migrating his XP VM to new laptops for the last 13 years. XP was the last version of windows that would run his invoicing software and launching a VM is less painful for him than purchasing/learning/configuring new software. He is in his 60s now and close to retirement so for the cost of beer and pizza he has managed to avoid a huge headache.
As mentioned by several people already, the migration process from 2 to 3 was really painful.
It will be easier for the Python core maintainers, but introducing a breaking change with the intent that that version 2 will be deprecated has probably caused more distress overall, especially maintainers of other open source projects which rely on a specific python version.
Let's never have the same kind of breaking change again, please.
> It will be easier for the Python core maintainers, [...]
It wasn't not even that easy, to be frank. 2to3 is not a small project [1] and they had (alas, incorrectly) assumed that there were enough people to have enough tests to be benefited from 2to3, only to be found massively wrong. In hindsight no one had a good idea about language evolution; the modern concensus is a single interpreter supporting multiple parallel versions, also called "editions", but (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it.
Python had that long before js, with __future__ imports.
But editions & friends are for syntax. Python 3 changed semantics all around. Editions are not an option when you’re changing the langage’s entire text model, especially without static typing.
Could they have at least minimised the blast radius? For example, make all of the text/Unicode changes for Python 3 and thats it.
Python 3.1 could then have introduced some of the other minor changes, perhaps behind a future import.
I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version. We couldn’t have done it without six.
It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake.
> Could they have at least minimised the blast radius? For example, make all of the text/Unicode changes for Python 3 and thats it.
Except "that's it" does a ton of work here, as the entire stdlib had to be audited and updated to correctly work with the new text model, plus other semantics change (e.g. the removal of total ordering).
That those were so large and wide-ranging in the first place is why the core team decided that fixing a bunch of other issues could go in as well e.g. reorganise the stdlib, move keywords to functions, remove deprecated stuff, etc...
Those ancillary changes were mostly pretty easy to implement.
> I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version.
It wouldn't have been more pleasant if the breaking changes had been spread over multiple versions, you'd still have eaten all of those, and the early adopter would have gotten even more pain as they'd have needed to implement breaking changes on every update.
> We couldn’t have done it without six.
We did just fine without six migrating a hundreds-kloc-scale repo from 2.7 to 3.5, we basically copied what Werkzeug did, with our own "minisix" module for just the bits we needed (some lifted from werkzeug's and others bespoke e.g. we needed a compatibility layer for CSV).
We could not, however, have done without the 3.0 - 3.3 improvements, the existence of 2.7, or the community experience.
And tests. Having a well-tested codebase was absolutely essential to a comfy migration, the less a corner of the codebase was tested, the more migration bug kept popping up afterwards.
> It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake.
No, doing all those breaking changes version after version would have been a lot worse. IME the vast, vast majority of the pain (short and long term) came from the semantics changes, not the syntactic ones. Those were relatively easy: do the syntactic change, enable the corresponding lint (so colleagues don't reintroduce the old construct out of habit), commit. A few of the API changes were frustrating (like base64 which was clearly mis-converted by the core team but what do you do).
- the text model stuff took by far the longest and had by far the longest tail (as in things we missed during the migration), this is in part because not all of the changes or Python 3 APIs are ideal, but mostly it's just a lot of runtime things in a lot of paths if your software leverages strings a lot
- total ordering removal broke a surprising amount of things, and also had a pretty long tail of bugs, mostly in that `None` would get mixed in with other values and get fed to comparison operators, sorting, or min/max
- `round` was a surprise contender for pain in the ass, not just because it changed the rounding method (to banker's rounding, from I don't remember what) but also because P2's round() always returned floats while P3 returns integers if precision is missing / None
- changes to dict iteration also broke a few things, there were multiple pieces of code with implicit reliance on iteration order, although for the most part they were issues in numerical stability of floats, which broke tests using strict equality on floats
- division changes also broke a few things as it wasn't always easy to recognise whether operands were always floats or could be int, also `//` being floor division rather than integer division led to a few behavioural changes
We also did a few things like banning the builtins which had changed behaviour (open, map, filter, ...) for the duration of the migration, the issues they caused were subtle enough as to not be worth it. They were...
> Could they have at least minimised the blast radius? For example, make all of the text/Unicode changes for Python 3 and thats it.
Most of the other changes were kind of trivial though. print → print(), some stdlib name changes: that's easy stuff; you can automate that, even provide a compatibility shim. The division operator was a bit more tricky, but I never encountered code that broke due to it (although I'm sure some did, it didn't seem like a wide-spread issue). Plus you could do most of these things in Python 2.
> It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible.
You should have seen the initial discussions surrounding Python 3000, as it was called back then. The most common thing Guido had to say was "No, that's far too large of a change, we're not designing a new language", even for things where it was agreed on that the situation wasn't ideal. The community came up with some pretty wild proposals, and a lot was kept out of Python 3.
The difficult changes between Python 2 and Python 3 are not static information, at least not without fully typing the entire codebase, which was even less feasible in Python 2 than it is in Python 3.
The idea of compilers and interpreters providing backward compatibility options to support programs written using older dialects did not originate in Javascript.
That only works with “syntactic” updates, not with deep semantics changes.
Those are why the core team pulled the trigger on an overhaul update. And they were by far the biggest issues during migration, anything which would be gated behind an edition or opt-in was easy.
> In hindsight no one had a good idea about language evolution; the modern concensus is a single interpreter supporting multiple parallel versions, also called "editions", but (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it.
Perl did it since ages. You just put
use v5.24
in header and interpreter enabled those language features on file by file basis
Perl's behavior was utterly broken. It worked more like how a C/C++ compiler handles language standards (by lying to you) than actually enforcing behaviors.
From the toolchain and language developer side, nobody liked Perl's approach. That's why there was so much effort around Perl 6 (which became Raku) at around the same time.
That isn't even remotely true and definitely was not why so much effort into Perl 6 happened. Perl 6 was a different language with Perl-ish semantics and why it was renamed to Raku to get it out of the Perl namespace. The "use version" behavior has been used forever and is not "utterly broken". It isn't even "broken" at all.
At Yahoo, we had a wrapper so we could have 5.6 and 5.8 installed (and later 5.8 and 5.10), and if you had use 5.6, you'd get 5.6, and use 5.8, you'd get 5.8, and neither, you'd get the default set for that machine.
If you tried to run a 5.8 script on a machine without 5.8, you'd get an error as should be expected. If you ran a use 5.6 script on a 5.8/5.10 machine, I think you'd get 5.8, but I can't remember... 5.10 was less disruptive than 5.8.
I see people saying that the migration was painful, but my own experience with two separate Python 2->3 migrations were so smooth they might as well have been a minor version bumps.
Some time ago we migrated a few hundred thousands lines of Py2 spread over dozens of services. Easiest case possible: every service can be converted and tested separately.
Took a calendar year of ca-a-a-arefully migrating these one by one, all while delivering stuff and making sure nothing falls apart along the way.
It was easy if you were working on your own packages. It was quite difficult if you were dependent on external packages that had no plan to upgrade or if your company had internal packages which were Python 2-only and required in most services.
See my other comment, we started a Python 2 project in late 2013 (Python 3.0 was released in 2008) because of all the dependencies we wanted to use were not ported (yet, no clue if they ever were).
Or all the people maintaining large python in house projects that had a certain expectation of how many people it takes to maintain a mature largely finished project which is suddenly much higher. I supported our framework adding a feature here and there, on boarding a new app team here and there for like 10% of my time. To switch it all to Python 3 would have been major work, high risk, and no real benefit. We either still run Python 2 now, or have migrated to Go (slightly more verbose code and slightly slower dev time, for much lower compute foot print).
To be fair it was a place where the Python fans advocated for its use but to management was always a bit skeptical, so we had to make it all seem easy. We had good solutions to internationalization which all broke with Python 3.
Python 3.0 was released almost 15 years ago and it was clear at that point that Python 2 would be shown the door eventually. My empathy for any complaints regarding that switch is exactly 0, even for open source projects.
I've had to deal with a lot of stupidity regarding Python, and that started with assuming that /usr/bin/python is Python 2. Why? Because that was the default in distros for just way too long, and some of the problems only got fixed after the distros went with Python 3 as the default. Obviously some people need to get the rug pulled under their feet before they start moving.
Depending from which version you're upgrading from. I think there are breaking changes between 3.7 (or 3.8?) and 3.10 regarding the asyncio loop. I had (simple) servers cause exceptions because they used `loop` as a keyword argument which got removed without proper deprecation.
In the 3.9 release notes it says "Aliases to Abstract Base Classes in the collections module, like collections.Mapping alias to collections.abc.Mapping, are kept for one last release for backward compatibility. They will be removed from Python 3.10" but the 3.10 release notes don't really mention it. Apparently it's been emitting a DeprecationWarning since Python 3.3 (2012), but still... I like to have things "just work" even if I return to them after 5 or 10 years.
There were some breaking changes within Python 3.10, mostly esoteric features that didn't seem to be used that much. Lots of deprecation warnings though and some C API changes that did break at least one package I used.
When I worked on patching/rebuilding all the Python packages for Alpine's upgrade to Python 3.10 I remember quite a few things being broken and needing patches (maybe about 5% of the overall packages), but a lot of those were also test failures due to the new deprecations, so while some stuff did truly break, it was pretty rare.
Glad this is finally mostly over, but... that was bad. I wonder if the energy that had to be put into this migration by everyone involved was worth what seems to be relatively small improvements.
The print-as-a-statement was ugly but convenient and didn't seem like a big deal, the integer division was something that you could live with once you knew about it (and you still need to know what the current behavior is), and the new string types make code more verbose, and the way they were handled introduced countless hard to find bugs during the migration.
The fact that such a talk exists shows that the improvements aren't worth the switch for most users. If they were, they wouldn't need convincing.
Yes, Python 3 is (mostly) better than Python 2. No, it's not even remotely worth the amount of confusion and work it caused. If Python 3 had brought massive performance improvements, or proper support for multiple threads, then maybe it would have been. As it stands, Python's biggest weaknesses remain unaddressed.
> The fact that such a talk exists shows that the improvements aren't worth the switch for most users. If they were, they wouldn't need convincing.
That's reading far too much into a tongue in cheek presentation title.
Besides, good devs are typically reasonably skeptical of new and shiny.
And when you've got a large Python 2.7 codebase, and Python 3 is backwards incompatible, you'd be looking at a lot of work to upgrade, so damn straight you'd need convincing to make the investment.
Hell, look at how many Java codebases are still 1.8 because of the upgrade cost of going past that, even though modern Java has many compelling features.
And expecting Python 3 to remove the GIL is incredibly unrealistic. How much Python code in the wild implicitly depends on the behaviour of the GIL? All of it.
A lot of codebases could be converted from Python 2 to Python 3 with just some fairly mechanical work and light refactoring on top. GIL would be a thing of a completely different magnitude.
OCaml 5 recently solved the global lock problem by introducing concurrent domains in which one or more threads share a lock. So old code keeps spawning threads with a shared lock, but new code can work with single-thread domains and manage in which domains does old code run.
Python 3 paved the path for the changes that you mentioned you wished for, some of which are being tackled now. It's not only about what Python 3 offered at the time, but also about what it made possible in the long run.
While that specific example is stupid the general question is very valid. I can't see what paved the way. The most invasive change was the unicode one and even that seems irrelevant to eg the GIL.
That's a bit cynical since I think if we were still dealing with the issues python 3 fixed today, I'd be very disappointed in python overall.
And, the latest iterations of Python3 have real usability improvements - the error message improvements alone make development a lot nicer. And the perf improvements and jit coming I think in 3.13 will really make more people consider py3 for their new projects.
> No, it's not even remotely worth the amount of confusion and work it caused.
Have you worked with pervasively non-ASCII text in Python 2? Like on a machine where any file might suddenly turn out to be non-ASCII (even if it’s only in a comment), not just data inside a few carefully-patrolled fences? Outside of a few well-behaved libraries (Flask), my experience was that it was utterly impossible. More than half of my time debugging straightforward code was spent chasing down and fixing UnicodeErrors, knowing that some will still remain. And that includes code that used nothing but the standard library.
Now you might think that’s not important, and though I disagree (especially when talking about “most users”) that would be fair. But it is a substantial improvement going from 2 to 3.
There is another universe where Python 3 only fixed the unicode strings, and the transition was a huge success and over in a couple of years.
The print() function was absolutely not worth all the carnage it caused. They could have just kept the print statement around and introduced a printing function with a different name, for example.
The fact that they removed syntax for Python 3 meant that all kinds of scientific computing code had to be transitioned even if it didn't use any strings!
For printing it's not a problem because a string containing unicode will be transparently printed in both Python 2 and 3 (at least on Linux/Mac, not sure how Windows deals with it).
It's also not a problem if you only have ASCII strings.
The print function was the simplest part of the transition. For one you could find and replace all instances of it fairly safely across your codebase and you could quickly discover any places where things broke because of a bad replace. That part took like an hour for a large codebase.
The Unicode handling on the other hand took ages because string handling is not easily searchable and replacing it requires understanding what the piece of code actually does.
So you have it exactly backwards: if the only migration was the print function then it would be a quick and relatively painless process. But fixing the hard thing which was the broken str/byte model was the hard transition.
I am curious if you actually went through this transition or if your opinion on this is more based on observation of others’ work.
Python is a scripting language. Imagine if bash suddenly changed its syntax so that that instead of printing using echo, you now have to write echo(). Sure it’s just a pair of parentheses. But you probably have thousands of shell scripts on your computer. Are you really going to get your hands dirty and risk editing them without knowing what they do?
Of course not, you’re just going to revert back to the version of bash that doesn’t require parentheses and sit on it for as long as you possibly can.
Code exists in three states: actively maintained, done/mature, and obsolete.
Python 3 attacked mature code, forcing developers to either go back into active maintenance or to obsolete it early.
I did transition some Python 2 code to 3. But mostly it was uneconomical and risky, so the vast majority of it sat on Python 2.7 until it was no longer needed.
I feel that most of the people who had a positive experience with Python 3 were people doing webdev using Django. They were working on monolithic actively maintained projects and so transitioning to v3 was just a normal maintenance activity.
You are correct that my experience is mostly though not exclusively we Django projects. I guess maintaining your code pays off in the long run.
Again, you are arguing that running 2to3 and testing your code was hard. I guess that may be true. But having hunted for a ton of subtle Unicode bugs that problem was very worth fixing.
> There is another universe where Python 3 only fixed the unicode strings, and the transition was a huge success and over in a couple of years.
Didn't the unicode changes go on until at least Python 3.6? The issue that most C APIs did not care about their byte strings having any encoding at all seemed to cause one headache after another, especially on Posix, where everything touching the operating system could return random bytes. Reducing the scope to unicode wouldn't have sped up the transition one iota.
Not the one you replied to, but we were doing things mostly without strings (audio processing, stuff with bits) and python 2 worked perfectly fine. We had started this project as late as 2013 because the important libraries had not been ported yet and when the stuff wrapped up in 2017... I'm not sure if we could have switched by then.
So personally I'm ok with Python 3 but I actually never ran into the described problems despite using Python 2 for years, so it was at least not a universal problem that needed to be solved in such a breaking way.
So not a single Cyrillic path in your INI or CSV files, or in the user input (CP_ANSI) to be printed onto the Windows console (CP_OEM, unless you’re in IDLE or redirecting the output, in which case CP_ANSI)? (The standard library’s CSV module was particularly atrocious at Unicode as I remember.) Not a single Chinese comment in your hardware descriptions? I envy you, but that’s a still a very specific corner of the world.
We were working exclusively with filenames (and files) that we generated ourselves, moving them around in our own infrastructure, and writing all of our own configs, yes :P
I didn't insinuate that there weren't problems or that I wanted to keep Python 2 - just that it was perfectly normal to use it and not run into problems. I think I only remember a handful of encoding problems with my other python 2 adventures...
Look, the breaking change to python3 was a disaster.
There’s no need to sugar coat it.
It was incompetently managed, in a way that made the technical success of the work look like a failure.
That’s on them. They screwed up.
I lived through it; I feel no particular need to smile and nod and say “it wasn’t that bad”. It was bad.
Things are good now! Python 3 is great, it’s well supported and the people involved all learnt a lot about how important managing communities is, as well as writing good code.
So, all round, a success!
…but, if you don’t acknowledge failures in the past, you’re doomed to repeat them.
> Now you might think that’s not important
I don’t think your experience with Python 2 is not important; I had similar issues with it.
No one is seriously going back to Python 2 at this point.
However, the process is a lesson worth studying; and the take away is not “what a technical success!”
… it is: never do this.
Big scary breaking migrations are bad, they hurt your community, and leave bitterness behind them.
Even if the result is better, technically; there are other factors that need to be considered too.
> No one is seriously going back to Python 2 at this point.
No, the community is stupid and wrong. My machine learning professor (actual professor, not a teaching assistant) only used Python 2 in 2017. Imagine how many people had their opinions shaped by someone like them.
> … it is: never do this.
By that standard, we would never get things like angular 2 or Raku.
Would it be better if we renamed Python 3 to some other name and pretended it was a completely new language?
> Would it be better if we renamed Python 3 to some other name and pretended it was a completely new language?
Quite possibly yes? It doesn't run python2 code, it can't be inter-linked with python2 code, so is it really the same language? The hybrid dialect that works in both does exist but you have to have compatibility shims in a few places.
By comparison, most C compilers will let you use C89 and C17 code in the same project provided you set the compiler options, and C# provided the "netstandard" target for libraries to smooth the transition between Framework and Core runtimes.
I mean, Russ basically said that about all the work people were doing on the golang package manager and did his own thing, and it worked out. Good job! Sometimes, a smart person can do something that is better when they're deeply steeped in the domain.
> No, the community is stupid.
I personally find that most communities have a lot of very clever, very thoughtful people in them, and if you listen, you can learn a lot from them.
Just because they don't agree with you, doesn't make them stupid.
I think that its basic respect to listen to your community, acknowledge them and think about what they say, even if you go your own way.
...but, if you (as a maintainer) just think they (your community of users) are just a bunch of idiots, do whatever you like. They are idiots if they hang around in a community like that.
Yes, I have and it was simple but with plenty of footguns for developers who had not groked how you were supposed to work with strings. I am happy we moved away from that but I am not happy with the solution Python 3 picked. I think e.g. Ruby 1.9 came up with a nicer solution to the problem and one which broke much less code.
Uh, Python 2 was aboslutely fine at working with non-ASCII text as long as the only operations you try to do on it are "take the length" and "reproduce a string verbatim". Notably, it did not forcibly explode when encountering a character appearing in a comment or string literal or whatever that couldn't be interpreted as utf-8. This was a real step back for code that, oh, say, used filesystem interfaces to look at untrusted filenames.
Not if “reproduce the [possibly Unicode] string verbatim” includes concatenating it with others in order to insert it into some sort of context, that will absolutely blow up if you’re not careful and unfortunate input data comes in (I hope that happens before the code enters production!).
For comments and string literals in Python code, note that Python 3 changed the default from Latin-1 to UTF-8, but you can still use -*- coding: -*- to change that. Bytes literals outside of ASCII range you’ll still have to escape. As for program input that’s possibly invalid UTF-8 but not in places you care about, you’ll need to set errors='surrogateescape' or similar explicitly (or possibly set an 8-bit encoding like Latin-1, though not if there’s a chance of UTF-8 in places you do care about).
Paths are a bit painful, yes, both because there’s nothing on Unix systems that precludes /home/alice having filenames in UTF-8 and /home/boris in KOI8-R, and because NT paths are not byte sequences at all (they are WCHAR, that is 16-byte-number, sequences, in no way required to be representable in the current CP_ANSI). Having uninterpreted byte sequences as filenames would work to solve the former issue, and it does: I think most functions in os will accept bytes paths as input and treat that as a signal that you want bytes paths as output (if any). For the latter the cleanest solution is probably an abstract Path type—and you get one, in the standard library’s pathlib. That didn’t work all that well in early Python 3 versions, but it does now. (And IIUC inspired Rust’s abstract-type solution to the problem.)
This is true, almost all "string" operations require you to know extra information about the string's "type"—which, yes, includes its encoding.
Python 3 takes this fact and runs with the stance that strings should, by default, carry their encoding with them (and that encoding is specifically, more or less, a sequence of quasi-Unicode code points with unspecified internal representation). But this is an intensely half-baked solution to the wrong problem, because you need to know much more about a string than its encoding for just about any string operation you'd want to do. Just with Unicode code point concatenation, you can run into plenty of trouble with:
- strings containing control characters
- strings with unpaired surrogates (which okay, shouldn't really be there, but get used for various reasons anyway)
- strings containing combining characters
- strings in different natural languages
- strings in languages with ligatures
- strings with ligature-like emoji
(and this is obviously not remotely close to exhaustive.)
Would you still feel that way if Python had a series of upgrade versions that "hand-held" the migration and giving warnings about behavior that was going to break?
This talks title probably comes from the awful python 3.0+ versions. 3.0 was so bad, they had to fix and revive certain in the later versions. IIRC 3.3 was the first version who was considered decently enough, that someone you could use it for more serious things.
Also, the fact that there is a talk with such a name does not mean anything other than that someone is out there trying to clarify what the transition is about.
(There could have been a video "Windows 7: Trust me, it's better than Windows XP" and you probably still would say that it showed that improvements aren't worth the switch for most users.)
Unless I'm missing something what in here supposed to justify a decade of pain? Hell most of it could have been easily done in python 2 and slowly via warning and errors transition codebases.
The changes Python made was made in Perl without breaking compatibility, you just wrote say
use v5.24
in header to use given feature set (defaulted to something old to not break old stuff)
The Py3 approach was terrible and wasted untold amount of hours just because you had to migrate everything, you couldn't just upgrade codebase piece by piece like in case of Perl.
At the very least they should've just made new one be under .py3 and just make Python3 interpreter transpile Py2 code at runtime
Isn't that effectively what was done with Python? You'd do "#!/usr/bin/env python" for old code and "#!/usr/bin/env python3" for new code. Rather than it being wrapped up in a single entry-point, you had the different runtimes and library sets.
No, it's completely different. The newer Perl versions correctly continued to execute the old but already working scripts.
The Python mess was, IMO, a typical example of bureaucracy inventing for itself new but previously unnecessary work to justify its existence, so I agree that that the decisions of how to introduce Py3 features caused (and still cause) waste of immense amount of the hours world-wide that could have been used more productively. Sad.
For sure. I'd argue it was debatable to even "normalizing" all the built in packages to be consistent.
I think very few people had trouble remembering them, and if you're using one, you'll be looking up documentation for anything with more than a few methods.
It really is frustrating in these situations with high level languages. It's super possible to do the majority of changes they did in non breaking ways.
Hell, they do it already with the __future__ package. Where did all that compatibility work go in python3?
> No, it's completely different. The newer Perl versions correctly continued to execute the old but already working scripts.
There was some breakage between 5.6, 5.8, and 5.10; Unicode is hard, and it takes time to get it right. But I think the key difference I've heard as a mostly python avoider is the intent was for code written for 5.6 to probably work in 5.8 and 5.10 and if it doesn't, for there to be a way to have one file that works for all versions.
From what I understand, it's not easy to have a python module/script that works in 2 and 3, and you can't go to 3 unless your dependencies do, so if you have a lot of dependencies (as is modern), you're stuck on 2. Your dependencies won't want to move to 3 either, because their users are stuck on 2, so if they just switch to 3, they're droping users; instead they need to support two parallel versions of their code. Most perl modules didn't have to do anything special to support 5.6 and 5.8, but if they did, it was usually small and it could be done within the file with conditional compilation --- I don't think that was an option for python.
No, Perl 6 is a completely different language (now called Raku, because otherwise it sounds like a newer version of Perl 5, and never intended to be compatible with Perl 5). But because of the Perl 6 situation, even a change the size of the Python 2 to 3 change would be a "Perl 5" version update.
I think having Python 2 vs. 3 is a clearer demarcation, myself, but the Perl community cares a LOT about backwards compatibility.
The people working on Perl still support running in old mode. You can grab the latest Perl 5 release and run old-style Perl code in it. The people behind Python don't support Python 2.
Just because the old way was deprecated doesn't mean that it wasn't possible to run old and new code next to each other.
Also, Perl 5 is a very different example. While I haven't followed Perl for quite a while, ISTR that Perl had a very similar transition from 4->5 (I personally remember spending a lot of time trying to get old code working on newer OSes). Also, AFAIK but I may be wrong, Perl 6 has not really taken off like Python 3 did.
That means the piece of software will work also in the future.
Personally I only use python for throwaway scripts. Whatever is written in Python likely won’t work in 10 years anymore. So they’re all lost in time. Whereas I can continue some of my C++ eternity projects and they still work perfectly fine.
You don't. The directives in Perl code tell the Perl interpreter, even the very latest one and it's the only one installed on your machine, how it should intepret loading the code.
If you don't write special in your code, Perl doesn't have a "say" method (because it didn't have one in Perl 5.8). If you declare in the code that you're code for Perl 5.10, magically there are all the Perl 5.10 features, including "say". You can also declare you want specific features, rather than declare a specific Perl version.
$ perl -v | head -2 | tail -1
This is perl 5, version 30, subversion 3 (v5.30.3) built for x86_64-cygwin-threads-multi
$ perl -e 'say 1'
Number found where operator expected at -e line 1, near "say 1"
(Do you need to predeclare say?)
syntax error at -e line 1, near "say 1"
Execution of -e aborted due to compilation errors.
$ perl -e 'use 5.010; say 1'
1
$ perl -e 'use feature "say"; say 1'
1
The best thing about this is you can mix and match libraries from earlier and newer Perl versions. If a library says they want Perl 5.10, they get Perl 5.10, even if you want both Perl 5.30 and that library.
If only Python were as reliable for backwards compatibility.
The Perl community is going through their own Py2->3 moment, and they're doing it with great care. They're going to create Perl 7, which will be Perl 5 where the "defaults" make the language look more like Perl 5.32 rather than Perl 5.8 [1]. Modern Perl scripts don't need so much boilerplate to enable modern features. They know exactly the trouble this will cause; if Perl 7 ever becomes the default Perl interpreter, the only one, it _will not run_ lots of very old Perl scripts untouched, because the defaults will make years worth of language changes suddenly visible to those scripts. Even if the fix is just to slap "use 5.008" on every script, they worry about it. That's a much better attitude than the Python developers.
> The directives in Perl code tell the Perl interpreter, even the very latest one and it's the only one installed on your machine, how it should intepret loading the code
> That's a much better attitude than the Python developers.
You do realize that this comes at a cost, right?
A lot of decisions in Python3 (and further) are based on the desire to keep the interpreter simple.
Python Core Team doesn't have unlimited resources, and even if they did, the cost of complexity grows non-linearly.
py2 -> py3 transition did not go as planned, but that doesn't mean that following Perl route would have had better outcomes.
i think it’s clear this was a huge mistake. none of these changes were critical to pythons current success. python stalled for 10 years if not more because of these non-BC changes. it will fragmented indefinitely. this is a good case study of how one big ego can completely derail a massive project for decades
It won't be fragmented indefinitely, that claim might've carried some weight five years ago, but these days I see very few Python libraries that still support Python 2.7. And, if they do, it strongly implies that they've not been maintained for some time. (See also, a dependency on six) - with the caveat that sometimes you don't need to continually upgrade a library that fills its desired niche perfectly.
Hell, if you're still using 3.6 you're limited in your ability to upgrade dependencies, 3.8 is generally the minimum supported Python version these days.
The last wxPython release that even supported py27 was from close to three years ago, and you would have known if you spent half a minute looking into it before making an absurd claim.
I tried using the library less than a year ago. I must have missed something. This would be great news… if the way you told me didn’t make me feel like shit.
Is that true? I have little data, but I'd imagine most are just working on small/mid size scripts that can be upgraded easily. There's really not many big projects your average joe will be coding in python, from what I've seen.
Even if that's true, it's extremely obvious that Python is the language of choice for some very hot subfields at the moment—things like ML, computer vision, etc.
It may have lost a lot of people who got frustrated by the 2->3 transition, but Python is clearly not in a bad place as a language today.
Python is still fragmented 12 years on. It might eventually get fixed, but it still isn't.
Most actively maintained Python code works with Python 3 now, but there's still a whole lot of even maintained code which uses `#!/usr/bin/env python`.
A lot of devs were busy with the busywork of migration rather than improving their libraries. For a long while you often got stuck between one critical library that has no alternative only supporting 2, with a second critical library for some other thing that also has no alternative only supporting 3. So doing your project in Python wasn't feasible.
The removal of implicit conversion between bytes and strings via ascii did not introduce bugs, it showed were you already had bugs that you maybe did not yet notice.
That's only true if you ever received Unicode input. There are plenty of uses of strings that never do - enums, DNS domain names, URLs, HTTP parsing, email addresses (from any sane provider) etc.
You can't use Unicode characters in HTTP messages - IDNs and IRIs are encoded into ASCII before being sent on the wire (using punycode for IDNs and percent encoding for IRIs).
As for RFC6531, my understanding is that virtually no email provider implements it, because of the same risks that make browsers often show IDNs as their punycode version - Unicode is extremely easy to use to confuse people maliciously or accidentally, since it contains vast amounts of duplicate characters (e.g. Latin a and Cyrillic а), even larger amounts of similar looking characters (e.g. Latin r and Cyrillic г), and even characters that are ambiguous unless you choose a locale! (the CJK problem, where the same Unicode codepoint can represent different characters based on the locale - whose locale to use when communicating between two machines being the implementer's problem).
Also, I'm not saying that Python shouldn't have had proper suport for separating byte arrays from encoded strings. I was only pointing out that there were actual legitimate use cases where a valid Python2 program was broken by the Python 3 Unicode string transition, whereas the GP was claiming the Python 2 had to have been buggy already.
Edit:reading around more, it seems that RFC6531 is getting some traction, and many providers accept sending to/receiving from internationalized emails even if they don't themselves allow you to have a Unicode email (e.g. you can't have айа@gmail.com, but you can correspond with someone having such an email at a different provider). So, email was a bad example in my list. The rest still stand.
No doubt some things broke "needlessly", or that things couldn't have been done better, but I don't see how it could have been avoided since there is no way to distinguish between "I know that this string will always be ASCII" vs. "this string can contain non-ASCII".
For example, what if I want to enter "ουτοπία.δπθ.gr" in an application, via direct user input, a config file, or something else? Or display that to the user from punycode? No one expects users to convert that manually to "xn--kxae4bafwg.xn--pxaix.gr", and no one will understand that when being displayed, so any generic IDN/domain name library will have to deal with non-ASCII character sets.
The same holds true for email addresses: "ουτοπία <a@example.com>" is an email address. Sure, this may get encoded as "=?UTF-8?q?…?=" in the email header (not always, "bare" UTF-8 is becoming more common) but you still want to display that, accept it as user input, etc. People sometimes to forget that the name part of an email address is widely used and that any serious email system will have to deal with it, and non-ASCII input has been common there for a long time.
In specific applications you can often "get away" by ignoring non-ASCII input because you sometimes don't need it. For example I'm working on some domain name code right now which can because everything is guaranteed to be ASCII or "punycode ASCII", so it's all good. But in generic libraries – such as those in Python's stdlib – that's much harder.
Strings are still strings in Python 3; if you do 'foo' == EnumValue then that will work fine in Python 2 and 3. If 'foo' is from an unknown source: yeah, you might get a bytes type in Python 3 and an error, but that's the entire point. Turns out that in practice, it can contain >0xff more often than you'd think.
Certainly today DNS domain names, URLs, and email addresses can – and do – contain >0xff input, and for some of these that was the case in 2008 as well (URLs and email addresses – IDN didn't come until a bit later).
The Python 2 situation was untenable and lead to a great many bugs; "decode errors" were something I regularly encountered in Python programs as a user. In hindsight, the migration effort for existing codebases was understated and things could have been done better with greater attention to compatibility, but the problem it addressed was very real.
DNS-the-protocol still doesn't support non-ASCII input, but DNS-as-people-use-it does. I expanded on that in another comment I just posted, so I won't repeat it here: https://news.ycombinator.com/item?id=34230218
But is also very rare that you ask directly for domain name resolution in the first place; you are usually instead using something directly or indirectly that happens to be remote, and that eventually happens to have a punycode encoded non-ascii hostname or top level domain. But there's no garantee that you (or the libraries, or the libraries that your libraries use...) are only handling the ascii punycode.
I can count on my fingers the number of places I invoked manual DNS lookups inside production code.
I've seen way too many cases (possibly resulting from 2to3 autoconversion) where the code ran without errors, you just couldn't log in because the xsrf token was "b'123456'" instead of "123456".
Even a decade ago, you would have needed to support Unicode if you handled any of those strings. IDN domain names existed as far back as 2003, so unless you could guarantee that everything was in A-label form already, you would need to worry about that (which affects URLs and email addresses as well). URL paths might be Unicode if no one normalized it to percent-encoding first. And HTTP headers--like email headers--could well be non-ASCII despite the standard prohibiting unencoded non-ASCII text because the real world is full of shitty implementations that don't follow standards, and the internet community generally runs on the principle that it's better to force everybody else to try to make sense of the result than tell those people to fix their code.
Indeed, I just had to port some code that had been running happily for years at the south pole to py3 (due to EL8 upgrade and not wanting to install the py2 world). It was something talking to a HV supply over a serial port which of course only spits out bytes, but then needed to parse using string handling. It wasn't that hard to port but it took a few tries to find all the places necessary, requiring debugging over a rather slow ssh connection (that is only up when the satellite is up).
For speakers of languages that cannot be written in ASCII-only, the importance of the improved unicode handling really cannot be overstated, especially for people new to programming.
Heh, I still regularly write workarounds to deal with the Python 3 transition. These days, the main cause of problems is the disagreement of what the python 3 shebang is. A lot of especially Google code "supports" Python 3, but expects `#!/usr/bin/env python` to work. That means messing with environment variables and files so that `python` is available as a wrapper script which just executes `python3`.
I don't understand why there apparently wasn't a clearly stated preference about `python` being Python 2 and `python3` being Python 3 from the Python org.
This really sucks but at least pyenv exists. Basically distros care so little about their users we have to use external tools to use python2.
"Everyone has has a chance to migrate" . Doesn't work that way, just yesterday I spent a few hours trying to get scripts that are really good and would take me forever to migrate that are in python2 to work on an ubuntu based distro and this is with pyenv, custom venv,etc... just when you get python syntax worked out you are faced with deprecated or uselessly renamed imports and modules.
This is the shitty aspect of using foss stuff. I don't pay for it so I have to accept their decission. But in the broader sense, inability to influence things like this even with a ton of other people willing to pay for it is frustrating.
You know what would be great? If the community can vote with money pledges. So it you have 1000 people willing to donate $10/month to support python on sysv init then if volunteers don't want the money we get an external person getting paid 10k/month to support these packages even if the corporate types, developers and "devops"/sysadmin people have no use for it.
The free part of foss is expensive but I think the middle ground is to collectively pay for the support. I try very hard to avoid python2, I am not a developer and I very easily lose at least 1h/month on average to resolve these issues.
Do you all know why 2023 will not be the year of the Linux desktop? Because desktops are a commercial market and consumers/users cannot sway what the Linux desktop experience is. They either accept things as they are or pay for them by spending time reporting bugs, troubleshooting stuff, asking questions on random sites,etc... a lot of this has to do with change and maintenance requiring human time and resources. It's great when there are no issues and volunteers donate their time but it would be great to be able to pay with money also to support features or resolve issues. You can even make sure multiple people pay an equal amount to avoid wealthy people having too much or an influence but it still will not affect regular volunteer work.
I was hoping Ubuntu and Fedora would have such features.
> Do you all know why 2023 will not be the year of the Linux desktop? Because desktops are a commercial market and consumers/users cannot sway what the Linux desktop experience is.
And... you believe that Microsoft and Apple are swayed by users? More than Red Hat and Canonical? I don't buy that for one second.
For sure, market studies, focus groups,etc... look at windows 7 for example, fixed a lot of complaints users had about vista but to your point, their customers that have influence are those that pay the most which are bigcorps with MS and multimedia editors with apple. Do you know why MS is bloated and "slower"? Because of all the technology they support from decades ago and all the interpperability layers in between. A vbscript from 2000 still works to automate stuff on windows 11 even with the advent of powershell for example. They could save money and crapton of security issues by getting rid of the windows script host just as an example but they would make paying enterprise users unhappy and less likely to buy more from MS.
> For sure, market studies, focus groups,etc... look at windows 7 for example, fixed a lot of complaints users had about vista
That's true. Now look at 8, 10, and 11 and tell me with a straight face that they still care.
> Do you know why MS is bloated and "slower"? Because of all the technology they support from decades ago and all the interpperability layers in between.
Then surely the comparison is RHEL, where companies pay for compatibility and stability, not "we <3 bleeding edge" Fedora or "volunteers maintaining a coherent system" Debian.
> That's true. Now look at 8, 10, and 11 and tell me with a straight face that they still care.
Do you see the Metro start screen anywhere on Windows 10 or 11?
If they weren't swayed by user input, why did they throw away all that time, effort, and energy put into the the Metro UI and go back to the start menu paradigm?
Yes they do care, about profit that is. Doing random stuff users want isn't usually profitable though which is why I like the configurability of Linux. But if users who pay up demand something they will listen unless it contradicts with a more profitable business strategy.
> Then surely the comparison is RHEL, where companies pay for compatibility and stability, not "we <3 bleeding edge" Fedora or "volunteers maintaining a coherent system" Debian.
Unfortunately, RHEL and SLED are both targeting enterprise users, very little of that is desktop as well.
You know who else succeeded in the Linux desktop that isn't really considered proper Linux? ChromeOS on the ChromeBook.
If only these new laptop makers like purism or system76 took a similar radical approach where they would indeed listen to paying users more directly better than existing bigcorps, that would give Linux on the desktop a chance at market share gains that are significant. Then I would be comfortable gifting a Linux laptop to a non-technical user who can call a support line when having issues or needing a feature.
You shouldn’t be using the system-provided Python for anything except system-wide tools that can be packaged and marked dependent on the system Python package (and any Python packages which are packaged as system packages - don’t use “pip install” system-wide). The system-provided Python is mostly there to support system packages/tools that happen to be written in Python.
Stand-alone applications should use pyenv/venv because you can’t guarantee which version of Python is the system-wide one (and you don’t necessarily want to tie your app to the distro’s release schedule). Dependencies should always be in a venv as to not interfere with the system-wide ones or other applications.
If the removal of Python from distro repos breaks your things then it’s a wake up call and you’ve been doing it wrong the whole time. Fortunately it’s an easy fix.
> You shouldn’t be using the system-provided Python for anything except system-wide tools
That does not sound like a user friendly programming language.
The version compatibility of Python is a joke and has been after 2.7 to 3 to. Which is the main reason I don't use Python for anything but bash-like scripts.
One major problem is that some version of Python was supplied with most Linux/Unix distributions - this allowed people to get started without thinking about this problem and then end up too deep in the hole by the time they realise.
Other languages or OSes that don't have their interpreter bundled forced you to think about the problem from the start and build a way to ship a stable version of the runtime with your application.
Python, as a server-side programming language is not too bad. You can standardise on a version and bundle that interpreter version with your application itself in whatever artefact you produce (whether a Docker container, tarball with Python binaries + source or entire VM image with the right Python version installed).
Desktop distribution is a pain, but that's not limited to Python; any language that can't compile to a native binary will have this problem - even Java, despite having much better backwards compatibility, still isn't prefect and sometimes requires an older JRE to be installed on your target machine.
As a matter of practicality I agree but for one not everyone knows this and for another distros like debian don't support what you are saying. We are resorting to pyenv because distros refuse to maintain different application specific python environments. You can pin python versions and deps per package if needed.
Your distro exists to provide other people's software. The distro doesn't develop the software themselves. The Python 2 organization has decided to end Python 2. It is completely and decidedly unsupported by anyone as of over two years ago.
If you want to complain about Python 2 being unsupported by the Python foundation, you're free to. But no distro can in good conscience keep distributing a thoroughly unsupported piece of software that's probably full of security bugs.
Regarding the "year of the Linux desktop" thing: Apple is much more aggressive than any Linux distro when it comes to dropping deprecated stuff and forcing their users to move on. Apple hasn't shipped Python 2 for a long time, they ship Python 3.
I liked how Canonical did it: the command `python` would not execute, you have to use `python3` explicitly. This means that your old Python 2 code would not get executed if it were triggered by some shell script via the `python` command, you'd have to fix the shell script and become aware that you still need to migrate the Python script.
This is also how Debian did it, too. Actually it's probably propagated from Debian Unstable to Ubuntu.
You need to install either "python-is-python2" or "python-is-python3" packages to get proper system-wide symlinks. Also, it was strictly verboten to give either of two packages as dependencies to workaround migration procedures.
Debian Testing doesn't have Python2 at least for a year now.
I've upgraded debian thru the ages and /usr/bin/python defaulted to python2. Didn't need to install anything for that to happen, maybe if you installed debian from scratch
Another mistake in Py3 migration was keeping the interpreter name as python so when you had it in path you had a choice of new stuff breaking or old stuff breaking
> I've upgraded debian thru the ages and /usr/bin/python defaulted to python2
It was, in the past.
When python2.7 deprecated and removed in Debian testing, python ceased to exist as a command. You need to install these packages (python-is-python{2,3}) to get the python command and point to the correct interpreter you like.
This behavior is consistent in two Debian testing systems, installed 6 and 9 years ago, respectively.
> Another mistake in Py3 migration was keeping the interpreter name as python...
I installed python-as-python3 myself, after python2 disappeared from system to fix an oddball python3 script which bundled as a part of a bigger software package which is not in official repositories.
Maybe they have fixed it, but I didn't bother to check till now. I'm not even sure which program it was.
So, it was an addition to a clean system and didn't inadvertently directed python2 code to python3 interpreter.
I like Python 3. It eliminated a whole category of encoding/decoding errors. Even Python 2 codebases benefited from it, as libraries were updated to handle Unicode better in an effort to achieve compatibility with Python 3. I didn't experience much pain migrating codebases to it, but I'm just speaking for myself here.
Congratulations to Debian on upgrading to Python 3!
Yeah, coming from C#, Python 2's unicode support was so bad I almost abandoned it immediately as a Chinese speaker (and to make it worse, I use Windows). You literally can't use IDLE for learning/testing properly half of time due to encoding issues.
And what surprised me most is that every time I mentioned this, there would be lots people telling me how this is a superior design because you can operate string like bytes. I mean, it of course has its upside, but I don't think it's worth it if you care even slightly beyond ASCII.
We had crazy amounts of code handling unicode support and conversion from our ecommerce site to our ERP system (running on Windows using some Windows code page thing). With Python 3 all that went away, you can now just seamlessly parse text from one system to another.
For me, the unicode handling alone was worth the time spend migrating from Python 2. That was a decade ago, to finding that the "python" command still launches a Python 2.7 interpreter in 2023 is just beyond belief. Personally I feel like they should have yanked Python 2 in Jessie (Debian 8) in 2015, more realistically in Stretch in 2017.
The biggest danger in Python 2's unicode handling is that incorrect things somewhat worked (until you got a non-ascii character at which point it exploded or produced incorrect behaviour).
I'm sure you could do things well in Python 2 with proper combinations of encode/decode, but it wasn't obvious where you even needed those because with ascii text, things "just worked" transparently. With Python 3 it's very obvious where you need encoding/decoding because bytes != str.
You could do things correctly in Python 2, but as soon as you used any third-party library in your project, chances were it is going to explode underneath you anyway.
In the early 2000s, I maintained an py2, wxPython app with the users having the system encoding win-1250; the effort to patch this was unbelievable. The migration to python3-style handling forced everyone to think about these issues, not just few people for which things were crashing. Even just popularizing the issue was great, until then, many maintainers of third-party libraries didn't even understand what is the problem that you want to "needlessly complicated" fix in their libs.
> Personally I feel like they should have yanked Python 2 in Jessie (Debian 8) in 2015, more realistically in Stretch in 2017.
For example GnuRadio started supporting Python 3 with GnuRadio 3.8 released in 2019, and then you had to port all your programs using it to this version. So no, in 2017, the ecosystem was not ready.
FWIW, in Linux, this problem does not exist. Everything is UTF-8 and Python 2 would work just fine (and always did).
In order to support Windows better, Python 3 introduced support for UCS-4 (or worse, UTF-16) strings (depending on a compilation setting when Python was compiled) and they had to introduce extra string types to distinguish readable strings from binary strings ("bytes").
These extra types made Python 3 a lot harder to teach (I teach 30 person classes every year).
So it's not all roses now.
In the end, I got used to it, BUT I just gave up asking encode()/decode() questions at the exams. Very few people understand it, or care enough (and I understand why--it's a ridiculous thing to have). You only need it if your OS somehow slept through the introduction of UTF-8, which is backward compatible with ASCII, resilient even if there are transfer errors and can encode all unicode characters.
Encoding problem used to be really common in UNIX (and before that, in mainframes), but with the introduction of UTF-8, all encoding problems I had vanished and never appeared again.
Even Windows 10 has an UTF-8 mode now and the Windows API functions that end in "A" can be made to use UTF-8.
Now, in a sense, Python 3 has this entire complication for no reason.
That said, Python 3 is ok to use now--and, conceptually, distinguishing byte strings from unicode strings is better (for example so that you don't accidentially print the former to the terminal). It just uses up brain cycles that you could be using for solving your actual problems.
> I just gave up asking encode()/decode() questions at the exams. Very few people understand it, or care enough (and I understand why--it's a ridiculous thing to have).
I get it from the "pass the exam" perspective, since that's one more thing to worry about.
But from my experience in teaching others, doing the conversion between bytes and string implicitly (à la Python 2's way) hinders actual understanding of this very important concept, and it's quite harmful in further study.
Bytes should be considered as a separate, more low-evel thing, away from int/float/strings; at the very least, it should be considered as bits/hex numbers. If you want strings, you explicitly encode/decode them in a way, even if everything is UTF-8.
On top of that, "byte string" is just a confusing concept. It might works for English speaker (by "it's a ridiculous thing to have" I assume you mean that, "'english'.encode() is just b'english', why bother?"), not at all for Chinese speakers, even in UTF-8. There is no b'中文' -- only b'\xe4\xb8\xad\xe6\x96\x87' which has zero meanings in their own.
And even from an easy-to-use perspective: most people don't even work on bytes often nowadays. A more abstract "string" type is all they need, without worrying about how it works under the hood (and if they do, they need to understand how encode/decode works properly anyway).
I'm working with Germans, Japanese and Polish that use a lot of special characters, including Kanji, umlauts, extra quote characters etc. I need the non-ASCII parts and had no problem with them in Python 2 on Linux (now, C++ libraries that reinvented their own string classes: many problems; C libraries: no problems).
The point is when bytes is str, everything works just fine in Python 2 Linux with UTF-8 locale (which are used in all modern Linux distributions). No need to have a distinction between bytes and str.
That how the rest of the OS works, too. Even a lot of Gtk, Glib and so on (for example the GNOME desktop environment) assume that you are in an UTF-8 locale for file names, for example.
> A more abstract "string" type is all they need, without worrying about how it works under the hood (and if they do, they need to understand how encode/decode works properly anyway).
Ehh, we had students write drivers for measurement apparatuses and they all used Python 2 str (without being prompted to do so). No encode or decode anywhere. Of the students, almost no one who tried Python 3 for that stayed with it (instead they were using Python 2). There was just no upside for this use case.
I agree that, long term, having a distinction str vs bytes makes sense. But then you ARE juggling things that the OS doesn't need--it's basically busywork in Linux.
I'm not trying to minimize your experience--but I don't think it would happen if you tried python2 on Linux today. Not sure it was worth it breaking compat for that.
Honestly, I just found Python 3 to introduce a whole bunch of complexity when working with text. Why can't it just be a byte buffer? Why must you complicate interacting with the OS so much? Text is just a buffer of mostly UTF-8 encoded bytes, why make it needlessly hard?
I think two aspects have played a significant part in the migration pain:
1. The amount of breaking changes. If it was possible to do a little at a time, over a series of releases, it's a bit more manageable.
2. Being an interpreted language makes it difficult/painful to break compatibility over a series of releases, as all applications and libraries have to be kept source compatible. Source compatibility also makes it a pain for library maintainers, as you can't just cross-compile the library to a older ABI.
Maybe support for consuming libraries based on an intermediate, compiled representation, would make it easier for the library eco-system to follow breaking changes over a series of releases, and in turn make it more feasible for applications to handle a small amount of breaking changes over a longer release timeline.
They also changed the C interface at the same time -- and many Python libraries had non-trivial amounts of C code in them because Python was and is so slow.
It was also for a time impossible (or close to impossible) to run the same Python code in 2.x and 3.x. It helped when later 2.x versions acquired more 'from __future__ import'.
They really aimed for making it maximally hard to support v2 and v3 at the same time which created a nasty coordination problem: why upgrade if the libraries aren't ready and why upgrade the libraries if there are no users.
Easy: Introduce breaking new version, but freeze and continue to support old versions (e.g. a python3 install should include the ability to fallback to python2). A real world example of this philosophy are the D3D APIs. Each new major D3D version had a completely new API, but old D3D versions are supported forever.
Python source files then somehow need to announce what python runtime version they expect.
Having to support old versions forever also should make the maintainers think twice before breaking compatibility again.
Feature flags or at worse, transpile. Perl did the first (in form of just writing what version features you want to use in header) and they also went thru "UTF8ication" of the language.
Even if you don't get all old code to work, having say 80% of it working means to start using Py3 you'd need to only initally rewrite 20% of the codebase, run tests and off you go, then migrate rest slowly over time.
Angular2 is a model for breaking changes IMO. The naming convention is a travesty, but the upgrade process was really elegant. The biggest bits:
- Added some stuff to AngularJS to make it more Angular2 compatible so that you could get closer to the target without taking in the breaking changes
- A thing called ngUpgrade let you dual-boot AngularJS and Angular2 together, so that you could transition parts of your program over time without needing to redo the whole thing
- Well documented migration guides for each step, outlining what you need to do in order to move from X version to Y version.
I've never used any version of Angular, but from what I've seen people complain about it was also a huge breaking change - so much of peoples' apps had to be rewritten they ended up just switching to React or Vue or something.
Fundamentally yes it was a huge shift. Practically, no, you dont need to rewrite a bunch _if_ you followed the upgrade path laid out as I described in my original post. They backported a lot of stuff so that you could do the rewrite incrementally.
You mean the way rakudo took over the entire perl ecosystem when it finally arrived?
I suspect a lot of the controversial decusions made doing the 2to3 migration by the python foundation was deliberately designed to avoid a situation similar to what happened to perl when the time came to cleanup some of the old cruft from the language.
You underestimate how big of a change Python 3 actually was. Changing how strings and bytes are handled has a huge blast radius. It broke almost every library or application I worked on. Everything else was peanuts to that semantic change.
It was a necessary change too. Supporting Unicode strings or mixed-encoded strings was absolutely awful in Python 2 and almost not worth doing. Python 3 made that work properly.
Arguably, the only real mistake was using UTF-16 instead of UTF-8 for strings. But back then, nobody knew what would take off and Windows natively uses UTF-16, so...
Python 3 had a HUGE impact on existing code, yes. Everything broke. But Python 3 didn't actually change much for the people writing new Python code, it just broke the old code.
Raku is a completely different language from Perl. Writing Raku instead of Perl 5 is like writing Rust instead of C++.
Well, trying to sell a project to the higher-ups is a lot easier when you can say
> Hey our interpreter for our backend code will be eventually deprecated. It is currently version 2 and we need to move to version 3"
Then the execs say "we are on 2?? and there is a 3??! what are we waiting for?!" A whole lot easier than trying to say:
> Hey, there is a new interpreter out there called COBRA. We need to rewrite our whole codebase to use it, there isn't a tremendous amount of benefit, but its the new thing from the same people and eventually everyone will use it. Trust me.
To which the execs ask you to stop fooling around and get back to work.
It was a pretty easy sell for us because Django ended Python 2 support in 1.11 So in reality the conversation actually went more like "hey you know the web framework we use for all of our services. yeah, that will be stuck at the same version for forever with no security releases and unable to use any new integrations unless you devote 2 engineers for 3 months and take the medicine now instead of digger a deeper hole"
But it was a lot easier for the CEO to digest `2 -> 3` as a clear path to the future as opposed to switching to a new language.
Much harder.for us as our code base predates Django so we (years before I was hired) have our own framework, our own orm, … so we’d basically have to port everything all at once, with no clue if we other half is correct until you’ve ported both.
Ha ha, well that was almost the CEOs verbatim reaction at a multi-billion dollar company. It helps if you are a financial services company and also add on "and if we dont do it, our web frameworks will reach EOL and have no more security updates"
Oh, I've heard that a lot from execs. What they also say afterwards is - "we expect you to do it without slowing the pace of adding new features and fixing bugs!!"
Meanwhile we are still living our Python 2 like moments in .NET and JVM ecosystems, with the huge amount of projects stuck in .NET Framework and Java 8, with no end in sight of them ever moving forward.
It's not as big of a deal, though. Code written for java8 is still source backwards compatible on newer platforms. And in .net/jvm you're not depending on some interpreter on the OS level and don't have code bundled with the OS, as has been the big issue with lots of small python scripts included in various distros.
On .NET Framework you surely depend on OS integration, that is one of the major design changes in .NET Core.
Starting with Java 9, finally the long deprecated APIs started to be removed.
In any case, the issue is that we are forced to write C# 7 and Java 8 code.
Some well known examples, Sitecore, SharePoint WebParts, SQL Server SP extensions, Dynamics, Office AddIns, VS plugins, Solr extensions (many places keep using Solr on Java 8 even if more recent versions exist), Android (unless one goes Kotlin or targets version 12), Forms and WPF 3rd party components (the versions on .NET Core are incompatible due to new designer)...
Yeah, I hoped MS would be more supportive of .NET Standard 2.0 during transition period.
I am stuck on .NET48 at least for another two years (because of webforms frontend) and the last version of EF Core that supports .NET Standard 2.0 is 3.1. They dropped .NET Standard 2.0 like a rock... for what? Few default interface implementations.
Or nullability attributes that are missing in .NET Fx, they could also be defined externally, like R# does. Nope, sucks to be you.
My previous, and only, software developement job involved migrating a monolithic .NET Framework web app to .NET 5.0
Compared to Python 2->3, the breaking changes were more obvious as they involved entire strucutres and paradigms being deprecated. Python 2->3 felt a lot more subtle despite being just as damaging.
It was a bit of a shock when I upgraded my Debian systems to Bullseye (11) from Buster (10) a year or so ago. I discovered that the wiki I've been using for years, Moin Moin, was removed because it didn't support Python 3. I've a lot in the wiki and was in no position to migrate, so I made a wiki only VM and keep it at "oldstable". Since none of it is internet-facing, I can keep it this was forever, even once all support ends. I'd rather not have to have another VM though.
I had a VM with MediaWiki way back when and I was able to convert it to markdown and a private GitHub repository pretty easily with pandoc. I'm not familiar with Moin Moin but maybe there's a conversion tool you can use to do the same thing?
MoinMoin2 exists and works on Python 3, but sadly there's still no stable release and it requires slightly older Python packages right now. We've also had strange problems with index (Whoosh library) performance on a large wiki that we had to hack around, but it might be a good migration path for you.
And if you don't want to use moin2, it can convert the custom markup to Markdown, which you can then use elsewhere.
It looks like the MoinMoin people decided to rewrite a lot of stuff for Moin 2 (which does support Python 3), but that hasn't been finished yet and has been in the works since about 2010.
It craps me off that I still need to look up which direction to encode/decode strings and other things like bytes and ascii even after programming Python 3 for more than 10 years.
It's a huge usability fail on the part of Python that it's not obvious and easily memorable how to do this.
You encode strings to bytes, you decode bytes to strings. A string is an abstract representation of text, which can be encoded into whatever supported format you'd like; bytes are a concrete representation of binary data, such as text encoded in UTF8, which, if it represents text, can be decoded to obtain a string.
If you think about it, that's the only way that makes sense; if you wanted to go from, say, UTF8 to UTF16, you'd need to have an intermediate step that was abstract and formatless. Similarly, when you're operating on text, you don't want it to have a particular format; the format should only exist at the edges, where you take text in and output it. The format is a way to communicate with other programs, not components within your program.
My advice would be, if that doesn't stick in your head (totally legitimate), write down a code example and stick it in a personal wiki or something. I find it really helps me to understand something once and more or less copy paste it forever.
A simplistic summary is that you always decode to unicode and always encode to bytes.
Another way to think about it is that Unicode is an idealized character model that is reified into a byte encoding, usually UTF-8, for practical usage.
It seems to me that this is something a lot of developers don't think too much about until they run into some obscure encoding/decoding issue, but it's kind of a fundamental aspect of programming that's worth spending some time grokking.
Would you care to elaborate on what it is and why it's better for the benefit of those of us who don't know Go?
It's tricky to look up a different solution you aren't familiar since the terminology is different. What I'm seeing currently doesn't really look that different to Python, but with different names for the types and such.
Taking a stab at it; This looks simple, and runes are great (similar to Python):
But something that bothers me is that converting an array of bytes to a string looks to implicitly be ASCII, or possibly UTF8. This document doesn't say.
This document acknowledges encodings and clarifies that literals are UTF8, but makes no mention of byte arrays:
I've poked through as many SO questions as I care to and am not clear on how to use different encodings. Looks to me like this is a difference in philosophy; Python would say, explicit is better then implicit, and I'm inclined to agree.
If I'm not understanding properly I'm happy to be corrected.
I don't have skin in the game, but at least the .Net way is quite explicit, which I like:
byte[] unicodeBytes = Encoding.UTF8.GetBytes(inputString);
// Perform the conversion from one encoding to the other.
byte[] asciiBytes = Encoding.Convert(Encoding.UTF8, Encoding.ASCII, unicodeBytes);
string outputString = Encoding.ASCII.GetString(asciiBytes);
utf8_bytes: bytes = bytes(input_string, encoding="utf-8")
# Perform the conversion from one encoding to the other.
unicode_string: str = str(utf8_bytes, encoding="utf-8")
ascii_bytes: bytes = bytes(unicode_string, encoding="ascii")
# The conversion could also be written as:
ascii_bytes: bytes = utf8_bytes.decode("utf-8").encode("ascii")
output_string: str = str(ascii_bytes, encoding="ascii")
The biggest difference is that the conversion step requires you to explicitly decode the bytes to a unicode string and then encode the unicode string back to bytes rather than providing a convert() method that does this internally.
Perhaps a convenience method would be nice, something like this, but it somewhat obscures the intermediate decode-to-unicode step:
I can't use Linux as a daily driver because of the Python 2/3 breaking change. My personal wiki (WikidPad) just doesn't work properly under linux due to the changes that happened in Python and then wxWindows. There are ports that barely work, but all the dialog boxes to adjust anything just error out.
It's really sad that the Python folks committed suicide this way, all over a unicode fetish.
Having primarily used Python 3 since 2016 with some very specific (rare) exceptions and having found the switch remarkably pain free it's difficult to empathize with those still going on about this like it was massive, and yet there are major banks still struggling with the transition in key areas so there must be something.
Is there a risk it becomes like one of those situations where certain people use a topic simply to make conversation? Or the modern exaggeration to avoid seeming boring? Or a bunch of "oh, it's gonna cost ya" contractors plying the age old lines of car mechanics negotiating basically anything. Someone below referred to carnage. I wonder how far off we are to someone claiming it was actual violence.
I worked on a code base which had a large dependency not port to Python 3 until 2018, and at the same time they made a huge ton of changes to the APIs, most of which were not documented well. So there was both a big refactoring to handle binary data transfer between it and the application, and also a ton of other unrelated things just to make it work. It's things like that that made it difficult to port for most people.
The lack of type checking and general runtime failures made the transition particularly difficult if you didn't have a bazillion unit tests covering all your large codebases.
I was lucky it didn't affect me much, but I could sympathize if somebody had hundreds of thousands of lines of python2 legacy code that needs to be migrated..
I really feel like the community was a difficult pain in the ass during the 2->3 migration. There were breaking changes but I’m sure the Python maintainers didn’t expect the community to react so badly.
Are there some valid reasons that made 2->3 migration insanely hard for some projects? I remember seeing blog articles whining about print vs print(), but surely there are some more important stuff.
> Are there some valid reasons that made 2->3 migration insanely hard for some projects?
Print was a non-issue once you could do "from __future__ import print_function". It was the unicode migration.
All string handling from I/O required at least checking.
The dependency issue was the most crippling, though: because you can't load Python 2 code from Python 3, before your project can begin migrating you have to wait for all your dependencies to update as well. This may involve them changing the types that get passed over the API from 'str' to 'bytes' or vice versa, which is potentially a breaking change.
Migrating where there’s a 1-1 mapping of old API to new API is easy, such as the print to print() change. The hard part is when there’s a 1-N mapping, where a single name in the old API becomes multiple names in the new API. These are the most useful changes to make, because they can split apart unrelated concepts that had been erroneously represented by the same object, but require a human to determine which use case applies.
For python2, the str class could either contain a sequence of characters, or could contain a sequence of bytes. These are often conflated when using ASCII encoded text, but are very different for any other encoding. In python3, these are represented by the str class for a sequence of characters, and by the bytes class for a sequence of bytes. It’s so much nicer to work with, but required manually separating out the earlier use case.
Right, essentially the Python developers decided to define a sizable portion of their developer base as "bad people" due to their particular use of the string API and then spent years going to war with them.
The war continues to this day apparently with this Debian news. Somehow you can use fortran77 compilers without people getting angry but you can't use Python2
It’s not that simple: as the person you’re responding to explained, this was leaky because in simple cases you could pretend those were equivalent but then things would break around interface points when someone introduced things like filenames with Unicode.
I had clean Python 2 codebases which had been Unicode-safe for years and had good test coverage, so porting to Python 3 was trivial and mostly automated. The projects which were hard were the sloppy ones which conflated bytes and Unicode, and the poor development culture which lead to that also tended to mean they had limited testing, orphaned dependencies, monkey patching, etc. which made the migration hard. I also found a fair number of cases where that process fixed other bugs which had been ignored for years.
Guido has done some pycon keynotes in the recent past and said himself he takes some of the blame for the transition. He says he greatly underestimated the impact of breaking changes and the inertia to get the community to change code. In particular not shipping tools to help automate the migration, or even thinking through a migration path with targeted backwards compatibility/back ports from the start (these eventually came quite a few releases and years into the transition). I don't think it's fair to say the community was entirely at fault for the length of time.
They are a thing, the ast module was created to help with the transition and tooling to automate some of the conversion: https://docs.python.org/3/library/ast.html IIRC it wasn't there at the start though.
A codemod is an automated refactor. Codemods are typically done by writing scripts that manipulate the AST of code in files or by writing a regex to find and replace.
The name originates from an internal script at Facebook called codemod. It had a public open source fork if you are curious.
I think both of you are looking at some of the same problems from different angles, and there are some interesting ways in which the community shifted over the same time period. I suspect some of Guido’s instincts were guided by the earlier community which was more early-adopters, and simply younger. The 1 to 2 migration was easier because the median developer was far more motivated to switch - you weren’t using it if you weren’t a fan.
Python’s rise brought tensions there in multiple ways - business users started wanting longer-term support and tools for migrating large code bases, but relatively few would pay for them, and some maintainers had either drifted away or retired, or simply had less time available due to things like starting families or getting promoted. The 2 to 3 migration really highlighted the gap between commercial open source and hobbyist projects, and I’m sure now anyone would fundraise for support having seen how that worked out.
The other thing we’ve been seeing is a switch to faster releases industry wide and I think that showed some serious tech debt. Projects which have CI & decent tests had a much easier time shipping updates or having two release series for an while, and the 2 to 3 migration really highlighted that. Some of the griping was really showing that older projects often had the frictional disincentive of that tool. That’s still relevant because of the way security issues are forcing more frequent updates now.
The python maintainers went out of their way to break things - and were smug about it. That’s what it felt like to the community.
A big part of 3 was Unicode strings. In 2 you could mark a string as Unicode with a u””. This would have been a great way to let libraries and code work with both 2 and 3. They banned this, but you had to still use b””. This was the attitude and just one example.
The list of stuff that got hard / slow was long. Migration path was unnecessarily difficult as was compatible code. Somewhere around 3.4 it was like a light switch flipped and they started being more reasonable. ASCII handling improved (yes, not all internet protocol stuff is Unicode), they began making it easier to target 2/3 etc. but it was horrible to start. I’ve thankfully forgotten some of the details :)
> Migration path was unnecessarily difficult as was compatible code.
I feel this take is outright wrong. Python provided it's 2to3 tool[1] which took care of the bulk of the work required to port Python2 code to Python3. The only code that was not supported was eggregious errors in the code that worked by coincidence, such as handling bytes as strings and vice-versa. Porting old code to Python3 is a breeze that consisted of running 2to3, run tests with python3, and if anything breaked then just touch up the code to get it to work. I know it because I personally ported half a dozen projects throughout the years.
I'll go as far as to claim that most problems porting old python code to 3 were either upstream dependencies dragging their feet or internal human/organizational issues.
This is totally false - code produced by 2to3 no longer ran on 2. So you ended up w a chicken and egg problem. Things like u would allow folks to keep compatibility with 2 while working in 3.
Your comment is a perfect illustration of the issues. Lots of user blaming. No actual solution. As I said, it did start to get massively better at some point. Instead of condescending lectures on org issues they for example began allowing u”” in 3, which did not mess up Unicode handling in 3
> This is totally false - code produced by 2to3 no longer ran on 2.
I don't see where GP made a claim that it would. Why would one expect code written for a newer version, using features that don't exist in an older version, to run under the older version?
This was the issue. Why should a new version make it hard to run old code for no particular reason except to break things.
Folks using u”” were already being careful with Unicode . Supposedly Unicode was important enough for 3 to default to it.
The cost benefit of then destroying code using u”” made no sense to me. It was unnecessary. This was followed by the oh just upgrade thing. I’d love to see the “quick” upgrade guidos employer did. My guess is a total lie that it was easy and it probably took years and VC type money.
It’s not about using new features, it’s about not unnecessarily breaking old or allowing for cross compatibility.
> using features that don't exist in an older version
Except they broke the old way for existing features, required you to use the new way and in most cases you could manually write code so it would still work with both. Meanwhile the only tooling provided by the python 3 crowd screwed over anyone with an existing customer base stuck on python 2, which at least early on should have been a foreseeable problem. Not to forget that anyone stuck on python 2 was literally Satan and projects where outright "shamed" to drop python 2 support.
Given dependency chains the business case for these “quick” switches was often poor as well. Yes, Dropbox had $$ and guido - but that wasn’t everyone and most code was 2 at the time.
Many folks would have been ok updating to 3 compatible approaches if it didn’t blow up their 2 story. That’s what ultimately happened in lots of cases when it became more doable. The rip and replace everything at once was a weird goal
If you published libraries on pypi you were in exactly that nightmare scenario. How do you support users on both versions of python (because let's be real, it took _ten years_ to change so you had to support both major versions) without two totally separate codebases and all the massive headaches that entails. There were some partial solutions like the library six and all kinds of related tricks to write python that was compatible with both 2 and 3 at the same time.
> This is totally false - code produced by 2to3 no longer ran on 2.
That's what "breaking changes" and "backwards incompatible" means.
What exactly is hard to grasp?
> So you ended up w a chicken and egg problem.
You really don't. Your code and your upstream dependencies need to be ported to python3. Once your dependencies are updated, all that's missing is you doing your job.
Python2 has been on the path to deprecation for how long? A decade?
You're fabricating problems where there are none.
> Your comment is a perfect illustration of the issues. Lots of user blaming.
There is no issue. When I had to port projects to Python3, I just ported them. No drama, no hangup. You're pretending there were problems where there were none, and you're throwing a tantrum when this fact is pointed out to you.
There is really no justification for this. It's high time people like you stop making up excuses and start to own up their misconceptions, misjudgements, and mistakes.
Some projects were rewritten from scratch in entirely different tech stacks in less time the likes of you complained they could not update their projects from python2 to python3.
“It's high time people like you stop making up excuses and start to own up their misconceptions, misjudgements, and mistakes.”
What is it about python 3 that brings out this lecture stuff.
I would really like to see the inside of this “easy” change at even the bigger most well resourced places. Google / Facebook / Dropbox etc.
At small places where a program is working and programmer is gone they are not investing in ports.
Anyways - they did eventually get a clue. It’s much easier NOW to handle this, but it wasn’t at the start. Yes, they did put u”” back in. No it did not destroy the world.
> Yes, they did put u”” back in. No it did not destroy the world.
This saga isn't completely over, over here: We still have code running python2 that we don't have the room to upgrade, and libraries that have to work with both python2 and python3. Meanwhile, people are pushing "code consistency" to the point where people outside our team are taking those libraries and running the "black" formatter on them - but "black" doesn't understand python2, so it removes the "u" prefix and breaks the library.
I like black and python 3 - but that’s pretty funny! In fairness black is opinionated on purpose - even if it understood u”” it’d remove it if it’s targeting 3 probably because it tries to reformat to one style. Amazing how long the ripples last though
>What is it about python 3 that brings out this lecture stuff.
I think some people convinced themselves that using "" without the u"" was disenfranchising to people who don't want to use ASCII to represent their language. This transformed a practical argument over syntax into a moral crusade.
You should not be using those dependencies today. Looks like they have been unmaintained for a decade and can contain a ton of vulnerabilities.
Though I find it hard to believe that there are some useful dependencies that have not been ported and don't have better alternatives. Typically a "never ported" dependency is never ported because it is deprecated in favor of something better.
Does no one work in a business? Does no one work with science instruments purchased more than 5 years ago? Businesses have lots of useful dependencies that have not ported. If you are running a payroll system and it’s working, there is a real aversion to going to a new code base, and the cost benefit is not there for many businesses. You do realize python 2 is downloaded 3 million times per month now? Yes that’s down - but it’s not nothing - probably 300 - 400 thousand plus users easily
> Ok, so what happens when several of your dependencies never ported?
If you're consuming dependencies which were never updated in the past decade then you have more worrying problems to deal with than porting your code to python3.
So why did for example Ruby not have such a painful transition from 1.8 to 1.9/2.0? My theory is that the Ruby people make the migration in many steps, it started even back in at least Ruby 1.8.6 and they did it with a lot of care on how to make the transition as painless as possible while the Python team just broke everything and bet on that tools like 2to3 would just magically solve it.
Right. I think a key miss was underestimating value of code that would work w both 2 and 3. This avoids need to upgrade all at once.
Relax things a tiny bit in 3, back port to 2 via futures, get a library like six going. It could have been a ton easier.
In fairness by 3.5 maybe they’d realized this, but a lot of lecturing was done before. Initially recommendation was to upgrade all dependencies and programs to 3 and not try and make dependencies work w both
I think rubys TDD culture at the time also probably helped. My (purely unsupported) opinion is that the test suites were likely more fleshed out on ruby so its less of a risk to switch. That said, I still saw lots of 1.8.6 installs in the wild for quite some time after 2.0 hit.
I think this was the right decision for the long run though. What we have now is the right situation where the strings we work with in the language are 'magic' and we don't need to think about representation or encoding or whatever. And then at the periphery we encode/decode based on what the outside world is working on. It has become clear that this is the 'correct' approach for programming languages and is almost universally the approach in new languages. E.g. Rust does something similar (they don't hide that the internal encoding is UTF-8, but it's consistent).
I'm glad that this is what we have for Python now, despite the pain it caused. We'll be using Python for another few decades probably, so we'll reap the benefits.
> The python maintainers went out of their way to break things - and were smug about it. That’s what it felt like to the community.
The other side of that coin is that the python maintainers went out of their way to push back the Python 2 EOL. They kept on pushing the date back and back again, as per the sunset page:
"We did not want to hurt the people using Python 2. So, in 2008, we announced that we would sunset Python 2 in 2015, and asked people to upgrade before then. Some did, but many did not. So, in 2014, we extended that sunset till 2020."
The drawn out demise of Python 2 was, frankly, painful.
I have no time for whinging snowflakes complaining 12 years (2020-2008) was not enough time to migrate their code to Python 3. Hell, even the original 7 years (2015-2008) should have been long enough for 99.999999% of the community.
If they hadn’t extended there was talk of actually forking python. Basically people needed a 2.x compatible python, and most features being put into 3 could have been backported. So you’d maybe keep u””, keep print, add futures stuff for folks targeting both and then pull in 3 stuff to reduce pressure to move. They were smart not to drop 2 cold.
> I have no time for whinging snowflakes complaining 12 years (2020-2008) was not enough time to migrate their code to Python 3. Hell, even the original 7 years (2015-2008) should have been long enough for 99.999999% of the community.
I'm old enough to have read this when it came out...and it changed my view on backwards compatability (from Joel Spolsky of Trello, FogBugz, and StackOverflow fame)
If a company has production Python2 application / service (with hundreds of thousands of LOC), what business value does it bring to migrate it to Python3?
At that point, if you've got to make severe changes, folks might decide to use a language that doesn't impose breaking changes (and business cost) on them. YMMV.
I feel like a lot of the people who had this reaction were reacting to being told (correctly) that how they handled strings was broken. A big reason that projects had difficulty upgrading, especially difficulty using the automatic 2->3 upgrade tooling, was because of broken handling of unicode and broken handling of binary strings. I remember reading a quick assessment of a large open-source project that there were thousands of calls to a single string-handling function that would each need to be inspected by hand to determine whether it was correct usage that would be correctly translated by the 2->3 converter or buggy usage that needed to be fixed before running the 2->3 converter, with some people guessing a 50/50 split. The effort was going to be gargantuan, and though the ultimate cause and responsibility was poor design choices in Python 2, the immediate cause was that the code was wrong, and some people didn't appreciate hearing that.
No one really blamed programmers for getting strings wrong in Python 2. It was unreasonably hard and extremely rare for people to get it right, so there was not really any shame in Python 2 programs and libraries being pervasively broken in that way. But some people still felt it was smug and condescending to make reference to this elephant in the room. This put the Python maintainers in a bad position, because they were trying to explain the benefits of upgrading and also the costs and difficulties, both of which were intimately tied to the brokenness of current codebases.
As a user and spectator to the whole drama, therefore, I felt that some maintainers got overly defensive about issues that virtually every codebase suffered from. And some of them took a "so what" attitude towards bugs involving non-Western languages, saying that if they personally didn't care whether their code handled Chinese or Bengali filenames then it wasn't a bug to do random things with them, which sounded just as bad ten years ago as it does now.
In python2 you can easily mix encoded and unencoded strings. This doesn't actually work, but at worst it'll produce mangled strings. You used to see this sort of thing online quite a lot with webpages having all sorts of weird characters and question marks showing up.
Python3 enforces the difference between encoded and unencoded strings. It forces you to deal correctly with unicode. If your code base was already handling unicode correctly it wasn't much hassle to migrate. If your code base was making a mess of unicode handling --like many at the time were-- you'd run into that headfirst.
There are many codebases in the world that never need to deal with Unicode. Therefore, new language features which deal with Unicode are unwelcome when they replace the existing string API (which was often used to handle raw bytes)
The biggest problem is fundamental to Python and a lot of other scripting languages: no static typing. Running Python 2 code direct on Python 3? All looks good, until _somewhere_ you use a method of str that isn't on bytes, or vice versa, or something that was a list before is now an iterable and it blows up in your face... but _only_ if code execution reaches that line. So you have to test _everything_. And if you're a library/module, that means testing with _every user_.
Secondly, it seems to me the Python team didn't want to make it possible to write code that works correctly on both Python 2 and 3 simultaneously. They wanted you to write code that only works on Python 3 and tell every single Python 2 user to get lost. Essentially, they wanted to co-opt you as their agent of change; burn your own reputation by abandoning your userbase (or worse, start proselytising Py3 at them) or double your own workload to maintain two codebases. Who wants that?
It's fundamentally an attitude problem. A desire for there to be only one right way to do things, a change of mind of what the right way is, and kicking the chair out from anyone doing it the old right way; the only compromise being to kicking the chair very very slowly. In Java, you can still run code that uses Vector and Hashtable from 1996, they're not even deprecated. They're used by _very_ little code since generic List and Map APIs superceded them 2004, but they're still there.
The early Python 3 releases were abysmal, much slower than Python 2 even without considering the bugs. That alone was bad. But there was also real world problems, like how some syscall wrappers would not take bytestrings where they should have, so you literally could not open files unless you had a valid unicode representation of their file name. That's fun when all you want to do is some data processing.
The core team was swamped with other work around the migration and the community found their response to real world problems lacking. They did things like declaring Python 2 dead long before 3 was usable. Then they refused to fix SSL problems with Python 2, which everyone desperately needed, and went out of their way to oppose a community-led release to fix those problems.
It all ended well. There was another Python 2 release, and Python 3.4 came and fixed most problems people had. It was still slower, but performance picked up and by 3.7 it was just as good if not better. But it was a good ten years during which the response to real world issues could have been a lot better.
All in all, the migration could have happened much smoother. They could have learned from other languages. But ten years is not unreasonable for backwards incompatible changes for a major programming language. The idea that the process could be accelerated by neglecting the old version does not work.
In my personal opinion, the single thing that could have made the transition better for the community would have been the possibility to run Python 2 code in Python 3. It was discussed a lot before the transition and was deemed impossible because of conflicting data types. But it's not impossible, it's just (a lot of) work. It's another type conversion, and Python can already do those. The impossibility to link to old libraries made the conversion a flag day for most code bases, and those are really hard.
512 comments
[ 20.1 ms ] story [ 8184 ms ] threadThe LTS project[0] plans for a minimum of 5 year support from the date of initial release, so Aug 2026 is based on the Aug 2021 release date + 5 years. The rough-schedule on the LTS wiki also uses "June 2026" for now.
That's what we've been showing so far on https://endoflife.date/debian (With a disclaimer about what LTS means). Suggestions welcome!
[0]: https://wiki.debian.org/LTS
The effort for implementation and package maintainers to support both versions was considerable, resulting in some notable packages only supporting one or the other.
In the words of the Python maintainers:
We needed to sunset Python 2 so we can help Python users by improving Python faster.
We released Python 2.0 in 2000. We realized a few years later that we needed to make big changes to improve Python. So in 2006, we started Python 3.0. Many people did not upgrade, and we did not want to hurt them. So, for many years, we have kept improving and publishing both Python 2 and Python 3.
But this makes it hard to improve Python. There are improvements Python 2 can't handle. And we have less time to work on making Python 3 better and faster.
And if many people keep using Python 2, then that makes it hard for the volunteers who use Python to make software. They can't use the good new things in Python 3 to improve the tools they make.
https://www.python.org/doc/sunset-python-2/
Of course, "We want this code to run forever the way it is" is a value option that a company can choose - just not with any language it wants. And somehow, people seem a bit reluctant to switch from Python to C to be able to keep running the same programs forever, guaranteed.
Effectively, however, python2 and python3 are different languages. Since they're interpreted, there's now a problem with '#!/usr/bin/env python' in your script: which python version is being invoked? Which language version are modules targeting? It's a mess.
But the poster child for screwing up a language version upgrade is still Perl.
The first 3.x releases were in no way good enough to replace 2.x and should not be counted like this.
Great that Debian finally pulls the plug on it, that means that most if not all of the downstream distros (such as Ubuntu) will probably follow suit, that's a big fraction of the whole Linux eco-system.
pip
Did you know pip can execute arbitrary code when installing packages?
https://www.anaconda.com/
Some would consider it cheating but I haven't got enough hours in the day. If I were to write some ad copy for Anaconda it would be 'in several years of frequent use this particular user didn't have to resolve a single conflict, it all just worked'.
Which runs counter to just about all of my other experiences with Python and its package managers/distributions/toolchains, some of those stories would end with 'and as a result the whole server was hosed starting from what should have been a routine certificate update' and other such nice adventures.
If you're writing Python code, `poetry` has been the least bad package manager in my experience. It has a lock-file, so it should help avoid supply-side attacks (like the PyTorch one from a few days ago [1]), and it means your local environment and CI/CD environment should have the same packages installed.
I'd also recommend setting the [`virtualenvs.in-project`][2] setting to `true`, to store the `.venv` in the same folder as your code, so that when you delete your code, all of the downloaded packages get deleted too. Otherwise they'll just stick around in your user directory and use up a lot of disk space.
[1]: https://pytorch.org/blog/compromised-nightly-dependency/
[2]: https://python-poetry.org/docs/configuration/#virtualenvsin-...
I know that Python 2 has stuck around for far longer than anyone wanted and that people were (are?) hesitant to migrate from 2 to 3, but I'm not sure what decisions contributed to that.
So IMO, basically what should have happened is accounting for valid use of features that 3 wanted to kill, and actually offering a path forward, from day one, so everyone could do a migration and actually have it work.
Edit: Also (but connected), 3.0 (and IIRC a couple thereafter) should have been marked as release candidates, because they were.
For us it was a change in string handling that stopped us being able to upgrade. Literally 10,000 things would have to be checked. We saw that red hat was supporting python 2 until 2025 so we decided to keep using python 2 until then. I think that once red hat stops supporting python 2 someone else will keep it running.
Python 2 still makes the world go round.
I've been slowly porting a lot of my legacy code from 2 to 3 on an as-needed basis, there's a few bits though that are just too huge/crusty/etc that they will never get ported.
But it would be a mistake to consider Python anything but luxury glue no matter how much you've used it for production code, it isn't and never will be a bullet proof language, there are just too many ways in which the runtime can surprise you and the long term maintainability of any Python codebase is always going to be questionable, in part because you will most likely have dependencies on stuff that will silently disappear or change out from under you while you're not watching.
I've been writing Python code pretty much since it was first released and there isn't a single piece of code that I ever put together that worked longer than a few years out of the box without some breaking change. This as opposed to just about every other language that I've used, which I find very frustrating because Python could be just about perfect. And I'm still sore about significant whitespace ;)
The best practice, which prevents this issue, is to use envs, (as well as pinning your dependencies for any given project in a committable artifact).
(Seems like a lot of people are using system Python, which isn't really recommended.)
The whole idea that you need a multitude of environments to support applications written in the same interpreted language is where it breaks, that should not have been necessary if backwards compatibility had not been an issue.
I really like the language, but all the tooling is seriously broken.
Fwiw what I've settled on that seems the most stable is to use pyenv as the "outer layer", as it shims the python interpreter along with any venvs so you don't get conflicts / heisen-incompatibility. Lets you use anaconda envs as children as well. The real trick, I think, is to pick one system as the canonical master env controller and don't deviate.
The code typesetting is for the humans not the machines.
That is a lot worse than formatting squabbles, especially for a glue language where you are expected to be able to copy snippets for short scripts all the time.
The libraries are very powerful, but a lot of them are hacks. Numpy and Pandas totally rely on magic and overloading the array indexing operator. You can't express a problem the natural way or it will be slow, you have to think the numpy way. I find it especially confusing if you create expressions with numpy arrays. Am I operating on them element wise, or creating a "cross product", or am I creating some kind of magic indexing object? Matplotlib is also annoying, there are so many ways to do things and half of it is object oriented, while half of it uses global state.
One thing I like about Python are the web frameworks (Django, Flask). Async is also good, although it took almost 10 years for vanilla Python to catch up with what Twisted already had.
About code breaking I also have the opposite experience. I wrote a little Gtk app in the late 2000s and it worked with little or no changes for many Python and Gtk versions, thanks to the dynamism of Python. That would be completely impossible with a compiled language.
I do think that "The libraries are very powerful, but a lot of them are hacks. Numpy and Pandas totally rely on magic and overloading the array indexing operator. You can't express a problem the natural way or it will be slow, you have to think the numpy way. I find it especially confusing if you create expressions with numpy arrays. Am I operating on them element wise, or creating a "cross product", or am I creating some kind of magic indexing object? Matplotlib is also annoying, there are so many ways to do things and half of it is object oriented, while half of it uses global state." is an illustration of my point: the language itself isn't powerful enough to do the job so you rely on a lot of libraries written in different languages to glue it all together.
An 'elegant' language would provide a way to do so without all of these leaky abstractions.
To me Clojure is such a language, but it doesn't nearly have the kind of ease of use that Python has.
The language is 100% capable. Matplotlib's API is horrible and confusing because one of their design goals was/is to be similar to matlab (that's where the "mat" in it's name comes from) to allow people who are comfortable with matlab to switch more easily.
Matlab's plotting facilities are a mess of global state, so matplotlib has copied it.
> An 'elegant' language would provide a way to do so without all of these leaky abstractions.
It would have to do something else, first: Exist.
In 10+ years of programming, I've yet to ever see a implementation of a complex abstraction that was comprehensive and complete enough that I didn't find a corner case requiring me to reach through it and deal with platform specific something-or-other.
I think I like Python, because coming from C, a lot of things that are tedious there are effortless in Python (Strings! Lists and dicts, libraries, and so on.)
That's why each and every number crunching problem will eventually make use of the various escape hatches to call libraries written in a language that is performant.
As for strings: BASIC also had strings. That doesn't mean that having strings (or even dicts) is what makes a language elegant.
The problem really is that there are only so many sweet spots for programming languages and usually those come with limitations with respect to the domain you want to use them for. Pick any two: expressive, fast, easy to use.
At least not until the day they suddenly didn’t.
FWIW I found it nearly useless - whereas biting the bullet and refactoring it so that it could be unit tested - even just a few tests - made a world of difference.
That's been happening more and more recently to me. Python 2 really was lax about letting you mix types and get shit done fast, but Python 3 is a different beast entirely. Really opinionated.
Oftentimes there's... Bodges and hacks that made shit work that I never documented.
For Java 8 especially, the later reason is particularly amusing, as Oracle's extended support is now slated to go to 2030 (16yrs after release, for comparison Py2.7 got 10yrs), but Java 21, the next Java LTS that should release next year, will be on Oracle's standard 8 year extended support cycle, and thus extended support only goes til 2031. So why would anyone decide, much less how would one justify to a corporate overlord, to spend all the time and resources to refactor their code, upgrade libraries (assuming the libraries themselves upgrade), update build systems, perform extensive regression testing, and everything else, just for a version that would be EOL one year after what they just upgraded from?
The longer you delay upgrades, the bigger the delta, the more work it will be to update. Instead of doing two medium sized updates, one today and another one in 2030, you'll be doing a huge one in 2030. That might be a trade off you might be willing to make today, but you might regret it in 2029 when you're halfway through a multiyear migration project where all feature work is on hold.
The actual migration should you choose to do it was almost trivial. The official conversion script did most if not all the work. The changes were all cosmetic, like "print 42" changing to "print(42)".
I'd say the uproar was about the minorness itself. The syntax had these tiny but otherwise incompatible changes, and for many people, for no good reason. Fo example it was deemed that, as in the example above, print shall not be a statement anymore, but a function instead. Why? Unclear. Meanwhile, Python 3 offered few concrete improvements to many users
So I think the 2->3 moan, of which I was a party, was about that: minor, unnecessary changes that introduced little to no benefit, and yet backwards-incompatible.
If that were it, it'd have been over a lot sooner. For some packages, bytes vs Unicode strings was bad. For many more, layers of dependencies were an impediment. The trivial stuff wasn't the problem.
But for example a big stumbling block was the scientific programming community, from which I also stem. There were no major syntactical reasons there and yet people were up in arms, for the reasons I describe above.
Most people I know migrated only when python2 literally wouldnt work for them anymore, like they needed a feature of a library only released for python3.
print("goodbye, sweet prince")
Lots of abuse of how weakly/poorly/forgivingly typed Py2 is.
Doing the same processes in Py3 is actually headache inducing because of the much stronger typing system.
What lesson was learned? It seems like a weird corollary of the Streisand Effect: if you announce something's EOL, you'll get more clingy users that keep the product out of the grave, creating a drag on maintenance budgets, but, maybe, if you just stop maintaining and walk away, dependent users will either notice and switch away, or die themselves, and the death will ultimately be quicker and quieter.
(1) As an example: https://news.ycombinator.com/item?id=26168596
LOL that's an ancient bit of copy pasta. Here's a page from over two decades ago that was already describing it as a "classic troll".
https://everything2.com/title/BSD+is+dying
Seriously though, that was one interesting language, the number of weird features (the editor is a string) that you'll come across in MUMPS make it fascinating just to study and wonder what inputs into the decision making process around the design trade offs were.
In case you're not trolling: FreeBSD optimizations used by Netflix to serve video at 800Gb/s [pdf][1]
[1] https://news.ycombinator.com/item?id=33449297
Usenet activity is a very odd measure of deadness to use today.
Although maybe that is exactly what you mean: that the BSDs were already dying in the Usenet era and are still not dead? I think even if they were dying back then, over the course of the past 20 years they may have found their niches to heal and survive in.
Also poeple dont like to change shit that runs well, thats why we have so much legacy stuff around.
In the real world, it doesnt matter if your whole damn business is still running their warehouse system on a DOS program running in a god damn DOS emulator on a Win xp VM running on some linux box. What matters is that it works.
Those are from the 1980s.
Been long enough for everyone to migrate to python3, and this finally happened.
It will be easier for the Python core maintainers, but introducing a breaking change with the intent that that version 2 will be deprecated has probably caused more distress overall, especially maintainers of other open source projects which rely on a specific python version.
Let's never have the same kind of breaking change again, please.
It wasn't not even that easy, to be frank. 2to3 is not a small project [1] and they had (alas, incorrectly) assumed that there were enough people to have enough tests to be benefited from 2to3, only to be found massively wrong. In hindsight no one had a good idea about language evolution; the modern concensus is a single interpreter supporting multiple parallel versions, also called "editions", but (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it.
[1] https://github.com/python/cpython/tree/main/Lib/lib2to3
Not that I've any proof but I kind of find it hard to believe there were no other examples before
But editions & friends are for syntax. Python 3 changed semantics all around. Editions are not an option when you’re changing the langage’s entire text model, especially without static typing.
Python 3.1 could then have introduced some of the other minor changes, perhaps behind a future import.
I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version. We couldn’t have done it without six.
It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake.
Except "that's it" does a ton of work here, as the entire stdlib had to be audited and updated to correctly work with the new text model, plus other semantics change (e.g. the removal of total ordering).
That those were so large and wide-ranging in the first place is why the core team decided that fixing a bunch of other issues could go in as well e.g. reorganise the stdlib, move keywords to functions, remove deprecated stuff, etc...
Those ancillary changes were mostly pretty easy to implement.
> I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version.
It wouldn't have been more pleasant if the breaking changes had been spread over multiple versions, you'd still have eaten all of those, and the early adopter would have gotten even more pain as they'd have needed to implement breaking changes on every update.
> We couldn’t have done it without six.
We did just fine without six migrating a hundreds-kloc-scale repo from 2.7 to 3.5, we basically copied what Werkzeug did, with our own "minisix" module for just the bits we needed (some lifted from werkzeug's and others bespoke e.g. we needed a compatibility layer for CSV).
We could not, however, have done without the 3.0 - 3.3 improvements, the existence of 2.7, or the community experience.
And tests. Having a well-tested codebase was absolutely essential to a comfy migration, the less a corner of the codebase was tested, the more migration bug kept popping up afterwards.
> It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake.
No, doing all those breaking changes version after version would have been a lot worse. IME the vast, vast majority of the pain (short and long term) came from the semantics changes, not the syntactic ones. Those were relatively easy: do the syntactic change, enable the corresponding lint (so colleagues don't reintroduce the old construct out of habit), commit. A few of the API changes were frustrating (like base64 which was clearly mis-converted by the core team but what do you do).
- the text model stuff took by far the longest and had by far the longest tail (as in things we missed during the migration), this is in part because not all of the changes or Python 3 APIs are ideal, but mostly it's just a lot of runtime things in a lot of paths if your software leverages strings a lot
- total ordering removal broke a surprising amount of things, and also had a pretty long tail of bugs, mostly in that `None` would get mixed in with other values and get fed to comparison operators, sorting, or min/max
- `round` was a surprise contender for pain in the ass, not just because it changed the rounding method (to banker's rounding, from I don't remember what) but also because P2's round() always returned floats while P3 returns integers if precision is missing / None
- changes to dict iteration also broke a few things, there were multiple pieces of code with implicit reliance on iteration order, although for the most part they were issues in numerical stability of floats, which broke tests using strict equality on floats
- division changes also broke a few things as it wasn't always easy to recognise whether operands were always floats or could be int, also `//` being floor division rather than integer division led to a few behavioural changes
We also did a few things like banning the builtins which had changed behaviour (open, map, filter, ...) for the duration of the migration, the issues they caused were subtle enough as to not be worth it. They were...
Most of the other changes were kind of trivial though. print → print(), some stdlib name changes: that's easy stuff; you can automate that, even provide a compatibility shim. The division operator was a bit more tricky, but I never encountered code that broke due to it (although I'm sure some did, it didn't seem like a wide-spread issue). Plus you could do most of these things in Python 2.
> It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible.
You should have seen the initial discussions surrounding Python 3000, as it was called back then. The most common thing Guido had to say was "No, that's far too large of a change, we're not designing a new language", even for things where it was agreed on that the situation wasn't ideal. The community came up with some pretty wild proposals, and a lot was kept out of Python 3.
The difficult changes between Python 2 and Python 3 are not static information, at least not without fully typing the entire codebase, which was even less feasible in Python 2 than it is in Python 3.
https://docs.python.org/3/library/__future__.html
That only works with “syntactic” updates, not with deep semantics changes.
Those are why the core team pulled the trigger on an overhaul update. And they were by far the biggest issues during migration, anything which would be gated behind an edition or opt-in was easy.
Perl did it since ages. You just put
in header and interpreter enabled those language features on file by file basisFrom the toolchain and language developer side, nobody liked Perl's approach. That's why there was so much effort around Perl 6 (which became Raku) at around the same time.
If you tried to run a 5.8 script on a machine without 5.8, you'd get an error as should be expected. If you ran a use 5.6 script on a 5.8/5.10 machine, I think you'd get 5.8, but I can't remember... 5.10 was less disruptive than 5.8.
I don't know where this discrepancy comes from.
(And before anyone asks, a big part of why it’s so big is that it’s very old. So old it predates Django, so, in house framework, in house ORM, etc.)
Took a calendar year of ca-a-a-arefully migrating these one by one, all while delivering stuff and making sure nothing falls apart along the way.
Not pleasant. Can't recommend.
It's been 14 years and dropping python2 is still controversial.
It's a story of how not to do a major language upgrade.
It's not quite as bad as "Perl 6" which killed off the whole language, but it's up there with massive mistakes which has cost the industry dearly.
To be fair it was a place where the Python fans advocated for its use but to management was always a bit skeptical, so we had to make it all seem easy. We had good solutions to internationalization which all broke with Python 3.
We cannot deny everything that happend before Python 3 was released.
3.8 is a bigger C ABI roadblock than 3.10.
When I worked on patching/rebuilding all the Python packages for Alpine's upgrade to Python 3.10 I remember quite a few things being broken and needing patches (maybe about 5% of the overall packages), but a lot of those were also test failures due to the new deprecations, so while some stuff did truly break, it was pretty rare.
[1] https://docs.python.org/3/whatsnew/3.10.html (see the removed section especially)
The print-as-a-statement was ugly but convenient and didn't seem like a big deal, the integer division was something that you could live with once you knew about it (and you still need to know what the current behavior is), and the new string types make code more verbose, and the way they were handled introduced countless hard to find bugs during the migration.
Core developer Brett Cannon's talk "Python 3.3: Trust Me, It's Better than 2.7" goes over most of the differences at the time (2013):
https://www.youtube.com/watch?v=f_6vDi7ywuA
The fact that such a talk exists shows that the improvements aren't worth the switch for most users. If they were, they wouldn't need convincing.
Yes, Python 3 is (mostly) better than Python 2. No, it's not even remotely worth the amount of confusion and work it caused. If Python 3 had brought massive performance improvements, or proper support for multiple threads, then maybe it would have been. As it stands, Python's biggest weaknesses remain unaddressed.
That's reading far too much into a tongue in cheek presentation title.
Besides, good devs are typically reasonably skeptical of new and shiny.
And when you've got a large Python 2.7 codebase, and Python 3 is backwards incompatible, you'd be looking at a lot of work to upgrade, so damn straight you'd need convincing to make the investment.
Hell, look at how many Java codebases are still 1.8 because of the upgrade cost of going past that, even though modern Java has many compelling features.
And expecting Python 3 to remove the GIL is incredibly unrealistic. How much Python code in the wild implicitly depends on the behaviour of the GIL? All of it.
About as much as depended on Python 2-specific features?
Removing the GIL would be bloody fantastic, but it's far more involved than 2 to 3.
More like it shows that many people like GP are only aware of the superficial changes.
And, the latest iterations of Python3 have real usability improvements - the error message improvements alone make development a lot nicer. And the perf improvements and jit coming I think in 3.13 will really make more people consider py3 for their new projects.
Have you worked with pervasively non-ASCII text in Python 2? Like on a machine where any file might suddenly turn out to be non-ASCII (even if it’s only in a comment), not just data inside a few carefully-patrolled fences? Outside of a few well-behaved libraries (Flask), my experience was that it was utterly impossible. More than half of my time debugging straightforward code was spent chasing down and fixing UnicodeErrors, knowing that some will still remain. And that includes code that used nothing but the standard library.
Now you might think that’s not important, and though I disagree (especially when talking about “most users”) that would be fair. But it is a substantial improvement going from 2 to 3.
The print() function was absolutely not worth all the carnage it caused. They could have just kept the print statement around and introduced a printing function with a different name, for example.
The fact that they removed syntax for Python 3 meant that all kinds of scientific computing code had to be transitioned even if it didn't use any strings!
It's also not a problem if you only have ASCII strings.
The Unicode handling on the other hand took ages because string handling is not easily searchable and replacing it requires understanding what the piece of code actually does.
So you have it exactly backwards: if the only migration was the print function then it would be a quick and relatively painless process. But fixing the hard thing which was the broken str/byte model was the hard transition.
I am curious if you actually went through this transition or if your opinion on this is more based on observation of others’ work.
Of course not, you’re just going to revert back to the version of bash that doesn’t require parentheses and sit on it for as long as you possibly can.
Code exists in three states: actively maintained, done/mature, and obsolete.
Python 3 attacked mature code, forcing developers to either go back into active maintenance or to obsolete it early.
I did transition some Python 2 code to 3. But mostly it was uneconomical and risky, so the vast majority of it sat on Python 2.7 until it was no longer needed.
I feel that most of the people who had a positive experience with Python 3 were people doing webdev using Django. They were working on monolithic actively maintained projects and so transitioning to v3 was just a normal maintenance activity.
Again, you are arguing that running 2to3 and testing your code was hard. I guess that may be true. But having hunted for a ton of subtle Unicode bugs that problem was very worth fixing.
Didn't the unicode changes go on until at least Python 3.6? The issue that most C APIs did not care about their byte strings having any encoding at all seemed to cause one headache after another, especially on Posix, where everything touching the operating system could return random bytes. Reducing the scope to unicode wouldn't have sped up the transition one iota.
So personally I'm ok with Python 3 but I actually never ran into the described problems despite using Python 2 for years, so it was at least not a universal problem that needed to be solved in such a breaking way.
I didn't insinuate that there weren't problems or that I wanted to keep Python 2 - just that it was perfectly normal to use it and not run into problems. I think I only remember a handful of encoding problems with my other python 2 adventures...
There’s no need to sugar coat it.
It was incompetently managed, in a way that made the technical success of the work look like a failure.
That’s on them. They screwed up.
I lived through it; I feel no particular need to smile and nod and say “it wasn’t that bad”. It was bad.
Things are good now! Python 3 is great, it’s well supported and the people involved all learnt a lot about how important managing communities is, as well as writing good code.
So, all round, a success!
…but, if you don’t acknowledge failures in the past, you’re doomed to repeat them.
> Now you might think that’s not important
I don’t think your experience with Python 2 is not important; I had similar issues with it.
No one is seriously going back to Python 2 at this point.
However, the process is a lesson worth studying; and the take away is not “what a technical success!”
… it is: never do this.
Big scary breaking migrations are bad, they hurt your community, and leave bitterness behind them.
Even if the result is better, technically; there are other factors that need to be considered too.
No, the community is stupid and wrong. My machine learning professor (actual professor, not a teaching assistant) only used Python 2 in 2017. Imagine how many people had their opinions shaped by someone like them.
> … it is: never do this.
By that standard, we would never get things like angular 2 or Raku.
Would it be better if we renamed Python 3 to some other name and pretended it was a completely new language?
However, I am firmly in the Python 3 camp. It is a better language (all the newfangled bullshit notwithstanding).
Quite possibly yes? It doesn't run python2 code, it can't be inter-linked with python2 code, so is it really the same language? The hybrid dialect that works in both does exist but you have to have compatibility shims in a few places.
By comparison, most C compilers will let you use C89 and C17 code in the same project provided you set the compiler options, and C# provided the "netstandard" target for libraries to smooth the transition between Framework and Core runtimes.
Sometimes.
I mean, Russ basically said that about all the work people were doing on the golang package manager and did his own thing, and it worked out. Good job! Sometimes, a smart person can do something that is better when they're deeply steeped in the domain.
> No, the community is stupid.
I personally find that most communities have a lot of very clever, very thoughtful people in them, and if you listen, you can learn a lot from them.
Just because they don't agree with you, doesn't make them stupid.
I think that its basic respect to listen to your community, acknowledge them and think about what they say, even if you go your own way.
...but, if you (as a maintainer) just think they (your community of users) are just a bunch of idiots, do whatever you like. They are idiots if they hang around in a community like that.
For comments and string literals in Python code, note that Python 3 changed the default from Latin-1 to UTF-8, but you can still use -*- coding: -*- to change that. Bytes literals outside of ASCII range you’ll still have to escape. As for program input that’s possibly invalid UTF-8 but not in places you care about, you’ll need to set errors='surrogateescape' or similar explicitly (or possibly set an 8-bit encoding like Latin-1, though not if there’s a chance of UTF-8 in places you do care about).
Paths are a bit painful, yes, both because there’s nothing on Unix systems that precludes /home/alice having filenames in UTF-8 and /home/boris in KOI8-R, and because NT paths are not byte sequences at all (they are WCHAR, that is 16-byte-number, sequences, in no way required to be representable in the current CP_ANSI). Having uninterpreted byte sequences as filenames would work to solve the former issue, and it does: I think most functions in os will accept bytes paths as input and treat that as a signal that you want bytes paths as output (if any). For the latter the cleanest solution is probably an abstract Path type—and you get one, in the standard library’s pathlib. That didn’t work all that well in early Python 3 versions, but it does now. (And IIUC inspired Rust’s abstract-type solution to the problem.)
Python 3 takes this fact and runs with the stance that strings should, by default, carry their encoding with them (and that encoding is specifically, more or less, a sequence of quasi-Unicode code points with unspecified internal representation). But this is an intensely half-baked solution to the wrong problem, because you need to know much more about a string than its encoding for just about any string operation you'd want to do. Just with Unicode code point concatenation, you can run into plenty of trouble with: - strings containing control characters - strings with unpaired surrogates (which okay, shouldn't really be there, but get used for various reasons anyway) - strings containing combining characters - strings in different natural languages - strings in languages with ligatures - strings with ligature-like emoji (and this is obviously not remotely close to exhaustive.)
The first python 3 versions were not great and were missing some things. I think 3.3 didn't even had pip builtin
I ask, because I know I would have wanted that.
Also, the fact that there is a talk with such a name does not mean anything other than that someone is out there trying to clarify what the transition is about.
(There could have been a video "Windows 7: Trust me, it's better than Windows XP" and you probably still would say that it showed that improvements aren't worth the switch for most users.)
The Py3 approach was terrible and wasted untold amount of hours just because you had to migrate everything, you couldn't just upgrade codebase piece by piece like in case of Perl.
At the very least they should've just made new one be under .py3 and just make Python3 interpreter transpile Py2 code at runtime
The Python mess was, IMO, a typical example of bureaucracy inventing for itself new but previously unnecessary work to justify its existence, so I agree that that the decisions of how to introduce Py3 features caused (and still cause) waste of immense amount of the hours world-wide that could have been used more productively. Sad.
I think very few people had trouble remembering them, and if you're using one, you'll be looking up documentation for anything with more than a few methods.
It really is frustrating in these situations with high level languages. It's super possible to do the majority of changes they did in non breaking ways.
Hell, they do it already with the __future__ package. Where did all that compatibility work go in python3?
There was some breakage between 5.6, 5.8, and 5.10; Unicode is hard, and it takes time to get it right. But I think the key difference I've heard as a mostly python avoider is the intent was for code written for 5.6 to probably work in 5.8 and 5.10 and if it doesn't, for there to be a way to have one file that works for all versions.
From what I understand, it's not easy to have a python module/script that works in 2 and 3, and you can't go to 3 unless your dependencies do, so if you have a lot of dependencies (as is modern), you're stuck on 2. Your dependencies won't want to move to 3 either, because their users are stuck on 2, so if they just switch to 3, they're droping users; instead they need to support two parallel versions of their code. Most perl modules didn't have to do anything special to support 5.6 and 5.8, but if they did, it was usually small and it could be done within the file with conditional compilation --- I don't think that was an option for python.
.. but you can't use old libraries in new code.
I think having Python 2 vs. 3 is a clearer demarcation, myself, but the Perl community cares a LOT about backwards compatibility.
Also, Perl 5 is a very different example. While I haven't followed Perl for quite a while, ISTR that Perl had a very similar transition from 4->5 (I personally remember spending a lot of time trying to get old code working on newer OSes). Also, AFAIK but I may be wrong, Perl 6 has not really taken off like Python 3 did.
That's terrible practice.
Personally I only use python for throwaway scripts. Whatever is written in Python likely won’t work in 10 years anymore. So they’re all lost in time. Whereas I can continue some of my C++ eternity projects and they still work perfectly fine.
If you don't write special in your code, Perl doesn't have a "say" method (because it didn't have one in Perl 5.8). If you declare in the code that you're code for Perl 5.10, magically there are all the Perl 5.10 features, including "say". You can also declare you want specific features, rather than declare a specific Perl version.
The best thing about this is you can mix and match libraries from earlier and newer Perl versions. If a library says they want Perl 5.10, they get Perl 5.10, even if you want both Perl 5.30 and that library.If only Python were as reliable for backwards compatibility.
The Perl community is going through their own Py2->3 moment, and they're doing it with great care. They're going to create Perl 7, which will be Perl 5 where the "defaults" make the language look more like Perl 5.32 rather than Perl 5.8 [1]. Modern Perl scripts don't need so much boilerplate to enable modern features. They know exactly the trouble this will cause; if Perl 7 ever becomes the default Perl interpreter, the only one, it _will not run_ lots of very old Perl scripts untouched, because the defaults will make years worth of language changes suddenly visible to those scripts. Even if the fix is just to slap "use 5.008" on every script, they worry about it. That's a much better attitude than the Python developers.
[1] https://www.perl.com/article/announcing-perl-7/
> That's a much better attitude than the Python developers.
You do realize that this comes at a cost, right?
A lot of decisions in Python3 (and further) are based on the desire to keep the interpreter simple.
Python Core Team doesn't have unlimited resources, and even if they did, the cost of complexity grows non-linearly.
py2 -> py3 transition did not go as planned, but that doesn't mean that following Perl route would have had better outcomes.
Hell, if you're still using 3.6 you're limited in your ability to upgrade dependencies, 3.8 is generally the minimum supported Python version these days.
I tried using the library less than a year ago. I must have missed something. This would be great news… if the way you told me didn’t make me feel like shit.
Where would they go from python? Node?
It may have lost a lot of people who got frustrated by the 2->3 transition, but Python is clearly not in a bad place as a language today.
Most actively maintained Python code works with Python 3 now, but there's still a whole lot of even maintained code which uses `#!/usr/bin/env python`.
https://www.w3.org/International/articles/idn-and-iri/
https://www.rfc-editor.org/rfc/rfc6531
etc...
Just because you can get away with ignoring the non english speaking world doesn't mean Python should pretend it can.
As for RFC6531, my understanding is that virtually no email provider implements it, because of the same risks that make browsers often show IDNs as their punycode version - Unicode is extremely easy to use to confuse people maliciously or accidentally, since it contains vast amounts of duplicate characters (e.g. Latin a and Cyrillic а), even larger amounts of similar looking characters (e.g. Latin r and Cyrillic г), and even characters that are ambiguous unless you choose a locale! (the CJK problem, where the same Unicode codepoint can represent different characters based on the locale - whose locale to use when communicating between two machines being the implementer's problem).
Also, I'm not saying that Python shouldn't have had proper suport for separating byte arrays from encoded strings. I was only pointing out that there were actual legitimate use cases where a valid Python2 program was broken by the Python 3 Unicode string transition, whereas the GP was claiming the Python 2 had to have been buggy already.
Edit:reading around more, it seems that RFC6531 is getting some traction, and many providers accept sending to/receiving from internationalized emails even if they don't themselves allow you to have a Unicode email (e.g. you can't have айа@gmail.com, but you can correspond with someone having such an email at a different provider). So, email was a bad example in my list. The rest still stand.
For example, what if I want to enter "ουτοπία.δπθ.gr" in an application, via direct user input, a config file, or something else? Or display that to the user from punycode? No one expects users to convert that manually to "xn--kxae4bafwg.xn--pxaix.gr", and no one will understand that when being displayed, so any generic IDN/domain name library will have to deal with non-ASCII character sets.
The same holds true for email addresses: "ουτοπία <a@example.com>" is an email address. Sure, this may get encoded as "=?UTF-8?q?…?=" in the email header (not always, "bare" UTF-8 is becoming more common) but you still want to display that, accept it as user input, etc. People sometimes to forget that the name part of an email address is widely used and that any serious email system will have to deal with it, and non-ASCII input has been common there for a long time.
In specific applications you can often "get away" by ignoring non-ASCII input because you sometimes don't need it. For example I'm working on some domain name code right now which can because everything is guaranteed to be ASCII or "punycode ASCII", so it's all good. But in generic libraries – such as those in Python's stdlib – that's much harder.
How would you send an email to the customer support of flüge.de? How would you parse that domain?
IDN is here, and it's here to stay.
Certainly today DNS domain names, URLs, and email addresses can – and do – contain >0xff input, and for some of these that was the case in 2008 as well (URLs and email addresses – IDN didn't come until a bit later).
The Python 2 situation was untenable and lead to a great many bugs; "decode errors" were something I regularly encountered in Python programs as a user. In hindsight, the migration effort for existing codebases was understated and things could have been done better with greater attention to compatibility, but the problem it addressed was very real.
Is that true? My understanding is that DNS never supported non-ASCII and so punycode was invented.
I can count on my fingers the number of places I invoked manual DNS lookups inside production code.
At least python3 got that right, unicode by default.
I don't understand why there apparently wasn't a clearly stated preference about `python` being Python 2 and `python3` being Python 3 from the Python org.
"Everyone has has a chance to migrate" . Doesn't work that way, just yesterday I spent a few hours trying to get scripts that are really good and would take me forever to migrate that are in python2 to work on an ubuntu based distro and this is with pyenv, custom venv,etc... just when you get python syntax worked out you are faced with deprecated or uselessly renamed imports and modules.
This is the shitty aspect of using foss stuff. I don't pay for it so I have to accept their decission. But in the broader sense, inability to influence things like this even with a ton of other people willing to pay for it is frustrating.
You know what would be great? If the community can vote with money pledges. So it you have 1000 people willing to donate $10/month to support python on sysv init then if volunteers don't want the money we get an external person getting paid 10k/month to support these packages even if the corporate types, developers and "devops"/sysadmin people have no use for it.
The free part of foss is expensive but I think the middle ground is to collectively pay for the support. I try very hard to avoid python2, I am not a developer and I very easily lose at least 1h/month on average to resolve these issues.
Do you all know why 2023 will not be the year of the Linux desktop? Because desktops are a commercial market and consumers/users cannot sway what the Linux desktop experience is. They either accept things as they are or pay for them by spending time reporting bugs, troubleshooting stuff, asking questions on random sites,etc... a lot of this has to do with change and maintenance requiring human time and resources. It's great when there are no issues and volunteers donate their time but it would be great to be able to pay with money also to support features or resolve issues. You can even make sure multiple people pay an equal amount to avoid wealthy people having too much or an influence but it still will not affect regular volunteer work.
I was hoping Ubuntu and Fedora would have such features.
And... you believe that Microsoft and Apple are swayed by users? More than Red Hat and Canonical? I don't buy that for one second.
That's true. Now look at 8, 10, and 11 and tell me with a straight face that they still care.
> Do you know why MS is bloated and "slower"? Because of all the technology they support from decades ago and all the interpperability layers in between.
Then surely the comparison is RHEL, where companies pay for compatibility and stability, not "we <3 bleeding edge" Fedora or "volunteers maintaining a coherent system" Debian.
Do you see the Metro start screen anywhere on Windows 10 or 11?
If they weren't swayed by user input, why did they throw away all that time, effort, and energy put into the the Metro UI and go back to the start menu paradigm?
> Then surely the comparison is RHEL, where companies pay for compatibility and stability, not "we <3 bleeding edge" Fedora or "volunteers maintaining a coherent system" Debian.
Unfortunately, RHEL and SLED are both targeting enterprise users, very little of that is desktop as well.
You know who else succeeded in the Linux desktop that isn't really considered proper Linux? ChromeOS on the ChromeBook.
If only these new laptop makers like purism or system76 took a similar radical approach where they would indeed listen to paying users more directly better than existing bigcorps, that would give Linux on the desktop a chance at market share gains that are significant. Then I would be comfortable gifting a Linux laptop to a non-technical user who can call a support line when having issues or needing a feature.
Stand-alone applications should use pyenv/venv because you can’t guarantee which version of Python is the system-wide one (and you don’t necessarily want to tie your app to the distro’s release schedule). Dependencies should always be in a venv as to not interfere with the system-wide ones or other applications.
If the removal of Python from distro repos breaks your things then it’s a wake up call and you’ve been doing it wrong the whole time. Fortunately it’s an easy fix.
That does not sound like a user friendly programming language.
The version compatibility of Python is a joke and has been after 2.7 to 3 to. Which is the main reason I don't use Python for anything but bash-like scripts.
Other languages or OSes that don't have their interpreter bundled forced you to think about the problem from the start and build a way to ship a stable version of the runtime with your application.
Python, as a server-side programming language is not too bad. You can standardise on a version and bundle that interpreter version with your application itself in whatever artefact you produce (whether a Docker container, tarball with Python binaries + source or entire VM image with the right Python version installed).
Desktop distribution is a pain, but that's not limited to Python; any language that can't compile to a native binary will have this problem - even Java, despite having much better backwards compatibility, still isn't prefect and sometimes requires an older JRE to be installed on your target machine.
If you want to complain about Python 2 being unsupported by the Python foundation, you're free to. But no distro can in good conscience keep distributing a thoroughly unsupported piece of software that's probably full of security bugs.
Regarding the "year of the Linux desktop" thing: Apple is much more aggressive than any Linux distro when it comes to dropping deprecated stuff and forcing their users to move on. Apple hasn't shipped Python 2 for a long time, they ship Python 3.
You need to install either "python-is-python2" or "python-is-python3" packages to get proper system-wide symlinks. Also, it was strictly verboten to give either of two packages as dependencies to workaround migration procedures.
Debian Testing doesn't have Python2 at least for a year now.
Another mistake in Py3 migration was keeping the interpreter name as python so when you had it in path you had a choice of new stuff breaking or old stuff breaking
It was, in the past.
When python2.7 deprecated and removed in Debian testing, python ceased to exist as a command. You need to install these packages (python-is-python{2,3}) to get the python command and point to the correct interpreter you like.
This behavior is consistent in two Debian testing systems, installed 6 and 9 years ago, respectively.
> Another mistake in Py3 migration was keeping the interpreter name as python...
I installed python-as-python3 myself, after python2 disappeared from system to fix an oddball python3 script which bundled as a part of a bigger software package which is not in official repositories.
Maybe they have fixed it, but I didn't bother to check till now. I'm not even sure which program it was.
So, it was an addition to a clean system and didn't inadvertently directed python2 code to python3 interpreter.
I guess nowadays, as the py2 compatibility is a thing of the past, most programs would crash when using that interpreter version.
Congratulations to Debian on upgrading to Python 3!
And what surprised me most is that every time I mentioned this, there would be lots people telling me how this is a superior design because you can operate string like bytes. I mean, it of course has its upside, but I don't think it's worth it if you care even slightly beyond ASCII.
For me, the unicode handling alone was worth the time spend migrating from Python 2. That was a decade ago, to finding that the "python" command still launches a Python 2.7 interpreter in 2023 is just beyond belief. Personally I feel like they should have yanked Python 2 in Jessie (Debian 8) in 2015, more realistically in Stretch in 2017.
apt install python-is-python3
I'm sure you could do things well in Python 2 with proper combinations of encode/decode, but it wasn't obvious where you even needed those because with ascii text, things "just worked" transparently. With Python 3 it's very obvious where you need encoding/decoding because bytes != str.
In the early 2000s, I maintained an py2, wxPython app with the users having the system encoding win-1250; the effort to patch this was unbelievable. The migration to python3-style handling forced everyone to think about these issues, not just few people for which things were crashing. Even just popularizing the issue was great, until then, many maintainers of third-party libraries didn't even understand what is the problem that you want to "needlessly complicated" fix in their libs.
The problem is not the end-user invoking the command.
The problem is scrips expecting `#!/usr/bin/env python` to invoke python-2.
For example GnuRadio started supporting Python 3 with GnuRadio 3.8 released in 2019, and then you had to port all your programs using it to this version. So no, in 2017, the ecosystem was not ready.
In order to support Windows better, Python 3 introduced support for UCS-4 (or worse, UTF-16) strings (depending on a compilation setting when Python was compiled) and they had to introduce extra string types to distinguish readable strings from binary strings ("bytes").
These extra types made Python 3 a lot harder to teach (I teach 30 person classes every year).
So it's not all roses now.
In the end, I got used to it, BUT I just gave up asking encode()/decode() questions at the exams. Very few people understand it, or care enough (and I understand why--it's a ridiculous thing to have). You only need it if your OS somehow slept through the introduction of UTF-8, which is backward compatible with ASCII, resilient even if there are transfer errors and can encode all unicode characters.
Encoding problem used to be really common in UNIX (and before that, in mainframes), but with the introduction of UTF-8, all encoding problems I had vanished and never appeared again.
Even Windows 10 has an UTF-8 mode now and the Windows API functions that end in "A" can be made to use UTF-8.
Now, in a sense, Python 3 has this entire complication for no reason.
That said, Python 3 is ok to use now--and, conceptually, distinguishing byte strings from unicode strings is better (for example so that you don't accidentially print the former to the terminal). It just uses up brain cycles that you could be using for solving your actual problems.
I get it from the "pass the exam" perspective, since that's one more thing to worry about.
But from my experience in teaching others, doing the conversion between bytes and string implicitly (à la Python 2's way) hinders actual understanding of this very important concept, and it's quite harmful in further study.
Bytes should be considered as a separate, more low-evel thing, away from int/float/strings; at the very least, it should be considered as bits/hex numbers. If you want strings, you explicitly encode/decode them in a way, even if everything is UTF-8.
On top of that, "byte string" is just a confusing concept. It might works for English speaker (by "it's a ridiculous thing to have" I assume you mean that, "'english'.encode() is just b'english', why bother?"), not at all for Chinese speakers, even in UTF-8. There is no b'中文' -- only b'\xe4\xb8\xad\xe6\x96\x87' which has zero meanings in their own.
And even from an easy-to-use perspective: most people don't even work on bytes often nowadays. A more abstract "string" type is all they need, without worrying about how it works under the hood (and if they do, they need to understand how encode/decode works properly anyway).
There was no conversion. `bytes` and `str` were the same type.
http://docs.python.org/whatsnew/2.6.html#pep-3112-byte-liter... says:
> Python 2.6 adds bytes as a synonym for the str type, and it also supports the b'' notation.
I just checked in Python 2.7:
I'm working with Germans, Japanese and Polish that use a lot of special characters, including Kanji, umlauts, extra quote characters etc. I need the non-ASCII parts and had no problem with them in Python 2 on Linux (now, C++ libraries that reinvented their own string classes: many problems; C libraries: no problems).The point is when bytes is str, everything works just fine in Python 2 Linux with UTF-8 locale (which are used in all modern Linux distributions). No need to have a distinction between bytes and str.
That how the rest of the OS works, too. Even a lot of Gtk, Glib and so on (for example the GNOME desktop environment) assume that you are in an UTF-8 locale for file names, for example.
> A more abstract "string" type is all they need, without worrying about how it works under the hood (and if they do, they need to understand how encode/decode works properly anyway).
Ehh, we had students write drivers for measurement apparatuses and they all used Python 2 str (without being prompted to do so). No encode or decode anywhere. Of the students, almost no one who tried Python 3 for that stayed with it (instead they were using Python 2). There was just no upside for this use case.
I agree that, long term, having a distinction str vs bytes makes sense. But then you ARE juggling things that the OS doesn't need--it's basically busywork in Linux.
I'm not trying to minimize your experience--but I don't think it would happen if you tried python2 on Linux today. Not sure it was worth it breaking compat for that.
That's not true at all. I remember all kinds of encoding errors when dealing with the FS, the network or any user input when using Linux.
Unless you're talking specifically about IDLE?
Debian had Py3 for ages. It wasn't upgrade, that happened ages ago, just removing old packages kept for compatibility
If they had started a new project called COBRA and just eventually had it over take python the world would be all over it.
But massive breaking changes going from 2 to 3 was a disaster for their audience.
Worse still you could hear the acrimony in Pythons communications on the topic.
1. The amount of breaking changes. If it was possible to do a little at a time, over a series of releases, it's a bit more manageable.
2. Being an interpreted language makes it difficult/painful to break compatibility over a series of releases, as all applications and libraries have to be kept source compatible. Source compatibility also makes it a pain for library maintainers, as you can't just cross-compile the library to a older ABI.
Maybe support for consuming libraries based on an intermediate, compiled representation, would make it easier for the library eco-system to follow breaking changes over a series of releases, and in turn make it more feasible for applications to handle a small amount of breaking changes over a longer release timeline.
It was also for a time impossible (or close to impossible) to run the same Python code in 2.x and 3.x. It helped when later 2.x versions acquired more 'from __future__ import'.
They really aimed for making it maximally hard to support v2 and v3 at the same time which created a nasty coordination problem: why upgrade if the libraries aren't ready and why upgrade the libraries if there are no users.
Python source files then somehow need to announce what python runtime version they expect.
Having to support old versions forever also should make the maintainers think twice before breaking compatibility again.
The one of the things Microsoft appear to have mastery of is making sure users and devs don't feel breaking changes.
Drivers generally as well as D3D.
Even if you don't get all old code to work, having say 80% of it working means to start using Py3 you'd need to only initally rewrite 20% of the codebase, run tests and off you go, then migrate rest slowly over time.
- Added some stuff to AngularJS to make it more Angular2 compatible so that you could get closer to the target without taking in the breaking changes
- A thing called ngUpgrade let you dual-boot AngularJS and Angular2 together, so that you could transition parts of your program over time without needing to redo the whole thing
- Well documented migration guides for each step, outlining what you need to do in order to move from X version to Y version.
I suspect a lot of the controversial decusions made doing the 2to3 migration by the python foundation was deliberately designed to avoid a situation similar to what happened to perl when the time came to cleanup some of the old cruft from the language.
It was a necessary change too. Supporting Unicode strings or mixed-encoded strings was absolutely awful in Python 2 and almost not worth doing. Python 3 made that work properly.
Arguably, the only real mistake was using UTF-16 instead of UTF-8 for strings. But back then, nobody knew what would take off and Windows natively uses UTF-16, so...
Raku is a completely different language from Perl. Writing Raku instead of Perl 5 is like writing Rust instead of C++.
But it was a lot easier for the CEO to digest `2 -> 3` as a clear path to the future as opposed to switching to a new language.
Nope, the real life conversation would be "Is 2 still supported ? Yes ? Well then we will migrate when it won't"
Python doesn't exist anymore, everyone's using COBRA. We can dedicate a dozen engineer months to switch to COBRA.
Or thousands to maintain python ourselves...
Starting with Java 9, finally the long deprecated APIs started to be removed.
In any case, the issue is that we are forced to write C# 7 and Java 8 code.
Some well known examples, Sitecore, SharePoint WebParts, SQL Server SP extensions, Dynamics, Office AddIns, VS plugins, Solr extensions (many places keep using Solr on Java 8 even if more recent versions exist), Android (unless one goes Kotlin or targets version 12), Forms and WPF 3rd party components (the versions on .NET Core are incompatible due to new designer)...
I am stuck on .NET48 at least for another two years (because of webforms frontend) and the last version of EF Core that supports .NET Standard 2.0 is 3.1. They dropped .NET Standard 2.0 like a rock... for what? Few default interface implementations.
Or nullability attributes that are missing in .NET Fx, they could also be defined externally, like R# does. Nope, sucks to be you.
Compared to Python 2->3, the breaking changes were more obvious as they involved entire strucutres and paradigms being deprecated. Python 2->3 felt a lot more subtle despite being just as damaging.
And if you don't want to use moin2, it can convert the custom markup to Markdown, which you can then use elsewhere.
That's my impression from https://moinmo.in/MoinMoin2.0 anyway.
It's a huge usability fail on the part of Python that it's not obvious and easily memorable how to do this.
If you think about it, that's the only way that makes sense; if you wanted to go from, say, UTF8 to UTF16, you'd need to have an intermediate step that was abstract and formatless. Similarly, when you're operating on text, you don't want it to have a particular format; the format should only exist at the edges, where you take text in and output it. The format is a way to communicate with other programs, not components within your program.
My advice would be, if that doesn't stick in your head (totally legitimate), write down a code example and stick it in a personal wiki or something. I find it really helps me to understand something once and more or less copy paste it forever.
This is a key reason why people hated Python 3 so much IMO.
For context I’m a huge advocate of Python 3, except for this (and lambda functions, but that’s another whine session).
https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
A simplistic summary is that you always decode to unicode and always encode to bytes.
Another way to think about it is that Unicode is an idealized character model that is reified into a byte encoding, usually UTF-8, for practical usage.
It seems to me that this is something a lot of developers don't think too much about until they run into some obscure encoding/decoding issue, but it's kind of a fundamental aspect of programming that's worth spending some time grokking.
The problem is that encode and decode are directional constructs that change depending on context.
That’s a poor UI in Python.
It's tricky to look up a different solution you aren't familiar since the terminology is different. What I'm seeing currently doesn't really look that different to Python, but with different names for the types and such.
Taking a stab at it; This looks simple, and runes are great (similar to Python):
https://golangdocs.com/strings-in-golang
But something that bothers me is that converting an array of bytes to a string looks to implicitly be ASCII, or possibly UTF8. This document doesn't say.
This document acknowledges encodings and clarifies that literals are UTF8, but makes no mention of byte arrays:
https://go.dev/blog/strings
I've poked through as many SO questions as I care to and am not clear on how to use different encodings. Looks to me like this is a difference in philosophy; Python would say, explicit is better then implicit, and I'm inclined to agree.
If I'm not understanding properly I'm happy to be corrected.
Perhaps a convenience method would be nice, something like this, but it somewhat obscures the intermediate decode-to-unicode step:
Isn't it true that encode always means text -> bytes and decode always means bytes -> text?
https://xkcd.com/353/
It's really sad that the Python folks committed suicide this way, all over a unicode fetish.
Is there a risk it becomes like one of those situations where certain people use a topic simply to make conversation? Or the modern exaggeration to avoid seeming boring? Or a bunch of "oh, it's gonna cost ya" contractors plying the age old lines of car mechanics negotiating basically anything. Someone below referred to carnage. I wonder how far off we are to someone claiming it was actual violence.
I was lucky it didn't affect me much, but I could sympathize if somebody had hundreds of thousands of lines of python2 legacy code that needs to be migrated..
Are there some valid reasons that made 2->3 migration insanely hard for some projects? I remember seeing blog articles whining about print vs print(), but surely there are some more important stuff.
Print was a non-issue once you could do "from __future__ import print_function". It was the unicode migration.
All string handling from I/O required at least checking.
The dependency issue was the most crippling, though: because you can't load Python 2 code from Python 3, before your project can begin migrating you have to wait for all your dependencies to update as well. This may involve them changing the types that get passed over the API from 'str' to 'bytes' or vice versa, which is potentially a breaking change.
For python2, the str class could either contain a sequence of characters, or could contain a sequence of bytes. These are often conflated when using ASCII encoded text, but are very different for any other encoding. In python3, these are represented by the str class for a sequence of characters, and by the bytes class for a sequence of bytes. It’s so much nicer to work with, but required manually separating out the earlier use case.
The whole 2 to 3 debacle was only because somebody thought that naming your string class 'unicode' doesn't sound nice enough or something.
The war continues to this day apparently with this Debian news. Somehow you can use fortran77 compilers without people getting angry but you can't use Python2
I had clean Python 2 codebases which had been Unicode-safe for years and had good test coverage, so porting to Python 3 was trivial and mostly automated. The projects which were hard were the sloppy ones which conflated bytes and Unicode, and the poor development culture which lead to that also tended to mean they had limited testing, orphaned dependencies, monkey patching, etc. which made the migration hard. I also found a fair number of cases where that process fixed other bugs which had been ignored for years.
edit: This talk he did from pycascades 2018 in particular I remember seeing and being a good retrospective: https://www.youtube.com/watch?v=Oiw23yfqQy8
The name originates from an internal script at Facebook called codemod. It had a public open source fork if you are curious.
https://github.com/facebookarchive/codemod
Python’s rise brought tensions there in multiple ways - business users started wanting longer-term support and tools for migrating large code bases, but relatively few would pay for them, and some maintainers had either drifted away or retired, or simply had less time available due to things like starting families or getting promoted. The 2 to 3 migration really highlighted the gap between commercial open source and hobbyist projects, and I’m sure now anyone would fundraise for support having seen how that worked out.
The other thing we’ve been seeing is a switch to faster releases industry wide and I think that showed some serious tech debt. Projects which have CI & decent tests had a much easier time shipping updates or having two release series for an while, and the 2 to 3 migration really highlighted that. Some of the griping was really showing that older projects often had the frictional disincentive of that tool. That’s still relevant because of the way security issues are forcing more frequent updates now.
A big part of 3 was Unicode strings. In 2 you could mark a string as Unicode with a u””. This would have been a great way to let libraries and code work with both 2 and 3. They banned this, but you had to still use b””. This was the attitude and just one example.
The list of stuff that got hard / slow was long. Migration path was unnecessarily difficult as was compatible code. Somewhere around 3.4 it was like a light switch flipped and they started being more reasonable. ASCII handling improved (yes, not all internet protocol stuff is Unicode), they began making it easier to target 2/3 etc. but it was horrible to start. I’ve thankfully forgotten some of the details :)
I feel this take is outright wrong. Python provided it's 2to3 tool[1] which took care of the bulk of the work required to port Python2 code to Python3. The only code that was not supported was eggregious errors in the code that worked by coincidence, such as handling bytes as strings and vice-versa. Porting old code to Python3 is a breeze that consisted of running 2to3, run tests with python3, and if anything breaked then just touch up the code to get it to work. I know it because I personally ported half a dozen projects throughout the years.
I'll go as far as to claim that most problems porting old python code to 3 were either upstream dependencies dragging their feet or internal human/organizational issues.
[1] https://docs.python.org/3/library/2to3.html
Your comment is a perfect illustration of the issues. Lots of user blaming. No actual solution. As I said, it did start to get massively better at some point. Instead of condescending lectures on org issues they for example began allowing u”” in 3, which did not mess up Unicode handling in 3
I don't see where GP made a claim that it would. Why would one expect code written for a newer version, using features that don't exist in an older version, to run under the older version?
Folks using u”” were already being careful with Unicode . Supposedly Unicode was important enough for 3 to default to it.
The cost benefit of then destroying code using u”” made no sense to me. It was unnecessary. This was followed by the oh just upgrade thing. I’d love to see the “quick” upgrade guidos employer did. My guess is a total lie that it was easy and it probably took years and VC type money.
It’s not about using new features, it’s about not unnecessarily breaking old or allowing for cross compatibility.
Except they broke the old way for existing features, required you to use the new way and in most cases you could manually write code so it would still work with both. Meanwhile the only tooling provided by the python 3 crowd screwed over anyone with an existing customer base stuck on python 2, which at least early on should have been a foreseeable problem. Not to forget that anyone stuck on python 2 was literally Satan and projects where outright "shamed" to drop python 2 support.
Many folks would have been ok updating to 3 compatible approaches if it didn’t blow up their 2 story. That’s what ultimately happened in lots of cases when it became more doable. The rip and replace everything at once was a weird goal
That's what "breaking changes" and "backwards incompatible" means.
What exactly is hard to grasp?
> So you ended up w a chicken and egg problem.
You really don't. Your code and your upstream dependencies need to be ported to python3. Once your dependencies are updated, all that's missing is you doing your job.
Python2 has been on the path to deprecation for how long? A decade?
You're fabricating problems where there are none.
> Your comment is a perfect illustration of the issues. Lots of user blaming.
There is no issue. When I had to port projects to Python3, I just ported them. No drama, no hangup. You're pretending there were problems where there were none, and you're throwing a tantrum when this fact is pointed out to you.
There is really no justification for this. It's high time people like you stop making up excuses and start to own up their misconceptions, misjudgements, and mistakes.
Some projects were rewritten from scratch in entirely different tech stacks in less time the likes of you complained they could not update their projects from python2 to python3.
What is it about python 3 that brings out this lecture stuff.
I would really like to see the inside of this “easy” change at even the bigger most well resourced places. Google / Facebook / Dropbox etc.
At small places where a program is working and programmer is gone they are not investing in ports.
Anyways - they did eventually get a clue. It’s much easier NOW to handle this, but it wasn’t at the start. Yes, they did put u”” back in. No it did not destroy the world.
This saga isn't completely over, over here: We still have code running python2 that we don't have the room to upgrade, and libraries that have to work with both python2 and python3. Meanwhile, people are pushing "code consistency" to the point where people outside our team are taking those libraries and running the "black" formatter on them - but "black" doesn't understand python2, so it removes the "u" prefix and breaks the library.
I think some people convinced themselves that using "" without the u"" was disenfranchising to people who don't want to use ASCII to represent their language. This transformed a practical argument over syntax into a moral crusade.
This isn't strawmanning - this is literally the case for a codebase I'm working with today.
Though I find it hard to believe that there are some useful dependencies that have not been ported and don't have better alternatives. Typically a "never ported" dependency is never ported because it is deprecated in favor of something better.
If you're consuming dependencies which were never updated in the past decade then you have more worrying problems to deal with than porting your code to python3.
Relax things a tiny bit in 3, back port to 2 via futures, get a library like six going. It could have been a ton easier.
In fairness by 3.5 maybe they’d realized this, but a lot of lecturing was done before. Initially recommendation was to upgrade all dependencies and programs to 3 and not try and make dependencies work w both
I'm glad that this is what we have for Python now, despite the pain it caused. We'll be using Python for another few decades probably, so we'll reap the benefits.
Nah, I'm good.
The other side of that coin is that the python maintainers went out of their way to push back the Python 2 EOL. They kept on pushing the date back and back again, as per the sunset page:
"We did not want to hurt the people using Python 2. So, in 2008, we announced that we would sunset Python 2 in 2015, and asked people to upgrade before then. Some did, but many did not. So, in 2014, we extended that sunset till 2020."
The drawn out demise of Python 2 was, frankly, painful.
I have no time for whinging snowflakes complaining 12 years (2020-2008) was not enough time to migrate their code to Python 3. Hell, even the original 7 years (2015-2008) should have been long enough for 99.999999% of the community.
They had to push back the EOL because they screwed up.
I'm old enough to have read this when it came out...and it changed my view on backwards compatability (from Joel Spolsky of Trello, FogBugz, and StackOverflow fame)
"Code doesn't rust": https://www.joelonsoftware.com/2000/04/06/things-you-should-...
If a company has production Python2 application / service (with hundreds of thousands of LOC), what business value does it bring to migrate it to Python3?
At that point, if you've got to make severe changes, folks might decide to use a language that doesn't impose breaking changes (and business cost) on them. YMMV.
Not just talking ASCII was one of the first ways.
Drift in 3rd party library support is another.
Security support of the language and libraries is a massive one.
Simply put as hard as you try to stand still the rest of the world is not going to.
I feel like a lot of the people who had this reaction were reacting to being told (correctly) that how they handled strings was broken. A big reason that projects had difficulty upgrading, especially difficulty using the automatic 2->3 upgrade tooling, was because of broken handling of unicode and broken handling of binary strings. I remember reading a quick assessment of a large open-source project that there were thousands of calls to a single string-handling function that would each need to be inspected by hand to determine whether it was correct usage that would be correctly translated by the 2->3 converter or buggy usage that needed to be fixed before running the 2->3 converter, with some people guessing a 50/50 split. The effort was going to be gargantuan, and though the ultimate cause and responsibility was poor design choices in Python 2, the immediate cause was that the code was wrong, and some people didn't appreciate hearing that.
No one really blamed programmers for getting strings wrong in Python 2. It was unreasonably hard and extremely rare for people to get it right, so there was not really any shame in Python 2 programs and libraries being pervasively broken in that way. But some people still felt it was smug and condescending to make reference to this elephant in the room. This put the Python maintainers in a bad position, because they were trying to explain the benefits of upgrading and also the costs and difficulties, both of which were intimately tied to the brokenness of current codebases.
As a user and spectator to the whole drama, therefore, I felt that some maintainers got overly defensive about issues that virtually every codebase suffered from. And some of them took a "so what" attitude towards bugs involving non-Western languages, saying that if they personally didn't care whether their code handled Chinese or Bengali filenames then it wasn't a bug to do random things with them, which sounded just as bad ten years ago as it does now.
Can you be very precise about what you mean by "broken" or "incorrect" code?
Python3 enforces the difference between encoded and unencoded strings. It forces you to deal correctly with unicode. If your code base was already handling unicode correctly it wasn't much hassle to migrate. If your code base was making a mess of unicode handling --like many at the time were-- you'd run into that headfirst.
Secondly, it seems to me the Python team didn't want to make it possible to write code that works correctly on both Python 2 and 3 simultaneously. They wanted you to write code that only works on Python 3 and tell every single Python 2 user to get lost. Essentially, they wanted to co-opt you as their agent of change; burn your own reputation by abandoning your userbase (or worse, start proselytising Py3 at them) or double your own workload to maintain two codebases. Who wants that?
It's fundamentally an attitude problem. A desire for there to be only one right way to do things, a change of mind of what the right way is, and kicking the chair out from anyone doing it the old right way; the only compromise being to kicking the chair very very slowly. In Java, you can still run code that uses Vector and Hashtable from 1996, they're not even deprecated. They're used by _very_ little code since generic List and Map APIs superceded them 2004, but they're still there.
The core team was swamped with other work around the migration and the community found their response to real world problems lacking. They did things like declaring Python 2 dead long before 3 was usable. Then they refused to fix SSL problems with Python 2, which everyone desperately needed, and went out of their way to oppose a community-led release to fix those problems.
It all ended well. There was another Python 2 release, and Python 3.4 came and fixed most problems people had. It was still slower, but performance picked up and by 3.7 it was just as good if not better. But it was a good ten years during which the response to real world issues could have been a lot better.
All in all, the migration could have happened much smoother. They could have learned from other languages. But ten years is not unreasonable for backwards incompatible changes for a major programming language. The idea that the process could be accelerated by neglecting the old version does not work.
In my personal opinion, the single thing that could have made the transition better for the community would have been the possibility to run Python 2 code in Python 3. It was discussed a lot before the transition and was deemed impossible because of conflicting data types. But it's not impossible, it's just (a lot of) work. It's another type conversion, and Python can already do those. The impossibility to link to old libraries made the conversion a flag day for most code bases, and those are really hard.