In my experience, companies haven't yet migrated from Python 2 for the same reason they haven't yet migrated from COBOL. It has nothing to do with Python 3's benefits or flaws.
It is horrible compromise of a bad situation. 99% of the time file name bytes are UTF-8 decodable. What do you do in the other case? You could return a different type, but then that makes dealing with it harder. You could make a new filesystem name type, but that would make simple APIs complicated. Using a new control character is probably the best one could do.
Arguing that python3’s str model doesn’t work well with POSIX’s “any bag of bytes can be a filename” model. Plus new rants about surrogateescape which the author has learned since publishing the last article.
The author sure has a penchant for flamebait titles.
No comment on the titles, but the issues are real (the linked page explains some of it decently [1]). I know I've found it excruciatingly difficult to write Python code that handles non-ASCII stdio correctly, especially one that might display on a terminal, especially a portable manner. Some of the compatibility issues are inherently hard problems in any language, but others are Python-related, and it didn't get better in Python 3.
> I've found it excruciatingly difficult to write Python code that handles non-ASCII stdio correctly, especially one that might display on a terminal, especially a portable manner, and it didn't get better in Python 3.
I've been doing just that in Python 3 for half a decade at least -- got a fairly popular open source CLI application that hasn't seen a Unicode complaint for years. It doesn't come for free on all possible configurations (yeah, every developer with a wide enough userbase has seen the dreaded "'ascii' codec can't encode characters in position ...: ordinal not in range" at some point) but it's definitely not "excruciatingly difficult". Bad things only happen sys.stdout.encoding isn't utf-8, which is rare on * nix systems -- fixed by setting PYTHONIOENCODING or setting locale to * .UTF-8. Encoding on windows is always a headache (not specific to Python at all) but somehow we seemed to have managed to steer clear too. Anyway, just check sys.stdout.encoding.
Meanwhile, this article is about problems that arise when you have garbage filenames that are neither Unicode nor whatever Microsoft's encoding; I wouldn't expect the average user to deal with files like that in day-to-day usage.
I think the issue comes from PHP magic with non-ASCII values. PHP tries to guess and never fails.
Python instead doesn't do this for you.
Anyway, some non-english speaker countries developers feel like programming language developers still have different encodings as second citizens like this was the sixties.
I hope this changes some time in the future although 10 years ago I already hoped for today to be that future.
Let me quote the sys.stdout/sys.stderr encoding rules:
> The character encoding is platform-dependent. Non-Windows platforms use the locale encoding (see locale.getpreferredencoding()).
> On Windows, UTF-8 is used for the console device. Non-character devices such as disk files and pipes use the system locale encoding (i.e. the ANSI codepage). Non-console character devices such as NUL (i.e. where isatty() returns True) use the value of the console input and output codepages at startup, respectively for stdin and stdout/stderr. This defaults to the system locale encoding if the process is not initially attached to a console.
> The special behaviour of the console can be overridden by setting the environment variable PYTHONLEGACYWINDOWSSTDIO before starting Python. In that case, the console codepages are used as for any other character device.
> Under all platforms, you can override the character encoding by setting the PYTHONIOENCODING environment variable before starting Python or by using the new -X utf8 command line option and PYTHONUTF8 environment variable. However, for the Windows console, this only applies when PYTHONLEGACYWINDOWSSTDIO is also set.
> Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess.
But on *nix it's basically based on the second part of your locale. If it's anything.UTF-8 (and your terminal talks UTF-8) there won't be a problem. If you set your locale to C or something.ASCII or whatever you can't really blame Python for trying to encode the output according to what you claim to prefer.
I fail to see how you can "guess" the terminal's expected encoding otherwise.
> Bad things only happen sys.stdout.encoding isn't utf-8
Yes, so your code will break when the caller or user sets it to something else.
> which is rare on * nix systems
"UNIX only" is not exactly what I meant by "portable"...
> It doesn't come for free but it's definitely not "excruciatingly difficult".
It's not when you're ignoring the difficult parts!
> Encoding on windows is always a headache (not specific to Python at all)
It's a headache in Windows, but in some cases encoding is worse in Python 3. Sadly I've never sat down to make a library of all the examples I come across, but here's one:
Try predicting what this shell script should print on each platform. (It's Bash/Batch cross-compatible, so once you tell me your prediction, try running in both.)
> Yes, so your code will break when the caller or user sets it to something else.
No, it will break when the encoding is set to something where the output text simply can't be encoded. There's simply no such thing as “” in the ASCII locale, or the C locale, so when you try to print those you get an encoding error; if the encoding is set to cp1252 it would be fine.
If you don't want to encode text according to user's preferred encoding, just encode as utf-8 regardless and write the bytes. Let me say this again: if you want to print text, print text; if you want to print a byte stream, print a byte stream. Python3 doesn't make this trivial but it's also not terribly hard.
But by printing utf-8 byte stream under all scenarios, you would be conveniently ignoring the fact that default console encoding on Windows is cp437, not cp65001, so non-ASCII parts of the utf-8 byte stream would be garbled on Windows consoles by default. (And I see garbled Unicode filenames from programs written in other languages all too often when I'm on a Windows console.) It's a damned if you do, damned if you don't situation. Interestingly, the default now is utf-8 on Windows consoles, unless PYTHONLEGACYWINDOWSSTDIO is set. They probably decided that printing (the occasional) garbage is better than correctness.
> Try predicting what this shell script should print on each platform.
Again, not surprising when you realize the default encoding on (en_US) Windows is cp1252. If you want to write a byte 0xdd, don't use '\xDD' which is U+00DD which of course is encoded differently in utf-8 (0xc3 0x9d) and cp1252 (0xdd).
PYTHONIOENCODING is an override env var you can set. cp1252 being the default on Windows is of course not Python-specific, it’s convention set by Microsoft. In fact, you do realize cp1252 aka Latin-1 was even the default encoding used by the HTML spec before the advent of HTML5 (postdates Python3)? Creators of Python3 certainly didn’t set this default to mess with you.
No, this is just insane Python behavior. There's clearly nothing in the Windows I/O path that's doing this, and if it's a "convention", it's not even one that Python 2 has followed!
Hell, even Python 2's behavior of raising an error is more sane than just encoding in cp1252 silently and expecting every developer to somehow know the resulting byte sequence is going to be different on each platform:
python3 -c "print(u'\u00DD')"
Obviously, I'm not the only one who thinks that whatever this (ancient?) so-called "convention" is, it's actively harmful enough to avoid in 2019.
And as if that's not enough, that's not even the end of it. Even literally writing a string to a file (with no console or stdio in between!!) ends up producing a completely different file depending on which platform the code is run on:
How is any programmer supposed to deal with this insanity? A program writes to a file, but produces files with completely different contents depending on the platform? Heaven help you if you try to interacting with a program in another language. At least with newlines, as painful as they are, most people know and recognize how to deal with them. But dealing with that mess is enough, and not a reason to make the situation even worse?! It's almost as if they took the CRLF issue and then went "Hmm, that's not fair to CR or LF, we need to do this to even more characters."
Somehow we have all this nonsense in the name of "fixing" strings from Python 2. Instead of finding a better convention or at least leaving the existing behavior alone, Python 3—which is intended to let people write cross-platform code!—actively embraced it and botched Python 2's safe behavior, making programs produce garbled output completely silently... and blaming it on the stupid developer for trying to write a broken cross-platform program without first getting a PhD on the history of Code Pages on Windows.
Interestingly, I had exactly the opposite situation: I always had codec problems when using Python 2, and they were all solved with Python 3.
Perhaps the main difference is the native language of the developer and his/her clients/customers: If you and your customers are native english speakers, encoding problems were dramatically less probable before because all letters in use were ASCII anyway - and the UTF-8 positions of ASCII characters stayed the same (e.g. 'A' is still 0x41)- so buggy encoding didn't tend to break things for them. So even older Windows versions encoding filenames in latin1 etc. were never a problem to begin with - something that was always a hazzle for me (natively German speaking, with umlauts etc.).
Python 3 forces UTF-8 everywhere, which solved oodles of quirks for everyone but natively english speaking people. And thankfully encoding bugs now tend to break things mostly on startup, not just if a customer uploads a funky file from an old OS, two months into deployment.
If only. My problems with Python 3 were due to things like crashing when piping stdout to a file because for some reason the Python-chosen encoding for that was ASCII-only.
Another was discovering that the Python 3 default encoding for text files on Windows was CP-1252, so it would crash outputting any non-English text to a file.
It's a little better now, but the defaults were awful when I first started using Python 3. I expected it to be Unicode by default. It was not.
Yes, I omitted that my Python 2 (mostly Django or bottle) ran on Linux with UTF-8. I suspect your piping case could be due to either the OS not using UTF-8 as locale or the application behind the pipe ignoring the OS' locale.
These encoding problems will probably stay with us for quite a while, regardless of the language we write our applications in.
Just to elaborate, on a fresh Windows 7 install with en_CA locale, this was the behaviour:
python3 -c 'print("é")'
é
python3 -c 'print("é")' > out.txt
(UnicodeEncodeError) 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128)
I felt that was unacceptable behaviour for my program, but it was beyond my ability to fix. Though, I think this has been addressed in more recent Python releases.
"Python does not cater to my favourite edge case" != fundamental problem
In the days before UTF-8-everywhere, file names with anything besides alphanumerics and safe symbols like dashes or underscores were always a problem. If you had special characters in your filenames, you were almost certain to run in to problems, since their meaning varied greatly depending on how the system was set up - codepages and whatnot.
But this is only a problem if you have such files, and unless you’ve kept files around for decades, you don’t. So young programmers can grow up never having problems with this. Everything will be UTF-8 and it’ll just work.
And as for broken old file names, who cares? Fix your file names and move on. There’s no reason that Python 3 should have workarounds for problems that were solved over a decade ago.
> And as for broken old file names, who cares? Fix your file names and move on.
what if you are trying to process "foreign"(as in, not created by you) filenames, trying to validate/conform with python? I mean its a great DoS vector, which is difficult to protect against with python.
it'll crash. Which the point the article is trying to make.
> unless you’ve kept files around for decades, you don’t.
Thats both untrue and not very helpful. Its perfectly possible to bump into files like this.
Then you can use the 'bytes' variant of calls. If you process arbitrary data, there's lots of things you need to think about. This is just one of them. And if you'd crash otherwise, you should have exception handling in place.
I understand it's a problem and it's not trivial to support this, but it's not difficult either.
> what if you are trying to process "foreign"(as in, not created by you) filenames, trying to validate/conform with python? I mean its a great DoS vector, which is difficult to protect against with python.
1. Have catch-all exception handling. Exception may not be your fault, exception-caused "crashing" whatever that means is entirely your fault.
The truth is that 99% of programmers can do everything they need to in ASCII and the other 1% are working on tools to handle Unicode itself. It’s a mistake and as soon as it goes the way of the <blink> tag the better. At least that tag was amusing for a short while...
Always felt like a strange argument to me. I grew up bilingual, neither of these languages was English. Never in my life has it occurred to me to name a file I created with something else than ascii characters.
Reminds of a guy talking about maintaining an employee directory for a large multinational. All the names had to be displayed and searchable in the correct language. And his instrumentation said that everyone was using ascii when searching the directory.
Me neither, but I don't write the applications only for myself :) Most normal people are not aware of these problems, e.g. because they only have used Windows for their whole life and never crossed OS boundaries. Then they never experience the encoding problems that teached us the hard way how to properly name files.
> Never in my life has it occurred to me to name a file I created with something else than ascii characters.
Yeah, now talk to someone whose native language doesn't use the Latin alphabet. In fact, just look at open source projects created by CJK natives and the unfortunate romanizations they often come up with.
48 comments
[ 4.9 ms ] story [ 98.7 ms ] threadDidn’t that story finish?
not just rant, its a specific, well reasoned bug report.
https://changelog.complete.org/archives/10053-the-incredible...
https://news.ycombinator.com/item?id=21606416
Arguing that python3’s str model doesn’t work well with POSIX’s “any bag of bytes can be a filename” model. Plus new rants about surrogateescape which the author has learned since publishing the last article.
The author sure has a penchant for flamebait titles.
[1] http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/
I've been doing just that in Python 3 for half a decade at least -- got a fairly popular open source CLI application that hasn't seen a Unicode complaint for years. It doesn't come for free on all possible configurations (yeah, every developer with a wide enough userbase has seen the dreaded "'ascii' codec can't encode characters in position ...: ordinal not in range" at some point) but it's definitely not "excruciatingly difficult". Bad things only happen sys.stdout.encoding isn't utf-8, which is rare on * nix systems -- fixed by setting PYTHONIOENCODING or setting locale to * .UTF-8. Encoding on windows is always a headache (not specific to Python at all) but somehow we seemed to have managed to steer clear too. Anyway, just check sys.stdout.encoding.
Meanwhile, this article is about problems that arise when you have garbage filenames that are neither Unicode nor whatever Microsoft's encoding; I wouldn't expect the average user to deal with files like that in day-to-day usage.
Python instead doesn't do this for you.
Anyway, some non-english speaker countries developers feel like programming language developers still have different encodings as second citizens like this was the sixties.
I hope this changes some time in the future although 10 years ago I already hoped for today to be that future.
> The character encoding is platform-dependent. Non-Windows platforms use the locale encoding (see locale.getpreferredencoding()).
> On Windows, UTF-8 is used for the console device. Non-character devices such as disk files and pipes use the system locale encoding (i.e. the ANSI codepage). Non-console character devices such as NUL (i.e. where isatty() returns True) use the value of the console input and output codepages at startup, respectively for stdin and stdout/stderr. This defaults to the system locale encoding if the process is not initially attached to a console.
> The special behaviour of the console can be overridden by setting the environment variable PYTHONLEGACYWINDOWSSTDIO before starting Python. In that case, the console codepages are used as for any other character device.
> Under all platforms, you can override the character encoding by setting the PYTHONIOENCODING environment variable before starting Python or by using the new -X utf8 command line option and PYTHONUTF8 environment variable. However, for the Windows console, this only applies when PYTHONLEGACYWINDOWSSTDIO is also set.
https://docs.python.org/3/library/sys.html#sys.stdout
locale.getpreferredencoding doc is vague
> Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess.
https://docs.python.org/3/library/locale.html#locale.getpref...
But on *nix it's basically based on the second part of your locale. If it's anything.UTF-8 (and your terminal talks UTF-8) there won't be a problem. If you set your locale to C or something.ASCII or whatever you can't really blame Python for trying to encode the output according to what you claim to prefer.
I fail to see how you can "guess" the terminal's expected encoding otherwise.
Yes, so your code will break when the caller or user sets it to something else.
> which is rare on * nix systems
"UNIX only" is not exactly what I meant by "portable"...
> It doesn't come for free but it's definitely not "excruciatingly difficult".
It's not when you're ignoring the difficult parts!
> Encoding on windows is always a headache (not specific to Python at all)
It's a headache in Windows, but in some cases encoding is worse in Python 3. Sadly I've never sat down to make a library of all the examples I come across, but here's one:
Try predicting what this shell script should print on each platform. (It's Bash/Batch cross-compatible, so once you tell me your prediction, try running in both.)
No, it will break when the encoding is set to something where the output text simply can't be encoded. There's simply no such thing as “” in the ASCII locale, or the C locale, so when you try to print those you get an encoding error; if the encoding is set to cp1252 it would be fine.
If you don't want to encode text according to user's preferred encoding, just encode as utf-8 regardless and write the bytes. Let me say this again: if you want to print text, print text; if you want to print a byte stream, print a byte stream. Python3 doesn't make this trivial but it's also not terribly hard.
But by printing utf-8 byte stream under all scenarios, you would be conveniently ignoring the fact that default console encoding on Windows is cp437, not cp65001, so non-ASCII parts of the utf-8 byte stream would be garbled on Windows consoles by default. (And I see garbled Unicode filenames from programs written in other languages all too often when I'm on a Windows console.) It's a damned if you do, damned if you don't situation. Interestingly, the default now is utf-8 on Windows consoles, unless PYTHONLEGACYWINDOWSSTDIO is set. They probably decided that printing (the occasional) garbage is better than correctness.
> Try predicting what this shell script should print on each platform.
Again, not surprising when you realize the default encoding on (en_US) Windows is cp1252. If you want to write a byte 0xdd, don't use '\xDD' which is U+00DD which of course is encoded differently in utf-8 (0xc3 0x9d) and cp1252 (0xdd).
By "the encoding" you're referring to PYTHONIOENCODING, right?
Somehow PYTHONIOENCODING is "not specific to Python at all"?
PHP doesn't do it this way:
Ruby doesn't do it this way either: Neither does MSYS2's Bash: Neither does Node.js: Hell, even Python 2's behavior of raising an error is more sane than just encoding in cp1252 silently and expecting every developer to somehow know the resulting byte sequence is going to be different on each platform: Obviously, I'm not the only one who thinks that whatever this (ancient?) so-called "convention" is, it's actively harmful enough to avoid in 2019.And as if that's not enough, that's not even the end of it. Even literally writing a string to a file (with no console or stdio in between!!) ends up producing a completely different file depending on which platform the code is run on:
How is any programmer supposed to deal with this insanity? A program writes to a file, but produces files with completely different contents depending on the platform? Heaven help you if you try to interacting with a program in another language. At least with newlines, as painful as they are, most people know and recognize how to deal with them. But dealing with that mess is enough, and not a reason to make the situation even worse?! It's almost as if they took the CRLF issue and then went "Hmm, that's not fair to CR or LF, we need to do this to even more characters."Somehow we have all this nonsense in the name of "fixing" strings from Python 2. Instead of finding a better convention or at least leaving the existing behavior alone, Python 3—which is intended to let people write cross-platform code!—actively embraced it and botched Python 2's safe behavior, making programs produce garbled output completely silently... and blaming it on the stupid developer for trying to write a broken cross-platform program without first getting a PhD on the history of Code Pages on Windows.
Perhaps the main difference is the native language of the developer and his/her clients/customers: If you and your customers are native english speakers, encoding problems were dramatically less probable before because all letters in use were ASCII anyway - and the UTF-8 positions of ASCII characters stayed the same (e.g. 'A' is still 0x41)- so buggy encoding didn't tend to break things for them. So even older Windows versions encoding filenames in latin1 etc. were never a problem to begin with - something that was always a hazzle for me (natively German speaking, with umlauts etc.).
Python 3 forces UTF-8 everywhere, which solved oodles of quirks for everyone but natively english speaking people. And thankfully encoding bugs now tend to break things mostly on startup, not just if a customer uploads a funky file from an old OS, two months into deployment.
If only. My problems with Python 3 were due to things like crashing when piping stdout to a file because for some reason the Python-chosen encoding for that was ASCII-only.
Another was discovering that the Python 3 default encoding for text files on Windows was CP-1252, so it would crash outputting any non-English text to a file.
It's a little better now, but the defaults were awful when I first started using Python 3. I expected it to be Unicode by default. It was not.
These encoding problems will probably stay with us for quite a while, regardless of the language we write our applications in.
The author is an individual. He is not bound to craft titles for the approval of HN...
Nit: isn't Python generally considered to be strongly, dynamically typed?
In the days before UTF-8-everywhere, file names with anything besides alphanumerics and safe symbols like dashes or underscores were always a problem. If you had special characters in your filenames, you were almost certain to run in to problems, since their meaning varied greatly depending on how the system was set up - codepages and whatnot.
But this is only a problem if you have such files, and unless you’ve kept files around for decades, you don’t. So young programmers can grow up never having problems with this. Everything will be UTF-8 and it’ll just work.
And as for broken old file names, who cares? Fix your file names and move on. There’s no reason that Python 3 should have workarounds for problems that were solved over a decade ago.
what if you are trying to process "foreign"(as in, not created by you) filenames, trying to validate/conform with python? I mean its a great DoS vector, which is difficult to protect against with python.
it'll crash. Which the point the article is trying to make.
> unless you’ve kept files around for decades, you don’t.
Thats both untrue and not very helpful. Its perfectly possible to bump into files like this.
Then you can use the 'bytes' variant of calls. If you process arbitrary data, there's lots of things you need to think about. This is just one of them. And if you'd crash otherwise, you should have exception handling in place.
I understand it's a problem and it's not trivial to support this, but it's not difficult either.
1. Have catch-all exception handling. Exception may not be your fault, exception-caused "crashing" whatever that means is entirely your fault.
2. Use os.fsdecode. https://docs.python.org/3/library/os.html#os.fsdecode
3. Don't process random untrusted filenames. Sanitize if it's some sort of HTTP upload, for instance.
Possible, sure, but highly unlikely. Unless you consider black-hat hackers, it’s pretty rare for modern software to be fed decades-old files.
Broken logic for a very common case != "Python does not cater to my favourite edge case"
>And as for broken old file names, who cares?
People with actual products / production code.
>Fix your file names and move on.
Is this satire?
This is the totally normal experience of every programmer from every country.
The only people pushing Unicode, ironically, are white native English speakers who think they’re saving the world.
What experience? Never naming a file in "something else than ascii characters"?
It might be true for programmers. Users, which 99.9% of them are not programmers, do it ALL THE TIME in any country in the world.
And why shouldn't they? Should they learn english just to name files, or use transliterations of their language names for the contents of the file?
>The only people pushing Unicode, ironically, are white native English speakers who think they’re saving the world.
Spoken like a person with no experience outside an English speaking country or developer echo bubble whatsoever.
Yeah, now talk to someone whose native language doesn't use the Latin alphabet. In fact, just look at open source projects created by CJK natives and the unfortunate romanizations they often come up with.
Gee I wonder why ...