I am wondering what would happen, if someone swapped out the glibc implementation? Would this uncover the deeper technical reason for the implementation? Would suddenly bugs occur elsewhere?
It probably just shows how old the glibc is and like most software entropy is increasing over time.
The comment block mentions locales so I'd think non-C locales and extended-ascii (though non-ascii is also possible) e.g. in ISO-8859-9, 0xDD is İ (uppercase dotted i) and 0xFD is ı (lowercase dotless i) which you would expect to be an alphabetic letter in the turkic locale.
Or à á é è í ì ò ó ù ú in pretty much anywhere in Europe other than the England (and also places like Greece, Serbia and Bulgaria which have a full alphabet of their own to handle in isalnum :)).
Musl assumes your locale is either C or UTF-8, glibc doesn't. Sure almost everything is UTF-8 these days and has been in the last 15 years, but claiming that the glibc code makes no sense is utterly condescending and I expected ddevault to know better than that.
I also wouldn't be surprised if those countries that have a non-Latin alphabet were still using single byte character encodings since UTF-8 would double the size of their text.
> Or à á é è í ì ò ó ù ú in pretty much anywhere in Europe country other than the England (and also places like Greece, Serbia and Bulgaria which have a full alphabet of their own to handle in isalnum :)).
That's more complicated because depending on the language these may or may not be considered separate letter e.g. the french alphabet has 26 letters: while required by the grammar, diacritics and ligatures are "extras". This means isalnum wouldn't necessarily return true for, say, ç in the french locale (I've no idea what it actually does).
Turkish however does have 29 letters, and the dotted / dotless i is very much part of the alphabet (so are ç, ş, ğ, ö and ü).
> Musl assumes your locale is either C or UTF-8
UTF-8 is an encoding, not a locale. And Musl just plain assumes the C locale, it clearly has no support whatsover for anything else.
Which, for what that's worth, is a respectable choice: posix locales are really really bad, so not bothering with them is not necessarily a negative, but at the same time dinging glibc for supporting locales or old platform is not very honest.
> isalnum wouldn't necessarily return true for, say, ç in the french locale (I've no idea what it actually does).
It returns true, otherwise "advance to the next word" or "grep -w" would be utterly broken. ("grep -w ça" would not match anything, and in fact it probably won't match anything under musl).
Other European languages also treat characters with diacritics as separate letters, for example the Czech alphabet has 42 letters (one of which is "ch" which doesn't have its own Unicode character as far as I remember; if it did there would be more fun to be had with normalization and title case).
> And Musl just plain assumes the C locale, it clearly has no support whatsover for anything else.
Yeah what I meant is that if you are reading UTF-8 you won't be passing chars in the 128-255 range to the ctype functions. But musl doesn't implement Unicode wchar_t either.
> Yeah what I meant is that if you are reading UTF-8 you won't be passing chars in the 128-255 range to the ctype functions. But musl doesn't implement Unicode wchar_t either.
The trap is that ctype functions take int, that's exactly the mistake TFAA made: they passed in an integer assuming isalnum would work with codepoints (which doesn't actually make sense since int is only required to be 16 bits).
You're right I looked up only iswalnum and iswdigit, and trusted masklinn on the rest. :-) So I remembered right, musl assumes an encoding of Unicode where 128-255 is invalid.
I guess the glibc complexity with a LUT is to support non-ascii character sets? It seems quite silly to compare a general solution to a hard coded special case.
Any big standard lib I laid my eyes on (gcc, clang, MS headers) seems to be a mess. I too wonder if the complexity is really necessary, but I guess so? Macros in my experience is ofent to fool the compiler to generate better code. Maybe that is a legacy reason?
99% of the time the answer is speed. It's just way faster to evaluate and build all of this at compile time than at runtime.
I understand both sides of the argument here - on one hand you have fast and unreadable code and on the other hand slower and more readable code. The problem here is 1. the leaky abstraction of characters and code points and 2. loss of type safety due to C's weak typing rules, which is a bit lost in the "hurr durr messy code."
Windows.h is still full of obsolete ancient macros that can’t be removed for compatibility. Things like references to near and far pointers (still!) and how every Win32 function with string params is a macro for the real A/W-suffixed functions, and all the A-suffixed functions now intentionally crash on post-98 systems.
- Focus on performance, thus playing "weird" tricks with macros, lookup tables etc.
- especially in headers: protect against users doing crazy sh*t
- standards sometimes change or see extensions
- certainly more :)
First two items can be re-evaluated form time to time, but especially GNU often aims to run "anywhere"
The third item comes from the fact that users can create macros for many things and only some names (leading _ and capital letter or __) are reserved for standards or C++ allowing to overload the comma operator, which can have weird effects. Thus to be conformant to the standards some ugliness is required.
The fourth point combined with different compatibility requirements requires tricks to switch between different implementations based on compiler flags/macros (for instance `#define _GNU_SOURCE` or `#define _POSIX_C_SOURCE=...`)
If that were the case, wouldn't a developer either be using uchar types or wchar_t which have their own specific variants of character related functions?
But what the duck? How does that make sense for Chinese?
This is bullshit. Great, now isalnum returns true for ä if I'm using de_DE.latin1 or whatever, but is still nonsense for any non-western language. And the price we pay for that is a messed up pile of garbage that nobody understands and can segfault.
It's not just Western languages. It's how text was encoded until the 90s and early 2000s for pretty much every language except Chinese, Japanese and Korean (maybe Thai, I am not sure). That's the "famous" CJK acronym for languages that really did require double byte character sets. For example Russian, Greek, Hebrew, and Indic languages all have alphabets that will fit very easily in 128 codepoints (Vietnamese too, which is Latin but with a large number of diacritics).
Not sure, I still see mail that is not UTF-8 encoded, probably some clients are "optimizing" if they only see an occasional accented letter. And for non-Latin languages UTF-8 is pretty much a two-fold increase in size so there may still be usage of single-byte character sets; legacy DBCS aren't entirely dead in China and Japan either.
The default locale for pretty much any language on most Linux distros is utf8 at least. Legacy systems exist, as you point out especially in those CJK countries, I guess the transition to Unicode was much more annoying there. But incidentally those old MBCS charsets are those where isalnum et al never made sense in the first place.
I'd argue the usefulness of all that glibc code is very close to zero. Sure, the glibc folks can pat themselves on the back for covering the POSIX spec so well but I really prefer the Linux approach here; follow POSIX where it makes sense but omit the insanity.
They would pat themselves on the back for writing useful code when it was useful. They didn't write it in 2010, it dates back to the 80s. The comparison to Linux also makes little sense; talking about locales, Linux still supports Microsoft code pages for the FAT file system.
Then it is time to consider removing this code. If you need it use an old glibc.
Your comparison to Linux makes no sense either. It's not that the code pages themselves are the problem, you can use them for conversion and whatnot. But pretending a collection of functions that is unable to do anything meaningful in the present day and crashes for extra points is fine, because it was written long ago just isn't. The musl solution is correct. If you need to handle strings in a locale-aware manner, use a dedicated lib. Better yet don't use C.
With musl, an Italian person wouldn't be able to use "grep -w ciò" and a Greek person wouldn't be able to use "grep -w" on Greek text at all. So okay it's not a big deal, a Thai person cannot use "grep -w" on Thai text either, but the musl solution is correct for specific use cases where locale doesn't matter.
And the crash is a problem with C but it's not a problem with the function. There's a different function, wisalnum, that doesn't crash with out of bounds values, but it also doesn't do Unicode on musl.
grep -w is pretty useless either way. Ok I might sound like I just try to brush off the issue here but it's not intuitive anyways that it needs to know the locale to properly match in the first place. The Italian person might be caught off guard when trying to find a Swedish word in a file, because you need to remember to switch the locale first. And if grep actually uses isalnum for -w then the Italian and Greek and Thai people will be out of luck either way since they all use utf8 today and ò is two bytes. Good look if you work with a multi-language file and have to constantly switch. But that's not even a problem specifically with POSIX or glibc but language in general; there are languages that don't have any separators between words so you already lost.
The crash is a problem with the function, specifically with glibc. You might even put into question why POSIX defines out of range values as UB instead of requiring it to return false but that's yet another can of worms we better not open here. I fully blame not doing any range checks and doing an oob array access on glibc. UB doesn't force you to create an implementation that crash and burns... Returning false would be great. Maybe abort if you're anal. Have it return something random if you're concerned about speed. But don't freaking crash! The libc should work with the dev, not against them.
> But what the duck? How does that make sense for Chinese?
It doesn't.
> This is bullshit.
Welcome to POSIX locales.
> Great, now isalnum returns true for ä if I'm using de_DE.latin1 or whatever, but is still nonsense for any non-western language.
Technically it's already nonsense for western languages as ISO-8859 is long outdated, and any codepage other than ISO-8859-1 will not fit inside a uchar when decoded from UTF-8. Also there were non-western languages which fit in 8-bit encodings for which isalnum would work fine.
> And the price we pay for that is a messed up pile of garbage that nobody understands and can segfault.
Welcome to POSIX locales. And also C, where not reading and understanding the implication of every word in the specification means you're bad and therefore deserve everything you get. In this case, POSIX clearly specifies that input values outside of EOF or "unsigned char" is UB.
>So the fix is obvious at this point. Okay, fine, my bad. My code is wrong. I apparently cannot just hand a UCS-32 codepoint to isalnum and expect it to tell me if it’s between 0x30-0x39, 0x41-0x5A, or 0x61-0x7A.
If that's what you want, just go ahead and write that.
Let's be specific, then, and exclude any software that is built with GNU make? First thing that goes is the Linux kernel (which can only be built with GNU make). As far as I know, Alpine still relies on the Linux kernel, so that's out.
BSD utils balk if you get the order of command-line switches "wrong" (an attribute shared with the Git CLI). The time wasted dealing with that alone is worth installing the GNU utils.
Speaking as someone who wrote a Standard C library like this many years ago, I can confirm. All of the work is done inside setlocale(), switching (or constructing) tables. The character type functions devolve to a single table lookup.
For those interested: The technique is used in P. J. Plauger's 1992 The Standard C Library and has been around a long time.
Here's the thing: isalnum has never been a bottleneck in my code. The sum amount of runtime that glibc's optimizations have saved my programs is almost certainly significantly less than the amount of time I had to spend untangling glibc's bullshit when it broke. Isn't "programmer time is more expensive than computer time" the lie that we've built this whole mess on anyway?
And even if you want to use a LUT, other implementations have incorporated a LUT without this segfault issue, for instance plan 9:
Absolutely, isalnum et al are probably never going to be a bottleneck. I was curious whether it even made a difference at all, after all, the naive implementation doesn't really do a lot.
And I fully agree with you that this glibc code is awful. As evidenced by the Plan 9 implementation, table lookups can be done simple as well, and probably perform just the same.
At first glance, the plan9 code looks broken to me: EOF is commonly defined as -1, which gets converted to unsigned char (ie 0xFF) before lookup. If your encoding uses the full 8-bit codespace, that's the wrong thing to do...
That’s probably because they do not care about non-US encodings.
I dearly wish entire world would run on us-ascii, it would make software so much easier... but unfortunately there are whole countries which disagree with me.
I absolutely loathe and despise the arrogant 'your code must be bullshit' attitude that runs throughout this article.
When you encounter other peoples' code and it looks a bit complicated, and not the 'obvious and simple' thing, your first thought should never be 'they wrote bullshit code', but instead 'what have I missed? What did I forget to consider for this function?'
I don't know anything about the glibc authors, or their code style, but if you go through your life assuming incompetence if someone wrote a function differently to your initial thoughts, then you are an arrogant prick.
I don't have a problem with strong language. Drew easily comes across as arrogant sometimes, and I don't agree with everything he says.
But if there's something I'm sick of it's that stupid "well the spec says it's UB so let's just write code that FUCKS THE DEV as hard as possible." attitude in certain circles regarding a one letter programming language.
The code in question is a classic array lookup with missing bounds check. If you write code like that and then blame people for using it wrong because the spec says so, you're just as arrogant, just in a different way.
I actually loved it. I found it very entertaining.
Tangent: surprises me how often comment people lose their shit when you utter the slightest word of criticism about some pet {language,code, tech}, but are absolutely fine, supportive and enabling when someone verbally eviscerates the living shit out of another human based on what they said, or their opinion.
How is attacking a non human idea and no one in particular bad, but attacking a specific person viciously ok? I don't get that and I never did.
Even though I agree with what you are saying, it is also worth looking at the issue through the eyes of the author.
The function that they were calling can be summarized as, "check to see if a value is in a set," only to receive an initially inexplicable segfault. They had a clear example of what that could look like, someone else's code from a common library, yet spent a considerable amount of time untangling the code in the library where the bug popped up.
It is wrong to assume incompetence on the part of the glibc authors, particularly when the comparison library has a much more limited scope and can afford to avoid "bullshit code", but the source of the frustration is clear. I would hardly define them as an arrogant prick because of that.
The Glibc code shown is really unforgiving, so I don't mind the author's humor.
My opinion is that there is only one excuse for writing so unreadable code for such simple functionality (even taking account the legacy): that would be to use a language completely unfit for the task. That should not be the case here ... or is it? Why so many macros?
glibc is and quite probably has to be written in C (the clue is in the name), and it has to work on a great many platforms and architectures, serving an absolutely huge amount of software. Is there any other software linked with and providing vital infrastructure to tens of thousands of other programs? (Maybe Windows MSVCRT.)
I've written tons and tons of software for Linux, and I believe only once have I seen a bug in glibc[1].
65 comments
[ 285 ms ] story [ 2184 ms ] threadIt probably just shows how old the glibc is and like most software entropy is increasing over time.
Musl assumes your locale is either C or UTF-8, glibc doesn't. Sure almost everything is UTF-8 these days and has been in the last 15 years, but claiming that the glibc code makes no sense is utterly condescending and I expected ddevault to know better than that.
I also wouldn't be surprised if those countries that have a non-Latin alphabet were still using single byte character encodings since UTF-8 would double the size of their text.
That's more complicated because depending on the language these may or may not be considered separate letter e.g. the french alphabet has 26 letters: while required by the grammar, diacritics and ligatures are "extras". This means isalnum wouldn't necessarily return true for, say, ç in the french locale (I've no idea what it actually does).
Turkish however does have 29 letters, and the dotted / dotless i is very much part of the alphabet (so are ç, ş, ğ, ö and ü).
> Musl assumes your locale is either C or UTF-8
UTF-8 is an encoding, not a locale. And Musl just plain assumes the C locale, it clearly has no support whatsover for anything else.
Which, for what that's worth, is a respectable choice: posix locales are really really bad, so not bothering with them is not necessarily a negative, but at the same time dinging glibc for supporting locales or old platform is not very honest.
It returns true, otherwise "advance to the next word" or "grep -w" would be utterly broken. ("grep -w ça" would not match anything, and in fact it probably won't match anything under musl).
Other European languages also treat characters with diacritics as separate letters, for example the Czech alphabet has 42 letters (one of which is "ch" which doesn't have its own Unicode character as far as I remember; if it did there would be more fun to be had with normalization and title case).
> And Musl just plain assumes the C locale, it clearly has no support whatsover for anything else.
Yeah what I meant is that if you are reading UTF-8 you won't be passing chars in the 128-255 range to the ctype functions. But musl doesn't implement Unicode wchar_t either.
The trap is that ctype functions take int, that's exactly the mistake TFAA made: they passed in an integer assuming isalnum would work with codepoints (which doesn't actually make sense since int is only required to be 16 bits).
It looks like it does. iswalpha uses some kind of table to return its response. Is that implementation somehow too simplistic?
I understand both sides of the argument here - on one hand you have fast and unreadable code and on the other hand slower and more readable code. The problem here is 1. the leaky abstraction of characters and code points and 2. loss of type safety due to C's weak typing rules, which is a bit lost in the "hurr durr messy code."
Do you have a source for this claim?
I find it hard to believe that Microsoft would drop support for ANSI applications.
- Support for weird compilers
- Focus on performance, thus playing "weird" tricks with macros, lookup tables etc.
- especially in headers: protect against users doing crazy sh*t
- standards sometimes change or see extensions
- certainly more :)
First two items can be re-evaluated form time to time, but especially GNU often aims to run "anywhere"
The third item comes from the fact that users can create macros for many things and only some names (leading _ and capital letter or __) are reserved for standards or C++ allowing to overload the comma operator, which can have weird effects. Thus to be conformant to the standards some ugliness is required.
The fourth point combined with different compatibility requirements requires tricks to switch between different implementations based on compiler flags/macros (for instance `#define _GNU_SOURCE` or `#define _POSIX_C_SOURCE=...`)
> an unsigned char or […] EOF
so isalnum is "their own specific variants of character related functions" for uchar, and locale-aware.
This is bullshit. Great, now isalnum returns true for ä if I'm using de_DE.latin1 or whatever, but is still nonsense for any non-western language. And the price we pay for that is a messed up pile of garbage that nobody understands and can segfault.
But all the platforms that use glibc have moved to utf8 over a decade ago. So even for western languages that code doesn't make sense anymore.
I'd argue the usefulness of all that glibc code is very close to zero. Sure, the glibc folks can pat themselves on the back for covering the POSIX spec so well but I really prefer the Linux approach here; follow POSIX where it makes sense but omit the insanity.
Your comparison to Linux makes no sense either. It's not that the code pages themselves are the problem, you can use them for conversion and whatnot. But pretending a collection of functions that is unable to do anything meaningful in the present day and crashes for extra points is fine, because it was written long ago just isn't. The musl solution is correct. If you need to handle strings in a locale-aware manner, use a dedicated lib. Better yet don't use C.
And the crash is a problem with C but it's not a problem with the function. There's a different function, wisalnum, that doesn't crash with out of bounds values, but it also doesn't do Unicode on musl.
The crash is a problem with the function, specifically with glibc. You might even put into question why POSIX defines out of range values as UB instead of requiring it to return false but that's yet another can of worms we better not open here. I fully blame not doing any range checks and doing an oob array access on glibc. UB doesn't force you to create an implementation that crash and burns... Returning false would be great. Maybe abort if you're anal. Have it return something random if you're concerned about speed. But don't freaking crash! The libc should work with the dev, not against them.
It doesn't.
> This is bullshit.
Welcome to POSIX locales.
> Great, now isalnum returns true for ä if I'm using de_DE.latin1 or whatever, but is still nonsense for any non-western language.
Technically it's already nonsense for western languages as ISO-8859 is long outdated, and any codepage other than ISO-8859-1 will not fit inside a uchar when decoded from UTF-8. Also there were non-western languages which fit in 8-bit encodings for which isalnum would work fine.
> And the price we pay for that is a messed up pile of garbage that nobody understands and can segfault.
Welcome to POSIX locales. And also C, where not reading and understanding the implication of every word in the specification means you're bad and therefore deserve everything you get. In this case, POSIX clearly specifies that input values outside of EOF or "unsigned char" is UB.
>So the fix is obvious at this point. Okay, fine, my bad. My code is wrong. I apparently cannot just hand a UCS-32 codepoint to isalnum and expect it to tell me if it’s between 0x30-0x39, 0x41-0x5A, or 0x61-0x7A.
If that's what you want, just go ahead and write that.
The musl isalnum code is completely locale-unaware. Which you might argue is a good thing as POSIX locales are awful, but still.
The answer is a pretty big yes. Glibc's isalnum is about 6-7 times faster than musl's.
Benchmark code: https://paste.sr.ht/~minus/18b44cfe58789bc1fb69494130e859a11...
For those interested: The technique is used in P. J. Plauger's 1992 The Standard C Library and has been around a long time.
And even if you want to use a LUT, other implementations have incorporated a LUT without this segfault issue, for instance plan 9:
https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/...
https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/...
And I fully agree with you that this glibc code is awful. As evidenced by the Plan 9 implementation, table lookups can be done simple as well, and probably perform just the same.
When you encounter other peoples' code and it looks a bit complicated, and not the 'obvious and simple' thing, your first thought should never be 'they wrote bullshit code', but instead 'what have I missed? What did I forget to consider for this function?'
I don't know anything about the glibc authors, or their code style, but if you go through your life assuming incompetence if someone wrote a function differently to your initial thoughts, then you are an arrogant prick.
But if there's something I'm sick of it's that stupid "well the spec says it's UB so let's just write code that FUCKS THE DEV as hard as possible." attitude in certain circles regarding a one letter programming language.
The code in question is a classic array lookup with missing bounds check. If you write code like that and then blame people for using it wrong because the spec says so, you're just as arrogant, just in a different way.
Tangent: surprises me how often comment people lose their shit when you utter the slightest word of criticism about some pet {language,code, tech}, but are absolutely fine, supportive and enabling when someone verbally eviscerates the living shit out of another human based on what they said, or their opinion.
How is attacking a non human idea and no one in particular bad, but attacking a specific person viciously ok? I don't get that and I never did.
Well it's not an absolute, is it? There's context. Some specific people deserve to be attacked.
The function that they were calling can be summarized as, "check to see if a value is in a set," only to receive an initially inexplicable segfault. They had a clear example of what that could look like, someone else's code from a common library, yet spent a considerable amount of time untangling the code in the library where the bug popped up.
It is wrong to assume incompetence on the part of the glibc authors, particularly when the comparison library has a much more limited scope and can afford to avoid "bullshit code", but the source of the frustration is clear. I would hardly define them as an arrogant prick because of that.
My opinion is that there is only one excuse for writing so unreadable code for such simple functionality (even taking account the legacy): that would be to use a language completely unfit for the task. That should not be the case here ... or is it? Why so many macros?
I've written tons and tons of software for Linux, and I believe only once have I seen a bug in glibc[1].
[1] And it was both a horrible bug, and also an extremely weird corner case: https://bugzilla.redhat.com/show_bug.cgi?id=563103#c8