It's not that it's hard, it's just people don't go out of their way to escape UTF-8 glyphs into ASCII when dealing with exotic glyphs in a text editor. It's more mundane and tedious, but not 'hard'.
Try working with raw UTF-8 in JS and find yourself in a world of pain. Mathias Bynens talks about these gotchas here:
Favourite unicode fact: properly rendering unicode requires understanding of the current geopolitical situation (Depending on whom you accept as a country and whom you do not, two country-code-letters may or may not render as a flag. This changes sometimes in today's world.). https://esham.io/2014/06/unicode-flags
The font designer has to include a flag for each supported country. Often a given font is missing lots of flags for reasons that have nothing to do with whether the designer recognizes a given country or not, just a question of priorities; perhaps only 100 out of 200 flags are supported.
Imho, unicode should stay out of politics. Country flags, vaccine syringes and pregnant men should have nothing to do with how computers handle text and writing systems.
If it were that easy - sadly everything that's in any way related to the way we communicate and we relate to the world. Just look at the kerfuffle about skin tones... Everything is political if you are looking for a reason to fight.
Language is a very sensitive topic - in Central Asian countries, using Latin, Cyrillic or Perso-Arabic script for instance has very strong political connotations, same in the Balkans. The world is just like that
From what I recall the problem was on some devices it was rendered as a “sci-fi zapper” or squirtgun and on others it was a fairly realistic depiction of a gun. Leading to some misunderstandings…
There is no way to avoid it. It is very obvious that deciding whether "vaccine syringes" are political (and therefore excluded) or not is itself a political decision.
There's a certain kind of extremist who claims that their contentious positions aren't political, but the fact that there's an argument, and that you can point to mainstream coverage of it, strongly suggests that they're full of shit and everyone can see it.
What a strange position. The fact that an argument exists just shows that some people want to argue. Anyone can start an argument about anything. It’s hardly a good base to make a decision.
The good news is that Unicode has stayed out of politics. Such things you're complaining about are graphemes comprised of multiple combined code points, and it's up to an environment to support them or not (Android and iOS infamously do this in China: text won't render the same within and without China.)
This is actually a genius decision on their part! It's telling that for Han unification they stomped all over Japanese/Korean/Vietnamese glyphs but still kept the simplified and traditional forms separate.
No kidding, you have not lived until you try and explain UTF-8 to people who only believes in what they called "doublebyte".
You think they get it, but surprise happens when a database load fails when loading Chinese Character "string" into a field sized calculated based upon 2 bytes per character.
Thank god for emojis! Those people would say, "No one in our org would use chinese" and refuse to fix things... but now I just point them to latest message from upper management which contain emoji or two.
(And emoji are such a fine example - once they ate on the table, you need support for combining characters, characters outside of BMP, ligatures.. a large part of Unicode spec)
It's a very strange series of events that led us here, with various stages of 'oh no' and 'oh yes' events happening.
Like Apple trying to region-lock emojis, or whoever decided merging CJK together was a good idea, or that plan9 dev who scribbled out UTF8 on a napkin.
It's terrible, and we IMHO owe that to some introductory university courses to Java (plus some Win32 veterans). I got very close to being rejected by a professor that was obstinately convinced that Unicode "characters" were 2 bytes because it drunk the Kool Aid in the '90s about Java's `char` type representing a Unicode character. Ugh. I still get angry by thinking back at that sometimes
I can relate, I remember a teacher stating "you never have to worry about the amount of memory". This was in the late 90s, I then asked "So I can load a 20 gig data file into memory", he said yes.
If you found this essay interesting, you owe it to yourself to check out this super entertaining talk "Plain Text"[0] from NDC 2022 by Dylan Beattie. Rabbit hole warning: This video caused me to lose an entire Sunday watching Dylan's talks on YouTube, which are uniformly awesome.
I also really enjoy Dylan Beattie's work. For those with some spare time, who might like to see a true "rockstar" programmer, you may like "The Art of Code"[0].
Another good talk! I also really loved "Email vs Capitalism, or, Why We Can't Have Nice Things"[0] which has one of the best audience participation gimmicks I have ever seen in a talk.
This is exactly what I came here to post, too. I believe he's unusually prolific because he's part of the organisation of these conferences - but his delivery pays off regardless. I actually sent exactly the same video to a colleague this a few days ago.
The logical next step here is to realize that if you want to be truly internationalized, pretty much every single method of the string class in your favorite language is an antipattern and should be used with extreme caution. Seriously!
There were many discussions on how to handle strings in the forums too.
Remarkably, it is not possible to access "abc"[1] as it has unpredictable performance; instead one has to build the index and making this should make the developer realize the operation is costly.
All in all most beginners in Swift hate working with strings because it’s not intuitive at first, but to be fair in almost all languages strings handling is not done properly.
Oh, interesting! The fact that "abc"[1] doesn't work is a great sign, since my contention is that "the character at index i" is not a well-defined concept.
Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results.
I'd take "we don't do unicode at all" or "we only support BMP" or "we don't support composite characters" any day over pretend-support (but then inevitably breaking when the program wasn't tested with anything non-ASCII)
(ninjaedit: to see how prevalent it is, even gigantic message apps such as discord make this mistake. There are users on discord who you can't add as friends because the friend input field is limited to 32.... something - probably bytes, yet elsewhere the program allows the name to be taken. This is easy to do with combining characters)
I was tempted to include unicode when I enhanced the URL/Domain generating strategies in Hypothesis to correctly generate from the set of all valid TLDs a few years ago! But I decided against it since technically the canonical storage form doesn't contain the unicode characters as it should have had Punycode (https://en.wikipedia.org/wiki/Punycode ) conversion done by the "input" system be it a URL bar of your browser or whatever input handling is in play. Assuming all domain processors are able to correctly Punycode was a bit too much, so I kept it to the set of basic valid characters without needing Punycode conversion, and if someone wanted to fuzz test fully they could just add a layer that generates from full unicode.
Why are people so interested in ‘lengths’ of strings?
The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated.
𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes of UTF8 and as wide as about 9 Latin characters. (an England flag emoji) is about a character wide, but takes 28 bytes to store in UTF8.
The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?
There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle.
Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker.
The rules for grapheme clusters are well-defined and documented. I’ve only implemented a forward iterator on graphemes in the finl_unicode rust crate, but backwards would not be that difficult either. If I’m looking at a byte stream, I have to first make sure that I’m not in the middle of a multi-byte sequence (easily enough accomplished), you need to do some forwards and backwards iteration to determine where the grapheme boundaries are, and likely you will want a library to do this for you, but it’s something that should take a reasonably skilled programmer in about a day or two.
Well don't ask me, ask the service operators who limit their fields to X bytes!:D Whatever their reasoning, if they want to limit strings to a certain length, they should make it consistent so the user won't run into problems where it's accepted in one place then it runs into an error or another part of the service refuses to accept it.
Saying it's the ONLY thing anyone should care about is bound to over-generalize but I do find myself agreeing the vast majority of cases are really trying to guarantee something about byte or pixel dimensions. I think the reason we often end up defaulting to string length is, for most western text, it was often much easier to deal with and Emoji is, somewhat intentionally, a key driver in breaking that assumption even for those that don't care about supporting Cuneiform or have any idea or plans on trying to support other languages from the start.
Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how people think of their characters.
Is there a real world case? I'd be excited to learn it.
What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)
There was probably an existing character set that had precomposed characters and Unicode always includes existing character sets for round trip idempotency.
Yeah, Chinese characters radicals are not an alphabet per se, while Hangul has separate, standalone letters. It's literally a matter of composing them in a fixed way, compared to Chinese characters which tend to go bananas on how many thousands of ways they can be composed.
Exactly this. When Unicode 1.0 came out, there was no standardized way to indicate that f i should compose into fi, let alone that ᄋ ㅡ ᄀ should compose into 윽. IIRC, composing of East Asian scripts into han or hangul happened at the firmware level, not at the app or OS level. If you didn’t have the hardware built into your keyboard, you simply could not type anything in East Asian scripts. A lot of the inconsistencies in Unicode (like the difference in how vowels are handled in Thai vs Indic scripts) come down to how the original 8-bit encodings worked pre-Unicode.
Not especially. I was around for a lot of it (I shared an office with one of the contributors to Unicode 1.0 back in the late 80s and used to have a massive binder with every ECMA code page in it (which included a lot of non-European codes). The other thing about Han unification that people don’t realize is that there were five different East Asian 16-bit encodings (the original draft of ISO-10646 had a 32-bit encoding which included all of them). I worked on a project in the early 90s where we needed multiple-script capabilities (Roman, Cyrillic and Japanese) and ended up using JIS because it was a well-defined encoding that we could buy fonts that employed it (although we still had to contract with Bitstream to get a version of their font that was split into 8-bit segments for our software). There was JIS (Japanese), the Korean code (which included both Jamo and pre-composed Hangul syllables) and not one, not two, but three, count ’em, three different Chinese encodings: Mainland China, Hong Kong and Taiwan. Since the original goal of Unicode was to try to be a 16-bit fixed-width encoding (as opposed to the competing 32-bit fixed-width encoding of ISO-10646), there was really no choice but to do Han Unification. Even so, the five 16-bit encodings still linger.
Your app, unless it's a text editor or UI toolkit, should not care.
The easiest way to support Unicode is to avoid supportimg it. Leave text editing to UI widgets. Leave truncation to web browsers. Avoid fancy marquee effect on the terminal. Your program may look less fancy but it will automatically support unicode, even future extensions.
Unless you deal with very old data, just use UTF-8 [0]. Mac terminal uses UTF-8 by default, most Linuxes use UTF-8 by default, and if you are stuck with Windows, there are libraries which let you use UTF-8 everywhere.
And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just hardcode two common encodings (like 8859-1 and 8859-15)... I used to deal with encodings a lot, and the only time I actually had data loss from encoding problems was from misguided apps which assume 8859-1 (and tried to convert to ASCII by stripping accents). Please either assume UTF-8 or assume nothing and keep data as-is.
And that's it, _most_ software does not need to know more. Let's look at some examples:
- For file archiver, assume utf-8 everywhere and don't worry about encoding at all (unless you planning to deal with very old archives, in this case you'd need encoding support. But for something made from scratch, don't bother.) _Especially_ don't add any encoding support for "extract to console" function -- if I need a different encoding I can pipe into iconv myself, thank you very much.
- For API client, you don't need to worry about encoding either.. A very old HTTP server might return data in something other than utf-8 but (1) your HTTP library likely handles it already (2) how many such servers are left, anyway?
- For XML analyzer, use proper XML library, they will handle encodings for you.
- For web dashboard or a database, keep everything in utf-8 and things will just work.
- For ETL-like applications, many modern data sources are already in UTF-8.. and if they are not, convert everything to utf-8 ASAP.
In other words: while it's useful to know that encodings exist, unless you are working with very old data or legacy systems, utf-8 is the only thing you need.
Do you have any specific examples, or are you just saying general statements?
Back in the day, I worked quite a bit various encodings (my language had 2 primary one and 2 secondary one, and it was a guess which one the text was), and the data loss usually happened from programs that tried to support encodings.
When program would not touch encodings, there might be some mojibake and unreadable text, but you could generally fix things.. I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss.
It's the programs which were "able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8" which caused data loss. There were so many cases when I ended up with directory full of "?????????" or "aoeoaoao" files and that was it, there is no way to recover.
So please don't add Unicode support unless you have to, it is not ideology but rather the results of hard practical experience.
I used to work on a platform at a large financial services firm; it was essentially complete ignorant of anything Unicode with respect to string handling, strings were null-terminated byte streams. The platform had CSV import capability for tabular data, and it had an integrated pivot table capability based on some widgets that had been grafted onto it.
Some of the users in Hong Kong discovered that you could import CSVs with Unicode text (e.g. index compositions with Chinese company names) and they'd display in the pivot table widgets and even be exportable to reports. But only most names. Some names were truncated or turned into garbage, and I was called upon to help debug this.
My first reaction was frank amazement that this "worked" at all: apparently, the path from the dumb CSV import code through to the Unicode-aware pivot table was sufficiently clean that much of the encoded text made it through OK. I can't remember the precise details now but I think the problem turned out to be embedded nulls from UTF-16 encoding and so was completely insoluble without a gut renovation of the platform.
UTF-8 is made in such a way that processing based on ASCII and bytes mostly works. All non-ASCII characters are just a sequence of bytes >127, and the system can recover if code points are split or truncated.
So if you just move strings around and concatenate them, you don't even need to be aware of Unicode, even things like templating can work by just processing bytes. Of course, when you get to editing and rendering, you get all the complexity.
Thank you for being the first article I've ever actually read to explain the difference between NFC, NFD, NFKD and NFKC in a way that I actually understood. I was a little bored through the whole UCS/UTF* history lesson because I knew a lot of it already, but the normalization and collation examples were definitely worth it
Agreed, and it would be even better if it mentioned some real-world normalization issues like it does for the UCS encodings. I learned about it the hard way when dealing with Apple filesystems: https://eclecticlight.co/2021/05/08/explainer-unicode-normal...
Man, this gave me such a big headache when using a NAS with MacOS. I have a bunch of file names that use Hangul, and it resulted in a mess that I never fully sorted out.
Currently working on a language, I feel dizzy after reading this.
My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries.
I fail to see this as part of a programming language.
One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it."
Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points).
This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type.
For another good alternative to this approach, see Ruby.
> separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it."
Yes.
Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as
s(5)Hello
This, in turn, is encapsulated in UTF-8 XML. I was coding this in Rust, which led to the discovery that the C++ code it is emulating had problems.
Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check.
Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as
> "Strings are always UTF-8. Anything else is a bug. Deal with it."
That’s how I did it: everything is utf8 (symbols and strings), but you can specifically indicate you want a byte array or nibble or word array etc. And they all work the same transparently.
You’re definitely wrong. You’re designing a language that only works for “Americans” by default.
Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English?
You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American. And even if you’re American, people will still throw emdashes and Unicode quote symbols and emojis into your text. I’d suggest that if a language doesn’t have a proper character type, it’s not a useful language for the majority of the world.
I mean the reserved words in my lang are in English, and the user-created identifiers must be ASCII at the moment. I could extend the latter to some wider range but my question was more about providing built-in Unicode string handling.
My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes.
Apologies if I come off a bit strong on this one. I deal mostly with non “American” text so it’s a pain point. Unicode is so messy, trying to push handling it on to each individual programmer is not an ideal situation.
Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively.
Imagine being from a country where a language’s built in Strcmp doesn’t work?
It depends on the level and domain of your language. A low-level language can get by with just byte arrays. A mid-level language should probably handle at least some encodings, and provide codepoint access. A high-level language should handle locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!).
People expect Strings. You don't have to provide strings, but you should be aware that people expect them and if you provide something else but don't clearly label it as not strings there's an excellent chance they'll go "Oh, strings" and then they'll complain that it doesn't work the way they expected.
If I can write "foo".contains('o') or similar in your language, but then you pretend you don't have strings, nobody will believe you and they're quite right to complain when, inevitably, lots of trickier stuff doesn't work because you didn't bother.
WUFFS doesn't have strings. But WUFFS isn't a general purpose language. Fortran has strings. C has strings (they're terrible but they're clearly strings).
Regarding the title -- anecdotally, everyone I know is sacred of encoding issues, and I don't know anyone who claims they have a great understanding of Unicode or think it is easy (including myself). It is often overlooked for sure -- people don't realize there is a problem in the code until they run into a bug, ane it turns out they are treating strings wrong from the very beginning.
Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.
But UTF-8 is just a way to encode a number as a variable-length string of octets. Why would you be unable to encode, say, a terminating U+D800 as a string of three bytes at the end of a UTF-8 stream?
Still potentially annoying if you deal with some other code that expects UTF-8 proper and you pass it a wtf-8 string that fails the lifted verification.
That would be great, but that article is about recommendations for applications running on Windows, not about what valid file names applications may encounter. Here's a counter-example: https://github.com/golang/go/issues/32334
No, I mean Windows API honors UTF-16 surrogate pairs, and processes them correctly. It doesn't produce invalid UTF-16 strings either. Apps may not support UTF-16 properly, and that's not on Windows, is it?
NTFS, on the other hand, has no dictated format for filename encoding. It just stores raw bytes as filenames, so anything can be a filename on NTFS, including invalid strings if the caller decides to do so. That's not on Windows either, otherwise, we should add Linux to the list too as ext4 and most other file systems also don't care about filename encoding.
Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words).
It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the edge case where you do have file names that aren't UTF-16.
This is basically a result of Windows's file API being fixed in 1993, a good few years before Unicode decided that 16 bits wasn't enough for everyone, and hence a good few years before UTF-16 existed.
The problem is that if the operating system allows sequences that are not valid UTF-16, then you cannot just state "it uses UTF-16": anything that is expecting UTF-16 and only UTF-16 is going to break on valid file names.
"That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. If you treat everything as if it must be UTF-8, your application may just break when it encounters a name that isn't UTF-8. (This detail gets messy because there are configurations of, eg, ext4 and ZFS that limit valid file names to be valid UTF-8.)
Does this actually matter? Who is out there, in the wild, creating file paths that use incredibly cursed random sequences of bytes? Maybe it's just me, but I don't really see the need to accommodate people who do stupid things to see what breaks.
On Windows, it doesn’t matter much because, although the file system doesn’t validate Unicode, the usual system calls that work with file names do, so it’s very rare to encounter non-Unicode file names.
On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only support UTF-8 paths.
I'll just say: consider yourself very lucky to not having encountered it. (Especially if you've spent any time downloading random zip archives from the internet...)
I have quite a number of times. There's no one "thing"; it's usually some legacy app that write these names. And when you get a program that refuses to open any non-UTF16 compliant file name, you'll have a time of grief.
Windows used to use UCS-2 because UTF-16 hadn't been invented yet. Any Windows after Windows 2000 supports UTF-16.
You can't just dismiss the whole operating system API space just because NTFS doesn't enforce filename encoding. Yes, you might encounter file names with invalid UTF sequences because some app might be allowing them to be created. That can also happen in Linux too. Then add Linux, and probably others to the list. Then, what's the use for this list?
I don't understand why this gets brought up. You still have to assume the filenames are encoded to something in order to display them, otherwise there would be no way to display a filename other than in hexadecimal. Windows always uses UTF-16 while Linux uses the locale (which defaults to UTF-8). It's not like working with filenames on Linux is somewhat easier or more straightforward, it's certainly not.
Indeed it is. One use of Unicode I do is for icons that can be used by console programs like (neo)vim. I was quite happy that xterm supports Unicode these days so I can use a fast terminal that supports OSC52 system clipboard integration(none of the newer gnome/KDE terminals do).
I was rather disappointed when I noticed my pretty Unicode icons would sometimes end up cut in half :-(
if you see URLs which project the microsoft editors choice of "single quote mark" into a triad of %xx; values, you've seen how people wind up in unexpected unicode/utf land, because of the text editor swapping ASCII for an uplift into either distorted iso-latin1, or unicode/utf8 depending.
Annoying. Had recently trouble with a translation file (Java properties format). Traditionally it is all ASCII with \u1234 escape sequences. Well except for that recently added "single quote" which is actually some unnecessary (and I assume unwanted) more exotic Unicode character.
Also a reason why I'm against Unicode identifier in source code. You got some source with characters like 'ö' in one variant and now your editor decides to use a different variant (e.g. compatibility character vs base character + combining decoration). And from my observation programming languages which claim to support Unicode tend to not make the effort to resolve those equivalents. At least for older versions of Go the specification clearly stated that they only compare Unicode code points. (don't know the current state)
I'll always take an excuse to link to one of my favorite StackOverflow answers, to the question "Why does modern Perl avoid UTF-8 by default?": <https://stackoverflow.com/a/6163129/2521092>. It's from 2011 and Perl-centric, of course, but skip down to "𝔸 𝕤 𝕤 𝕦 𝕞 𝕖 𝔹 𝕣 𝕠 𝕜 𝕖 𝕟 𝕟 𝕖 𝕤 𝕤" for a thorough, if opinionated, list.
My favorite, similar to the issue of case-folding, is the idea of something being a "letter" and "lowercase" but NOT "lowercase letter."
A reviewer for a scientific paper I wrote asked if I had considered the impact of UTF-9 on what I was doing. To this day I don't know if they were trolling me.
With the benefit of hindsight, if the standard was done over again either a lot of unnecessary difficulties mentioned in this article should be avoided or the standard split in two. It is arguably too ambitious, too inefficient, and too unrealistic for a number of these things to be handled as recommended in all contexts. There are many examples - in operating system kernels and system programming in general, to name two.
To start with, alternative representations for the same visibly identical character should have been excluded. In the base standard all supported characters should be precomposed - no modifiers. The article points out the difficulties in something as simple (and as security critical) as determining whether two strings refer to the same visibly identical characters.
A character set that does not make that trivial is not suitable for general use in system programming or for security critical identifiers (unfortunately). It massively complicates programming in many programming languages as well, and even UTF-32 is not sufficient to remedy the problem, as the article well notes.
The inability to handle and process any arbitrary string of bytes in a universal character set is a serious problem as well. The world is complicated, and the inability to pass through incorrectly coded data, alternatively coded data, and general binary data is a major limitation with serious consequences. System code such as device drivers or filesystems typically can't deal with inefficiencies or limitations like that.
In addition, there probably should be two types of uppercase and lowercase conversions, a simple, predictable one for system programming, and a more complex one that deals with considerations in languages that do not follow the normal rules.
String collation should be done at two levels as well, a simple code point level suitable for such things as prefix matches on database indexes, and a more complex variation for applications where linguistically sensitive collation is critical.
In general trying to solve all the higher end use cases in a large body of software that do not need to deal with them, should not need to deal with them, or cannot deal with them is impractical and has exacerbated the common string processing issues we see today. A lightweight standard - perhaps a subset or profile of Unicode that supported arbitrary binary data as opaque codepoints - could be helpful in a lot of contexts.
Pity those bad design decisions like surrogate pair pollute the future for generations instead of being cleanly cut off with a painful, but temporary transition period
Another important subject are confusables. "A" looks same for Latin A, Greek Alpha and Cyrillic A. If you need users to identify names by sight, you need to do something about confusables. It is even a security problem, for example for hostnames, imagine a hacker build your bank's website with a similar looking URL.
119 comments
[ 3.2 ms ] story [ 172 ms ] threadTry working with raw UTF-8 in JS and find yourself in a world of pain. Mathias Bynens talks about these gotchas here:
https://mathiasbynens.be/notes/javascript-unicode
Language is a very sensitive topic - in Central Asian countries, using Latin, Cyrillic or Perso-Arabic script for instance has very strong political connotations, same in the Balkans. The world is just like that
Besides, why do you care what funny symbols people use in discussions that don’t involve you?
You think they get it, but surprise happens when a database load fails when loading Chinese Character "string" into a field sized calculated based upon 2 bytes per character.
(And emoji are such a fine example - once they ate on the table, you need support for combining characters, characters outside of BMP, ligatures.. a large part of Unicode spec)
Like Apple trying to region-lock emojis, or whoever decided merging CJK together was a good idea, or that plan9 dev who scribbled out UTF8 on a napkin.
[0]: https://www.youtube.com/watch?v=gd5uJ7Nlvvo
[0] https://youtu.be/6avJHaC3C2U
[0]: https://www.youtube.com/watch?v=mrGfahzt-4Q
Looks like the entire idea is to avoid using any ASCII/Extended ASCII control character bytes, except to represent those characters.
Not sure what the advantage of that was.
[1] https://en.wikipedia.org/wiki/UTF-1
[1]: https://www.swift.org/blog/utf8-string/
There were many discussions on how to handle strings in the forums too. Remarkably, it is not possible to access "abc"[1] as it has unpredictable performance; instead one has to build the index and making this should make the developer realize the operation is costly. All in all most beginners in Swift hate working with strings because it’s not intuitive at first, but to be fair in almost all languages strings handling is not done properly.
I'd take "we don't do unicode at all" or "we only support BMP" or "we don't support composite characters" any day over pretend-support (but then inevitably breaking when the program wasn't tested with anything non-ASCII)
(ninjaedit: to see how prevalent it is, even gigantic message apps such as discord make this mistake. There are users on discord who you can't add as friends because the friend input field is limited to 32.... something - probably bytes, yet elsewhere the program allows the name to be taken. This is easy to do with combining characters)
Placing a fuzzy tester like "hypothesis.strategies.characters" into the CI may also be revealing.
The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated.
𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes of UTF8 and as wide as about 9 Latin characters. (an England flag emoji) is about a character wide, but takes 28 bytes to store in UTF8.
Failing that, ask an AI.
There's no way to take an arbitrary utf-8 string and shorten it to <= 4000 bytes without losing something. Heck, even ASCII has \r\n which can leave you in a bad place if you cut it off in the middle.
Even if you do the Unicode stuff right and make sure you don't break a grapheme cluster, you're still cutting words in half, which, if you're doing something like pulling AP headlines and truncating them to fit on a screen, can have embarrassing consequences for one like 'FDA Chief Exposes Butter Industry Corruption'. Even if you make sure to break on word boundaries you're still at risk of turning 'US Navy Fires Nuclear Weapons Program Chief' into 'US Navy Fires Nuclear Weapons' on your news ticker.
Strings are language. Language is hard.
And what "various things" do you mean that are on top the char split?
Does it matter if it's Hangul?
Hmm, I could imagine that you have a precomposed character with two diacritical marks; if there were no precomposed character with the remaining diacritical you'd need to replace the precomposed character with the base character and diacritical. However I would imagine the better UX would be for backspace to remove the entire character (whether diacriticals were composed or not) because that's how people think of their characters.
Is there a real world case? I'd be excited to learn it.
... Hangul.
What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)
Imagine if hanzi had been encoded as radicals…
The easiest way to support Unicode is to avoid supportimg it. Leave text editing to UI widgets. Leave truncation to web browsers. Avoid fancy marquee effect on the terminal. Your program may look less fancy but it will automatically support unicode, even future extensions.
Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8. Otherwise you'll get text talking about cutting doorways 8œ feet tall or submitting a résumé.
Any software that needs to process information effectively needs to understand character sets, including Unicode.
And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just hardcode two common encodings (like 8859-1 and 8859-15)... I used to deal with encodings a lot, and the only time I actually had data loss from encoding problems was from misguided apps which assume 8859-1 (and tried to convert to ASCII by stripping accents). Please either assume UTF-8 or assume nothing and keep data as-is.
And that's it, _most_ software does not need to know more. Let's look at some examples:
- For file archiver, assume utf-8 everywhere and don't worry about encoding at all (unless you planning to deal with very old archives, in this case you'd need encoding support. But for something made from scratch, don't bother.) _Especially_ don't add any encoding support for "extract to console" function -- if I need a different encoding I can pipe into iconv myself, thank you very much.
- For API client, you don't need to worry about encoding either.. A very old HTTP server might return data in something other than utf-8 but (1) your HTTP library likely handles it already (2) how many such servers are left, anyway?
- For XML analyzer, use proper XML library, they will handle encodings for you.
- For web dashboard or a database, keep everything in utf-8 and things will just work.
- For ETL-like applications, many modern data sources are already in UTF-8.. and if they are not, convert everything to utf-8 ASAP.
In other words: while it's useful to know that encodings exist, unless you are working with very old data or legacy systems, utf-8 is the only thing you need.
[0] http://utf8everywhere.org/
Back in the day, I worked quite a bit various encodings (my language had 2 primary one and 2 secondary one, and it was a guess which one the text was), and the data loss usually happened from programs that tried to support encodings.
When program would not touch encodings, there might be some mojibake and unreadable text, but you could generally fix things.. I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss.
It's the programs which were "able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8" which caused data loss. There were so many cases when I ended up with directory full of "?????????" or "aoeoaoao" files and that was it, there is no way to recover.
So please don't add Unicode support unless you have to, it is not ideology but rather the results of hard practical experience.
> I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss.
If you do the first, it is impossible to do the second.
Are you just disagreeing for the sake of being disagreeable? Why would you do that?
Some of the users in Hong Kong discovered that you could import CSVs with Unicode text (e.g. index compositions with Chinese company names) and they'd display in the pivot table widgets and even be exportable to reports. But only most names. Some names were truncated or turned into garbage, and I was called upon to help debug this.
My first reaction was frank amazement that this "worked" at all: apparently, the path from the dumb CSV import code through to the Unicode-aware pivot table was sufficiently clean that much of the encoded text made it through OK. I can't remember the precise details now but I think the problem turned out to be embedded nulls from UTF-16 encoding and so was completely insoluble without a gut renovation of the platform.
So if you just move strings around and concatenate them, you don't even need to be aware of Unicode, even things like templating can work by just processing bytes. Of course, when you get to editing and rendering, you get all the complexity.
My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries.
I fail to see this as part of a programming language.
Am I wrong here ?
Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points).
This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type.
For another good alternative to this approach, see Ruby.
Yes.
Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as
This, in turn, is encapsulated in UTF-8 XML. I was coding this in Rust, which led to the discovery that the C++ code it is emulating had problems.Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check.
Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as
per the spec.That’s how I did it: everything is utf8 (symbols and strings), but you can specifically indicate you want a byte array or nibble or word array etc. And they all work the same transparently.
Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English?
You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American. And even if you’re American, people will still throw emdashes and Unicode quote symbols and emojis into your text. I’d suggest that if a language doesn’t have a proper character type, it’s not a useful language for the majority of the world.
UTF-8 is a reasonable compromise though.
My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes.
I get your concern though and realize parsing
would be a problem.Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively.
Imagine being from a country where a language’s built in Strcmp doesn’t work?
> Imagine being from a country where a language’s built in Strcmp doesn’t work?
Problem is it's seemingly a massive headache. Citing the article, "Unicode handling is always best left to a library".
I could bundle the recommended lib (ICU) into my lang's stdlib but it seems quite big...
EDIT: and most languages made before 1990 too
If I can write "foo".contains('o') or similar in your language, but then you pretend you don't have strings, nobody will believe you and they're quite right to complain when, inevitably, lots of trickier stuff doesn't work because you didn't bother.
WUFFS doesn't have strings. But WUFFS isn't a general purpose language. Fortran has strings. C has strings (they're terrible but they're clearly strings).
By your logic, couldn't I pretend then to have strings insomuch as C when "foo".contains('o') can be transpiled by my lang as
[1] https://simonsapin.github.io/wtf-8/#utf-8
https://learn.microsoft.com/en-us/windows/win32/intl/surroga...
NTFS, on the other hand, has no dictated format for filename encoding. It just stores raw bytes as filenames, so anything can be a filename on NTFS, including invalid strings if the caller decides to do so. That's not on Windows either, otherwise, we should add Linux to the list too as ext4 and most other file systems also don't care about filename encoding.
It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the edge case where you do have file names that aren't UTF-16.
This is basically a result of Windows's file API being fixed in 1993, a good few years before Unicode decided that 16 bits wasn't enough for everyone, and hence a good few years before UTF-16 existed.
A file path is just a sequence of bytes. Don't try to validate it in any sort of encoding, it's just bytes that the OS uses to identify a file.
"That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. If you treat everything as if it must be UTF-8, your application may just break when it encounters a name that isn't UTF-8. (This detail gets messy because there are configurations of, eg, ext4 and ZFS that limit valid file names to be valid UTF-8.)
On Linux, it’s not as rare as you might expect to end up with a few non-UTF–8 file names, though they’ll normally be on things you won’t often touch directly, so things like desktop software can reasonably only support UTF-8 paths.
I'm getting strong XKCD "Workflow"[0] vibes out of the idea that someone would have unintelligible paths on their system
[0] https://xkcd.com/1172/
I have quite a number of times. There's no one "thing"; it's usually some legacy app that write these names. And when you get a program that refuses to open any non-UTF16 compliant file name, you'll have a time of grief.
That works fine until it becomes literally impossible when your language/library/framework forces you into picking an encoding.
Even Microsoft's own APIs that add "UTF-8 support" choke when they encounter file names with invalid UTF-16 sequences.
You can't just dismiss the whole operating system API space just because NTFS doesn't enforce filename encoding. Yes, you might encounter file names with invalid UTF sequences because some app might be allowing them to be created. That can also happen in Linux too. Then add Linux, and probably others to the list. Then, what's the use for this list?
truly an appropriate name
In practice, UTF-16 validation is rare.
I was rather disappointed when I noticed my pretty Unicode icons would sometimes end up cut in half :-(
Also a reason why I'm against Unicode identifier in source code. You got some source with characters like 'ö' in one variant and now your editor decides to use a different variant (e.g. compatibility character vs base character + combining decoration). And from my observation programming languages which claim to support Unicode tend to not make the effort to resolve those equivalents. At least for older versions of Go the specification clearly stated that they only compare Unicode code points. (don't know the current state)
My favorite, similar to the issue of case-folding, is the idea of something being a "letter" and "lowercase" but NOT "lowercase letter."
A reviewer for a scientific paper I wrote asked if I had considered the impact of UTF-9 on what I was doing. To this day I don't know if they were trolling me.
To start with, alternative representations for the same visibly identical character should have been excluded. In the base standard all supported characters should be precomposed - no modifiers. The article points out the difficulties in something as simple (and as security critical) as determining whether two strings refer to the same visibly identical characters.
A character set that does not make that trivial is not suitable for general use in system programming or for security critical identifiers (unfortunately). It massively complicates programming in many programming languages as well, and even UTF-32 is not sufficient to remedy the problem, as the article well notes.
The inability to handle and process any arbitrary string of bytes in a universal character set is a serious problem as well. The world is complicated, and the inability to pass through incorrectly coded data, alternatively coded data, and general binary data is a major limitation with serious consequences. System code such as device drivers or filesystems typically can't deal with inefficiencies or limitations like that.
In addition, there probably should be two types of uppercase and lowercase conversions, a simple, predictable one for system programming, and a more complex one that deals with considerations in languages that do not follow the normal rules.
String collation should be done at two levels as well, a simple code point level suitable for such things as prefix matches on database indexes, and a more complex variation for applications where linguistically sensitive collation is critical.
In general trying to solve all the higher end use cases in a large body of software that do not need to deal with them, should not need to deal with them, or cannot deal with them is impractical and has exacerbated the common string processing issues we see today. A lightweight standard - perhaps a subset or profile of Unicode that supported arbitrary binary data as opaque codepoints - could be helpful in a lot of contexts.
https://news.ycombinator.com/item?id=32497414
https://en.wikipedia.org/wiki/Spread_of_the_Latin_script#21s...
Any Indians here? I heard that most Hindi texting is done with a weird Latin character slang these days?