Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
We really ought to deprecate UTF-16 someday. The fact that it pretends to be a fixed-length encoding has caused all sorts of bugs over the years, with many people assuming n(UTF-16 codepoints) == n(characters) which breaks when the string contains non-BMP characters.
And also, for personal aesthetic reasons I hate that it limits the Unicode codepoint range to an awkward non-power-of-two number (0x110000 codepoints in total). UTF-8 and UTF-32's 2^31 is much more natural.
I wonder what it would take. For things like Java, JSA, and c# there would have to be things like a parallel utf8 api (yes please), but the really hard parts is the things that are as old as time (windows).
I don't think it will ever happen, but one can dream.
I think that wouldn't change much. They would just make use of more grapheme clusters.
For emojies they already make heavy use of the Zero-Width-Joiner. So a woman firefighter is the woman emoji + ZWJ + fire engine. Sure the UTF-8000 approach is much better encoding size wise.
I predict eventual convergence between UTF-whatever and most popular tokenizer for whatever LLM escapes to become world-ruling AGI.
I mean, if someone's seriously going to try encoding birdsong and dog barks, at this point they're basically reinventing tokens for multi-modal language models.
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
> "UTF-8" support was often just single byte, so it was little more than ASCII
"Single byte UTF-8" is ASCII. That's one of its most important properties.
> Even now it's common for programs to choke on the optional BOM
And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.
> BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing
How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM? It seems like one would have to read until finding a code point that would be illegal under one ordering (but files might not include such a code point).
Similarly, a UTF-8 BOM is a useful flag to distinguish UTF-8 from other text encodings. You are right that the ambiguity goes away if those other encodings do, but people don't want to rewrite their legacy files. Some people don't want to use two bytes for common non-ASCII characters, so they are really attached to ISO-8859 or Windows-1252 or koi8r or whatever. CJK languages have their own encodings that are more efficient for their languages. UTF-8 is great for English speakers, but it's a compromise for everyone else, so they might reasonably want incompatible systems for their own use. UTF-8 BOM is a good "magic" sequence to detect encoding as long as people have non-UTF-8 files.
> How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM?
Simple: switch text files to UTF-8 as the only text encoding, and get rid of the legacy UCS-2/UTF-16, Extended-ASCII-with-code-pages and all the other ancient region-specific encoding hacks that popped up in the 70s and 80s.
This general switch to UTF-8 should have happend in the mid-to-late 1990s, and Microsoft alone is to blame for dragging this shit along for the next three decades. If all Microsoft tools would only save text data as UTF-8 starting by the end of the last century, but still support reading all sorts of encodings for a while, the transition would long be finished by now without imposing a radical deadline. Alas, that never happened.
Your opinion on the BOM isn't wrong, but it's also not germaine to whether or not you're actually following the spec. The spec is the spec. If you don't like it you can get the spec changed. You don't get to ignore the spec and then claim support. That's not how standards work. "I don't like it," isn't a good explanation.
Otherwise I'd be inclined to fix the spelling error in the HTTP referrer.
CJK characters is the one big weak point of UTF-8. It forces them to be 3 bytes large instead of the local codepage which is 2 bytes per character, or UTF-16 which is also 2 bytes per character.
IMHO a 30 percent size increase for sharing text data is really not a problem when most data sharing (by volume) happens via audio, video and image data which is infinitely bigger.
It's still a joy to see how frigging elegant and extensible the UTF-8 specification is. And even without the esoteric 0xFF lead byte, the regular UTF-8 encoding with a 0xFE lead byte (11111110) would still have plenty of headroom (36 bits) compared to the current 21 bits for UNICODE.
C0 and C1 can do so though according to the author.
> because all 4 of 2-byte UTF-8's mandatory content bits lie in the first-and-final start byte, we can explicitly rule out 11000000 (0xC0) and 11000001 (0xC1) as permanently invalid bytes. They will never ever appear anywhere in a valid UTF-8000 code unit!
256-bit addresses would be long enough proper cryptographic addressing. (128 is barely not enough due to how they're allocated)
512 would also for cryptographic addresses within cryptographic subnets (or 3 128-bit layers plus some extra bits), and 1024 would allow for up to 7 layers. It's not a completely silly idea.
Yggdrasil subnet addresses only have 56 bits of cryptographic entropy; there's a limited workaround where you increase work by searching for one with a zero prefix, but the cost to clone one can't be more than 2^56 times the cost to create it.
But the nice thing about UTF-8 is that this proposal isn't really a "complication", but a minimal and natural extension of the original idea to allow more than one lead byte.
Allowing more than one lead byte is a complication of the existing standard. As many others have pointed out, we have plenty of coding space without that (e.g. by allowing 5- and 6-byte UTF-8 again), so the case for the extra complexity is currently not compelling.
UTF-8 originally supported up to six-byte encodings (see eg. RFC 2279), but it was restricted to four bytes in 2003 in order to match UTF-16 constraints :(
The number of glyphs available by adding additional bytes drops exponentially because each subsequent byte has one less bit available.
So I think if we ever were in a situation where > 1 million code points isn’t enough, then we should look at an entirely new way to serialise those code points.
I don't quite get it. 5-byte utf-8 encoding gets extra 5 bits compared to 4 byte, and 6-byte gets extra 10 bits. If you were thinking about bits in leading byte, then yes, you are losing one bit for every extra trailing byte, but you also get 6 bits from it. So adding a byte gives you extra 5 bits.
It seems when designing Unicode most energy went into emoji. And there was nothing left for fancy things like fixed-length string buffers. The only explaination why UTF8 Buffers aren't compatible with UTF16 Buffers... is a really strong emoji...
The internal string encoding of a programming language doesn't matter as long as it supports UTF-8 at the boundaries. E.g. the text encoding standard on the web is clearly UTF-8, even though JS strings may be internally stored as UTF-16 (or any other encoding).
Same on macOS/iOS btw: AFAIK NSString is internally UTF-16, but I've never seen a UTF-16 text file on macOS, it's all UTF-8 (unless the file originated on Windows of course).
On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
In regular unicode, a grapheme can be made up of an arbitrary number of codepoints (and thus an arbitrary number of bytes), which does cause issues at times.
Without endorsing this specific idea, I think maybe after over half a century of C that this argument shouldn't get in the way of a new standard. Pretty much every other language has managed to solve this problem, and the people who write new projects in C/C++ have decided they're not concerned about buffer overflows, so if someone decides to start a new project using something like this (or go out of their way to add support for it to an existing project), that's kind of on them. The rest of computing shouldn't get stuck in 1972 forever.
I'm failing to see why an embedded environment would have any need for a new encoding format, which is kind of my point: the types of things that are going to be written in C are not the ones going to be adopting completely new backwards-incompatible standards anyhow. If we refuse to try something based on how it would interact of the ecosystem that would likely never consider adopting something like it in the first place, we're literally fixing our computing to constraints from half a century ago and counting. Nobody who is going to write C would stop because of something like a new Unicode scheme with much larger encoding widths, so why should that be an argument against it happening?
A reasonable person would assume you’re talking about UTF-8000. It’s not completely new: RFC2279, the original UTF-8 proposal, worked exactly like UTF-8000 for codepoints 31 bits or smaller in size. It’s not backwards-incompatible: UTF-8000 is exactly like UTF-8 for 1, 2, and 3-byte long codepoints, and like UTF-8 codepoints for 4-byte long codepoints with a value of 0x10_ffff or smaller (so all UTF-8 codepoints encoded with the first byte being 0b1111_00xx or starting with the bytes 0b1111_0100 0b1000_xxxx).
It’s a backwards compatible way of encoding numbers in UTF-8 larger than 0x10_ffff or (0x7fff_ffff with the original RFC2279 proposal).
I agree that C isn’t the best language to start a new programming project in. There are things I don’t like about Rust, mainly that there’s only one implementation of it, but if I were to start a new project needing the speed of a system programming language, it makes a lot of sense.
The original comment that I responded to said this:
> On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
My response was that I don't see "it would be hard to write correct code for this in C" to be a particularly useful objection. I'm honestly a bit confused by the reaction to my comments because it seems that every new reply seems to have an orthogonal objection to the previous one, but none of them seem to be actually addressing the point I've been trying to make.
I agree C has a lot of problems. The nice thing about C is that it’s a rigorous standard (C90, C99, C23, etc.) with multiple actively maintained implementations (GCC, LLVM/clang, TCC, etc.) so standards compliant C code should (in theory) compile and run for the foreseeable future.
Python, Rust, Perl, etc. are all languages with a single actively maintained implementation and haven’t had the rigorous standardization C and POSIX utilities (Awk, /bin/sh, sed, etc.) have had.
> standards compliant C code should (in theory) compile and run for the foreseeable future
Yes, but the problem is that we have decades of examples of showing that even the best programmers in the world in practice will write code that isn't "standards compliant" in a way that prevents UB.
You don't have a buffer overflow problem if you read it in a memory-safe way i.e. read it in chunks and realloc when you reach the size of your allocation.
What you will have is a potential denial-of-service attack - although this one isn't particularly great because there's zero amplification (they might as well just send garbage into your firewall)
Memory allocation?? Why? Real world, you'll bump into limits based on fonts long before you'll get a buffer that's too large for the stack and worth using the memory allocator for. I can't see any reason to support more than 2^64 characters and lots of headaches from trying to beyond that. You check your buffer writes and reject the character if it's too long.
I think the DoS has pretty common amplification vectors in the form of APIs that split or otherwise copy (e.g. materializing code points for Unicode regex searching).
I don’t think there’s harm is speccing out the arbitrary encoding and then having a different spec that references that spec but puts hard limits on it. Many rfcs are like that.
"your first 2 extensions (5 and 6 bytes) were clearly envisioned.
the standard (up to 4 bytes) was created to cover the size of
unicode. i thought any more description would be a waste of
paper. i think your extension from 7 to 8 bytes is a little hoaky.
i requires reading the whole string rather than "knowing" the
number of follow on bytes. so, i think the only thing new is the
7 byte version.
i appreciate the mail, but i really dont think it is useful. it is
like replacing ipv6 with ipv50."
Important to recognize that characters have individuality, that's why there can only be a limited number of them. Anything without this property - any generic form of encoding - is not characters, it's something else like images. If it's not in any alphabet it shouldn't be in unicode, you should use an escape tag for image data instead. (Emojis probably shouldn't, but they do behave like an alphabet)
There cannot be 4 trillion characters because humans would need to know all of them and humans cannot know that many things.
It’s a fairly simple encoding which is UTF-8 compatible and allows codepoints to be arbitrarily large. It’s a real encoding proposal with a simple rigorous specification. I don’t have a use case for needing arbitrarily large codepoints, but for people who do, this makes more sense than hacks such as having surrogates in Unicode’s hi-bit private use area.
- fully-customizable emojis (think of a RPG-like character customization screen)
- heck, why not full jpegs/gifs?
- some unicode programming script (running Doom)
- ?
That said, some very minor (HN-style) nitpick:
> Otherwise for an n byte code unit this is (5n+1) / 8n, that is 5n+1 content bits out of a total of 8n bits from n bytes. We can rewrite this as (5/8) + 1/(8n) which moderately quickly approaches 5/8 = 62.5%. It is nice that this limit is nonzero and does not depend on n.
Isn't a limit by definition no longer dependent on n?
U+E000–U+F8FF, U+F0000–U+FFFFD, and U+100000–U+10FFFD can already provide you with your own emoji, as that range has been reserved for private use. Extending the range further might make sense if you need even more space in your program, but that's a lot of space already.
I agree with you that Unicode urgently needs a scripting capability (*), but my plan was to just implement it using invisible tag characters [1] or something like that - but of course implementing it in a single codepoint is the much more elegant solution.
Another one: encode instructions on how to draw the glyph into the text itself. The string becomes both the text and the font. Why not make it turing complete and as powerful/complex as TTF.
Imagine someone using the same fully customized emoji multiple times in the same text. Seems like a waste of space. Maybe better to encode just a UUID, and send the image codebook separately.
Self-synchronization in UTF-8 is intuitively a great thing to have, yet I don’t remember actively relying on it ever. Does anyone have a good example of when it‘s useful?
Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.
However, Rust makes use of it for fast safety checks. Because rust strings must be valid utf8, if you want to take a substring at some range, eg "Hello, World!"[7..12] then it's very simple to just check bytes 7 and 12 and see if they are the start of a codepoint, no other scanning or parsing is required.
I don’t see how UTF-8000 doesn’t have it. The first byte of any code point is either 0xxxxxxx or 10xxxxxxx, which is distinct from all non-first bytes which are 11xxxxxx. Thus any UTF-8000 sub-bytestring must necessarily have the start aligned at a code point boundary, at which point all the subsequent bytes are interpreted as codepoints in the same way.
> Does anyone have a good example of when it‘s useful?
It prevents vulnerabilities where an incorrect offset into a string could result in characters being read that aren’t in the original string (which could defeat a prior sanitization of the string).
1. You can partition an input file at any offsets, parallelize, and adjust partition boundaries to a valid offset independently.
Without the property, parallelization is hard.
This is how mapreduce has been used to process large text files, except at line boundaries.
Now, for this that might not be a useful enough property, given that we already do similar things for newlines, and UTF-8 guarantees ASCII is always recognizable and hence newlines are always recognizable.
2. It might have been more useful in the era of dial-up where we still had occasional corrupted bytes in the transmission.
3. It helps regain sanity if e.g. a background process outputs bytes that get interleaved at the tty. For example, cat a large text file, the write boundaries won't always align at UTF-8 boundaries, then have a background process output get interleaved in an unfortunate way. If it self-synchronizes, it'll knock itself back into sync after a small amount of garbage.
I was about to immediately suggest UCS-X, and then pleased that the author offers a fair comparison to UCS-X and many other alternatives. Great. Too bad UCS-X is possibly more memorable name.
IIRC UTF-8 proposal used to go to 6 bytes as they already felt that not having part of the code point in the first byte was an issue somehow.
I think at least 7 should be allowed. 42 bits could contain all sorts of information.
"UTF-8000" is still ANSI-safe and although it loses some of the properties such as a header code only appearing once in a byte sequence, it is perfectly viable to encode arbitrarily sized unsigned integers.
It's not even true as being able to tell the character length from the first byte is not a property that extending UTF-8 past 36 bit payloads preserves.
Perhaps we can use a generalized Unicode-like encoding space for LLM tokens. A tokenization scheme uses up a few hundred thousand entries, with say one thousand new entries added annually. These can be called amojis, meaning AI mojis.
If it's just a general Transformation Format and Unicode will never assign those code points, it'd be just TF-8. I can think of a second way to expand that abbreviation.
107 comments
[ 0.29 ms ] story [ 60.9 ms ] threadNot until they decide to expand the emoji range, allocate space for all past and future fictional languages, as well as birdsong and dog barks.
Someone at the consortium is rubbing their hands with glee with all the newfound space.
But honestly, cool hack! If you invent a method to encode large numbers into bytes, why limit yourself to 24-bit numbers?
Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
For compatibility with UTF-16:
* https://datatracker.ietf.org/doc/html/rfc3629#section-12* https://en.wikipedia.org/wiki/UTF-16
The original spec had 31 bits (the UTF-32/UCS-4 range):
* https://datatracker.ietf.org/doc/html/rfc2279
* https://en.wikipedia.org/wiki/UTF-32
And also, for personal aesthetic reasons I hate that it limits the Unicode codepoint range to an awkward non-power-of-two number (0x110000 codepoints in total). UTF-8 and UTF-32's 2^31 is much more natural.
I don't think it will ever happen, but one can dream.
For emojies they already make heavy use of the Zero-Width-Joiner. So a woman firefighter is the woman emoji + ZWJ + fire engine. Sure the UTF-8000 approach is much better encoding size wise.
I mean, if someone's seriously going to try encoding birdsong and dog barks, at this point they're basically reinventing tokens for multi-modal language models.
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
"Single byte UTF-8" is ASCII. That's one of its most important properties.
> Even now it's common for programs to choke on the optional BOM
And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.
How should a reader infer the bye order for a UCS-2 or UTF-16 file without a BOM? It seems like one would have to read until finding a code point that would be illegal under one ordering (but files might not include such a code point).
Similarly, a UTF-8 BOM is a useful flag to distinguish UTF-8 from other text encodings. You are right that the ambiguity goes away if those other encodings do, but people don't want to rewrite their legacy files. Some people don't want to use two bytes for common non-ASCII characters, so they are really attached to ISO-8859 or Windows-1252 or koi8r or whatever. CJK languages have their own encodings that are more efficient for their languages. UTF-8 is great for English speakers, but it's a compromise for everyone else, so they might reasonably want incompatible systems for their own use. UTF-8 BOM is a good "magic" sequence to detect encoding as long as people have non-UTF-8 files.
Simple: switch text files to UTF-8 as the only text encoding, and get rid of the legacy UCS-2/UTF-16, Extended-ASCII-with-code-pages and all the other ancient region-specific encoding hacks that popped up in the 70s and 80s.
This general switch to UTF-8 should have happend in the mid-to-late 1990s, and Microsoft alone is to blame for dragging this shit along for the next three decades. If all Microsoft tools would only save text data as UTF-8 starting by the end of the last century, but still support reading all sorts of encodings for a while, the transition would long be finished by now without imposing a radical deadline. Alas, that never happened.
Otherwise I'd be inclined to fix the spelling error in the HTTP referrer.
> because all 4 of 2-byte UTF-8's mandatory content bits lie in the first-and-final start byte, we can explicitly rule out 11000000 (0xC0) and 11000001 (0xC1) as permanently invalid bytes. They will never ever appear anywhere in a valid UTF-8000 code unit!
512 would also for cryptographic addresses within cryptographic subnets (or 3 128-bit layers plus some extra bits), and 1024 would allow for up to 7 layers. It's not a completely silly idea.
Yggdrasil subnet addresses only have 56 bits of cryptographic entropy; there's a limited workaround where you increase work by searching for one with a zero prefix, but the cost to clone one can't be more than 2^56 times the cost to create it.
Especially with the layering, I would think you could put this in a routing header?
This is why we need the KISS enforcers.
Some day we'll need this when we finally realise we are not alone in the universe. Alien glyphs ftw.
https://xkcd.com/1953/
So I think if we ever were in a situation where > 1 million code points isn’t enough, then we should look at an entirely new way to serialise those code points.
https://artoria2e5.github.io/XB18030/
It seems when designing Unicode most energy went into emoji. And there was nothing left for fancy things like fixed-length string buffers. The only explaination why UTF8 Buffers aren't compatible with UTF16 Buffers... is a really strong emoji...
Same on macOS/iOS btw: AFAIK NSString is internally UTF-16, but I've never seen a UTF-16 text file on macOS, it's all UTF-8 (unless the file originated on Windows of course).
So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
Without endorsing this specific idea, I think maybe after over half a century of C that this argument shouldn't get in the way of a new standard. Pretty much every other language has managed to solve this problem, and the people who write new projects in C/C++ have decided they're not concerned about buffer overflows, so if someone decides to start a new project using something like this (or go out of their way to add support for it to an existing project), that's kind of on them. The rest of computing shouldn't get stuck in 1972 forever.
Sometimes the environment forces a language choice.
A reasonable person would assume you’re talking about UTF-8000. It’s not completely new: RFC2279, the original UTF-8 proposal, worked exactly like UTF-8000 for codepoints 31 bits or smaller in size. It’s not backwards-incompatible: UTF-8000 is exactly like UTF-8 for 1, 2, and 3-byte long codepoints, and like UTF-8 codepoints for 4-byte long codepoints with a value of 0x10_ffff or smaller (so all UTF-8 codepoints encoded with the first byte being 0b1111_00xx or starting with the bytes 0b1111_0100 0b1000_xxxx).
It’s a backwards compatible way of encoding numbers in UTF-8 larger than 0x10_ffff or (0x7fff_ffff with the original RFC2279 proposal).
I agree that C isn’t the best language to start a new programming project in. There are things I don’t like about Rust, mainly that there’s only one implementation of it, but if I were to start a new project needing the speed of a system programming language, it makes a lot of sense.
> On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
My response was that I don't see "it would be hard to write correct code for this in C" to be a particularly useful objection. I'm honestly a bit confused by the reaction to my comments because it seems that every new reply seems to have an orthogonal objection to the previous one, but none of them seem to be actually addressing the point I've been trying to make.
Python, Rust, Perl, etc. are all languages with a single actively maintained implementation and haven’t had the rigorous standardization C and POSIX utilities (Awk, /bin/sh, sed, etc.) have had.
Yes, but the problem is that we have decades of examples of showing that even the best programmers in the world in practice will write code that isn't "standards compliant" in a way that prevents UB.
What you will have is a potential denial-of-service attack - although this one isn't particularly great because there's zero amplification (they might as well just send garbage into your firewall)
"your first 2 extensions (5 and 6 bytes) were clearly envisioned. the standard (up to 4 bytes) was created to cover the size of unicode. i thought any more description would be a waste of paper. i think your extension from 7 to 8 bytes is a little hoaky. i requires reading the whole string rather than "knowing" the number of follow on bytes. so, i think the only thing new is the 7 byte version.
i appreciate the mail, but i really dont think it is useful. it is like replacing ipv6 with ipv50."
Nobody needs more than 4.47 trillion characters. (famous last words)
There cannot be 4 trillion characters because humans would need to know all of them and humans cannot know that many things.
edit: replaced false prophet site with wiki link for the original one and true timecube.com...lost to the sands of ...time.
It’s a fairly simple encoding which is UTF-8 compatible and allows codepoints to be arbitrarily large. It’s a real encoding proposal with a simple rigorous specification. I don’t have a use case for needing arbitrarily large codepoints, but for people who do, this makes more sense than hacks such as having surrogates in Unicode’s hi-bit private use area.
I mean, I can derive from the text that this can encode arbitrarily large integers. So what? What's the point? I also understand that this is for fun.
- fully-customizable emojis (think of a RPG-like character customization screen)
- heck, why not full jpegs/gifs?
- some unicode programming script (running Doom)
- ?
That said, some very minor (HN-style) nitpick:
> Otherwise for an n byte code unit this is (5n+1) / 8n, that is 5n+1 content bits out of a total of 8n bits from n bytes. We can rewrite this as (5/8) + 1/(8n) which moderately quickly approaches 5/8 = 62.5%. It is nice that this limit is nonzero and does not depend on n.
Isn't a limit by definition no longer dependent on n?
I agree with you that Unicode urgently needs a scripting capability (*), but my plan was to just implement it using invisible tag characters [1] or something like that - but of course implementing it in a single codepoint is the much more elegant solution.
(*) in RFC 04-01-2027
[1] https://en.wikipedia.org/wiki/Tags_(Unicode_block)
I, for one, welcome our new time-traveling overlords.
- Unicode-as-a-character, meaning: a whole text displayed as if it were one character, made for infinite zooming
> - ?
Bad Apple, oh, wait: https://github.com/hsfzxjy/Bad-Apple-Font
Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.
However, Rust makes use of it for fast safety checks. Because rust strings must be valid utf8, if you want to take a substring at some range, eg "Hello, World!"[7..12] then it's very simple to just check bytes 7 and 12 and see if they are the start of a codepoint, no other scanning or parsing is required.
It prevents vulnerabilities where an incorrect offset into a string could result in characters being read that aren’t in the original string (which could defeat a prior sanitization of the string).
1. You can partition an input file at any offsets, parallelize, and adjust partition boundaries to a valid offset independently.
Without the property, parallelization is hard.
This is how mapreduce has been used to process large text files, except at line boundaries.
Now, for this that might not be a useful enough property, given that we already do similar things for newlines, and UTF-8 guarantees ASCII is always recognizable and hence newlines are always recognizable.
2. It might have been more useful in the era of dial-up where we still had occasional corrupted bytes in the transmission.
3. It helps regain sanity if e.g. a background process outputs bytes that get interleaved at the tty. For example, cat a large text file, the write boundaries won't always align at UTF-8 boundaries, then have a background process output get interleaved in an unfortunate way. If it self-synchronizes, it'll knock itself back into sync after a small amount of garbage.
I think at least 7 should be allowed. 42 bits could contain all sorts of information.
"UTF-8000" is still ANSI-safe and although it loses some of the properties such as a header code only appearing once in a byte sequence, it is perfectly viable to encode arbitrarily sized unsigned integers.
I don’t actually know if this is LLM-generated, but phrasing like this is weirdly triggering to me now