The author Paul McCann (polm) is one of my favourite programmers out there!
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article...
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
It occurs to me that we can use 彊 to mean "a completely unknown concept that cannot be named". For example if you ask, "when Cthulhu rises from its slumber, what is thefirst thing it will do? Probably it will 彊.
Before that, there was the case of ÿ and Ÿ in the base IBM character set (see https://en.wikipedia.org/wiki/Ÿ). The former exists in a handful of proper nouns in French and Hungarian (as in about five proper nouns), but only as a y with a diæresis, which would normally be handled with a Unicode modifier symbol. It is also used in a couple of endangered languages in South America, which would not have influenced the character getting in to a PC character set. It is thought that possibly it was supposed to represent the Dutch "ij" character, but apparently that is never written as "ÿ" in print. Ÿ seems to not be used by any language.
á (á) is also an 'a' with a diacritical acute accent. If you mean that ÿ should not have a precomposition in Unicode, well, why not, especially if it really is used in "a handful of proper nouns in French and Hungarian".
Remember, the reason we have combining marks is that that is in fact how many of these characters were composed in actual use, especially with typewriters. Heck, 1966 US-ASCII / ECMA-6 (1985), section 5, describes the use of backspace/overstrike in US-ASCII for composition of such characters! That comes from typewriter use. And that's where compose-key sequences generally come from, too.
So it's not at all surprising that given that ÿ has _some_ use, therefore a) it is a valid glyph to construct with combining diaeresis, and b) that it has a precomposed codepoint in Unicode.
See also https://en.wikipedia.org/wiki/A_Book_from_the_Sky (a 1988 art project based on intentionally making up ghost characters by hand and printing a whole book with nonexistent but aesthetically plausible Chinese characters). Amusingly, Wikipedia mentions that over a dozen of them (out of about 4,000) were later found to accidentally exist!
It’s really a complete and total nothing burger. Extra code points were added, might have been an issue when we were trying to cap the total number below needing some arbitrarily fixed number of bytes for convenience, but now that’s no longer the case and they’re just a historical oddity that costs nothing to maintain and certainly don’t “haunt” in the sense of “ keep popping up and causing problems” in any way.
I appreciate the article nevertheless, of course, but I do feel that it would probably be more meaningful to someone that has at least a basic understanding of Japanese.
24 comments
[ 0.19 ms ] story [ 36.1 ms ] threadHe’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
[0] https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ [2] https://spacy.io/
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Why didn't they simly replace the original bad one?
> nine hundred pages. Imagine tracking down a single character without a page reference
Not that hard to imagine, OCR existed back then?
https://en.wikipedia.org/wiki/A_Book_from_the_Sky
á (á) is also an 'a' with a diacritical acute accent. If you mean that ÿ should not have a precomposition in Unicode, well, why not, especially if it really is used in "a handful of proper nouns in French and Hungarian".
Remember, the reason we have combining marks is that that is in fact how many of these characters were composed in actual use, especially with typewriters. Heck, 1966 US-ASCII / ECMA-6 (1985), section 5, describes the use of backspace/overstrike in US-ASCII for composition of such characters! That comes from typewriter use. And that's where compose-key sequences generally come from, too.
So it's not at all surprising that given that ÿ has _some_ use, therefore a) it is a valid glyph to construct with combining diaeresis, and b) that it has a precomposed codepoint in Unicode.
I've seen YouTube videos on this topic before.
> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me.
It's from Unicode version 3.1 (published 2001) so this is surprising.
Major previous discussions:
110 comments: https://news.ycombinator.com/item?id=17637375
130 comments: https://news.ycombinator.com/item?id=24951130
180 comments: https://news.ycombinator.com/item?id=32095502
I appreciate the article nevertheless, of course, but I do feel that it would probably be more meaningful to someone that has at least a basic understanding of Japanese.