- It'd be nice to have additional transformations: NFKC, NFKD, and maybe some others like CLDR transliteration, confusable mapping, ...
- Characters outside the BMP get broken down as surrogate pairs, which is not what you want. Non-BMP characters they should be treated as a single codepoint; surrogate pairs are an implementation artifact of JavaScript's internal UTF-16 representation.
Well, of UTF-16 itself, not specific to Javascript. What they're exposing are the 16-bit Code Units of UTF-16, which happen to be (reserved) Unicode Code Points, but are not Unicode Scalar Values.
I guess I don't know what this software is for? But the references to the normalisation forms makes me assume it's about Unicode and so it ought to work in the actual Code Points, not these surrogates.
It might be useful to say what it is. Especially when you're on a mobile when you're presented with 2 exactly the same breakdowns and what looks like the bottom of the page.
Character rendering and recognition breaks with characters outside plane 0 (BMP), however, normalization seems to still work correctly. For example, [◌𑄮] u+11131 u+11127 (two characters) is interpreted as five characters but normalized correctly to [◌𑄮] u+11123 (one character) which is nevertheless interpreted as three characters for some reason.
7 comments
[ 5.6 ms ] story [ 30.6 ms ] thread- It'd be nice to have additional transformations: NFKC, NFKD, and maybe some others like CLDR transliteration, confusable mapping, ...
- Characters outside the BMP get broken down as surrogate pairs, which is not what you want. Non-BMP characters they should be treated as a single codepoint; surrogate pairs are an implementation artifact of JavaScript's internal UTF-16 representation.
I guess I don't know what this software is for? But the references to the normalisation forms makes me assume it's about Unicode and so it ought to work in the actual Code Points, not these surrogates.
That said, very interesting. A cool tool to have in your belt.