Too bad there isn't a standard lib that will flag any non-ASCII characters to be rendered big like those oversized sprites in the "giant" level of Super Mario 3.
So you scroll through source code and hit this weirdly spaced line with a big "h" character and go-- oh-- that's some buggy crap in there.
Then maybe have a map for ranges that you want to include in your allowable set, and we're all good to go. :)
That's only the lexer check. We have further lints that run after parsing to detect Unicode confusables in identifiers as well, as well as two lints for text flow codepoints that can change what the code looks like in editors (one for comments and one for string literals).
Better yet, highlight characters out of context. Not everyone writes in languages that are fully representable in ASCII, yet confusable characters are still an issue.
Potentially this is a big problem. Especially with OCR, transliterations from and across different languages, characters missing their diacriticals, etc.
Whilst I was aware of the problem I wasn't aware that it's as big as it is. Those of us who use Latin scripts are reasonably familiar with the common ones such as o, O and 0 [l/c, u/c alpha & zero], but there's some tricky ones even in these Latin scripts that many of us get wrong.
As I discovered a while back - but I can't remember how - most of us get the World War Two abbreviation WWII wrong (myself included). The 'II' is not two alpha characters as we almost inevitability use but rather it should be the Unicode characters for Roman numerals. Even then I cannot remember if the correct transliteration for Arabic numeral 2 is supposed to to be Roman numeral '1' used twice/repeated or if Roman numeral '2' actually has its own
Roman numerical glyph (I suspect the latter is correct). There are many more instances like this too, the dash, minus sign, en and em dash for instance.
Yes, I could look them up but it's a nuisance to do so on-the-fly and that's the whole point/trouble.
It seems to me we need much better proofing tools that would flag errors or potential errors. I reckon we've been very poorly served in this regard in that there no simple software tools available of the quality we need.
It's not only symbols or characters we need to correct but also typos that don't show up on spelling checkers such as for and fro and the big troublemakers it's and its. Spelling and grammar checkers should automatically highlight or flag such words whether their usage is correct or not.
Unicode does have both upper- and lower-case Roman numerals, U+2160 to U+2188.
But I doubt that the Romans had any clearer of a sense than you of whether "II" was one letter or two. It probably varied across eras and regions, and most people probably didn't think one way or another about it. It's probably not such a big deal to write II when you meant U+2161.
However, I definitely do agree that we are badly lacking in good software tools to help us enrich the set of letters that we can easily use. Imagine if you could type "II" and your text editor could offer to replace it with "Ⅱ".
On the other hand, font support is a limiting factor. For most purposes, II should looks similar to Ⅱ. But in our current world, you are at the mercy of whatever fonts that happen to implement Ⅱ, which might be totally different from how II looks.
Yeah, right. I don't think the Romans would have given it much of a second thought if at all. It's only an issue from our modern perspective, preserving data integrity in electronic records is the principal reason. Whilst 'I' '1' or 'l' are essentially just vertical strokes or lines and there's not much to distinguish between them, especially so with the glyphs of some sanserif typefaces, ascetics also play a role and likely more so with certain other characters. Incidentally, here the spelling checker threw up rôle as an alterative which is a somewhat related point.
FYI, I've somewhat expanded on my earlier comment in reply to CharlesW.
> …most of us get the World War Two abbreviation WWII wrong (myself included). The 'II' is not two alpha characters as we almost inevitability use but rather it should be the Unicode characters for Roman numerals.
FWIW, this example is really, really far down on the list of things you'll be going to typographic and/or semantic hell for. It doesn't impede communication, and I'd guess that using the Roman numeral glyphs would actually break stuff that assumes the usual/normal Latin representation.
From https://en.wikipedia.org/wiki/Roman_numerals#Unicode: "The Unicode standard, however, includes special Roman numeral code points for compatibility only, stating that '[f]or most purposes, it is preferable to compose the Roman numerals from sequences of the appropriate Latin letters.'"
> Potentially this is a big problem.
I wonder how far context can go to mitigate this in OCR applications?
FWIW, this example is really, really far down on the list of things you'll be going to typographic and/or semantic hell for. It doesn't impede communication,...
Totally agree, it's just that on occasions I've seen others raise the matter as an excellent example of how complex Unicode can be. Using a well-known, well used example clearly illustrates the point.
However, the matter of data integrity and machine learning is another matter altogether as meanings can easily be misinterpreted if wrong characters are used. One of the biggest problems of course is the transliterations of scripts both ancient and modern. This is a serious issue for historians and archivists and it's the reason for why Unicode makes such distinctions.
Considering these serious limitations, it's little wonder why OCR is not used as much as it ought to be. Take any text PDF on the Internet Archive that's derived from an image scan of a book and you'll find the error rate to be enormous. Clearly, these errors are why the IA offers books in both text and 'raw' image forms. Similarly, many newspaper publishers have usergroups of readers who correct scans of historical newspapers.
Obviously, this is very time consuming and somewhat defeats the purpose of OCR, it still however is a more accurate and much faster process than retyping the text.
The only real longterm solution is AI/machine learning and even then it won't be perfect. Statistics (hit/error rates for any given text) will come into play. Texts that are converted with, say, a deemed accuracy of 99.99% [my guesstimate] would be exempt from further correction but still marked with '99.99%' in the attached metadata - here the metadata is everything. Others will be flagged for correction with the potential errors flagged.
This needs new conversion format rules for OCR, as I see it, both metadata and the original scans should always be embedded within the 'converted' file. As far as I'm aware whilst work has progressed along these lines we're still a long way from a practical working solution.
You shouldn't try to use every weird dedicated character Unicode has. Usually they are included for full round-trip compatibility with other encodings. (like Ω ohm sign vs Ω omega which is preferred). About Roman numerals Unicode standard says: For most purposes, it is preferable to compose the Roman numerals
from sequences of the appropriate Latin letters.
Before the Romans, the Greeks were already using phonetic letters as numerals. They did saw Iota as either a number or a phoneme depending on the context.
Looking at these older sytems, it looks like the romans just mixed both approaches, either on purpose or not. (I have absolutely no evidence to support this, it's a guess).
Confusables are used by attackers to make malicious domains and URLs apppear innocuous. If you show such things to users, it’s good to highlight confusables.
Unfortunately, this is only single-character confusables. There are an enormous number of permutations missing (ѐ [450] vs è [E8]). The confusable matching isn't even reflexive (A confuse B, but B doesn't confuse A).
There are the NKFC and NKFD normalizations which can resolve most of the confusing things so that, e.g., Ⅵ will become VI, Å will become Å, etc. although it doesn’t resolve B and В to the same character.
Because fuck those people who want to use non-English names, amirite?
Seriously, it's 2022, we have better solutions than ASCII by now. And for what it's worthful, even in ASCII some chars like l, I and 1 are quite confusable.
DNS support is no where near Unicode. At best we get local language support for non-ASCII domains, but everyone else will see it as punycode. Of course, phishing is one good reason why it's this.
> Internationalized domain names are stored in the Domain Name System (DNS) as ASCII strings using Punycode transcription.
One useful table that AFAIK isn’t in the Unicode consortium’s tables is “characters that exist primarily or only for round trip compatibility”.
An example would be the “Roman numerals” set that are discussed elsewhere in this HN discussion. Others include “ghost characters” and obsolete keitai emojis from the 90s that only appear in obsolete character tables. Mostly these should be normalised to modern / canonical versions.
37 comments
[ 3.3 ms ] story [ 87.1 ms ] threadSo you scroll through source code and hit this weirdly spaced line with a big "h" character and go-- oh-- that's some buggy crap in there.
Then maybe have a map for ranges that you want to include in your allowable set, and we're all good to go. :)
Source: https://github.com/gc/confusables Demo: https://confusables.gc.codes/
Too many raw items to process.
Any good guesses what this is about?
EDIT: Got it, the number of strings confusable with the input.
Whilst I was aware of the problem I wasn't aware that it's as big as it is. Those of us who use Latin scripts are reasonably familiar with the common ones such as o, O and 0 [l/c, u/c alpha & zero], but there's some tricky ones even in these Latin scripts that many of us get wrong.
As I discovered a while back - but I can't remember how - most of us get the World War Two abbreviation WWII wrong (myself included). The 'II' is not two alpha characters as we almost inevitability use but rather it should be the Unicode characters for Roman numerals. Even then I cannot remember if the correct transliteration for Arabic numeral 2 is supposed to to be Roman numeral '1' used twice/repeated or if Roman numeral '2' actually has its own Roman numerical glyph (I suspect the latter is correct). There are many more instances like this too, the dash, minus sign, en and em dash for instance.
Yes, I could look them up but it's a nuisance to do so on-the-fly and that's the whole point/trouble.
It seems to me we need much better proofing tools that would flag errors or potential errors. I reckon we've been very poorly served in this regard in that there no simple software tools available of the quality we need.
It's not only symbols or characters we need to correct but also typos that don't show up on spelling checkers such as for and fro and the big troublemakers it's and its. Spelling and grammar checkers should automatically highlight or flag such words whether their usage is correct or not.
But I doubt that the Romans had any clearer of a sense than you of whether "II" was one letter or two. It probably varied across eras and regions, and most people probably didn't think one way or another about it. It's probably not such a big deal to write II when you meant U+2161.
However, I definitely do agree that we are badly lacking in good software tools to help us enrich the set of letters that we can easily use. Imagine if you could type "II" and your text editor could offer to replace it with "Ⅱ".
On the other hand, font support is a limiting factor. For most purposes, II should looks similar to Ⅱ. But in our current world, you are at the mercy of whatever fonts that happen to implement Ⅱ, which might be totally different from how II looks.
MS Word has had this for decades.
FYI, I've somewhat expanded on my earlier comment in reply to CharlesW.
FWIW, this example is really, really far down on the list of things you'll be going to typographic and/or semantic hell for. It doesn't impede communication, and I'd guess that using the Roman numeral glyphs would actually break stuff that assumes the usual/normal Latin representation.
From https://en.wikipedia.org/wiki/Roman_numerals#Unicode: "The Unicode standard, however, includes special Roman numeral code points for compatibility only, stating that '[f]or most purposes, it is preferable to compose the Roman numerals from sequences of the appropriate Latin letters.'"
> Potentially this is a big problem.
I wonder how far context can go to mitigate this in OCR applications?
> Unicode Utilities: Confusables
Totally agree, it's just that on occasions I've seen others raise the matter as an excellent example of how complex Unicode can be. Using a well-known, well used example clearly illustrates the point.
However, the matter of data integrity and machine learning is another matter altogether as meanings can easily be misinterpreted if wrong characters are used. One of the biggest problems of course is the transliterations of scripts both ancient and modern. This is a serious issue for historians and archivists and it's the reason for why Unicode makes such distinctions.
Considering these serious limitations, it's little wonder why OCR is not used as much as it ought to be. Take any text PDF on the Internet Archive that's derived from an image scan of a book and you'll find the error rate to be enormous. Clearly, these errors are why the IA offers books in both text and 'raw' image forms. Similarly, many newspaper publishers have usergroups of readers who correct scans of historical newspapers.
Obviously, this is very time consuming and somewhat defeats the purpose of OCR, it still however is a more accurate and much faster process than retyping the text.
The only real longterm solution is AI/machine learning and even then it won't be perfect. Statistics (hit/error rates for any given text) will come into play. Texts that are converted with, say, a deemed accuracy of 99.99% [my guesstimate] would be exempt from further correction but still marked with '99.99%' in the attached metadata - here the metadata is everything. Others will be flagged for correction with the potential errors flagged.
This needs new conversion format rules for OCR, as I see it, both metadata and the original scans should always be embedded within the 'converted' file. As far as I'm aware whilst work has progressed along these lines we're still a long way from a practical working solution.
We know the Romans were influenced by the Greek, and the Greek by the Minoan.
Both the Greek and Minoan had their own numerals:
https://en.wikipedia.org/wiki/Aegean_numerals https://en.wikipedia.org/wiki/Greek_numerals
Before the Romans, the Greeks were already using phonetic letters as numerals. They did saw Iota as either a number or a phoneme depending on the context.
Looking at these older sytems, it looks like the romans just mixed both approaches, either on purpose or not. (I have absolutely no evidence to support this, it's a guess).
Maybe the problem is really older than we think?
I've been developing a normalization library for Ethereum Name Service: https://adraffy.github.io/ens-normalize.js/test/resolver.htm...
Seriously, it's 2022, we have better solutions than ASCII by now. And for what it's worthful, even in ASCII some chars like l, I and 1 are quite confusable.
> Internationalized domain names are stored in the Domain Name System (DNS) as ASCII strings using Punycode transcription.
https://en.wikipedia.org/wiki/Internationalized_domain_name
https://en.wikipedia.org/wiki/Punycode
An example would be the “Roman numerals” set that are discussed elsewhere in this HN discussion. Others include “ghost characters” and obsolete keitai emojis from the 90s that only appear in obsolete character tables. Mostly these should be normalised to modern / canonical versions.
https://github.com/wanderingstan/Confusables
For example, "𝓗℮𝐥1೦" matches "Hello".
"Hello" gets turned into the following regex of character classes:
[H\Ｈ\ℋ\ℌ\ℍ\𝐇\𝐻\𝑯\𝓗\𝕳\𝖧\𝗛\𝘏\𝙃\𝙷\Η\𝚮\𝛨\𝜢\𝝜\𝞖\Ⲏ\Н\Ꮋ\ᕼ\ꓧ\𐋏\Ⱨ\Ң\Ħ\Ӊ\Ӈ] [e\℮\ｅ\ℯ\ⅇ\𝐞\𝑒\𝒆\𝓮\𝔢\𝕖\𝖊\𝖾\𝗲\𝘦\𝙚\𝚎\ꬲ\е\ҽ\ɇ\ҿ] [l\\|\∣\⏽\￨1\\۱\𐌠\\𝟏\𝟙\𝟣\𝟭\𝟷I\Ｉ\Ⅰ\ℐ\ℑ\𝐈\𝐼\𝑰\𝓘\𝕀\𝕴\𝖨\𝗜\𝘐\𝙄\𝙸\Ɩ\ｌ\ⅼ\ℓ\𝐥\𝑙\𝒍\𝓁\𝓵\𝔩\𝕝\𝖑\𝗅\𝗹\𝘭\𝙡\𝚕\ǀ\Ι\𝚰\𝛪\𝜤\𝝞\𝞘\Ⲓ\І\Ӏ\\\\\\\\\ⵏ\ᛁ\ꓲ\𖼨\𐊊\𐌉\\\ł\ɭ\Ɨ\ƚ\ɫ\\\\\ŀ\Ŀ\ᒷ\🄂\⒈\\⒓\㏫\㋋\㍤\⒔\㏬\㍥\⒕\㏭\㍦\⒖\㏮\㍧\⒗\㏯\㍨\⒘\㏰\㍩\⒙\㏱\㍪\⒚\㏲\㍫\lj\IJ\‖\∥\Ⅱ\ǁ\\𐆙\⒒\Ⅲ\𐆘\㏪\㋊\㍣\Ю\⒑\㏩\㋉\㍢\ʪ\₶\Ⅳ\Ⅸ\ɮ\ʫ\㏠\㋀\㍙] [l\\|\∣\⏽\￨1\\۱\𐌠\\𝟏\𝟙\𝟣\𝟭\𝟷I\Ｉ\Ⅰ\ℐ\ℑ\𝐈\𝐼\𝑰\𝓘\𝕀\𝕴\𝖨\𝗜\𝘐\𝙄\𝙸\Ɩ\ｌ\ⅼ\ℓ\𝐥\𝑙\𝒍\𝓁\𝓵\𝔩\𝕝\𝖑\𝗅\𝗹\𝘭\𝙡\𝚕\ǀ\Ι\𝚰\𝛪\𝜤\𝝞\𝞘\Ⲓ\І\Ӏ\\\\\\\\\ⵏ\ᛁ\ꓲ\𖼨\𐊊\𐌉\\\ł\ɭ\Ɨ\ƚ\ɫ\\\\\ŀ\Ŀ\ᒷ\🄂\⒈\\⒓\㏫\㋋\㍤\⒔\㏬\㍥\⒕\㏭\㍦\⒖\㏮\㍧\⒗\㏯\㍨\⒘\㏰\㍩\⒙\㏱\㍪\⒚\㏲\㍫\lj\IJ\‖\∥\Ⅱ\ǁ\\𐆙\⒒\Ⅲ\𐆘\㏪\㋊\㍣\Ю\⒑\㏩\㋉\㍢\ʪ\₶\Ⅳ\Ⅸ\ɮ\ʫ\㏠\㋀\㍙] [o\ం\ಂ\ം\ං\०\੦\૦\௦\౦\೦\൦\๐\໐\၀\\۵\ｏ\ℴ\𝐨\𝑜\𝒐\𝓸\𝔬\𝕠\𝖔\𝗈\𝗼\𝘰\𝙤\𝚘\ᴏ\ᴑ\ꬽ\ο\𝛐\𝜊\𝝄\𝝾\𝞸\σ\𝛔\𝜎\𝝈\𝞂\𝞼\ⲟ\о\ჿ\օ\\\\\\\\\\\\\\\\\\\\\ഠ\ဝ\𐓪\𑣈\𑣗\𐐬\\ø\ꬾ\ɵ\ꝋ\ө\ѳ\ꮎ\ꮻ\ꭴ\\ơ\œ\ɶ\∞\ꝏ\ꚙ\ൟ\တ]
Commit: https://github.com/wanderingstan/Confusables/commit/c4121428...
https://unicode-org.github.io/icu-docs/apidoc/released/icu4j...
https://unicode-org.github.io/icu-docs/apidoc/released/icu4c...
PyICU (≥ 1.9) has bindings for this (class icu.SpoofChecker).