Is this one of those things that everyone agrees (apart from the emotionally invested people with the power to change it) is absolutely completely infuriatingly obvious and should be changed immediately?
If this proposal is accepted, \w will still not match the apostrophe character you use in coding (Ascii 0x27, unicode U+0027); Rather, it will match an apostrophe character (U+02BC) that represents a part of a word, intended for conveying the semantics of the apostrophe when used in books and language, not in common code.
U+0027 (APOSTROPHE) is a string (or character) literal delimiter in some environments, and it might be sane (though inconvenient, for typing) to have an environment where U+2018 and U+2019 (LEFT and RIGHT SINGLE QUOTATION MARK) were character/string delimiters, but there is no sane reason U+02BC (APOSTROPHE, MODIFIER LETTER) should ever be a delimiter of any kind.
This article is about the argument that Unicode Committee recommendation that U+2019 (RIGHT SINGLE QUOTATION MARK) is the preferred character for apostrophe in English text is wrong (since, among other things, it breaks detecting matched pairs of quotation marks), and that the preferred character for that use should be U+02BC (APOSTROPHE, MODIFIER LETTER).
There are other situations where pairs of quotation marks can be unmatched. For example, in English it’s common to have an opening quote at the beginning of each paragraph of a multi‐paragraph quote, but only close it on the final paragraph.
Far‚ far too often. I work in a mailhouse‚ and clients will often send requirements in .docx files‚ and data in xlsx files‚ which ‘helpfully’ convert to Windows-1252. “O’Hea” seems to be the worst offender.
Oh god. Never, ever copy-paste from Word documents when running SMS campaigns. Unicode characters are rampant and take up 6x as much character data, so when you're already pushing the limits with your message you end up going well over the 155 char limit for a message.
What? Apostrophe is U+0027. The distinctions created by these extra Unicode characters will never be faithfully reflected in most text, because English speakers only have one key to type them with and don't care.
The author's point directly conflicts with your suggestion of using U+0027. The author correctly states that the apostrophe when used inside a contracted English word should be treated as a modifier letter. There is a Unicode apostrophe character set aside and labelled as exactly this--U+02BC.
The author's point makes sense to me. I agree and my initial thought is that the correct thing to do is recommend U+02BC be the apostrophe.
This is something for the future. It won't be faithfully reflected in text for years or decades.
There's plenty of existing software that already takes those keyboard keys and converts them to a different character based on context. Changing which character is used for the conversion shouldn't be a big deal.
U+0027 has worked for me everywhere I've used it, and no one has complained that I was using the wrong character.
On the other hand, I've had to track down a ton of mysterious bugs caused by software "helpfully" converting characters out of the standard ASCII set, unbeknownst to the user (and they look basically the same)... only a hexdump shows the truth.
Perl 5.22 adds a new kind of word boundary, spelled "\b{wb}" as opposed to plain "\b", that recognizes that apostrophes can occur in the middle of words.
Since there's only one key on the keyboard that represents both single-quote and apostrophe, software has to guess which one you mean from context. I don't think there's anything Unicode alone can do to fix this underlying problem.
> Since there's only one key on the keyboard that represents both single-quote and apostrophe, software has to guess which one you mean from context.
That's not at all true; while "guess from context" is one option, using modifier keys is also an option; there's a lot of software that does this for things which don't have distinct keys. Certainly seen this used for different widths of spaces, and different widths of dashes, even though the keyboard has only the space bar and the hyphen.
Only a small fraction of users know how (and are willing) to use modifier keys like that. I don't think it's practical to assume they'll change their behavior in this case.
sadly, even the use of the shift key seems beyond the ability of most users. just witness all the posts in online forums that look like this.
Fortunately, it does seem to have changed over the years (although I'm not sure to what extent things like auto-correct/auto-capitalisation contribute), so perhaps in the far future when keyboard layouts evolve enough, everyone will be using Unicode apostrophes and single quotes...
The author bases his entire argument on the assumption that "don't" is obviously a single word in English and that there is no doubt about that.
I for one was quite surprised about that. When doing word segmentation for NLP it is common to split these contractions.
Plenty of people argue that it is a single word because there are spaces around it or simply because it is a contraction. Other argue it is 2 words since they are of different word classes.
20 comments
[ 5.2 ms ] story [ 63.1 ms ] threadThis article is about the argument that Unicode Committee recommendation that U+2019 (RIGHT SINGLE QUOTATION MARK) is the preferred character for apostrophe in English text is wrong (since, among other things, it breaks detecting matched pairs of quotation marks), and that the preferred character for that use should be U+02BC (APOSTROPHE, MODIFIER LETTER).
It's enough to drive you crazy.
The author's point makes sense to me. I agree and my initial thought is that the correct thing to do is recommend U+02BC be the apostrophe.
This is something for the future. It won't be faithfully reflected in text for years or decades.
On the other hand, I've had to track down a ton of mysterious bugs caused by software "helpfully" converting characters out of the standard ASCII set, unbeknownst to the user (and they look basically the same)... only a hexdump shows the truth.
That's not at all true; while "guess from context" is one option, using modifier keys is also an option; there's a lot of software that does this for things which don't have distinct keys. Certainly seen this used for different widths of spaces, and different widths of dashes, even though the keyboard has only the space bar and the hyphen.
Fortunately, it does seem to have changed over the years (although I'm not sure to what extent things like auto-correct/auto-capitalisation contribute), so perhaps in the far future when keyboard layouts evolve enough, everyone will be using Unicode apostrophes and single quotes...
I for one was quite surprised about that. When doing word segmentation for NLP it is common to split these contractions.
Plenty of people argue that it is a single word because there are spaces around it or simply because it is a contraction. Other argue it is 2 words since they are of different word classes.