> Unicode 13.0 adds 5,930 characters, for a total of 143,859 characters. These additions include 4 new scripts, for a total of 154 scripts, as well as 55 new emoji characters.
So how far off is Unicode from being 'done'? At what point will they be able to stop adding characters and scripts?
My wife and I call each other "donkey", and some years ago we used the horse emoji, which was low resolution enough to look as a donkey if you squinted. But modern emojis are too high resolution and it definitely looks like a horse now. So I feel your pain.
Emoji: It's like Kanji, only without agreed upon semantic meaning or pronunciation. I pity historians of the future who have to try and decipher this garbage.
Ha, imagine if some idiots designed a whole country so that its correct functioning depended upon trying to decipher the meaning of symbols which lack semantic meaning decades or even centuries later.
Oh wait, that's the United States of America. What does constitute a "well regulated militia" and how exactly did those symbols come to mean people you wouldn't trust with a sharp stick are allowed semi-automatic hand guns?
Symbols mean whatever people want them to mean, that's the extent to which Humpty is right when he lectures Alice. Whether it's an Eggplant emoji or "Ugandan discussions" the symbol is not the meaning of the symbol.
"214 graphic characters that provide compatibility with various home computers from the mid-1970s to the mid-1980s and with early teletext broadcasting standards"
This part is dear to me, as I helped craft it. It includes 2x3 videotext mosaic characters that will make it much easier to draw large text have have better quality charts in text terminal interfaces.
And, of course, the ability to properly encode documents that were generated in computers in the 70's and 80's that contained those platform-specific characters.
For 14 we are planning on adding symbols from the Sharp MZ series and the large text characters (3x3 cells) of HP terminals.
The available space is closer to 2^20 (0-10FFFF, minus surrogate pairs, depending on whether you are talking about Unicode scalar values or code points).
German, a European language that has been more or less standardized for several centuries, with a Latin-based alphabet, added a new letter (ẞ) to its alphabet in 2017. As long as that continues to happen, Unicode will have to add new characters, even if no more ancient scripts are discovered and no new writing systems are developed for currently unwritten languages.
Small precision for those who don't know the context: the Eszett (which comes from the ligature of 'ss') existed for centuries already in German writing. 2017 is just the date of its official integration in the alphabet, so it's not a 'new' letter created from scratch. I say that because I remember learning it at school a few decades ago (even if at the time we were warned the subject was touchy), and I was surprised it wasn't standardized that earlier.
The Eszett (ß) has been already standardized for several decades (since 1986: with ISO 8859-1 aka latin-1).
The newly added letter is the "capital letter Eszett", which did not exist until recently. One could argue that this new letter is not really needed, as Eszett does not appear in capitalized form except when a word is in all-caps, and was then simply written as "SS".
Standardisation was earlier than that :-) ISO-646 is the 7-bit predecessor to the 8-bit ISO-8859. It dates back to the late 1960s. Like 8859, 646 has several variants, such as ISO-646-DE which has ß where ~ is in ascii. (The trigraphs in C are partly to work around transcoding issues between ISO-646 and EBCDIC variants.)
And some more precision: this is about the uppercase Eszett. The lowercase variant has existed for ages, and before 2017 was officially uppercased into SS or SZ.
It still is officially capitalized to SS under normal circumstances; capital ẞ is an allowed alternative: "Bei Schreibung mit Großbuchstaben schreibt man SS. Daneben ist auch die Verwendung des Großbuchstabens ẞ möglich." (Deutsche Rechtschreibung § 25 E 3)
SZ hasn't officially been an option at least since 1996.
ß is both visually and from its name (‘s’ ‘z’) a ligature of s and z (“Eszett” in English is “ess-zed). An ss ligature would look like a double integral sign. “Sz” seems like a better way to represent that sound, so I don’t know why it morphed into “ss”
Greek has two characters for s, one for use in the middle and one for the ends of words. English lost this in the late 18th or early 19th century (look in the Declaration of Independence for examples). German kept it longer, at least in Fraktur, which included other standard ligatures, even in handwritten text, such as ch, tz et al. The Umlaut mark can also be considered a ligature for E which is how it was originally drawn.
That letter, the capital eszett, has existed in German typefaces since at least 1905:
> Historical typefaces offering a capitalized eszett mostly date to the time between 1905 and 1930. The first known typefaces to include capital eszett were produced by the Schelter & Giesecke foundry in Leipzig, in 1905/06. Schelter & Giesecke at the time widely advocated the use of this type, but its use remained very limited.
Eszett is usually just a lower-case form; it is most often upper-cased to SS, or two capital letter esses, being an example of how changing case does not always preserve the number of letters in a text. Capital eszett is very rare, but it was in uncommon usage in German text, and so it was added to Unicode.
I think that most of the changes in Unicode 13 are not from the evolution of human written language. I don't know anyone who's ever written "blueberries" by drawing a picture of some blueberries in the middle of their text.
Emojis literally are an evolution in human written language. They started with youth texting and are now showing up in business emails. I predict that within 50 years we'll see emojis as a routine component of New York Times articles.
I'm not going to hold my breath on this one. Emojis have an air of informality that is not appropriate in many circumstances. Imagine writing a death notice with emojis
Now you're getting into the definition of "writing". I would say I've only seen emoji typed, not written. (Before anyone asks: yes, I've seen cuneiform written. I have some interesting friends.) If you count any visual communication that is typed on a phone under the greater umbrella of "writing", then we could also include colors, styles, orientation, funny fonts, image memes, animation, etc. There's no end to the possible visual communication that people might want to transmit digitally.
Where do you draw the line? I draw it at "anything in or using a language that people might write in the absence of computers, which they would then reasonably want to store and transmit using a computer". I don't include "any possible visual communication that can occur using a computer". That's far too broad to define "text", or be part of any existing "language", which are the stated goals of Unicode.
From the perspective of a person who uses an alphabetical language, such as English, sure Unicode can be "done". But if your language is based on ideograms, like Chinese, then it'll never be "done". As words are created they need to be encoded.
Again, that's great and I understand that (I've studied Japanese), but that's only part of the new version. They're not adding pictures of "mousetrap" and "olives" and "toilet plunger" because any existing language needs to write these.
Furthermore, I'm really starting to question the way CJK is encoded. We don't make every English word a separate codepoint. 97% of these CJK ideographs are just different combinations of the same few radicals. Korean seems especially weird, as they have both individual radicals and every precomposed triple (in a block that's been rearranged once or twice, on the basis that nobody was really using it yet). I'm not saying we should nix all precomposed Hanzi/Kanji, exactly, because that's a very convenient way for programs to handle text, but it seems like this system is becoming increasingly awkward for non-western languages.
I feel there's a fundamental flaw when our "universal" text encoding system can't handle the regular creation of new words in a well-understood way, for languages spoken by 1/3rd of the world's population. It's like we're issuing hardware patches for a software problem.
It is, I do not like the way CJK is being doubt with. Not to mention fonts dont include All the CJK variants of the fonts when I use the same word but need a JK variant because that is how it was suppose to be used.
Even the "C" has traditional and simplified variant.
Fortunately I think Unicode is pretty much done for Alphabetical languages. Someday if CJK design Unicode isn't good enough breaking it off to something better isn't entire impossible.
Well, it was a can of worms once they took the existing characters which for the most part were culturally very tied to Japan. Now everyone had access to those fun little icons and in turn a lot of people felt concepts from their cultural surrounding underrepresented.
And while every Unicode announcement gets derided because they added more emoji, it's still just a small subset of the standard. Back when they added them I wasn't much of a fan, but by now I think it was the right decision. I still don't use them, but I've heard they're quite popular in younger age groups.
I believe all scripts that are in extant use today are complete; if not, the missing extant scripts are down to the scripts with very few literate people (thousands or fewer).
Many of the additions today, barring emoji, are covering historical usage. This includes things like Medieval scribal annotations, a different set of numbers for the Ottomans, and the Mayan script. It will still be over a decade for the historical work to be complete, since there is often a lot of actual research that needs to be done to understand how an ancient writing system works, which has to come before you can even put together a coherent proposal for a new script.
I think that's a good framework to think about Unicode being "done". But even extant scripts aren't done; consider the Bopomofo additions here in Unicode 13. And it's not clear what "done" even means for Chinese characters.
The Script Encoding Initiative [0] is a UC Berkeley project to add Unicode support for uncommon and historical scripts. They have a list of remaining scripts which is encouragingly short [1], and most of the scripts on that list have proposals in progress so in theory they should be adopted soon
You just gotta feel for the Unicode staff, setting out to improve communication across all of humanity. Turns out what we're REALLY interested in are funny emojis.
I suspect emojis as characters will soon be replaced with graphics. Consumers are annoyed or confused when emojis look slightly different on different platforms.
I'm confused. How would this solve the problem? Whilst looking different is clearly an issue (e.g. the gun being depicted a water gun), I'm not sure what your solution would look like? Images hosted somewhere? You'd get consistency, but you'd lose universality. It goes without saying, but you can use Emoji practically _anywhere_ that support text. That's huge.
Although it depends on having the same level of emoji support across all the relevant systems/devices. If you send a message with Unicode 13 emoji to me and I read it on my phone that's stuck on Unicode 10, it's not much use. Whereas if these silly little images were represented by links to a canonical repository somewhere, newly-added images could automatically work even in pre-existing systems.
On the other hand, that'd be dependent on connectivity.
Yeh, so the current system seems better, because, although it's versioned, it doesn't need to constantly communicate with some central server. Also, another drawback is fonts need to be updated, but again, I think the benefits outweigh the costs.
Arguably it might be better to include a limited, space-optimized vector image format (something like HVIF, Haiku Vector Icon Format[0]) along with text rather than try to enumerate emoji, although there are also advantages to the current approach (smaller size for supported characters and meaning indpendent of representation). For the web there is already data: url and similar inline image options would be possible in many situations even without the basic text format including images.
I will regret this but... Animated ligature emojis or some sorta crontraption... Animated characters is the next stage. If emojis are different between carriers meh.
Course I find it silly that they swapped a real gun emoji for a water gun. How will my friends know I intend to go hunting and not play water guns with my nephews or whatever.
Uhh...perhaps try using actual words if you want to make your meaning clear? After all, if you used a "real gun" emoji, your friends might just think you were planning a massacre...
I usually do, I was kind of poking fun at how people use only emojis to talk, although with the sort of limits that Twitter and SMS has, you sometimes have to cut messages short. I meant using a single emoji in addition to other words.
I think if you ship the font in your app to get consistent emoji renderings across platforms (no need to dump unicode for this) you might have appeased the people who want emojis to look the same on all platforms, but then you will enrage the people who want emojis on all of their apps to look the same.
Interesting. I’d think the opposite: I bet the number of times the average user looks at their own message on someone else’s device is pretty low, while they might have a chance to notice the differences in emoji between app A and app B on their own device several times per day.
> I bet the number of times the average user looks at their own message on someone else’s device is pretty low
That is not the issue. The issue is that users expect the emoji they send to look the same to the receiver.
Geeks know that emojis are "characters" and therefor might look slightly different in different fonts. But this is not how the average user sees it. They see emojis as small images and expect them to transfer unaltered just like any other image they put into a message.
Unified rendering is not really a goal of Unicode. That's better left to the individual platform providers, who have to balance this against keeping their own user experience consistent over time and across devices.
Well you jest maybe, but the vast majority of new characters are not emojis. The Creative Commons symbol, for example. And 200 glyphs from the vintage Apple II, Amstrad CPC and Commodore Amiga computers. There are "only" 55 new emojis. FWIW I love new emojis and can't wait for WhatsApp/Android to add the new ones :-)
I'm personally hyped by the new characters for Creative Common, that could come in handy in data processing (at least in my field).
It's cool the Bopomofo was extended to support Cantonese (didn't know this existed tbh), because it means multilingual Madanrin-Hokkien-Cantonese can use a uniform spelling for word and character readings.
And finally Khitan small script, while ghetto, was an interesting offspring of the Chinese script that targeted a Mongolian language.
PS: I won't comment on the ridiculous emoji shitshow.
> Symbols added in this release (which aren't implemented as emojis) include a Creative Commons symbol, as well as other related glyphs for non-commercial licences, or to indicate where attribution is required.
So you can now indicate license requirements, such as attribution, share-alike, or non-commercial use only. Those can apply to a vast number of works, so does nice to have easily-accessed standard symbols for it.
It also adds symbols for some older machines, which will make discussing them easier.
And yes, it has new emojis. But it is worth noting that the total number of emojis is a very small portion of the Unicode characters.
The CC logo and icons as a matter of trademark law are governed by CC’s trademark policy, which specifies conditions of their use “in a manner reasonable to the medium and context.”
According to Unicode criteria for encoding symbols, a trademark weakens, but does not disqualify, the case for encoding. As such, we would like to address this point directly, in addition to emphasizing the criteria that strengthens the case for encoding. So as to be sure there exists no misunderstanding, CC is seeking inclusion of the CC logo and icons in the Unicode standard notwithstanding that CC asserts trademark rights in its logo and icons.
The aims of Unicode and the CC trademark policy are not antagonistic to one another.
It is perfectly consonant with the purposes for Unicode to allow trademarked logos and icons into its standard without jeopardizing trademark rights of the requesting organization. Especially where, as here, there is a clear and public trademark policy by the submitting organization in place that clarifies when and how the logo and icons may be used, and such marks are ubiquitous.
As such, they were accepted and celebrated by MoMA into its permanent collection,6 alongside universal icons such as the @ symbol and the International Symbol for Recycling—both of which are encoded in the Unicode standard. Encoding the CC logo and icons in UCS would more easily enable creators to mark their works as consistent with CC’s trademark reasonable to the medium and context, in this case within text-based editors.
Was excited when I read your comment, but then looked at the new update and couldn't find an encircled, vertically-flipped C (i.e. copyleft licenses) among the new additions :(
"Support for these legacy computing symbols includes 212 characters added in
Version 13.0 to provide compatibility with a wide range of early home computers, or
“microcomputers,” manufactured from the mid-1970s to the mid-1980s. These symbols
also cover the teletext broadcasting standard originally developed in the early 1970s, and
the Minitel standard developed in the 1980s. This collection of early microcomputer symbols includes support for the character sets of Amstrad CPC, Apple 8-bit, Atari 8 and 16-
bit, Commodore 8 and 16-bit, MSX, Yamaha, RISC OS, and Tandy"
95 comments
[ 2.4 ms ] story [ 187 ms ] threadSo how far off is Unicode from being 'done'? At what point will they be able to stop adding characters and scripts?
I wonder why Gardiner's descriptions never made it into Unicode? https://en.wikipedia.org/wiki/List_of_Egyptian_hieroglyphs#E
O, wait, there are censored genitalia in D block ...
My wifi SSID is <horse U+1F40E><unicorn U+1F984>, which is a barely satisfactory approximation.
(FWIW the FTP site is still up: ftp://ftp.unicode.org/Public/13.0.0/ )
Oh wait, that's the United States of America. What does constitute a "well regulated militia" and how exactly did those symbols come to mean people you wouldn't trust with a sharp stick are allowed semi-automatic hand guns?
Symbols mean whatever people want them to mean, that's the extent to which Humpty is right when he lectures Alice. Whether it's an Eggplant emoji or "Ugandan discussions" the symbol is not the meaning of the symbol.
This part is dear to me, as I helped craft it. It includes 2x3 videotext mosaic characters that will make it much easier to draw large text have have better quality charts in text terminal interfaces.
And, of course, the ability to properly encode documents that were generated in computers in the 70's and 80's that contained those platform-specific characters.
For 14 we are planning on adding symbols from the Sharp MZ series and the large text characters (3x3 cells) of HP terminals.
Found a unicode consortium tweet (!) with a picture of the whole block:
https://twitter.com/unicode/status/1085613123183071232?lang=...
* http://pelulamu.net/unscii/ (https://news.ycombinator.com/item?id=18478350)
The newly added letter is the "capital letter Eszett", which did not exist until recently. One could argue that this new letter is not really needed, as Eszett does not appear in capitalized form except when a word is in all-caps, and was then simply written as "SS".
SZ hasn't officially been an option at least since 1996.
ß is both visually and from its name (‘s’ ‘z’) a ligature of s and z (“Eszett” in English is “ess-zed). An ss ligature would look like a double integral sign. “Sz” seems like a better way to represent that sound, so I don’t know why it morphed into “ss”
Greek has two characters for s, one for use in the middle and one for the ends of words. English lost this in the late 18th or early 19th century (look in the Declaration of Independence for examples). German kept it longer, at least in Fraktur, which included other standard ligatures, even in handwritten text, such as ch, tz et al. The Umlaut mark can also be considered a ligature for E which is how it was originally drawn.
That letter, the capital eszett, has existed in German typefaces since at least 1905:
> Historical typefaces offering a capitalized eszett mostly date to the time between 1905 and 1930. The first known typefaces to include capital eszett were produced by the Schelter & Giesecke foundry in Leipzig, in 1905/06. Schelter & Giesecke at the time widely advocated the use of this type, but its use remained very limited.
Eszett is usually just a lower-case form; it is most often upper-cased to SS, or two capital letter esses, being an example of how changing case does not always preserve the number of letters in a text. Capital eszett is very rare, but it was in uncommon usage in German text, and so it was added to Unicode.
Do you think human written language is ‘done’ and will never evolve?
But you wait until Maya script gets into Unicode. You'll have at least three different Maya codepoints for death. (-:
Where do you draw the line? I draw it at "anything in or using a language that people might write in the absence of computers, which they would then reasonably want to store and transmit using a computer". I don't include "any possible visual communication that can occur using a computer". That's far too broad to define "text", or be part of any existing "language", which are the stated goals of Unicode.
Furthermore, I'm really starting to question the way CJK is encoded. We don't make every English word a separate codepoint. 97% of these CJK ideographs are just different combinations of the same few radicals. Korean seems especially weird, as they have both individual radicals and every precomposed triple (in a block that's been rearranged once or twice, on the basis that nobody was really using it yet). I'm not saying we should nix all precomposed Hanzi/Kanji, exactly, because that's a very convenient way for programs to handle text, but it seems like this system is becoming increasingly awkward for non-western languages.
I feel there's a fundamental flaw when our "universal" text encoding system can't handle the regular creation of new words in a well-understood way, for languages spoken by 1/3rd of the world's population. It's like we're issuing hardware patches for a software problem.
Even the "C" has traditional and simplified variant.
Fortunately I think Unicode is pretty much done for Alphabetical languages. Someday if CJK design Unicode isn't good enough breaking it off to something better isn't entire impossible.
* https://www.emojis.com/food/fruit/
There will always be another Emoji that someone, somewhere wants to add.
And while every Unicode announcement gets derided because they added more emoji, it's still just a small subset of the standard. Back when they added them I wasn't much of a fan, but by now I think it was the right decision. I still don't use them, but I've heard they're quite popular in younger age groups.
Many of the additions today, barring emoji, are covering historical usage. This includes things like Medieval scribal annotations, a different set of numbers for the Ottomans, and the Mayan script. It will still be over a decade for the historical work to be complete, since there is often a lot of actual research that needs to be done to understand how an ancient writing system works, which has to come before you can even put together a coherent proposal for a new script.
[0]: https://linguistics.berkeley.edu/sei/index.html [1]: https://linguistics.berkeley.edu/sei/scripts-not-encoded.htm...
Edit: Apparently HN doesn't support unicode?
I was trying to say:
U+1F637 or U+1F912 not good enough?
On the other hand, that'd be dependent on connectivity.
[0] https://blog.leahhanson.us/post/recursecenter2016/haiku_icon...
Course I find it silly that they swapped a real gun emoji for a water gun. How will my friends know I intend to go hunting and not play water guns with my nephews or whatever.
That is not the issue. The issue is that users expect the emoji they send to look the same to the receiver.
Geeks know that emojis are "characters" and therefor might look slightly different in different fonts. But this is not how the average user sees it. They see emojis as small images and expect them to transfer unaltered just like any other image they put into a message.
1. https://imgur.com/a/iMHYtjB
When they were really simple icons, it made sense to treat them as characters. But today they are detailed illustrations.
I need to check my combining class stuff, it seems.
I'm personally hyped by the new characters for Creative Common, that could come in handy in data processing (at least in my field).
It's cool the Bopomofo was extended to support Cantonese (didn't know this existed tbh), because it means multilingual Madanrin-Hokkien-Cantonese can use a uniform spelling for word and character readings.
And finally Khitan small script, while ghetto, was an interesting offspring of the Chinese script that targeted a Mongolian language.
PS: I won't comment on the ridiculous emoji shitshow.
> Symbols added in this release (which aren't implemented as emojis) include a Creative Commons symbol, as well as other related glyphs for non-commercial licences, or to indicate where attribution is required.
So you can now indicate license requirements, such as attribution, share-alike, or non-commercial use only. Those can apply to a vast number of works, so does nice to have easily-accessed standard symbols for it.
It also adds symbols for some older machines, which will make discussing them easier.
And yes, it has new emojis. But it is worth noting that the total number of emojis is a very small portion of the Unicode characters.
(This is why there is no Apple logo or character for TAFKATAFKAP, for one thing.)
https://creativecommons.org/wp-content/uploads/2016/10/CC-Un...
The CC logo and icons as a matter of trademark law are governed by CC’s trademark policy, which specifies conditions of their use “in a manner reasonable to the medium and context.” According to Unicode criteria for encoding symbols, a trademark weakens, but does not disqualify, the case for encoding. As such, we would like to address this point directly, in addition to emphasizing the criteria that strengthens the case for encoding. So as to be sure there exists no misunderstanding, CC is seeking inclusion of the CC logo and icons in the Unicode standard notwithstanding that CC asserts trademark rights in its logo and icons.
The aims of Unicode and the CC trademark policy are not antagonistic to one another. It is perfectly consonant with the purposes for Unicode to allow trademarked logos and icons into its standard without jeopardizing trademark rights of the requesting organization. Especially where, as here, there is a clear and public trademark policy by the submitting organization in place that clarifies when and how the logo and icons may be used, and such marks are ubiquitous.
Though the CC logo and icons are trademarked, they are a widely-used functional marking tool that indicates permissive use as an alternative to the well-known © symbol, followed by the icons that represent those terms.
As such, they were accepted and celebrated by MoMA into its permanent collection,6 alongside universal icons such as the @ symbol and the International Symbol for Recycling—both of which are encoded in the Unicode standard. Encoding the CC logo and icons in UCS would more easily enable creators to mark their works as consistent with CC’s trademark reasonable to the medium and context, in this case within text-based editors.
For a long time on its homepage placed message banner:
> New versions of all fonts will be posted the release of Unicode 13, March 2020.
[0] https://aur.archlinux.org/packages/ttf-symbola
[1] http://users.teilar.gr/~g1951d/
I predict that within 10 years, some group will start a new simplified text encoding standard which is just for text.
"Support for these legacy computing symbols includes 212 characters added in Version 13.0 to provide compatibility with a wide range of early home computers, or “microcomputers,” manufactured from the mid-1970s to the mid-1980s. These symbols also cover the teletext broadcasting standard originally developed in the early 1970s, and the Minitel standard developed in the 1980s. This collection of early microcomputer symbols includes support for the character sets of Amstrad CPC, Apple 8-bit, Atari 8 and 16- bit, Commodore 8 and 16-bit, MSX, Yamaha, RISC OS, and Tandy"