> Ask the people behind your Operating System and those who design your favourite fonts to start supporting Unicode 9!
Surely not every font has to create glyphs for every Unicode character...how does that work? Is there a kind of "fall-back" font for characters not implemented?
Systems have a way of finding fall-back fonts. At least on OS X (err, macOS) they will satisfy glyph requests in order from the font you specify, some built-in fonts, and then any fonts that have the required glyphs. Finally, a "last resort" font is used to fill in any remaining glyphs[1].
A result of this is that if you ask for a character such as "𓀴" (U+13034 EGYPTIAN HIEROGLYPH A044) in a monospace font, the symbol you get back can be variable width.
On my Mac it gets displayed with "Noto Sans Egyptian Hieroglyhs" which appears to be included in the system (/Library/Application Support/Apple/Fonts/Language Support) but can be downloaded for other platforms from here https://www.google.com/get/noto/
The basic problem is that Unicode characters (which may consist of as many codepoints as you like) have varying width. For instance Chinese has characters that are displayed over a width of two normal monospace characters.
I have a font for that, but the character is unreadably small while it's perfectly fine for latin characters.
Many other unicode symbols also suffer from this problem. E.g. ␀ is the printable version of the unprintable NUL (\0) control character, but it's so small at 13.3px / 10pt CSS font size that it's difficult to distinguish from the other control pictures.
That looks like your browser is somehow misinterpreting the character set of the page. What you're seeing appears to be something like code page 437 (https://en.wikipedia.org/wiki/Code_page_437) rather than unicode.
What browser/os are you using? Is it possible you're going through a proxy that is altering the HTTP headers?
I didn't know there's a whole ligature as a single code point U+FDFD available, thanks sscotth. The code point is also easy to remember.
(It, accidentally, also most probably represents the words first said by the Orlando shooter when calling 911, when the translation in the transcription is compared.)
This isn't even possible for most font formats. There are over 100k assigned code points, but TTF and OTF both use unsigned 16-bit glyph indices, giving a maximum of 65,536 glyphs per font. Not all code points map to glyphs, but some glyphs are also not directly associated to code points (for example, some Unicode combining marks require separate glyphs in certain combinations).
There are some, and I have tried to make one myself [1] to learn OpenType and scripts. It is hard and takes much time and effort to get a good one (Noto fonts [2] are probably the closest font that covers the entire Unicode and looks great). I admit that it is still hard even when you don't have the looks-good requirement.
I wonder if HN is going to complain about the frivolity and uselessness of these the way they incessantly complain about emoji every time a Unicode thread comes up?
As the story mentions regarding the off symbol (a circle), there are many visually identical code points that have different semantic meanings. But in this case, they added an additional semantic meaning to an existing code point.
So which is it? Does each code point represent a visual image? A semantic meaning? Both? It depends? Something else?
I've tried to decipher that on my own and only learned that the answer to these sorts of questions are complicated, because it's very complicated to represent all written human language via one set of rules.
So I know some of the answers to my questions above, but I'm hoping someone with real expertise can provide the fundamental rules/policies - if there are any.
That is not the compromise struck, though; there are even many Cyrillic glyphs that are visually identical to those in Latin, but assigned differing codepoints.
There are multiple reasons for that, one of which is compatibility with previous encodings and standards. If a previous encoding Unicode wanted to be compatible with encoded these as different characters, Unicode needs these to have separate code points for them too.
I didn't dispute that. I just state that trying to remain compatible for the sake of being compatible is a great way to design a convuluted and difficult to understand standard.
Of course, but lack backward compatibility is a great way to make sure a standard is not adopted. For example he reason that UTF-8 'won' is that it has a great backward compatibility story with other ASCII based encodings and systems.
Perhaps Unicode could just have tables listing all relevant sequences of symbols, instead. So "latin letters lowercase" would list the codepoints for a-z in order, for example. Would no longer matter if the codepoints themselves are sequential or not.
(And relevant to my country, "Swedish characters lowercase" would map to latin letters lowercase + åäö.)
Characters have a script associated with them (e.g. Latin), and caseness is also part of a character's properties.
Now, language-specific subsets¹ of those are a bit iffy to deal with. Especially when text can contain loan words from other languages, so in my experience it's rarely a useful thing to ask for.
¹ Yes, subsets. Latin letters lowercase is not the set abcdefghijklmnopqrstuvwxyz. It is the set
How do you condense that again into language-specific subsets? Every letter that appears in a word in a dictionary? Then at least é belongs to German as well, even though it's usually not considered part of the German Latin subset. Unicode stays clear of that issue by simply not defining what script subsets a character belongs to (rightfully so, IMHO).
A pointer would let you ask "does a character lie between 0x12 and 0xBC" but would not hold the character itself; it would make possible to implement different "characters" with the same representation.
But how serious is the problem? How many times do you need to test if a given character is one of the 26 allowed letters of the English alphabet, and where you implement it by testing it against the range?
Typically you write it as "islower_english(c)" once, and be done with it. Is that really hard?
If you do think that's a serious problem, then what of those programmers who need to test for lowercase letters in "España", "München", "Diyarbakır", and "façade"?
I mean, if you really want to get into it, it was a huge pain in the ass at a time when paying the cost of a call to islower_english was much more expensive than a hardware less-than instruction.
We've broadly moved beyond that, but there's still value in grouping sets together in a way that makes certain kinds of frequent tests less computationally expensive than they would be if codepoints were randomly distributed.
Yes, but they have alternate italic forms, for example. Sure, some one of the glyphs like с doesn't have an alternate italic form. Since the other ones do, it would be weird to only assign a separate codepoint to some of them and overlap the others. It would be a workable solution, but still weird.
Which is a bad idea because the characters don't look right unless you use a Japanese font. But if you want to write an article comparing Japanese and Chinese characters, you have to use two different fonts.
Try looking up han-unification and its justification and you'll see the exact opposite approach to encoding characters into unicode.
For CJK characters, they unified all semantically similar han-characters, even when they have visual forms that are quite different between Japanese, Chinese and Korean.
If you want to write Japanese and Chinese in the same document, you need to mark up the section to tell the system that renders it, to render different visual forms for similar codepoints depending on whether they are used in Japanese or Chinese.
> For CJK characters, they unified all semantically similar han-characters, even when they have visual forms that are quite different between Japanese, Chinese and Korean.
This isn't true. 青 and 靑 are the same character written differently; they have their own codepoints. Ditto for a huge number of simplified Chinese characters; 语 is mainland Chinese and 語 is the same character in Japanese.
It is true for lots of characters (so I guess I was being a little hyperbolic when I said "all"), and you cannot rely on choosing the correct code points in order to have a text display Japanese or Chinese. You need to tell your rendering program (often through choice of font) if things are to be rendered with Japanese or Chinese forms.
I wouldn't know how to show you examples here, as 直 will 直 display the same since they have the same code point, but different number of strokes in japabese and chinese.
Aren't they putting the disunified characters into the U+2xxxx plane now?
Han unification is generally seen as a bad choice in retrospect, but it was something Unicode had to do when it looked like 2^16 codepoints were all they were going to get.
Never heard of that, but I would appreciate if all the characters with different glyphs had different codepoints. Do you have a source? Do you know what happens to the "unified" code-points?
It is true to some extent. While 青 and 靑 have different codepoints, there are plenty of characters of the same codepoint that are rendered differently depends on the language specificed:
Han characters that are traditionally viewed as variants of one another, or that are simplified from more complex logograms (such as 龜, which was simplified into 亀 in Japan and 龟 in mainland China) tend to have different codepoints, but the stylistically different ones usually belong to the same codepoint.
Han Unification "rules" were an inconsistent mess, but I do know that in Japanese 靑 was at one time a printer's simplification of 青, so you could find either in texts, and the Consortium tended to encode a character separately if you could find printed examples of both in the same language.
"every code point represents a semantic meaning" is completely consistent with the notion that some code points e.g. have differing visual representation depending on their position in the word.
The emoji code points can be represented differently on different systems given their meaning.
So it makes sense to have different emojis for different 'meanings'.
The 'moon' switch here does no mean 'moon' - it means 'standby' or whatever.
It may look noticeably different on different systems.
Think from a design perspective: you have 5 emojis to represent 'clouds, sky, earth' etc. - and the a different set of 5 to represent 'on, off, sleep, shutdown'. Those icons will be markedly different in terms of representation, groupings, colour coding, underlying functionality if they are integrated into an experience in any meaningful way.
Text your car with the 'shutdown' symbol to tell it to shut down.
Your bot texts your friend with a moon symbol to tell him you're asleep. Or whatever.
But that doesn't explain the inconsistency in the current case.
So if a system wants to render "on" differently than "straight vertical line", that's possible.
However, if "off" should be rendered differently than "circle", that's not possible. (Or only possible with out-of-band information or modifier characters which would still have to be defined)
It's a mess. If you want to write a document in Japanese that talks about a Chinese character which is written differently than its Japanese version, you can, or can't, achieve this in Unicode, depending on the character, its history, and the mood of the consortium the day it was assigned.
The reality is that Unicode is governed by people, some of those people are grumpy reductionists who push for a minimum of symbols and a maximum of meaning-overloads, and others are more liberal and tend to advocate the opposite, and the result is a compromise, and is in areas very messy.
Do note that they did not include a generic "off" symbol, they included the IEEE 1621 off symbol - which must be rendered as a circle; while on the other hand the IEEE 1621 on symbol must be rendered in a manner that is often different from just "straight vertical line" in particular regarding the corners of that line.
maybe there should be like a universal unicode for all the icons that apps might need that represent common functionality and they should be animated to represent an either on or off state for some of them. also top brand square logos should be added into the unicode as well. and then different forks/variations can be submitted for the unicode and be accepted if they are useful and good looking. also they should all be black and white and have the same theme/look similar. u like my idea?? :DD
https://materialdesignicons.com/ makes a font available with a reasonably large selection of brand and functionality icons, using unicode private ranges (also svg or whatever).
I don't think i personally agree with brand logos in Unicode.
I'm still waiting for a Unicode codepoint for Love Symbol #2 (aka The Artist Formerly Known as Prince). There are codepoints for dead Chinese emperors, there should be one for Prince.
I already ranted about unicode earlier today, my main argument is, that unicode is what happens if everybody qualified thinks: "That's a great idea, of course you have to handle X and Y and Z and I just remember that I forgot to fill out several warranty cards."
This blog post is a nice example, I have absolutely no idea how these new code points are supposed to look like, since I only spend an afternoon to implement the unicode best practices from the Arch wiki, instead of subscribing to some unicode standard mailing list. (Except the one symbol which was redefined to a symbol that does not carry the semantic meaning of "standby symbol" anywhere outside of the unicode standard.)
In my opinion there are two ways forward, one burn the entire thing. Or alternatively, force the unicode committee to produce an authoritative and complete font, in triplicate, and in their own blood.
Well, if you want to know what they can look like, the blogpost has images, an embedded webfont and links to the reference font for the new symbols. And AFAIK providing reference images that are freely usable is required for all new symbol proposals.
It does not work on Firefox with noScript. (And as a matter of fact, prohibiting random blog posts from rendering unicode, executing complex numerical calculations on my graphics card or delivering exploits is kind of the purpose of noScript...)
NoScript does not prevent pages from "rendering Unicode", whatever that's supposed to mean. I use it myself, and the only thing it does is selectively block Javascript.
I think that the BMP -- Basic Multilingual Plane or the first 16-bit of Unicode characters -- is pretty reasonable, and covers fairly well everything we may consider as text (all alphabets in current use plus mathematical symbols). Anything beyond that, from emojis and pictograms to ancient Greek musical notation is pretty... weird.
I think it would have made much more sense to have something like image tags: a special codepoint would introduce a link to a URL containing a sequence of glyphs, followed by an index into that sequence. Those glyphs would be guaranteed not to change (in any meaningful way), and devices would be free to cache them. This way, anything that isn't real text, would standardize representation, too, instead of just a vague "meaning". Another standard could relate those glyphs to one another in some way, giving them standard semantics and means of translation (i.e. "Egyptian hieroglyphics"). This would also allow each of those (emojis or hieroglyphics) to evolve their standards independent of a single universal standard that means little.
I mostly agree. In my opinion, separating code points from encodings (without providing a replacement for txt files) was the original sin of unicode. That just adds a lot of complexity with very little gain. (Suppose we would have standardized from ASCII to a 64 bit per character standard encoding, then text files would grow by a factor of 8 and nothing important would have happened.^1)
^1 I am pretty sure that some use cases actually profit a lot, but neither text file formats (since most text does not contain 2^64 different characters zip would work nicely) nor networking (since most data on the internet is either video or torrents) seem to be among them. So probably they would not be huge fields.
It's not bad, but it's complicated, as it requires an O(n) algorithm to jump to a specific character. Unicode should have been capped at 16 bits, and doubling text files in size is fine. An alternate representation of simplified UTF-8 would have kept compatibility with old ASCII files.
Doubling text files is a waste for the most part, but what makes it tolerable is compression. Still 16 bits would not be enough, it's only 65536 different code pages, less than half of what is currently in Unicode. 24 bit is sufficiently out of allignment with modern hardware and algorithms, so 32 bit it is for efficiency. That is now 4 times the size, and compression is now a requirement.
In any case, Utf8 has a place, and if you want easy manipulation and search, convert it to Utf32 - it's fixed width.
> Still 16 bits would not be enough, it's only 65536 different code pages, less than half of what is currently in Unicode.
But that's only because Unicode has significantly ventured well beyond what we consider to be text. The BMP is enough to represent all text (including math).
BMP is mostly filled with symbols from logographic languages, and currently has around 100 free code points. 16 bits simply isn't enough for the scope of capturing all written language of the history.
But I don't think all written languages in history should have the same treatment when it comes to standardized data representation, or should all be standardized by the same body. It's OK to have alphabets no one has used for thousands of years other than specialized researchers standardized separately from Latin or Chinese alphabets.
But if you don't what would the point of a unified standard be? What should happen to glyphs that noone have used outside of research for 100 years? 1000 years?
They should all be standardized under separate standards. You can call them "extended text", if you like. Mathematical notation isn't really supported by Unicode, either (just mathematical symbols), and that's fine. Math should have its own standard, and so should hieroglyphs, emojis, and musical notation.
What happens if you want to use two different extended text code points in one blog post? How would they interact? How do they avoid assigning the same codepoint to different symbols.
How would browsers support this? What's the actual plan, not just a handwave? Do you think it'd be more efficient to have to support 6 different standards than one?
> What happens if you want to use two different extended text code points in one blog post?
There are no codepoints if it's not text. How do you use codepoints for embedding a video or a picture on your blog? You don't! But, if you want to treat something as if it were text, then I suggested doing something similar to an XML namespace: "the next segment is hieroglyphics, you can get their glyphs from here, and these are their indices...". That "extended text" is still not text, and it still doesn't use any Unicode codepoints, but it can work according to similar principles.
> Do you think it'd be more efficient to have to support 6 different standards than one?
Then why don't we let the Unicode Consortium take over standardizing video or audio? If something isn't text, why is it standardized by a text standardization body?
If you think of Unicode as a standard that enables the ease of exchange of textual data among parties, would that change your mind? People have been putting emojis alongside text (think SMS) long before Unicode put them into the standard at the request of Google and Apple.
People have been putting images alongside text since the invention of print. That still doesn't make the images text. So it's really great that for a few years now some people are embedding icons in their SMS messages, but I think that incorporating those fashionable 2-5-year-old icons into a standard that's mostly about standardizing hundreds-of-years-old text doesn't make much sense.
You can exchange text with embedded icons just as easily without requiring OS vendors to come up with their own versions of vaguely-defined pictures by... simply embedding pictures.
I can already see the people on whatever would be the HN 20 years from now complaining how "bloated" Unicode is, full of thousands of symbols that no one ever uses, and calling to replace the whole thing, costing the industry even more money to replace a standard yet again.
If compression would be a requirement for 32 bits, then it was definitely a requirement for 8-bit text 25 years ago when memory and bandwidth were typically 1/1000th of today. Of course it often wasn't, and isn't. And where it is, it's still a requirement with UTF-8.
ASCII is only 7 bit so UTF-8 is fully compatible, at least when compared to ISO 2022 and other similar horrors from the same era. Are you thinking of other encodings?
Emojis and hieroglyphs aside, 16bit was not enough for CJK characters. It's a real world problem---a character set that can't spell people's name or location can't be universally adopted.
How do you define "character"? You typically need to take into account grapheme clusters / composed character sequences, which make jump-to-character O(n) regardless of encoding.
> it requires an O(n) algorithm to jump to a specific character
If you are trying to index into a string by "character" you are almost certainly already doing it wrong. Meaningful indexing almost always has to be by grapheme cluster. See Swift's string API as a great example of this done right.
That's nice, except the BMP doesn't encode all of Chinese, and it includes a number of weird control characters for compatibility with ASCII. Like Vertical Tab. Who uses Vertical Tab anymore?
The dream of a 16-bit Unicode washed up on the rocks of CJK scripts. It's dead and it isn't going to be revived. You can argue for a simpler standard, with fewer assigned codepoints, but the original BMP isn't it and was never going to be it.
The Unicode tables include examples for all graphical code points: http://unicode.org/charts/. If you really wanted you can make them into a font (most of them seem to be vectorized), but since I'm guessing you see most of the added code points as useless why do you care if they show up as boxes? What harm is this stuff causing or going to cause to the standard? We have hundreds of thousands of unassigned code points.
Meanwhile, a lot of the "Ys and Zs" added to Unicode have proved to be extremely useful. Unicode's math operator and letter-styling support is what made MathJax (and more generally MathML) possible. They've also helped big time when it comes to accessibility (e.g. screen readers) for mathematics on the internet. Should we have shunted that off to another standard and made the creators of screen readers completely restructure their offerings so they can deal with Unicode characters and "Mathicode" characters? Assuming anyone bothered to implement it, how would that be better than just adding a Unicode category and spending a meager amount of space?
Your answer illustrates my point perfectly, first the lack of a reasonable fall back mechanism. Of course I can start a hex editor, get the utf-8 encoding and then look up the code point ( and theoretically add that character to a open font). A default font would just ship with every OS out there, and suddenly there would be a working fall back.
Second mathematical symbols, consider the case were I get a text file considering mostly of ASCII 7 and some mathematical symbols which may render as mathematical symbols or as Chinese characters, since there is no way to specify the encoding in a text file and so I have to guess the encoding. (That is not helped by the roughly 17 standardized encodings that mostly agree with utf-8.)
I checked your link, then I proceeded to the terms of service and IANAL there they claim that all 'unicode software' is basically MIT licensed. (Please check with a lawyer before you conclude that word is MIT licensed.)
Fonts
The shapes of the reference glyphs used in these code charts are not prescriptive. Considerable variation is to be
expected in actual fonts. The particular fonts used in these charts were provided to the Unicode Consortium by a number
of different font designers, who own the rights to the fonts.
See http://www.unicode.org/charts/fonts.html for a list.
Terms of Use
You may freely use these code charts for personal or internal business uses only. You may not incorporate them either
wholly or in part into any product or publication, or otherwise distribute them without express written permission from
the Unicode Consortium. However, you may provide links to these charts.
The fonts and font data used in production of these code charts may NOT be extracted, or used in any other way in any
product or publication, without permission or license granted by the typeface owner(s).
The Unicode Consortium is not liable for errors or omissions in this file or the standard itself. Information on characters
added to the Unicode Standard since the publication of the most recent version of the Unicode Standard, as well as on
characters currently being considered for addition to the Unicode Standard can be found on the Unicode web site.
Now you can try to track down the actual typeface owners one by one (this alone seems hopeless) and even then I doubt that you can get permission from all of them.
> Second mathematical symbols, consider the case were I get a text file considering mostly of ASCII 7 and some mathematical symbols which may render as mathematical symbols or as Chinese characters, since there is no way to specify the encoding in a text file and so I have to guess the encoding.
What does that have to do with Unicode adding anything? Are you really claiming that if we threw out Unicode like you recommend, and (if I'm understanding your point correctly) choose an encoding for the new version that looks nothing like ASCII the encoding mess would get better? I think continuing the migration of most transmission of text to UTF-8 and explicitly specifying encodings for everything that needs to stick with Latin-1, etc. is a better option, unless you propose codifying the new encoding in law to force adoption.
But all mathematical symbols, except for styled math letters (which would have been equally well served by simply rendering them in italics or in a special math font), were already in the original 16-bit Unicode, as are any characters humans normally associate with "text" (all alphabets except for Egyptian hieroglyphics and other extinct alphabets). How useful is it to standardize hieroglyphics, ancient greek musical notation, and emojis as standard text characters esp. without standardizing their screen representation?
> How useful is it to standardize hieroglyphics, ancient greek musical notation, and emojis as standard text characters esp. without standardizing their screen representation?
In the same way it's useful to standardize letters in various alphabets without standardizing their screen representation. There is semantic content associated with each of these symbols that persists even if there is significant variation in how they are presented. Of the ones you list, emojis are the only ones where this is any more a problematic approach than it is for letters in various alphabets. And as people who don't approve of Unicode adding emojis like to point out, emojis aren't that critical so having some loss in the translation isn't a huge deal.
Remember that before emoji standardization various cell phone manufacturers (particularly in Japan if I remember correctly) started using codepoints for whatever they pleased. The alternative to Unicode not standardizing them was to have a repeat of the OEM font gold rush in the SMP.
> styled math letters (which would have been equally well served by simply rendering them in italics or in a special math font)
That was my first reaction as well, but there are a few problems with that approach:
* Math italic characters look very different from normal italics, and are shaped and kerned very differently because they are commonly used for single-letter variables which will be juxtaposed together in expressions. If your goal is to be able to preserve some math formulas in a purely line based text format, preserving this aspect makes a big difference in readability.
* Many of the math letters and "letter-like symbols" have associated semantic content (like bold for vectors), which it makes sense to preserve. MathML alleviates this to a significant degree but I don't believe these codepoints were intended only for MathML usage.
* On the technical side, OpenType math fonts need to carry associated metadata for many of these characters. Putting them in separate fonts complicates this, since these tables need to refer to glyphs (general codepoints are unsuitable in a number of cases) and each font file would have a different glyph address space.
> In the same way it's useful to standardize letters in various alphabets without standardizing their screen representation.
I disagree. Say the name of a letter in any alphabet, and people will draw it in ways that are similar enough for automatic recognition. This is not true for pictograms and emojis.
> The alternative to Unicode not standardizing them was to have a repeat of the OEM font gold rush in the SMP.
I disagree. The alternative is a much simpler and faster standardization, of the kind I offered here: https://news.ycombinator.com/item?id=11958903 There is absolutely no need for a fixed codepoint for most of the non-BMP characters.
> If your goal is to be able to preserve some math formulas in a purely line based text format, preserving this aspect makes a big difference in readability.
So is rendering text in Arial vs. Comic Sans, but they haven't made separate codepoints for those.
Also, where this makes a lot of difference, would count as "specialized usage". I don't think it makes sense to have a single universal standard to standardize all specialized usage of human-readable data.
> I disagree. The alternative is a much simpler and faster standardization, of the kind I offered here: https://news.ycombinator.com/item?id=11958903 There is absolutely no need for a fixed codepoint for most of the non-BMP characters.
So you think instituting a system based on links not rotting would better preserve meaning? Not to mention that:
* Every text renderer that doesn't support your codepoint now displays a full URL, instead of a box, making text using these emojis very difficult to read.
* Instead of making implementation easy by requiring nothing new of text shaping libraries, they now have to be able to both connect to the internet and tie into a file cache.
The supplementary space was already there when we got to emojis, and UTF-8 and UTF-16 already had to deal with SMP codepoints for some of the less common CJK characters. Not everything above 0xFFFF is "weird" non-human language stuff. If the choice was "stick with UCS-2 and be totally fine language wise, or add more bits just for emojis and pictograms" I'd probably agree with you. If you think that's what happened, your timeline for this process is way off.
> So is rendering text in Arial vs. Comic Sans, but they haven't made separate codepoints for those.
Sure, but the different letter types carry crucial meaning in math formulas. "sup" in upright letters is the math operator supremum, "sup" in math italics is s * u * p. This kind of thing applies to every one of the mathematical letter variants.
> Also, where this makes a lot of difference, would count as "specialized usage". I don't think it makes sense to have a single universal standard to standardize all specialized usage of human-readable data.
You're zooming way out on this one. Math symbols have a lot more in common with letters and "normal" symbols than "all specialized usage of human-readable data". Remember that when Unicode added the math symbols, things like MathJax were simply impossible. Being able to write at least some formulas, which consist of letters and symbols, without losing tons of semantic information seems like exactly the kind of thing character encodings should do.
> Every text renderer that doesn't support your codepoint now displays a full URL, instead of a box, making text using these emojis very difficult to read.
It can still display a box.
> Instead of making implementation easy by requiring nothing new of text shaping libraries, they now have to be able to both connect to the internet and tie into a file cache.
The question is implementation of what. I think that it is not an onerous requirement from applications that need to display emojis or ancient Egyptian hieroglyphs. Their OS could provide this service for them just as it allows them the use of fonts.
> The supplementary space was already there when we got to emojis, and UTF-8 and UTF-16 already had to deal with SMP codepoints for some of the less common CJK characters. Not everything above 0xFFFF is "weird" non-human language stuff.
That is a good point, one of which I was not aware, but I still don't think it justifies standardization of Chinese characters, emojis, and ancient Greek musical notation by the same standards body.
> Sure, but the different letter types carry crucial meaning in math formulas.
The use of italics in text may also carry crucial meaning. But if a textual representation as sup is supported, I don't see why the specialized rendering should be supported, too, but for math and not plain text.
> Being able to write at least some formulas, which consist of letters and symbols, without losing tons of semantic information seems like exactly the kind of thing character encodings should do.
I agree, but I think that that semantic information is preserved when writing N or NN instead of 𝑵 or ℕ. Considering that Unicode isn't enough to write most mathematical formulas convenient forms anyway and requires a specialized renderer anyway, I don't see the reason for this extra effort.
If it knows about your new codepoint. Everyone using an implementation that doesn't yet support it is going to show the full URL. If history repeats itself, these implementations will be the majority for at least a decade.
> The question is implementation of what. I think that it is not an onerous requirement from applications that need to display emojis or ancient Egyptian hieroglyphs.
Except they need to do literally nothing different. Hieroglyphs are vectorized, and emojis are either bitmapped (which existing font formats already supported) or vectorized. None of this required a single line of code in any text shaping library to change. Text shaping libraries generally don't even need to understand Unicode categories or other metadata: font files already contain all the relevant information (directionality, combining mark, etc.).
> The use of italics in text may also carry crucial meaning. But if a textual representation as sup is supported, I don't see why the specialized rendering should be supported, too, but for math and not plain text.
If you scrub all bold and italics from text, do any of the words turn into different words? That's what happens with sup (or other named operators). Same thing for blackboard letters, fraktur, etc.
> I agree, but I think that that semantic information is preserved when writing N or NN instead of 𝑵 or ℕ. Considering that Unicode isn't enough to write most mathematical formulas convenient forms anyway and requires a specialized renderer anyway, I don't see the reason for this extra effort.
My point is it was better than nothing, which was the alternative when Unicode added these. By adding mathematical characters that worked exactly the same as all other characters, we could get some of the advantages of MathML without needing everybody to implement a special math renderer or learn any new markup, just download new font files.
If getting everyone to accept MathML or something similar in an expedited fashion was a reasonable proposition, then I might agree that they should've kept it out of Unicode. Those were not the facts on the ground when this decision was made. Note that even now that we have MathML, the few browsers that support it (IIRC Firefox & Safari only) have complete shit implementations that look terrible.
The crux of this is that making changes to a standard to support something new and propagating the changes is super hard; getting new standards to get accepted and implemented is a Herculean effort. Unicode's expansion into these domains required nobody to do anything differently, let alone decide to up and write an implementation of a completely different standard. I think this is a case were our alternative was to let the perfect be the enemy of the good, or at least the working.
> That's what happens with sup (or other named operators).
And that's what happens when scrubbing subscripts or binomial coefficients. When you want to represent math as text, you need to change your representation (add multiplication signs, forgo subscripts, use confusing parentheses etc.). This is still true with Unicode. The contribution of the non-BMP special math characters is quite minimal.
> The crux of this is that making changes to a standard to support something new and propagating the changes is super hard; getting new standards to get accepted and implemented is a Herculean effort.
Sure, but the emoji craziness continues, and there's little sign it would ever stop. Instead of saying "this isn't text; if you want, call it 'special text', escape it, and let a different body standardize it", the body entrusted with standardizing text representation worries about how to represent a picture of two people kissing as text. What next? Kids would want to add tunes to their text messages. Would the Unicode Consortium add code points for MIDI? And maybe managers would want to standardize code points for organizational diagrams. Would that be the consortium’s responsibility, too? The BMP contains all the characters for reasonable text-art.
Unicode has added 1791 emojis[1]. Note that it has used significantly fewer codepoints than this for emojis, because some (e.g. flags) are done via a small number of combining marks (flags use 26 code points for letters in country codes). They've been slowing down the rate of additional emojis since they started doing this. Do you really think they'll accelerate at some point and use up the almost 1 million unassigned code points Unicode has left? Not to mention the fact that many UTF8/UTF16 implementations are already fine with full 32-bit code points, instead of the 21 bits Unicode I'm using above (they have said they'll only ever use 21 bits, but the option is there).
Suffice it to say this getting "out of control" and eating up the remaining space in our lifetimes, or our children's lifetimes would be pretty impressive. This means that the worst we have to fear is more boxes, assuming OSes don't keep up with their fallback fonts. Saying adding more symbols-designed-to-be-just-kind-of-placed-in-line-with-other-symbols-like-they-always-have-been is the first step towards MIDI and organization diagrams is like saying you're vegetarian because it's a slippery slope from eating meat to eating people.
Also note that unlike every other excess of the Unicode standard, these would require massive changes to the code that handles text. This means that if Unicode decided to do this, you wouldn't have to worry about negative effects because nobody would implement it.
I'm not afraid of running out of codepoints. I just think it is misguided that non-text is standardized as text. It just doesn't make sense. I don't know how people will communicate 50 years from now, but I think it's funny that even then, text strings would still need to support all those vegetables and hand gestures. Written text is pretty much eternal; emojis and pictograms? I doubt it. It doesn't make sense for them to get a similar treatment.
> Remember that before emoji standardization various cell phone manufacturers (particularly in Japan if I remember correctly) started using codepoints for whatever they pleased.
Things were way worse than that: to add emoji to text, NTT DoCoMo used private-use codepoints, AU used embedded image tags and Softbank wrapped emoji codes in SI/SO escape sequences.
It hasn't disappeared (Google Ngram viewer shows a consistent occurrence of "led" since 1800). It's just a common error due to (1) inconsistency with "read" (same spelling for present and past tense) and (2) the noun "lead" being pronounced "led".
Unicode symbols... seems like we should've developed them the way languages develop: start with the most important symbols, ones for food, water, shelter, danger, etc, then expanded them into the abstract mess they are today.
Emoji were not developed haphazardly. They evolved naturally in Japan, then were adopted by the rest of the world. That is why there are so many Asia / Japan themes in the standard emoji set. The problem is Westerners don't understand the Japanese emotion behind the symbol. The symbol for bookbag looks exactly like a Japanese school kid's backpack. It's why there is a kimono. Bamboo wind chimes. Tsunami. Shinkansen... I could go on and on.
In some respect, they are getting jumbled up because of international pressures for the base emoji set to be stretched into a be-all for the global market. An example is Taco. There are tacos in Japan. They are hard to find and when you do find one, you definitely don't want to eat one there. Mexican food is one of the rare cuisines the Japanese don't do better.
But why? The trend towards putting icons into Unicode may be a mistake. Unless it's a symbol one uses in a sentence, there's no real reason to have it in Unicode. Unicode should not be viewed as a standard clip art library.
If I understand correctly, Unicode only provides the semantic meaning, not the actual rendering. The font provides information for how to render it. Am I right?
Yes, though they provide guidance on how they should look, and in most cases, there is only one font on the system that has a glyph for some of these more esoteric code points, and it usually provides a reasonable representation.
I believe they are much more legitimate symbols to put to Unicode than Emoji. They are standardized (IEEE 1621), they frequently appear in the running text (of particular kind, but widespread enough), and they have distinct semantics from existing characters and symbols. In terms of worthiness they are on par with math symbols, enough for justifying the inclusion.
What is "plain text format" though? If 'text' isn't limited to Western ASCII characters (which it very obviously shouldn't be considering many people use other character sets), then the idea of a text standard should be to encode all the glyphs people use, so "plain text format" becomes a canonical list of all the communicative symbols in all languages. That's what Unicode aims to be.
In my opinion, if they're used for communication, it doesn't seem unreasonable that such a canon of characters should include universal iconographic symbols like the standby icon.
I hear you. If you want pictures then use a markup language. Unfortunately it is too late now. We finally had an almost universally supported character set, and then we ruined it with levitating men in business suits. Recent Unicode versions introduce far more technical challenges than they solve. For instance, now that code points can come with colour, there are conflicting requirements between the requested text colour and the intrinsic colour of a symbol.
This is about icon fonts and font rendering and has nothing to do with Unicode. Code points don't have colour; there's no colour requirements anywhere in the spec.
Your fonts don't have to support the entirety of Unicode. That's why we have font stacks and fallbacks.
U+1F499 BLUE HEART
U+1F49A GREEN HEART
U+1F49B YELLOW HEART
U+1F49C PURPLE HEART
U+1F53D DOWN-POINTING SMALL RED TRIANGLE
U+1F536 LARGE ORANGE DIAMOND
U+1F537 LARGE BLUE DIAMOND
...
How did the Unicode Consortium turn around. I remember 10 years ago they were refusing to add standard media icons because
>The scope of the Unicode Standard (and ISO/IEC 10646) does
not extend to encoding every symbol or sign that bears
meaning in the world.
>This list has been round and round and round on this -- regular as clockwork, about once a year, the topic comes up again. And I see no indication that the UTC or WG2 are any closer to concluding that bunches of icons should start being included in the character encoding standards simply on the basis of their being widespread and recognizable icons.
>Where is the defensible line between "Fast Forward" and
"Women's Restroom" or "Right Lane Merge Ahead" or
"Danger Crocodiles No Swimming"?
Unicode is supposed to include symbols that appear in "running text", not standalone icons. So no on traffic signs for instance. (There are exceptions for historical reasons. And emoji are a totally separate story.)
I'm not sure about the 'running text' thing, but in my view Traffic Signs are not globally universal (yet), so you'd have to have regional variants which is impractical.
In all seriousness, I'm not sure emoji's really belong in text encoding. Even though it's more convenient, based on where they're most frequently used I don't think they need to be universal.
The "universal" in Unicode means that it aspires to include all symbols used in any form of text; not that it should only include symbols that are used in all forms of text.
1) everybody uses them on their phones, they're in Unicode, consistent and compatible between devices and messaging programs. In the far flung future, researchers will be able to study their linguistic role in communication, confident in understanding what the characters were.
2) everybody uses them on their phones, they're proprietary fonts and codepoints (in the Unicode private use area if you're luck, just random data if you're not), there's no consistency between phone models, manufacturers, or cell networks. Future researchers can pound sand.
We were at #2 pre-Unicode. It was a goddamn mess, especially in Japan. Lord knows why anyone would prefer it. There's no value in being a snob about what kinds of incredibly frequently used characters we think are Worthy of inclusion, imo.
3) People who love colourful images will use stickers in Facebook Messenger, LINE, Viber, and soon iMessage. I'm sure WeChat has them too.
It's basically like 2), except we've moved from proprietary codepoints to proprietary protocols.
I don't mind characters like and or even good old ︎ (which has always been too tiny for its own good). These work in black and white, in different artistic styles, and they're a fairly limited set.
But now we're going down the road where we get new stuff like tacos and unicorns every year. And even though Unicode is an industry standard, the pictures need to look like Apple's bitmaps to avoid confusion, and the Unicode standard changes so often that you have to manually keep track of who can already see and whose computer/phone/browser/messenger software is too old.
Oops, thanks. Well, that explains why I've never seen Emoji on Hacker News. And I've missed the edit window, so I can't fix my post.
Should have been:
> I don't mind characters like ((yellow smiling Emoji)) and ((thumbs up Emoji)) or even good old ︎((pre-Emoji Unicode smiley)) (which has always been too tiny for its own good). These work in black and white, in different artistic styles, and they're a fairly limited set.
> But now we're going down the road where we get new stuff like tacos and unicorns every year. And even though Unicode is an industry standard, the pictures need to look like Apple's bitmaps to avoid confusion, and the Unicode standard changes so often that you have to manually keep track of who can already see ((upside-down smiling Emoji)) and whose computer/phone/browser/messenger software is too old.
Emoji were added to Unicode for compatibility with various mobile phones, so they would have a standard encoding. That's how Unicode ended up with the poop emoji for example - they didn't sit around thinking "what we really need is...". Since people really, really want more emoji, Unicode is sort of stuck constantly adding more. If you want to propose new emoji, the rules are at http://www.unicode.org/emoji/selection.html
Text symbols (as opposed to emoji) have different rules. Basically, the symbol needs to be used in "running text" (i.e. normal text), like "containers with [recycling symbol] can be recycled" or "he bid 2[club]". Traffic signs for example are not normally used in the middle of text, so they aren't encoded in Unicode. To get the Bitcoin symbol encoded, I needed to show that it was used in text, not just as a standalone icon. The full rules for symbols in Unicode are at http://www.unicode.org/pending/symbol-guidelines.html
TL;DR: Don't argue "Why does Unicode have a poop emoji but no symbol for X?" - the rules are totally different for emoji and symbols.
Edit: does HN strip out arbitrary Unicode characters now? I originally had Unicode characters in place of [recycling symbol] and [club], but they disappeared when I submitted.
IIRC HN might have a character whitelist to prevent overloads of combining or layout-altering characters, and not have to worry about the behavior of newly added characters. There were some comment threads a few years ago that were just stacks of hundreds of combining diacritics that would crash some rendering engines and create odd decorated text on others.
Tom Scott has a great video about the history of emoji. Basically, some companies in far east countries were encoding these icons in various proprietary codes for their messaging systems. The ecosystem became widespread and consistent enough that the unicode consortium saw it fit to include these emoji in the standard.
The snowman, on the other hand, is a weather symbol for snow, I assume. It appears alongside other symbols for meteorological phenomena, so I imagine was added around the same time and with similar reasoning: http://www.fileformat.info/info/unicode/block/miscellaneous_...
Honestly, I think "SVG over UTF" makes a lot more sense. It's impossible to make a character set that supports every character known to man, because that just adds undue effort on every computer maker, ect, to keep up.
So why don't we pick a very good set: perhaps every letter in every language in common use for the past 200 years? Then, for the oddball symbols that someone wants to mix in text, there can be some kind of SVG-like convention. This allows publishing textual information without requiring that every device maker updates their device to support a 1-off symbol.
But if the shape of embedded in the text, font choice becomes meaningless.
> undue effort on every computer maker, ect, to keep up.
The effort to update the font files every few years? Unless you insist on supporting a new Unicode version the second it comes out, I don't see the big effort here? Of course there is effort for font makers, but this is quite centralised.
> This allows publishing textual information without requiring that every device maker updates their device to support a 1-off symbol.
The main purpose of Unicode is to encode the information. How the information is turned into its visual counterpart is outside the scope of unicode. For what it's worth this could be done by linking unicode code points to matching SVGs in a document. Wait, exactly that is already a W3C standard: https://www.w3.org/TR/SVG/fonts.html
What about the oddest oddballs whose "symbols" are animations http://www.reactiongifs.com/r/tww.gif? They are used a lot on reddit sometimes even with sound.
Because it's easier to throw in random icons than to actually accomplish the goal of "every letter in every language in common use for the past 200 years", or even "past 20 years".
Or, put another way:
'We have an unambiguous, cross-platform way to represent “PILE OF POO” (), while we’re still debating which of the 1.2 billion native Chinese speakers deserve to spell their own names correctly.'
This is a link by the article's author that is intended to make it easier for us to add useful symbols: https://github.com/jloughry/Unicode
I recommend you use it to add any glyphs that you feel are being neglected.
That article raises an interesting issue about a character in the author's name that is missing from Unicode. Unfortunately the article is (how to put this?) not constructive. The complex reasons that Unicode excluded the character are described in [1]. If the author addresses those issues, there's a much better chance to get the desired character into Unicode.
Correct me if I'm wrong, but isn't the Han Unification project more about unifying semantically distinct, but visually identical characters under the same codepoint (rather than grouping together similar-looking codepoints as the article suggests)? As far as I'm aware it's more along the lines of reusing the codepoint for 'a' when encoding both English and Spanish text. Am I mistaken in thinking this?
I didn't see special Unicode symbols for the SI units in the link you gave or in a more general search. I found no match for "ampere" which is the SI base unit for electric current, or for "candela", used for luminous intensity.
> Unicode also has encoded U+212B Å ANGSTROM SIGN. However, that is canonically equivalent to the ordinary letter Å. The duplicate encoding at U+212B is due to round-trip mapping compatibility with an East-Asian character encoding, but is otherwise not to be used.
I was actually wondering about the electrical symbols for logic gates, such as AND, OR, NOR, XOR, NOT, etc. I would hope they were universally accepted by now and would help when writing books or describing logic. A quick Duck Duck search revealed nothing...?
Electrical symbols in general don't do so well when scaled down to the size of text. Plus, it is very uncommon to encounter the electrical gate symbols inline with text - usually the symbols are sitting in a separate circuit diagram.
Nevertheless, Unicode does have all of the logical symbols from mathematics, which are pretty commonly understood:
The success of the unicodepowersymbol proposal inspired me to suggest a couple characters to Unicode (the Bitcoin sign and IBM's group mark from 1960s mainframes, which were accepted). The point is that Unicode really is open to proposals from random people; you don't need to part of a big company to influence Unicode.
Heck, Unicode is a mess already. But that's mostly because (a) language and scripts are messy, and (b) the original aim was to unify and encompass all existing character sets, and some of those were messy as well.
While there was considerable uproar over Emoji and there still often is over yet another fifteen symbols that everyone thinks no one would ever need or use, the bulk of the Unicode character set is still scripts for human languages. And some of those are only relevant for a very small minority, say, archaeologists. But that's fine. There's enough space, we're nowhere near to running out and Unicode enabled all sorts of cool things in computing that simply were not possible before or only with awful hacks and workarounds.
Conversion between different character codes became much easier. (Å (U+00C5) is canonically the same as Å (U+212B) but the latter exist only for round-trip compatibility.)
Unicode defines normalization algorithms (is é its own character, or e with a modifier character?).
I can have a document which combines English, Russian, Arabic, and Chinese, and expect it to be readable and editable by many different tools.
> I can have a document which combines English, Russian, Arabic, and Chinese
English combines with top-down Chinese, and English combines with right-to-left Arabic, but top-down Chinese and right-to-left Arabic don't combine properly in the same document using Unicode -- the Arabic will be written bottom-up instead of top-down when embedded in the top-down Chinese.
I meant something simpler, like: The word 'computer' in English is 计算者 [jì suàn zhě] in Chinese, Компьютер in Russian, and حاسوب in Arabic."
Try that without Unicode.
It's of course possible with TeX, and no doubt other solutions. Which is why I added "and expect it to be readable and editable by many different tools".
(As a real-world use case, look at Knuth's "The Art of Computer Programming" and see how he credits people using their full names, in their own written language.)
Do you know when you would use 计算着 over 电脑[dian nao]?
计算着 I guess more literally translates to "one who computes", whereas 电脑 translates to "electric brain" which is a way more fun image, but I have no idea how the usage varies.
计算着 [jisuanji] is the older usage that harkens back to the days when computers were mainframes and terminals and not in every household. Generally everyone says 电脑 [diannao] these days.
I am not a native speaker and terrible at googling this, but I've once heard that 電腦 was originally the Taiwanese way of saying it and 計算機 the Chinese way, in the same way that 'program' is still called 程式 on one side of the strait and 程序 on the other, or 'internet' is either 網絡 or 網路. The names of movies and video games are also generally different (PRC, HK, TW).
A text file, a webpage, or a database table can only contain textual data in a given encoding.
That's because every byte stored in the file, for example byte number 188, either means "¼" (as it does in ISO/IEC 8859-1, aka. Latin-1 or ANSI), or it means "ỳ" (as in ISO/IEC 8859-14) or "ｼ" (in JIS X 0201, one of the many Japanese encodings that were devised over the years.)
How do you know which encoding a certain file uses? In general YOU CAN'T and this was the source of many problems and "solutions" which caused even more problems over the years.
Well then, how did you mix symbols from different alphabets, say in a dictionary or in a post that talks about them, like this very post? YOU COULDN'T, short of doing ugly hacks and other subterfuges, like using GIFs for all foreign characters.
Unicode gave a distinct number (or "codepoint") to every character and symbol known to man (within reasonable limits) and this allowed a lot of things that we take for granted nowadays, including this very post, were I just copied and pasted various symbols from their Wikipedia pages and just expect it to work.
Fun fact: Japanese even has a word for foreign characters, "gaiji", and it's extremely common in Japanese ePUBs to use small square images very frequently for characters not in the current font, using the term "gaiji" in the CSS class names used for these characters. And at least one mainstream ePUB reader has special code to detect these gaiji and adjust its rendering to make them behave better.
Similarly, the word "loanword" (used to describe words borrowed from other languages) is gairaigo 外来語 which is literally 外来 (gairai) "foreign" + 語 (go) "word/language". Japanese is filled with words where you can often figure out the meaning purely from the characters used!
Someone I know worked at a company that sells a content management system with version control. Bigcos use it to keep and update lots of marketing materials and manuals. Those responsible for product x or area y can see what's been added by others, and update their own versions, and record that.
In this way, new FAQs and other updates spread to all the markets easily.
It's vastly easier to do this stuff when all the documents use the same text encoding. Even if noone can read everything, the fact that everything uses the same encoding means that any pair of languages you can read are technically readable.
Although I had never worked with the Unicode Consortium, I [submitted a proposal][1] for an international symbol for an observer and it was eventually accepted.
Its one thing to have absolute, iron fisted control over your own platform - its another to intentionally seek to limit people's self expression on other platforms by influencing the standard in this way.
Are you saying that a company with voting rights shouldn't be allowed to have influence on what goes into Unicode? That doesn't make any sense. Also, if you read the article, Apple wasn't the only party in favor of nixing the emoji.
> Are you saying that a company with voting rights shouldn't be allowed to have influence on what goes into Unicode?
One might have a right to do something and yet be wrong to do it.
Apple had every right to do what they did, but they were completely, totally and undeniably in the wrong to do it. Everyone associated with their action should be ashamed. Honestly, they should all resign: their behaviour demonstrates that they have no business being associated with this sort of work.
Your comment is ridiculously extreme. They were not "undeniably" in the wrong. You think they were wrong, but that is an highly subjective opinion. In fact, I don't even agree that they were wrong to do this at all. I think it's perfectly reasonable to argue against the inclusion of more gun imagery in Unicode.
Also, if you think Apple was wrong, you must also think that Microsoft was (they voiced support), and everyone else at the meeting who agreed with the move. As the article says,
> Davis confirmed to BBC News on Monday that "there was consensus to remove" the emojis, but that he couldn't comment on the details.
So it's clearly not just Apple that thought this was the appropriate move.
Thank goodness for apple and other members which rejected the starter pistol/rifle proposal. Imagine how many mass shootings we've prevented by people not being able to communicate their plans using the rifle emoji.
Just like how North Korea isn't "undeniably" wrong in censoring speech to such an extreme. This is a very 1984-esque "solution" to a problem -- don't want to acknowledge positive use of guns? Good news, we can just erase them from our language! The way we treat emoji has some very serious similarities to Newspeak.
That's absurd. Nobody's censoring anything here. Apple not wanting to add a new gun emoji is in no way preventing you from talking about guns. Emoji isn't a replacement for English and nobody is forcing you to "speak" in all emoji.
I assume they reserve the unicode character, but anyone who wants to use it decides what it looks like (so the rifle could look different on different platforms, which isn't a big issue)
This is always true for Emoji. The platforms always decide what their presentation of emoji will look like, just as they determine what font they will use for the unicode traditional letters. In this case, Apple and others decided they didn't want a rifle emoji, so it was moved to the 'black and white symbols' section so that those platforms wouldn't be 'missing' an emoji, which has other technical implications.
An emoji cannot be an emoji without having emoji presentation. There is a rifle character that resembles, in form and function, the other Olympic emoji, it's just not sitting in the set reserved for emoji. Any platform that wants to pick it up can use it, they can even put it next to all their other emoji if they want to. Apple, et al did not 'prevent' Rifle from being encoded. They just decided they wouldn't pick it up as an emoji for their platform, and the Unicode conference decided to move it to a different category based on that decision.
There are literally thousands of characters that Apple doesn't encode for their platforms. I haven't heard whether Apple will be supporting:
Osage, a Native American language
Nepal Bhasa, a language of Nepal
Fulani and other African languages
The Bravanese dialect of Swahili, used in Somalia
The Warsh orthography for Arabic, used in North and West Africa
Tangut, a major historic script of China
All of which were added to Unicode 9.
>The two characters will still be part of the Unicode spec, but they'll be classed as black-and-white "symbols" instead of regular emoji
You don't even need to be part of a big company to go to their meetings. I went to their conference in San Jose just to see the proposal for the Chinese Take-out box/chopsticks/fortune cook emojis. I met a ton of nice people, too.
When do we stop handing out IPv4s by the hundreds of thousands? "When we run out". When do we stop doing 120km/h? "When we run out of road". When do we stop spending money? "When we're flat out broke".
(Note: I am not advocating for or against the unicode case, I'm making a point about the specific mindset used to justify this)
I phrased it rather crudely. I really meant "when there is no more to add". With the caveat that running out is the other option.
I think the question represents an equally terrible mindset, "when do we stop?" How can I answer that, honestly? Is there a number? Or is it when a certain date has come and gone? How do we pick that number? Or that date? Why do we need to stop? I understand why we might want to rate-limit the adoption of new glyphs, but I don't see why we'd ever draw a line in the sand and say, "Ok, now it's frozen." Imagine if that had happened before the creation of the Euro as a currency.
You don't have to be able to read the word ON to recognize it as a symbol. "Circle next to zigzag-thing" is as good as circle with line sticking out of it.
There's also precedent. Continental Europe standardized on "STOP" on stop signs back in the 70's, even though no continental language has "stop" in it.
Excuse me? No continental European language has the word stop in it? You should probably learn more languages before making claims like that.
Stop is a word in Dutch (the first recognisable use of the word I could find dates back to 1287). And German has stopf. I couldn't find a date for that, because my German isn't good enough to read their etymology dictionary, but it's source is Old High Germanic, so it's safe to say that stop has been around in continental Europe in Germanic languages since Medieval time at the very least.
Edit: A quick further Google search reveals that Norwegian and Danish (and thus most likely Sweden too) have the variation "stoppe" derived from Low German.
Seems a bit disingenuous to claim it was adopted despite "not being an existing word" if two languages use the exact rendering on the sign and 4 others use/have words that are so closely related that they have the same spelling, but 1 or 2 additional letters.
Further edit: French apparently adopted the word stop from English...in 1792
The German Wikipedia, at https://de.wikipedia.org/wiki/Stoppschild , gives an example of "stop" used in the Protectorate of Bohemia and Moravia (after the German occupation of Czechoslovakia) in 1939, though it says the sign was an imported variant.
> As in Dutch stoppen, the sense “to stop” is figurative from water flow being stopped by plugging. Only in this figurative meaning has the form been adopted into standard German proper, under the reinforcing influence of English to stop.
https://en.wiktionary.org/wiki/stop gives a list of continental European languages where 'stop' is part of the language. Nearly all borrow from the English. Not Dutch, however.
German stop signs used the word "HALT" before. My German dictionary defines "stopf" as darning yarn, and "stoppen" as stop. Not quite the same spelling. Typing "stop" into French Google translate autocompletes to "stoppé". I wouldn't be in the least surprised if the American spelling crept into many European languages as a result of the sign being ubiquitous for 41 years - it would be surprising if it didn't.
I wouldn't underestimate the influence on the language of the American occupation after the war, either, nor the global influence of American business since the war. English words have crept in everywhere.
Because it's a zero and the | is a one. Arabic numerals are more universal, and the convention of 0 for off and 1 for on was established precisely to avoid picking a language. Then the combined glyph for an on/off button was created, along with the similar broken circle glyph for on/standby. Those have squarish proportions, so the corresponding 0 and 1 glyphs are needed to match those proportions. Hence the four symbols now existing in Unicode (well, 3½)
> the convention of 0 for off and 1 for on was established precisely to avoid picking a language
I figured that politics was the case. What is less supportable is working backwards from that to concoct a rational reason. People who understand digital electronic conventions are highly unlikely to not recognize "ON" and "OFF". Furthermore, anyone who does not know either would find "ON" just as easy to learn as "|".
We could also use "OHM" instead of Ω and "EUR" instead of €, but symbols provide more concise representations of meaning, especially in the case of a combined ON/OFF button. Much easier to fit the universal circle with 1 in it than "ON/OFF". The broken circle with 1 in it is way smaller than "ON/STANDBY". The other two symbols are then necessary to keep the proportions consistent across all four related glyphs.
That's the only argument I've seen for it that makes any sense. I'll counter by saying only "ON" is needed, not "ON/OFF", as the off part is implicit. Same goes for STANDBY.
This has been standard practice for a long time, I'm not just making things up on the fly. BTW, SBY is a standard abbreviation for STANDBY used by the military, if space is a problem. And SBY is a lot more google-able than an icon.
It's not obvious that the "0" is a zero, as opposed to the letter "O" (cyrillic, latin) or the something entirely misleading, as this gesture: http://i.imgur.com/6KZ1nKG.jpg
I suspect the vertical slash | has as many lookalikes as the O, but I won't go there. I do want to mention that, as you demonstrated, the "1" that means on is more often represented as the Latin I or just a vertical stroke as in |, making it just as hard to ID as a numeral, especially if you don't know that the 1/0 are derived from the binary logic gates.
I would argue that the words "ON" and "OFF" when seen as glyphs are much less ambiguous than I/O.
It can be treated as a glyph. Why would it be inappropriate?
And besides, languages the world over use plenty of words borrowed from English, and English itself is loaded with borrow words from other languages.
I've thought the mania for icons to replace common words since the Mac to be silly. Why is a picture of a Kleenex box more understandable than 'PRINT'? I have no idea what half the icons on my iPhone mean.
No way to google icons, either. I know, I'm supposed to learn them by pressing them to see what happens, but as someone who has learned not to learn how to operate machinery that way, I find it distasteful.
428 comments
[ 2.8 ms ] story [ 315 ms ] threadSurely not every font has to create glyphs for every Unicode character...how does that work? Is there a kind of "fall-back" font for characters not implemented?
https://en.wikipedia.org/wiki/Fallback_font
Not always true though, and can depend on the application using the right system calls for rendering text (for example).
A result of this is that if you ask for a character such as "𓀴" (U+13034 EGYPTIAN HIEROGLYPH A044) in a monospace font, the symbol you get back can be variable width.
[1]: https://en.wikipedia.org/wiki/Fallback_font
013
034
OSX 10.11.4
The basic problem is that Unicode characters (which may consist of as many codepoints as you like) have varying width. For instance Chinese has characters that are displayed over a width of two normal monospace characters.
See this hacker news post: https://news.ycombinator.com/item?id=10206380
So the problem with emojis is that they aren't really specified to be full width or half width characters: https://ianrenton.com/blog/adventures-in-emoji/
Many other unicode symbols also suffer from this problem. E.g. ␀ is the printable version of the unprintable NUL (\0) control character, but it's so small at 13.3px / 10pt CSS font size that it's difficult to distinguish from the other control pictures.
␀ ␁ ␂ ␃ ␄ ␅ ␆ ␇ ␈ ␉ ␊ ␋ ␌ ␍ ␎ ␏ ␐ ␑ ␒ ␓ ␔ ␕ ␖ ␗ ␘ ␙ ␚ ␛ ␜ ␝ ␞ ␟ ␠ ␡
How those look on display w/o pixel scaling: https://i.imgur.com/lAAyjXu.png
http://i.imgur.com/kKfLYgx.jpg
Interestingly in Safari the hieroglyph is shown (still small though), I wonder why Chrome can't find it?
[0] http://www.fileformat.info/info/unicode/char/2660/index.htm [1] http://www.fileformat.info/info/unicode/char/2406/index.htm
What browser/os are you using? Is it possible you're going through a proxy that is altering the HTTP headers?
Biang from Biang biang noodles is also fairly complex (though not supported by Unicode)
"appearance of a dragon in flight"
And actually it is in the Arabic "name of God," thanks!
https://en.wiktionary.org/wiki/%D8%A8%D8%B3%D9%85_%D8%A7%D9%...
I didn't know there's a whole ligature as a single code point U+FDFD available, thanks sscotth. The code point is also easy to remember.
(It, accidentally, also most probably represents the words first said by the Orlando shooter when calling 911, when the translation in the transcription is compared.)
Is this the biggest glyph in Unicode?
http://i.imgur.com/g7igcwP.png
(M-x ucs-insert FDFD)
http://m.imgur.com/tKnyPad
http://i.imgur.com/WemOI28.png
http://imgur.com/oQEHbA7
[1] https://github.com/lifthrasiir/unison
[2] https://www.google.com/get/noto/
Thank you for stepping up and making a difference.
So which is it? Does each code point represent a visual image? A semantic meaning? Both? It depends? Something else?
I've tried to decipher that on my own and only learned that the answer to these sorts of questions are complicated, because it's very complicated to represent all written human language via one set of rules.
So I know some of the answers to my questions above, but I'm hoping someone with real expertise can provide the fundamental rules/policies - if there are any.
Well the answer is clear: each code point represents one visual image, to which is associated one or more meanings.
(I don't know anything about Unicode, so maybe it already has that.)
Programatically, it is much easier to say "does a character lie between 0x12 and 0xBC" than to create a function like `isSymbolForTrafficInEurope()`
(And relevant to my country, "Swedish characters lowercase" would map to latin letters lowercase + åäö.)
Now, language-specific subsets¹ of those are a bit iffy to deal with. Especially when text can contain loan words from other languages, so in my experience it's rarely a useful thing to ask for.
¹ Yes, subsets. Latin letters lowercase is not the set abcdefghijklmnopqrstuvwxyz. It is the set
abcdefghijklmnopqrstuvwxyzªºßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ āăąćĉċčďđēĕėęěĝğġģĥħĩīĭįıijĵķĸĺļľŀłńņňʼnŋōŏőœŕŗřśŝşšţťŧũūŭůűųŵ ŷźżžſƀƃƅƈƌƍƒƕƙƚƛƞơƣƥƨƪƫƭưƴƶƹƺƽƾƿdžljnjǎǐǒǔǖǘǚǜǝǟǡǣǥǧǩǫǭǯǰdzǵǹǻǽǿ ȁȃȅȇȉȋȍȏȑȓȕȗșțȝȟȡȣȥȧȩȫȭȯȱȳȴȵȶȷȸȹȼȿɀɂɇɉɋɍɏɐɑɒɓɔɕɖɗɘəɚɛɜɝɞɟɠɡɢ ɣɤɥɦɧɨɩɪɫɬɭɮɯɰɱɲɳɴɵɶɷɸɹɺɻɼɽɾɿʀʁʂʃʄʅʆʇʈʉʊʋʌʍʎʏʐʑʒʓʕʖʗʘʙʚʛʜʝʞʟ ʠʡʢʣʤʥʦʧʨʩʪʫʬʭʮʯʰʱʲʳʴʵʶʷʸˠˡˢˣˤᴀᴁᴂᴃᴄᴅᴆᴇᴈᴉᴊᴋᴌᴍᴎᴏᴐᴑᴒᴓᴔᴕᴖᴗᴘᴙᴚᴛᴜᴝ ᴞᴟᴠᴡᴢᴣᴤᴥᴬᴭᴮᴯᴰᴱᴲᴳᴴᴵᴶᴷᴸᴹᴺᴻᴼᴽᴾᴿᵀᵁᵂᵃᵄᵅᵆᵇᵈᵉᵊᵋᵌᵍᵎᵏᵐᵑᵒᵓᵔᵕᵖᵗᵘᵙᵚᵛᵜᵢᵣᵤ ᵥᵫᵬᵭᵮᵯᵰᵱᵲᵳᵴᵵᵶᵷᵹᵺᵻᵼᵽᵾᵿᶀᶁᶂᶃᶄᶅᶆᶇᶈᶉᶊᶋᶌᶍᶎᶏᶐᶑᶒᶓᶔᶕᶖᶗᶘᶙᶚᶛᶜᶝᶞᶟᶠᶡᶢᶣᶤᶥᶦ ᶧᶨᶩᶪᶫᶬᶭᶮᶯᶰᶱᶲᶳᶴᶵᶶᶷᶸᶹᶺᶻᶼᶽᶾḁḃḅḇḉḋḍḏḑḓḕḗḙḛḝḟḡḣḥḧḩḫḭḯḱḳḵḷḹḻḽḿṁṃṅṇ ṉṋṍṏṑṓṕṗṙṛṝṟṡṣṥṧṩṫṭṯṱṳṵṷṹṻṽṿẁẃẅẇẉẋẍẏẑẓẕẖẗẘẙẚẛẜẝẟạảấầẩẫậắằẳẵặẹ ẻẽếềểễệỉịọỏốồổỗộớờởỡợụủứừửữựỳỵỷỹỻỽỿⁱⁿₐₑₒₓₔₕₖₗₘₙₚₛₜⅎↄⱡⱥⱦⱨⱪ ⱬⱱⱳⱴⱶⱷⱸⱹⱺⱻⱼⱽꜣꜥꜧꜩꜫꜭꜯꜰꜱꜳꜵꜷꜹꜻꜽꜿꝁꝃꝅꝇꝉꝋꝍꝏꝑꝓꝕꝗꝙꝛꝝꝟꝡꝣꝥꝧꝩꝫꝭꝯꝰꝱꝲꝳꝴꝵ ꝶꝷꝸꝺꝼꝿꞁꞃꞅꞇꞌꞎꞑꞓꞡꞣꞥꞧꞩꟸꟹꟺfffiflffifflſtstａｂｃｄｅｆｇｈｉｊｋｌｍｎｏｐｑｒ ｓｔｕｖｗｘｙｚ
How do you condense that again into language-specific subsets? Every letter that appears in a word in a dictionary? Then at least é belongs to German as well, even though it's usually not considered part of the German Latin subset. Unicode stays clear of that issue by simply not defining what script subsets a character belongs to (rightfully so, IMHO).
Also, why are you doing that check? Is it to see if something is lowercase? If so, your check will get the wrong answer for lowercase letters like å.
Unicode does have a way to check if something is uppercase/lowercase, when that distinction exists. This is in UnicodeData.txt.
You might try asking an old IBM programmer just how "fine" they felt dealing with EBCDIC...
But how serious is the problem? How many times do you need to test if a given character is one of the 26 allowed letters of the English alphabet, and where you implement it by testing it against the range?
Typically you write it as "islower_english(c)" once, and be done with it. Is that really hard?
If you do think that's a serious problem, then what of those programmers who need to test for lowercase letters in "España", "München", "Diyarbakır", and "façade"?
We've broadly moved beyond that, but there's still value in grouping sets together in a way that makes certain kinds of frequent tests less computationally expensive than they would be if codepoints were randomly distributed.
EDIT: Plus, if it were that important, IBM could implement the function in hardware. (Perhaps they did.)
For CJK characters, they unified all semantically similar han-characters, even when they have visual forms that are quite different between Japanese, Chinese and Korean.
If you want to write Japanese and Chinese in the same document, you need to mark up the section to tell the system that renders it, to render different visual forms for similar codepoints depending on whether they are used in Japanese or Chinese.
This isn't true. 青 and 靑 are the same character written differently; they have their own codepoints. Ditto for a huge number of simplified Chinese characters; 语 is mainland Chinese and 語 is the same character in Japanese.
I wouldn't know how to show you examples here, as 直 will 直 display the same since they have the same code point, but different number of strokes in japabese and chinese.
https://en.m.wikipedia.org/wiki/Han_unification
Han unification is generally seen as a bad choice in retrospect, but it was something Unicode had to do when it looked like 2^16 codepoints were all they were going to get.
https://en.wikipedia.org/wiki/Han_unification#Examples_of_la...
Han characters that are traditionally viewed as variants of one another, or that are simplified from more complex logograms (such as 龜, which was simplified into 亀 in Japan and 龟 in mainland China) tend to have different codepoints, but the stylistically different ones usually belong to the same codepoint.
> the stylistically different ones usually belong to the same codepoint
Fair enough. Do you happen to know why 青 and 靑 weren't unified?
Look it's pretty simple, every code point represents a semantic meaning, except for:
1. those characters who also encode the width of their visual image (U+FF00..FFEF)
2. the one that means 'unknown' (U+FFFD)
3. those characters that change their visual representation depending on their position in the word (U+FB50..U+FDFF,U+FE70..U+FEFF)
4. those that change the visual image of another code point (U+FE00..U+FE0F)
5. those characters that have a visual image as their semantic meaning (too many to list)
6. those that are designated to have no semantic meaning at all (U+FDD0..U+FDEF)
7. those that have a meaning only in pairs (U+D800..U+DFFF)
8. miscellaneous
"every code point represents a semantic meaning" is completely consistent with the notion that some code points e.g. have differing visual representation depending on their position in the word.
So? How is that different from any regular character in real life?
101 for example means the number 101, an introductory class in university, slang for "anything introductory" in general, etc.
And let's not get started on the meanings of letters, e.g. a and e.
The emoji code points can be represented differently on different systems given their meaning.
So it makes sense to have different emojis for different 'meanings'.
The 'moon' switch here does no mean 'moon' - it means 'standby' or whatever.
It may look noticeably different on different systems.
Think from a design perspective: you have 5 emojis to represent 'clouds, sky, earth' etc. - and the a different set of 5 to represent 'on, off, sleep, shutdown'. Those icons will be markedly different in terms of representation, groupings, colour coding, underlying functionality if they are integrated into an experience in any meaningful way.
Text your car with the 'shutdown' symbol to tell it to shut down.
Your bot texts your friend with a moon symbol to tell him you're asleep. Or whatever.
So if a system wants to render "on" differently than "straight vertical line", that's possible.
However, if "off" should be rendered differently than "circle", that's not possible. (Or only possible with out-of-band information or modifier characters which would still have to be defined)
It's a mess. If you want to write a document in Japanese that talks about a Chinese character which is written differently than its Japanese version, you can, or can't, achieve this in Unicode, depending on the character, its history, and the mood of the consortium the day it was assigned.
The reality is that Unicode is governed by people, some of those people are grumpy reductionists who push for a minimum of symbols and a maximum of meaning-overloads, and others are more liberal and tend to advocate the opposite, and the result is a compromise, and is in areas very messy.
I don't think i personally agree with brand logos in Unicode.
Seconded, brands come and go, as do their logos, many large corporations have gone through several logo changes.
Pan-Am is the main one that springs to mind (or it's because I've just re-watched Blade Runner AGAIN...)
So just change the font you use to render Unicode. The Unicode itself can remain the same.
http://nymag.com/selectall/2016/04/princes-legendary-floppy-...
I don't know the codepoint for it though. Does anyone (other than the guy mentioned in the article) actually have this 'Prince Font' ?
This blog post is a nice example, I have absolutely no idea how these new code points are supposed to look like, since I only spend an afternoon to implement the unicode best practices from the Arch wiki, instead of subscribing to some unicode standard mailing list. (Except the one symbol which was redefined to a symbol that does not carry the semantic meaning of "standby symbol" anywhere outside of the unicode standard.)
In my opinion there are two ways forward, one burn the entire thing. Or alternatively, force the unicode committee to produce an authoritative and complete font, in triplicate, and in their own blood.
I think it would have made much more sense to have something like image tags: a special codepoint would introduce a link to a URL containing a sequence of glyphs, followed by an index into that sequence. Those glyphs would be guaranteed not to change (in any meaningful way), and devices would be free to cache them. This way, anything that isn't real text, would standardize representation, too, instead of just a vague "meaning". Another standard could relate those glyphs to one another in some way, giving them standard semantics and means of translation (i.e. "Egyptian hieroglyphics"). This would also allow each of those (emojis or hieroglyphics) to evolve their standards independent of a single universal standard that means little.
^1 I am pretty sure that some use cases actually profit a lot, but neither text file formats (since most text does not contain 2^64 different characters zip would work nicely) nor networking (since most data on the internet is either video or torrents) seem to be among them. So probably they would not be huge fields.
In any case, Utf8 has a place, and if you want easy manipulation and search, convert it to Utf32 - it's fixed width.
But that's only because Unicode has significantly ventured well beyond what we consider to be text. The BMP is enough to represent all text (including math).
There are no codepoints if it's not text. How do you use codepoints for embedding a video or a picture on your blog? You don't! But, if you want to treat something as if it were text, then I suggested doing something similar to an XML namespace: "the next segment is hieroglyphics, you can get their glyphs from here, and these are their indices...". That "extended text" is still not text, and it still doesn't use any Unicode codepoints, but it can work according to similar principles.
> Do you think it'd be more efficient to have to support 6 different standards than one?
Then why don't we let the Unicode Consortium take over standardizing video or audio? If something isn't text, why is it standardized by a text standardization body?
You can exchange text with embedded icons just as easily without requiring OS vendors to come up with their own versions of vaguely-defined pictures by... simply embedding pictures.
I can already see the people on whatever would be the HN 20 years from now complaining how "bloated" Unicode is, full of thousands of symbols that no one ever uses, and calling to replace the whole thing, costing the industry even more money to replace a standard yet again.
If you are trying to index into a string by "character" you are almost certainly already doing it wrong. Meaningful indexing almost always has to be by grapheme cluster. See Swift's string API as a great example of this done right.
The dream of a 16-bit Unicode washed up on the rocks of CJK scripts. It's dead and it isn't going to be revived. You can argue for a simpler standard, with fewer assigned codepoints, but the original BMP isn't it and was never going to be it.
I concede this point. I still don't see why the Unicode Consortium should spend effort standardizing non-text as text.
Meanwhile, a lot of the "Ys and Zs" added to Unicode have proved to be extremely useful. Unicode's math operator and letter-styling support is what made MathJax (and more generally MathML) possible. They've also helped big time when it comes to accessibility (e.g. screen readers) for mathematics on the internet. Should we have shunted that off to another standard and made the creators of screen readers completely restructure their offerings so they can deal with Unicode characters and "Mathicode" characters? Assuming anyone bothered to implement it, how would that be better than just adding a Unicode category and spending a meager amount of space?
Second mathematical symbols, consider the case were I get a text file considering mostly of ASCII 7 and some mathematical symbols which may render as mathematical symbols or as Chinese characters, since there is no way to specify the encoding in a text file and so I have to guess the encoding. (That is not helped by the roughly 17 standardized encodings that mostly agree with utf-8.)
Tip: a quicker way is to copy the unrendered box and Google it.
No, last time I checked you are not legally allowed to do that.
What does that have to do with Unicode adding anything? Are you really claiming that if we threw out Unicode like you recommend, and (if I'm understanding your point correctly) choose an encoding for the new version that looks nothing like ASCII the encoding mess would get better? I think continuing the migration of most transmission of text to UTF-8 and explicitly specifying encodings for everything that needs to stick with Latin-1, etc. is a better option, unless you propose codifying the new encoding in law to force adoption.
In the same way it's useful to standardize letters in various alphabets without standardizing their screen representation. There is semantic content associated with each of these symbols that persists even if there is significant variation in how they are presented. Of the ones you list, emojis are the only ones where this is any more a problematic approach than it is for letters in various alphabets. And as people who don't approve of Unicode adding emojis like to point out, emojis aren't that critical so having some loss in the translation isn't a huge deal.
Remember that before emoji standardization various cell phone manufacturers (particularly in Japan if I remember correctly) started using codepoints for whatever they pleased. The alternative to Unicode not standardizing them was to have a repeat of the OEM font gold rush in the SMP.
> styled math letters (which would have been equally well served by simply rendering them in italics or in a special math font)
That was my first reaction as well, but there are a few problems with that approach:
* Math italic characters look very different from normal italics, and are shaped and kerned very differently because they are commonly used for single-letter variables which will be juxtaposed together in expressions. If your goal is to be able to preserve some math formulas in a purely line based text format, preserving this aspect makes a big difference in readability.
* Many of the math letters and "letter-like symbols" have associated semantic content (like bold for vectors), which it makes sense to preserve. MathML alleviates this to a significant degree but I don't believe these codepoints were intended only for MathML usage.
* On the technical side, OpenType math fonts need to carry associated metadata for many of these characters. Putting them in separate fonts complicates this, since these tables need to refer to glyphs (general codepoints are unsuitable in a number of cases) and each font file would have a different glyph address space.
I disagree. Say the name of a letter in any alphabet, and people will draw it in ways that are similar enough for automatic recognition. This is not true for pictograms and emojis.
> The alternative to Unicode not standardizing them was to have a repeat of the OEM font gold rush in the SMP.
I disagree. The alternative is a much simpler and faster standardization, of the kind I offered here: https://news.ycombinator.com/item?id=11958903 There is absolutely no need for a fixed codepoint for most of the non-BMP characters.
> If your goal is to be able to preserve some math formulas in a purely line based text format, preserving this aspect makes a big difference in readability.
So is rendering text in Arial vs. Comic Sans, but they haven't made separate codepoints for those.
Also, where this makes a lot of difference, would count as "specialized usage". I don't think it makes sense to have a single universal standard to standardize all specialized usage of human-readable data.
So you think instituting a system based on links not rotting would better preserve meaning? Not to mention that:
* Every text renderer that doesn't support your codepoint now displays a full URL, instead of a box, making text using these emojis very difficult to read.
* Instead of making implementation easy by requiring nothing new of text shaping libraries, they now have to be able to both connect to the internet and tie into a file cache.
The supplementary space was already there when we got to emojis, and UTF-8 and UTF-16 already had to deal with SMP codepoints for some of the less common CJK characters. Not everything above 0xFFFF is "weird" non-human language stuff. If the choice was "stick with UCS-2 and be totally fine language wise, or add more bits just for emojis and pictograms" I'd probably agree with you. If you think that's what happened, your timeline for this process is way off.
> So is rendering text in Arial vs. Comic Sans, but they haven't made separate codepoints for those.
Sure, but the different letter types carry crucial meaning in math formulas. "sup" in upright letters is the math operator supremum, "sup" in math italics is s * u * p. This kind of thing applies to every one of the mathematical letter variants.
> Also, where this makes a lot of difference, would count as "specialized usage". I don't think it makes sense to have a single universal standard to standardize all specialized usage of human-readable data.
You're zooming way out on this one. Math symbols have a lot more in common with letters and "normal" symbols than "all specialized usage of human-readable data". Remember that when Unicode added the math symbols, things like MathJax were simply impossible. Being able to write at least some formulas, which consist of letters and symbols, without losing tons of semantic information seems like exactly the kind of thing character encodings should do.
It can still display a box.
> Instead of making implementation easy by requiring nothing new of text shaping libraries, they now have to be able to both connect to the internet and tie into a file cache.
The question is implementation of what. I think that it is not an onerous requirement from applications that need to display emojis or ancient Egyptian hieroglyphs. Their OS could provide this service for them just as it allows them the use of fonts.
> The supplementary space was already there when we got to emojis, and UTF-8 and UTF-16 already had to deal with SMP codepoints for some of the less common CJK characters. Not everything above 0xFFFF is "weird" non-human language stuff.
That is a good point, one of which I was not aware, but I still don't think it justifies standardization of Chinese characters, emojis, and ancient Greek musical notation by the same standards body.
> Sure, but the different letter types carry crucial meaning in math formulas.
The use of italics in text may also carry crucial meaning. But if a textual representation as sup is supported, I don't see why the specialized rendering should be supported, too, but for math and not plain text.
> Being able to write at least some formulas, which consist of letters and symbols, without losing tons of semantic information seems like exactly the kind of thing character encodings should do.
I agree, but I think that that semantic information is preserved when writing N or NN instead of 𝑵 or ℕ. Considering that Unicode isn't enough to write most mathematical formulas convenient forms anyway and requires a specialized renderer anyway, I don't see the reason for this extra effort.
If it knows about your new codepoint. Everyone using an implementation that doesn't yet support it is going to show the full URL. If history repeats itself, these implementations will be the majority for at least a decade.
> The question is implementation of what. I think that it is not an onerous requirement from applications that need to display emojis or ancient Egyptian hieroglyphs.
Except they need to do literally nothing different. Hieroglyphs are vectorized, and emojis are either bitmapped (which existing font formats already supported) or vectorized. None of this required a single line of code in any text shaping library to change. Text shaping libraries generally don't even need to understand Unicode categories or other metadata: font files already contain all the relevant information (directionality, combining mark, etc.).
> The use of italics in text may also carry crucial meaning. But if a textual representation as sup is supported, I don't see why the specialized rendering should be supported, too, but for math and not plain text.
If you scrub all bold and italics from text, do any of the words turn into different words? That's what happens with sup (or other named operators). Same thing for blackboard letters, fraktur, etc.
> I agree, but I think that that semantic information is preserved when writing N or NN instead of 𝑵 or ℕ. Considering that Unicode isn't enough to write most mathematical formulas convenient forms anyway and requires a specialized renderer anyway, I don't see the reason for this extra effort.
My point is it was better than nothing, which was the alternative when Unicode added these. By adding mathematical characters that worked exactly the same as all other characters, we could get some of the advantages of MathML without needing everybody to implement a special math renderer or learn any new markup, just download new font files.
If getting everyone to accept MathML or something similar in an expedited fashion was a reasonable proposition, then I might agree that they should've kept it out of Unicode. Those were not the facts on the ground when this decision was made. Note that even now that we have MathML, the few browsers that support it (IIRC Firefox & Safari only) have complete shit implementations that look terrible.
The crux of this is that making changes to a standard to support something new and propagating the changes is super hard; getting new standards to get accepted and implemented is a Herculean effort. Unicode's expansion into these domains required nobody to do anything differently, let alone decide to up and write an implementation of a completely different standard. I think this is a case were our alternative was to let the perfect be the enemy of the good, or at least the working.
And that's what happens when scrubbing subscripts or binomial coefficients. When you want to represent math as text, you need to change your representation (add multiplication signs, forgo subscripts, use confusing parentheses etc.). This is still true with Unicode. The contribution of the non-BMP special math characters is quite minimal.
> The crux of this is that making changes to a standard to support something new and propagating the changes is super hard; getting new standards to get accepted and implemented is a Herculean effort.
Sure, but the emoji craziness continues, and there's little sign it would ever stop. Instead of saying "this isn't text; if you want, call it 'special text', escape it, and let a different body standardize it", the body entrusted with standardizing text representation worries about how to represent a picture of two people kissing as text. What next? Kids would want to add tunes to their text messages. Would the Unicode Consortium add code points for MIDI? And maybe managers would want to standardize code points for organizational diagrams. Would that be the consortium’s responsibility, too? The BMP contains all the characters for reasonable text-art.
Suffice it to say this getting "out of control" and eating up the remaining space in our lifetimes, or our children's lifetimes would be pretty impressive. This means that the worst we have to fear is more boxes, assuming OSes don't keep up with their fallback fonts. Saying adding more symbols-designed-to-be-just-kind-of-placed-in-line-with-other-symbols-like-they-always-have-been is the first step towards MIDI and organization diagrams is like saying you're vegetarian because it's a slippery slope from eating meat to eating people.
Also note that unlike every other excess of the Unicode standard, these would require massive changes to the code that handles text. This means that if Unicode decided to do this, you wouldn't have to worry about negative effects because nobody would implement it.
[1]: http://unicode.org/emoji/charts/full-emoji-list.html
1: http://large.stanford.edu/courses/2011/ph241/dunn2/
Things were way worse than that: to add emoji to text, NTT DoCoMo used private-use codepoints, AU used embedded image tags and Softbank wrapped emoji codes in SI/SO escape sequences.
Japanese, Oriental, Korean, and Emoji.
It's a bad JOKE.
Well done.
When did the word "led" disappear from the English language?
I want to strangle "reporters" who write articles for the NYT, Washington Post, etc. who get this wrong all the time.
In some respect, they are getting jumbled up because of international pressures for the base emoji set to be stretched into a be-all for the global market. An example is Taco. There are tacos in Japan. They are hard to find and when you do find one, you definitely don't want to eat one there. Mexican food is one of the rare cuisines the Japanese don't do better.
"To start the device, press the ⏻ button on the device face"
There, used in a sentence.
"Every possible image" (e.g. an elephant icon in running text) is not.
A standard clip art library that covers universally understood symbols sounds like something that would be very useful.
What is "plain text format" though? If 'text' isn't limited to Western ASCII characters (which it very obviously shouldn't be considering many people use other character sets), then the idea of a text standard should be to encode all the glyphs people use, so "plain text format" becomes a canonical list of all the communicative symbols in all languages. That's what Unicode aims to be.
In my opinion, if they're used for communication, it doesn't seem unreasonable that such a canon of characters should include universal iconographic symbols like the standby icon.
Your fonts don't have to support the entirety of Unicode. That's why we have font stacks and fallbacks.
>The scope of the Unicode Standard (and ISO/IEC 10646) does not extend to encoding every symbol or sign that bears meaning in the world.
>This list has been round and round and round on this -- regular as clockwork, about once a year, the topic comes up again. And I see no indication that the UTC or WG2 are any closer to concluding that bunches of icons should start being included in the character encoding standards simply on the basis of their being widespread and recognizable icons.
>Where is the defensible line between "Fast Forward" and "Women's Restroom" or "Right Lane Merge Ahead" or "Danger Crocodiles No Swimming"?
(http://www.unicode.org/mail-arch/unicode-ml/y2005-m08/0371.h...)
Now it looks they add whatever somebody thinks of. I guess it's related to the liberation from the BMP.
Until Unicode has a half-star character, it won't even be able to encode the average newspaper.
Recent article on the Unicode/emoji debate:
https://www.buzzfeed.com/charliewarzel/inside-emojigeddon-th...
In all seriousness, I'm not sure emoji's really belong in text encoding. Even though it's more convenient, based on where they're most frequently used I don't think they need to be universal.
1) everybody uses them on their phones, they're in Unicode, consistent and compatible between devices and messaging programs. In the far flung future, researchers will be able to study their linguistic role in communication, confident in understanding what the characters were.
2) everybody uses them on their phones, they're proprietary fonts and codepoints (in the Unicode private use area if you're luck, just random data if you're not), there's no consistency between phone models, manufacturers, or cell networks. Future researchers can pound sand.
We were at #2 pre-Unicode. It was a goddamn mess, especially in Japan. Lord knows why anyone would prefer it. There's no value in being a snob about what kinds of incredibly frequently used characters we think are Worthy of inclusion, imo.
3) People who love colourful images will use stickers in Facebook Messenger, LINE, Viber, and soon iMessage. I'm sure WeChat has them too. It's basically like 2), except we've moved from proprietary codepoints to proprietary protocols.
I don't mind characters like and or even good old ︎ (which has always been too tiny for its own good). These work in black and white, in different artistic styles, and they're a fairly limited set.
But now we're going down the road where we get new stuff like tacos and unicorns every year. And even though Unicode is an industry standard, the pictures need to look like Apple's bitmaps to avoid confusion, and the Unicode standard changes so often that you have to manually keep track of who can already see and whose computer/phone/browser/messenger software is too old.
> characters like and or even good old ︎ (which
Should have been:
> I don't mind characters like ((yellow smiling Emoji)) and ((thumbs up Emoji)) or even good old ︎((pre-Emoji Unicode smiley)) (which has always been too tiny for its own good). These work in black and white, in different artistic styles, and they're a fairly limited set.
> But now we're going down the road where we get new stuff like tacos and unicorns every year. And even though Unicode is an industry standard, the pictures need to look like Apple's bitmaps to avoid confusion, and the Unicode standard changes so often that you have to manually keep track of who can already see ((upside-down smiling Emoji)) and whose computer/phone/browser/messenger software is too old.
Heck, I'd be unhappy. I love adding emoticon and emoji and fun things to my emails.
They are mostly in Unicode for use in SMS, but there are plenty of use cases in other forms of text.
Text symbols (as opposed to emoji) have different rules. Basically, the symbol needs to be used in "running text" (i.e. normal text), like "containers with [recycling symbol] can be recycled" or "he bid 2[club]". Traffic signs for example are not normally used in the middle of text, so they aren't encoded in Unicode. To get the Bitcoin symbol encoded, I needed to show that it was used in text, not just as a standalone icon. The full rules for symbols in Unicode are at http://www.unicode.org/pending/symbol-guidelines.html
For the snowman in particular, it was added to Unicode because it was a symbol used in the character set for Japanese TV broadcasts, see http://www.unicode.org/L2/L2007/07391-n3341.pdf
TL;DR: Don't argue "Why does Unicode have a poop emoji but no symbol for X?" - the rules are totally different for emoji and symbols.
Edit: does HN strip out arbitrary Unicode characters now? I originally had Unicode characters in place of [recycling symbol] and [club], but they disappeared when I submitted.
The snowman, on the other hand, is a weather symbol for snow, I assume. It appears alongside other symbols for meteorological phenomena, so I imagine was added around the same time and with similar reasoning: http://www.fileformat.info/info/unicode/block/miscellaneous_...
So why don't we pick a very good set: perhaps every letter in every language in common use for the past 200 years? Then, for the oddball symbols that someone wants to mix in text, there can be some kind of SVG-like convention. This allows publishing textual information without requiring that every device maker updates their device to support a 1-off symbol.
> undue effort on every computer maker, ect, to keep up.
The effort to update the font files every few years? Unless you insist on supporting a new Unicode version the second it comes out, I don't see the big effort here? Of course there is effort for font makers, but this is quite centralised.
The main purpose of Unicode is to encode the information. How the information is turned into its visual counterpart is outside the scope of unicode. For what it's worth this could be done by linking unicode code points to matching SVGs in a document. Wait, exactly that is already a W3C standard: https://www.w3.org/TR/SVG/fonts.html
Or, put another way:
'We have an unambiguous, cross-platform way to represent “PILE OF POO” (), while we’re still debating which of the 1.2 billion native Chinese speakers deserve to spell their own names correctly.'
https://modelviewculture.com/pieces/i-can-text-you-a-pile-of...
[1] http://www.unicode.org/L2/L2004/04252-khanda-ta-review.pdf
If people actually used these, it would make searching text for formulae much easier. Wikipedia editors and academic publishers, please note.
Also, there's no Unicode for screwdriver. Perhaps iFixit would like to campaign for that?
Congratulations on getting the power symbols in! When @edent writes "Will update ... when I stop dancing", was it "I got the power"?
I don't see how using them for anything else would have any use. I never searched for units when searching for formulas
BTW, just because a character exist doesn't mean it's the best choice for ordinary use. As https://en.wikipedia.org/wiki/%C3%85#Symbol_for_.C3.A5ngstr.... points out:
> Unicode also has encoded U+212B Å ANGSTROM SIGN. However, that is canonically equivalent to the ordinary letter Å. The duplicate encoding at U+212B is due to round-trip mapping compatibility with an East-Asian character encoding, but is otherwise not to be used.
I was actually wondering about the electrical symbols for logic gates, such as AND, OR, NOR, XOR, NOT, etc. I would hope they were universally accepted by now and would help when writing books or describing logic. A quick Duck Duck search revealed nothing...?
https://en.wikipedia.org/wiki/Logic_gate#Symbols
Nevertheless, Unicode does have all of the logical symbols from mathematics, which are pretty commonly understood:
While there was considerable uproar over Emoji and there still often is over yet another fifteen symbols that everyone thinks no one would ever need or use, the bulk of the Unicode character set is still scripts for human languages. And some of those are only relevant for a very small minority, say, archaeologists. But that's fine. There's enough space, we're nowhere near to running out and Unicode enabled all sorts of cool things in computing that simply were not possible before or only with awful hacks and workarounds.
Could you give an example? I don't know anything about this stuff.
Unicode defines normalization algorithms (is é its own character, or e with a modifier character?).
I can have a document which combines English, Russian, Arabic, and Chinese, and expect it to be readable and editable by many different tools.
English combines with top-down Chinese, and English combines with right-to-left Arabic, but top-down Chinese and right-to-left Arabic don't combine properly in the same document using Unicode -- the Arabic will be written bottom-up instead of top-down when embedded in the top-down Chinese.
I meant something simpler, like: The word 'computer' in English is 计算者 [jì suàn zhě] in Chinese, Компьютер in Russian, and حاسوب in Arabic."
Try that without Unicode.
It's of course possible with TeX, and no doubt other solutions. Which is why I added "and expect it to be readable and editable by many different tools".
(As a real-world use case, look at Knuth's "The Art of Computer Programming" and see how he credits people using their full names, in their own written language.)
計算機 is a calculator, not a computer, in Taiwan: https://tw.images.search.yahoo.com/search/images?p=計算機
Compare with Baidu in the PRC: http://image.baidu.com/search/index?tn=baiduimage&ie=utf-8&w...
Edit: Oops, I didn't even realise that the parent poster asked about 計算著. That one I've never seen.
That's because every byte stored in the file, for example byte number 188, either means "¼" (as it does in ISO/IEC 8859-1, aka. Latin-1 or ANSI), or it means "ỳ" (as in ISO/IEC 8859-14) or "ｼ" (in JIS X 0201, one of the many Japanese encodings that were devised over the years.)
How do you know which encoding a certain file uses? In general YOU CAN'T and this was the source of many problems and "solutions" which caused even more problems over the years.
Well then, how did you mix symbols from different alphabets, say in a dictionary or in a post that talks about them, like this very post? YOU COULDN'T, short of doing ugly hacks and other subterfuges, like using GIFs for all foreign characters.
Unicode gave a distinct number (or "codepoint") to every character and symbol known to man (within reasonable limits) and this allowed a lot of things that we take for granted nowadays, including this very post, were I just copied and pasted various symbols from their Wikipedia pages and just expect it to work.
In this way, new FAQs and other updates spread to all the markets easily.
It's vastly easier to do this stuff when all the documents use the same text encoding. Even if noone can read everything, the fact that everything uses the same encoding means that any pair of languages you can read are technically readable.
[1] http://www.unicode.org/charts/PDF/U2B00.pdf
2BD2 GROUP MARK 2015-May-05 Accepted 2016-May-29 Stage 6
The bitcoin symbol is in there too.
http://imgur.com/Sx0lkM8
A bit early.
Although I had never worked with the Unicode Consortium, I [submitted a proposal][1] for an international symbol for an observer and it was eventually accepted.
[1]: http://hypertexthero.com/logbook/2015/01/international-symbo...
(Top result: http://www.cbc.ca/news/trending/rifle-emoji-dropped-unicode-...)
Its one thing to have absolute, iron fisted control over your own platform - its another to intentionally seek to limit people's self expression on other platforms by influencing the standard in this way.
Obviously not.
One might have a right to do something and yet be wrong to do it.
Apple had every right to do what they did, but they were completely, totally and undeniably in the wrong to do it. Everyone associated with their action should be ashamed. Honestly, they should all resign: their behaviour demonstrates that they have no business being associated with this sort of work.
Also, if you think Apple was wrong, you must also think that Microsoft was (they voiced support), and everyone else at the meeting who agreed with the move. As the article says,
> Davis confirmed to BBC News on Monday that "there was consensus to remove" the emojis, but that he couldn't comment on the details.
So it's clearly not just Apple that thought this was the appropriate move.
There are literally thousands of characters that Apple doesn't encode for their platforms. I haven't heard whether Apple will be supporting:
All of which were added to Unicode 9.>The two characters will still be part of the Unicode spec, but they'll be classed as black-and-white "symbols" instead of regular emoji
http://arstechnica.com/apple/2016/06/apple-and-microsoft-pus...
"Until it's full" reminds me of departments that feel they must spend their entire budget.
When do we stop handing out IPv4s by the hundreds of thousands? "When we run out". When do we stop doing 120km/h? "When we run out of road". When do we stop spending money? "When we're flat out broke".
(Note: I am not advocating for or against the unicode case, I'm making a point about the specific mindset used to justify this)
I think the question represents an equally terrible mindset, "when do we stop?" How can I answer that, honestly? Is there a number? Or is it when a certain date has come and gone? How do we pick that number? Or that date? Why do we need to stop? I understand why we might want to rate-limit the adoption of new glyphs, but I don't see why we'd ever draw a line in the sand and say, "Ok, now it's frozen." Imagine if that had happened before the creation of the Euro as a currency.
Seriously, equipment has been marked ON and OFF for a hundred years. Is there any evidence anyone was confused with this more than with O and | ?
https://en.m.wikipedia.org/wiki/Power_symbol
Stop is a word in Dutch (the first recognisable use of the word I could find dates back to 1287). And German has stopf. I couldn't find a date for that, because my German isn't good enough to read their etymology dictionary, but it's source is Old High Germanic, so it's safe to say that stop has been around in continental Europe in Germanic languages since Medieval time at the very least.
Edit: A quick further Google search reveals that Norwegian and Danish (and thus most likely Sweden too) have the variation "stoppe" derived from Low German.
Seems a bit disingenuous to claim it was adopted despite "not being an existing word" if two languages use the exact rendering on the sign and 4 others use/have words that are so closely related that they have the same spelling, but 1 or 2 additional letters.
Further edit: French apparently adopted the word stop from English...in 1792
https://en.wikipedia.org/wiki/Stop_sign#Stop_signs_around_th... shows examples of "Stop" used in Germany and Italy in the 1950s.
The German Wikipedia, at https://de.wikipedia.org/wiki/Stoppschild , gives an example of "stop" used in the Protectorate of Bohemia and Moravia (after the German occupation of Czechoslovakia) in 1939, though it says the sign was an imported variant.
https://en.wiktionary.org/wiki/stoppen#German points out:
> As in Dutch stoppen, the sense “to stop” is figurative from water flow being stopped by plugging. Only in this figurative meaning has the form been adopted into standard German proper, under the reinforcing influence of English to stop.
https://en.wiktionary.org/wiki/stop gives a list of continental European languages where 'stop' is part of the language. Nearly all borrow from the English. Not Dutch, however.
I wouldn't underestimate the influence on the language of the American occupation after the war, either, nor the global influence of American business since the war. English words have crept in everywhere.
I figured that politics was the case. What is less supportable is working backwards from that to concoct a rational reason. People who understand digital electronic conventions are highly unlikely to not recognize "ON" and "OFF". Furthermore, anyone who does not know either would find "ON" just as easy to learn as "|".
(That's why I referred to "ON" as a 'glyph'.)
This has been standard practice for a long time, I'm not just making things up on the fly. BTW, SBY is a standard abbreviation for STANDBY used by the military, if space is a problem. And SBY is a lot more google-able than an icon.
Ok, here's my last argument then. It's aesthetically pleasing. I admit that's probably the weakest argument, but also the hardest to refute :)
I suspect the vertical slash | has as many lookalikes as the O, but I won't go there. I do want to mention that, as you demonstrated, the "1" that means on is more often represented as the Latin I or just a vertical stroke as in |, making it just as hard to ID as a numeral, especially if you don't know that the 1/0 are derived from the binary logic gates.
I would argue that the words "ON" and "OFF" when seen as glyphs are much less ambiguous than I/O.
And besides, languages the world over use plenty of words borrowed from English, and English itself is loaded with borrow words from other languages.
I've thought the mania for icons to replace common words since the Mac to be silly. Why is a picture of a Kleenex box more understandable than 'PRINT'? I have no idea what half the icons on my iPhone mean.
No way to google icons, either. I know, I'm supposed to learn them by pressing them to see what happens, but as someone who has learned not to learn how to operate machinery that way, I find it distasteful.