Have you ever read with other people, like in school or a book club, or been somewhere that there were other people around? It's an interesting move by the author; the loneliness epidemic hasn't gone unnoticed.
It's cute, and provides a hint of human connection that is otherwise absent on the web "hey, another human is reading this too!" which you probably know but something about seeing the pointer move makes it feel real.
Probably not the greatest during a hacker news hug of death, but if I read that article some other time and saw one of the moving pointers, I would think it was really cool.
I've been drawing circles for over a minute now and no one has joined me yet, so I conclude those movements are random rather than made by intelligent beings. :)
I did the same for a while while I was reading. From another comment, the position only seems to update once a second, so it'll be hard for someone to notice your movements.
Anytime tonsky's site gets posted here, I'm reminded by how awful it is, which is ironic given his UI/UX background. The site's lightmode is a blinding saturated yellow, and if you switch into darkmode, it's an even less readable "cute" flashlight js trick. I don't know why he thought this was a good idea. Thank god for Firefox reader mode.
I'm having a hard time reconciling "he knows what he is doing" with him making his site practically unusable without a reader mode, which by the way, not every browser supports (especially on mobile).
Don't even think of switching on the dark (night) mode with that attitude! :D
I really enjoyed the tongue in cheek design. I think every modern browser either allows you to turn on reader mode (especially on mobile) or just turn off CSS. This particular article works excellently even in w3m.
> Don't even think of switching on the dark (night) mode with that attitude! :D
:D
> I really enjoyed the tongue in cheek design. I think every modern browser either allows you to turn on reader mode (especially on mobile) or just turn off CSS. This particular article works excellently even in w3m.
Firefox Focus on mobile does not have Reader Mode.
Fine, sure. Cute - turn on reader mode. Now the images that are supposed to be sitting over yellow background are dark gray images over a slightly less dark gray background.
The decision to design a serious (read: not-tongue-in-cheek) topic with these "quirky" tricks sucks, JMHO.
Works like a normal website with JavaScript disabled. I didn't even know it did fancy junk until reading the comments here. NoScript saves the day again! I don't know how people can browse the web without it.
It is some time ago since I last used it, but I found that too many websites that I want to read require Javascript to even show you the main body of text, or a reasonable layout. Is that different now?
No. Just whitelist the main domain for sites that are obviously broken. Then try one or two likely subdomains if that's not enough. In the rare cases where it still wants more crap enabled, then it's usually not worth the effort, close tab and move on to something else. As you build up a whitelist over time, it becomes pretty rare that you need to interact with it more than a couple times per day. Yeah, it takes some effort, but it's worth it to nuke cookie banners and sticky headers and videos and all the other crap people do with JS.
I already have that routine with uBlock Origin. I don't think NoScript offers all of uBO's functionality, and I certainly won't do the same dance for two extensions, but I'll look into uBO's abilities to specifically block JS.
Makes sense! I use both, uBO just does its thing and I never interact with it. NoScript handles blocking & whitelisting javascript. It's totally possible uBO has a similar feature and I just don't know about it.
I never understood how people can browser the we WITH IT. Even 10 years ago. today more then ever basically every website needs JS to work properly. I basically never come across a page where I have the urge to disable JS. I have a large list of adblock lists active that also help getting rid of cookie banners and other shit.
I can not imagine manually approving JS for every site. And again doing the inverse and have noscript installed to deny one website a year does not seem to be worth it for me. In this case I can also just use a adblock rule to block that specific script or all .js files from the domain I guess. So I really no not need NoScript.
By default the script for the page itself is whitelisted, it is just the third party scripts that are blocked. This works fairly often, but there are a few sites that you can also globally unblock because they provide value. One example is mathjax, used to format equations on many pages.
Many sites don't need JS at all, like the OP of this thread for example. For a lot of sites, disabling JS actually gives a better experience than leaving it enabled, again like the OP of this thread. It's a trade-off, but I find most uses of JS are so bad it's worth putting up with whitelisting. For example, I don't see cookie pop-ups, I don't see videos, disabling JS kills most of those stupid sticky headers that web designers love so much, and whatever too-clever crap the OP of this thread was doing is completely bypassed. The web is so, so much better with JS off by default.
For those sites that do need JS, NoScript's whitelist feature makes it quick & easy to fix. The first time I visit a new site, if it is obviously broken, then I whitelist the main domain. If that doesn't fix it, then I whitelist a couple likely-looking domains (often sites import JS from similar domains, or from common library domains). That's enough to get probably 90+% of websites working, while still leaving most garbage JS disabled. The remaining ~10% of websites that need a dozen domains whitelisted are probably not worth visiting anyway, so I just move on at that point. Or NoScript even lets you temp-whitelist everything for a given tab and just put up with the misery to get whatever I need from that one site. Since the whitelist persists forever, and I don't visit hundreds of different websites every day, after some time it becomes pretty rare that I need to whitelist more than one or two things per day.
You maintain an adblock blacklist, I maintain a NoScript whitelist. Not so different :)
It is obviously a joke (and a good one, I dare say). The fact that people seem to take it seriously says something about the contemporary state of webdesign :)
2. I don't think it's a very good joke to post long-form content on your blog with the expectation that it's basically unreadable without a reader mode.
> The fact that people seem to take it seriously says something about the contemporary state of webdesign :)
Mind expanding and what it says exactly about contemporary web design?
Whether I take it seriously or not doesn't change the fact that it's still damn hard to read anything.
I can agree that most modern web design is bad. I can also agree that the web design on tonsky's site is bad, but OK, I acknowledge that it is intentionally bad; so bad that it's unreadable. I had myself a chuckle now, and next time I see a link to tonsky's site, I'll click on it, chuckle, and immediately leave.
It's deeply ironic that an article about dealing with text properly has images which are part of the article text and yet have no alt-text, rendering parts of the article unreadable in reader mode if the server is slow.
Good times. If you click on the sun switch the entire UI gets zeroed out and you get to use on:hover mouse shtick to read the UI through a fuzzy radius. Is Yoko Ono designing websites now?
> If you can't handle refreshing or merely clicking it again, that's you having a problem, not the site having a problem.
No, it's the site's problem. The contrast between the blinding radioactive yellow background and the font is eye straining and doesn't meet the WCAG standards for accessible text. And the dark mode is unusable. The joke would be funnier if there was a real dark mode or if the light mode was readable.
You said the joke made you leave. The normal color scheme is not part of the joke, and I'm sorry it hurts your eyes. I won't try to defend the eye hurting.
> doesn't meet the WCAG standards for accessible text.
Oh, which part of the standards? When I punch #000000 on #FDDB29 into contrast checkers I get good results.
The single quote isn't as good but all the rest has those colors.
I'm using the Firefox Accessibility Tools. But you're right, I mistook the accessibility warning for the quote/header text for the body text. #000 > #FDDB29 does pass unfortunately.
It's quite possibly the worst web page presentation I've come across in a long time - aside from the fact it looks like some bug has caused my OS to leave a random trail of mouse pointers all over the screen, some of them even move around, making me doubt my sanity when I'm quite sure I'm holding the mouse still.
And the less said about the colours the better. There's no way I was going to put up with that long enough to read all the text on it.
Too bad the Linux and the Mac pointer look so similar. But when you give them different background colors, it becomes more obvious which platform dominates, like:
>That gives us a space of about 11 million code points.
About 170,000, or 15%, are currently defined. An additional 11% are reserved for private use. The rest, about 800,000 code points, are not allocated at the moment. They could become characters in the future.
The ConScript Unicode Registry is a volunteer project to coordinate the assignment of code points in the Unicode Private Use Areas (PUA). Why does tengwar have to be in the PUA, why not make it a first-class charset? It's not just a minor conlang a small group of geeks invented on a weekend, it's a well-established piece of the modern culture, isn't it?
According to a sibling to what you replied to, it's because the shapes of the glyphs are still under copyright by known-litigious rightsholders and the Unicode consortium doesn't want to subject font authors to that.
That's a higher bar than having seen it, I think. I also had to look it up, but as soon as I saw the images in Wikipedia I knew that it's from Lord of the Rings.
Honestly, I wouldn't have thought that would be an issue to the Unicode folks. They have already allowed things (emoji) that have no place being in the standard, as they aren't even text.
According to a sibling to what you replied to, it's because the shapes of the glyphs are still under copyright by known-litigious rightsholders and the Unicode consortium doesn't want to subject font authors to that.
The problem with Tengwar (and Klingon) is the problem with a lot of pop culture right now: copyright. The Tolkien Estate still exists and still litigiously upholds what it can of their copyright terms. CBS Viacom (Paramount) still claim a copyright interest in all the written forms of Klingon.
Copyright is not technically violated simply by encoding the characters into a plane such as one of Unicode's, that's an easy open and shut fair use, but Unicode principals have stated they don't want to pass on the copyright burden to font authors either, which would be sued if they tried to paint some of those characters. (Why encode something that fonts aren't allowed to produce?) That should also be fair use, but the law is complicated and copyright still so often today leans in favor of the Estates and major Corporations rather than fair use and the public commons.
(ETA: I'm hugely in favor that "conlang", constructed language, scripts such as these should be encoded by Unicode. I wish someday we fix the copyright problems of them.)
They are logically equal (that is, they represent the same text in an abstract way), but computing this equality in practice is expensive, because you first need to normalize the strings then compare.
Most languages, when comparing strings, skip the normalization and just compare string bytes as is (or, if the string is interned, compare just the pointer)
You can easily do the comparison dynamically with checking for combining marks, and then do the proper lookup. No need to normalize everything, or even store the normalized variant. Though in a filesystem or username lookup you would only store it normalized.
> I just not sure why they put in the "Angstrom symbol" to begin with.
Frequently, the answer to this is "some obscure character set had this as a distinct symbol." In this case, blame the Japanese: https://en.wikipedia.org/wiki/JIS_X_0208
> People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to?
Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11.
Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last time I checked), I have always been setting the locale to En-US even though I have never been to America. This leads to a number of annoyances though. E.g. LibreOffice always creates new documents for the Letter paper format and I have to switch it to A4 manually every time. It's even worse on Linux where locales appear to be less easy to customize than in Windows. Windows always offered a handy configuration dialog to granularly tweak your locale choosing what measures system you prefer, whether your weeks begin on sundays or mondays and even define your preferred date-time format templates fully manually.
A less-spoken about problem is Windows' system-wide setting for the default legacy codepage. I happen to use single-language legacy (non-Unicode) apps made by people from a number of very different countries. Some apps (e.g. I can remeber the Intel UHD Windows driver config app) even use this setting (ignoring the system locale and system UI language) to detect your language and render their whole UI in it.
> English (USA), English (UK)
This deserves a separate discussion. I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects. To us presence of a huge number of these (don't forget en-AU, en-TT, en-ZW etc - there are more!) in the options lists brings only annoyance, especially when one chooses some non-US one and this opens another can of worms.
By the way I wonder how do string capitalization and comparision functions manage to work on computers of people who use both English and Turkish actively (Turkish locale distinguishes between dotted and undotted İ).
> Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last time I checked), I have always been setting the locale to En-US even though I have never been to America. This leads to a number of annoyances though. E.g. LibreOffice always creates new documents for the Letter paper format and I have to switch it to A4 manually every time. It's even worse on Linux where locales appear to be less easy to customize than in Windows. Windows always offered a handy configuration dialog to granulatly tweak your locale choosing what measures system you prefer, whether your weeks begin on sundays or mondays and even define your preferred date-time format templates fully manually.
There's the English (Denmark) locale for that on some platfoms.
> I doubt many English speakers care to distinguish between English dialects
It's worthwhile purely for the sake of autocorrect/typo highlighting in text-editing software. I don't miss the days of spelling a word correctly in my version of English but still being stuck with the visual noise of red highlighting up and down the document because it doesn't conform to US English.
Yeah I'd rather not have my British English dialect seen as second-class in a world of American English ideally which is what having a red document full of 'errors' implies in those sorts of situations.
It's sometimes not a trivial distinction either, for example I've heard of cases where surprised British redditors have found themselves banned from American subreddits for being homophobic when they were actually talking innocently enough about cigarettes!
I would think a lot of mods, who are either Highly Online Americans or their weirdo equivalents in other countries, are well aware of the UK usage, but simply expect Brits to give it up in order to avoid offending Americans and the global Reddit community that largely takes American-style sensitivity as its orthodoxy. And considering that Reddit corporate feels that anything that could stir up such outrage is bad for business, mods of popular subreddits may well feel pressured to come down hard on these matters.
It doesn't matter if you use UK or US spelling you are wrong. I wish we would adopt the international phonetic alphabet I might have a chance of spelling things correctly.
> This deserves a separate discussion. I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects.
While that is generally (though not always) true, I would assume it's really a stand in for the much more relevant zh locales.
It is also rather relevant to es locales (america spanish has diverged quite a bit from europe spanish hence the creation of es-419), definitely french (canadian french, to a lesser extend belgian and swiss), and german (because swiss german). And it might be relevant for ko if north korea ever stops being what it is.
As an Irish person, while we have en_IE which is great (and solves most of the problems you list re: Euro-centric defaults + English), I'd still quite like to have an even more broad / trans-language / "cosmopolitan" locale to use.
I mainly type in English but occasionally other languages - I use a combination of Mac & Linux - macOS has an (off-by-default but enable-able) lang-changer icon in the tray that is handy enough, but still annoying to have to toggle. Linux is much worse.
Mac also has quite a nice long-press-to-select-special character that at least makes for accessible (if not efficient) typing in multiple languages while using an English locale. Mobile keyboards pioneered this (& Android's current one even does simultanous multi-lang autocomplete, though it severely hurts accuracy).
---
> I doubt many English speakers care to distinguish between English dialects.
I think you'll find the opposite to be true. US English spellings & conventions are quite a departure from other dialects, so typing fluidly & naturally in any non-US dialect is going to net you a world of autocorrect pain in en_US. To the extent it renders many potentially essential spelling & grammar checkers completely unusable.
I write in multiple languages daily on Linux, including English, Russian, and Chinese. Switching keyboards (at least with gnome) is a simple super-space.
While in my default (English) layout, it is easy enough to add in accents other characters using the compose key (right alt). So right-alt+'+a = á or right-alt+"+u = ü. I much prefer this over the long press as I can do it quickly and seamlessly without having to wait on feedback. Granted, it is not as discoverable, but once you are comfortable, it in my opinion is a better system.
I can 2nd this as an American who now resides in Europe. My first laptop I brought with me, and was defaulted to en_US, but my replacement is en_GB (Apple doesn't have en_NL, for good reason).
I don't find it "unusable", though. I could change it back to en_US, but it has actually been interesting to see all of my American spellings flagged by autocorrect. Each time I write authorize instead of authorise it is an act of stubborn group affinity!
> US English spellings & conventions are quite a departure from other dialects.
As far as the written, formal language is concerned, English really has only three dialects: US American, Canadian, and everywhere else. There are some other subtle differences (such as "robots" for traffic lights in South Africa, or "minerals" for fizzy drinks in Ireland¹), but that's pretty much it.
¹ Yes, this isn't just slang in Ireland: the formal, pre-recorded announcements on trains use it: "A trolley service will operate to your seat, serving tea, coffee, minerals and snacks." The corresponding Irish announcement renders it mianraí. Food service on trains stopped during covid and has not yet resumed, so I'm working from distant memory now.
> As far as the written, formal language is concerned, English really has only three dialects
This is true, but I don't see why the "formal" qualifier is needed here :) There are much more than 3 dialects of English, both written & spoken.
Especially there's a fair few extremely common notable differences in (casual, written) Irish English: the word "amn't" (among other less common contractions), the alternative present tense of the verb "to be" (i.e. "do be"), various regional plurals of "you", and - perhaps the most common - prepositional pronouns, etc. etc.
I guess it's a question as to how many varieties of spelling you want to make available as "translations" in software (e.g. color vs colour, tire vs tyre).
There's plenty of regional variants just within the US, but "en_us" covers the whole country.
That's a fair point - even in tiny tiny Ireland there's many regional dialects, with larger countries there'll typically be far more.
I guess the simple answer to that is: how much interest is there in maintenance. I don't think there's any compelling reason not to create something: if there's insufficient interest in maintenance that's an imperfect but reasonable proxy for utility.
I'm not aware of any maintained en_US_Xyz languages but it might be pretty cool if someone started. There's precedence in a few other languages, like no_NO_NY, zh_Hans_HK, etc.
Well, quite. If we include any one or more of the following three categories — formal spoken language, informal spoken language, informal written language — then there's definitely far more than three dialects of English. But formal spoken language really has only the three.
> I have always been setting the locale to En-US even though I have never been to America. This leads to a number of annoyances though. E.g. LibreOffice always creates new documents for the Letter paper format and I have to switch it to A4 manually every time
> I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects. To us presence of a huge number of these (don't forget en-AU, en-TT, en-ZW etc - there are more!) in the options lists brings only annoyance, especially when one chooses some non-US one and this opens another can of worms.
Well, you just explained what this plethora of options is about. It's not just about how you spell flavor/flavour. It's a lot of different defaults for how you expect your OS to present information to you. Default paper size, but also how to write date and time, does the week start on Monday, Sunday, or something else, etc.
> I doubt many English speakers care to distinguish between English dialects
I think you'd be surprised how many english (UK) people will get pissed off when their spell-checker starts removing the "u" from colour or flavour, or how many English (US) people get pissed off when the spellchecker starts suggesting random "u"s to words.
additionally to that, locale isn't just about language. English (US) and English (UK) decides whether your dates get formatted DD-MM-YY or MM-DD-YY, whether your numbers have the thousands broken by commas or spaces, and a host of other localization considerations with a lot more significance than just the dialect of english.
I worked for BP for a while (well, as a contracted coder) and I got quite used to the UK spell check correcting everything to its idiom. Everything seemed wrong once I returned a world that dismissed the value of the letter 'U' and preferred the letter 'Z' over 'S'. Also missed the normalizing of drinking beer at lunch.
> Also missed the normalizing of drinking beer at lunch.
Perhaps you're an old-timer? I worked in the city in the early 80s; lunch in the pub was routine, and sometimes required. By the end of the 80s, that was at best frowned on. Over the last 20 years, having alcohol on your breath after lunch would have been a disciplinary issue, unless you were entertaining a client, at least in the places I worked.
It's likely because I was in the exploration frontier of Alaska. This was late nineties, probably an operation run by people who worked in the city in the '80's and who could continue the old ways far away from the social glare of the head office. :D
I write daily in US English, Australian English, and Austrian German. Most of the time, a specific document is in one dialect/language or another: not mixed, although sometimes that's not true.
I can understand that the conflation of spelling, word choices, time and date formatting, default paper sizes, measurement units, etc, etc, is convenient, and works a lot of the time, but it really doesn't work for me at all.
That said, I appreciate that I occupy a very small niche.
> I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects.
Most people in the UK care - a population nearly twice that of California, and larger than the native speakers of any non-top-20 language. If you care enough to support e.g. Italian you should support en_UK.
> This deserves a separate discussion. I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects. To us presence of a huge number of these (don't forget en-AU, en-TT, en-ZW etc - there are more!) in the options lists brings only annoyance, especially when one chooses some non-US one and this opens another can of worms.
I definitely do. The biggest difference, as everyone else, has pointed out is the US vs UK spellings.
Realistically, though, beyond that country is a poor indicator for everything else. I want to use DD/MM/YYYY date format in English, but DD.MM.YYYY date format in German. I want to use $1,000 in English, but 1.000 $ in German. This isn't dependent on the country I live in, this is dependant on a combination of a country and language - that could be the country I'm living in, or the country I grew up in (mostly US date format vs not), and it's either the language I'm actively typing in, or the language of the document I'm reading, or the language I'm thinking in (but a computer can't exactly handle that).
Trying to guess the correct combination is tricky, especially if a document is in two languages (e.g., a quotation), and users are lazy and won't switch their IME unless they have to.
What this means in slightly more practical terms is that setting a single "locale" for my device doesn't make sense, but rather I should be able to choose a locale per language (or possibly spelling preferences by language and formatting options by language as separate choices). I'd then pick a language to use the device in, and it would use that languages locale, and tell apps that this language is the preferred language. If an app doesn't provide my preferred language, pull the preferred locale from settings for a language it does support, otherwise use the default set by developers. For some apps, it's a bit more complex, particularly if I'm creating content. GMail or Office would be two good examples, where the UI language might be in English, but the emails or documents are in German, or a combination of German and English.
Even then, I'm sure there are people who need something even more flexible than that.
At the moment, if I set my language to English but my Country to Germany on my iPhone, for example, things occasionally get confused. My UK banking app, for example, pulled the decimal separator from my locale settings for a while and then refused to work because "£9,79" (or whatever it was) isn't a valid amount of money, and I couldn't see a way to fix that without switching my Country in the phone settings. I imagine they fixed it by ignoring my configured locale and always using en-GB, thus defeating the whole point of a locale in the first place.
So yeah, these days it's fairly common to not have a single "locale" that you work in - it's quite possible to want to use two or more but nothing is really set up to handle that well.
Unix style locale (set by env vars) is flexible, that can set per app. Android and iOS now support locale per app recently IIRC. Windows locale settings is global and some requires reboot.
Unicode is a total mess. In a sane system, "extended grapheme clusters" would equal "codepoints" and it wouldn't make a difference for 99% of languages. Now we ended up with grapheme clusters, normalization, decomposition, composition, Zalgo text, etc. But instead of deprecating this nonsense, Unicode doubled down with composed Emojis.
I feel its the same as with any long standing computer system we have today. It was designed as more and more of the world came online and all the growing pains it came with. Could it be built from scratch today better? Yes. Will it? No. I suspect it will be around long after we are all dead. Same with IPv4 :V
With something as large as a end user language format for input, this is a change we ourselves cannot make, just as using another calendar for dates. Just because I want to use the year 2002023 calendar with 29.5 days per month, doesn't make it useful to others or myself really.
The problems with Unicode are mostly to do with internal inconsistencies and churn, problems that usually only affect programmers.
1. Different ways to encode the same visually indistinguishable set of characters as code points leading to normal forms, text that compares unequal even when it appears to be identical, the disastrous "grapheme clusters" concept and so on.
2. Many different ways to encode the same sequence of code points as bytes. Not only UTF-32/16/8 but also curiousities like "modified UTF-8".
3. Emoji. A fractal of disasters:
3.a. Updates frequently. Neither Unicode nor software in general was built on the assumption that something as basic as the alphabet changes every year. If you send someone an emoji, can their device draw it? Who knows! In practice this means messaging apps can't rely on the OS system fonts or text handling libraries anymore which is a drastic regression in basic functionality.
3.b. (Ab)uses composition so much it's practically a small programming language, e.g. flags are composed of the two letter country code spelled using special characters. People are represented as as generic person plus skin color patch, families are represented using composed individual people etc.
3.c. Meaning of a character is theoretically specified but can subtly depend on the font used, e.g. people use a fruit emoji in visual puns because of how it looks specifically on Apple devices, so a "sentence" can make no sense if it's rendered with a different font.
3.d. Unbounded in scope. There's no reason the Unicode committee won't just keep adding new pictograms forever.
3.e. Encoded beyond the BMP which in theory every correct program should handle but in practice some don't because nobody except a few academics used characters beyond it much until emoji came along.
3.f. Disagreement over single vs double width chars, can only know this via hard-coded tables, matters for terminals and code editors.
Some of these can potentially be cleaned up outside of the Unicode consortium in backwards compatible ways. You could have a programming language that automatically normalized strings to fully composed form when deserializing from bytes, and then automatically folded semantically identical code points together (this would be a small efficiency win for some languages too). You could campaign to build a consensus around a specific normal form, like how UTF-8 gained consensus as a transfer encoding. You could also define a fork of Unicode (using private use areas?) that allocates a single code point to the characters that are unnecessarily using composition today but don't yet have one and then just subset out the concept of composition entirely.
Emoji are a big problem. It's tempting to say that these should not be encoded as characters at all. Instead there could be a set of code points that define bounds that contain a tiny binary subset of SVG, enough to recreate the Apple pixel art somewhat closely. Emoji would always be transmitted as inlined vector art. Text rendering libraries would call out to a little renderer for each encoded glyph, using a fast fingerprinting algorithm to deduplicate the bytes to an internal notion of a character. To avoid wire bloat, text can simply be compressed with a pre-agreed zstd or Brotli dictionary that contains whatever images happen to be popular in the wild. At a stroke this would avoid backwards compat problems with new emoji, enabling programs working with text to be upgraded once and then never again, eliminate all the ridiculous political committee bike-shedding over what gets added, let apps go back to using system text support and get rid of the bajillion edge cases that emoji have spewed all over the infrastructure.
For most software it doesn't really matter either.
I've written unicode-aware software for over a decade, doing a wide variety of programs, and I've never had to bother with all that mess.
If I'm parsing strings I'm looking for stuff in the 7-bit ASCII range which maps neatly onto the Unicode representations, and so I just need to take care to preserve the rest.
The only trouble I've had is that a lot of programmers haven't learned, or don't get, that text encoding is a thing and that it needs to be handled.
So they'll hand me an XML they claim is UTF-8 encoded, except that XML header was just copypasta and the actual XML document is encoded in some other system encoding like Windows-1252. Or worse, a mix of both.
Honestly I like ipv4 better than v6. I like having a NAT and easy addresses like 192.168.1.3 instead of fe80::210:5aff:feaa:20a2. They didn't need to mess with those things just to expand the address space, like how utf8 didn't require remapping ASCII.
IPv4.1 should have just had 39 bits, to be written like 999.999.999.999. (I know this wouldn't have actually had much effect, nobody is going to add new routes in the middle of "class A" spaces that already existed, so it would just give those that already had IP addresses more IP addresses. Additionally, people really abuse decimal addresses in horrifying ways; for example, Fios steals 192.168.1.100-192.168.1.150 for its TV service, and that range doesn't really correspond to anything that you can mask off in binary. It only makes sense in decimal, which is not what any underlying machinery uses. They should have given themselves a /26 or something. You get 3 for yourself (modulo the broadcast and gateway address), and they get 1 for TV.)
Having it actually be decimal might've been nice, but at this point people are used to the 1-254 range, and I think the least jarring addition of extra bits would be to simply extend it for the addresses that need them (and not for the ones that don't). So you could have 123.444.3.254 or longer like 123.444.3.254.12.43.
The writing systems were already like this when we got them. Unicode's "total mess" mostly just reflects that. Of course it would be convenient for you, the programmer, if the users wanted the software to do whatever was easiest for you, but obviously they want what's easiest for them, not you.
you can't not handle devanagari, tamil (or like half the scripts across the Indian subcontinent and oceania) or hangul. even the IPA, used by linguists every day, would be particularly bad to deal with if we couldn't write things like /á̤/, and some languages already don't have the precomposed diacritics for all letters (like ǿ), so the idea of a world with only precomposed letter forms is more of a exponential explosion in the character set
> so the idea of a world with only precomposed letter forms is more of a exponential explosion in the character set
"Exponential explosion" is really putting it too strong; it's perfectly possible to just add ǿ and á̤ and a bunch of other things. The combinations aren't infinite here.
The problem with e.g. Latin script isn't necessarily that combining characters exist, but that there's two ways to represent many things. That really is just a "mess": use either one system or the other, but not both. Hangul has similar problems.
Devanagari doesn't have any pre-compose characters AFAIK, so that's fine.
That's really the "mess": it's a hodgepodge of different systems, and you can't even know which system to use a lot of the time because it's not organised ("look it up in a large database"), and even taking in to account historical legacy I don't think it really needed to be like this (or is even an unfixable problem today, strictly speaking).
At least they deprecated ligatures like st and fl, although recently I did see ij being used in the wild.
They certainly are. Languages are a creative space driven by the human imagination. Give people enough time and they'll build new combinations for fun or for profit or for research or for trying to capture a spoken word/tone poem in just the right sort of exciting way. You may frown on "Zalgo text" [1] (and it is terrible for accessibility), but it speaks to a creative mood or three.
The growing combinatorial explosion in Unicode's emoji space isn't an accident or something unique to emoji, but a characteristic that emoji are just as much a creative language as everything else Unicode encodes. The biggest difference is that it is a living language with a lot of visible creative work happening in contemporary writing as opposed to a language some monks centuries ago decided was "good enough" and school teachers long ago locked some of the creative tools in the figurative closets to keep their curriculum simpler and their days with fewer headaches.
Well, in theory it's infinite, but in reality it's not of course.
We've got 150K assigned codepoints assigned, leaving us with 950K unassigned codepoints. There's truly massive amounts of headroom.
To be honest I think this argument is rather too abstract to be of any real use: if it's a theoretical problem that will never occur in reality then all I can say is: <shrug-emoji>.
But like I said: I'm not "against" combining marks, purely in principle it's probably better, I'm mostly against two systems co-existing. In reality it's too late to change the world to decomposed (for Latin, Cyrillic, some others) because most text already is pre-composed, so we should go full-in on pre-composed for those. With our 950k unassigned codepoints we've got space for literally thousands of years to come.
Also this is a problem that's inherent in computers: on paper you can write anything, but computers necessarily restrict that creativity. If I want to propose something like a "%" mark on top of the "e" to indicate, I don't know, something, then I can't do that regardless of whether combining characters are used, never mind entirely new characters or marks. Unicode won't add it until it sees usage, so this gives us a bit of a catch-22 with the only option being mucking about with special fonts that use private-use (hoping it won't conflict with something else).
The Unicode committees have addressed this for languages such as Latin, Cyrillic, and others and stated outright that decomposed forms should be preferred and decomposition canonical forms are generally the safest for interoperability and operations such as collation (sorting) and case folding (lowercase to uppercase transformations).
Unicode can't get rid of the many precombined characters for a huge number of backward compatibility reasons (including compatibility with ancient Mainframe encodings such as EBCDIC which existed before computer fonts had ligature support), but they've certainly done what they can to suggest the "normal" forms in this decade should "prefer" the decomposed combinations.
> If I want to propose something like a "%" mark on top of the "e" to indicate, I don't know, something, then I can't do that regardless of whether combining characters are used
This is where emoji as a living language actually shines a living example: It's certainly possible to encode your mark today as a ZWJ sequence, say «e ZWJ %», though you might want to consider for further disambiguation/intent-marking adding a non-emoji variation selector such as Variation Selector 1 (U+FE00) to mark it as "Basic Latin"-like or "Mathematical Symbol"-like. You can probably get away with prototyping that in a font stack of your choosing using simple ligature tools (no need for private-use encodings). A ZWJ sequence like that in theory doesn't even "need" to ever be standardized in Unicode if you are okay with the visual fallback to something like "e%" in fonts following Unicode standard fallback (and maybe a lot of applications confused by the non-recommended grapheme cluster). That said, because of emoji the process for filing new proposals for "Recommended ZWJ Sequences" is among the simplest Unicode proposals you can make. It's not entirely as Catch-22 on "needs to have seen enough usage in written documents" as some of the other encoding proposals.
Of course, all of that is theory and practice is always weirder and harder than theory. Unicode encoding truly living languages like emoji is a blessing and it does enable language "creativity" that was missing for a couple of decades in Unicode processes and thinking.
> The Unicode committees have addressed this for languages such as Latin, Cyrillic, and others and stated outright that decomposed forms should be preferred
Yes, and that only makes things worse since the overwhelming majority of documents (99.something% last time I checked) uses pre-composed. Also AFAIK just about everyone just ignores that recommendation.
I suppose "e ZWJ %" is a bit better than Private Use as it will appear as "e%" if you don't have font support, but the fundamental problem of "won't work unless you spend effort" remains. For a specific niche (math, language study, something else) that's okay, but for "casual" usage: not so much. "Ship font with the document" like PDF and webfonts do is an option, but also has downsides and won't work in a lot of contexts, and still requires extra effort from the author.
I'm not saying it's completely impossible, but certainly harder than it used to be, arguably much harder. I could coin a new word right here and now (although my imagination is failing me to provide a humorous example at this moment) and if people like it, it will see usage. In 1960s HN when we would have exchanged these things over written letters, and it would have been trivial to propose a "e with % on top" too, but now we need to resort to clunky phrases like this (even for typewriters you can manually amend things, if you really wanted to).
Or let me put it this way: something like ‽ would see very little chance of being added to Unicode if it was coined today. Granted, it doesn't see that much use, but I do encounter it in the wild on occasion and some people like it (I personally don't actually, but I don't want to prevent other people from using it).
None of this is Unicode's fault by the way, or at least not directly – this is a generic limitation of computers.
> Yes, and that only makes things worse since the overwhelming majority of documents (99.something% last time I checked) uses pre-composed.
It shouldn't matter what's in the wild in documents. That's why we have normalization algorithms and normalization forms. Unicode was built for the ugly reality of backwards compatibility and that you can't control how people in the past wrote. These precomposed characters largely predate Unicode and were a problem before Unicode. Unicode won in part because it met other encodings where they were rather than where they wished they would be. It made sure that mappings from older encodings could be (mostly) one-to-one with respect to code points in the original. It didn't quite achieve that in some cases, but it did for, say, all of EBCDIC.
Unicode was never in the position to fix the past, they had to live with that.
> This is a classic "reality should adjust to the standard" type of thinking.
Not really. The Unicode standard suggests the normal/canonical forms and very well documented algorithms (including directly in source code in the Unicode committee-maintained/approved ICU libraries) to take everything seen in the wilds of reality and convert them to a normal form. It's not asking reality to adjust to the standard, it is asking developers to adjust to the algorithms for cleanly dealing with the ugly reality.
> Or let me put it this way: something like ‽ would see very little chance of being added to Unicode if it was coined today.
Posted to HN several times has been the well documented proposal process from start to finish (it succeeded) of getting common and somewhat less common power symbols encoded in Unicode. It's a committee process. It certainly takes committee time. But it isn't "impossible" to navigate and is certainly higher than "little chance" if you've got the gumption to document what you want to see encoded and push the proposal through the committee process.
Certainly the Unicode committee picked up a reputation for being hard to work with in the early oughts when the consortium was still fighting the internal battles over UCS-2 being "good enough" and had concerns about opening the "Astral Plane". Now that the astral plane is open and UTF-16 exists, the committee's attitude is considered to be much better, even if its reputation hasn't yet shifted from those bad old days.
> None of this is Unicode's fault by the way, or at least not directly – this is a generic limitation of computers.
Computers do anything we program them to do and in general people find a way regardless of the restrictions and creative limitations that get programmed. I've seen MS Paint drawn symbols embedded in Word documents because the author couldn't find the symbol they needed or it didn't quite exist. It's hard to use such creative problem solving in HN's text boxes, but that from some viewpoints is just as much a creative deficiency in HN's design. It's not an "inherent" problem to computers. When it is a problem they pay us software developers to fix it. (If we need to fix it by writing a proposal to a standards committee such as the Unicode Consortium, that is in our power and one of our rights as developers. Standards don't just bind in one-direction, they also form an agreement of cooperation in the other.)
The thing with normalization is that it's not free, and especially for embedded use cases people seem quite opposed to this. IIRC it requires about ~100K of binary size, ~20K of memory, and some non-zero number of CPU cycles. This is negligible for your desktop computer, but for embedded use cases this matters (or so I've been told).
This comes up in specifications that have a broad range of use cases; when I was involved in this my idea was to just spec things so that there's only one allowed form; you'll still need a small-ish table for this, but that's fine. But that's currently hard because for a few newer Latin-adjacent alphabets some letters cannot be represented without a combining character.
So then you have either the "accept that two things which seem visually similar are not identical" (meh) or "exclude embedded use cases" (meh).
I never really found a good way to unify these use cases. I've seen this come up a few times in various contexts over the years.
> Posted to HN several times has been the well documented proposal process from start to finish (it succeeded) of getting common and somewhat less common power symbols encoded in Unicode.
Would this work for an entirely new symbol I invent today? It's not really the Unicode people that are "difficult" here as such, they just ask for demonstrated usage, which is entirely reasonable, and that's hard to get (or: harder than it was before computers) especially for casual usage. I'm sure that if some country adopts/invents a new script today, as seems to be happening in West-Africa at in recent years, the Unicode people are more than amendable to work with that, but "I just like ‽" is a rather different type of thing.
> Would this work for an entirely new symbol I invent today? It's not really the Unicode people that are "difficult" here as such, they just ask for demonstrated usage, which is entirely reasonable, and that's hard to get (or: harder than it was before computers) especially for casual usage.
Sure, they want demonstrated usage as inline in the flow of text as textual elements as opposed to purely iconography or design elements (because such things are outside of Unicode's remit, modulo some old Wingdings encoded for compatibility reasons and the fine line between emoji are expressive text and also emoji are useful for iconography in many cases). But at this point (again in contrast to the UCS-2/no-Astral-plane days) the committees don't seem to care how it was mocked up (do it on a chalkboard, do it in paint, do it in LaTeX drawing commands, whatever gets the point across) or how "casual" or infrequent the usage is, so long as you can state the case for "this is a text element" (not an icon!) used in living creative language expression. There's more "provenance" requirements for dead languages and they'll want some number of academic citations, but for living languages they've come to be flexible (no hard requirements) on the number of examples they need from the wild and where those are sourced from. Showing it in old classic documents/manuals/books, for instance, helps the case greatly, but the committees today no longer seem as limited to just what can be used to demonstrate usage. "I just like it" is obviously not a rock solid proposal/defense to bring to a committee (any committee, really), but that doesn't mean that is impossible for the committee to be swayed by someone making a strong enough "I just like it" case if they demonstrate well enough why they like it and how they use it and how they think other people will use it (and how those uses aren't just iconography/decorative elements but useful in the inline context of textual language).
The problem is not that you need character composition for some writing systems. It's that there are no rules that would help with everything having an unique representation internally.
Even "put the code points forming the composed character in descending numerical order" would be better than nothing. If it was there from the start.
However, the Unicode commitee is too busy adding new emojis to make their standard sane.
There are rules for that, Unicode has standards (not only formal, but easily usable in most software libraries) for canonical forms that will collapse all the variations to a single representation.
But, of course, unicode can't define that the standard will cover only the canonical forms, and couldn't do that since the start, as it needed backwards compatibility with various pre-unicode encodings which had mutually incompatible principles, so it needed support for both composed and decomposed versions of the same characters.
because the current mess means all their old stuff still works. ASCII is good so long as you only need English (or any other latin languages without the various accents), which was good enough for a long time - and ASCII was also carefully designed to make programming easier - flip one bit changes lower/uppercase for example, but there are more things it makes easy. By the time we realized we actually care about the rest or the world it was too late to make a nice system.
The real world never respected your artificial ASCII limitations, so that part never worked because people always needed more. But the original comment states that composition is a source of mess, not the ASCII having the same code point, and that's the puzzling part
It's unnecessary complexity and a security nightmare. Have you ever tried to implement Unicode normalization? A single bug in your code and malformed text can crash your application or worse.
That's tricky, for sure. My 'workaround' has long been converting codepoints into byte sequences and creating a character dictionary from that. Based on the source corpus, this dictionary can be further expanded/compressed and used for downstream processing.
They kind of have to don’t they? Otherwise we’ll become space-limited way too fast? Especially with how quickly new emojis are being made and all their variants.
Not just emojis, in general I believe Unicode has just said they're not going to add new pre-composed characters and that using combining characters is the Right Way™ to do things (well, the only way for newer scripts).
One of the downsides of writing down specifications is that they tend to attract people with Very Strong Opinions on the One And Only Right Way and will argue it to no end, and essentially "win" the argument just by sheer verbosity and persistence.
That's certainly what I've seen happen in a few cases, and is what happens on e.g. Wikipedia as well at times.
But yeah, emojis is even worse. Something things can look rather different depending on which invisible variation selector is present. We've got tons and tons of unassigned codepoints and we need to resort to these tricks to save a few of them?
Firefighter is "(man|woman|person) + ZWJ + firetruck". Clever, I guess. Construction worker is "Construction worker (+ ZWJ + (male sign|female sign))?" (absence is gender-neutral). Why are there 2 systems to encode this? Sigh...
All of this is too clever by a mile.
[1]: HN will strip stuff, but try something like:
echo $'↔\ufe0f ↔\ufe0e'
May not display correctly in terminal, but can xclip it to a browser – screenshot: https://imgur.com/a/iFmBDQk
This is NOT a complaint about that they added diversity as such, in principle I'm all for that, it's just that few seem to actually use these emojis, and both in terms of code and UI it all gets pretty complex; there's 98 combinations to choose from here.
I don't really get why <heart> or <killing lips> or <kissing face> isn't enough. That's actually what most people seem to use anyway, because who finds it convenient to pick all the correct genders and skin tones from the UI for both people?
Less than that since a default skin color can be set in most apps. I'm sure setting a gender will come soon so the entire first part of that emoji can be auto-guessed. Then its just showing the other options in the UI. Really all of this is UI design as even with the 98 combinations you can still display it as 4/5 options you drill down.
> who finds it convenient to pick all the correct genders and skin tones from the UI for both people?
I just checked and searching "kissing" in my iOS emoji keyboard inside Messenger showed just 4 of the emoji's your describing - defaulting both skin tones to my settings and then the four M/F pair ups. Plus some non-related kissing emojis like the cat kissing.
But that's kind of wrong, no? The entire point is that you can choose both sides individually. What if you set it to black and want to kiss some white bloke?
If anything that only underscores my point that it's too complex and that no one is using them (certainly not as intended anyway).
That seems like a lot of effort when you could have sent <kissing-lips>, <kissing-cat>, <kissing-face>, <heart>, or any number of other emojis, which is what my point was.
But precomposing all the potential combinations is less sane than the current mess (and you can outlaw Zalgo in the standard if you think it's a serious issue)
Also, the % should measure people, not languages, that would greatly decrease the imaginary 99%
I tried to read the articles since it seemed interesting. After exactly 30 seconds trying it I had to leave the page. Impossible to read more than two sentences with all those pointer moving there - and for a folk with ADHD even more difficult. Sorry, but I couldn't make it :(
Not everyone runs Linux, and not every browser has a reader mode. This should not be the solution. There should definitely be an option to disable all these features, especially the dark mode toggle, that one's a fun premise, but horrific for usability.
Well, there is a new fact that I learned and immediately hated.
The fuck were authors thinking...
I am now firmly convinced people developing unicode hate developers. I suspected it before just due to how messy it was (same character having different encodings ? Really ? Fuck you), but this cements it.
Yeah this is a big problem for me right now trying to pick fonts and characters for CJK. I have a bunch of bugs to fix that will require sending the locale down to the text itemization code.
Well C is locale dependent. And one does not break backwards compatibility with C for fear of badness. So naturally Unicode must be locale dependent too.
This is pretty good. One thing I would add is to mention that Unicode defines algorithms for bidirectional text, collation (sorting order), line breaking and other text segmentation (words and sentences, besides grapheme clusters). The main point here is to know that there are specifications one should take into account when topics like that come up, instead of just inventing your own algorithm.
The author seem to hate people which concentration issues and/or various visual sicknesses.
That coloration tools shows the moving mouse coursers of other participants even if they aren't needed/wanted is already pretty bad, why bring it to a website?
This seems like good feedback but it could really be phrased more constructively. I doubt the author “hates” any such thing and you know it too. “Didn’t design with such in mind”, sure. You can do better.
It's tracking every visitors' cursor and sharing it with every other visitor.
Why would a frontend developer demonstrate their ability to do frontend programming on their personal, not altogether super-serious blog? I meant that rhetorically but it's a flex. I agree, not the best design in the world if you're catering for particular needs, but simple and fun enough. You should check out dark mode.
In that vein, I think it's okay if we let people have fun. That might not work for everyone, but why should we let perfect be the worst enemy of fun?
because it shows that they don't understand important design aspects
while it doesn't really show off their technical skills because it could be some plugin or copy pasted code, only someone who looks at the code would know better. But if someone care enough about you to look at your code you don't need to show of that skill on you normal web-site and can have some separate tech demo.
> okay if we let people have fun
yes people having fun is always fine especially if you don't care if anyone ever reads your blog or looks at it for whatever reason (e.g. hiring)
but the moment you want people to look at it for whatever reason then there is tension
i.e. people don't get hired to have fun
and if you want others to read you blog you probably shouldn't assault them with constant distractions
I assume the creator didn't anticipate this amount of readers at the same time and having one or two other cursors on the page does sound fun and not too distracting. They should probably limit the maximum amount of other cursors displayed to a sensible amount
Not every website, even technical ones, need to have an eye towards professional advancement. Sometimes they're just for fun. I welcome it, as it's a thing that gets more rare on the web as time goes by.
Creative expression allows us to push ourselves, both in what we think we can do, and often the technical aspects about how we do it too. Even if the idea doesn't stick, you've tried something new.
In a world of Tailwinds and Bootstraps and the same five templates copied again and again and again, let's celebrate the people willing to push things and learn from their inevitable but ultimately valuable mistakes. And let's have some fun along the way.
It's revenge against anyone with certain kinds of visual impairments and/or concentration issues because the ex-spouse of the author which turned out to be a terrible person had such.
(sarcasm try 2)
It's revenge against anyone using JS on the net with the author trying to subtle hint that JS is bad.
(realistic)
It's probably on of:
- the website is a static view of some collaborative tool which has that functionality build in by default
- some form of well intended but not well working functionality add to the site as it was some form of school/study project, in that case I'm worried about the author suffering unexpected very much higher cost due to it ending up on HN ...
Hi, author here. In case you really want to know: no, it’s custom-made and works exactly as intended. There are two main reasons:
1. Fun. Modern internet is boring, most blog posts are just black text on white background. Hard to remember where you read what. And you can’t really have fun without breaking some expectations.
2. Sense of community. Internet is a lonely place, and I don’t necessarily like that. I like the feeling of “right now, someone else reading the same thing as I do”. It’s human presence transferred over the network.
I understand not everybody might like it. Some people just like when things are “normal” and everything is the same. Others might not like feeling of human presence. For those, I’m not hiding my content, reader mode is one click away, I make sure it works very well.
As for “unexpectedly ended up on HN”, it’s not at all unexpected. Practically every one of my general topic articles ends up here. It’s so predictable I rely on HN to be my comment section.
2. I only understood that it was actual other people's mouse cursors when I read that here. So it didn't really engender a sense of community, although after some time I did think you are very good at modelling actual human mouse movements. Now that I know it, it's pretty neat though.
The cursors will only be a problem during front page HN traffic. And the opt-out for people who care is reader mode / disable js / static mirror. Not sure if there's any better way to appease the fun-havers and the plain content preferrers at the same time. Maybe a "hide cursors" button on screen? I, for one, had a delightful moment poking other cursors.
I don't know what you people are talking about. I'm just glad I always browse with Javascript turned off. If you didn't see the writing on the wall and permanently turn Javascript off around 2006, you have no right to complain about anything.
Meanwhile, ironic irony is ironic: "Hey, idiots! Learn to use Unicode already! Usability and stuff! Oh, btw, here is some extremely annoying Javascript pollution on your screen because we are all still children, right? Har har! Pranks are so kewl!!!1!"
I stopped in the middle of reading the post just for this. It was so distracting I was unable to focus on the text. It's a fun gimmick, but the result is that someone who wanted to read the post, stopped in the middle.
> The only modern language that gets it right is Swift:
I disagree.
What is the "right" things is use-case dependent.
For UI it's glyph bases, kinda, more precise some good enough abstraction over render width. For which glyphs are not always good enough but also the best you can get without adding a ton of complexity.
But for pretty much every other use-case you want storage byte size.
I mean in the UI you care about the length of a string because there is limited width to render a strings.
But everywhere else you care about it because of (memory) resource limitations and costs in various ways. Weather that is for bandwidth cost, storage cost, number of network packages, efficient index-ability, etc. etc. In rare cases being able to type it, but then it's often us-ascii only, too.
Arguably, you don’t need any (default) length at all, just different views or iterators. When designing a string type today, I wouldn’t add any single distinguished length method.
Swift string type has got many different views, like UTF-8, UTF-16, Unicode Scalar, etc… so if you want to count the bytes or cut over a specific byte you still can.
as in they should things you can just use by-default without thinking about it
as swift is deeply rooted in UI design having a default of glyphs make sense
and as rust is deeply rooted in unix server and system programming utf-8 bytes make a lot of sense
through the moment your language becomes more general purpose you could argue having a default in any way is wrong and it should have multiple more explicit methods.
There was no string.length in Swift for a while. Then they added one that just does what the user expects, get the number of grapheme clusters. If a user figures out that this isn't what they want, they can go use the other length method.
The most common reason I can think of a server wanting the string length is because it's enforcing a character limit in some field, in which case it does what the user expects. You aren't often manually managing memory in Swift. On top of that, Swift on server is probably rare to begin with.
It has distinct .chars .codes and .bytes that you can specify depending on the use case. And if you try to use .length is complains asking you to use one of the other options to clarify your intent.
my \emoji = "\c[FACE PALM]\c[EMOJI MODIFIER FITZPATRICK TYPE-3]\c[ZERO WIDTH JOINER]\c[MALE SIGN]\c[VARIATION SELECTOR-16]";
say emoji; #Will print the character
say emoji.chars; # 1 because on character
say emoji.codes; # 5 because five code points
say emoji.encode('UTF8').bytes; # 17 because encoded utf8
say emoji.encode('UTF16').bytes; # 14 because encoded utf16
Swift made an effort to handle grapheme clusters but severely over-complicated strings by exposing performance details to users. Look at the complex SO answers to what should be simple questions, like finding a substring: https://news.ycombinator.com/item?id=32325511 , many of which changed several times between Swift versions
I was working on an app in Swift that needed full emoji support once. Team ended up writing our own string lib that stores things as an array of single-character Swift strings.
You still see all the answers from old versions sitting around, often at the top. Part of it is because of how often they changed such fundamental things. String length changed 3 times. Every other language figured these things out before the initial non-beta release.
Also, realized "needed full emoji support" sounds silly. It needed to do a lot of string manipulation, with extended grapheme clusters in mind, mainly for the purpose of emojis.
Rust was given as one of the examples and Rust's .len() behaviour is chosen based on three very reasonable concerns:
1. They want the String type to be available to embedded use-cases, where it's not reasonable to require the embedding of the quite large unicode tables needed to identify grapheme boundaries. (String is defined in the `alloc` module, which you can use in addition to `core` if your target has a heap allocator. It's just re-exported via `std`.)
2. They have a policy of not baking stuff that is defined by politics/fiat (eg. unicode codepoint assignments) into stuff that requires a compiler update to change. (Which is also why the standard library has no timezone handling.)
3. People need a convenient way to know how much memory/disk space to allocate to store a string verbatim. (Rust's `String` is just a newtype wrapper around `Vec<u8>` with restricted construction and added helper functions.)
That's why .len() counts bytes in Rust.
Just like with timezone definitions, Rust has a de facto standard place to find a grapheme-wise iterator... the unicode-segmentation crate.
Just a nitpick because the page says: "Unicode is a standard that aims to unify all human languages, both past and present, and make them work with computers." but of course unicode is only relevant to written languages as opposed to spoken languages (and signed languages)
I wish that was the only thing wrong with that page
The E0000-E007F block is the "Tags" block, which is used for flag emojis.
But there is not a code for each flag. Instead there is a code for each ASCII character.
A flag sequence is formed from U+1F3F4 (Black Flag), followed by at least two tags that form a country/region code, and then U+E007F (End tag).
So, yes this is weird, because the emoji is dependent on the decoder. It was made this way to keep Unicode independent of geopolitics.
I wondered about how to do simple text centering / spacing justification given graphemes showing string lengths that don't match up human-perceived characters, like in 'Café' (python len('Café') returns 5, even though we see four letters).
(apparently the article talks about this however the blog post is largely unreadable due to dozens of animated arrow pointers jumping all over the screen)
This is quite a good write up. An answer to one of the author's questions:
> Why does the fi ligature even have its own code point? No idea.
On of the principles of Unicode is round trip compatibility. That is you should be able to read in a file encoded with some obsolete coding system and write it out again properly. Maybe frob it a bit with your unicode-based tools first. This is a good principle, though less useful today.
So the fi ligature was in a legacy encoding system and thus must be in Unicode. That's also why things like digits with a circle around them exist: they were in some old Japanese character set. Nowadays we might compose them with some zwj or even just leave them to some higher level formatting (my preference).
The circled digits as code points are very nice to have precisely because they are available in applications that don't support them otherwise... which is actually most of the software I can think of (Notepad, Apple Notes, chat applications, most websites, etc).
Can you write them with iOS keyboard? Or when you say Apple Notes and chat apps you just mean from desktop?
Edit ①: seems the answer is not with the default iOS keyboard, but possible to paste it and perhaps possible with a third party keyboard that I'm not keen on trying (unless I hear of a keyboard that's both genuinely useful / better than default, and that doesn't send keystrokes to the developer - though I can't remember if the latter is even a risk on iOS, better go search about that next..)
Speaking of third party keyboards, I’m still upset about what happened to Nintype[0]. I’ve never ever been able to type faster on mobile than with it’s intuitive hybrid input style of sliding and tapping, paired with AI that was actually good. It used to be quite performant, fully customizable, and it worked beautifully as a replacement for default on jailbroken iOS.
Today, it’s buggy $5 abandonware that only makes me sad when I am reminded of it.
EDIT: Here[1] is a blog post that claims it's still the best keyboard in 2023. I actually might give it another shot... Not holding my breath though.
*EDIT: Looks like another dedicated fan has actually taken it upon themself to revive the project, under the new name Keyboard71[2].
I’m really considering repurchasing (I definitely owned it previously, no idea what happened), can you describe specifically what the main bugs are for you? I’d be happy if I could use it solely for occasionally writing long notes, not as a replacement for all text inputs.
Really not looking to burn another $5, I’d greatly appreciate any thoughts/concerns at all.
You can type ① with the UniChar keyboard app on iOS. It at least claims it doesn’t transmit information. As it’s only useful for special characters I don’t worry because I can’t use it for normal typing anyway.
My point was that, had they not been legacy characters (or had RT compatibility been disregarded) Unicode could still have supported them as composed characters. Though I personally still feel they are a kind of ligature or graphic, but luckily for everyone else I’m not the dictator of the world :-).
We should be careful: someone on HN could write a proposal that they should be considered pre composed forms that should also have an un-composed sequence… so there could in future be not just 1 in a circle but 1 ZWJ circle, circle ZWJ 1 all considered the same…I can imagine some HN readers being pranksters like that.
This implies that they're obsolete, but they're not -- they're still in very common use today. You can type them in Japanese by typing まる (maru, circle) and the number, then pick it out of the IME menu. Some IMEs will bring them up if you just type the number and go to the menu, too. :)
> So the fi ligature was in a legacy encoding system and thus must be in Unicode.
Most of the pre-composed latin ligatures are generally from EBCDIC codepages. People in the ancient Mainframe era wanted nice typesetting too, but computer fonts with ligature support were a much later invention.
You can see fi and several others directly in EBCDIC code page 361:
Thanks. Some alphabets have precomposed ligatures that aren't really letters, like old German alphabets with tz, ch, ss (I only know how to type the last one, ß, because the others have died out over the last hundred years).
Actually in German (at least) ä, ö and ü really are actually ligatures for ae, oe, and ue -- the scribes started to write the E's on their sides above the base letters, and over time the superscript "E"s became dots or dashes. Often they are described the other way around: "you can type oe if you can't type ö." That's what my kid was told in school!
But Ö and ß aren't really part of the alphabet in German, while, say, in Swedish, ä and ö became actual letters of the alphabet. English got W that way too.
That's sounds a bit false to me. The Umlaute (ä,ö, ü) and the "eszett" ß are actually part of the German alphabet[1]. Also it is kinda weird to describe them as ligatures of the original letters and the diaeresis, because while this is what they started out as a long time ago, they are just their own letters now (as opposed to "real" stylistic ligatures like combining fi into one glyph). The advice your kid was told that they can be replaced with ae, oe and ue is correct - it is a replacement nowadays.
They sure are letters, but they aren't generally thought of as being in the alphabet (which seems to be why they are just kinda tacked on after a space on wikipedia) and get ordered as if they where just the base letter (mostly)
Letters which are sorted separately from what we'd think of as the base characters in English (they appear at the end of the alphabet, as W X Y Z Æ Ø Å, with C often omitted in Norwegian).
By contrast, my French dictionary has énorme nestled between enorgueillir and enquiérir. (Looking for an example does underscore some of the patterns in the language: page after page of ét~ with only a few et~ and one êt~ among them; pages of ex~ with no éx~ at all.)
Similarly in Swedish, W was not considered a letter but just a variant of V, so in phone books etc all the W names were mixed in with V names. This was changed in 2006 due to an increase in English loanwords.
C'mon that page is highly technical, really just listing the letters or glyphs used for forming printed text. In reality, if you walk into any first grade classroom you see pictures of the letters A-Z with pictures (Apfel, Bär, uwv) and then after the end maybe around the corner, what, Öl? I can't even remember. When the kids recite the letters they don't recite äöüß. TBH I really only remember this because when kiddo was that age the Neue Rechtscribung transition was mid-process and the parents were angrily divided so I was kinda paying attention.
Also, though it's hardly authoritative, my kids' school taught English through immersion from grade 1 too, and both German and English teachers said "same alphabet".
As bmicraft pointed out, even in that wikipedia chart those inflected letters are spaced apart from the others. Yes, they are letterforms, but not part of the "alphabet" -- they don't even have a sorting like the Swedish Ä or W do.
And you can switch in running text from using the marker for umlaut (dots or bar, not semantically dieresis) or a normal "e" without anyone blinking. There's no problem reading a Swiss book even though ß refuses to cross the border. Though I personally prefer to read Äpfel and Bär rather than Aepfel and Baer, really, they are the same.
> When the kids recite the letters they don't recite äöüß.
On a somehow related side note, I read that "&", which is derived from a ligature of the Latin conjunction "Et" (meaning "and"), was named "ampersand" in English as a mondegreen for "and per se and" as it used to be placed at the end of the English alphabet recitation.
You sent me on an enjoyable wild goose chase but it appears that only ß and ff are in unicode: tz, ch, ck have to be handled completely in rendering.
I have my music teacher's German schoolbook from around 1915 and it lists them all as letters (the whole book is in Fraktur). I have various old books in Fraktur and once could read them. I imagine that if I sat down and tried to read one it would come back, but at the moment I have to thin a little to read just the titles!
571 comments
[ 2.8 ms ] story [ 348 ms ] threadProblem solved!
But any more saturation and it would go all mustard-electric on me.
That's an interesting observation on variation of saturation response. Feels like useful knowledge for ... web site designers. Or any color crafter.
It's black text on black background (I'm on mobile Firefox on Android).
Future _improvement_ idea: the mouse cursors are shared, so the light switch should be, too! Let me play with the light with everyone
Deleting that makes the while thing a lot less stressful.
Wow, what abominable mix of decimal and hexadecimal.
doh.
So the author is using hex for the last four digits and decimal for the remaining ones.
eg https://www.npr.org/2023/05/02/1173418268/loneliness-connect...
Probably not the greatest during a hacker news hug of death, but if I read that article some other time and saw one of the moving pointers, I would think it was really cool.
[0] WS connection is on `wss://tonsky.me/pointers?id=XXXXXX&page=/blog/unicode/&platform=XXX`
I really enjoyed the tongue in cheek design. I think every modern browser either allows you to turn on reader mode (especially on mobile) or just turn off CSS. This particular article works excellently even in w3m.
:D
> I really enjoyed the tongue in cheek design. I think every modern browser either allows you to turn on reader mode (especially on mobile) or just turn off CSS. This particular article works excellently even in w3m.
Firefox Focus on mobile does not have Reader Mode.
I should also add that the night mode won't be as fun on mobile either, just checked and I don't think it works with pointer events for the effect
The decision to design a serious (read: not-tongue-in-cheek) topic with these "quirky" tricks sucks, JMHO.
I can not imagine manually approving JS for every site. And again doing the inverse and have noscript installed to deny one website a year does not seem to be worth it for me. In this case I can also just use a adblock rule to block that specific script or all .js files from the domain I guess. So I really no not need NoScript.
For those sites that do need JS, NoScript's whitelist feature makes it quick & easy to fix. The first time I visit a new site, if it is obviously broken, then I whitelist the main domain. If that doesn't fix it, then I whitelist a couple likely-looking domains (often sites import JS from similar domains, or from common library domains). That's enough to get probably 90+% of websites working, while still leaving most garbage JS disabled. The remaining ~10% of websites that need a dozen domains whitelisted are probably not worth visiting anyway, so I just move on at that point. Or NoScript even lets you temp-whitelist everything for a given tab and just put up with the misery to get whatever I need from that one site. Since the whitelist persists forever, and I don't visit hundreds of different websites every day, after some time it becomes pretty rare that I need to whitelist more than one or two things per day.
You maintain an adblock blacklist, I maintain a NoScript whitelist. Not so different :)
As others have pointed out, reader mode works as expected.
2. I don't think it's a very good joke to post long-form content on your blog with the expectation that it's basically unreadable without a reader mode.
> The fact that people seem to take it seriously says something about the contemporary state of webdesign :)
Mind expanding and what it says exactly about contemporary web design?
Whether I take it seriously or not doesn't change the fact that it's still damn hard to read anything.
The same as when political satire is indistinguishable from actual politics. It means that the real things has sort of become a joke itself.
not even a proper flashlight. it updates when the mouse moves, so you're SOL if you scroll on desktop.
To each their own.
> It made me close the browser tab.
If you can't handle refreshing or merely clicking it again, that's you having a problem, not the site having a problem.
No, it's the site's problem. The contrast between the blinding radioactive yellow background and the font is eye straining and doesn't meet the WCAG standards for accessible text. And the dark mode is unusable. The joke would be funnier if there was a real dark mode or if the light mode was readable.
> doesn't meet the WCAG standards for accessible text.
Oh, which part of the standards? When I punch #000000 on #FDDB29 into contrast checkers I get good results.
The single quote isn't as good but all the rest has those colors.
1.1 million?
Why is Tengwar still not in Uniclde officially? What's the problem with it?
Copyright is not technically violated simply by encoding the characters into a plane such as one of Unicode's, that's an easy open and shut fair use, but Unicode principals have stated they don't want to pass on the copyright burden to font authors either, which would be sued if they tried to paint some of those characters. (Why encode something that fonts aren't allowed to produce?) That should also be fair use, but the law is complicated and copyright still so often today leans in favor of the Estates and major Corporations rather than fair use and the public commons.
(ETA: I'm hugely in favor that "conlang", constructed language, scripts such as these should be encoded by Unicode. I wish someday we fix the copyright problems of them.)
Most languages, when comparing strings, skip the normalization and just compare string bytes as is (or, if the string is interned, compare just the pointer)
Fortunately, it seems like it's marked as deprecated: https://en.wikipedia.org/wiki/Angstrom#Symbol
Frequently, the answer to this is "some obscure character set had this as a distinct symbol." In this case, blame the Japanese: https://en.wikipedia.org/wiki/JIS_X_0208
Which is why there's an 'mm' and 'cm' and other random symbols: https://www.compart.com/en/unicode/block/U+3300
Ideally - the "English-World" locale is supposedly meant for us, cosmopolitans. It's included with Windows 10 and 11.
Practically, as "English-World" was not available in the past (and still wasn't available on platforms other than Windows the last time I checked), I have always been setting the locale to En-US even though I have never been to America. This leads to a number of annoyances though. E.g. LibreOffice always creates new documents for the Letter paper format and I have to switch it to A4 manually every time. It's even worse on Linux where locales appear to be less easy to customize than in Windows. Windows always offered a handy configuration dialog to granularly tweak your locale choosing what measures system you prefer, whether your weeks begin on sundays or mondays and even define your preferred date-time format templates fully manually.
A less-spoken about problem is Windows' system-wide setting for the default legacy codepage. I happen to use single-language legacy (non-Unicode) apps made by people from a number of very different countries. Some apps (e.g. I can remeber the Intel UHD Windows driver config app) even use this setting (ignoring the system locale and system UI language) to detect your language and render their whole UI in it.
> English (USA), English (UK)
This deserves a separate discussion. I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects. To us presence of a huge number of these (don't forget en-AU, en-TT, en-ZW etc - there are more!) in the options lists brings only annoyance, especially when one chooses some non-US one and this opens another can of worms.
By the way I wonder how do string capitalization and comparision functions manage to work on computers of people who use both English and Turkish actively (Turkish locale distinguishes between dotted and undotted İ).
There's the English (Denmark) locale for that on some platfoms.
https://unix.stackexchange.com/questions/62316/why-is-there-...
It's worthwhile purely for the sake of autocorrect/typo highlighting in text-editing software. I don't miss the days of spelling a word correctly in my version of English but still being stuck with the visual noise of red highlighting up and down the document because it doesn't conform to US English.
It's sometimes not a trivial distinction either, for example I've heard of cases where surprised British redditors have found themselves banned from American subreddits for being homophobic when they were actually talking innocently enough about cigarettes!
> This deserves a separate discussion. I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects.
While that is generally (though not always) true, I would assume it's really a stand in for the much more relevant zh locales.
It is also rather relevant to es locales (america spanish has diverged quite a bit from europe spanish hence the creation of es-419), definitely french (canadian french, to a lesser extend belgian and swiss), and german (because swiss german). And it might be relevant for ko if north korea ever stops being what it is.
I mainly type in English but occasionally other languages - I use a combination of Mac & Linux - macOS has an (off-by-default but enable-able) lang-changer icon in the tray that is handy enough, but still annoying to have to toggle. Linux is much worse.
Mac also has quite a nice long-press-to-select-special character that at least makes for accessible (if not efficient) typing in multiple languages while using an English locale. Mobile keyboards pioneered this (& Android's current one even does simultanous multi-lang autocomplete, though it severely hurts accuracy).
---
> I doubt many English speakers care to distinguish between English dialects.
I think you'll find the opposite to be true. US English spellings & conventions are quite a departure from other dialects, so typing fluidly & naturally in any non-US dialect is going to net you a world of autocorrect pain in en_US. To the extent it renders many potentially essential spelling & grammar checkers completely unusable.
While in my default (English) layout, it is easy enough to add in accents other characters using the compose key (right alt). So right-alt+'+a = á or right-alt+"+u = ü. I much prefer this over the long press as I can do it quickly and seamlessly without having to wait on feedback. Granted, it is not as discoverable, but once you are comfortable, it in my opinion is a better system.
Not for me!
Right-alt+'+a = â
Right-alt+'+u = û
I don't find it "unusable", though. I could change it back to en_US, but it has actually been interesting to see all of my American spellings flagged by autocorrect. Each time I write authorize instead of authorise it is an act of stubborn group affinity!
As far as the written, formal language is concerned, English really has only three dialects: US American, Canadian, and everywhere else. There are some other subtle differences (such as "robots" for traffic lights in South Africa, or "minerals" for fizzy drinks in Ireland¹), but that's pretty much it.
¹ Yes, this isn't just slang in Ireland: the formal, pre-recorded announcements on trains use it: "A trolley service will operate to your seat, serving tea, coffee, minerals and snacks." The corresponding Irish announcement renders it mianraí. Food service on trains stopped during covid and has not yet resumed, so I'm working from distant memory now.
This is true, but I don't see why the "formal" qualifier is needed here :) There are much more than 3 dialects of English, both written & spoken.
Especially there's a fair few extremely common notable differences in (casual, written) Irish English: the word "amn't" (among other less common contractions), the alternative present tense of the verb "to be" (i.e. "do be"), various regional plurals of "you", and - perhaps the most common - prepositional pronouns, etc. etc.
There's plenty of regional variants just within the US, but "en_us" covers the whole country.
I guess the simple answer to that is: how much interest is there in maintenance. I don't think there's any compelling reason not to create something: if there's insufficient interest in maintenance that's an imperfect but reasonable proxy for utility.
I'm not aware of any maintained en_US_Xyz languages but it might be pretty cool if someone started. There's precedence in a few other languages, like no_NO_NY, zh_Hans_HK, etc.
> I doubt many English speakers (let alone those who don't live in a particular anglophone country) care to distinguish between English dialects. To us presence of a huge number of these (don't forget en-AU, en-TT, en-ZW etc - there are more!) in the options lists brings only annoyance, especially when one chooses some non-US one and this opens another can of worms.
Well, you just explained what this plethora of options is about. It's not just about how you spell flavor/flavour. It's a lot of different defaults for how you expect your OS to present information to you. Default paper size, but also how to write date and time, does the week start on Monday, Sunday, or something else, etc.
I think you'd be surprised how many english (UK) people will get pissed off when their spell-checker starts removing the "u" from colour or flavour, or how many English (US) people get pissed off when the spellchecker starts suggesting random "u"s to words.
additionally to that, locale isn't just about language. English (US) and English (UK) decides whether your dates get formatted DD-MM-YY or MM-DD-YY, whether your numbers have the thousands broken by commas or spaces, and a host of other localization considerations with a lot more significance than just the dialect of english.
Perhaps you're an old-timer? I worked in the city in the early 80s; lunch in the pub was routine, and sometimes required. By the end of the 80s, that was at best frowned on. Over the last 20 years, having alcohol on your breath after lunch would have been a disciplinary issue, unless you were entertaining a client, at least in the places I worked.
ı I
I symphatize with people who get this wrong. (I just saw some YouTube video have a title TÜRKIYE in a segment)
Even google keyboard can't seem to distinguish between I and İ. When I type "It", it suggests "İt's" which is quite pathetic.
I can understand that the conflation of spelling, word choices, time and date formatting, default paper sizes, measurement units, etc, etc, is convenient, and works a lot of the time, but it really doesn't work for me at all.
That said, I appreciate that I occupy a very small niche.
Most people in the UK care - a population nearly twice that of California, and larger than the native speakers of any non-top-20 language. If you care enough to support e.g. Italian you should support en_UK.
I definitely do. The biggest difference, as everyone else, has pointed out is the US vs UK spellings.
Realistically, though, beyond that country is a poor indicator for everything else. I want to use DD/MM/YYYY date format in English, but DD.MM.YYYY date format in German. I want to use $1,000 in English, but 1.000 $ in German. This isn't dependent on the country I live in, this is dependant on a combination of a country and language - that could be the country I'm living in, or the country I grew up in (mostly US date format vs not), and it's either the language I'm actively typing in, or the language of the document I'm reading, or the language I'm thinking in (but a computer can't exactly handle that).
Trying to guess the correct combination is tricky, especially if a document is in two languages (e.g., a quotation), and users are lazy and won't switch their IME unless they have to.
What this means in slightly more practical terms is that setting a single "locale" for my device doesn't make sense, but rather I should be able to choose a locale per language (or possibly spelling preferences by language and formatting options by language as separate choices). I'd then pick a language to use the device in, and it would use that languages locale, and tell apps that this language is the preferred language. If an app doesn't provide my preferred language, pull the preferred locale from settings for a language it does support, otherwise use the default set by developers. For some apps, it's a bit more complex, particularly if I'm creating content. GMail or Office would be two good examples, where the UI language might be in English, but the emails or documents are in German, or a combination of German and English.
Even then, I'm sure there are people who need something even more flexible than that.
At the moment, if I set my language to English but my Country to Germany on my iPhone, for example, things occasionally get confused. My UK banking app, for example, pulled the decimal separator from my locale settings for a while and then refused to work because "£9,79" (or whatever it was) isn't a valid amount of money, and I couldn't see a way to fix that without switching my Country in the phone settings. I imagine they fixed it by ignoring my configured locale and always using en-GB, thus defeating the whole point of a locale in the first place.
So yeah, these days it's fairly common to not have a single "locale" that you work in - it's quite possible to want to use two or more but nothing is really set up to handle that well.
The problems with Unicode are mostly to do with internal inconsistencies and churn, problems that usually only affect programmers.
1. Different ways to encode the same visually indistinguishable set of characters as code points leading to normal forms, text that compares unequal even when it appears to be identical, the disastrous "grapheme clusters" concept and so on.
2. Many different ways to encode the same sequence of code points as bytes. Not only UTF-32/16/8 but also curiousities like "modified UTF-8".
3. Emoji. A fractal of disasters:
3.a. Updates frequently. Neither Unicode nor software in general was built on the assumption that something as basic as the alphabet changes every year. If you send someone an emoji, can their device draw it? Who knows! In practice this means messaging apps can't rely on the OS system fonts or text handling libraries anymore which is a drastic regression in basic functionality.
3.b. (Ab)uses composition so much it's practically a small programming language, e.g. flags are composed of the two letter country code spelled using special characters. People are represented as as generic person plus skin color patch, families are represented using composed individual people etc.
3.c. Meaning of a character is theoretically specified but can subtly depend on the font used, e.g. people use a fruit emoji in visual puns because of how it looks specifically on Apple devices, so a "sentence" can make no sense if it's rendered with a different font.
3.d. Unbounded in scope. There's no reason the Unicode committee won't just keep adding new pictograms forever.
3.e. Encoded beyond the BMP which in theory every correct program should handle but in practice some don't because nobody except a few academics used characters beyond it much until emoji came along.
3.f. Disagreement over single vs double width chars, can only know this via hard-coded tables, matters for terminals and code editors.
Some of these can potentially be cleaned up outside of the Unicode consortium in backwards compatible ways. You could have a programming language that automatically normalized strings to fully composed form when deserializing from bytes, and then automatically folded semantically identical code points together (this would be a small efficiency win for some languages too). You could campaign to build a consensus around a specific normal form, like how UTF-8 gained consensus as a transfer encoding. You could also define a fork of Unicode (using private use areas?) that allocates a single code point to the characters that are unnecessarily using composition today but don't yet have one and then just subset out the concept of composition entirely.
Emoji are a big problem. It's tempting to say that these should not be encoded as characters at all. Instead there could be a set of code points that define bounds that contain a tiny binary subset of SVG, enough to recreate the Apple pixel art somewhat closely. Emoji would always be transmitted as inlined vector art. Text rendering libraries would call out to a little renderer for each encoded glyph, using a fast fingerprinting algorithm to deduplicate the bytes to an internal notion of a character. To avoid wire bloat, text can simply be compressed with a pre-agreed zstd or Brotli dictionary that contains whatever images happen to be popular in the wild. At a stroke this would avoid backwards compat problems with new emoji, enabling programs working with text to be upgraded once and then never again, eliminate all the ridiculous political committee bike-shedding over what gets added, let apps go back to using system text support and get rid of the bajillion edge cases that emoji have spewed all over the infrastructure.
I've written unicode-aware software for over a decade, doing a wide variety of programs, and I've never had to bother with all that mess.
If I'm parsing strings I'm looking for stuff in the 7-bit ASCII range which maps neatly onto the Unicode representations, and so I just need to take care to preserve the rest.
The only trouble I've had is that a lot of programmers haven't learned, or don't get, that text encoding is a thing and that it needs to be handled.
So they'll hand me an XML they claim is UTF-8 encoded, except that XML header was just copypasta and the actual XML document is encoded in some other system encoding like Windows-1252. Or worse, a mix of both.
For site local, fec0::3. Yeah site-local is discouraged but you can still do it. Or you can slightly misuse fd00::3.
You only get those latter 16 hex characters if you explicitly don't want to choose addresses.
"Exponential explosion" is really putting it too strong; it's perfectly possible to just add ǿ and á̤ and a bunch of other things. The combinations aren't infinite here.
The problem with e.g. Latin script isn't necessarily that combining characters exist, but that there's two ways to represent many things. That really is just a "mess": use either one system or the other, but not both. Hangul has similar problems.
Devanagari doesn't have any pre-compose characters AFAIK, so that's fine.
That's really the "mess": it's a hodgepodge of different systems, and you can't even know which system to use a lot of the time because it's not organised ("look it up in a large database"), and even taking in to account historical legacy I don't think it really needed to be like this (or is even an unfixable problem today, strictly speaking).
At least they deprecated ligatures like st and fl, although recently I did see ij being used in the wild.
They certainly are. Languages are a creative space driven by the human imagination. Give people enough time and they'll build new combinations for fun or for profit or for research or for trying to capture a spoken word/tone poem in just the right sort of exciting way. You may frown on "Zalgo text" [1] (and it is terrible for accessibility), but it speaks to a creative mood or three.
The growing combinatorial explosion in Unicode's emoji space isn't an accident or something unique to emoji, but a characteristic that emoji are just as much a creative language as everything else Unicode encodes. The biggest difference is that it is a living language with a lot of visible creative work happening in contemporary writing as opposed to a language some monks centuries ago decided was "good enough" and school teachers long ago locked some of the creative tools in the figurative closets to keep their curriculum simpler and their days with fewer headaches.
[1] https://en.wikipedia.org/wiki/Zalgo_text
We've got 150K assigned codepoints assigned, leaving us with 950K unassigned codepoints. There's truly massive amounts of headroom.
To be honest I think this argument is rather too abstract to be of any real use: if it's a theoretical problem that will never occur in reality then all I can say is: <shrug-emoji>.
But like I said: I'm not "against" combining marks, purely in principle it's probably better, I'm mostly against two systems co-existing. In reality it's too late to change the world to decomposed (for Latin, Cyrillic, some others) because most text already is pre-composed, so we should go full-in on pre-composed for those. With our 950k unassigned codepoints we've got space for literally thousands of years to come.
Also this is a problem that's inherent in computers: on paper you can write anything, but computers necessarily restrict that creativity. If I want to propose something like a "%" mark on top of the "e" to indicate, I don't know, something, then I can't do that regardless of whether combining characters are used, never mind entirely new characters or marks. Unicode won't add it until it sees usage, so this gives us a bit of a catch-22 with the only option being mucking about with special fonts that use private-use (hoping it won't conflict with something else).
Unicode can't get rid of the many precombined characters for a huge number of backward compatibility reasons (including compatibility with ancient Mainframe encodings such as EBCDIC which existed before computer fonts had ligature support), but they've certainly done what they can to suggest the "normal" forms in this decade should "prefer" the decomposed combinations.
> If I want to propose something like a "%" mark on top of the "e" to indicate, I don't know, something, then I can't do that regardless of whether combining characters are used
This is where emoji as a living language actually shines a living example: It's certainly possible to encode your mark today as a ZWJ sequence, say «e ZWJ %», though you might want to consider for further disambiguation/intent-marking adding a non-emoji variation selector such as Variation Selector 1 (U+FE00) to mark it as "Basic Latin"-like or "Mathematical Symbol"-like. You can probably get away with prototyping that in a font stack of your choosing using simple ligature tools (no need for private-use encodings). A ZWJ sequence like that in theory doesn't even "need" to ever be standardized in Unicode if you are okay with the visual fallback to something like "e%" in fonts following Unicode standard fallback (and maybe a lot of applications confused by the non-recommended grapheme cluster). That said, because of emoji the process for filing new proposals for "Recommended ZWJ Sequences" is among the simplest Unicode proposals you can make. It's not entirely as Catch-22 on "needs to have seen enough usage in written documents" as some of the other encoding proposals.
Of course, all of that is theory and practice is always weirder and harder than theory. Unicode encoding truly living languages like emoji is a blessing and it does enable language "creativity" that was missing for a couple of decades in Unicode processes and thinking.
Yes, and that only makes things worse since the overwhelming majority of documents (99.something% last time I checked) uses pre-composed. Also AFAIK just about everyone just ignores that recommendation.
This is a classic "reality should adjust to the standard" type of thinking. Previous comments about that: https://news.ycombinator.com/item?id=36984331
I suppose "e ZWJ %" is a bit better than Private Use as it will appear as "e%" if you don't have font support, but the fundamental problem of "won't work unless you spend effort" remains. For a specific niche (math, language study, something else) that's okay, but for "casual" usage: not so much. "Ship font with the document" like PDF and webfonts do is an option, but also has downsides and won't work in a lot of contexts, and still requires extra effort from the author.
I'm not saying it's completely impossible, but certainly harder than it used to be, arguably much harder. I could coin a new word right here and now (although my imagination is failing me to provide a humorous example at this moment) and if people like it, it will see usage. In 1960s HN when we would have exchanged these things over written letters, and it would have been trivial to propose a "e with % on top" too, but now we need to resort to clunky phrases like this (even for typewriters you can manually amend things, if you really wanted to).
Or let me put it this way: something like ‽ would see very little chance of being added to Unicode if it was coined today. Granted, it doesn't see that much use, but I do encounter it in the wild on occasion and some people like it (I personally don't actually, but I don't want to prevent other people from using it).
None of this is Unicode's fault by the way, or at least not directly – this is a generic limitation of computers.
It shouldn't matter what's in the wild in documents. That's why we have normalization algorithms and normalization forms. Unicode was built for the ugly reality of backwards compatibility and that you can't control how people in the past wrote. These precomposed characters largely predate Unicode and were a problem before Unicode. Unicode won in part because it met other encodings where they were rather than where they wished they would be. It made sure that mappings from older encodings could be (mostly) one-to-one with respect to code points in the original. It didn't quite achieve that in some cases, but it did for, say, all of EBCDIC.
Unicode was never in the position to fix the past, they had to live with that.
> This is a classic "reality should adjust to the standard" type of thinking.
Not really. The Unicode standard suggests the normal/canonical forms and very well documented algorithms (including directly in source code in the Unicode committee-maintained/approved ICU libraries) to take everything seen in the wilds of reality and convert them to a normal form. It's not asking reality to adjust to the standard, it is asking developers to adjust to the algorithms for cleanly dealing with the ugly reality.
> Or let me put it this way: something like ‽ would see very little chance of being added to Unicode if it was coined today.
Posted to HN several times has been the well documented proposal process from start to finish (it succeeded) of getting common and somewhat less common power symbols encoded in Unicode. It's a committee process. It certainly takes committee time. But it isn't "impossible" to navigate and is certainly higher than "little chance" if you've got the gumption to document what you want to see encoded and push the proposal through the committee process.
Certainly the Unicode committee picked up a reputation for being hard to work with in the early oughts when the consortium was still fighting the internal battles over UCS-2 being "good enough" and had concerns about opening the "Astral Plane". Now that the astral plane is open and UTF-16 exists, the committee's attitude is considered to be much better, even if its reputation hasn't yet shifted from those bad old days.
> None of this is Unicode's fault by the way, or at least not directly – this is a generic limitation of computers.
Computers do anything we program them to do and in general people find a way regardless of the restrictions and creative limitations that get programmed. I've seen MS Paint drawn symbols embedded in Word documents because the author couldn't find the symbol they needed or it didn't quite exist. It's hard to use such creative problem solving in HN's text boxes, but that from some viewpoints is just as much a creative deficiency in HN's design. It's not an "inherent" problem to computers. When it is a problem they pay us software developers to fix it. (If we need to fix it by writing a proposal to a standards committee such as the Unicode Consortium, that is in our power and one of our rights as developers. Standards don't just bind in one-direction, they also form an agreement of cooperation in the other.)
This comes up in specifications that have a broad range of use cases; when I was involved in this my idea was to just spec things so that there's only one allowed form; you'll still need a small-ish table for this, but that's fine. But that's currently hard because for a few newer Latin-adjacent alphabets some letters cannot be represented without a combining character.
So then you have either the "accept that two things which seem visually similar are not identical" (meh) or "exclude embedded use cases" (meh).
I never really found a good way to unify these use cases. I've seen this come up a few times in various contexts over the years.
> Posted to HN several times has been the well documented proposal process from start to finish (it succeeded) of getting common and somewhat less common power symbols encoded in Unicode.
Would this work for an entirely new symbol I invent today? It's not really the Unicode people that are "difficult" here as such, they just ask for demonstrated usage, which is entirely reasonable, and that's hard to get (or: harder than it was before computers) especially for casual usage. I'm sure that if some country adopts/invents a new script today, as seems to be happening in West-Africa at in recent years, the Unicode people are more than amendable to work with that, but "I just like ‽" is a rather different type of thing.
Sure, they want demonstrated usage as inline in the flow of text as textual elements as opposed to purely iconography or design elements (because such things are outside of Unicode's remit, modulo some old Wingdings encoded for compatibility reasons and the fine line between emoji are expressive text and also emoji are useful for iconography in many cases). But at this point (again in contrast to the UCS-2/no-Astral-plane days) the committees don't seem to care how it was mocked up (do it on a chalkboard, do it in paint, do it in LaTeX drawing commands, whatever gets the point across) or how "casual" or infrequent the usage is, so long as you can state the case for "this is a text element" (not an icon!) used in living creative language expression. There's more "provenance" requirements for dead languages and they'll want some number of academic citations, but for living languages they've come to be flexible (no hard requirements) on the number of examples they need from the wild and where those are sourced from. Showing it in old classic documents/manuals/books, for instance, helps the case greatly, but the committees today no longer seem as limited to just what can be used to demonstrate usage. "I just like it" is obviously not a rock solid proposal/defense to bring to a committee (any committee, really), but that doesn't mean that is impossible for the committee to be swayed by someone making a strong enough "I just like it" case if they demonstrate well enough why they like it and how they use it and how they think other people will use it (and how those uses aren't just iconography/decorative elements but useful in the inline context of textual language).
Even "put the code points forming the composed character in descending numerical order" would be better than nothing. If it was there from the start.
However, the Unicode commitee is too busy adding new emojis to make their standard sane.
But, of course, unicode can't define that the standard will cover only the canonical forms, and couldn't do that since the start, as it needed backwards compatibility with various pre-unicode encodings which had mutually incompatible principles, so it needed support for both composed and decomposed versions of the same characters.
There' your problem right there. Plural formS. It's not canonical if there are more of it.
What on earth are you doing that it's leading to crashes? Are you not validating the result?
https://www.reddit.com/r/apple/comments/37e8c1/malicious_tex...
In fact, I don't know that there's any reason to believe normalization happens at all in the process of executing this.
Not just emojis, in general I believe Unicode has just said they're not going to add new pre-composed characters and that using combining characters is the Right Way™ to do things (well, the only way for newer scripts).
One of the downsides of writing down specifications is that they tend to attract people with Very Strong Opinions on the One And Only Right Way and will argue it to no end, and essentially "win" the argument just by sheer verbosity and persistence.
That's certainly what I've seen happen in a few cases, and is what happens on e.g. Wikipedia as well at times.
But yeah, emojis is even worse. Something things can look rather different depending on which invisible variation selector is present. We've got tons and tons of unassigned codepoints and we need to resort to these tricks to save a few of them?
Firefighter is "(man|woman|person) + ZWJ + firetruck". Clever, I guess. Construction worker is "Construction worker (+ ZWJ + (male sign|female sign))?" (absence is gender-neutral). Why are there 2 systems to encode this? Sigh...
All of this is too clever by a mile.
[1]: HN will strip stuff, but try something like:
May not display correctly in terminal, but can xclip it to a browser – screenshot: https://imgur.com/a/iFmBDQkBut the technical implementation? Yeah, that could have gone a lot better IMHO.
One must also wonder if some things really had to be added in the first place, e.g. for people kissing it's:
This is NOT a complaint about that they added diversity as such, in principle I'm all for that, it's just that few seem to actually use these emojis, and both in terms of code and UI it all gets pretty complex; there's 98 combinations to choose from here.I don't really get why <heart> or <killing lips> or <kissing face> isn't enough. That's actually what most people seem to use anyway, because who finds it convenient to pick all the correct genders and skin tones from the UI for both people?
Oh. So that's why HN discussion always looks sane. They strip the pollution.
Less than that since a default skin color can be set in most apps. I'm sure setting a gender will come soon so the entire first part of that emoji can be auto-guessed. Then its just showing the other options in the UI. Really all of this is UI design as even with the 98 combinations you can still display it as 4/5 options you drill down.
> who finds it convenient to pick all the correct genders and skin tones from the UI for both people?
I just checked and searching "kissing" in my iOS emoji keyboard inside Messenger showed just 4 of the emoji's your describing - defaulting both skin tones to my settings and then the four M/F pair ups. Plus some non-related kissing emojis like the cat kissing.
But that's kind of wrong, no? The entire point is that you can choose both sides individually. What if you set it to black and want to kiss some white bloke?
If anything that only underscores my point that it's too complex and that no one is using them (certainly not as intended anyway).
In the Windows 11 emoji picker it works like this:
1. Search "kissing". See two generic yellow people kissing. Notice a blue dot in the bottom right corner.
2. Clicking the emoji brings up previously used versions of the kissing emoji, with a + button.
3. Clicking + brings up a dialog like I described previously. Two generic figures at the top, then a row of skin tones.
4. You can click on each generic person and choose a gender, then select a skin tone. You can do this for each person in the group.
5. Click done. This emoji is now in your default emoji list and you won't need to recreate it again.
Also, the % should measure people, not languages, that would greatly decrease the imaginary 99%
Well, there is a new fact that I learned and immediately hated.
The fuck were authors thinking...
I am now firmly convinced people developing unicode hate developers. I suspected it before just due to how messy it was (same character having different encodings ? Really ? Fuck you), but this cements it.
But making it clear which glyph to use is also a key feature!
Or at least they have a vicious indifference to us. Unicode is a nightmare.
That coloration tools shows the moving mouse coursers of other participants even if they aren't needed/wanted is already pretty bad, why bring it to a website?
through it also wasn't meant to be constructive critique
Why would a frontend developer demonstrate their ability to do frontend programming on their personal, not altogether super-serious blog? I meant that rhetorically but it's a flex. I agree, not the best design in the world if you're catering for particular needs, but simple and fun enough. You should check out dark mode.
In that vein, I think it's okay if we let people have fun. That might not work for everyone, but why should we let perfect be the worst enemy of fun?
because it shows that they don't understand important design aspects
while it doesn't really show off their technical skills because it could be some plugin or copy pasted code, only someone who looks at the code would know better. But if someone care enough about you to look at your code you don't need to show of that skill on you normal web-site and can have some separate tech demo.
> okay if we let people have fun
yes people having fun is always fine especially if you don't care if anyone ever reads your blog or looks at it for whatever reason (e.g. hiring)
but the moment you want people to look at it for whatever reason then there is tension
i.e. people don't get hired to have fun
and if you want others to read you blog you probably shouldn't assault them with constant distractions
Living by that motto is hugely self-destructive.
Creative expression allows us to push ourselves, both in what we think we can do, and often the technical aspects about how we do it too. Even if the idea doesn't stick, you've tried something new.
In a world of Tailwinds and Bootstraps and the same five templates copied again and again and again, let's celebrate the people willing to push things and learn from their inevitable but ultimately valuable mistakes. And let's have some fun along the way.
It's revenge against anyone with certain kinds of visual impairments and/or concentration issues because the ex-spouse of the author which turned out to be a terrible person had such.
(sarcasm try 2)
It's revenge against anyone using JS on the net with the author trying to subtle hint that JS is bad.
(realistic)
It's probably on of:
- the website is a static view of some collaborative tool which has that functionality build in by default
- some form of well intended but not well working functionality add to the site as it was some form of school/study project, in that case I'm worried about the author suffering unexpected very much higher cost due to it ending up on HN ...
1. Fun. Modern internet is boring, most blog posts are just black text on white background. Hard to remember where you read what. And you can’t really have fun without breaking some expectations.
2. Sense of community. Internet is a lonely place, and I don’t necessarily like that. I like the feeling of “right now, someone else reading the same thing as I do”. It’s human presence transferred over the network.
I understand not everybody might like it. Some people just like when things are “normal” and everything is the same. Others might not like feeling of human presence. For those, I’m not hiding my content, reader mode is one click away, I make sure it works very well.
As for “unexpectedly ended up on HN”, it’s not at all unexpected. Practically every one of my general topic articles ends up here. It’s so predictable I rely on HN to be my comment section.
Then the second thought was: I should again start to block js by default as much as I can.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
The cursors will only be a problem during front page HN traffic. And the opt-out for people who care is reader mode / disable js / static mirror. Not sure if there's any better way to appease the fun-havers and the plain content preferrers at the same time. Maybe a "hide cursors" button on screen? I, for one, had a delightful moment poking other cursors.
Meanwhile, ironic irony is ironic: "Hey, idiots! Learn to use Unicode already! Usability and stuff! Oh, btw, here is some extremely annoying Javascript pollution on your screen because we are all still children, right? Har har! Pranks are so kewl!!!1!"
I disagree.
What is the "right" things is use-case dependent.
For UI it's glyph bases, kinda, more precise some good enough abstraction over render width. For which glyphs are not always good enough but also the best you can get without adding a ton of complexity.
But for pretty much every other use-case you want storage byte size.
I mean in the UI you care about the length of a string because there is limited width to render a strings.
But everywhere else you care about it because of (memory) resource limitations and costs in various ways. Weather that is for bandwidth cost, storage cost, number of network packages, efficient index-ability, etc. etc. In rare cases being able to type it, but then it's often us-ascii only, too.
defaults matter
as in they should things you can just use by-default without thinking about it
as swift is deeply rooted in UI design having a default of glyphs make sense
and as rust is deeply rooted in unix server and system programming utf-8 bytes make a lot of sense
through the moment your language becomes more general purpose you could argue having a default in any way is wrong and it should have multiple more explicit methods.
That time has passed. If you want to know the length of a string, you really should indicate what length type you mean.
It has distinct .chars .codes and .bytes that you can specify depending on the use case. And if you try to use .length is complains asking you to use one of the other options to clarify your intent.
I was working on an app in Swift that needed full emoji support once. Team ended up writing our own string lib that stores things as an array of single-character Swift strings.
This was true while Swift was developing but it's been stable now for several years. At some point that complaint is no longer valid.
Rust was given as one of the examples and Rust's .len() behaviour is chosen based on three very reasonable concerns:
1. They want the String type to be available to embedded use-cases, where it's not reasonable to require the embedding of the quite large unicode tables needed to identify grapheme boundaries. (String is defined in the `alloc` module, which you can use in addition to `core` if your target has a heap allocator. It's just re-exported via `std`.)
2. They have a policy of not baking stuff that is defined by politics/fiat (eg. unicode codepoint assignments) into stuff that requires a compiler update to change. (Which is also why the standard library has no timezone handling.)
3. People need a convenient way to know how much memory/disk space to allocate to store a string verbatim. (Rust's `String` is just a newtype wrapper around `Vec<u8>` with restricted construction and added helper functions.)
That's why .len() counts bytes in Rust.
Just like with timezone definitions, Rust has a de facto standard place to find a grapheme-wise iterator... the unicode-segmentation crate.
I wish that was the only thing wrong with that page
But there is not a code for each flag. Instead there is a code for each ASCII character. A flag sequence is formed from U+1F3F4 (Black Flag), followed by at least two tags that form a country/region code, and then U+E007F (End tag).
So, yes this is weird, because the emoji is dependent on the decoder. It was made this way to keep Unicode independent of geopolitics.
Read more: <https://en.wikipedia.org/wiki/Tags_(Unicode_block)>
Skipping over UTF-32-BE and UTF-32-LE there...
(I mean, it might not be an issue if it's just being used as an internal representation, but still)
Found this! good to know about. https://pypi.org/project/grapheme/ "A Python package for working with user perceived characters. "
(apparently the article talks about this however the blog post is largely unreadable due to dozens of animated arrow pointers jumping all over the screen)
> Why does the fi ligature even have its own code point? No idea.
On of the principles of Unicode is round trip compatibility. That is you should be able to read in a file encoded with some obsolete coding system and write it out again properly. Maybe frob it a bit with your unicode-based tools first. This is a good principle, though less useful today.
So the fi ligature was in a legacy encoding system and thus must be in Unicode. That's also why things like digits with a circle around them exist: they were in some old Japanese character set. Nowadays we might compose them with some zwj or even just leave them to some higher level formatting (my preference).
Edit ①: seems the answer is not with the default iOS keyboard, but possible to paste it and perhaps possible with a third party keyboard that I'm not keen on trying (unless I hear of a keyboard that's both genuinely useful / better than default, and that doesn't send keystrokes to the developer - though I can't remember if the latter is even a risk on iOS, better go search about that next..)
Today, it’s buggy $5 abandonware that only makes me sad when I am reminded of it.
EDIT: Here[1] is a blog post that claims it's still the best keyboard in 2023. I actually might give it another shot... Not holding my breath though.
*EDIT: Looks like another dedicated fan has actually taken it upon themself to revive the project, under the new name Keyboard71[2].
[0] https://apps.apple.com/us/app/nintype/id796959534
[1] https://maxleiter.com/blog/nintype
[2] https://www.reddit.com/r/keyboard71/
Really not looking to burn another $5, I’d greatly appreciate any thoughts/concerns at all.
https://unichar.app
We should be careful: someone on HN could write a proposal that they should be considered pre composed forms that should also have an un-composed sequence… so there could in future be not just 1 in a circle but 1 ZWJ circle, circle ZWJ 1 all considered the same…I can imagine some HN readers being pranksters like that.
This implies that they're obsolete, but they're not -- they're still in very common use today. You can type them in Japanese by typing まる (maru, circle) and the number, then pick it out of the IME menu. Some IMEs will bring them up if you just type the number and go to the menu, too. :)
① Draw some circles
② Draw the rest of the owl
Commentary: ① is simple but ② is masking many complex steps that are necessary to draw an owl.
Most of the pre-composed latin ligatures are generally from EBCDIC codepages. People in the ancient Mainframe era wanted nice typesetting too, but computer fonts with ligature support were a much later invention.
You can see fi and several others directly in EBCDIC code page 361:
https://en.wikibooks.org/wiki/Character_Encodings/Code_Table...
Actually in German (at least) ä, ö and ü really are actually ligatures for ae, oe, and ue -- the scribes started to write the E's on their sides above the base letters, and over time the superscript "E"s became dots or dashes. Often they are described the other way around: "you can type oe if you can't type ö." That's what my kid was told in school!
But Ö and ß aren't really part of the alphabet in German, while, say, in Swedish, ä and ö became actual letters of the alphabet. English got W that way too.
[1] https://de.wikipedia.org/wiki/Deutsches_Alphabet
By contrast, my French dictionary has énorme nestled between enorgueillir and enquiérir. (Looking for an example does underscore some of the patterns in the language: page after page of ét~ with only a few et~ and one êt~ among them; pages of ex~ with no éx~ at all.)
Also, though it's hardly authoritative, my kids' school taught English through immersion from grade 1 too, and both German and English teachers said "same alphabet".
As bmicraft pointed out, even in that wikipedia chart those inflected letters are spaced apart from the others. Yes, they are letterforms, but not part of the "alphabet" -- they don't even have a sorting like the Swedish Ä or W do.
And you can switch in running text from using the marker for umlaut (dots or bar, not semantically dieresis) or a normal "e" without anyone blinking. There's no problem reading a Swiss book even though ß refuses to cross the border. Though I personally prefer to read Äpfel and Bär rather than Aepfel and Baer, really, they are the same.
On a somehow related side note, I read that "&", which is derived from a ligature of the Latin conjunction "Et" (meaning "and"), was named "ampersand" in English as a mondegreen for "and per se and" as it used to be placed at the end of the English alphabet recitation.
They still ꜩ on some German street signs. I can't find ch in Unicode though (could just be my old eyes).
I have my music teacher's German schoolbook from around 1915 and it lists them all as letters (the whole book is in Fraktur). I have various old books in Fraktur and once could read them. I imagine that if I sat down and tried to read one it would come back, but at the moment I have to thin a little to read just the titles!
Replace "digits with a circle around them" with "emojis" and that's also true.