for i := 0; i <= x.tctl_b(); i++ {
v, e := x.getg[i].pop()
if e != nil {
log.Fatalf("%v\nloop", e)
}
}
So is this
imul 8(%rsp), %rax
So is this
shopt -s globstar && grep -sn '\xFF' **
Whether we translate the meaning of the above into Mandarin, English, Spanish, Japanese, Swedish, Farsi, Swhaili, Hindi, Latin, Klingon, Dothraki, Sindarin or any other spoken, written or invented language we chose to be comfortable with, makes exactly zero difference.
I mean use English as a universal language for users interacting with computers. It is a good way to force people to learn and use English. Like how American materiel with English documentation did so before.
Sadly the language is part of thought process and English isn't really universal enough when it comes to representing every mode of thoughts. Whatever Chomsky et al argued is just a language equivalent of Turing completeness, like technically Scheme and Swift and Visual BASIC all compile into x86 and are Turing complete but you can't make a simple conversion tool to go between them.
Emoji was originally added because of Unicode's mission. It wasn't some rando who thought, "we should put a smiley face emoji". It was because Japanese phones used JIS encodings that had them in it.
Unicode, and its encoding in utf-8 are fine. Being able to parse every character of every written language, including symbols, using the same logic, was one of the best things that happened to computing.
I've noticed that a large chunk of the people who reject caring about Unicode support are those who never have to deal with the consequences of forcing ASCII and English/Western ideals on everyone.
That's an incredibly English (speaking) centric view of machine design. Why should our 26 letter alphabet with only two cases be the restrictive set of characters that everybody in the world has to write terminal commands and filenames and code in? Are you saying that all code should be written in English? Even without bringing up languages with different writing systems, 7 bit ASCII doesn't allow you to correctly write words in many western languages.
Hmm fair point. I think maybe stuff like LaTeX is better for domain-specific though. Still we could have a much simpler unicode with a max 2-byte character and characters with consistent width, no joining/merging/shaping, eliminate the need for harfbuzz if we just refuse to support non-English human languages on computers.
That’s what we had before Unicode, and it was hell. Every country had their own opinions on what to put in the blank spaces in the codepoint map. As for a max 2-byte encoding, UCS-2 tried that. Turns out 65k points isn’t enough (even ignoring emoji), and now we have the hack known as UTF-16.
Unicode currently encompasses over 144 thousand codepoints. How many are emoji? Only ~3.5k. So even if we removed emoji, 16 bits still wouldn’t be nearly enough.
Instead of saying we should go back to the past, one should first see why we don’t do the things we did in that past
> English is the language of the world and we should use computers to push everyone to learn
This (and everything you wrote after) is incredibly bigoted. English is only just the most spoken language in the world. 1.35bn people have English as a 1st or 2nd language whereas 1.12bn people have Mandarin Chinese[1]. And even if it is the most popular language (by a narrow margin) why should that mean that everyone else has to speak it?
Let's imagine a world where the Chinese government got everyone (Cantonese etc.) retrained to speak Mandarin over the next 10 years so that it overtook English. Would you then change your view to think that everything should be written in Mandarin? No, you just want everyone else to adopt your own self centred approach so that you don't have to take into account anyone else's preferences.
Because it's an effective tool to maintain American cultural hegemony and I want to do that. It's an effective way to spread American values and further American interests and computers are a good way to do it. I reject using chinese as a global language because it furthers chinese influence.
It's an effective way to get every country to make their own text encoding, thereby returning us all to the dark ages of computing where you actually had to care about this stuff instead of just using unicode for everything.
I mean seriously, text encodings as a propaganda tool?
Because it's easier to visualize / talk about than some weird (to us) Hangul script but the problem is exactly the same: english isn't the only language on earth.
I would prefer to see issues on real text if they even exist. I don't care about some missing combining support for emojis esp for probably a quickly moving target.
Yes, I don't understand why non 'text' characters such as flags and emoji were allowed in Unicode. They appear to me to be a fundamentally different concept.
That said, I have always assumed that these problems we get with emoji also have equivalents that show up in other languages, it's just that we speak English and so only see them with Emoji. Maybe it's good that Emoji were included because it actually stresses our Unicode implementations!
And this is all before you introduce text that is written right-to-left like arabic scripts.
> Yes, I don't understand why non 'text' characters such as flags and emoji were allowed in Unicode.
Just my 2 cents, but if ASCII includes, among other things, "characters" for linefeed, 4 unspecified "device control characters", EOT (End of Transmission), and a character to ring a bell, then I guess its okay to put some smileys and flags into Unicode.
RTL is a nightmare for those of us who work with it. The problem is not to work with RTL. I enjoy that.
The problem is the massive quantity of LTR stuff that is in RTL languages still (English words, programming language keywords, math, music, certain conventions, ...)
Anyway yes multi-character glyphs occur in various forms (like accents, sometimes: é, ñ...) but I always thought that the burden of supporting them was on the fonts, not on the Terminal or the Unicode standard itself.
Coincidentally, this would be my response to the root comment as well. If anything more complicated than a box doesn't belong in a Terminal, how do we expect speakers of non-western languages to use the Terminal?
I specififcally wrote "my terminal", not "a Terminal".
I am well aware, and supportive, of the need to encode the entirety of human script and written symbolics in a unified format.
And the reason why my terminal doesn't need anything beyond basic modifying diacritic marks and single-codepoint symbols (like blocks and line characters for text-user-interfaces) is because I do not use it to write prose is a variety of languages, I use it to communicate with a machine, administrating systems, designing backend software, and analysing machine-written logfiles.
I knew that kind of comment would pop up, and oddly enough, I have the exact opposite opinion.
I am NOT a native English speaker. I cannot even write my name with ASCII letters.
To me, the situation looks exactly the reverse of your comment, I see the urge of putting Unicode everywhere as an essentially white savior syndrome that nobody asked for.
I want to be able to write name in Word or other text processing software, but nothing more. The command line, network protocols, etc. There are not the realm of fancy dynamic length characters. I want simple glyphs with fixed size bytes mapping. I don't want to embed a complicated Unicode parsing library in every software.
When I'm in the realm of programming, I'm not here to do politics. I'm here for the most reasonable, straightforward, working solution for a set of problems. If that means accepting a fixed set of characters, I'm 100% fine with that.
You may not care about putting your non-ASCII name in a file, but others do. As a programmer, you shouldn’t be preventing the user from doing what they want when other software lets them do it just fine. Especially things as simple as file names. You can’t say you’re just solving problems when you ignore the requirements of those problems. Internationalization requires Unicode support.
The cat is out of the bag, and if your software doesn’t support Unicode. you’re limiting what your users can do with it. Maybe that’s fine with you and a certain program, but it’s not for a lot of people.
You also don’t need to embed full Unicode parsing/support libs if you just treat the user’s input as a byte stream. When programmers try to get fancy and ToUppercase() a byte blob with no regard for what that blob represents - that’s when we have problems.
> The command line, network protocols, etc. There are not the realm of fancy dynamic length characters.
So how does one write, say, the backend for a reservation system managing thousands of hotel room bookings every hour, all over the world? I'm pretty sure there will be ALOT of customers with non-ascii names, not to mention adresses and hotel names. How does one write a system like twitter, where messages in every known system of writing need to be be sent, processed, stored, delivered, commented, displayed?
And btw. network protocols and backends work with arbitrary data all the time: Audio & video streaming, networked gaming, sensor readouts, images. Even plaintext webpages are often transmitted in compressed form. So if these systems are the "realm" of arbitrary bytestreams to encode everything from classical music, over sha-hashes, to gifs of dancing dogs, they may as well encode a few hundred different systems of writing and some smiley faces.
I always wonder why this complexity is necessary. For family, just make 1 static graphic with husband, wife, son, daughter. For color just make them all yellow. No mess of joining things like this.
I know, right? Just make the family 2 guys, one black and one white, with 4 kids of various skin tones. Everyone will definitely be happy with that representation. It’s a legitimate family after all.
I want to laugh but this is sadder than it is funny.
Your definition of a “normal” family is literally heterosexual and white* (certainly not interracial). This is exactly why people want emoji to be more representative. They don’t agree that their families are “not normal”.
*Make no mistake, “yellow” is still a proxy for white skin. If the skin tone chosen were, say, dark orange, it would be exactly as realistic as yellow but white people would have a hard time identifying with the color and would be upset.
My definition of a normal family is straight because that's what the vast majority are... if you want to convey a concept easily use the common version. No issue with orange skin instead, or grey, or whatever else. Lots of advertisements recently use blue, green, pink, purple, whatever to purposely not be specific skin tones.
I’m not sure what to say to you here. You’re quite literally saying that it should just please the majority. Not for any technical reason, either. Representing families of different sorts is entirely feasible technically. I could totally understand a “don’t bother supporting emoji at all” stance, because that would actually simplify, but that’s not what you’re saying.
> Lots of advertisements recently use blue,…
I can’t recall ever seeing this. When I see commercials, they’re generally real people, plus that one gecko.
I haven’t seen this in print advertisements either but I don’t see a lot of print advertisements in general.
Look up "globohomo art style", originally developed for Facebook as "Alegria". It's the same one with distorted people and all that. The point is they purposely make them representative of nobody to avoid this problem.
Flip phone emojis started in mid-1990s as carrier-specific custom characters on unused areas on Shift-JIS, and initially the implementations were largely the same. Wireless carriers then realized it could be used as differentiators, like by adding more specific mojis, colored mojis and animated GIF emojis, and then each strains of emojis that carriers offer grew into different dinosaurs over a decade and half until forcibly unified into a common ground in 2010s basically by Apple and Google.
Apple got into the emoji game when they entered Japanese market with iPhone 3G on SoftBank, and Google then helped it standardized in Unicode. At that exact point everyone was on the same page. And at next instant, they realized that they are not differentiating on emoji.
>For example, iTerm2 considers the "rosette" emoji to have width 1
It's not the case here, but for some emoji there's another issue: Unicode 9 changed the width for some codepoints (mostly emoji) from 1 to 2, and iTerm until very recently (don't know if it's released yet) defaulted to the Unicode 8 widths, with an opt-in escape sequence to change to Unicode 9.
>This approach comes from the wcwidth utility, and the comment at the top of the C source file provides further insight into the difficulties faced here.
That's link goes to Markus Kuhn's implementation from 2007. It supports Unicode 5, and is by now woefully out of date. You don't want to use it anymore.
Most terminals have their own definition, and the annoying part is that the client application and the terminal need to have theirs in sync or they get weird glitches when moving the cursor.
Shameless plug: Fish's solution is widecharwidth[0], which is a python script that parses the Unicode data files and generates a wcwidth for C++, Javascript and Rust. It's still a wcwidth, meaning that it has issues with joining code points, but it's at least a start. It's up-to-date with Unicode 14 and, unless they change the data format (again) should be easy to update to future Unicode releases.
It's public domain and used by at least fish and WezTerm.
> All terminal emulators I tested consider codepoints with an "East Asian Width" of N to have cell width of 1. This is incorrect in the case of Emoji Presentation Sequences - Unicode recommends they should be always treated as "East Asian Wide" (W).
According to the link you shared it's a bug in Xft and not st:
> Please don't bother reporting this bug to st, but notify the upstream Xft developers about fixing this bug.
Huh, very interesting. Never even considered emojis in the terminal before. Hope these issues get fixed though (even though I personally don't use emojis in my terminal).
kitty does indeed seem to do best among the four terminals I tested (Kitty, wezTerm, Konsole and Terminator) https://imgur.com/X0cD3jA The motorboat in Kitty is quite small though?
That's because kitty correctly prints not an emoji motorboat, but rather a text motorboat; the \U0001F6E5 defaults to text presentation and that is why east_asian_width returns 'N'.
You can get the fancy one with print("\U0001F6E5\uFE0Fhello"). However, the bug that the post alludes to is that not all terminals override the east_asian_width to 'W' when they see "\uFE0F". For example on gnome-terminal the "h" overlaps with the right half of the boat.
Please try DomTerm (https://domterm.org). The 2.9.4 AppImage (https://github.com/PerBothner/DomTerm/releases/tag/2.9.4) should have the needed support for grapheme clusters and hopefully work on reasonably up-to-date Linux systems. Of course there are more recent fixes and improvements if you don't mind building from source.
On my terminal I can delete family members. With the family emoji from the article of each back space removes a child, three key presses are needed to remove all 3 people in the emoji. It makes sense but very funny to see.
This is why I've stuck with Terminal.app over the years — there's basically one feature I would like (deeper color support) but otherwise I've missed out on tons of correctness or performance issues. The other features are nice but I really value a rock-solid terminal more.
How do terminals handle CJK characters? I remember opening something with those in konsole, and I think they distinctly occupied two columns, so I assume some width variation is supported in terminals.
Emojis are different from chars insofar as that the former are always (not by necessity, but in practice) multi-coloured. A set of mono-coloured emojis is probably a good idea.
Weird take on "avoid, if possible." I've just added a default Emoji font and begun to use them for little indicators of which machine I'm SSHed into. They work fine?
DomTerm (https://domterm.org) does a pretty good job IMNSO. See screenshot here: https://domterm.org/Features.html . DomTerm mostly delegates to the browser and the font how to compose extended Grapheme Clusters.
Unfortunately, as far as I know there is no "complete" monospace font set that handles emoji. Ideally, you want a font with two character widths, with double-width for emoji, hanji (CJK characters), and similar. Instead the browser will substitute these characters from some variable-width font, and then the spacing will be off.
DomTerm handles this by putting double-width characters as well as Extended Grapheme Clusters in a separate span that is forced to have the correct width. I created a JavaScript library https://github.com/PerBothner/unicode-properties based on other people's code but optimized for DomTerm's needs: It provides both East Asian Width (for recognizing double-width characters) and character classes (for grapheme clusters) in a single efficient trie structure.
The DomTerm equivalent of tmux's "select mode" is grapheme-cluster-aware, so left/right-arrow will correctly move over an entire grapheme cluster.
Is there an equivalent to the Acid2 CSS browser test[1] for terminal Unicode compliance? It would be great to move forward to a place where good Unicode support is table stakes for new as well as old terminals.
84 comments
[ 2.3 ms ] story [ 128 ms ] threadWill do. If its more complicated than a box-character, it doesn't belong in my terminal :-)
Or that I can't type my address, which contains ø?
Sincerely,
The rest of the world.
This is "computer language":
So is this So is this Whether we translate the meaning of the above into Mandarin, English, Spanish, Japanese, Swedish, Farsi, Swhaili, Hindi, Latin, Klingon, Dothraki, Sindarin or any other spoken, written or invented language we chose to be comfortable with, makes exactly zero difference.Why do we want to force programmers around the world to learn and use english?
That's how we get COBOL
Poop emoji might be dumb, but it doesn’t seem to hurt anything and if it makes developers support Unicode better, awesome.
Clearly the article is demonstrating that this hasn't come true.
Take something as simple as names: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...
Math is the language of the universe.
How am I supposed to represent/parse/write mathematical symbols like ∊ or ℚ or π or µ or ⊃ or ⎷ or ⋦ or ∑ when I only have ASCII?
Unicode currently encompasses over 144 thousand codepoints. How many are emoji? Only ~3.5k. So even if we removed emoji, 16 bits still wouldn’t be nearly enough.
Instead of saying we should go back to the past, one should first see why we don’t do the things we did in that past
This (and everything you wrote after) is incredibly bigoted. English is only just the most spoken language in the world. 1.35bn people have English as a 1st or 2nd language whereas 1.12bn people have Mandarin Chinese[1]. And even if it is the most popular language (by a narrow margin) why should that mean that everyone else has to speak it?
Let's imagine a world where the Chinese government got everyone (Cantonese etc.) retrained to speak Mandarin over the next 10 years so that it overtook English. Would you then change your view to think that everything should be written in Mandarin? No, you just want everyone else to adopt your own self centred approach so that you don't have to take into account anyone else's preferences.
[1] https://www.statista.com/statistics/266808/the-most-spoken-l....
I mean seriously, text encodings as a propaganda tool?
That said, I have always assumed that these problems we get with emoji also have equivalents that show up in other languages, it's just that we speak English and so only see them with Emoji. Maybe it's good that Emoji were included because it actually stresses our Unicode implementations!
And this is all before you introduce text that is written right-to-left like arabic scripts.
Just my 2 cents, but if ASCII includes, among other things, "characters" for linefeed, 4 unspecified "device control characters", EOT (End of Transmission), and a character to ring a bell, then I guess its okay to put some smileys and flags into Unicode.
The problem is the massive quantity of LTR stuff that is in RTL languages still (English words, programming language keywords, math, music, certain conventions, ...)
Anyway yes multi-character glyphs occur in various forms (like accents, sometimes: é, ñ...) but I always thought that the burden of supporting them was on the fonts, not on the Terminal or the Unicode standard itself.
I am well aware, and supportive, of the need to encode the entirety of human script and written symbolics in a unified format.
And the reason why my terminal doesn't need anything beyond basic modifying diacritic marks and single-codepoint symbols (like blocks and line characters for text-user-interfaces) is because I do not use it to write prose is a variety of languages, I use it to communicate with a machine, administrating systems, designing backend software, and analysing machine-written logfiles.
I am NOT a native English speaker. I cannot even write my name with ASCII letters.
To me, the situation looks exactly the reverse of your comment, I see the urge of putting Unicode everywhere as an essentially white savior syndrome that nobody asked for.
I want to be able to write name in Word or other text processing software, but nothing more. The command line, network protocols, etc. There are not the realm of fancy dynamic length characters. I want simple glyphs with fixed size bytes mapping. I don't want to embed a complicated Unicode parsing library in every software.
When I'm in the realm of programming, I'm not here to do politics. I'm here for the most reasonable, straightforward, working solution for a set of problems. If that means accepting a fixed set of characters, I'm 100% fine with that.
The cat is out of the bag, and if your software doesn’t support Unicode. you’re limiting what your users can do with it. Maybe that’s fine with you and a certain program, but it’s not for a lot of people.
You also don’t need to embed full Unicode parsing/support libs if you just treat the user’s input as a byte stream. When programmers try to get fancy and ToUppercase() a byte blob with no regard for what that blob represents - that’s when we have problems.
So how does one write, say, the backend for a reservation system managing thousands of hotel room bookings every hour, all over the world? I'm pretty sure there will be ALOT of customers with non-ascii names, not to mention adresses and hotel names. How does one write a system like twitter, where messages in every known system of writing need to be be sent, processed, stored, delivered, commented, displayed?
And btw. network protocols and backends work with arbitrary data all the time: Audio & video streaming, networked gaming, sensor readouts, images. Even plaintext webpages are often transmitted in compressed form. So if these systems are the "realm" of arbitrary bytestreams to encode everything from classical music, over sha-hashes, to gifs of dancing dogs, they may as well encode a few hundred different systems of writing and some smiley faces.
Because lots of families don't look like that, and people want to use emoji that reflect them and theirs
I want to laugh but this is sadder than it is funny.
Your definition of a “normal” family is literally heterosexual and white* (certainly not interracial). This is exactly why people want emoji to be more representative. They don’t agree that their families are “not normal”.
*Make no mistake, “yellow” is still a proxy for white skin. If the skin tone chosen were, say, dark orange, it would be exactly as realistic as yellow but white people would have a hard time identifying with the color and would be upset.
> Lots of advertisements recently use blue,…
I can’t recall ever seeing this. When I see commercials, they’re generally real people, plus that one gecko.
I haven’t seen this in print advertisements either but I don’t see a lot of print advertisements in general.
Flip phone emojis started in mid-1990s as carrier-specific custom characters on unused areas on Shift-JIS, and initially the implementations were largely the same. Wireless carriers then realized it could be used as differentiators, like by adding more specific mojis, colored mojis and animated GIF emojis, and then each strains of emojis that carriers offer grew into different dinosaurs over a decade and half until forcibly unified into a common ground in 2010s basically by Apple and Google.
Apple got into the emoji game when they entered Japanese market with iPhone 3G on SoftBank, and Google then helped it standardized in Unicode. At that exact point everyone was on the same page. And at next instant, they realized that they are not differentiating on emoji.
Turning Unicode into clipart collection is not.
It's not the case here, but for some emoji there's another issue: Unicode 9 changed the width for some codepoints (mostly emoji) from 1 to 2, and iTerm until very recently (don't know if it's released yet) defaulted to the Unicode 8 widths, with an opt-in escape sequence to change to Unicode 9.
>This approach comes from the wcwidth utility, and the comment at the top of the C source file provides further insight into the difficulties faced here.
That's link goes to Markus Kuhn's implementation from 2007. It supports Unicode 5, and is by now woefully out of date. You don't want to use it anymore.
Most terminals have their own definition, and the annoying part is that the client application and the terminal need to have theirs in sync or they get weird glitches when moving the cursor.
Shameless plug: Fish's solution is widecharwidth[0], which is a python script that parses the Unicode data files and generates a wcwidth for C++, Javascript and Rust. It's still a wcwidth, meaning that it has issues with joining code points, but it's at least a start. It's up-to-date with Unicode 14 and, unless they change the data format (again) should be easy to update to future Unicode releases.
It's public domain and used by at least fish and WezTerm.
[0]: https://github.com/ridiculousfish/widecharwidth
Sounds like a bug which could be fixed.
One of the simplest workarounds is to ensure that the default emoji font is black&white; the linked issue above suggests other workarounds.
Also in the official st FAQ: https://git.suckless.org/st/file/FAQ.html
(search for "when trying to render emoji")
You can get the fancy one with print("\U0001F6E5\uFE0Fhello"). However, the bug that the post alludes to is that not all terminals override the east_asian_width to 'W' when they see "\uFE0F". For example on gnome-terminal the "h" overlaps with the right half of the boat.
worked in putty too
But I have a thing for using built-in tools and Terminal.app gets a lot of the basics right. And has for so many years.
Here's a quick guide:
https://towardsdatascience.com/converting-emojis-to-text-a28...
https://github.com/microsoft/terminal/issues?q=is%3Aissue+is...
Unfortunately, as far as I know there is no "complete" monospace font set that handles emoji. Ideally, you want a font with two character widths, with double-width for emoji, hanji (CJK characters), and similar. Instead the browser will substitute these characters from some variable-width font, and then the spacing will be off.
DomTerm handles this by putting double-width characters as well as Extended Grapheme Clusters in a separate span that is forced to have the correct width. I created a JavaScript library https://github.com/PerBothner/unicode-properties based on other people's code but optimized for DomTerm's needs: It provides both East Asian Width (for recognizing double-width characters) and character classes (for grapheme clusters) in a single efficient trie structure.
The DomTerm equivalent of tmux's "select mode" is grapheme-cluster-aware, so left/right-arrow will correctly move over an entire grapheme cluster.
[1] https://en.wikipedia.org/wiki/Acid2