> Both UCS-2 and UTF-16 are character encodings for Unicode.
UCS-2 is not an encoding that is generally compatible with Unicode. It's kind of like saying that 7-bit ASCII and UTF-8 are character encodings for Unicode.
Sure, but if I buy a product released in 2012 that advertises "Unicode" support, and it turns out that it only supports a version of it from the era of IE4, I'm probably going to whine.
From Wikipedia (http://en.wikipedia.org/wiki/UTF-8): "UTF-8 (UCS Transformation Format — 8-bit[1]) is a variable-width encoding that can represent every character in the Unicode character set."
Very nice write up. I was actually looking for something like this about a week ago, and was referred to the ECMAScript spec (section 8.4) which talked about "UTF-16 code units" - which I believe is just UCS-2. If this is the case, I kind of wonder if the spec should be updated to make things a little more clear, since the issue isn't straight forward for those who don't know a lot about unicode.
Because Windows uses UTF-16 as its Unicode encoding. Why? Because Unicode support in Windows started in NT, and that used UCS2. Why? Because when NT was first being developed in 1989/1990, UTF-8 hadn't been invented yet, and wouldn't be for another few years. And UTF-8 didn't have much traction at all until after 2000, by which time it was too late.
Also, Windows has some problems with UTF-8 due to MB_LEN_MAX[0] being 2. Which cannot be changed because that would break the ABI. (e.g. "char foo[MB_LEN_MAX * len + 1];") Those problems aren't totally insurmountable as Windows does have UTF-8 code pages which work, but it's not totally plain sailing.
In any case, going back now would be an admission of a mistake.
Thus, Windows is UTF-16. And the infection spreads....
This means that for applications that want to store binary data as efficiently as possible in localStorage (e.g. Offline Wikipedia https://news.ycombinator.com/item?id=3409512), you can pack two bytes into each string character. ECMAScript strings are just arrays of 16-bit unsigned integers (e.g., '\ud800' is a valid JS string but is not valid UTF-16).
15 comments
[ 2.7 ms ] story [ 23.5 ms ] threadUCS-2 is not an encoding that is generally compatible with Unicode. It's kind of like saying that 7-bit ASCII and UTF-8 are character encodings for Unicode.
They can only encode a subset of Unicode, sure, but they’re still character encodings for Unicode, right?
From the Unicode website (http://www.unicode.org/faq/utf_bom.html#UTF8): "UTF-8 is the byte-oriented encoding form of Unicode."
From Wikipedia (http://en.wikipedia.org/wiki/UTF-8): "UTF-8 (UCS Transformation Format — 8-bit[1]) is a variable-width encoding that can represent every character in the Unicode character set."
JavaScript’s internal encoding is closer to UCS-2 than it is to UTF-16, but it doesn’t guarantee that a string is valid UCS-2 (or valid UTF-16).
http://wiki.ecmascript.org/doku.php?id=strawman:support_full...
Also, Windows has some problems with UTF-8 due to MB_LEN_MAX[0] being 2. Which cannot be changed because that would break the ABI. (e.g. "char foo[MB_LEN_MAX * len + 1];") Those problems aren't totally insurmountable as Windows does have UTF-8 code pages which work, but it's not totally plain sailing.
In any case, going back now would be an admission of a mistake.
Thus, Windows is UTF-16. And the infection spreads....
[0] http://www.kernel.org/doc/man-pages/online/pages/man3/MB_LEN...
JavaScript strings are like UCS-2, except that they allow surrogate characters.