Totally agreeing on the benefits. But remember one thing: You english speaking people out there have it easy because of points 1-3 in the article.
This stops being the case the moment you are dealing with any non-ascii character. At that point, some assumptions stop being valid, like the fact that it stops being the case that every document is a valid UTF-8 encoded document.
If you treat arbitrary encoded data as UTF-8, depending on your environment, you will get thrown exceptions at or you will see question marks in various designs all over the place.
Combine this with the fact that browsers sometimes are not quite sure of what they are doing:
I've seen them sending latin1 but telling the server it's utf-8 or the other way around.
The rails people tried to detect utf-8-ness using the snowman and lately a checkmark (http://railssnowman.info/).
Finally, keep in mind that it's impossible to accurately detect the encoding if it isn't utf-8 without actually doing language analysis.
Soon, you'll notice that utf-8 isn't magically solving problems.
It's funny how many english speaking people talking to english audiences think that slapping a "; charset=utf-8" to their content-type headers suddenly makes their site utf-8 compliant.
As long as their content is 7 bit english and their users send data in 7 bit english, they could as well just have left the charset alone or set it to ASCII as ASCII = utf-8 as long as the first bit isn't set.
The application I'm maintaining, while being multi lingual, is, thankfully, targeting countries with languages that can be expressed in latin-1, so that's what we are (still) using.
I made various attempts at going utf-8, but in the end, between browsers lying and external third-party APIs still insisting on latin1, these efforts never bore fruit.
Still, I prefer site to have naïve charset=UTF-8 slapped that at least allows me to use non-ASCII characters mostly correctly most of the time, than to deal with ISO-8859-1 site (like Amazon, grr) which leaves me no choice but to omit or transliterate most characters.
I don't see any solution to character encoding problems while web browsers actively lie about what they are sending. But that's not a problem with UTF-8, or even with any specific encoding.
of course not. I was just saying that the fact that every ASCII document is also a UTF-8 document only makes life easier for people dealing with ASCII.
The moment you leave that safe area, all the usual problems will come and haunt you.
We specialized in the exportation of sport shoes and other products(clothing, bag,sunglasses,watches,belts,etc )which have great enjoyed popularty in the world market Many of our goods are on sales ,we can guarantee the crediblity by Pay-pal and delivery time .we would like to make a long termship.
1)Name : The perfect gift.
2)Grade : AAA+
3)Package : in original boxes
4)Color : various
5)MOQ : 1 pc
6)Payment : Pay-pal/Western Union / Credit card/Moneygram
7)Ship-ping : 4-7 days with guarantee of customs clearance, drop shipp-ing is accepted to customers'demands.
8)Who-lesale: very low price, you can make a small order first to test quality and service.
I used to think UTF was a great thing. But I came to realize it isn't good to go through all this trouble just to keep using a 70s US-centric API. UTF is error prone and the problems are often subtle and hard to catch.
Also, the same API has serious issues with buffer overflows and off by 1. It would be great to move on.
API as in the standard C API and obviously the part for ASCII string management. The UTF encoding was designed to keep that API working with Unicode. How would an encoding standard be an API? (Application Programming Interface)
I can't believe your smug attack gets voted up. You just proved you're either an idiot or a troll. You add nothing to the discussion.
If you're representing text which contains lots of characters that aren't in ASCII, like say Chinese, UTF-8 will consume much more storage than necessary. There are many languages where non-ASCII characters are extremely common.
He misses a very useful property of UTF-8 as well, it never contains null bytes. This trips up all sorts of heuristics for detecting binary files in various programs if you use e.g. UTF-16 for mostly ASCII text, since it then will contain lots of nulls.
UTF-8 is a very clever way to avoid problems on systems suffering under the mistaken assumption that text is 8-bit byte strings (cough UNIX cough), but that doesn't make it the ideal choice every time.
It is still very common for cross platform tools to not handle file names with non-ASCII characters for example. Both Mercurial and Git suffered from this last I looked.
The reason is that they treat file names as byte strings instead of text in some encoding, and therefore cannot translate to the proper encoding on platforms which treat file names as Unicode text, like Windows and OS X. OS X also uses a somewhat unconventional normalization form, which means you need to handle normalization as well.
The page takes 37KB in GBK, 39KB in UTF-8 and 73KB in UTF-16! (UTF-16 doubles cost of HTML markup but only saves ⅓ in text)
Even in pure Chinese plain text UTF-16 doesn't win by much. I've tested some random article: 18KB in UTF-16, 25KB in UTF-8. It's down to 9.5KB vs 10KB after gzipping.
I'm not surprised it doesn't make much of a difference with HTML since the markup contains so much ASCII. But 25 vs 18 kB is almost 40%. That might not be insignificant depending on how much text you're storing.
But it's a nitpick really, I just thought it he should have noted some of the disadvantages to UTF-8 as well.
> UTF-8 sequences sort in code point order.
You can verify this by inspecting the encodings in the table above. This means that Unix tools like join, ls, and sort (without options) don't need to handle UTF-8 specially.
mean?
Isn't ordering a property that is external to the encoding and language dependent?
(Though AFAICT ls on osx seems to sort fine italian and hungarian alphabets, even if it's unable to handle character length properly :) )
The only way to sanely handle unicode is to load it using a codec in the language of your choice, and manipulate it using the tools your language supplies for manipulating strings OR read all the insane specs for unicode and implement your own. Treating unicode as bytes will end in tears.
The article is correct if by substring you mean exact sequence of code points. This is still remarkable, because even that is not always possible in variable-width encodings.
Problem of searching for semantically equivalent text (taking into account decomposed forms, ligatures, etc.) is higher-level and applies to all Unicode encodings (i.e. using UCS-2 or UTF-32 doesn't solve it either).
Danger, Danger Will Robinson! UTF-8 strings can include overlong encodings (e.g. using 3 bytes to encode a code point that's normally encoded in 1 byte).
The Unicode standard disallows creation of these forms, but until recently (http://www.unicode.org/versions/corrigendum1.html), only "strongly discouraged" interpretation of them. I do not doubt that there is code in the wild which will interpret them as characters.
Recommending that people manipulate utf-8 directly is a bad idea for lots of reasons. It's fairly easy to end up with mixed normalizations and lots of other things which will come back to bite you. Just about everything, including GNU command line utilities, support proper unicode handling, why would you want to go back into the stone ages of dealing with bytes directly?
If you are doing any comparing of Unicode strings, or anything beyond the simplest manipulation of them, I recommend running them through the appropriate-for-your-task normalization process: http://unicode.org/reports/tr15/
You should understand what the normalization forms are. Which I say despite having only a bit of a vague idea myself. But I feel bad about it. (And actually plan on fixing it soon for one of my projects anyhow.)
UTF-8 was designed, in front of my eyes, on a
placemat in a New Jersey diner one night in September or so 1992.
What happened was this. We had used the original UTF from ISO 10646
to make Plan 9 support 16-bit characters, but we hated it. We were
close to shipping the system when, late one afternoon, I received a
call from some folks, I think at IBM - I remember them being in Austin
- who were in an X/Open committee meeting. They wanted Ken and me to
vet their FSS/UTF design. We understood why they were introducing a
new design, and Ken and I suddenly realized there was an opportunity
to use our experience to design a really good standard and get the
X/Open guys to push it out. We suggested this and the deal was, if we
could do it fast, OK. So we went to dinner, Ken figured out the
bit-packing, and when we came back to the lab after dinner we called
the X/Open guys and explained our scheme. We mailed them an outline
of our spec, and they replied saying that it was better than theirs (I
don't believe I ever actually saw their proposal; I know I don't
remember it) and how fast could we implement it? I think this was a
Wednesday night and we promised a complete running system by Monday,
which I think was when their big vote was.
24 comments
[ 2.2 ms ] story [ 48.1 ms ] threadThis stops being the case the moment you are dealing with any non-ascii character. At that point, some assumptions stop being valid, like the fact that it stops being the case that every document is a valid UTF-8 encoded document.
If you treat arbitrary encoded data as UTF-8, depending on your environment, you will get thrown exceptions at or you will see question marks in various designs all over the place.
Combine this with the fact that browsers sometimes are not quite sure of what they are doing:
I've seen them sending latin1 but telling the server it's utf-8 or the other way around.
The rails people tried to detect utf-8-ness using the snowman and lately a checkmark (http://railssnowman.info/).
Finally, keep in mind that it's impossible to accurately detect the encoding if it isn't utf-8 without actually doing language analysis.
Soon, you'll notice that utf-8 isn't magically solving problems.
It's funny how many english speaking people talking to english audiences think that slapping a "; charset=utf-8" to their content-type headers suddenly makes their site utf-8 compliant.
As long as their content is 7 bit english and their users send data in 7 bit english, they could as well just have left the charset alone or set it to ASCII as ASCII = utf-8 as long as the first bit isn't set.
The application I'm maintaining, while being multi lingual, is, thankfully, targeting countries with languages that can be expressed in latin-1, so that's what we are (still) using.
I made various attempts at going utf-8, but in the end, between browsers lying and external third-party APIs still insisting on latin1, these efforts never bore fruit.
The moment you leave that safe area, all the usual problems will come and haunt you.
( http://www.yessoso.com )
We specialized in the exportation of sport shoes and other products(clothing, bag,sunglasses,watches,belts,etc )which have great enjoyed popularty in the world market Many of our goods are on sales ,we can guarantee the crediblity by Pay-pal and delivery time .we would like to make a long termship.
1)Name : The perfect gift. 2)Grade : AAA+ 3)Package : in original boxes 4)Color : various 5)MOQ : 1 pc 6)Payment : Pay-pal/Western Union / Credit card/Moneygram 7)Ship-ping : 4-7 days with guarantee of customs clearance, drop shipp-ing is accepted to customers'demands. 8)Who-lesale: very low price, you can make a small order first to test quality and service.
( http://www.yessoso.com )
Online Contact . thank you.
Also, the same API has serious issues with buffer overflows and off by 1. It would be great to move on.
Again, down-votes of disagreement for contrarian opinions. Lovely HN.
I can't believe your smug attack gets voted up. You just proved you're either an idiot or a troll. You add nothing to the discussion.
He misses a very useful property of UTF-8 as well, it never contains null bytes. This trips up all sorts of heuristics for detecting binary files in various programs if you use e.g. UTF-16 for mostly ASCII text, since it then will contain lots of nulls.
UTF-8 is a very clever way to avoid problems on systems suffering under the mistaken assumption that text is 8-bit byte strings (cough UNIX cough), but that doesn't make it the ideal choice every time.
It is still very common for cross platform tools to not handle file names with non-ASCII characters for example. Both Mercurial and Git suffered from this last I looked.
The reason is that they treat file names as byte strings instead of text in some encoding, and therefore cannot translate to the proper encoding on platforms which treat file names as Unicode text, like Windows and OS X. OS X also uses a somewhat unconventional normalization form, which means you need to handle normalization as well.
I don't think that's a problem in practice. Taking for example:
http://www.baidu.com/s?wd=%D0%C2+%CE%C5
The page takes 37KB in GBK, 39KB in UTF-8 and 73KB in UTF-16! (UTF-16 doubles cost of HTML markup but only saves ⅓ in text)
Even in pure Chinese plain text UTF-16 doesn't win by much. I've tested some random article: 18KB in UTF-16, 25KB in UTF-8. It's down to 9.5KB vs 10KB after gzipping.
But it's a nitpick really, I just thought it he should have noted some of the disadvantages to UTF-8 as well.
> UTF-8 sequences sort in code point order. You can verify this by inspecting the encodings in the table above. This means that Unix tools like join, ls, and sort (without options) don't need to handle UTF-8 specially.
mean? Isn't ordering a property that is external to the encoding and language dependent? (Though AFAICT ls on osx seems to sort fine italian and hungarian alphabets, even if it's unable to handle character length properly :) )
EDIT: got it _code point order_ I'm an idiot
5. Substring search is just byte string search.
Nope. The same character sequences can have multiple utf-8 representations. Yes, this comes up in the real world, especially on the web.
Reference: http://en.wikipedia.org/wiki/Unicode_equivalence#Normal_form...
The only way to sanely handle unicode is to load it using a codec in the language of your choice, and manipulate it using the tools your language supplies for manipulating strings OR read all the insane specs for unicode and implement your own. Treating unicode as bytes will end in tears.
Problem of searching for semantically equivalent text (taking into account decomposed forms, ligatures, etc.) is higher-level and applies to all Unicode encodings (i.e. using UCS-2 or UTF-32 doesn't solve it either).
The Unicode standard disallows creation of these forms, but until recently (http://www.unicode.org/versions/corrigendum1.html), only "strongly discouraged" interpretation of them. I do not doubt that there is code in the wild which will interpret them as characters.
You should understand what the normalization forms are. Which I say despite having only a bit of a vague idea myself. But I feel bad about it. (And actually plan on fixing it soon for one of my projects anyhow.)
UTF-8 was designed, in front of my eyes, on a placemat in a New Jersey diner one night in September or so 1992.
What happened was this. We had used the original UTF from ISO 10646 to make Plan 9 support 16-bit characters, but we hated it. We were close to shipping the system when, late one afternoon, I received a call from some folks, I think at IBM - I remember them being in Austin - who were in an X/Open committee meeting. They wanted Ken and me to vet their FSS/UTF design. We understood why they were introducing a new design, and Ken and I suddenly realized there was an opportunity to use our experience to design a really good standard and get the X/Open guys to push it out. We suggested this and the deal was, if we could do it fast, OK. So we went to dinner, Ken figured out the bit-packing, and when we came back to the lab after dinner we called the X/Open guys and explained our scheme. We mailed them an outline of our spec, and they replied saying that it was better than theirs (I don't believe I ever actually saw their proposal; I know I don't remember it) and how fast could we implement it? I think this was a Wednesday night and we promised a complete running system by Monday, which I think was when their big vote was.
sorted(map(encode, values)) == sorted(values)
</ shameless plug >