It's garbage anyway, which you can (non-reliably) guess by there being a Korean character in the middle of C/J kanji. (Kanji are not completely gone from Korean, but mostly.)
If UTF-8 decoding fails, then it's time to pull out the fancy statistical tools to (unreliably) guess an encoding. But that should be a fallback, not the first thing you try.
I think there are a bunch of encodings that just repurposed a few ASCII characters as different characters - someone on this page was giving the example of some Swedish encoding where {}| were replaced with three accented Swedish letters. There are probably a bunch of others. In those cases, the text will decode fine as UTF-8, but it will display the wrong thing.
I meant, some messages will decode fine as UTF-8, but in some other messages there may be letters which don't fit in 7 bits. So some simple testing, especially with English words, will show it to work fine. But as soon as a non-7-bit characters creeps, it will stop working fine.
> Many charsets decode fine as UTF-8 as long as the message happens to fit in ASCII.
At which point the message is effectively ASCII. UTF-8 is a superset of ASCII, so "decoding" ASCII as UTF-8 is fine.
(Yes, I know there are some Japanese text encodings where 0x5c is decoded as "¥" instead of "\". But they're sometimes treated as backslashes even though they look like ¥ symbols so handling them "correctly" is complicated.)
I'm guessing they're thinking of Extended ASCII (the 8-bit one that's actually multiple different encodings, but the lower half is shared with 7-bit ASCII and so that part does fit in UTF-8 while the upper half likely won't if the message actually uses it).
"Fun" fact: some video subtitle formats (ASS specifically) use "text{\b1}bold" to format things — but since they were primarily used to subtitle Japanese anime, this frequently became "text{¥b1}bold". Which is all good and well, except when those subtitles moved to UTF-8 they kept the ¥. So now you have to support ¥ (0xC2 0xA5) as a markup/control character in those subtitles.
The definition GP is using most likely refers to non-ASCII sequences that validly decode as UTF-8, because virtually every major charset in practical use has ASCII as a subset.
ASCII (or ISO646) has local variants [0] that replace certain characters, like $ with £, or \ with ¥. The latter is still in use in Japan. That’s why "ASCII" is sometimes clarified as "US-ASCII".
I'm skeptical. Any charset that uses bytes 128-255 as characters is unlikely to successfully decode to UTF-8. Are there really many others that only use 0-127, or most text ends up only using 0-127?
> If UTF-8 decoding fails, then it's time to pull out the fancy statistical tools to (unreliably) guess an encoding.
Don't really even need to do that. There's only a handful other encodings still in common use, just try each of them as fallbacks and see which one works without errors, and you'll manage the vast majority of what's not UTF-8.
(We recently did just that for a system that handles unreliable input, I think I remember our fallback only has 3 additional encodings before it gives up and it's been working fine)
These are actually interpreted as the corresponding C1 control codes by Windows, so arguably not invalid in practice, just formally reserved to be reassigned to other characters in the future.
The person you're replying to sort of addresses this, though not completely.
Since UTF-8 is a variable-length encoding, it somewhat naturally has some error detection built in. Fixed-length encodings don't really have that, and for some of them, any byte value, 0 to 255, in any position, is valid. (Some have a few byte values that are invalid or reserved, but the point still stands.)
So you could very easily pick a "next most common encoding" after UTF-8 fails, try it, find that it works (that is, no bytes are invalid in that encoding), but it turns out that's still not actually the correct encoding. The statistics-based approach will nearly always yield better results. Even a statistics-based approach that restricts you to a few possible encodings that you know are most likely will do better.
Actually, I can just assume UTF-8, since that's what the world standardized on. Just like I can assume the length of a meter or the weight of a gram. There is no need to have dozens of incompatible systems.
You can start by assuming UTF-8, then move on to other heuristics if UTF-8 decoding fails. UTF-8 is "picky" about the sequence of bytes in multi-byte sequences; it's extraordinarily unlikely that text in any other encoding will satisfy its requirements.
(Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)
Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.
I don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.
Encoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's not right -- where did those strings come from?
Taking an ID3 tag example, if you are mass-converting/sanitizing/etc. tag titles and other similar metadata, the strings are often very short, sometimes only even a single codepoint or character, and proper assumptions of encoding can not be relied on because so many people violate specs and put whatever they want in there, which is the whole point of wanting to sanitize the info in the first place.
I don't think that's true. Looking at how it's encoded[0], it seems similar to many other country/language-specific encodings: bytes 0-127 are the control chars and latin alphabet and symbols, and is more-or-less ASCII, then 128-255 represent characters specific to the language at hand.
The only way you'd successfully decode Shift-JIS as UTF-8 is if it essentially is just latin-alphabet text (though the yen symbol would incorrectly display as a '\'). If it includes any non-trival amount of Japanese in it, it'll fail to decode as UTF-8.
As for whether or not you can then (after it fails to decode as UTF-8) use statistical analysis to reliably figure out that it's in fact Shift-JIS, and not something else, I can't speak to that.
I in fact can assume it. If the assumption is wrong then that's someone else's problem. 15 years ago I wrote a bunch of code using uchardet to detect encodings and it was a pretty useful feature at the time. In the last decade everything I've touched has required UTF-8 unless it's been interoperating with a specific legacy system which has some other fixed charset, and it's never been an issue.
> would you guess the unit instead of specifying what you expect?
It depends on the circumstances. It might be the least bad thing to do. Or not.
But that wasn't my point. I replied to this:
> I can assume the length of a meter or the weight of a gram
Sure, the length of a meter and the "weight" of a gram are both standardized. (To be very picky, "gram" is a mass, not a weight. The actual weight depends on the "g" constant, which on average is 9.81 m/s^2 on earth, but can vary about 0.5%.)
So if you know the input is in meters, you don't need to do any further processing.
But dealing with input text files with an unknown encoding is like dealing with input lengths with an unknown unit.
So while UTF-8 itself might be standardized, it is not the same as all input text files always being in UTF-8.
You can choose to say that all input text files must be in valid UTF-8, or the program refuses to load them. Or you can use silent heuristics. Or something inbetween.
I don't understand the outraged tone. Asking developers to write actually good software shouldn't be viewed as some kind of crazy imposition. We don't have to write everything like it is running on a spacecraft (which I never claimed), but we should try to make it actually good. For example, if there was a web browser with a security compromise and the makers left it unfixed for a long time, there would be consequences. Saying "well, it's just a browser" wouldn't cut it.
More to the point, what situation can you think of where guessing measurement units is a good idea? In a CNC machine? Maps program? Somewhere else? You seem to have omitted the actual counterargument part from your counterargument, while adding a hearty dash of misplaced outrage.
The way you get that reality is you do the opposite of the recommendation of Postel’s law: be very picky about what you consume and fail loudly if it’s not UTF-8
There’s a difference between asssuming and not making a distinction.
Very few developers I’ve met know could make a distinction. They’d see a few off characters and think it’s some one-off bug but it’s because they’re both assuming an encoding.
Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to.
> Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to.
Unless you want it fixed immediately, then a million dollars should motivate almost any developer to spend a month learning, a month doing, and a few years on vacation. A billion is incomprehensible.
Default UTF-8 is better than the linked suggestion of using a heuristic, but failing catastrophic when old data is encountered is unacceptable. There must be a fallback.
(Note that the heuristic for "is this intended to be UTF-8" is pretty reliable, but most other encoding-detection heuristics are very bad quality)
Off-topic, but the bit numbering convention is deliciously confusing.
Little-endian bytes (lowest byte is leftmost) and big-endian bits (bits contributing less numerical value are rightmost) are normal, but the bits are referenced/numbered little-endian (first bit is leftmost even though it contributes the most numerical value). When I first read the numbering convention I thought it was going to be a breath of fresh air of someone using the much more sane, but non-standard, little-endian bits with little-endian bytes, but it was actually another layered twist. Hopefully someday English can write numbers little-endian, which is objectively superior, and do away with this whole mess.
It actually is if we did not need to consider historical baggage.
Especially in programming where we already use in-band encoding like 0x to denote a hex string or 0b to denote a binary string. I like using 1{s}, e.g. 1x to denote a little-endian hex string and 1b to denote a little-endian binary string, to denote little-endian encoding.
But, even ignoring programming, it is still better in normal use. The Arabic language got it right writing little-endian (Arabic numbers are written the same, but Arabic is a right-to-left language so it is actually little-endian), and the European languages just stole it stupid by copying the form instead of the function.
From what I have seen, low numbers in Arabic are spoken/written little-endian (twenty five is five and twenty). Apparently German as well. The internet claims that historically large numbers in Arabic were also written out (as in when using number words rather than numerals) little-endian.
"Although generally found in text written with the Arabic abjad ("alphabet"), numbers written with these numerals also place the most-significant digit to the left, so they read from left to right (though digits are not always said in order from most to least significant[10]). The requisite changes in reading direction are found in text that mixes left-to-right writing systems with right-to-left systems."
Basically everyone uses x86 bit numbering. It has the pleasant property that the place value of every bit is always 2^n (or -2^n for a sign bit), and zero-extending a value doesn't change the numbering of its bits.
The more I thought it through, even assuming x86, I guess there’s just no “correct” way to casually reference bit positions when we read them in the opposite order from the machine. Are they being referenced from the perspective of a human consumer of text, or the machine’s perspective as a consumer of bits? If I were writing that content, I’d have a difficult time deciding on which to use. If I were writing for a lay person, referencing left-to-right seems obvious, but in this case where the audience is primarily developers, it becomes much less obvious.
Thanks for giving me an example of an architecture where the bits are labelled backwards, I'd never encountered that before. I've always appreciated that the bit number represents 2 to the power of that number.
How about assume utf-8, and if someone has some binary file they'd rather a program interpret as some other format, they turn it into utf-8 using a standalone program first. Instead of burning this guess-what-bytes-they-might-like nonsense into all the software.
We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.
> should be asking the user to select explicit input and output formats
It depends on the requirements.
If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.
> Isn't the point of the article to describe how an engineer would create such a tool?
Honestly, no, because the tool that it's suggesting how to write isn't one that will even come close to doing a good job.
If you want to write such a tool, the first thing you need to do is to understand what the correct answer is. And to do that, you need to sample your input set to figure out what the correct answer should be for several inputs where it matters. There's unfortunately no easy way to avoid that work; universal charset detection isn't really a thing that works all that well.
>universal charset detection isn't really a thing that works all that well.
This seems like something LLMs would be good at. A mundane use of them, but I bet they'd be really good at determining that the input has the wrong encoding. Then the program would iterate through encodings, from most probable to least, and select the one that the LLM likes the most. Granted, this means your tool will be 1GB or more. But hey, thems the breaks.
Yeah, that could be an interesting use of LLMs. It could at least tell you which languages might be present in the input text.
In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|".
So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.
I'm glad that didn't stick around, like ¥/₩ being used for directory separators like \. I can't imagine trying to read source code with those substitutions.
Encrypting without zipping doesn't leak any information about the content. You can't rule out certain byte sequences (other than by total length) just by looking at the ciphertext length.
If "oui" compresses to two bytes and "non" compresses to one byte, and then you go over them with a stream cipher, which is which:
This has nothing to do with compression. If you use "yes" and "no" instead of "oui" and "non" (which just happen to be three characters each) and you compress "yes" to "T" and "no" to "F" then the uncompressed text will be the leaky one.
Yes, and my example was an example meant to prove the opposite idea. The point is that it is irrelevant whether you compress or not. You can leak information either way.
It's in the article if you would bother to read it LOL. "simply measuring the size of packets without decoding them can identify whole words and phrases with a high rate of accuracy . . . [the researchers] can search for chosen phrases within the encrypted data"
Cryptography noob here: I'm confused by "Encrypting without zipping doesn't leak any information about the content." Logically speaking, if we compress first and therefore "the content" will now refer to "the zipped content", doesn't this mean we still can't get any useful information?
Encryption generally leaks the size of the plaintext.
This is true in both the compressed and non-compressed case. However with compression the size of the plaintext depends on the contents, so the leak of the size can matter more than when not using compression.
Even without compression this can matter sometimes. Imagine compressing "yes" vs "no".
> Encryption generally leaks the size of the plaintext.
Ah, I see. Naïvely, this seems like a really bad thing for an encryption algorithm to do—is there no way around it? Like, why is encryption different from hashing in this regard?
There are methods, but they are generally very inefficient bandwidth wise in the general case. The general approach is to add extra text (pad) so that all messages are a fixed size (or e.g. some power of 2). The higher the fixed size is, the less information is leaked and the less efficient it is. E.g. if you pad to 64mb but need to transmit a 1mb message, that is 63mb of extra data to transmit.
Part of the problem (afaik) is we lack good math tools to analyze the trade offs of different padding size vs how much extra privacy they provide. This makes it hard to reason about how much padding is "enough".
Another approach is adding a random amount of padding. This can be defeated if you can force the victim to resend messages (which you then average out the size of).
Hashing is different because you don't have to reconstruct the message from the hash. With encryption the recipient needs to decrypt the message eventually and get the original back. However there is no way to transmit (a maximally compressed) message in less space then it takes up.
There are special cases where this doesn't apply e.g. if you have a fixed transmission schedule where you send a sprcific number of bytes on a specific agreed upon schedule.
Not OP, but 'zipping and encrypting' one thing (a file for example) does not leak information by itself. The problem comes when an adversary is able to see the length of your encrypted data, and then can see how that length changes over time - especially if the attacker can control part of the input fed to the compressor.
So if you compressed the string "Bob likes yams" and I could convince you to append a string to it and compress again, then I could see how much the compressed length changed.
If the string I gave you was something already in your data then the string would compress more than it would if the string I gave you was not already in your data - "Bob likes yams and potatoes" will be larger than "Bob likes yams likes Bob".
If the only thing I can see about your data is the length and how it changes under compression - and I can get you to compress that along with data that I hand to you - then eventually I can learn the secret parts of your data.
It's an extension of the chosen-plaintext attack, and so requires the attacker to be able to send custom text that they know is in the encrypted payload. If the unencrypted payload is "our-secret-data :::: some user specified text", then the attacker can eventually determine the contents of our-secret-data by observing how the size of the encrypted response changes as they change the text when the compression step matches up with a part of the secret data. It can be defeated by adding random-length padding after compression and before the encryption step, though.
Yes, of course it leaks more information than encryption without compression, because that’s just encryption which doesn’t leak anything.
In an enormous number of real world cases adversaries can end up including attacker-controller input alongside secret data. In that case you can guess at secret data and if you guess correctly, you get smaller compressed output. But even without that, imagine the worst case: a 1TB file that compresses to a handful of bytes. Pretty clearly the overwhelming majority of the text is just duplicate bytes. That’s information which is leaked.
Essentially if you zip something, repeated text will be deduplicated.
For example "FooFoo" will be smaller than "FooBar" since there is a repeated pattern in the first one.
The attacker can look at the file size and make guesses about how repetitive the text is if they know what the uncompressed or normal size is.
This gets more powerful if the attacker can insert some of their own plaintext.
For example if the plaintext is "Foo" and the attacker inserts "Fo" (giving "FooFo") the result will be smaller than if they inserted zq where there is no pattern. By making lots of guesses the attacker can figure out the secret part of the text a little bit at a time just by observing the size of the ciphertext after inserting different guesses.
That is not the only way. There are other ways of knowing partial contents of files and changes to files, depending on the situation. If the document is a known form in which one of five boxes is checked by the sender, it's probably not hard to rule out certain selections based on the ciphertext length, if not pin down the contents exactly.
I'm not sure i entirely understand your example (if there are 5 checkboxes and 1 checked, presumably length would be the same regardless which one of those are checked). However to your broader point, i agree there exist scenarios along those lines (e.g. fingerprinting known communication based on length), however most of them apply even better when not using compression.
The checkbox example is completely plausible. There is no guarantee that all checkboxes lead to the same number of bytes changed in the file when checked. What if the format makes a note of the page number wherever a checkbox is checked? 1X could be two bytes and 15X would be three.
And even if the format only stored the checkbox states as a single bit each (unlikely), compression algorithms don't care. They will behave differently on different byte sequences, which can easily lead to a difference in output length.
The attack you're referring to is not specific to compression. It's the same class of attack that can reveal keystrokes over older versions of ssh based on packet size and timing, even on uncompressed connections. Conversely, fixed-bitrate voice streams don't have the same vulnerability as variable-bitrate encodings even though they're still compressed.
The version of your checkbox example which is vulnerable without any formal data compression is when the checkbox is encoded in a field that is only included or changes in length if the value isn't the default, common in uncompressed variable-length encodings like JSON.
I'm sure that the people getting hacked care deeply about whether the attack they suffered was sui generis.
Also, zip/deflate etc was not designed to eliminate side channel leakage. Some compression schemes obviously (with padding) can mitigate leaks, but it has to be done deliberately
Any of it has to be done deliberately. The length of the data reveals something about its contents whether it's compressed or not.
The special concern with compression is when attacker-controlled data is compressed against secret data because then the attacker can measure the length multiple times and deduce the secret based not just on the length but on how the length changes when the secret is constant and the attacker-controlled data varies. This can be mitigated with random padding (makes the attack take many times more iterations because it now requires statistical sampling) or prevented by compressing the sensitive data and attacker-controlled data separately.
Only with "pure" CJK text in a flat text file; for most real-world situations you'll have enough ASCII text that UTF-8 will be smaller: HMTL/XML tags, email headers, things like that. I did some tests a few years back, and wasn't really able to come up with a real-world situation where UTF-16 is smaller. I'm sure some situations exist, but by and large, CJK users are better off with UTF-8.
Yep. I'm a heavy user of CJK languages and I don't give a damn about the slightly increased plaintext storage. Give me UTF-8 any day, every day. Legacy two-byte encodings can't represent all of the historical glyphs anyway, so there's no room for nationalist crap here.
Well, it's great that you did some tests a few years back, but I'm not sure how that qualifies you to make such a sweeping generalization about CJK text encoding. It's easy to dismiss UTF-16's benefits when you're only looking at a narrow slice of the real world, ignoring the vast amounts of pure CJK literature, historical archives, and user-generated content out there.
> they turn it into utf-8 using a standalone program first
I took the article to be for people who would be writing that "standalone program"?
I have certainly been in a position where I was the person who had to deal with input text files with unknown encodings. There was no-one else to hand off the problem to.
Then you don't have to worry about it since you won't get the work in the future? Someone else, with this presumably correct software, will always be able to do it for less, faster, and at a higher quality.
That's how business works...
If such a business competitor doesn't exist, then yes charge extra, and actually do the work correctly.
Am I missing something here? The work is ingesting documents from uncontrolled sources that might not all be UTF-8 and handling them correctly. Using an encoding guessing tool is the means to do that. In practice since there are only a few widely-used encodings and they're not terribly ambiguous it means that everything just works and users happily use the software.
This isn't some theoretical thing, we do this at $dayjob right now not only guessing the encoding but the file-type as well so that we can make sense out of whatever garbage our users upload. Everything from malformed CSV exports form Excel to PDFs that are just JPEGs of scanned documents. It works, and it works well.
And of course it does, the files our users are handing to us work on their machines. They can open them up and look at them in whatever local software they produced them with, there's no excuse for us to be unable to do the same.
The FCC ULS database records are stored in a combination of no fewer than three different encodings(1252, UTF8, and something else for a handful of German names) that vary per record.
When I brought this up they said something to the effect of: it's already unicodes it has tilde letters!
I did once have a file that had UTF8, Windows-1252, MARC8, and VT100 (really) all mixed up in it. I think the data had gone through multiple migrations between software in its past.
I had write to my own "clean this as well as possible" thing, and it did a good enough job.
> We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.
What ever happened to the Robustness Principle[1]? I think the entire comment section of this article has forgotten it. IMO the best software accepts many formats and "deals with it," or at least attempts to, rather than just exiting with "Hahah, Error 19923 Wrong Input Format. Try again, loser."
We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded. This is in some sense relatively recent, so word is still getting around, as the programming world does not move as quickly as it fancies itself to.
If you'd like to see why, read the HTML 5 parsing portion of the spec. Slowly and carefully. Try to understand what is going on and why. A skim will not reveal the issue. You will come to a much greater understanding of the problem.
Some study of what had happened when we tried to upgrade TCP (not the 4->6 transition, that's its own thing) and why the only two protocols that can practically exist on the Internet anymore are TCP and UDP may also be of interest.
Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec? Or is it genuinely unexplainable without experiencing something first-hand?
>>> the best software accepts many formats and "deals with it,"
>> We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded.
>> If you'd like to see why, read the HTML 5 parsing portion of the spec.
> Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec
I think the point was that the HTML 5 spec tries to parse all kinds of weird input instead of drawing a line in the sand and forcing the input to follow a simple format?
It isn't bad. In fact it's quite good. But it is very much a case of closing the barn door after the animals got out. You see in the standard that the effort was put to corral them back in, and I'm very glad they did, but it certainly was not free.
The explosion of the web happened in no small part because of how easy it was to write some HTML and get a basic, working webpage out of it. If you nested some tags the wrong way and the browser just put up an error page, rather than doing a (usually) pretty good job figuring out what you actually meant, people would get frustrated faster and not bother with it at all.
But imagine if our C/C++/Java/Rust/Go/etc. compilers were like "syntax error, but ehhhhh you probably meant to put a closing brace there, so let's just pretend you did". That would be a nightmare of bugs and security issues.
The difficulty in drawing a line in the sand and sticking to the spec, though, is that of user blame. Let's say you implement a spec perfectly -- even if you are the originator of the spec -- and then someone comes along and builds something of their own that writes out files that don't conform to the spec. Your software throws up an error and says "invalid file", but the other piece of software can read it back in just fine. Users don't know or care about specifications; they just know that your software "doesn't work" for the files they have, and the other software does. If you try to tell them that the file is bad, and the other software has a bug, they really won't care.
A possible solution to this is for large organizations to be intransigent about standards compliance. If your personal mail server rejects mail that isn't well-formed, you're just being a masochist because nobody is going to change for you. If Google does it, everybody else is going to fix their stuff because otherwise they can't send to gmail.
But imagine if our C/C++/Java/Rust/Go/etc. compilers were like "syntax error, but ehhhhh you probably meant to put a closing brace there, so let's just pretend you did". That would be a nightmare of bugs and security issues.
It proves that engineering is hard and that everything has costs and benefits, and you can't make them go away by just ignoring them. It turns out that "being robust" had a lot of costs we didn't see.
It also shows the harmfulness of binary black and white thinking in engineering. There are choices other than "just let everyone do whatever and hope all the different things picking up the pieces do it in more or less the same way" and "rigidly specify a spec and blow up the universe at the slightest deviation". Both of those easy-to-specify choices have excessive costs. There is no escape from hard design tasks. XHTML may always have been doomed to fail, but that is not to say that HTML had to be allowed to be as loosey-goosey as it is, either.
Had a gradient of failure been introduced rather than a rigid rock wall, things very likely wouldn't have gotten as badly out of hand as they did. If, for instance, a screwed up table was specified to deliberately render in a very aesthetically unappealing manner, but not crash the entire page the way XHTML did, people would have not come to depend so much on HTML being sloppy. The resulting broken page would still be somewhat usable, but there would have been motivation to fix it, rather than the world we actually live in where it all just seemed to work.
Being lenient is all well and good when the consequences are mild. When the consequences of misinterpreting or interpreting differently to a second implementation becomes costly, such as a security exploit, then the Robustness Principle becomes less obviously a win.
It's important to understand that every implementation will try to fix-up formatting problems in their own way unique to their particular implementation. From that you get various desync or reinterpretation attacks (eg. HTTP request smuggling).
I've tried telling users "sorry, your file isn't to spec", and they say "but it works with <competitor>", and that ideology flies right out the window, along with their money.
Exactly. "Accepting and trying" is how a lot of popular software won their market. Look at HN's favorite media player, VLC. In the past, media player software were horrible, refusing to play all but the most tightly constrained set of allowed containers/codecs. I remember spending the early 2000s trying to get Windows Media Player to play MPEGs by downloading codec packs and trying to cast secret spells into the Windows Registry. Yuck! Then along comes VLC which accepts whatever you throw at it, and that software is basically everywhere now. You can throw line noise at VLC and it will try its best to play something!
Yes, and that's why Postel's law is more of an empirical observation (a law of nature, if you will) on which software survives and which doesn't. You may dislike it but that won't make it go away.
I see it as the opposite, actually. They will pay you for a robust product: one that works for them. They have no care for the technical minutiae of your implementation, because they're just trying to do actually interesting things, with the help of your product. This is the customer perspective.
The trick is to enforce conformance right from the start in the first implementation of a format. Shipping a product that doesn't interop with the existing tools is a non-starter so the devs will have to fix their shit first.
As you say, unfortunately the genie cannot be put back in the bottle for formats that already have defective implementations in the wild.
The problem is that unless you restrict everyone else you'll get your own product not to interact with *new" tools too.
E.g. you produce valid .wat files, but my software which also outputs those has some bits screwed up.
My program can read both .wat but yours can't, but I have 5% market share.
Your users complain they sometimes receive files your software can't read while the competitor can. Do you tell them "well that file is invalid, tell whoever sent it to you to change the software they use"?
The genie can't stay in the bottle unless you have some sort of certification authority and even that may not be enough (see USB)
But the more every actor aims to follow the spec rather than reproduce everyone else's bugs, the less quickly the ecosystem devolves into a horrid tangled mess that's nigh impossible for a new entrant.
"So you're saying that I can have an advantage over everyone else if I implement all the spec features, plus one extra feature that adds convenience to our users?
And you're saying that by doing this, not only do I gain an advantage over the existing competition, but I also make it more difficult for more competitors to appear?
The customer does not know. They just want it to work. They may be using something that someone else gave them. The original source system of the file may not be changeable. But most importantly, their boss just wants it to work. or else.
(A text file containing only the ASCII bytes "Bush hid the facts", when opened in Windows Notepad, displays a sequence of CJK characters instead of the expected English sentence.)
This basically loses data integrity if it's wrong though.
You might want to do that with human input if it's helpful to the user - ie user enters a phone number and you strip dashes etc. But if it's machine to machine, it should just follow the spec.
That is fine in contexts where a wrong guess does no harm.
But that is not always the case, and e.g. silently "fixing" text encoding issues can often corrupt the data if you get it wrong.
By all means offer options of you want, but if you do flag very clearly to the user that they're taking a risk of corrupting the data unless any errors are very apparent and trivial to undo.
I've lived through dealing with non-UTF8 encoding issues and it was a truly gigantic pain in the ass. I'm much more on the side now of people who only want to deal with UTF8 and fully support software that tells any other encoding to go pound sand. The harder life gets for people who use other encodings (yes, particularly Microsoft) the more incentive they have to eventually get on board and stop costing everyone time and effort managing all this nonsense.
The article addresses this, that current thinking in many places is that the robustness principle / Postel's Law maybe wasn't the best idea.
If you reject malformed input, then the person who created it has to go back and fix it and try again. If you interpret malformed input the best you can (and get it right), then everyone else implementing the same thing in the future now also has to implement your heuristics and workarounds. The malformed input effectively becomes a part of the spec.
This is why HTML & CSS are the garbage dump they are today, and why different browsers still don't always display pages exactly alike. The reason HTML5 exists is because people finally just gave up and decided to standardize all the broken behavior that was floating around in the wild. Pre-HTML5, the web was an outright dumpster fire of browser compatibility issues (as opposed to the mere garbage dump we have today).
Anyway, it's not really important to try to convince you that Postel's Law is bad; what's important is that you know that many people are starting to think it's bad, and there's no longer any strong consensus that it was ever a good thing.
I doubt you can handle UTF-8 properly with that attitude.
The problems is, there is one very popular OS which is very hard to enforce UTF-8 everywhere, Microsoft Windows.
It's very hard to ensure all the software stack you are depending on it use Unicode version of Win32 API. Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. Even if you don't you have to ensure all the low level code you are depending on it do the same for you.
Oh and don't forget about the Unicode Normalizations. There is no THE UTF-8. There are bunch of UTF-8s with different Unicode normalizations. Apple macOS use NFD while other mostly use NFC.
These are Just some examples. When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.
Why would it break? If you just assume that the system codepage is UTF-8, then sure. If you specifically say in your manifest that you want UTF-8, then Windows (10+) will give you UTF-8 regardless of which locale it is:
> Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back.
Yes. You should convert your strings. Thankfully, UTF-16 is very difficult to confuse with UTF-8 because they're completely incompatible encodings. Conversion is (or should be) a relatively simple process in basically any modern language or environment. And personally, I've never run into a problem where the difference between NFC and NFD mattered. (Do you have an example?). The different forms are (or should be) visually completely identical for the user - at least on modern computers with decent unicode fonts.
The largest problem with UTF-8 (and its biggest strength) is how similar it is to ASCII. It is for this reason we should consider emoji to be a wonderful gift to software correctness everywhere. Correctly handling emoji requires that your software can handle unicode correctly - because they need multi-unit encoding with both UTF-16 and UTF-8. And emoji won't render correctly unless your software can also handle grapheme clusters.
> When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.
Check! If your application deals with text, throw your favorite multi-codepoint emoji into your unit testing data. (Mine is the polar bear). Users love emoji, and your software should handle it correctly. There's no excuse! Even the windows filesystem passes this test today.
> your software should handle it correctly. There's no excuse!
It is valid for the presentation of compound emoji can fallback to their component parts. You can't expect every platform to have an up to date database of every novel combination. A better test is emoji with color modifiers. Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix.
> You can't expect every platform to have an up to date database of every novel combination.
On modern desktop OSes and smart phones, I do expect my platform to have an up-to-date unicode database & font set. Certainly for something like the unicode polar bear, which was added in 2020. I'll begrudgingly look the other way for terminals, embedded systems and maybe video games... but generally it should just work everywhere.
Server code generally shouldn't interact with unicode grapheme clusters at all. I'm struggling to think of any common, valid reason to use a unicode character database in 'normal' backend server code.
> Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix.
I didn't know about that one. I'll have to try it out.
I thing being continuously updated should be tied to receiving new external content.
It's fine to have an embedded device that's never updated, but never receives new content - it doesn't matter that a system won't be able to show a new emoji because it doesn't have any content that uses that new emoji.
However, if it is expected to display new and updated content from the internet, then the system itself has to be able to get updated and actually get updated, there's no acceptable excuses for that - if it's going to pull new content, it must also pull new updates for itself.
As the user/owner of the device, no thanks. It should have code updates if and only if I ask it to, which I probably won't unless you have some compelling reason. For the device owner, pulling new updates by itself is just a built in backdoor/RCE exploit, and in practice those backdoors are often used maliciously. I'd much rather my devices have no way to update and boot from ROM.
The fact that we have to go as far back as CD players for a decent example illustrates my point - the "CD player" content distribution model is long dead, effectively nobody sells CD players or devices like CD players, effectively nobody distributes digital content on CDs or things like CDs (like, CD sales are even below vinyl sales) - almost every currently existing product receives content through a channel where updates could trivially be sent as well.
And if we're talking about how new products should be designed, then the "almost" goes away and they 100% wouldn't receive new content through anything like CDs, the issue transforms from an irrelevant niche (like CDs nowadays) to a nonexistent one.
> Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix.
I despise that Unicode retroactively applied default emoji presentation to existing symbols, breaking old text. Who the hell though that was a good idea?
Those are ok, but both of those emoji are represented as a single unicode codepoint. Some bugs (particularly UI bugs) only show up when multiple unicode characters combine to form a single grapheme cluster. I'd recommend something fancier.
I just tried it in gnome-terminal, and while the crying emoji works fine, polar bear or a country flag causes weird issues.
My native language uses some additional CJK chars on plane 2, and before ~2010s a lot of software had glitches beyond the basic plane of unicode. I am forever grateful for the "Gen Z" who pushed for Emojis.
Javascript's String.length is still semantically broken though. Too bad it's part of a unchangeable spec...
Yep. I don't use eslint, but if I did I would want a lint against any use of string.length. Its almost never what you want. Especially now that javascript supports unicode through [...str].
JavaScript’s recently added implementation of String[Symbol.iterator] iterates through Unicode characters. So for example, [...str] will split any string into a list of Unicode scalar values.
String.length is fine, since it counts UTF16 (UCS2?) code units. The attribute was only accidentally useful for telling how many characters were in a string for a long time, so people think it should work that way.
There's no definition of String.length that would be the obvious right choice. It depends on the use case. So probably better to let the application provide its own implementation.
> So probably better to let the application provide its own implementation.
I’d be very happy with the standard library providing multiple “length” functions for strings. Generally I want three:
- Length in bytes of the utf-8 encoded form. Eg useful for http’s content-length field.
- Number of Unicode codepoints in the text. This is useful for cursor positions, CRDT work, and some other stuff.
- Number of grapheme clusters in the text when displayed.
These should all be reasonably easy to query. But they’re all different functions. They just so happen to return the same result on (most) ascii text. (I’m not sure how many grapheme clusters \0 or a bell is).
Javascript’s string.length is particularly useless because it isn’t even any of the above methods. It returns the number of bytes needed to encode the string as UTF16, divided by 2. I’ve never wanted to know that. It’s a totally useless measure. Deceptively useless, because it’s right there and it works fine so long as your strings only ever contain ascii. Last I checked, C# and Java strings have the same bug.
The built-in string.length method is useless (it returns the number of char objects) and I agree that's a problem, but the solution is also built into the language, unlike in JS.
JS these days also has ways to iterate over codepoints and grapheme clusters. If you treat the string as an iterator, then its elements will be single-codepoint strings, on which you can call .codePointAt(0) to get the values. (The major JS engines can allegedly elide the allocations for this.) The codepoint count can be obtained most simply with [...string].length, or more efficiently by looping over the iterator manually.
The Intl.Segmenter API [0] can similarly yield iterable objects with all the grapheme clusters of a string. Also, the TextEncoder [1] and TextDecoder [2] APIs can be used to convert strings to and from UTF-8 byte arrays.
Possibly because the major software companies made it work on phones. Thus users saw it working on many apps and complain when your app fails to do this.
Google, Apple, IBM and MS also did a lot of localisation so their code bases deal with encoding.
It is FOSS Unix software that had the ASCII mindset, probably as C and C++ string types are ASCII and many programmers want to treat strings as arrays. The MacOS and Windows APIs do take UTF as their input not char * (agreed earlier versions did not put they have provided the UTF encodings for 25 years at least.
> I've never run into a problem where the difference between NFC and NFD mattered. (Do you have an example?)
The main place I've seen it get annoying is searching for some text in some other text. Unless you normalize the data you're searching through the same way as you normalize your search string.
After reading the above comment I went looking for Unicode documentation talking about the different normalisation formats. That was a point I read that surprised me because I hadn’t ever thought of it. They said search should be insensitive to normalisation form - so generally you should normalise all text before running a search.
That’s a great tip - obvious in hindsight but one I’d never considered.
We make some B2B software running on Windows, integrating with customer systems. We get a lot of interesting files.
About a decade ago I wrote some utility code for reading files, where it'll try to detect BOM first, if not scan for invalid UTF-8 sequences. If none are found assume UTF-8 else assume Windows-1252. Worked well for us so far.
Still get the occasional flat file in Windows-1252 with one random field containing UTF-8, so some special handling is needed for those cases. But that's rare.
Fortunately we don't have to worry about normalization for the most part. If we're parsing then any delimiters will be one of the usual suspects and the rest data.
Older versions of mac did enforce NFD for file names, but more recent names don't, at least at the OS level. But many apple programs, such as finder _will_ use NFD. Except that it isn't even Unicode standardized NFD, it is Apple's own modified version of it. And this can cause issues when for example you create a file in finder, then search for it using `find`, and type the name of the file the exact same way, but it can't find the file because find got an NFC form, but the actual file is in NFD.
OTOH, in many applications, you don't really care about the normalization form used. For example, if you are parsing a CSV, you probably don't need to worry about if one of the cells using using a single code point or two code points to represent that accented e.
Thanks, yet another quantum of knowledge that makes one's life irreversibly ever so slightly worse. But not as bad as encryption (and learning all the terrible ways most applications have broken implementations in)
I got a ticket about some "folders with Chinese characters" showing up on an SMB share at work, my first thought was a Unicode issue and sure enough when you combine two UTF-8/ASCII A-z code points together as one UTF-16 code point, it will usually wind up in the CJK Common Ideographs range of Unicode. Some crappy software had evidently bypassed the appropriate Windows APIs and just directly wrote a C-style ASCII string onto the filesystem without realizing that NTFS is UTF-16.
Microsoft Windows is a source of many a headache for me as almost every other client I write code for has to deal with data created by humans using MS Office. Ordinary users could be excused, because they are not devs but even devs don't see a difference between ASCII and UTF-8 and continue to write code today as if it was 1986 and nobody needed to support accented characters.
Do you know of a resource that explains character encoding in greater detail? Just for my own curiosity. I am learning web development and boy, they brow beat UTF-8 upon us which okay, I'll make sure that call is in my meta data, but none bother to explain how or why we got to that point, or why it seems so splintered.
This Joel On Software article [0] is a good starting point. Incredibly it's now over 20 years old so that makes me feel ancient! But still relevant today.
The suggestion that the web should just use utf-8 everywhere is largely true today. But we still have to interact with other software that may not use utf-8 for various legacy reasons - the CSV file example in the original article is a good example. Joel's article also mentions the solution discussed in the original article, i.e. use heuristics to deduce the encoding.
Some [1] may also consider working for any company/app that needs to display an emoji, to be a waste of at least one life (your life, and all your users' lives).
Not every encoding can make a round trip through Unicode without you writing ad hoc handling code for every single one. There's a number of reasons some of these are still in use and Unicode destroying information is one of them.
Non-technical users don't want to do that, and won't understand any of that. That's the unfortunate reality of developing software for people.
If Excel generates CSV files with some Windows-1234 encoding, then my "import data from CSV" function needs to handle that, in one way or another. A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding, they won't care that Microsoft is using obsolete or weird defaults. They will see it as a bug in my program and demand that I fix my software. Even if Excel offers them a choice in encoding, they won't understand any of that and more importantly they don't want to deal with that right now, they just want the thing to work.
>A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding...
They’ll probably switch to the application that does it for them and just works, instead of the one telling them to do something they don’t really understand.
Selecting UTF-8 from a dropdown on the export doesn't seem too onerous an ask. If that's the differentiator between you and your competitors, then you might have bigger problems.
I have sat in on and observed many in-person usability sessions with various applications and websites.
In can tell you right now anyone reading this site is in a completely different universe of tech competency than the general public, and even professionals who aren’t tech-focused.
You would have lost many of them simply with then jumble of letters “UTF-8”.
Yeah, I started writing software a "while" ago. I've encountered a handful of users (and charsets) in my time.
With CSV exports, there's already a good bit of training you have to do WRT the column layout, file format, headers (or not), cell value formats, etc. There's far more lift involved in training users there than ensuring they select "UTF-8" when they select CSV.
And, there's really nothing technical about it, as they don't need to understand what "UTF-8" actually means any more than they need to understand what "CSV" stand for. It's a simple ask in a list of asks.
I weigh this against having every developer now believe they have to check/convert every character set, which can be unreliable and produce garbage, BTW. And, speaking of garbage, there are some encodings that you won't be able to convert in any case, so it would be technically impossible to preserve data integrity without pushing some requirement on the source up to the user.
So, it's about tradeoffs. And, again, if asking users to choose UTF-8 is the difference between customers choosing your app or a competitor's, then you probably need to be more worried about that than your charset encoding.
Hell we just accept xlsx and xls natively. It sucks but it solves many issues. Also causes some, but you'll never get it out of users' heads that Excel files are not a data exchange format.
Definitely helps. Should be able to detect the charset from the Excel metadata, at least.
Of course, if that charset uses code points not available in your app's native charset, then you're kind of back to square one (unless your use case tolerates garbled or missing data).
I hope someday, with improved public education, we can have better users. Unlikely with Gen Z apparently having worse computer skills than the previous generation..
I think most people would rather get in their car, drive to the ocean, board a ship and join the navy, and come back after a few years abroad - than following some instructions on how to use the computer.
Pretty funny, and probably some truth to that being a user sentiment.
OTOH, they have to follow some process to use the software. For just the CSV export, they already have to ensure column orders, values, formats, maybe headers, etc. Selecting UTF-8 from a dropdown seems like the easy part.
Joining the navy and then be obliged to follow instructions on so many things you'd normally never even be ordered to do (because now you're in the Navy, sailor! and there are three ways to do any thing: the right way, the wrong way, and the Navy way, and guess which way we use here in the Navy?), just to not follow some instructions on how to use the computer... is something I can actually imagine some people doing.
Could this work? Implement handling of ancient excel files in your SaaS product, but charge extra dollar for parsing legacy formats and provide information how to export correct files from excel next time :)
Plus, Excel really likes to use semicolons instead of commas for comma-separated-files. That's another idiosyncrasy that programmers need to take into account.
I'm German and I hate this. The icing on top: Excel separates function arguments with a semicolon in German locale, too. Got me some head scratching. Examples are separated with comma, iirc.
> then my "import data from CSV" function needs to handle that, in one way or another.
It doesn't. Well, maybe "another".
Your function or even app doesn't need to handle it. Here's what we did on a bookkeeping app: remove all the heuristics, edge cases, broken-csv-handling and validation from all the CVS ingress points.
Add one service that did one thing: receive a CSV, and normalize it. All ingress points could now assume a clean, valid, UTF8 input.
It removed thousands of LOCs, hundreds of issues from the backlog. It made debugging easy and it greatly helped the clients even.
At some point, we offered their import runs for download, we added the [original name]_clean.csv our normalized versions. Got praise for that. Clients loved that, as they were often well aware of their internal mess of broken CVSs.
The point is to separate the cleaning stage from the import stage. Having a clean utf-8 csv makes debugging import issues so much easier. And there are several well working csv tools such as the Python one, that can not only detect character encodings but also record separators and various quotation idiosyncrasies that you also need to be aware of when dealing with Microsoft Office files. Other people have already thought long and hard about that stuff so you don't have to.
> We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.
Browsers used to have a menu option to choose the encoding you wanted to use to decode the page.
In Firefox, that's been replaced by the magic option "Repair Text Encoding". There is no justification for this.
They seem to be in the process of disabling that option too:
> Note: On most modern pages, the Repair Text Encoding menu item will be greyed-out because character encoding changes are not supported.
Probably because most websites now send a correct encoding header or meta Tag, so the user changing can only make it wrong. (Assuming no encoding header is wrong, which happens in reality)
It does happen a lot that old text content in non-UTF-8 encoding is mistakenly served explicitly marked as UTF-8. It is precisely in such circumstances that the Repair Text option is useful.
> Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden). With the current approach, the parser needs to know of one flag to force chardetng, which the parser has to be able to run in other situations anyway, to run.
> Elaborate UI surface for a niche feature risks the whole feature getting removed
> Telemetry [...] suggested that users aren’t that good at choosing correctly manually.
In other words, it's trying to protect users from themselves by dumbing down the browser. (Never mind that people who know what they are doing have probably also turned off telemetry...)
>> Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden).
There's no explanation of why you'd want this, or why it's security-relevant.
(Farther down, there's a mention of self-XSS, which definitely isn't relevant.)
>> Elaborate UI surface for a niche feature risks the whole feature getting removed
They've already removed the whole feature. That was easier to do after they mostly disabled it, not harder.
>> Telemetry showed users making a selection from the menu when the encoding of the page being overridden had come from a previous selection from the menu.
That would be an example of "working as expected". The removal of the ability to do this is the problem that disabling the encoding menu causes! Under the old, correct approach, you'd guess what the encoding was until you got it right. Under the new approach, the browser guesses for you, and if the first guess is wrong, screw you.
Do we really need 128 permutations just to express an alphabet of 26 letters?
I think we should use a 4 bit encoding.
0 - NUL
1-7 - aeiouwy
8 - space
9-12 - rst
13-15 - modifiers
When modifier bits are set, the values of the next half-byte change to represent the rest of the alphabet, numbers, symbols, etc. depending on the bits set.
As someone who has repeatedly had to deal with Unicode nonsense; I wholeheartedly agree. Also, you don’t need accents. You just need to know how to read and have context. See: live and live, read and read, etc.
Anyone got EBCDIC on their bingo cards? Because if the argument is "legacy encodings are still relevant in 2024" then we also need to bring EBCDIC (and EBCDIK and UTF-EBCDIC for more perverted fun) into the picture. Makes heuristics extra fun.
Or, you know, just say "nah, I can, those ancient stuff don't matter (outside of obligatory exceptions, like software archeology) anymore." If someone wants to feed me a KOI8-R or JIS X 0201 CSV heirloom, they should convert it into something modern first.
> Anyone got EBCDIC on their bingo cards? Because if the argument is "legacy encodings are still relevant in 2024"
I have a hobby interest in IBM mainframes and IBM i, so yes to EBCDIC for me. (I have encountered them professionally too, but only to a very limited extent.) In practice, I find looking for 0x40 (EBCDIC space) a useful heuristic. Even in binary files, since many mainframe data structures are fixed length space padded.
> then we also need to bring EBCDIC (and EBCDIK and UTF-EBCDIC for more perverted fun) into the picture. Makes heuristics extra fun.
Actual use of UTF-EBCDIC, while not nonexistent, has always been extremely rare. A person could spend an entire career dedicated to IBM mainframes and never encounter it
EBCDIK, at first I wondered if that was a joke, now I realise it is a name used for non-IBM Japanese EBCDIC code pages. Again, something one can spend a whole career in mainframes and never encounter – if one never works in Japan, if one works for a vendor whose products aren't sold in Japan, probably even if you work for a vendor whose products are sold in Japan but only to IBM sites (as opposed to Fujitsu/Hitachi sites)
I require UTF-8. If it isn't currently UTF-8, it's someone else's problem to transform it to UTF-8 first. If they haven't, and I get non-UTF-8 input, I'm fine bailing on that with a "malformed input - please correct" error.
Many Japanese websites have migrated from Shift-JIS to UTF-8, but this still ignores the fact that e.g. television captioning uses special characters[1] that are not found in UTF-8 or Shift-JIS. Windows itself has a habit of using its own Windows-932 encoding, which frequently causes problems in the Unix software I use. (e.g. Emacs fails at auto-detecting this format, and instructing Emacs to use Shift-JIS will result in decoding issues)
Windows-932 is the text encoding I dread the most. I wish literally everything could use Unicode, but we're not quite there yet. There's a reason encoding-japanese[1] on NPM has nearly 1 million weekly downloads. Unfortunately, I have to use it in one of my React Native applications since the one of servers I speak to returns text encoded as Shift-JIS and Hermes does not implement JavaScript's TextDecoder API. Shift-JIS is marginally better than the Windows code page, but I'd really prefer UTF-8.
More or less. The proportion of Japanese websites that use Shift-JIS is actually increasing, for example. (It's true that the absolute number of Japanese websites using UTF-8 is increasing, but that's misleading - it's only due to the overall growth of the web).
Any of the well-known han unification examples. People claim this can be solved with some kind of out-of-band font selection vaporware, but the kind of person who thinks it's fine to demand UTF-8 everywhere never actually stoops to implement this imaginary font selection functionality.
> implement this imaginary font selection functionality
We are already doing GSUB/GDEF tables and a lot of horrible stuff to display pretty glyphs on the screen. Hell, there are literal instructions for VM in TTF files to help with pretty rasterization on low-res screens.
Making font rendering library is hard, Nintendo hard.
That's just a nature of the beast. Fonts are messy and if we want one standard to deal with it once and for all, that means some compromises must be done. CJK is not some obscure alphabet. In this case, perfect is enemy of good.
I am just glad there really is only one (non-niche) standard, obligatory https://xkcd.com/927/
Right. One would think that misrendering a major world language used by over 100 million people would be an issue that warrants some attention. But too many of the HN crowd don't care.
> In this case, perfect is enemy of good.
It's not though. Unicode-only is not just imperfect, it's an outright regression for Japanese. Meanwhile traditional encoding-aware programs render Japanese just fine.
> It's not though. Unicode-only is not just imperfect, it's an outright regression for Japanese.
Unicode has variant selectors which can deal with region variants (e.g. https://imgur.com/a/syMcWNO) to deal with the issue.
Granted, it's not very widely used, but Unicode provided a solution. The onus is now on application developers and font providers.
Choosing to limit number of characters to 2 bytes was a good technical choice back in 1992, non-unified CJK wouldn't fit (that was a time of 2-4MB RAM and 100-200MB HDD). A solution to problem was later provided.
Using Shift-JIS is like doing on-IP, non-UDP protocol on the internet.
Sure, you can do it, but it's just non-perspective choice. A better invested time is on improving standard protocol, which is Unicode (e.g. conversion software).
> Unicode has variant selectors which can deal with region variants (e.g. https://imgur.com/a/syMcWNO) to deal with the issue.
They're officially deprecated and cause issues with breaking search etc.. So they're still below feature parity with using a traditional encoding. Traditional encoding support is also easier to test since documents with traditional encodings are more widespread and exist for many world languages, not just Japanese.
Well, I must nitpick: Shift JIS is actually just one of those out-of-band vaporware methods. Sure, it's better supported because of legacy, but new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either.
Of course, there is no correct solution (people in other comments seem to believe it exists a bit too quickly). Dynamic HTML page can load pieces of text written by people all across the world, it can't rely on any “main” language. Those people can't be automatically classified (based on some browser settings or location). Then there are always people who know both Chinese and Japanese (while using some other system locale, to make things more complex). It is wrong to assume that they should not be able to use Chinese forms and Japanese forms at once, even inside a single paragraph or phrase.
I wonder why Unicode has not simply introduced some “combining language form character” to make user choice stick. After all, there's a whole subsystem for emoji modification, and those things were once “weird Japanese texting customs”. As for complexity of handling Unicode text, it asymptotically reaches its maximum anyway.
Wait a second, there are “variation selectors” and some “Ideographic Variation Database”. Is it the solution? Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch…
> Shift JIS is actually just one of those out-of-band vaporware methods.
It's the opposite of vaporware; there's a whole bunch of well-known software that handles encodings correctly.
> new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either.
If you support encodings the way almost every programming language tells you too, you'll handle Shift-JIS just fine. If you use the "legacy" encodings approach and test even one non-English language you'll handle Japanese fine.
> Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch…
They're officially deprecated, IMEs and converters don't use them, naïve search implementations (which is to say most search implementations, because people do not test for an obscure edge case) break, ....
But there is no “naive” implementation of Unicode. It won't handle emoji, normalization, and thousands other things. People use The Library anyway. Adding another mutagen to it should be no different from the rest.
As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. A particularly evil solution would choose arbitrary existing combining characters that can't be declared “illegal” retroactively.
By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
> But there is no “naive” implementation of Unicode.
I assure you there are thousands if not millions of apps that implement search the naive way, by comparing bytes.
> It won't handle emoji, normalization, and thousands other things.
Yeah. But it works well enough for Americans, so the maintainers don't care enough to fix it.
> As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack.
And then what? What do you do with documents in Japanese that have already been converted without using your new codepoints? What do you do with programs that ignore your new standard? What do you do about search being broken in most apps with no prospect of fixing it? There are already some PUA codepoints for a proper implementation of Japanese, but most app authors don't even understand the problem, never mind being willing to support something that's "non-standard". Asking them to support traditional non-unicode encodings, which is something that's at least relatively well-known and standardised, and something they can test without knowing Japanese, is much easier.
> By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
No, not for arbitrary languages. If you want to mix arbitrary languages, you need some structure that's a rope of strings with an encoding per segment. But that's exactly the same thing that the unicode advocates claim is easy ("just have a sequence of spans with different lang tags"), and at least if you use traditional encodings then you don't have to do any such fancy business for the much more common case of a file that's entirely in Japanese (perhaps with some ASCII keywords).
The parent said "displayed correctly", not "encoded".
For example, if I want to talk about the fairly rare japanese name '刃一' (jinichi), there's a chance your computer displays it correctly, but there's also a chance your computer displays the chinese variant of the first character, making it look wrong. It's basically up to font-choice of your computer.
The "correct" way to fix that would be for me to be able to tag it with 'lang=ja', but Hacker News doesn't let me include html tags or some other 'language selector' in my comment, so I'm unable to indicate whether that's supposed to be the chinese or japanese variant of the character.
Most unicode text files don't have extra metadata indicating if a certain bit of text is japanese or chinese, so displaying it correctly by adding the correct 'lang' tag is impossible, especially since it's perfectly possible for one utf-8 text to mix both chinese and japanese.
The Shift-JIS codepoints for the characters of that name are understood to refer to Japanese characters, so fonts render them correctly. Encoding-aware programs have different representations (such as GB 18030 codepoints) for the similar-but-different Chinese characters that unicode-only programs tend to display these characters as, and so will render them differently.
If I understand you correctly, it means that the encoding itself serves as the metadata that indicates Chinese/Japanese. In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters?
> If I understand you correctly, it means that the encoding itself serves as the metadata that indicates Chinese/Japanese.
Only if you have some unicodebrained mentality where you consider a Chinese character that looks sort of similar to a Japanese character to be "the same". If you think of them as two different characters then they're just different characters, which may or may not be present in particular encodings (which is completely normal if you make a program that handles multiple encodings: not every character exists in every encoding)
> In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters?
Firstly, you have to "draw the rest of the fucking owl" and actually implement that language selection mechanism. Secondly, if you implement some clever extension mechanism on top of UTF-8 that's only needed for Japanese and can only really be tested by people who read Chinese or Japanese, realistically even if you implement it perfectly, most app makers won't use it or will use it wrong. Whereas if you implement encoding-awareness in the standard way that we've been doing for decades and test with even one non-default encoding, your application will most likely work fine for Japanese, Chinese and every other language even if you never test it with Japanese.
I didn't propose any solution or claim Shift-JIS fixes this. It doesn't really since a single Shift-JIS document can only encode the japanese variant, not both.
However, a unicode codepoint which acted as a "language hint" switch would solve this, and wouldn't require doubling the number of han codepoints.
There already are unicode variant selectors, but iiuc they only apply to a single character at a time, and no one actually uses them, so they're not very useful.
If the Unicode consortium haven't been able to come up with a way of encoding their name correctly, I don't see what hope I have of doing so.
Bonus - as soon as the Unicode consortium do find a way, my software should be able to handle it with no further changes. Well, it might need a recompile against a newer `libicu` as I don't think they maintain ABI backcompat between versions. But there's not much I can do about that.
Unicode can't do it without a breaking change. But if you support non-unicode encodings in the traditional, documented and standard way then your application will handle it fine. Who knows, one day a successor to unicode may come out that can handle all languages properly within a single encoding, in which case an application written to be encoding-aware will support it without even a recompile.
Are there Japanese characters missing in UTF-8? They should be added ASAP.
I know there's a weird Chinese/Japanese encoding problem where characters that kind-of look alike have the same character id, and the font file is responsible for disambiguation (terrible for multi-language content and we should really add more characters to create versions for each, but still the best we have).
IMHO the Unicode Consortium should standardize on using a variant selector for switching between Chinese and Japanese variants of unified Han characters. Best of both worlds: language-independent specificity while keeping the ability to have both in the same document.
Unless you're using an OS older than Windows 2000, or a linux distro from the 2000s, where some form of Unicode was not the default encoding, or maybe an ancient Win32 program compiled without "UNICODE" defined, it shouldn't be a problem. I specifically work with a lot of Japanese software and have not seen this problem in many years.
And even back in the mid 2000s, the only real problems I saw otherwise, were things like malformed html pages that assumed a specific encoding that they wouldn't tell you, or an MP3 file with an ID3 tag with CP932 shoved into it against the (v1) spec.
I also disagree with the author that Shift-JIS can be "good enough" hueristically detected due to its use of both 7 and 8-bit characters in both the high and low bytes to mean different things depending on what character is actually intended. Even string searching requires a complex custom-made version just for Shift-JIS handling.
What a bad, hyperbolic take. UTF-8 can encode the entire Unicode space. All you need is up-to-date libraries and fonts to display the codepoints correctly. It is backwards compatible forever. So requiring UTF-8 allows Japanese to represent their writing method exactly how it is and keep the scheme for a very long time with room to improve.
Japanese text in uft-8 is frequently rendered with the Chinese version of the kanji due to han unification, not "representing their writing method exactly". Shift-JIS encoding comunicates that the text is in Japanese via the encoding facilitating the correct font to be selected.
And it indeed facilitates, as in practice it works better than encoding in utf-8 that lacks an in-band way to communicate that, and out-of-band often fails/is ignored/doesn't exist.
My understanding is Unicode (and therefore UTF-8) can encode all the codepoints encodable by Shift JIS. I know that you need a language context to properly display the codepoints that have been Han Unified, so that could lead to display problems. But if we're trying to properly display a Japanese name, it's probably easier to put the appropriate language context in a UTF-8 document than it is to embed Shift JIS text into a UTF-8 document.
Realistically --- if someone hands me well marked Shift JIS content, I'm just going to reencode it as UTF-8 anyway... And if they hand me unmarked Shift JIS content, I'll try to see if I can decode it as UTF-8 and throw it away as invalid if not.
> if we're trying to properly display a Japanese name, it's probably easier to put the appropriate language context in a UTF-8 document than it is to embed Shift JIS text into a UTF-8 document.
You'd think that, but in practice I've found the opposite. Applications that use encodings managed to display things properly. Applications that hardcode UTF-8 don't.
> My understanding is Unicode (and therefore UTF-8) can encode all the codepoints encodable by Shift JIS
Trivia: There are some variants of ShiftJIS where this isn't entirely true. The traditional yuugen gaisha symbol, for example, which is analogous to U+32CF (LTD), is not supported. The /VENDORS/APPLE/JAPANESE.TXT file uses a PUA designator and then a sequence of four Unicode code points to convert it.
I get that you're referring to Han Unification, but if software doesn't display unified glyphs with the correct style, that's an issue with the font rendering system, not Unicode. Sure, the font rendering system's job may have been easier had Unicode made different choices, but encoding-wise, Unicode is no more ambiguous than any other encoding it's round trip compatible with. The font rendering system is free to assume all unified glyphs should be rendered in a Japanese style, just like it would have with a Japanese-centric encoding.
> Switching your entire encoding system to set a different font is by far the stupidest way to do it.
If it's stupid and it works, it's not stupid. I wish there were other reliable ways to have international programs display Japanese correctly, but there aren't.
To be clear, we're talking about a program/library that can handle both unicode and shift-JIS, and it will render a character that unicode considers identical in different ways depending on what encoding you loaded the character from, right?
Firefox (not the best example for a number of reasons, if you're going to follow up then I'll talk about a different one, but if you really want just one then it's the program I have to hand right now).
> To be clear, we're talking about a program/library that can handle both unicode and shift-JIS, and it will render a character that unicode considers identical in different ways depending on what encoding you loaded the character from, right?
Though if you're making any attempt to use valid tags you'll have <html lang="ja">, and that solves the problem for the web context at least as far as my testing goes.
> Though if you're making any attempt to use valid tags you'll have <html lang="ja">
Right, which is why that's a bad example. But it's the same for most "normal" applications (even e.g. text editors, or I remember hitting this kind of thing in WinRAR), and a lot of the time there isn't a standard way of indicating the language/locale in the file. Even within firefox, there are (admittedly rare) cases where you're viewing something that isn't HTML and doesn't have HTTP headers so using the encoding or manually setting the page language is still the only way to make it work - and applications that have a manual "file language" setting are the exception rather than the rule.
I'm no encoding geek, but have I'd say more than a passing familiarity with the issues involved. But I didn't know UTF-8 had in-band language signaling until today, so it perhaps doesn't surprise me that many applications implement it. (UI toolkits should, though... there's kinda no excuse for that.)
If you implement any kind of encoding support (that is, any kind of support for non-ASCII/non-unicode) you will probably have working Shift-JIS support even if you never test it, because Shift-JIS works the same as every other encoding you might test with. If you tested French or Spanish or really anything that wasn't English, you will display Japanese fine.
If you implement only unicode then you put yourself in a situation where Japanese is uniquely different from every other language, and your program will not work properly for Japanese unless you tested Japanese specifically.
This wouldn't solve your original problem since UTF8 also popular with Japanese, so just adding Shift-JIS isn't enough, so it comes down to the same basic thing: to support Japanese, you have to do some extra work to get and use extra info about a language, which is also possible within UTF8 counter to your initial broad claim of the opposite
> This wouldn't solve your original problem since UTF8 also popular with Japanese
Only among people who don't care. Implementing encoding support means your app supports a way to display Japanese properly. If you want to add more ways to display Japanese properly, go ahead, but that's supererogatory, whereas UTF8-only apps don't have a way to display Japanese properly at all.
> to support Japanese, you have to do some extra work to get and use extra info about a language, which is also possible within UTF8
It's not. There's some imaginary theorycrafted way in which it might notionally be possible within UTF-8, but not one UTF-8-only app has ever actually implemented support for displaying Japanese properly. The only approach to displaying Japanese properly that has ever actually been implemented in reality is to support multiple encodings (or to support only a Japanese encoding, but that has obvious downsides), and if you make your app encoding-aware then that's enough, you don't have to do anything else to be able to display Japanese properly for people who care about displaying Japanese properly (it's always possible for an app maker to go above and beyond, but I'm sure you'll agree the big difference is between an app that has a way to display proper Japanese at all and one that does not).
That works until you can't pay your bills unless you take a new contract where you have to deal with a large amount of historical text files from various sources.
Even then, at that point I'm writing an adapter to convert into UTF-8 JIT in front of my service. Or modifying my program when that contract comes up, but I'm not going to waste time proactively safeguarding against a what-if that may come up at some point.
> I'm not going to waste time proactively safeguarding against a what-if
The article doesn't say that you should. It clearly states that for many cases, the input format is known or explicitly stated in the input headers.
The article talks about cases where the input files are in an unknown input format. Even then, it states: "Perhaps there is a case to be made that csvbase's auto-detection should be a more explicit user interface action than a pre-selected combo-box."
But for the case where the requirements call for heuristics, the article then talks about how that can be done.
> at that point I'm writing an adapter to convert into UTF-8 JIT in front of my service.
And at that point you might need the advice in the article, right?
> at that point I'm writing an adapter to convert into UTF-8 JIT in front of my service
Right, and at that point you're probably gonna need these statistics-based heuristics to write that adapter. Unless you know specifically what other encoding each bit of input is. If you do, then, again, you are not the target audience for this article.
Better to assume UTF8 and fail with a clear message/warning. Sure you can offer to guess to help the end user if it fails, but as other people have pointed out, it’s been standard for a long time now. Even python caved and accepted it as the default: https://peps.python.org/pep-0686/
I wish I could live in the world where I could bluntly say "I will assume UTF-8 and ignore the rest of the world". Many Japanese documents and sites still use Shift JIS. Windows has this strange Windows-932 format that you will frequently encounter in CUE files outputted by some CD ripping software. ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208. These special characters are mostly icons used in traffic and weather reports, but transcoding to UTF-8 still causes trouble with these icons.
Most people on this site probably live in the world where everything is done in English. That's the norm for the vast majority of businesses and people in the US.
Even for those people, there is still a ton of old text files in Windows-1252 etc floating around.
You can choose to never work on projects where you have to deal with files like that.
But there may come a day where you have to choose between not paying your rent or writing a tool that converts old textfiles to UTF-8. At that point, it's nice to have references on the internet on how other people have actually dealt with it and what works. "Abort with an error" is not very useful advice then.
Why would you write a tool that does that instead of just digging up one that's already written? This sounds like the folly of writing ones own encryption library.
As far as implementing new tech goes, this sounds like about the easiest research project you could ask for. Grab a few and test em out. Not sure why you're making this out to be some kind of intractable problem.
I do. I cut my IT teeth on an old ass system from the 80s in the 00's. I remember having problems feeding the reports into modern systems. Goofy problems with eol and eof and some other hiccups. It wasn't that bad.
Honestly if you can't review/read/figure that out without writing a library of your own, you probably shouldn't be writing a library of your own in the first place.
You can't pick and use a library like this without understanding the underlying concepts. That goes for both encryption and the charset conversion issue. It's not always just plug and play.
There are examples of where people used encryption libraries in the wrong way and undermined the strength of the encryption (for example, CVE-2024-31497 in PuTTY).
A very big part of software development is dealing with leaky abstractions. We don't work with perfect black boxes. We need to understand enough of how things works in the lower layers to avoid problems. Note here that I wrote "enough", not "everything", or "write everything yourself".
I would not want a person writing software to handle charset conversion if he refuses to learn how the various encodings work, which charset will decode as another charset or not, etc.
Your example seems to be due to, paraphrasing, lack of a library rather than inability to choose one.
"older approach, PuTTY developers said, was devised at a time when Microsoft Windows lacked native support for a cryptographic random number generator."
So "enough of how things work" could just be "pick a modern encryption library" that doesn't come from the dark ages when t there were no random numbers
Same with encodings, it requires a much lower level understanding to pick a library, you can rely on the expertise of others
It's more like "I will assume UTF-8 and ignore edge case encoding problems which still arise in Japan, for some strange reason".
We are not running short on Unicode codepoints. I'm sure they can spare a few more to cover the Japanese characters and icons which invariably get mentioned any time this subject comes up on HN. I don't know why it hasn't happened and I won't be making it my problem to solve. Best I can do is update to version 16 when it's released.
I mention Japanese because I deal with Japanese text daily. I could mention some Chinese documents and sites using GBK to save space (since such encodings use exactly 2 bytes per character whereas the average size in UTF-8 is strictly larger than 2 bytes). But I am not very familiar with it. Overall, I would not say these are "strange reasons".
Other encodings exist, yes. But they can all be mapped to UTF-8 without loss of information[0]. If someone wants to save space, they should use compression, which will reduce any information, regardless of encoding, to approximately the same size. So it's perfectly reasonable to write software on the assumption that data encoded in some other fashion must be first reëncoded as UTF-8
[0]: Except Japanese, people hasten to inform us every time this comes up. Why? Why haven't your odd characters and icons been added to Unicode, when we have cuneiform? That's the strange part. I don't understand why it's the case.
Unicode did kind of dumb thing with CJK, the unified Chinese and Japanese kanjis makes displaying the CJK text much harder problem than it should be, as it now relies also on a language specific font to be displayed correctly[0]. I guess this could be bandaided by some sort of language marker in the UTF8 bytestring which then a text shaping engine would have to understand and switch the font accordingly..
Kind of a band-aid (it's necessary to stuff a variant selector after a CJK codepoint), but should work.
These decisions were made back in 1992 and codepoint in 16-bit was one of desired goals. Non-unified CJK wouldn't fit. In hindsight, it looks like a rather unfortunate decision, but having more codepoints that would fit to 16 bits could seriously hamper adoption and different standard would win (compute resources were far more limiting back then).
In either case, it's like 4 byte addressing in IPv4, in hindsight, 6+ bytes would be better, but what's done is done.
Edit: Even in 2000s, when C# was released, string was just a sequence of 16-bit code units (not codepoints), so they could deal with BMP without problems and astral planes were ... mostly DIY. They added Rune support (32-bit codepoint) only in .NET Core 3.0 (2019).
They aren't strange, but they are sort of self-inflicted, so it's not unreasonable for others to say, "we're not going to spend time and effort to deal with this mess".
I'm Russian. 20 years ago that meant having to deal with two other common encodings aside from UTF-8 (CP1251 and KOI8-R). 25 years ago, it was three encodings (CP866 was the third one). Tricks like what the article describes were very common. Things broke all the time anyway because heuristics aren't reliable.
These days, everything is in UTF-8, and we're vastly better off for it.
Unless the Unicode Consortium decides to undo the Han Unification stuff, I don't think it's going to get better for Japanese users, and programmers who build for a Japanese audience will have to continue to suffer with Shift-JIS.
There will be no undoing of anything, fortunately. Unicode is committed to complete backward compatibility, to the point where typos in a character name are supplemented with an alias, rather than corrected. Han Unification was an unforced error based on the proposition, which was never workable, that sixteen bits could work for everyone. This is entirely Microsoft's fault, by the way. But it shouldn't be, and won't be, fixed by breaking compatibility. That way lies madness.
There are two additional planes set aside for further Hanzi, the Supplementary and Tertiary Ideographic Planes, the latter is still mostly empty. Eventually the last unique ideograph used only to spell ten known surnames from the 16th century will also be added as a codepoint.
I view the continued use of Shift-JIS in Japan as part of a cultural trend, related to the continued and widespread use of fax machines, or the survival of floppy disks for many years after they were effectively dead everywhere else. That isn't at all intended as an insult, it's that matters Japanese stay within Japan to a high degree. Japanese technology has less outside pressure for cross-compatibility.
Shift-JIS covers all the corner cases of the language, and Unicode has been slow to do likewise, and it isn't like Japanese computers don't understand UTF-8, so people have been slow to switch. It's the premise of "unaware of how it works in the rest of the world" that I object to. It's really just Japan. Everywhere else, including the Chinese speaking parts of the world, there's Unicode data and legacy-encoded data, and the solution to the later is to encode it in the former.
> ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208.
Amazingly enough, ARIB STD-B24 is one of the major source of Unicode emojis. So transcoding would actually work for them! (I am aware of some exceptions, but semantically there is no loss.) Unicode and UTF-8 are truly eating every other legacy encoding else, so much so that it is becoming more reasonable to have a separate transcoding step.
But remember, adding emoji to a character encoding standard is Morally Bad, somehow, and Proof Of Intellectual Decay In The Modern World. Also, Unicode invented emoji to Sap And Impurify Our Bodily Fluids and there is no reason for any of them to exist.
If it's turtles all the way down and at every level you use utf-8, it's hard to see how any input with a different encoding (for the same underlying text) will not be detected before any unintended side effects were invoked.
At this point, I don't see any sufficiently good reason to not use utf-8 exclusively in any new system. Conversions to and from other encodings would only be done at well defined boundaries when I'm calling into dependencies that require non utf-8 input for whatever reason.
If you give me a computer timestamp without a timezone, I can and will assume it's in UTC. It might not be, but if it's not and I process it as though it is, and the sender doesn't like the results, that's on them. I'm willing to spend approximately zero effort trying to guess what nonstandard thing they're trying to send me unless they're paying me or my company a whole lot of money, in which case I'll convert it to UTC upon import and continue on from there.
Same with UTF-8. Life's too short for bothering with anything else today. I'll deal with some weird janky encoding for the right price, but the first thing I'd do is convert it to UTF-8. Damned if I'm going to complicate the innards of my code with special case code paths for non-UTF-8.
If there were some inherent issue with UTF-8 that made it significantly worse than some other encoding for a given task, I'd be sympathetic to that explanation and wouldn't be such a pain in the neck about this. For instance, if it were the case that it did a bad job of encoding Mandarin or Urdu or Xhosa or Persian, and the people who use those languages strongly preferred to use something else, I'd understand. However, I've never heard a viable explanation for not using UTF-8 other than legacy software support, and if you want to continue to use something ancient and weird, it's on you to adapt it to the rest of the world because they're definitely not going to adapt the world to you.
> For instance, if it were the case that it did a bad job of encoding Mandarin
I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.
A key aspect is that nowadays we rarely encode pure text - while other encodings are more efficient for encoding pure Mandarin, nowadays a "Mandarin document" may be an HTML or JSON or XML file where less than half of the characters are from CJK codespace, and the rest come from all the formatting overhead which is in the 7-bit ASCII range, and UTF-8 works great for such combined content.
That doesn't seem like the worst issue imaginable. I doubt there are too many cases where every byte counts, text uses a significant portion of the available space, and compression is unavailable or inefficient. If we were still cramming floppies full of text files destined for very slow computers, that'd be one thing. Web pages full of uncompressed text are still either so small that it's a moot point or so huge with JS, images, and fonts that the relative text size isn't that significant.
Which is all to say that you're right, but I can't imagine that it's more than a theoretical nuisance outside some extremely niche cases.
They shouldn't be non-existent. Zip-then-encrypt is not secure due to information leakage.
EDIT: also, it's not safe—message length is dependent on the values of the plaintext bytes, period. i'm not saying don't live dangerously, i'm just saying live dangerously knowing
The information leakage problem occurs when compression is done in the TLS layer, because then the compression context includes both headers (with cookies) and bodies (containing potentially attacker-controlled data). But if you do compression at the HTTP layer using its Transfer-Encoding then the compression context only covers the body, which is safe.
It can still leak data if attackers can get their input reflected. I.e. I send you a word, and then I get to observe a compressed and encrypted message including my word and sensitive data. If my word matches the sensitive data, the cyphertext will be smaller. Hence I can learn things about the cipgertext. That is no longer good encryption.
What you are talking about is generally referred to as the "BREACH" attack. While there may theoretically be scenarios where it is relavent, in practise it almost never is so the industry has largely decided to ignore it (its important to distinguish this from the CRIME attack which is about http headers instead of the response body which has a much higher liklihood of being exploitable while still being hard).
The reason its usually safe is that to exploit you need:
- a secret inside the html file
- the secret has to stay constant and cannot change (since it is adaptive attack. CSRF tokens and similar things usually change on every request so cannot be attacked)
- the attacker has to have a method to inject something into the html file and repeat it for different payloads
- the attacker has to be able to see how many bytes the response is (or some other side channel)
- the attacker is not one of the ends of the communication (no point to attack yourself)
Having all these requirements met is very unlikely.
For Asian languages, UTF-8 is basically the same size as any other encoding when compressed[0] (and you should be using compression if you care about space) so in practice there is no data size advantage to using non-standard encodings.
In addition, Chinese characters encode more information than English letters, so a text written in Chinese will generally consume fewer bytes than the same text in English even when using UTF-8.
(Consider: Horse is five letters, but 馬 is one character. Even at three bytes per character, Chinese wins.)
Presumably that derives from the overhead of encoding an english character as a full byte? Given there's only 26 characters normally, you could fit that into 5 bits instead, which funnily enough does actually line up with the chinese character encoding (5x5 vs 1x24).
It depends on the domain. If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else.
Unfortunately Google and many other companies have decided UTC is the only way, so this causes issues with ICS files that use that format sometimes when they are generating their helpful popups in the GMail inbox.
> If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else.
If you have to take medication (for instance, an antibiotic) every 24 hours, it must be taken at the same UTC hour, even if you took a train to a town in another timezone. Keeping the same local time even when the timezone changes would be wrong for that use case.
There are very few drugs where that's a requirement. Your kidneys and liver aren't smart enough to metabolize anything at precisely the same rate every day anyway.
If you're there for a while, you'll need to adapt anyway since your biorhythms will too. But there are plenty of other cases like a reminder to check something after dinner, or my standard wake-up alarm in the morning. Or if I plan to travel, book lunch at a nice place for 1pm, and put it in my calendar I just want it to be 1pm wherever I go, without caring about TZ changes.
Calendars, alarms, and reminders have some overlap here and floating time can be good for some cases.
I can't help myself. The grandest of nitpicks is coming your way. I'm sorry.
> If you give me a computer timestamp without a timezone, I can and will assume it's in UTC.
Do you mean, give you an _offset_? `2024-04-29T14:03:06.0000-8:00` the `-8:00` is an offset. It only tells you what time this stamp occurred relative to standard time. It does not tell you anything about the region or zone itself. While I have consumed APIs that give me the timezone context as part of the response, none of them are part of the timestamp itself.
The only time you should assume a timestamp is UTC is if it has the `z` at the end (assuming 8601) or is otherwise marked as UTC. Without that, you have absolutely no information about where or when the time has occurred -- it is local time. And if your software assumes a local timestamp is UTC, then I argue it is not the sender of that timestamp's problem that your software is broken.
My desire to meet you at 4pm has no bearing on if the DST switchover has happened, or my government decides to change the timezone rules, or if {any other way the offset for a zone can change for future or past times}. My reminder to take my medicine at 7pm is not centered on UTC or my physical location on the planet. Its just at 7pm. Every day. If I go from New York to Paris then no, I do not want your software to tell me my medicine is actually supposed to be at Midnight. Its 7pm.
But, assuming you aren't doing any future scheduling, calendar appointments, bookings, ticket sales, transportation departure, human-centric logs, or any of the other ways Local Time is incredibly useful -- ignore away.
It's always nice to see someone who actually understands time.
"Convert to UTC and then throw away the time zone" only works when you need to record a specific moment in time so it's crazy how often it's recommended as the universal solution. It really isn't that hard to store (datetime, zone) and now you're not throwing away information if you ever need to do date math.
Yeah, I've been trying to convince people forever to store time zones with timestamps when appropriate. If you record events from around the world and don't record what time zone they happened in you can't even answer basic questions like "what proportion happened before lunch time?"
People love simple rules and they will absolutely take things too far. Most developers learn "just use UTC!" and think that's the last thing they ever need to learn about time.
As I mentioned in another reply, "remind me every day at 7PM" isn't a timestamp. It's a formula for how to determine when the next timestamp is going to occur. Even those examples are too narrow, because it's really closer to "remind me the next time you notice that it's after 7PM wherever I happen to be, including if that's when I cross a time zone and jump instantly from 6:30PM to 7:30PM".
Consider my statement more in the context of logs of past events. The only time you can reasonably assume a given file is in a particular non-UTC TZ is when it came from a person sitting in your same city, from data they collected manually, and you're confident that person isn't a time geek who uses UTC for everything. Otherwise there's no other sane default when lacking TZ/offset data. (I know they're not the same, but they're similar in the sense that they can let you convert timestamps from one TZ to another).
> As I mentioned in another reply, "remind me every day at 7PM" isn't a timestamp. It's a formula for how to determine when the next timestamp is going to occur. Even those examples are too narrow, because it's really closer to "remind me the next time you notice that it's after 7PM wherever I happen to be, including if that's when I cross a time zone and jump instantly from 6:30PM to 7:30PM".
That's certainly fair in the context of a recurring event with some formula. I caution that a lot of people will still immediately reach for timestamps to calculate that formula, particularly for a next occurrence, and in the context if this conversation they would be given as an ISO8601 datetime based on Local Time. I would also caution that calendar events that have a distinct moment in time that they start are also prime for Local Time where a UTC-default mentality will cause errors.
> Consider my statement more in the context of logs of past events
From the stance of computer generated historical log data, I definitely agree that UTC everywhere is a sane default and safe to assume :)
(And, in your defense, I would definitely argue UTC-everywhere gets you 95% of the way there for 5% of the effort... I get why people make the tradeoff)
> (I know they're not the same, but they're similar in the sense that they can let you convert timestamps from one TZ to another).
More nitpicking on my part, again, I'm sorry, it lets you convert from one _offset_ to another or from an offset to UTC. Think Arizona being a special snowflake who (mostly!) doesn't observe DST. You can't assume all UTC-7 offsets are all Mountain Time.
> For instance, if it were the case that it did a bad job of encoding Mandarin
Please look up the issues caused by Han unification in Unicode. It’s an important reason why the Chinese and Japanese encodings are still used in their respective territories.
I haven't seen discussion of this point yet, but the post completely fails to provide any data to back up its assertion that charset detection heuristics works, because the feedback I've seen from people who actually work with charsets is that it largely doesn't (especially if you're based on naive one-byte frequency analysis). Okay, sure, it works if you want to distinguish between KOI8-R and Windows-1252, but what about Windows-1252 and Windows-1257?
I've done some charset detection, although it's been a while. Heuristics kind of work for somethings --- I'm a big fan of if it's decodable as utf-8, it's probably utf-8, unless there's zero bytes (in most text). If there's a lot of zero bytes, maybe it's UCS-2 or UTF-16, and you can try to figure out the byte order and if it decodes as utf-16.
If it doesn't fit in those categories, you've got a much harder guessing game. But usually you can't actually ask the source what it is, because they probably don't know and might not understand the question or might not be contactable. Usually, you have to guess something, so you may as well take someone else's work to guess, if you don't have better information.
Yeah. The fantastic python library ftfy ("fixes text for you", https://ftfy.readthedocs.io/en/latest/index.html), designed to fix mangled Unicode (mojibake, of many different varieties), mentions in its docs that heuristic encoding guessers are the cause of many of the problems ftfy is designed to fix. It's magical, by the way.
That section explains why not to use a specific naive charset detection library that doesn't have a strong prior for UTF-8. There's no basis for extrapolating that further.
487 comments
[ 2.3 ms ] story [ 386 ms ] threadIf UTF-8 decoding fails, then it's time to pull out the fancy statistical tools to (unreliably) guess an encoding. But that should be a fallback, not the first thing you try.
At which point the message is effectively ASCII. UTF-8 is a superset of ASCII, so "decoding" ASCII as UTF-8 is fine.
(Yes, I know there are some Japanese text encodings where 0x5c is decoded as "¥" instead of "\". But they're sometimes treated as backslashes even though they look like ¥ symbols so handling them "correctly" is complicated.)
[0] https://en.wikipedia.org/wiki/ASCII#7-bit_codes
Don't really even need to do that. There's only a handful other encodings still in common use, just try each of them as fallbacks and see which one works without errors, and you'll manage the vast majority of what's not UTF-8.
(We recently did just that for a system that handles unreliable input, I think I remember our fallback only has 3 additional encodings before it gives up and it's been working fine)
Since UTF-8 is a variable-length encoding, it somewhat naturally has some error detection built in. Fixed-length encodings don't really have that, and for some of them, any byte value, 0 to 255, in any position, is valid. (Some have a few byte values that are invalid or reserved, but the point still stands.)
So you could very easily pick a "next most common encoding" after UTF-8 fails, try it, find that it works (that is, no bytes are invalid in that encoding), but it turns out that's still not actually the correct encoding. The statistics-based approach will nearly always yield better results. Even a statistics-based approach that restricts you to a few possible encodings that you know are most likely will do better.
(Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)
The only way you'd successfully decode Shift-JIS as UTF-8 is if it essentially is just latin-alphabet text (though the yen symbol would incorrectly display as a '\'). If it includes any non-trival amount of Japanese in it, it'll fail to decode as UTF-8.
As for whether or not you can then (after it fails to decode as UTF-8) use statistical analysis to reliably figure out that it's in fact Shift-JIS, and not something else, I can't speak to that.
[0] https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map
If you're dealing with lengths, you can get input data in meters, centimeters, millimeters, inches, feet, etc.
If the input data is human heights, would you automatically assume meters even if the input data is "183"?
If the input data is the weight of humans, would you always assume grams, even if the input data is "75"?
It depends on the circumstances. It might be the least bad thing to do. Or not.
But that wasn't my point. I replied to this:
> I can assume the length of a meter or the weight of a gram
Sure, the length of a meter and the "weight" of a gram are both standardized. (To be very picky, "gram" is a mass, not a weight. The actual weight depends on the "g" constant, which on average is 9.81 m/s^2 on earth, but can vary about 0.5%.)
So if you know the input is in meters, you don't need to do any further processing.
But dealing with input text files with an unknown encoding is like dealing with input lengths with an unknown unit.
So while UTF-8 itself might be standardized, it is not the same as all input text files always being in UTF-8.
You can choose to say that all input text files must be in valid UTF-8, or the program refuses to load them. Or you can use silent heuristics. Or something inbetween.
It is not.
https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_...
Write all software like NASA needs to do for spacecraft?
Let me know when anyone has developed a web browser to those standards.
More to the point, what situation can you think of where guessing measurement units is a good idea? In a CNC machine? Maps program? Somewhere else? You seem to have omitted the actual counterargument part from your counterargument, while adding a hearty dash of misplaced outrage.
(yeah, my only grip with SI, but it's a frustrating one.)
-
You just won't always be right.
Very few developers I’ve met know could make a distinction. They’d see a few off characters and think it’s some one-off bug but it’s because they’re both assuming an encoding.
Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to.
Please pay up front.
Unless you want it fixed immediately, then a million dollars should motivate almost any developer to spend a month learning, a month doing, and a few years on vacation. A billion is incomprehensible.
(Note that the heuristic for "is this intended to be UTF-8" is pretty reliable, but most other encoding-detection heuristics are very bad quality)
Little-endian bytes (lowest byte is leftmost) and big-endian bits (bits contributing less numerical value are rightmost) are normal, but the bits are referenced/numbered little-endian (first bit is leftmost even though it contributes the most numerical value). When I first read the numbering convention I thought it was going to be a breath of fresh air of someone using the much more sane, but non-standard, little-endian bits with little-endian bytes, but it was actually another layered twist. Hopefully someday English can write numbers little-endian, which is objectively superior, and do away with this whole mess.
Upon reading this, I threw my laptop out the window.
Especially in programming where we already use in-band encoding like 0x to denote a hex string or 0b to denote a binary string. I like using 1{s}, e.g. 1x to denote a little-endian hex string and 1b to denote a little-endian binary string, to denote little-endian encoding.
But, even ignoring programming, it is still better in normal use. The Arabic language got it right writing little-endian (Arabic numbers are written the same, but Arabic is a right-to-left language so it is actually little-endian), and the European languages just stole it stupid by copying the form instead of the function.
https://en.wikipedia.org/wiki/Hindu%E2%80%93Arabic_numeral_s...
And having shattered the dirty window, your room was enlightened.
> 01000001
> Only the second and final bits are 1, or "on".
Isn’t it more accurate to say that the first and penultimate bits are 1, or “on”?
It works much better for handling discrete integers.
Once you get into bitfield instructions it is nice to have bit 0 be the "left most bit".
We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.
It’s much easier to tell the people with old cassette tapes to rip them, rather than try to put a tape player in every car.
I assume you mean "rip them", as in transcode to a different format?
In that case, you need a tool that takes the old input format(s) and convert them to the new format.
For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8.
Isn't the point of the article to describe how an engineer would create such a tool?
It depends on the requirements.
If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.
It becomes easier to just use https://pypi.org/project/ftfy/ on the input.
Honestly, no, because the tool that it's suggesting how to write isn't one that will even come close to doing a good job.
If you want to write such a tool, the first thing you need to do is to understand what the correct answer is. And to do that, you need to sample your input set to figure out what the correct answer should be for several inputs where it matters. There's unfortunately no easy way to avoid that work; universal charset detection isn't really a thing that works all that well.
But the point of the article is not the same thing as how well it achieved its goals.
This seems like something LLMs would be good at. A mundane use of them, but I bet they'd be really good at determining that the input has the wrong encoding. Then the program would iterate through encodings, from most probable to least, and select the one that the LLM likes the most. Granted, this means your tool will be 1GB or more. But hey, thems the breaks.
In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|".
So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.
If "oui" compresses to two bytes and "non" compresses to one byte, and then you go over them with a stream cipher, which is which:
A: ;
B: *&
1. the length of your phone call; and
2. what language you were speaking; oh and
3. half the words you said
(i.e. pwned)
https://web.archive.org/web/20080901185111/https://technolog...
How? Remember, the uncompressed text gets encrypted too.
That article is about voice calls. Totally different topic. Nothing to do with UTF-8.
This is true in both the compressed and non-compressed case. However with compression the size of the plaintext depends on the contents, so the leak of the size can matter more than when not using compression.
Even without compression this can matter sometimes. Imagine compressing "yes" vs "no".
Ah, I see. Naïvely, this seems like a really bad thing for an encryption algorithm to do—is there no way around it? Like, why is encryption different from hashing in this regard?
Part of the problem (afaik) is we lack good math tools to analyze the trade offs of different padding size vs how much extra privacy they provide. This makes it hard to reason about how much padding is "enough".
Another approach is adding a random amount of padding. This can be defeated if you can force the victim to resend messages (which you then average out the size of).
Hashing is different because you don't have to reconstruct the message from the hash. With encryption the recipient needs to decrypt the message eventually and get the original back. However there is no way to transmit (a maximally compressed) message in less space then it takes up.
There are special cases where this doesn't apply e.g. if you have a fixed transmission schedule where you send a sprcific number of bytes on a specific agreed upon schedule.
So if you compressed the string "Bob likes yams" and I could convince you to append a string to it and compress again, then I could see how much the compressed length changed.
If the string I gave you was something already in your data then the string would compress more than it would if the string I gave you was not already in your data - "Bob likes yams and potatoes" will be larger than "Bob likes yams likes Bob".
If the only thing I can see about your data is the length and how it changes under compression - and I can get you to compress that along with data that I hand to you - then eventually I can learn the secret parts of your data.
In an enormous number of real world cases adversaries can end up including attacker-controller input alongside secret data. In that case you can guess at secret data and if you guess correctly, you get smaller compressed output. But even without that, imagine the worst case: a 1TB file that compresses to a handful of bytes. Pretty clearly the overwhelming majority of the text is just duplicate bytes. That’s information which is leaked.
For example "FooFoo" will be smaller than "FooBar" since there is a repeated pattern in the first one.
The attacker can look at the file size and make guesses about how repetitive the text is if they know what the uncompressed or normal size is.
This gets more powerful if the attacker can insert some of their own plaintext.
For example if the plaintext is "Foo" and the attacker inserts "Fo" (giving "FooFo") the result will be smaller than if they inserted zq where there is no pattern. By making lots of guesses the attacker can figure out the secret part of the text a little bit at a time just by observing the size of the ciphertext after inserting different guesses.
Yes, there are scenarios where that matters. However the vast majority of usecases of utf-8 don't fit that or even use encryption at all.
And even if the format only stored the checkbox states as a single bit each (unlikely), compression algorithms don't care. They will behave differently on different byte sequences, which can easily lead to a difference in output length.
Also, it's already been done with voice calls with no attacker-controlled data: https://web.archive.org/web/20080901185111/https://technolog...
The version of your checkbox example which is vulnerable without any formal data compression is when the checkbox is encoded in a field that is only included or changes in length if the value isn't the default, common in uncompressed variable-length encodings like JSON.
Also, zip/deflate etc was not designed to eliminate side channel leakage. Some compression schemes obviously (with padding) can mitigate leaks, but it has to be done deliberately
The special concern with compression is when attacker-controlled data is compressed against secret data because then the attacker can measure the length multiple times and deduce the secret based not just on the length but on how the length changes when the secret is constant and the attacker-controlled data varies. This can be mitigated with random padding (makes the attack take many times more iterations because it now requires statistical sampling) or prevented by compressing the sensitive data and attacker-controlled data separately.
- not having to worry about byte order;
- and cruft like surrogate pairs;
- being able to pass the text through 8 bit string representations and even manipulate it as 8 bit in certain useful ways
Add bitmapped graphics to any text, and it will dominate the size.
Size of text is just noise in the world of streaming video, terabyte drives, fiber to the home ...
Browsers do, kind of https://mimesniff.spec.whatwg.org/#rules-for-identifying-an-...
I took the article to be for people who would be writing that "standalone program"?
I have certainly been in a position where I was the person who had to deal with input text files with unknown encodings. There was no-one else to hand off the problem to.
Me: "I'll make the program stop and ask the user."
Customer: "No. We have 10 million files. Start by using heuristics for batch processing."
Or "what do you mean ask them, they barely know what a jpeg is?!"
You just look bad if you can't do what every other software is able to do. Charging for it takes that to another level.
That's how business works...
If such a business competitor doesn't exist, then yes charge extra, and actually do the work correctly.
This isn't some theoretical thing, we do this at $dayjob right now not only guessing the encoding but the file-type as well so that we can make sense out of whatever garbage our users upload. Everything from malformed CSV exports form Excel to PDFs that are just JPEGs of scanned documents. It works, and it works well.
And of course it does, the files our users are handing to us work on their machines. They can open them up and look at them in whatever local software they produced them with, there's no excuse for us to be unable to do the same.
The FCC ULS database records are stored in a combination of no fewer than three different encodings(1252, UTF8, and something else for a handful of German names) that vary per record.
When I brought this up they said something to the effect of: it's already unicodes it has tilde letters!
I had write to my own "clean this as well as possible" thing, and it did a good enough job.
What ever happened to the Robustness Principle[1]? I think the entire comment section of this article has forgotten it. IMO the best software accepts many formats and "deals with it," or at least attempts to, rather than just exiting with "Hahah, Error 19923 Wrong Input Format. Try again, loser."
1: https://en.wikipedia.org/wiki/Robustness_principle
If you'd like to see why, read the HTML 5 parsing portion of the spec. Slowly and carefully. Try to understand what is going on and why. A skim will not reveal the issue. You will come to a much greater understanding of the problem. Some study of what had happened when we tried to upgrade TCP (not the 4->6 transition, that's its own thing) and why the only two protocols that can practically exist on the Internet anymore are TCP and UDP may also be of interest.
>> We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded.
>> If you'd like to see why, read the HTML 5 parsing portion of the spec.
> Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec
I think the point was that the HTML 5 spec tries to parse all kinds of weird input instead of drawing a line in the sand and forcing the input to follow a simple format?
HTML was a massive success, while at the same time when we tried having a strict spec with XHTML, we failed miserably.
I mean, the purist in me wants strictness but it seems lousyness wins.
The explosion of the web happened in no small part because of how easy it was to write some HTML and get a basic, working webpage out of it. If you nested some tags the wrong way and the browser just put up an error page, rather than doing a (usually) pretty good job figuring out what you actually meant, people would get frustrated faster and not bother with it at all.
But imagine if our C/C++/Java/Rust/Go/etc. compilers were like "syntax error, but ehhhhh you probably meant to put a closing brace there, so let's just pretend you did". That would be a nightmare of bugs and security issues.
The difficulty in drawing a line in the sand and sticking to the spec, though, is that of user blame. Let's say you implement a spec perfectly -- even if you are the originator of the spec -- and then someone comes along and builds something of their own that writes out files that don't conform to the spec. Your software throws up an error and says "invalid file", but the other piece of software can read it back in just fine. Users don't know or care about specifications; they just know that your software "doesn't work" for the files they have, and the other software does. If you try to tell them that the file is bad, and the other software has a bug, they really won't care.
That's how Perl ended up the way it is.
It also shows the harmfulness of binary black and white thinking in engineering. There are choices other than "just let everyone do whatever and hope all the different things picking up the pieces do it in more or less the same way" and "rigidly specify a spec and blow up the universe at the slightest deviation". Both of those easy-to-specify choices have excessive costs. There is no escape from hard design tasks. XHTML may always have been doomed to fail, but that is not to say that HTML had to be allowed to be as loosey-goosey as it is, either.
Had a gradient of failure been introduced rather than a rigid rock wall, things very likely wouldn't have gotten as badly out of hand as they did. If, for instance, a screwed up table was specified to deliberately render in a very aesthetically unappealing manner, but not crash the entire page the way XHTML did, people would have not come to depend so much on HTML being sloppy. The resulting broken page would still be somewhat usable, but there would have been motivation to fix it, rather than the world we actually live in where it all just seemed to work.
Being lenient is all well and good when the consequences are mild. When the consequences of misinterpreting or interpreting differently to a second implementation becomes costly, such as a security exploit, then the Robustness Principle becomes less obviously a win.
It's important to understand that every implementation will try to fix-up formatting problems in their own way unique to their particular implementation. From that you get various desync or reinterpretation attacks (eg. HTTP request smuggling).
As you say, unfortunately the genie cannot be put back in the bottle for formats that already have defective implementations in the wild.
E.g. you produce valid .wat files, but my software which also outputs those has some bits screwed up.
My program can read both .wat but yours can't, but I have 5% market share.
Your users complain they sometimes receive files your software can't read while the competitor can. Do you tell them "well that file is invalid, tell whoever sent it to you to change the software they use"?
The genie can't stay in the bottle unless you have some sort of certification authority and even that may not be enough (see USB)
And you're saying that by doing this, not only do I gain an advantage over the existing competition, but I also make it more difficult for more competitors to appear?
[takes notes]"
The customer does not know. They just want it to work. They may be using something that someone else gave them. The original source system of the file may not be changeable. But most importantly, their boss just wants it to work. or else.
Bush hid the facts
For those who don't know that reference: https://en.wikipedia.org/wiki/Bush_hid_the_facts
(A text file containing only the ASCII bytes "Bush hid the facts", when opened in Windows Notepad, displays a sequence of CJK characters instead of the expected English sentence.)
This basically loses data integrity if it's wrong though.
You might want to do that with human input if it's helpful to the user - ie user enters a phone number and you strip dashes etc. But if it's machine to machine, it should just follow the spec.
https://en.wikipedia.org/wiki/Robustness_principle#Criticism
Postel's Law doesn't pass a software engineering smell test.
The idea that software should guess and repair bad inputs is deeply flawed. It is a security threat and a source of enshittification.
I am so over this word.
But that is not always the case, and e.g. silently "fixing" text encoding issues can often corrupt the data if you get it wrong.
By all means offer options of you want, but if you do flag very clearly to the user that they're taking a risk of corrupting the data unless any errors are very apparent and trivial to undo.
If you reject malformed input, then the person who created it has to go back and fix it and try again. If you interpret malformed input the best you can (and get it right), then everyone else implementing the same thing in the future now also has to implement your heuristics and workarounds. The malformed input effectively becomes a part of the spec.
This is why HTML & CSS are the garbage dump they are today, and why different browsers still don't always display pages exactly alike. The reason HTML5 exists is because people finally just gave up and decided to standardize all the broken behavior that was floating around in the wild. Pre-HTML5, the web was an outright dumpster fire of browser compatibility issues (as opposed to the mere garbage dump we have today).
Anyway, it's not really important to try to convince you that Postel's Law is bad; what's important is that you know that many people are starting to think it's bad, and there's no longer any strong consensus that it was ever a good thing.
The problems is, there is one very popular OS which is very hard to enforce UTF-8 everywhere, Microsoft Windows.
It's very hard to ensure all the software stack you are depending on it use Unicode version of Win32 API. Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. Even if you don't you have to ensure all the low level code you are depending on it do the same for you.
Oh and don't forget about the Unicode Normalizations. There is no THE UTF-8. There are bunch of UTF-8s with different Unicode normalizations. Apple macOS use NFD while other mostly use NFC.
These are Just some examples. When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.
It depends. If you're writing an app, just add the necessary incantation to your manifest, and all the narrow char APIs start talking UTF-8 to you.
For a library, yeah.
https://learn.microsoft.com/en-us/windows/apps/design/global...
Yes. You should convert your strings. Thankfully, UTF-16 is very difficult to confuse with UTF-8 because they're completely incompatible encodings. Conversion is (or should be) a relatively simple process in basically any modern language or environment. And personally, I've never run into a problem where the difference between NFC and NFD mattered. (Do you have an example?). The different forms are (or should be) visually completely identical for the user - at least on modern computers with decent unicode fonts.
The largest problem with UTF-8 (and its biggest strength) is how similar it is to ASCII. It is for this reason we should consider emoji to be a wonderful gift to software correctness everywhere. Correctly handling emoji requires that your software can handle unicode correctly - because they need multi-unit encoding with both UTF-16 and UTF-8. And emoji won't render correctly unless your software can also handle grapheme clusters.
> When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.
Check! If your application deals with text, throw your favorite multi-codepoint emoji into your unit testing data. (Mine is the polar bear). Users love emoji, and your software should handle it correctly. There's no excuse! Even the windows filesystem passes this test today.
It is valid for the presentation of compound emoji can fallback to their component parts. You can't expect every platform to have an up to date database of every novel combination. A better test is emoji with color modifiers. Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix.
On modern desktop OSes and smart phones, I do expect my platform to have an up-to-date unicode database & font set. Certainly for something like the unicode polar bear, which was added in 2020. I'll begrudgingly look the other way for terminals, embedded systems and maybe video games... but generally it should just work everywhere.
Server code generally shouldn't interact with unicode grapheme clusters at all. I'm struggling to think of any common, valid reason to use a unicode character database in 'normal' backend server code.
> Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix.
I didn't know about that one. I'll have to try it out.
Case insensitive search
It's fine to have an embedded device that's never updated, but never receives new content - it doesn't matter that a system won't be able to show a new emoji because it doesn't have any content that uses that new emoji.
However, if it is expected to display new and updated content from the internet, then the system itself has to be able to get updated and actually get updated, there's no acceptable excuses for that - if it's going to pull new content, it must also pull new updates for itself.
And if we're talking about how new products should be designed, then the "almost" goes away and they 100% wouldn't receive new content through anything like CDs, the issue transforms from an irrelevant niche (like CDs nowadays) to a nonexistent one.
I despise that Unicode retroactively applied default emoji presentation to existing symbols, breaking old text. Who the hell though that was a good idea?
Mine is the crying emoji.
And after enough failures in breaking the system, the 100 emoji.
I just tried it in gnome-terminal, and while the crying emoji works fine, polar bear or a country flag causes weird issues.
My native language uses some additional CJK chars on plane 2, and before ~2010s a lot of software had glitches beyond the basic plane of unicode. I am forever grateful for the "Gen Z" who pushed for Emojis.
Javascript's String.length is still semantically broken though. Too bad it's part of a unchangeable spec...
Though you are right in that I don't know of a built-in way to count Unicode Scalar Values (USVs).
I’d be very happy with the standard library providing multiple “length” functions for strings. Generally I want three:
- Length in bytes of the utf-8 encoded form. Eg useful for http’s content-length field.
- Number of Unicode codepoints in the text. This is useful for cursor positions, CRDT work, and some other stuff.
- Number of grapheme clusters in the text when displayed.
These should all be reasonably easy to query. But they’re all different functions. They just so happen to return the same result on (most) ascii text. (I’m not sure how many grapheme clusters \0 or a bell is).
Javascript’s string.length is particularly useless because it isn’t even any of the above methods. It returns the number of bytes needed to encode the string as UTF16, divided by 2. I’ve never wanted to know that. It’s a totally useless measure. Deceptively useless, because it’s right there and it works fine so long as your strings only ever contain ascii. Last I checked, C# and Java strings have the same bug.
Length in encoded form can be found after encoding by checking the length of the binary content I guess.
I think for historical reasons access to codepoints can be useful, but it's rarely what one wants.
The built-in string.length method is useless (it returns the number of char objects) and I agree that's a problem, but the solution is also built into the language, unlike in JS.
The Intl.Segmenter API [0] can similarly yield iterable objects with all the grapheme clusters of a string. Also, the TextEncoder [1] and TextDecoder [2] APIs can be used to convert strings to and from UTF-8 byte arrays.
[0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
[1] https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder
[2] https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder
Before emoji, if somebody open a bug report like: "Your software doesn't handle UTF-8 correctly. It doesn't handle Japanese.",
the response was "Huh? We don't bother to support Japanese. Go pound sand. Close ticket with wontfix.".
Now it's "Your software doesn't handle UTF-8 correctly. It doesn't handle emoji" and we're like "Oh shit! My software can't handle my beloved emoji!"
Google, Apple, IBM and MS also did a lot of localisation so their code bases deal with encoding.
It is FOSS Unix software that had the ASCII mindset, probably as C and C++ string types are ASCII and many programmers want to treat strings as arrays. The MacOS and Windows APIs do take UTF as their input not char * (agreed earlier versions did not put they have provided the UTF encodings for 25 years at least.
The main place I've seen it get annoying is searching for some text in some other text. Unless you normalize the data you're searching through the same way as you normalize your search string.
That’s a great tip - obvious in hindsight but one I’d never considered.
You mean like opening a file by name?
About a decade ago I wrote some utility code for reading files, where it'll try to detect BOM first, if not scan for invalid UTF-8 sequences. If none are found assume UTF-8 else assume Windows-1252. Worked well for us so far.
Still get the occasional flat file in Windows-1252 with one random field containing UTF-8, so some special handling is needed for those cases. But that's rare.
Fortunately we don't have to worry about normalization for the most part. If we're parsing then any delimiters will be one of the usual suspects and the rest data.
It's actually worse than that.
Older versions of mac did enforce NFD for file names, but more recent names don't, at least at the OS level. But many apple programs, such as finder _will_ use NFD. Except that it isn't even Unicode standardized NFD, it is Apple's own modified version of it. And this can cause issues when for example you create a file in finder, then search for it using `find`, and type the name of the file the exact same way, but it can't find the file because find got an NFC form, but the actual file is in NFD.
OTOH, in many applications, you don't really care about the normalization form used. For example, if you are parsing a CSV, you probably don't need to worry about if one of the cells using using a single code point or two code points to represent that accented e.
What applications? Almost nobody writes their own implementations of encryption nowadays (nor should they). You mean openssl is "broken"?
(Kidding, mostly.)
This Joel On Software article [0] is a good starting point. Incredibly it's now over 20 years old so that makes me feel ancient! But still relevant today.
The suggestion that the web should just use utf-8 everywhere is largely true today. But we still have to interact with other software that may not use utf-8 for various legacy reasons - the CSV file example in the original article is a good example. Joel's article also mentions the solution discussed in the original article, i.e. use heuristics to deduce the encoding.
[0] https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
[1]Me, surely at least 1 other
[1] Me, for sure.
[1]you may find him at ::1
[2]HEAD~2
[3]Front-end work
If Excel generates CSV files with some Windows-1234 encoding, then my "import data from CSV" function needs to handle that, in one way or another. A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding, they won't care that Microsoft is using obsolete or weird defaults. They will see it as a bug in my program and demand that I fix my software. Even if Excel offers them a choice in encoding, they won't understand any of that and more importantly they don't want to deal with that right now, they just want the thing to work.
What if you just gave them instructions?
In can tell you right now anyone reading this site is in a completely different universe of tech competency than the general public, and even professionals who aren’t tech-focused.
You would have lost many of them simply with then jumble of letters “UTF-8”.
With CSV exports, there's already a good bit of training you have to do WRT the column layout, file format, headers (or not), cell value formats, etc. There's far more lift involved in training users there than ensuring they select "UTF-8" when they select CSV.
And, there's really nothing technical about it, as they don't need to understand what "UTF-8" actually means any more than they need to understand what "CSV" stand for. It's a simple ask in a list of asks.
I weigh this against having every developer now believe they have to check/convert every character set, which can be unreliable and produce garbage, BTW. And, speaking of garbage, there are some encodings that you won't be able to convert in any case, so it would be technically impossible to preserve data integrity without pushing some requirement on the source up to the user.
So, it's about tradeoffs. And, again, if asking users to choose UTF-8 is the difference between customers choosing your app or a competitor's, then you probably need to be more worried about that than your charset encoding.
Of course, if that charset uses code points not available in your app's native charset, then you're kind of back to square one (unless your use case tolerates garbled or missing data).
Gen Z has skills in what is available and popular today, just like the previous generation had.
OTOH, they have to follow some process to use the software. For just the CSV export, they already have to ensure column orders, values, formats, maybe headers, etc. Selecting UTF-8 from a dropdown seems like the easy part.
My accrued compatibility shims disagree.
Could this work? Implement handling of ancient excel files in your SaaS product, but charge extra dollar for parsing legacy formats and provide information how to export correct files from excel next time :)
It doesn't. Well, maybe "another".
Your function or even app doesn't need to handle it. Here's what we did on a bookkeeping app: remove all the heuristics, edge cases, broken-csv-handling and validation from all the CVS ingress points.
Add one service that did one thing: receive a CSV, and normalize it. All ingress points could now assume a clean, valid, UTF8 input. It removed thousands of LOCs, hundreds of issues from the backlog. It made debugging easy and it greatly helped the clients even.
At some point, we offered their import runs for download, we added the [original name]_clean.csv our normalized versions. Got praise for that. Clients loved that, as they were often well aware of their internal mess of broken CVSs.
Browsers used to have a menu option to choose the encoding you wanted to use to decode the page.
In Firefox, that's been replaced by the magic option "Repair Text Encoding". There is no justification for this.
They seem to be in the process of disabling that option too:
> Note: On most modern pages, the Repair Text Encoding menu item will be greyed-out because character encoding changes are not supported.
( https://support.mozilla.org/en-US/kb/text-encoding-no-longer... )
This note is logical gibberish; encoding isn't something that has to be supported by the page. Decoding is a choice by the browser!
https://hsivonen.fi/no-encoding-menu/
> Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden). With the current approach, the parser needs to know of one flag to force chardetng, which the parser has to be able to run in other situations anyway, to run.
> Elaborate UI surface for a niche feature risks the whole feature getting removed
> Telemetry [...] suggested that users aren’t that good at choosing correctly manually.
In other words, it's trying to protect users from themselves by dumbing down the browser. (Never mind that people who know what they are doing have probably also turned off telemetry...)
>> Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden).
There's no explanation of why you'd want this, or why it's security-relevant.
(Farther down, there's a mention of self-XSS, which definitely isn't relevant.)
>> Elaborate UI surface for a niche feature risks the whole feature getting removed
They've already removed the whole feature. That was easier to do after they mostly disabled it, not harder.
>> Telemetry showed users making a selection from the menu when the encoding of the page being overridden had come from a previous selection from the menu.
That would be an example of "working as expected". The removal of the ability to do this is the problem that disabling the encoding menu causes! Under the old, correct approach, you'd guess what the encoding was until you got it right. Under the new approach, the browser guesses for you, and if the first guess is wrong, screw you.
I think we should use a 4 bit encoding.
0 - NUL
1-7 - aeiouwy
8 - space
9-12 - rst
13-15 - modifiers
When modifier bits are set, the values of the next half-byte change to represent the rest of the alphabet, numbers, symbols, etc. depending on the bits set.
Or, you know, just say "nah, I can, those ancient stuff don't matter (outside of obligatory exceptions, like software archeology) anymore." If someone wants to feed me a KOI8-R or JIS X 0201 CSV heirloom, they should convert it into something modern first.
I have a hobby interest in IBM mainframes and IBM i, so yes to EBCDIC for me. (I have encountered them professionally too, but only to a very limited extent.) In practice, I find looking for 0x40 (EBCDIC space) a useful heuristic. Even in binary files, since many mainframe data structures are fixed length space padded.
> then we also need to bring EBCDIC (and EBCDIK and UTF-EBCDIC for more perverted fun) into the picture. Makes heuristics extra fun.
Actual use of UTF-EBCDIC, while not nonexistent, has always been extremely rare. A person could spend an entire career dedicated to IBM mainframes and never encounter it
EBCDIK, at first I wondered if that was a joke, now I realise it is a name used for non-IBM Japanese EBCDIC code pages. Again, something one can spend a whole career in mainframes and never encounter – if one never works in Japan, if one works for a vendor whose products aren't sold in Japan, probably even if you work for a vendor whose products are sold in Japan but only to IBM sites (as opposed to Fujitsu/Hitachi sites)
I require UTF-8. If it isn't currently UTF-8, it's someone else's problem to transform it to UTF-8 first. If they haven't, and I get non-UTF-8 input, I'm fine bailing on that with a "malformed input - please correct" error.
[1] section 1.2 in https://www.fontucom.com/pdf/AFSARIBR-Map.pdf
[1] https://www.npmjs.com/package/encoding-japanese
We are already doing GSUB/GDEF tables and a lot of horrible stuff to display pretty glyphs on the screen. Hell, there are literal instructions for VM in TTF files to help with pretty rasterization on low-res screens.
Making font rendering library is hard, Nintendo hard.
That's just a nature of the beast. Fonts are messy and if we want one standard to deal with it once and for all, that means some compromises must be done. CJK is not some obscure alphabet. In this case, perfect is enemy of good.
I am just glad there really is only one (non-niche) standard, obligatory https://xkcd.com/927/
Right. One would think that misrendering a major world language used by over 100 million people would be an issue that warrants some attention. But too many of the HN crowd don't care.
> In this case, perfect is enemy of good.
It's not though. Unicode-only is not just imperfect, it's an outright regression for Japanese. Meanwhile traditional encoding-aware programs render Japanese just fine.
Unicode has variant selectors which can deal with region variants (e.g. https://imgur.com/a/syMcWNO) to deal with the issue.
Granted, it's not very widely used, but Unicode provided a solution. The onus is now on application developers and font providers.
Choosing to limit number of characters to 2 bytes was a good technical choice back in 1992, non-unified CJK wouldn't fit (that was a time of 2-4MB RAM and 100-200MB HDD). A solution to problem was later provided.
Using Shift-JIS is like doing on-IP, non-UDP protocol on the internet.
Sure, you can do it, but it's just non-perspective choice. A better invested time is on improving standard protocol, which is Unicode (e.g. conversion software).
They're officially deprecated and cause issues with breaking search etc.. So they're still below feature parity with using a traditional encoding. Traditional encoding support is also easier to test since documents with traditional encodings are more widespread and exist for many world languages, not just Japanese.
Of course, there is no correct solution (people in other comments seem to believe it exists a bit too quickly). Dynamic HTML page can load pieces of text written by people all across the world, it can't rely on any “main” language. Those people can't be automatically classified (based on some browser settings or location). Then there are always people who know both Chinese and Japanese (while using some other system locale, to make things more complex). It is wrong to assume that they should not be able to use Chinese forms and Japanese forms at once, even inside a single paragraph or phrase.
I wonder why Unicode has not simply introduced some “combining language form character” to make user choice stick. After all, there's a whole subsystem for emoji modification, and those things were once “weird Japanese texting customs”. As for complexity of handling Unicode text, it asymptotically reaches its maximum anyway.
Wait a second, there are “variation selectors” and some “Ideographic Variation Database”. Is it the solution? Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch…
It's the opposite of vaporware; there's a whole bunch of well-known software that handles encodings correctly.
> new code that doesn't care about handling some lang metadata is not going to care about Shift JIS either.
If you support encodings the way almost every programming language tells you too, you'll handle Shift-JIS just fine. If you use the "legacy" encodings approach and test even one non-English language you'll handle Japanese fine.
> Can IMEs and converters simply stamp each variable character with invisible mark based on current input language? I suppose there's some catch…
They're officially deprecated, IMEs and converters don't use them, naïve search implementations (which is to say most search implementations, because people do not test for an obscure edge case) break, ....
As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack. A particularly evil solution would choose arbitrary existing combining characters that can't be declared “illegal” retroactively.
By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
I assure you there are thousands if not millions of apps that implement search the naive way, by comparing bytes.
> It won't handle emoji, normalization, and thousands other things.
Yeah. But it works well enough for Americans, so the maintainers don't care enough to fix it.
> As for official stance, maybe it's time for some group to agree on certain non-conflicting sequences, and implement them in some library/stack.
And then what? What do you do with documents in Japanese that have already been converted without using your new codepoints? What do you do with programs that ignore your new standard? What do you do about search being broken in most apps with no prospect of fixing it? There are already some PUA codepoints for a proper implementation of Japanese, but most app authors don't even understand the problem, never mind being willing to support something that's "non-standard". Asking them to support traditional non-unicode encodings, which is something that's at least relatively well-known and standardised, and something they can test without knowing Japanese, is much easier.
> By the way, is it possible to make proper quotes from other languages as described above when using Shift JIS?
No, not for arbitrary languages. If you want to mix arbitrary languages, you need some structure that's a rope of strings with an encoding per segment. But that's exactly the same thing that the unicode advocates claim is easy ("just have a sequence of spans with different lang tags"), and at least if you use traditional encodings then you don't have to do any such fancy business for the much more common case of a file that's entirely in Japanese (perhaps with some ASCII keywords).
For example, if I want to talk about the fairly rare japanese name '刃一' (jinichi), there's a chance your computer displays it correctly, but there's also a chance your computer displays the chinese variant of the first character, making it look wrong. It's basically up to font-choice of your computer.
The "correct" way to fix that would be for me to be able to tag it with 'lang=ja', but Hacker News doesn't let me include html tags or some other 'language selector' in my comment, so I'm unable to indicate whether that's supposed to be the chinese or japanese variant of the character.
Most unicode text files don't have extra metadata indicating if a certain bit of text is japanese or chinese, so displaying it correctly by adding the correct 'lang' tag is impossible, especially since it's perfectly possible for one utf-8 text to mix both chinese and japanese.
See https://heistak.github.io/your-code-displays-japanese-wrong/
Only if you have some unicodebrained mentality where you consider a Chinese character that looks sort of similar to a Japanese character to be "the same". If you think of them as two different characters then they're just different characters, which may or may not be present in particular encodings (which is completely normal if you make a program that handles multiple encodings: not every character exists in every encoding)
> In which case, why is it unreasonable to ask for the same for UTF-8, except using some more clearly specified way to indicate this (like lang="ja" etc), rather than encoding it all into separate characters?
Firstly, you have to "draw the rest of the fucking owl" and actually implement that language selection mechanism. Secondly, if you implement some clever extension mechanism on top of UTF-8 that's only needed for Japanese and can only really be tested by people who read Chinese or Japanese, realistically even if you implement it perfectly, most app makers won't use it or will use it wrong. Whereas if you implement encoding-awareness in the standard way that we've been doing for decades and test with even one non-default encoding, your application will most likely work fine for Japanese, Chinese and every other language even if you never test it with Japanese.
However, a unicode codepoint which acted as a "language hint" switch would solve this, and wouldn't require doubling the number of han codepoints.
There already are unicode variant selectors, but iiuc they only apply to a single character at a time, and no one actually uses them, so they're not very useful.
Bonus - as soon as the Unicode consortium do find a way, my software should be able to handle it with no further changes. Well, it might need a recompile against a newer `libicu` as I don't think they maintain ABI backcompat between versions. But there's not much I can do about that.
I know there's a weird Chinese/Japanese encoding problem where characters that kind-of look alike have the same character id, and the font file is responsible for disambiguation (terrible for multi-language content and we should really add more characters to create versions for each, but still the best we have).
And even back in the mid 2000s, the only real problems I saw otherwise, were things like malformed html pages that assumed a specific encoding that they wouldn't tell you, or an MP3 file with an ID3 tag with CP932 shoved into it against the (v1) spec.
I also disagree with the author that Shift-JIS can be "good enough" hueristically detected due to its use of both 7 and 8-bit characters in both the high and low bytes to mean different things depending on what character is actually intended. Even string searching requires a complex custom-made version just for Shift-JIS handling.
And it indeed facilitates, as in practice it works better than encoding in utf-8 that lacks an in-band way to communicate that, and out-of-band often fails/is ignored/doesn't exist.
Realistically --- if someone hands me well marked Shift JIS content, I'm just going to reencode it as UTF-8 anyway... And if they hand me unmarked Shift JIS content, I'll try to see if I can decode it as UTF-8 and throw it away as invalid if not.
You'd think that, but in practice I've found the opposite. Applications that use encodings managed to display things properly. Applications that hardcode UTF-8 don't.
Trivia: There are some variants of ShiftJIS where this isn't entirely true. The traditional yuugen gaisha symbol, for example, which is analogous to U+32CF (LTD), is not supported. The /VENDORS/APPLE/JAPANESE.TXT file uses a PUA designator and then a sequence of four Unicode code points to convert it.
In theory yes. In practice it doesn't.
> just like it would have with a Japanese-centric encoding.
The difference is that encoding-aware systems naturally use Japanese fonts for Japanese encodings and other fonts for other encodings.
Systems like what?
Switching your entire encoding system to set a different font is by far the stupidest way to do it.
Encoding-aware programs, UI toolkits, etc..
> Switching your entire encoding system to set a different font is by far the stupidest way to do it.
If it's stupid and it works, it's not stupid. I wish there were other reliable ways to have international programs display Japanese correctly, but there aren't.
Name just one please.
To be clear, we're talking about a program/library that can handle both unicode and shift-JIS, and it will render a character that unicode considers identical in different ways depending on what encoding you loaded the character from, right?
Firefox (not the best example for a number of reasons, if you're going to follow up then I'll talk about a different one, but if you really want just one then it's the program I have to hand right now).
> To be clear, we're talking about a program/library that can handle both unicode and shift-JIS, and it will render a character that unicode considers identical in different ways depending on what encoding you loaded the character from, right?
Yes.
Huh, today I learned.
Though if you're making any attempt to use valid tags you'll have <html lang="ja">, and that solves the problem for the web context at least as far as my testing goes.
Right, which is why that's a bad example. But it's the same for most "normal" applications (even e.g. text editors, or I remember hitting this kind of thing in WinRAR), and a lot of the time there isn't a standard way of indicating the language/locale in the file. Even within firefox, there are (admittedly rare) cases where you're viewing something that isn't HTML and doesn't have HTTP headers so using the encoding or manually setting the page language is still the only way to make it work - and applications that have a manual "file language" setting are the exception rather than the rule.
If you implement only unicode then you put yourself in a situation where Japanese is uniquely different from every other language, and your program will not work properly for Japanese unless you tested Japanese specifically.
Only among people who don't care. Implementing encoding support means your app supports a way to display Japanese properly. If you want to add more ways to display Japanese properly, go ahead, but that's supererogatory, whereas UTF8-only apps don't have a way to display Japanese properly at all.
> to support Japanese, you have to do some extra work to get and use extra info about a language, which is also possible within UTF8
It's not. There's some imaginary theorycrafted way in which it might notionally be possible within UTF-8, but not one UTF-8-only app has ever actually implemented support for displaying Japanese properly. The only approach to displaying Japanese properly that has ever actually been implemented in reality is to support multiple encodings (or to support only a Japanese encoding, but that has obvious downsides), and if you make your app encoding-aware then that's enough, you don't have to do anything else to be able to display Japanese properly for people who care about displaying Japanese properly (it's always possible for an app maker to go above and beyond, but I'm sure you'll agree the big difference is between an app that has a way to display proper Japanese at all and one that does not).
Then it's no longer "someone else's problem".
The article doesn't say that you should. It clearly states that for many cases, the input format is known or explicitly stated in the input headers.
The article talks about cases where the input files are in an unknown input format. Even then, it states: "Perhaps there is a case to be made that csvbase's auto-detection should be a more explicit user interface action than a pre-selected combo-box."
But for the case where the requirements call for heuristics, the article then talks about how that can be done.
> at that point I'm writing an adapter to convert into UTF-8 JIT in front of my service.
And at that point you might need the advice in the article, right?
Right, and at that point you're probably gonna need these statistics-based heuristics to write that adapter. Unless you know specifically what other encoding each bit of input is. If you do, then, again, you are not the target audience for this article.
Sounds like academia. "You" can't pay your bills anyway.
I got 99 problems, but charsets aint one of them.
I wish I could live in the world where I could bluntly say "I will assume UTF-8 and ignore the rest of the world". Many Japanese documents and sites still use Shift JIS. Windows has this strange Windows-932 format that you will frequently encounter in CUE files outputted by some CD ripping software. ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208. These special characters are mostly icons used in traffic and weather reports, but transcoding to UTF-8 still causes trouble with these icons.
You can choose to never work on projects where you have to deal with files like that.
But there may come a day where you have to choose between not paying your rent or writing a tool that converts old textfiles to UTF-8. At that point, it's nice to have references on the internet on how other people have actually dealt with it and what works. "Abort with an error" is not very useful advice then.
What are the common pitfalls that the library must deal with?
How do you know how to evaluate them if all you know is "not UTF-8? Abort with error" ?
Do you have any experience at all with different charsets under the hood by the way?
There are examples of where people used encryption libraries in the wrong way and undermined the strength of the encryption (for example, CVE-2024-31497 in PuTTY).
A very big part of software development is dealing with leaky abstractions. We don't work with perfect black boxes. We need to understand enough of how things works in the lower layers to avoid problems. Note here that I wrote "enough", not "everything", or "write everything yourself".
I would not want a person writing software to handle charset conversion if he refuses to learn how the various encodings work, which charset will decode as another charset or not, etc.
"older approach, PuTTY developers said, was devised at a time when Microsoft Windows lacked native support for a cryptographic random number generator."
So "enough of how things work" could just be "pick a modern encryption library" that doesn't come from the dark ages when t there were no random numbers
Same with encodings, it requires a much lower level understanding to pick a library, you can rely on the expertise of others
We are not running short on Unicode codepoints. I'm sure they can spare a few more to cover the Japanese characters and icons which invariably get mentioned any time this subject comes up on HN. I don't know why it hasn't happened and I won't be making it my problem to solve. Best I can do is update to version 16 when it's released.
[0]: Except Japanese, people hasten to inform us every time this comes up. Why? Why haven't your odd characters and icons been added to Unicode, when we have cuneiform? That's the strange part. I don't understand why it's the case.
0: https://heistak.github.io/your-code-displays-japanese-wrong/
Kind of a band-aid (it's necessary to stuff a variant selector after a CJK codepoint), but should work.
These decisions were made back in 1992 and codepoint in 16-bit was one of desired goals. Non-unified CJK wouldn't fit. In hindsight, it looks like a rather unfortunate decision, but having more codepoints that would fit to 16 bits could seriously hamper adoption and different standard would win (compute resources were far more limiting back then).
In either case, it's like 4 byte addressing in IPv4, in hindsight, 6+ bytes would be better, but what's done is done.
Edit: Even in 2000s, when C# was released, string was just a sequence of 16-bit code units (not codepoints), so they could deal with BMP without problems and astral planes were ... mostly DIY. They added Rune support (32-bit codepoint) only in .NET Core 3.0 (2019).
Adobe / Google seems to have a font https://ken-lunde.medium.com/improving-font-information-proc...
EDIT: Seems IVS is really old, but its still a problem so not holding my breath
I'm Russian. 20 years ago that meant having to deal with two other common encodings aside from UTF-8 (CP1251 and KOI8-R). 25 years ago, it was three encodings (CP866 was the third one). Tricks like what the article describes were very common. Things broke all the time anyway because heuristics aren't reliable.
These days, everything is in UTF-8, and we're vastly better off for it.
There are two additional planes set aside for further Hanzi, the Supplementary and Tertiary Ideographic Planes, the latter is still mostly empty. Eventually the last unique ideograph used only to spell ten known surnames from the 16th century will also be added as a codepoint.
I view the continued use of Shift-JIS in Japan as part of a cultural trend, related to the continued and widespread use of fax machines, or the survival of floppy disks for many years after they were effectively dead everywhere else. That isn't at all intended as an insult, it's that matters Japanese stay within Japan to a high degree. Japanese technology has less outside pressure for cross-compatibility.
Shift-JIS covers all the corner cases of the language, and Unicode has been slow to do likewise, and it isn't like Japanese computers don't understand UTF-8, so people have been slow to switch. It's the premise of "unaware of how it works in the rest of the world" that I object to. It's really just Japan. Everywhere else, including the Chinese speaking parts of the world, there's Unicode data and legacy-encoded data, and the solution to the later is to encode it in the former.
Amazingly enough, ARIB STD-B24 is one of the major source of Unicode emojis. So transcoding would actually work for them! (I am aware of some exceptions, but semantically there is no loss.) Unicode and UTF-8 are truly eating every other legacy encoding else, so much so that it is becoming more reasonable to have a separate transcoding step.
/s
At this point, I don't see any sufficiently good reason to not use utf-8 exclusively in any new system. Conversions to and from other encodings would only be done at well defined boundaries when I'm calling into dependencies that require non utf-8 input for whatever reason.
Same with UTF-8. Life's too short for bothering with anything else today. I'll deal with some weird janky encoding for the right price, but the first thing I'd do is convert it to UTF-8. Damned if I'm going to complicate the innards of my code with special case code paths for non-UTF-8.
If there were some inherent issue with UTF-8 that made it significantly worse than some other encoding for a given task, I'd be sympathetic to that explanation and wouldn't be such a pain in the neck about this. For instance, if it were the case that it did a bad job of encoding Mandarin or Urdu or Xhosa or Persian, and the people who use those languages strongly preferred to use something else, I'd understand. However, I've never heard a viable explanation for not using UTF-8 other than legacy software support, and if you want to continue to use something ancient and weird, it's on you to adapt it to the rest of the world because they're definitely not going to adapt the world to you.
I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.
Which is all to say that you're right, but I can't imagine that it's more than a theoretical nuisance outside some extremely niche cases.
Are basically non existent. Almost all modern web servers transparently compress html. Sending uncompressed text over the wire is extremely rare.
EDIT: also, it's not safe—message length is dependent on the values of the plaintext bytes, period. i'm not saying don't live dangerously, i'm just saying live dangerously knowing
The reason its usually safe is that to exploit you need:
- a secret inside the html file
- the secret has to stay constant and cannot change (since it is adaptive attack. CSRF tokens and similar things usually change on every request so cannot be attacked)
- the attacker has to have a method to inject something into the html file and repeat it for different payloads
- the attacker has to be able to see how many bytes the response is (or some other side channel)
- the attacker is not one of the ends of the communication (no point to attack yourself)
Having all these requirements met is very unlikely.
[0] https://utf8everywhere.org/#asian
(Consider: Horse is five letters, but 馬 is one character. Even at three bytes per character, Chinese wins.)
Unfortunately Google and many other companies have decided UTC is the only way, so this causes issues with ICS files that use that format sometimes when they are generating their helpful popups in the GMail inbox.
If you have to take medication (for instance, an antibiotic) every 24 hours, it must be taken at the same UTC hour, even if you took a train to a town in another timezone. Keeping the same local time even when the timezone changes would be wrong for that use case.
Calendars, alarms, and reminders have some overlap here and floating time can be good for some cases.
> If you give me a computer timestamp without a timezone, I can and will assume it's in UTC.
Do you mean, give you an _offset_? `2024-04-29T14:03:06.0000-8:00` the `-8:00` is an offset. It only tells you what time this stamp occurred relative to standard time. It does not tell you anything about the region or zone itself. While I have consumed APIs that give me the timezone context as part of the response, none of them are part of the timestamp itself.
The only time you should assume a timestamp is UTC is if it has the `z` at the end (assuming 8601) or is otherwise marked as UTC. Without that, you have absolutely no information about where or when the time has occurred -- it is local time. And if your software assumes a local timestamp is UTC, then I argue it is not the sender of that timestamp's problem that your software is broken.
My desire to meet you at 4pm has no bearing on if the DST switchover has happened, or my government decides to change the timezone rules, or if {any other way the offset for a zone can change for future or past times}. My reminder to take my medicine at 7pm is not centered on UTC or my physical location on the planet. Its just at 7pm. Every day. If I go from New York to Paris then no, I do not want your software to tell me my medicine is actually supposed to be at Midnight. Its 7pm.
But, assuming you aren't doing any future scheduling, calendar appointments, bookings, ticket sales, transportation departure, human-centric logs, or any of the other ways Local Time is incredibly useful -- ignore away.
"Convert to UTC and then throw away the time zone" only works when you need to record a specific moment in time so it's crazy how often it's recommended as the universal solution. It really isn't that hard to store (datetime, zone) and now you're not throwing away information if you ever need to do date math.
People love simple rules and they will absolutely take things too far. Most developers learn "just use UTC!" and think that's the last thing they ever need to learn about time.
Consider my statement more in the context of logs of past events. The only time you can reasonably assume a given file is in a particular non-UTC TZ is when it came from a person sitting in your same city, from data they collected manually, and you're confident that person isn't a time geek who uses UTC for everything. Otherwise there's no other sane default when lacking TZ/offset data. (I know they're not the same, but they're similar in the sense that they can let you convert timestamps from one TZ to another).
That's certainly fair in the context of a recurring event with some formula. I caution that a lot of people will still immediately reach for timestamps to calculate that formula, particularly for a next occurrence, and in the context if this conversation they would be given as an ISO8601 datetime based on Local Time. I would also caution that calendar events that have a distinct moment in time that they start are also prime for Local Time where a UTC-default mentality will cause errors.
> Consider my statement more in the context of logs of past events
From the stance of computer generated historical log data, I definitely agree that UTC everywhere is a sane default and safe to assume :)
(And, in your defense, I would definitely argue UTC-everywhere gets you 95% of the way there for 5% of the effort... I get why people make the tradeoff)
> (I know they're not the same, but they're similar in the sense that they can let you convert timestamps from one TZ to another).
More nitpicking on my part, again, I'm sorry, it lets you convert from one _offset_ to another or from an offset to UTC. Think Arizona being a special snowflake who (mostly!) doesn't observe DST. You can't assume all UTC-7 offsets are all Mountain Time.
You usually end up with having to deal with whatever eccentric sh!t that ultimately comes from the same source as the payment for the job.
Please look up the issues caused by Han unification in Unicode. It’s an important reason why the Chinese and Japanese encodings are still used in their respective territories.
See for example this effort in building a universal charset detector in Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1551276
If it doesn't fit in those categories, you've got a much harder guessing game. But usually you can't actually ask the source what it is, because they probably don't know and might not understand the question or might not be contactable. Usually, you have to guess something, so you may as well take someone else's work to guess, if you don't have better information.
It has an entire section of its documentation dedicated to why you should not use heuristic charset detection: https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-c...