-encoding encoding
Set the source file encoding name, such as EUC-JP and UTF-8. If
-encoding is not specified, the platform default converter is used.
where that last bit about the platform default converter is something I'd expect from a language that comes out of a squishier (like Python or POSIX) culture.
Internally, the model of a String is UTF-16-like, a sequence of unsigned 16-bit `char` values. However, it's not strictly UTF-16 because a Java String permits unpaired surrogates and invalid code points.
Externally, Java has historically dealt with whatever character set was in use on whatever platform it was running on. As time as gone on, most platforms have converged on UTF-8 for external data storage and interchange. Time for Java to do that too, at least by default.
The common mistake is relying on any 'default' for this. There's no such thing. This has caused me a huge amount of headaches over the years in the form of messed up strings in data exports, databases, or API outputs. There's always one place where somebody forgets to specify Utf-8 and relies on the default.
Which is often not Utf-8 on lots of operating systems depending on how they are configured. Even Linux has issues with this. Even in 2023. Most distributions set a default encoding of UTF-8 of course but if you start your server early in the boot process as root, none of that may be applied. If you write your own systemd scripts, be sure to make this explicit as well. I've found this out the hard way where the default was latin-1 and combined with somebody not doing what they should be doing, you end up dealing with corrupted data.
It's why not specifying a encoding when decoding bytes to a string is a mistake in Java that things like spotbugs check for and warn against. It's one of those things that IMHO should have been deprecated in Java ages ago. Instead they keep on adding more APIs that do the same thing. Removing it would probably break lots of code. But the point is that that code should already be considered broken and should be fixed.
Thankfully, Kotlin uses sensible default parameters in Kotlin extension functions that do similar things for this (of Utf-8). If you want something else, you have to be explicit about it. That's the only sane way to deal with character encodings IMHO. As long as you don't use Java APIs directly, things are fine.
I think the industry is moving towards UTF-8 everywhere as a default however the platform is misconfigured. Python certainly is. I think it is rare but not unheard of to find files in a different encoding and if that is the case it is not deliberate but an accident because of a misconfigured default encoding.
The industry moved more than two decades ago already. However operating systems continue to do the wrong thing for no good reason other than that they have been doing that since forever. UTF-8 was already a thing when Java launched in 1995. It was proposed as a standard three decades ago. As I mentioned, I've been dealing with weird UTF-8 related issues throughout my career. As soon if you have APIs that rely on the default and operating systems that don't set that to UTF-8, you end up with corrupted content.
There are all sorts of ways to make this happen: APIs that set a mime type but no character encoding, progams that open and save files, databases that don't default to using utf-8 (or like mysql actually have some non standard 3 byte version of utf-8 that doesn't deal with emojis and a lot of other characters). Linux distributions don't set UTF-8 as the character encoding until late in the boot process. IMHO there's no good reason why it should ever do anything else than UTF-8 in a shell from the microsecond it boots until it shuts down. Changing all that is hard however. Not for technical reasons but for reasons of backwards compatibility with systems that are arguably part of the problem here.
I think the root cause for this might be a blind spot that people in English speaking countries have since they don't use a lot of special characters and therefore don't see the problem. As soon as you start using Spanish, Swedish, German, etc. You are in trouble. And forget about any non latin scripts. Latin-1 is not appropriate for any of those. Utf-8 is a better default that should be usable everywhere. Some countries might prefer utf-16. But that's mainly a memory optimization that should not matter much in 2023.
> Internally, the model of a String is UTF-16-like, a sequence of unsigned 16-bit `char` values.
That is not 100% true anymore. Java 9 (September 2017) changed “the internal representation of the String class from a UTF-16 char array to a byte array plus an encoding-flag field. The new String class will store characters encoded either as ISO-8859-1/Latin-1 (one byte per character), or as UTF-16 (two bytes per character), based upon the contents of the string. The encoding flag will indicate which encoding is used.” (https://openjdk.org/jeps/254)
The model of a String, internal to the JVM, as manifested in the `java.lang.String` API, is still UTF-16-like. The internal representation has changed to be either UTF-16-like or Latin-1. JEP 254 didn't change the API. This is invisible to Java programs, and they can't tell which internal representation is being used. If a Java program wants a byte array from a String, it still has to go through the APIs that involve codeset conversion. There are suitable fast paths, though. For example, if the program requests Latin-1 and the internal representation is Latin-1, the result is just a straight copy of the internal array.
The bug in question has a number in the 8,000,000 range, but that doesn't mean there are 8M bugs. The bug numbering was carried over from the old Sun-internal bug database. That bug database was shared across a large number of Sun projects, not just Java, so it went through the numbers pretty quickly. I'm not entirely sure, but I think the numbering might have started at 1,000,000 in order to make sure the bug IDs all had seven digits.
In addition, there were several migrations to different databases, where "new" bugs were created at a particular range (say, 4,000,000) to distinguish them from "old" bugs that had been migrated from the previous system. Thus, there are large gaps in the numbering.
When OpenJDK migrated from the old Sun bug database to JIRA, something similar was done. New bugs were created starting in the 8,000,000 range. Most of the old Java-related bugs were migrated from the Sun bug database into JIRA, preserving their existing numbers. There are bugs in the 1,000,000 range that date back to 1994, predating Java 1.0, and well before Java was open-sourced.
The lowest-numbered bug in the current database is this one from 1994:
(This includes only JDK bugs. There are other projects in the same JIRA instance that have different numbering schemes.)
Finally, a single bug can generate multiple items in the JIRA database. For example, if a bugfix needs to be backported to an older release, this generates subtasks for each targeted release. There are about six actively maintained releases, so there's a bit of a multiplier effect. (Not every bugfix is backported, though.)
There are currently about 318,000 bugs (open and closed) in the JDK project.
Warning, controversial: I'm actually against this.
While I believe we should have a standard, I feel that having UTF-8 in source code brings more risks than benefits. [0]
Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` (yea I had to c/p that from somewhere). Searching for the former does not discover the latter.
Lastly, there is the issue of Intellisense, or whatever it is called in many languages. I have seen codebases in English with the aforementioned `é` in a function name. The only way for me to select that function name was with the arrow keys / mouse. I couldn't type it on my QWERTY.
And yes, I know that there are codebases which are non-English, and not even Latin. Those are very valid concerns to which I admittedly have no answer to.
depending on your OS / windowing system, you actually have easy ways to quickly get greek letters out of your (non-greek) keyboard without having to copy-paste them:
my favorite way is to assign a key or key combination of your choice to act as so-called dead_greek modifier, and then just press that modifier before a latin letter key to get the corresponding greek letter. For example, under linux/xorg, if you wanted AltGr+g to be your dead_greek, you can use xmodmap to set it so:
alternative 1: assign a key as a direct modifier (not a dead key) and add key combination defs for that to generate the characters you want … alternative 2: switchable keyboard layouts
can't test it currently as the machine I have here doesn't run wayland but xorg, but iirc: the general idea of using a dead_greek modifier should also work on wayland, but assigning a key for it unfortunately cannot be done with the simple non-xkb legacy X11 keyboard layout tools like xmodmap that still work on xorg.
That being said, wayland did take over part of xkb (the less old keyboard system) from xorg, and so from what I can gather from a quick search, the easiest way to assign a key as dead_greek in wayland would probably be with a file in $XDG_CONFIG_HOME/xkb/symbols/ like they do here for other key symbols: https://unix.stackexchange.com/questions/292868/how-to-custo...
I don't know any way to tell wayland to (re)load an xkb config on the fly (without logging out and back in) though. In particular, I doubt that setxkbmap would work for that like on xorg.
I can see wanting to stick to plain ASCII for identifiers, but what of embedded strings? Localization might prefer to be data-driven, but what of unit tests surrounding text management - including locale-specific sorting tests, glyph rendering, etc.? I don't think you're against making the source code UTF-8 per se, just "UTF-8" identifiers. Which seems quite valid. In rust you might write:
#![forbid(non_ascii_idents)]
pub fn fiancée() {}
And get an appropriate error:
error: identifier contains non-ASCII characters
--> src/lib.rs:3:8
|
3 | pub fn fiancée() {}
| ^^^^^^^
|
note: the lint level is defined here
--> src/lib.rs:1:11
|
1 | #![forbid(non_ascii_idents)]
| ^^^^^^^^^^^^^^^^
> Why would you not just put test input in a separate test-input file that you pipe in?
You just added a dependency on File I/O (can't easily run the test on filesystemless embedded/wasm targets), deserialization, the current working directory if using relative paths, filesystem layout if using absolute paths - we lose type checking as part of our compile step, so deserialization might fail, we lose intellisense...
> That way you can test multiple different encodings and all sorts of edge cases
Well, a mixture of half and fullwidth yen symbols (¥￥). A bit awkward to eyeball as they've been escaped via \u#### codes - dodging the whole "what encoding are our source files" problem - but given the graphical variants of that glyph, I could see keeping the escaped versions. Now, the array this is helping construct could certainly be deserialized from, say, a JSON or XML test-input file. But what would we gain, exactly, besides more boilerplate and context switching to wade through?
> While I'm also team ASCII, it's possible I'm not appreciating some edge case here.
To be clear I'm not saying you can't use files, and sometimes files are more appropriate and convenient than loading everything into code despite the caveats I mentioned... but I'm not seeing much of a boon for exiling test data to files here.
thanks. Interesting examples - and thanks for actually finding in-code examples :)
I didn't realize OpenJDK targets systems without filesystems. That's cool. I thought the Java Smart Card days are behind and the Java folks only care about server applications now. However having a requirement that all tests be inline seems a bit onerous. Do they inline images as well for testing BufferedImage and company?
I can see how an external file makes it a bit more of an "integration test" that would in effect be testing several moving pieces. I don't agree it would create more boiler plate or make things any less clear though. It makes it much easier to test many different complex and large inputs and to feed in new tests without needing to recompile. It's also easy to version control and introduce new tests with pathological cases. It presents a clear separation of test inputs from code - but maybe I can understand the counterargument. It does look neater to have everything together
makes it easy to embed arbitrary data with your code (or in this case, your tests, which are usually compiled separately) so no, you can easily test against files without access to a filesystem (beyond the access required to read the jars).
I still think having test cases with strings in the code is often a lot clearer personally.
(Unrelated: I wish more PLs made it easy to embed files within executables and access them with an OS-independent filesystem like API, it's often very useful.)
> I didn't realize OpenJDK targets systems without filesystems.
To be fair I'm not sure it does, really, per se. I know embedded Java is/was a thing, though, and it wouldn't suprise me if someone somewhere tortured their own personal fork into running a test suite on embedded stuff - if only for legacy support testing purpouses.
And poking around in that directory did make it clear some tests use files containing test vectors.
> I don't agree it would create more boiler plate
To get concrete again, I consider this boilerplate:
And this is merely (de)serializing pure unstructured strings without any kind of data format or failiable schema beyond charset encoding, making this a poster child for exiling test data to files (probably part of the reason why it was exiled to files!)
> It makes it much easier to test many different complex and large inputs and to feed in new tests without needing to recompile.
For bulk plain text I'd agree. For codebases with slow incremental builds, structured data might also benefit from being exiled for iteration speed. Or there can be benefits to skipping having a full developer environment.
But if incremental builds are fast (a worth goal), and if full developer environment is reasonably assumed (dev-focused unit/integration testing), compiler assistance with structured data is often more convenient, and has better error reporting for syntax errors etc. than what you'll get from many/most simple and straightforward uses of deserializers.
If you are specifically trying to test Unicode support, then you might want to try multiple formulations for the same glyph. E.g. you can make é either directly with 00e9 or with a combining mark 0065 0301. You can't tell the difference in a string literal, but escape codes will make it clear.
What's the difference between a variable name that's obscure because it has a special character and a variable name that's obscure for some other reason?
I wish a compose key was more widely available. It's actually moderately intuitive, on x windows, é is compose-e-'.
Yes, this. The files in the source repo are 99.9% ASCII. Anything that's not ASCII should be encoded in UTF-8 instead of some other encoding. The mailing list discussion refers to some issues we've had where characters in Shift-JIS (CP932) crept into the code base.
I don't think anyone is advocating using non-ASCII characters, such as accented characters or emoji, in identifiers.
Exactly, and this isn't a proposal for everybody else's Java software, it's a proposal for the JDK itself.
So this isn't even close to Rust's rule (all Rust source is UTF-8), let alone a declaration that Java is going to introduce data types with Korean names or new Russian method names on existing types, it's just hey, if any part of the JDK's own source code needs non-ASCII then it should be UTF-8, and if it doesn't need to be non-ASCII that's fine because ASCII is a strict subset of UTF-8.
I actually agree with this. Any byte higher than 127 in any source file should be a compile-time error, and it is so in any project where I can make the call.
> And yes, I know that there are codebases which are non-English
Non-English speaker here. Use English for identifiers and comments. Put magic strings in files other than source files, which is what you should be doing anyway if you want to seriously support i18n.
This makes it a PITA to use non-ASCII in string literals. They have applications beyond non-English language support. Unicode character escapes are opaque and impair readability. If a wizened language like C can do it gracefully, newer languages should be able to as well.
I don’t know, if you want to find any form of official documentation for your dependencies/frameworks, you will have to know English either way. It is just the de facto language of IT, and I think it is fair to have every (even internal-only by a foreign language speaking group) codebase entirely in English.
And I say it as someone whose native tongue is not English.
> While I believe we should have a standard, I feel that having UTF-8 in source code brings more risks than benefits. [0]
> Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` (yea I had to c/p that from somewhere). Searching for the former does not discover the latter.
If your search function can't handle different characters with the same meaning then it's already broken (e.g. usually you want search to be case insensitive). Likewise if similar-looking characters are an exploitable problem in your codebase then that problem already exists with e.g. l/I.
> Lastly, there is the issue of Intellisense, or whatever it is called in many languages. I have seen codebases in English with the aforementioned `é` in a function name. The only way for me to select that function name was with the arrow keys / mouse. I couldn't type it on my QWERTY.
In any decent system you can type it with compose-'-e like it looks like.
I'm all for making programmers dogfood the use of non-ascii characters (actually I'd like to see more use of non-unicode encodings, unless and until unicode stops screwing over Japanese). Maybe it'll give them more empathy for those of us who need to type things like this daily, and stop them doing things like flashy new packaging systems that break IME.
They might be alluding to issues from CJK unification. But I don’t believe that many practitioners working with Japanese text consider these concerns more important than the value that Unicode brings to the table.
I think that CJK unification leads to awkwardness where some fonts will basically render some kanji the “Chinese way” or the “Japanese way”, so you can’t really have a good font that covers both at once. There’s a good aesthetic argument about not using the same font for both languages, but Unicode outright precludes it I think
Virtually every Japanese organisation disagrees, you only have to look at a Japanese website to know as much (or are those somehow not "practitioners working with Japanese text"?). Most will use a non-Unicode encoding and/or resort to rendering important text as images, since it's impossible to rely on Unicode text rendering well.
I wonder if this can ever be fixed. If basically every Japanese organization disagrees with the Unicode consortium to the point that its undermining the use of unicode in entire countries, it might be worth unwinding CJK normalization.
This isn't the case obviously. Japanese organisations are involved in the Unicode process (something evidenced in the input for the several CJK Unified Ideograph Extensions). Thank to Unicode, many historical characters can be represented in digital text without resorting to placeholders or hacks.
Anecdotally, I've visited a group enthusiastically exploring the possibilities of the then-new CJK Unified Ideograph Extension B in Tokyo over a decade ago. The CJK issue mentioned here was a bigger problem then than now (most developers know how to handle it properly, e.g. with language declarations in HTML etc.).
People will use legacy encodings. That's just a fact of life. Sometimes out of inertia (i.e., it just works, so why bother changing it?), sometimes out of technical limitations of their pipeline, sometimes because of a lack of understanding, rarely out of principle. I'm stuck fixing an issue with some numb skull sending us SAML requests declared as UTF-8 containing some obsolete encoding instead, and this is for Dutch text! This just happens.
There is also nothing wrong with text in Shift-JIS, if that encoding encompasses all characters used.
From here [0], 94.1% of .jp websites use UTF-8 and 6.5% use Shift-JIS. You can find a list of popular sites still using Shift-JIS here [1].
If you change your PC or your phone locale to Japanese and access mostly Japanese website, the font display on most major OS works well. It's probably fair to assume that's a very common setup for Japanese users, and probably for Chinese and Korean users as well.
I think the CJK "problem" is more apparent for international users who don't have the locale setup or dealing with multiple languages plain text at the same time.
Your own link shows that both Shift-JIS and EUC-JP are on an upward trend, which isn't what we'd expect if Unicode was succeeding. I can't see it listing examples of .jp sites using UTF-8; would be interesting to see how many of them are relying on text-as-images.
Yeah what lmm says is not what happend/thought in Japan. Japanese still uses Shift_JIS because they love (/s) legacy software. Using Shift_JIS isn't a practical solution.
Random sample of some websites:
- Rakuten Bank -> Shift JIS
- NHK, the Ministry of Health and Labor Welfare, yahoo japan -> UTF-8
I don't really really know, but this random yahoo answers [0] page seems to agree with me that really you're gonna use UTF-8 for new stuff in general
One point that answer brings up that I hadn't thought of: older flipphones would have good shift-JIS support but not necessarily good unicode support
My guess is that "text as images" might partly be that, partly what you are saying in a legacy sense, and also partly for the same reason old sites in the US would do that: to make something look the way you want and you already have a photoshop render
[0]: https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q14...
> Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters?
Yes. Unicode uses codepoints that are primarily for Chinese characters to represent Japanese characters that they consider equivalent, even when those Japanese characters have different appearences; as a result, Japanese text in unicode looks bad (readable, but ugly) unless displayed in a specifically Japanese font (in which case you'd have the converse problem of Chinese text looking wrong). The unicode consortium suggests various vaporware approaches to combat this, but the thing that actually works is keeping Japanese text in Japanese encodings and Chinese text in Chinese encodings. (Of course this means that you need to be able to display text from multiple encodings in the same page if you want to display both languages on the same page, but all of the unicode consortium vaporware fixes require you to build something equivalently complex, and you wouldn't even be able to test it by using strings from two western encodings since it would be specific to Japanese and Chinese)
Even assuming that works, getting it to make its whole way through your whole tech stack is no easier (and more HTML-specific) than having spans with their own encoding.
This isn't about multiple encodings, this is about the one encoding, UTF-8, being able to represent multiple languages. Which is mostly can, except for han characters.
In my reply here, I can type in english and russian at once. Привет, мир.
Yet, if I try to type chinese on one line, and japanese on the next, I cannot do it. Hacker news does not let me enter "lang" tags, so I can only type either the chinese or the japanese variant of a kanji.
so yes it is about pages with multiple encodings. A span's smaller than a page!
The Unicode answer is "variation selectors" which are used for some historical variant kanji, but not for whole language switching. I suppose they could be used for that too though.
I don't know about HTML specifically; I meant spans as a general concept rather than a literal <span> tag. If the web stack has actually implemented mixing languages on the same page to the point where you can use it in a "normal" application then that's very cool (and if they've done it with their lang tags rather than by allowing mixed encodings, well, fine), but I've yet to see a site that actually has that up and running.
I can't read it, but there will be hundreds of articles on Japanese Wikipedia covering Chinese literature etc that have text in both languages, all in Unicode.
<html lang="ja">
Japanese text ...
<span lang="zh">
Chinese quote
</span>
...
</html>
is much easier than mixing encodings. With the above entirely in Unicode, it will be handled reliably by anything that can handle Unicode, and is still reasonably readable even if the Chinese text is shown in a Japanese font. Reading just the fourth line without the third will still show something 'OK'.
There's no difference between encoding a page in Shift_JIS and tagging the page as Japanese language (assuming your browser switches fonts correctly based on language tags). Chinese text in Shift_JIS isn't going to look correct either.
Sure, but every application handles encoding (or at least, every application did up until the recent UTF-8-only movement), whereas language tagging is web/html-specific.
If you're really hardcore processing multilingual text you need an equivalent to language tags anyway, because you need a dictionary for word-wrapping, date formatting, quote marks etc along with changing fonts (or glyph selectors in the same font.) TeX has them too.
But usually people only care about their language, so it goes by the system UI language and it works out.
> If you're really hardcore processing multilingual text you need an equivalent to language tags anyway, because you need a dictionary for word-wrapping, date formatting, quote marks etc along with changing fonts (or glyph selectors in the same font.) TeX has them too.
Sure. And once you're doing that you don't gain a lot of benefit from unicode AFAICS, because you have to track these spans of locale-specific text.
People's names are the thing you should be most careful to not do that with! Most Japanese people will generally put up with seeing the Chinese rendering of a character in ordinary text, but they really don't want you to do that to their names!
I think the suggested alternative though is having UTF-8 take care of this.
UTF-8 already has different codepoints for every other language, like I can type russian (День), greek (Ημέρα), english, even egyption hieroglyphs (𓀃) all of them in this one text field, and they all render right for both of us if we have a suitable font. If they don't render, it's unambiguous.
It's only chinese and japanese where I can type some characters, and depending on if you have a chinese or japanese font _first_ in your system, it might render wrong, and either way it's ambiguous to the computer without further metadata. The computer doesn't need metadata to know that "α" is "α", why does it need metadata to know if 直 is the chinese (http://www.hanzi5.com/bishun/76f4.html) or japanese (https://kanjivg.tagaini.net/viewer.html?kanji=%E7%9B%B4) character, two different looking characters that share the same codepoint.
Unicode does not have different codepoints for every language. Leaving aside scripts that aren't encoded at all, all languages using the Latin script get a single set of glyphs, and if you want Comic Sans MS or Carolingian miniscule or fine-grained control over whether "a" has a hook at the top or not, you need to specify a font that renders it the way you want, just like with 直.
Because Unicode is about the meaning of characters, not their appearance. If there were a context where the two different shapes of 直 have different meaning, Unicode would add a new codepoint to distinguish them. In fact "ɑ" without hook does have a separate codepoint because it is used in linguistics to mark a vowel different from "a".
If CYRILLIC SMALL LETTER A deserves a code point despite being so similar to LATIN SMALL LETTER A, why doesn't a Japanese character that actually looks different from a Chinese one get a codepoint? And the idea that something can have the "same meaning" across two languages is very wooly.
Why don't Japanese characters in seal script or handwriting get different codepoints from mincho font characters? They look different too.
Sometimes looking visually different doesn't matter because if you know how to write them by stroke order, you'll still be able to read it (which is how handwriting works, I think; I'm pretty bad at reading that…)
Hanzi simplification in the Mainland also complicated things, since I doubt they wanted to make all of those into different characters.
Snap/Flatpak/etc. often mean you can't use IME (input method extensions) with programs packaged with them unless the packager has jumped through certain hoops, which they usually haven't.
Ew, that's... extremely unfortunate. I'm sure there's some internal reason, but that feels like something the system should be able to add support for unilaterally, at least by default, without needing special packager effort:(
You'd need something that can actually parse the language to apply rules on language constructs like identifiers. I don't think EditorConfig supports this, but a linting tool would be able to.
While I agree that sticking with ASCII sidesteps a lot of potential issues, I don’t agree that “I don’t know how to type that character” is a valid excuse nowadays. There are various ways, from Compose keys to builtin OS functionality, that entering accented characters, at the very least, shouldn’t be a major hurdle. This is to encourage people to look into this, it’s really not that difficult.
> I feel that having UTF-8 in source code brings more risks than benefits. [0]
While this is a really fun theoretical attack, has this ever been encountered? A such patch would have to go trough a diff anyway before being accepted. And certainly a "comment" that looks like code has a high chance of getting rejected?
Does it make sense to restrict this feature (that is useful to everyone in the world, except those who only speak English) on account of a very theoretical risk?
> Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée`
Just in Italy, it can be written as: fidanzata, zita, morosa, ragazza, picciotta. Certainly the accent isn't the main problem here? Do you want to eliminate all synonyms? All languages other than English?
> I couldn't type it on my QWERTY.
Have you encountered this situation often? If so, just remap your keyboard.
I, for example, have a "w" key that is mostly used in shooter games, since the letter doesn't appear in my language.
> And yes, I know that there are codebases which are non-English, and not even Latin. Those are very valid concerns to which I admittedly have no answer to.
Perhaps allowing unicode in source files could be a solution.
> it also limits discoverability if `fiancee` now is written as `fiancée` […] Searching for the former does not discover the latter.
Let's book this under "falsehoods programmers believe about Unicode".
You can try either the code below or simply using your browser's search function for this very page and type in the string `fiancee` and see all variants, accent-marked or not, highlighted as results.
#!/usr/bin/env perl
use utf8;
use Unicode::Collate;
my $uc = Unicode::Collate->new(
normalization => undef, level => 1
);
printf "pos %d len %d\n",
$uc->index('Lettre à ma fiancée.docx', $_)
for 'fiancée', 'fiancee', 'lettre a ma-Fiancee _DOCX_';
__END__
pos 12 len 7
pos 12 len 7
pos 0 len 24
Pardon me if I butchered the French grammar, I do not speak the language and can't easily verify whether I made a mistake in adapting the code from a year ago. <https://news.ycombinator.com/item?id=30405840>
It isn't a falsehood but a valid remark regarding most tools. Tried searching "fiancee" in your example on VS Code and failed to find "fiancée". Of course you can consider "e=é" similar to how you can do case-insensitive search considering "e=E". Problem is most tools currently don't do this.
I think you have misunderstood what this issue is about. The Java Language Specification already specifies that Java programs are written using the Unicode character set and that letters in identifiers can be drawn from the entire Unicode character set. The issue is that there are many ways to encode Unicode characters and that a single encoding is not used across the code base. The suggestion is merely to normalize this encoding so that all files use the same, so as to not confuse the tooling and complicate the build process.
It's not about using more or less non-ASCII characters. It's about saving the source code files all in the same encoding (UTF-8) and standardizing around this encoding.
They want get rid of an encoding mess.
I don't know how one can be against this. Your code is probably saved in UTF-8 right now even if you don't use non-ASCII characters.
It's probably just that the JDK is a very old code base that was written from different OSes, without much thought on the encoding. Now, they want to standardize while making sure not to break anything.
What's more, the presence of non-ASCII characters in such a huge code base is almost inevitable so yeah, it matters if you want to avoid issues.
> Your code is probably saved in UTF-8 right now even if you don't use non-ASCII characters.
Nit: For a document that only contains ASCII characters, UTF-8 is identical to ASCII. So calling it UTF-8 is technically correct, but misleading -- you could equally well say the document is encoded using any of the other supersets of ASCII.
Indeed. I had that in mind when writing my comment, but it's good that you made this part explicit.
You'd better have your editor / tools set to use UTF-8 though in case some non-ascii characters end up in your file or else you might be in for some pain.
Not so controversial. The vast majority of code is written in plain ASCII (and english), and I hope it will go on. Using non ASCII character in your code is a double failure.
Take an example: "trouvé" instead of "found" as a variable name.
1- You're right. Most people won't be able to type "é", "à"... with their keyboard. Working on that code won't be easy.
2: What is the meaning of "trouvé" ? How people could figure out that variable name ? Average coder will search for english named functions or variable.
What you're against seems to be bad practices that could be more easily abused with UTF, rather than UTF itself.
I think the benefits could outweigh the harms, but some of it would indeed depend on good conventions being followed.
As for things like é, a function called fianc__e_accent_aigu__e would be far worse, and still a matter of convention. At least search engines could easily be configured to toggle accented letters as unaccented etc. And even nano has pretty good autocomplete facilities these days.
Where I find a big benefit with utf names is in reproducing standard mathematical formulae in code. But I agree that this implies a good way to input the required symbols that doesnt require you to search utf charmaps.
I wish Windows had better UTF-8 (and Unicode overall) support. The system-wide UTF-8 locale is still in beta after so many decades of suffering. Windows Console/Terminal/PowerShell still have constant issues with CJK.
I recently did something similar in C#. I ended up using .editorconfig to force an error in CI (and in Visual Studio) when a file isn't utf8; and to force Visual Studio to create new files in utf8.
For a minute I was wondering why they weren't making it UTF-7 like everything else. Then I realized that it's UTF-8. I'm sleepy haha.
However, in UTF-8, 7 bits are what is available to distinguish between characters that take only one byte. U+0000 through U+007F, or 0x00 through 0x7F. Ones starting with the first byte 0x80 or higher are chonkier. https://www.youtube.com/watch?v=NQFMeyWVe3g
I'm not sure that's the only reason. That is also lucky :) I guess I will have to read about those who made it. (The official story may not tell all.)
It's not ideal just for that reason but because 7 bits is a good amount to preserve for other characters, as well as the simplicity of only having to check one bit to tell it to look at the next byte to find what character it is. It could have been all but one to say that it isn't a single byte character. Then there would be 255 single byte characters and 256 two byte characters, minus however many you use to specify that there are more bytes.
It would be easy to make an encoding format where 0x00 or 0xFF (0b00000000 or 0b11111111) say to look at the next byte, but that would suck as a general text encoding format.
Their checklist is missing one item; look for mojibake and figure out how to re-encode it in UTF-8. I had to do this recently, and wow it was hard to figure out the right characters! I think I ended up Googling for the string and using that.
I've had good luck years ago just duplicating the string a few times and throwing it at Mozilla's character set detection stuff (there's a "chardet" python library implementing that, IIRC). My context was song metadata (title, artist, etc.) so your mileage might vary.
It's been the case for python for a while and nothing bad happened. Nobody started to use accentuated variables. But I finally could put my name in comments without having to put a # coding header, which is nice.
104 comments
[ 4.5 ms ] story [ 122 ms ] threadhttps://docs.oracle.com/javase/7/docs/technotes/tools/window...
where that last bit about the platform default converter is something I'd expect from a language that comes out of a squishier (like Python or POSIX) culture.Externally, Java has historically dealt with whatever character set was in use on whatever platform it was running on. As time as gone on, most platforms have converged on UTF-8 for external data storage and interchange. Time for Java to do that too, at least by default.
Which is often not Utf-8 on lots of operating systems depending on how they are configured. Even Linux has issues with this. Even in 2023. Most distributions set a default encoding of UTF-8 of course but if you start your server early in the boot process as root, none of that may be applied. If you write your own systemd scripts, be sure to make this explicit as well. I've found this out the hard way where the default was latin-1 and combined with somebody not doing what they should be doing, you end up dealing with corrupted data.
It's why not specifying a encoding when decoding bytes to a string is a mistake in Java that things like spotbugs check for and warn against. It's one of those things that IMHO should have been deprecated in Java ages ago. Instead they keep on adding more APIs that do the same thing. Removing it would probably break lots of code. But the point is that that code should already be considered broken and should be fixed.
Thankfully, Kotlin uses sensible default parameters in Kotlin extension functions that do similar things for this (of Utf-8). If you want something else, you have to be explicit about it. That's the only sane way to deal with character encodings IMHO. As long as you don't use Java APIs directly, things are fine.
There are all sorts of ways to make this happen: APIs that set a mime type but no character encoding, progams that open and save files, databases that don't default to using utf-8 (or like mysql actually have some non standard 3 byte version of utf-8 that doesn't deal with emojis and a lot of other characters). Linux distributions don't set UTF-8 as the character encoding until late in the boot process. IMHO there's no good reason why it should ever do anything else than UTF-8 in a shell from the microsecond it boots until it shuts down. Changing all that is hard however. Not for technical reasons but for reasons of backwards compatibility with systems that are arguably part of the problem here.
I think the root cause for this might be a blind spot that people in English speaking countries have since they don't use a lot of special characters and therefore don't see the problem. As soon as you start using Spanish, Swedish, German, etc. You are in trouble. And forget about any non latin scripts. Latin-1 is not appropriate for any of those. Utf-8 is a better default that should be usable everywhere. Some countries might prefer utf-16. But that's mainly a memory optimization that should not matter much in 2023.
That is not 100% true anymore. Java 9 (September 2017) changed “the internal representation of the String class from a UTF-16 char array to a byte array plus an encoding-flag field. The new String class will store characters encoded either as ISO-8859-1/Latin-1 (one byte per character), or as UTF-16 (two bytes per character), based upon the contents of the string. The encoding flag will indicate which encoding is used.” (https://openjdk.org/jeps/254)
so much history here, or just junk.
The bug in question has a number in the 8,000,000 range, but that doesn't mean there are 8M bugs. The bug numbering was carried over from the old Sun-internal bug database. That bug database was shared across a large number of Sun projects, not just Java, so it went through the numbers pretty quickly. I'm not entirely sure, but I think the numbering might have started at 1,000,000 in order to make sure the bug IDs all had seven digits.
In addition, there were several migrations to different databases, where "new" bugs were created at a particular range (say, 4,000,000) to distinguish them from "old" bugs that had been migrated from the previous system. Thus, there are large gaps in the numbering.
When OpenJDK migrated from the old Sun bug database to JIRA, something similar was done. New bugs were created starting in the 8,000,000 range. Most of the old Java-related bugs were migrated from the Sun bug database into JIRA, preserving their existing numbers. There are bugs in the 1,000,000 range that date back to 1994, predating Java 1.0, and well before Java was open-sourced.
The lowest-numbered bug in the current database is this one from 1994:
https://bugs.openjdk.org/browse/JDK-1169723
(This includes only JDK bugs. There are other projects in the same JIRA instance that have different numbering schemes.)
Finally, a single bug can generate multiple items in the JIRA database. For example, if a bugfix needs to be backported to an older release, this generates subtasks for each targeted release. There are about six actively maintained releases, so there's a bit of a multiplier effect. (Not every bugfix is backported, though.)
There are currently about 318,000 bugs (open and closed) in the JDK project.
While I believe we should have a standard, I feel that having UTF-8 in source code brings more risks than benefits. [0]
Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` (yea I had to c/p that from somewhere). Searching for the former does not discover the latter.
Lastly, there is the issue of Intellisense, or whatever it is called in many languages. I have seen codebases in English with the aforementioned `é` in a function name. The only way for me to select that function name was with the arrow keys / mouse. I couldn't type it on my QWERTY.
And yes, I know that there are codebases which are non-English, and not even Latin. Those are very valid concerns to which I admittedly have no answer to.
[0] https://krebsonsecurity.com/2021/11/trojan-source-bug-threat...
alternative 1: assign a key as a direct modifier (not a dead key) and add key combination defs for that to generate the characters you want … alternative 2: switchable keyboard layouts
That being said, wayland did take over part of xkb (the less old keyboard system) from xorg, and so from what I can gather from a quick search, the easiest way to assign a key as dead_greek in wayland would probably be with a file in $XDG_CONFIG_HOME/xkb/symbols/ like they do here for other key symbols: https://unix.stackexchange.com/questions/292868/how-to-custo...
I don't know any way to tell wayland to (re)load an xkb config on the fly (without logging out and back in) though. In particular, I doubt that setxkbmap would work for that like on xorg.
That way you can test multiple different encodings and all sorts of edge cases
While I'm also team ASCII, it's possible I'm not appreciating some edge case here.
You just added a dependency on File I/O (can't easily run the test on filesystemless embedded/wasm targets), deserialization, the current working directory if using relative paths, filesystem layout if using absolute paths - we lose type checking as part of our compile step, so deserialization might fail, we lose intellisense...
> That way you can test multiple different encodings and all sorts of edge cases
That can be done in code too. Let's take something concrete like https://github.com/openjdk/jdk/blob/master/test/jdk/java/uti... :
Well, a mixture of half and fullwidth yen symbols (¥￥). A bit awkward to eyeball as they've been escaped via \u#### codes - dodging the whole "what encoding are our source files" problem - but given the graphical variants of that glyph, I could see keeping the escaped versions. Now, the array this is helping construct could certainly be deserialized from, say, a JSON or XML test-input file. But what would we gain, exactly, besides more boilerplate and context switching to wade through?> While I'm also team ASCII, it's possible I'm not appreciating some edge case here.
To be clear I'm not saying you can't use files, and sometimes files are more appropriate and convenient than loading everything into code despite the caveats I mentioned... but I'm not seeing much of a boon for exiling test data to files here.
I didn't realize OpenJDK targets systems without filesystems. That's cool. I thought the Java Smart Card days are behind and the Java folks only care about server applications now. However having a requirement that all tests be inline seems a bit onerous. Do they inline images as well for testing BufferedImage and company?
I can see how an external file makes it a bit more of an "integration test" that would in effect be testing several moving pieces. I don't agree it would create more boiler plate or make things any less clear though. It makes it much easier to test many different complex and large inputs and to feed in new tests without needing to recompile. It's also easy to version control and introduce new tests with pathological cases. It presents a clear separation of test inputs from code - but maybe I can understand the counterargument. It does look neater to have everything together
https://www.jetbrains.com/help/idea/resource-files.html
makes it easy to embed arbitrary data with your code (or in this case, your tests, which are usually compiled separately) so no, you can easily test against files without access to a filesystem (beyond the access required to read the jars).
I still think having test cases with strings in the code is often a lot clearer personally.
(Unrelated: I wish more PLs made it easy to embed files within executables and access them with an OS-independent filesystem like API, it's often very useful.)
To be fair I'm not sure it does, really, per se. I know embedded Java is/was a thing, though, and it wouldn't suprise me if someone somewhere tortured their own personal fork into running a test suite on embedded stuff - if only for legacy support testing purpouses.
And poking around in that directory did make it clear some tests use files containing test vectors.
> I don't agree it would create more boiler plate
To get concrete again, I consider this boilerplate:
https://github.com/openjdk/jdk/blob/9fc518ff8cadbbb731a016d8...
https://github.com/openjdk/jdk/blob/9fc518ff8cadbbb731a016d8...
And this is merely (de)serializing pure unstructured strings without any kind of data format or failiable schema beyond charset encoding, making this a poster child for exiling test data to files (probably part of the reason why it was exiled to files!)
> It makes it much easier to test many different complex and large inputs and to feed in new tests without needing to recompile.
For bulk plain text I'd agree. For codebases with slow incremental builds, structured data might also benefit from being exiled for iteration speed. Or there can be benefits to skipping having a full developer environment.
But if incremental builds are fast (a worth goal), and if full developer environment is reasonably assumed (dev-focused unit/integration testing), compiler assistance with structured data is often more convenient, and has better error reporting for syntax errors etc. than what you'll get from many/most simple and straightforward uses of deserializers.
Escape sequences?
I wish a compose key was more widely available. It's actually moderately intuitive, on x windows, é is compose-e-'.
Are you “against” the link title or linked proposal?
I don't think anyone is advocating using non-ASCII characters, such as accented characters or emoji, in identifiers.
So this isn't even close to Rust's rule (all Rust source is UTF-8), let alone a declaration that Java is going to introduce data types with Korean names or new Russian method names on existing types, it's just hey, if any part of the JDK's own source code needs non-ASCII then it should be UTF-8, and if it doesn't need to be non-ASCII that's fine because ASCII is a strict subset of UTF-8.
> And yes, I know that there are codebases which are non-English
Non-English speaker here. Use English for identifiers and comments. Put magic strings in files other than source files, which is what you should be doing anyway if you want to seriously support i18n.
For languages limited to Latin-1 specifically, I always put a © in the header block as a canary for possible misinterpretation as Unicode or something else.
And I say it as someone whose native tongue is not English.
> Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` (yea I had to c/p that from somewhere). Searching for the former does not discover the latter.
If your search function can't handle different characters with the same meaning then it's already broken (e.g. usually you want search to be case insensitive). Likewise if similar-looking characters are an exploitable problem in your codebase then that problem already exists with e.g. l/I.
> Lastly, there is the issue of Intellisense, or whatever it is called in many languages. I have seen codebases in English with the aforementioned `é` in a function name. The only way for me to select that function name was with the arrow keys / mouse. I couldn't type it on my QWERTY.
In any decent system you can type it with compose-'-e like it looks like.
I'm all for making programmers dogfood the use of non-ascii characters (actually I'd like to see more use of non-unicode encodings, unless and until unicode stops screwing over Japanese). Maybe it'll give them more empathy for those of us who need to type things like this daily, and stop them doing things like flashy new packaging systems that break IME.
Can you say more about this? Are there still reasons to prefer shiftjis over unicode for japanese characters?
I think that CJK unification leads to awkwardness where some fonts will basically render some kanji the “Chinese way” or the “Japanese way”, so you can’t really have a good font that covers both at once. There’s a good aesthetic argument about not using the same font for both languages, but Unicode outright precludes it I think
Anecdotally, I've visited a group enthusiastically exploring the possibilities of the then-new CJK Unified Ideograph Extension B in Tokyo over a decade ago. The CJK issue mentioned here was a bigger problem then than now (most developers know how to handle it properly, e.g. with language declarations in HTML etc.).
People will use legacy encodings. That's just a fact of life. Sometimes out of inertia (i.e., it just works, so why bother changing it?), sometimes out of technical limitations of their pipeline, sometimes because of a lack of understanding, rarely out of principle. I'm stuck fixing an issue with some numb skull sending us SAML requests declared as UTF-8 containing some obsolete encoding instead, and this is for Dutch text! This just happens.
There is also nothing wrong with text in Shift-JIS, if that encoding encompasses all characters used.
If you change your PC or your phone locale to Japanese and access mostly Japanese website, the font display on most major OS works well. It's probably fair to assume that's a very common setup for Japanese users, and probably for Chinese and Korean users as well.
I think the CJK "problem" is more apparent for international users who don't have the locale setup or dealing with multiple languages plain text at the same time.
[0] https://w3techs.com/technologies/segmentation/tld-jp-/charac...
[1] https://w3techs.com/technologies/details/en-shiftjis
I don't really really know, but this random yahoo answers [0] page seems to agree with me that really you're gonna use UTF-8 for new stuff in general
One point that answer brings up that I hadn't thought of: older flipphones would have good shift-JIS support but not necessarily good unicode support
My guess is that "text as images" might partly be that, partly what you are saying in a legacy sense, and also partly for the same reason old sites in the US would do that: to make something look the way you want and you already have a photoshop render [0]: https://detail.chiebukuro.yahoo.co.jp/qa/question_detail/q14...
Yes. Unicode uses codepoints that are primarily for Chinese characters to represent Japanese characters that they consider equivalent, even when those Japanese characters have different appearences; as a result, Japanese text in unicode looks bad (readable, but ugly) unless displayed in a specifically Japanese font (in which case you'd have the converse problem of Chinese text looking wrong). The unicode consortium suggests various vaporware approaches to combat this, but the thing that actually works is keeping Japanese text in Japanese encodings and Chinese text in Chinese encodings. (Of course this means that you need to be able to display text from multiple encodings in the same page if you want to display both languages on the same page, but all of the unicode consortium vaporware fixes require you to build something equivalently complex, and you wouldn't even be able to test it by using strings from two western encodings since it would be specific to Japanese and Chinese)
In my reply here, I can type in english and russian at once. Привет, мир.
Yet, if I try to type chinese on one line, and japanese on the next, I cannot do it. Hacker news does not let me enter "lang" tags, so I can only type either the chinese or the japanese variant of a kanji.
> spans with their own encoding.
so yes it is about pages with multiple encodings. A span's smaller than a page!
The Unicode answer is "variation selectors" which are used for some historical variant kanji, but not for whole language switching. I suppose they could be used for that too though.
Mixing Shift_JIS and Big5 sounds like a recipe for corruption, but something similar was done in an old Russian and Japanese encoding: https://en.wikipedia.org/wiki/Shift_Out_and_Shift_In_charact...
RFC 2482 was a Unicode adaptation of this, but it was deprecated 12 years ago: https://www.rfc-editor.org/rfc/rfc6082.html
But usually people only care about their language, so it goes by the system UI language and it works out.
Sure. And once you're doing that you don't gain a lot of benefit from unicode AFAICS, because you have to track these spans of locale-specific text.
UTF-8 already has different codepoints for every other language, like I can type russian (День), greek (Ημέρα), english, even egyption hieroglyphs (𓀃) all of them in this one text field, and they all render right for both of us if we have a suitable font. If they don't render, it's unambiguous.
It's only chinese and japanese where I can type some characters, and depending on if you have a chinese or japanese font _first_ in your system, it might render wrong, and either way it's ambiguous to the computer without further metadata. The computer doesn't need metadata to know that "α" is "α", why does it need metadata to know if 直 is the chinese (http://www.hanzi5.com/bishun/76f4.html) or japanese (https://kanjivg.tagaini.net/viewer.html?kanji=%E7%9B%B4) character, two different looking characters that share the same codepoint.
Because Unicode is about the meaning of characters, not their appearance. If there were a context where the two different shapes of 直 have different meaning, Unicode would add a new codepoint to distinguish them. In fact "ɑ" without hook does have a separate codepoint because it is used in linguistics to mark a vowel different from "a".
Sometimes looking visually different doesn't matter because if you know how to write them by stroke order, you'll still be able to read it (which is how handwriting works, I think; I'm pretty bad at reading that…)
Hanzi simplification in the Mainland also complicated things, since I doubt they wanted to make all of those into different characters.
...what?
Can that be blocked with something like .editorconfig?
Yes, please mr I-Only-Speak-English, enlighten us. /s
> I feel that having UTF-8 in source code brings more risks than benefits. [0]
While this is a really fun theoretical attack, has this ever been encountered? A such patch would have to go trough a diff anyway before being accepted. And certainly a "comment" that looks like code has a high chance of getting rejected?
Does it make sense to restrict this feature (that is useful to everyone in the world, except those who only speak English) on account of a very theoretical risk?
> Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée`
Just in Italy, it can be written as: fidanzata, zita, morosa, ragazza, picciotta. Certainly the accent isn't the main problem here? Do you want to eliminate all synonyms? All languages other than English?
> I couldn't type it on my QWERTY.
Have you encountered this situation often? If so, just remap your keyboard.
I, for example, have a "w" key that is mostly used in shooter games, since the letter doesn't appear in my language.
> And yes, I know that there are codebases which are non-English, and not even Latin. Those are very valid concerns to which I admittedly have no answer to.
Perhaps allowing unicode in source files could be a solution.
How do you suggest I would achieve this in your non-unicode world?
[0] https://github.com/ltworf/vasttrafik-cli
edit: I had put the relevant emojis within the comment, as example, but it seems HN filtered them (angry emoji here)
Let's book this under "falsehoods programmers believe about Unicode".
You can try either the code below or simply using your browser's search function for this very page and type in the string `fiancee` and see all variants, accent-marked or not, highlighted as results.
Pardon me if I butchered the French grammar, I do not speak the language and can't easily verify whether I made a mistake in adapting the code from a year ago. <https://news.ycombinator.com/item?id=30405840>Edit: Source: https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html...
They want get rid of an encoding mess.
I don't know how one can be against this. Your code is probably saved in UTF-8 right now even if you don't use non-ASCII characters.
It's probably just that the JDK is a very old code base that was written from different OSes, without much thought on the encoding. Now, they want to standardize while making sure not to break anything.
What's more, the presence of non-ASCII characters in such a huge code base is almost inevitable so yeah, it matters if you want to avoid issues.
> Your code is probably saved in UTF-8 right now even if you don't use non-ASCII characters.
Nit: For a document that only contains ASCII characters, UTF-8 is identical to ASCII. So calling it UTF-8 is technically correct, but misleading -- you could equally well say the document is encoded using any of the other supersets of ASCII.
You'd better have your editor / tools set to use UTF-8 though in case some non-ascii characters end up in your file or else you might be in for some pain.
Take an example: "trouvé" instead of "found" as a variable name.
1- You're right. Most people won't be able to type "é", "à"... with their keyboard. Working on that code won't be easy. 2: What is the meaning of "trouvé" ? How people could figure out that variable name ? Average coder will search for english named functions or variable.
Same things about domain names btw.
PS : I am french.
I think the benefits could outweigh the harms, but some of it would indeed depend on good conventions being followed.
As for things like é, a function called fianc__e_accent_aigu__e would be far worse, and still a matter of convention. At least search engines could easily be configured to toggle accented letters as unaccented etc. And even nano has pretty good autocomplete facilities these days.
Where I find a big benefit with utf names is in reproducing standard mathematical formulae in code. But I agree that this implies a good way to input the required symbols that doesnt require you to search utf charmaps.
However, in UTF-8, 7 bits are what is available to distinguish between characters that take only one byte. U+0000 through U+007F, or 0x00 through 0x7F. Ones starting with the first byte 0x80 or higher are chonkier. https://www.youtube.com/watch?v=NQFMeyWVe3g
It's not ideal just for that reason but because 7 bits is a good amount to preserve for other characters, as well as the simplicity of only having to check one bit to tell it to look at the next byte to find what character it is. It could have been all but one to say that it isn't a single byte character. Then there would be 255 single byte characters and 256 two byte characters, minus however many you use to specify that there are more bytes.
It would be easy to make an encoding format where 0x00 or 0xFF (0b00000000 or 0b11111111) say to look at the next byte, but that would suck as a general text encoding format.