What would that even mean? A forked Java's strings could insist their implementation is UTF-8 encoded bytes, but that's strings, you're talking about char. Do you want char just to be a byte, like in C ? But Java already has a byte type.
A 32 bit "wide char" is both very wasteful under normal use and mostly useless for grapheme cluster (what business people probably actually mean by "character") handling.
In Go, strings are UTF8, but when you iterate over UTF8 characters like this:
for _, c := range str {
}
variable "c" is 32-bit. So it's not really wasteful, it fits in a register. It can be wasteful if you declare a slice of characters, like []rune, but I don't remember ever seeing it in practice.
In Java strings are stored in the heap either as compact strings for cases where everything fits in 8 bits, or as UTF-16. Iterating over them using chars will give you those 16 bit things, but there are other methods you can use to iterate over them as code points, just like you can in Go. The old char behaviour may not be what you really want, but it can’t be changed without breaking existing code.
Right. But this thread is about "OK, but what if we forked Java and could make breaking changes like that."
Plus, I'd bet that most (but of course, not all) code that iterates over a Java String using UTF-16 chars is broken; failing to properly handle surrogate pairs, and should instead use those iterate-over-them-as-codepoints/UTF-32 methods.
Most of the things you could do with a single codepoint aren’t valid unless you search for a group of unicode.IsMark(c) and process them together with the previous codepoint. Now I’m concerned that I don’t see that in the standard library.
You are correct that there are lots of things that you can't do without grouping codepoints into grapheme clusters. And it is unfortunate that the Go standard library doesn't have utilities for grouping codepoints into grapheme clusters; the Go team's response to that is largely "don't do those things; even if we had utilities for grouping codepoints into grapheme clusters, those things would be hard to do and you are likely to get it wrong; human-level text is complicated, so treat it as opaque as much as you can."
But also, there are still lots of things you can do with just codepoints. You can tokenize. You can trim whitespace. You can do case-conversions and comparisons. You can transcode to a different encoding (whether that be a different Unicode encoding, or something like a JSON where some codepoints will need to be \uXXXX-encoded).
So yeah, codepoints aren't always what you want, but they're what you want more often than Java's "usually codepoints, but sometimes surrogate pairs". Especially because I suspect that very few Java programmers bother to test their code with surrogate pairs.
I agree that iterating over UTF-16 code units can never be less painful than iterating over UTF-32 codepoints, but to be fair Java (like Win32) comes from more than fourteen years earlier when we still hoped UCS-2 would be enough for the world.
* If you're not really serious about using strings, why would you care?
* If you're really serious about actually doing things with Strings, you are gonna need to reimplement International Components for Unicode (or something similar in scope), and you have a free, libre and already working one for the "UTF-16 String Java" already. You don't have a working one for your custom fork with custom String handling.
If I had my druthers, String would be an interface with UTF-8, UTF-16, and UTF-32 (and UTF-7 on April Fool's Day) implementation classes. Then add Byte1, Byte2, and Byte4 as _unsigned_ primitives. Maybe have a wrapper class CodePoint to allow abstracting over all of them.
Java can't even bother adding unsigned integers or integer types smaller than 32-bits. How long have we been waiting for non-boxed primitives and runtime generics (Valhalla)? C# has had all (except generics) since day one. New "char" types aren't happening anywhere in the near future, unfortunately.
No, the `least` typedefs are irrelevant aliases for the exact type these days (except supposedly some 32-bit-only DSPs exist though. But 9-bit computing has been obsolete for longer than most of us have been alive, despite talking about it).
The `fast` typedefs are effectively programmer-opted-in to promotion to register size or something. But compiler optimizations often do the same thing on a per-operation level (signed overflow is UB, unsigned overflow masking can be deferred for addition/subtraction/multiplication and bitwise ops (though some can change the mask)).
It’s quite easy to go down an already discovered path and learn from its mistakes.
Also, Valhalla is said to be a 7 PhD combined-difficulty feat to pull off in a backwards compatible way, so it takes time. Java actually cares about that, instead of repeatedly pulling the plug on the whole platform like Microsoft likes to do.
Java predates Unicode 3.1 (I think I got the version correct: but basically when Java was created, Unicode had less than 65536 codepoints). So Java had 16 bits chars from the get go and to this day is still backward compatible with earlier Java code.
You can also ask to get the "codepoint", which returns you an int.
But, yup, the Java char primitive is 16 bits.
It's a SNAFU hard to understate. SNAFU doesn't even being to describe it. But to be honest Unicode in itself is probably the biggest SNAFU of our entire field.
> But to be honest Unicode in itself is probably the biggest SNAFU of our entire field.
Explain.
EDIT: Oh! I construed this as a negative sentiment about Unicode, but perhaps it was just: "Yeah, it's complicated because language is complicated". Thanks, responders!
It was always going to be messy to create something that covered all of the writing systems in the world, but things like this didn't help, and is probably what they're referring to:
I can't explain what the previous poster was thinking, but Unicode is frustrating to work with. The problem is, that frustration just replicates the reality of the human writing systems Unicode captures. Nobody invented weird nonsense to make your life harder in Unicode, millions of humans developed a variety of weird nonsense to make their lives easier, over thousands of years and Unicode captures most of it.
Case for example, why the heck would there be exactly two versions of A, one of which usually only appears at the start of words, and not always? Other people's squiggles don't do that, but notably several European writing systems (Latin and Cyrillic) do. People who use those systems think ABCDE and abcde are somehow "similar" even though they don't seem very similar at all. Then they're surprised other people don't do that.
> Nobody invented weird nonsense to make your life harder in Unicode
The current state of emoji, unfortunately, makes this no longer true. (That being said, it's also designed in a way that's mostly ignorable if you don't need it. But only mostly.)
Emoji is not an additional complexity, it builds on top of the same primitives like Chinese characters and whatnot. Unless you claim supporting non-european languages is unnecessary, then emojis are a perfectly good “making use of” of unused regions.
> Emoji... builds on top of the same primitives like Chinese characters
It does not, and if you think it does it's obvious you didn't read UAX 29. GB11 is the most complex rule and only applies to emoji (well, and other Extended_Pictographics if you don't want to call the legacy pictographics emoji, but no Chinese).
Seriously, the zoo of text encodings that reigned before widespread Unicode uptake was a pain the ass once you had data that went beyond 7 bits, especially for non-Roman scripts.
> the complexity of Unicode is only a mirror of the complexity of human scripts.
Oh please, Unicode didn't start out as an encoding covering human scripts - it was too late for that. It started out as an encoding to end all already existing encodings and that was a shit show with all kinds of weird and non printable symbols to begin with. It also didn't get better when Unicode started to add all kinds of shit like combining emojis into the mix.
> Unicode didn't start out as an encoding covering human scripts
> It started out as an encoding to end all already existing encodings
And those other encodings were encodings for a variety of human scripts.
Re: emojis, well, they're not that different from Kanjis, not really -- both are pictographs after all, one just has much more modern functionality (e.g., color).
In addition to what the other reply posted, Java has had compact strings for a while now[1], which uses 8-bit `byte` arrays for `String`s, and comes in handy for ISO-8859-1/Latin-1 strings to use a single byte per character.
One annoyance with that approach is that a single non-latin-1 codepoint is enough to double the size of the String; if it were an UTF-8 String, it would only add a couple of extra bytes.
At the expense of O(n) index access. At least Java and JS have compact strings. C# is always UTF-16 because they exposed too much implementation detail to be able to transparently do the compact string optimization.
The fact of the matter is that there are millions of places with loops like for(...) {charAt(i)} so it is by definition useful in practice. If Java had pushed iterators more strongly, perhaps it would be possible to make the switch.
OpenJDK seems pretty averse to including special case optimizations, otherwise I'm sure we would have intrinsics at least for ArrayList/HashMap containing boxed primitives. AFAIK they are still noticeably slower than plain arrays or manually inlined implementations.
For the most basic case you wouldn't even need JIT support - just keep a cache field of the last index accessed and where it ended and decode based on that.
The model of a Java String, as presented via the API, is as an array of 16-bit unsigned integers. Strings use a UTF-16-like encoding to represent non-BMP Unicode characters. The main difference from UTF-16 is that illegal sequences of 16-bit values (such as unpaired surrogates) are permitted.
The "UTF-8 by Default" refers to the default character set used for decoding and encoding, when Java Strings are read and written. Prior to JEP 400, the default character set was "platform specific." It was often UTF-8, but sometimes it was not, in which case hijinks ensued.
As noted elsewhere the internal representation of Java Strings is either UTF-16-like or is Latin-1, if all the characters of the String can be encoded in Latin-1. We think about using UTF-8 as an internal representation constantly. Doing so would potentially save space and reduce decoding/encoding overhead.
Unfortunately doing this is difficult. The key issue is that tons of code out there treats a String as a UTF-16 array (since that's what's in the API) so there are coding patterns like this:
for (int i = start; i < end; i++) {
char ch = str.charAt(i);
// do something with ch
}
If the internal representation were UTF-8, simplistically, `charAt(i)` would change from O(1) to O(N). Of course there are cleverer things one could do, such as converting to UTF-16 lazily. Now `charAt()` allocates memory and has variable latency. Well then partial conversion could be done and the current iteration point could be cached. This might work, but now String has state, and it's thread-specific as well. Etc.
> If the internal representation were UTF-8, simplistically, `charAt(i)` would change from O(1) to O(N).
16 bits is not enough space to store all unicode characters. Does that mean that java's charAt won't join chars for codepoints over U+FFFF?
Edit: Yeah, that's exactly what happens[1]. That's not a very nice implementation at all .
> Because 16-bit encoding supports 2^16 (65,536) characters, which is insufficient to define all characters in use throughout the world, the Unicode standard was extended to 0x10FFFF, which supports over one million characters. The definition of a character in the Java programming language could not be changed from 16 bits to 32 bits without causing millions of Java applications to no longer run properly. To correct the definition, a scheme was developed to handle characters that could not be encoded in 16 bits.
> The characters with values that are outside of the 16-bit range, and within the range from 0x10000 to 0x10FFFF, are called supplementary characters and are defined as a pair of char values.
I guess it hasn't came out because I don't personally work with a lot of user input or emojis in Java. When I do I'm normally using Python, and that uses Unicode transparently. Good to know.
Python also has this problem which you only find out once you're bitten by it. The encoding of all the built-in file manipulation machinery is platform specific so you either have to do `sys.setdefaultencoding("utf-8")` or add the encoding to every open call that uses text mode.
So kudos to every language that axes platform dependent encodings. Honestly platform dependent anything is so annoying to deal with. I would love if languages make it painfully explicit when you depend on platform defaults.
Even after Py2 -> Py3, the platform-dependent decoding sitting around for _so long_ was so annoying. Just constantly running into issues with file reads because servers were configured a certain way. "Just specify the encoding" is like... fine, but annoying when it's some third party script I just can't use.
I'm at least glad we had some guarantees from being Python 3.
Does Java have any sort of precedent for dynamically swapping out implementations at the first call of charAt to a more specialized string class? I have to imagine there's a lot of string stuff that _doesn't_ do this at all.
78 comments
[ 3.7 ms ] story [ 147 ms ] threadBetter late than never.
> Specify UTF-8 as the default charset without providing any means to change it — The compatibility impact of this change would be too high.
Nah, nobody is getting this correct right now. You should've swung for the fences.
I'm guessing it's not worth it, even aside from legal costs.
What would that even mean? A forked Java's strings could insist their implementation is UTF-8 encoded bytes, but that's strings, you're talking about char. Do you want char just to be a byte, like in C ? But Java already has a byte type.
Plus, I'd bet that most (but of course, not all) code that iterates over a Java String using UTF-16 chars is broken; failing to properly handle surrogate pairs, and should instead use those iterate-over-them-as-codepoints/UTF-32 methods.
https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...
You are correct that there are lots of things that you can't do without grouping codepoints into grapheme clusters. And it is unfortunate that the Go standard library doesn't have utilities for grouping codepoints into grapheme clusters; the Go team's response to that is largely "don't do those things; even if we had utilities for grouping codepoints into grapheme clusters, those things would be hard to do and you are likely to get it wrong; human-level text is complicated, so treat it as opaque as much as you can."
But also, there are still lots of things you can do with just codepoints. You can tokenize. You can trim whitespace. You can do case-conversions and comparisons. You can transcode to a different encoding (whether that be a different Unicode encoding, or something like a JSON where some codepoints will need to be \uXXXX-encoded).
So yeah, codepoints aren't always what you want, but they're what you want more often than Java's "usually codepoints, but sometimes surrogate pairs". Especially because I suspect that very few Java programmers bother to test their code with surrogate pairs.
https://manishearth.github.io/blog/2017/01/14/stop-ascribing... and the followup post https://manishearth.github.io/blog/2017/01/15/breaking-our-l...
* If you're not really serious about using strings, why would you care?
* If you're really serious about actually doing things with Strings, you are gonna need to reimplement International Components for Unicode (or something similar in scope), and you have a free, libre and already working one for the "UTF-16 String Java" already. You don't have a working one for your custom fork with custom String handling.
... other than short and byte, what are you missing? nibble?
https://docs.oracle.com/javase/tutorial/java/nutsandbolts/da...
Java's got byte (8 bits) and short (16 bits)? (Still signed, though, of course.)
Unless those are actually padded to 32 bits in the VM or something.
They are.
And since Java doesn't expose pointer guts, struct padding is purely a runtime optimization concern, not a language concern.
The `fast` typedefs are effectively programmer-opted-in to promotion to register size or something. But compiler optimizations often do the same thing on a per-operation level (signed overflow is UB, unsigned overflow masking can be deferred for addition/subtraction/multiplication and bitwise ops (though some can change the mask)).
Also, Valhalla is said to be a 7 PhD combined-difficulty feat to pull off in a backwards compatible way, so it takes time. Java actually cares about that, instead of repeatedly pulling the plug on the whole platform like Microsoft likes to do.
[I mean,make the strings use utf-8 instead of utf-16, which is a bit different than TFA]
You can also ask to get the "codepoint", which returns you an int.
But, yup, the Java char primitive is 16 bits.
It's a SNAFU hard to understate. SNAFU doesn't even being to describe it. But to be honest Unicode in itself is probably the biggest SNAFU of our entire field.
Explain.
EDIT: Oh! I construed this as a negative sentiment about Unicode, but perhaps it was just: "Yeah, it's complicated because language is complicated". Thanks, responders!
https://en.wikipedia.org/wiki/Han_unification
Case for example, why the heck would there be exactly two versions of A, one of which usually only appears at the start of words, and not always? Other people's squiggles don't do that, but notably several European writing systems (Latin and Cyrillic) do. People who use those systems think ABCDE and abcde are somehow "similar" even though they don't seem very similar at all. Then they're surprised other people don't do that.
The current state of emoji, unfortunately, makes this no longer true. (That being said, it's also designed in a way that's mostly ignorable if you don't need it. But only mostly.)
I understand why emoji is as complicated as it is. I don't think it was a good idea to bake that complication into Unicode.
It does not, and if you think it does it's obvious you didn't read UAX 29. GB11 is the most complex rule and only applies to emoji (well, and other Extended_Pictographics if you don't want to call the legacy pictographics emoji, but no Chinese).
Please stop taking dumps on Unicode. It does nothing for actually getting it implemented and implemented well.
Oh please, Unicode didn't start out as an encoding covering human scripts - it was too late for that. It started out as an encoding to end all already existing encodings and that was a shit show with all kinds of weird and non printable symbols to begin with. It also didn't get better when Unicode started to add all kinds of shit like combining emojis into the mix.
> It started out as an encoding to end all already existing encodings
And those other encodings were encodings for a variety of human scripts.
Re: emojis, well, they're not that different from Kanjis, not really -- both are pictographs after all, one just has much more modern functionality (e.g., color).
[1] https://openjdk.org/jeps/254
Ah yes, that feature which means we must have 2x memory overhead and/or dynamic dispatch, yet is also totally useless in practice.
For the most basic case you wouldn't even need JIT support - just keep a cache field of the last index accessed and where it ended and decode based on that.
The "UTF-8 by Default" refers to the default character set used for decoding and encoding, when Java Strings are read and written. Prior to JEP 400, the default character set was "platform specific." It was often UTF-8, but sometimes it was not, in which case hijinks ensued.
As noted elsewhere the internal representation of Java Strings is either UTF-16-like or is Latin-1, if all the characters of the String can be encoded in Latin-1. We think about using UTF-8 as an internal representation constantly. Doing so would potentially save space and reduce decoding/encoding overhead.
Unfortunately doing this is difficult. The key issue is that tons of code out there treats a String as a UTF-16 array (since that's what's in the API) so there are coding patterns like this:
If the internal representation were UTF-8, simplistically, `charAt(i)` would change from O(1) to O(N). Of course there are cleverer things one could do, such as converting to UTF-16 lazily. Now `charAt()` allocates memory and has variable latency. Well then partial conversion could be done and the current iteration point could be cached. This might work, but now String has state, and it's thread-specific as well. Etc.16 bits is not enough space to store all unicode characters. Does that mean that java's charAt won't join chars for codepoints over U+FFFF?
Edit: Yeah, that's exactly what happens[1]. That's not a very nice implementation at all .
> Because 16-bit encoding supports 2^16 (65,536) characters, which is insufficient to define all characters in use throughout the world, the Unicode standard was extended to 0x10FFFF, which supports over one million characters. The definition of a character in the Java programming language could not be changed from 16 bits to 32 bits without causing millions of Java applications to no longer run properly. To correct the definition, a scheme was developed to handle characters that could not be encoded in 16 bits.
> The characters with values that are outside of the 16-bit range, and within the range from 0x10000 to 0x10FFFF, are called supplementary characters and are defined as a pair of char values.
[1] https://docs.oracle.com/javase/tutorial/i18n/text/unicode.ht...
So kudos to every language that axes platform dependent encodings. Honestly platform dependent anything is so annoying to deal with. I would love if languages make it painfully explicit when you depend on platform defaults.
I'm at least glad we had some guarantees from being Python 3.
It’s Java, right? Everything is variable latency.
But language wars are almost always meaningless, both have their places and they definitely don’t occupy the same niche.