I was scrolling and scrolling, waiting for the author to mention the new methods, which of course every Android Dev had to migrate to at some point. And 99% of us probably thought how annoying this change is, even though it probably reduced the number of bugs for Turkish users :)
Unrelated, but a month ago I found a weird behaviour where in a kotlin scratch file, `List.isEmpty()` is always true. Questioned my sanity for at least an hour there... https://youtrack.jetbrains.com/issue/KTIJ-35551/
Ugh, I've had the exact same problem in a Java project, which meant I had to go through thousands and thousands of lines of code and make sure that all 'toLowerCase()' on enum names included Locale.ENGLISH as parameter.
As the article demonstrates, the error manifests in a completely inscrutable way. But once I saw the bug from a couple of users with Turkish sounding names, I zeroed in on it. And cursed a few times under my breath whoever messed up that character table so bad.
When I saw "Turkish alphabet bug", I just knew it was some version of toLower() gone horribly wrong.
(I'm sure there's a good reason, but I find it odd that compiler message tags are invariably uppercase, but in this problem code they lowercased it to go do a lookup from an enum of lowercase names. Why isn't the enum uppercase, like the things you're going to lookup?)
Everyone who has used Java has hit this before. Java really should force people to always specify the locale and get rid of the versions of the functions without locale parameters. There is so much hidden broken code out there.
I have always wondered why Turkey chose to Latinize in this way. I understand that the issue is having two similar vowels in Turkish, but not why they decided to invent the dotless I, when other diacritics already existed. Ĭ Î Ï Í Ì Į Ĩ and almost certainly a dozen other would've worked, unless there was already some significance to the dot in Turkish that's not obvious.
As a Turkish speaker who was using a Turkish-locale setup in my teenage years these kinds of bugs frustrated me infinitely. Half of the Java or Python apps I installed never run. My PHP webservers always had problems with random software. Ultimately, I had to change my system's language to English. However, US has godawful standards for everything: dates, measurement units, paper sizes.
When I shared computers with my parents I had to switch languages back-and-forth all the time. This helped me learn English rather quickly but, I find it a huge accessibility and software design issue.
If your program depends on letter cases, that is a badly designed program, period. If a language ships toUpper or a toLower function without a mandatory language field, it is badly designed too. The only slightly-better option is making toUpper and toLower ASCII-only and throwing error for any other character set.
While half of the language design of C is questionable and outright dangerous, making its functions locale-sensitive by all popular OSes was an avoidable mistake. Yet everybody did that. Just the existence of this behavior is a reason I would like to get rid of anything GNU-based in the systems I develop today.
I don't care if Unicode releases a conversion map. Natural-language behavior should always require natural language metadata too. Even modern languages like Rust did a crappy job of enforcing it: https://doc.rust-lang.org/std/primitive.char.html#method.to_... . Yes it is significantly safer but converting 'ß' to 'SS' in German definitely has gotchas too.
> Traditionally, ⟨ß⟩ did not have a capital form, and was capitalized as ⟨SS⟩. Some type designers introduced capitalized variants. In 2017, the Council for German Orthography officially adopted a capital form ⟨ẞ⟩ as an acceptable variant, ending a long debate.[4] Since 2024 the capital has been preferred over ⟨SS⟩.[5]
So this has been adopted since 2017. And Rust follows the unicode standard. It's not up to Rust, it's up to Unicode. And if that was what the mapping was in 2017, that's on Unicode, not Rust.
But I'm unsure if we can change the existing Unicode mapping without breaking backwards compatibility? I did learn that ẞ (the uppercase variant) is the same as SS (copy the SS and you'll see the ẞ highlighted, both upper and lowercase).
I am one of the maintainers is the Scala compiler, and this is one of the things that immediately jump to me when I review code that contains any casing operation. Always explicitly specify the locale. However, unlike TFA and other comments, I don't suggest `Locale.US`. That's a little too US-centric. The canonical locale is in fact `Locale.ROOT`. Granted, in practice it's equivalent, but I find it a little bit more sensible.
Also, this is the last remaining major system-dependent default in Java. They made strict floating point the default in 17; UTF-8 the default encoding some versions later (21?); only the locale remains. I hope they make ROOT the default in an upcoming version.
FWIW, in the Scala.js implementation, we've been using UTF-8 and ROOT as the defaults forever.
Ramazan Çalçoban sent his estranged wife Emine the text message:
Zaten sen sıkışınca konuyu değiştiriyorsun.
"Anyhow, whenever you can't answer an argument, you change the subject."
Unfortunately, what she thought he wrote was:
Zaten sen sikişınce konuyu değiştiriyorsun.
"Anyhow, whenever they are fucking you, you change the subject."
This led to a fight in which the woman was stabbed and died and the man committed suicide in prison.
Wouldn't at least the first issue be solved by using Unicode case folding instead of lowercase? Python, for example, has separate .casefold() and .lower() methods, and AFAIK casefold would always turn I into i, and is much more appropriate for this use case.
In C# programming, you are able to specify a culture every time you call a function such as numbers <-> strings, or case conversion. Or you specify the "Invariant Culture", which is basically US English. But the default culture is still based on your system's locale, you need to explicitly name the invariant culture everywhere. Because it involves a lot of filling in parameters for many different functions, people often leave it out, then their code breaks on systems where "," is the decimal separator.
You can also change the default culture to the invariant culture and save all the headaches. Save the localized number conversion and such for situations where you actually need to interact with localized values.
Interesting one. That and and relying on system character encodings is a source of subtle bugs. I've been bitten by that many times with e.g. XML parsing in the past. Modern Kotlin thankfully has very few (if any) places left where this can happen. Kotlin has parameters with default values. So anything that relies on a character encoding usually has a parameter of encoding set to UTF-8.
The bug here was the default Java implementation that Kotlin uses on JVM. On kotlin-js both toLowerCase() and lowercase() do exactly the same thing. Also, the deprecation mechanism in Kotlin is kind of cool. The deprecated implementation is still there and you could use it with a compiler flag to disable the error.
@Deprecated("Use lowercase() instead.", ReplaceWith("lowercase(Locale.getDefault())", "java.util.Locale"))
@DeprecatedSinceKotlin(warningSince = "1.5", errorSince = "2.1")
@kotlin.internal.InlineOnly
public actual inline fun String.toLowerCase(): String = (this as java.lang.String).toLowerCase()
/**
* Returns a copy of this string converted to lower case using Unicode mapping rules of the invariant locale.
*
* This function supports one-to-many and many-to-one character mapping,
* thus the length of the returned string can be different from the length of the original string.
*
* @sample samples.text.Strings.lowercase
*/
@SinceKotlin("1.5")
@kotlin.internal.InlineOnly
public actual inline fun String.lowercase(): String = (this as java.lang.String).toLowerCase(Locale.ROOT)
In September 2020, nearly a year after the coroutines bug had been fixed and forgotten
[…]
When they came to fix this issue, the Kotlin team weren’t leaving anything to chance. They scoured the entire compiler codebase for case-conversion operations—calls to capitalize(), decapitalize(), toLowerCase(), and toUpperCase()”
Bloody late, I would say. If something like this happened in OpenBSD, I think they would have done that, and more (the article doesn’t mention tooling to detect the introduction of new similar bugs ofof adding warnings to documentation), at the first spotting of such a bug.
How come no reviewer mentioned such things when the first fix was reviewed?
Wow this is bad. Even for a language like Java having vanilla strings as some sort of enum like value, and then even going further and downcasing them is a 100% bugmagnet waiting for the kaboom.
> If capitalize() was an ambiguous name, what should its replacement be called? Can you think of a name that describes the function’s behaviour more clearly?
In c#, setting every letter to its uppercase form is ToUpper, and I think capitalise is perfectly reasonable for setting the first character. I'm not sure I've ever referred to uppercasing a string as capitalising it
For a while when I made Minecraft mods, I had my test environment set to Turkish for this exact reason (there's some simple command-line parameter you can pass to the JVM). Half the other mods installed in this environment would have broken textures, but mine never did since I tested it.
I knew from the headline that this would be the Turkish I thing, but I couldn't fathom why a compiler would care about case-folding. "I don't know Kotlin, but surely its syntax is case-sensitive like all the other commonly used languages nowadays?"
> The code is part of a class named CompilerOutputParser, and is responsible for reading XML files containing messages from the Kotlin compiler. Those files look something like this:
26 comments
[ 3.0 ms ] story [ 47.4 ms ] threadUnrelated, but a month ago I found a weird behaviour where in a kotlin scratch file, `List.isEmpty()` is always true. Questioned my sanity for at least an hour there... https://youtrack.jetbrains.com/issue/KTIJ-35551/
As the article demonstrates, the error manifests in a completely inscrutable way. But once I saw the bug from a couple of users with Turkish sounding names, I zeroed in on it. And cursed a few times under my breath whoever messed up that character table so bad.
(I'm sure there's a good reason, but I find it odd that compiler message tags are invariably uppercase, but in this problem code they lowercased it to go do a lookup from an enum of lowercase names. Why isn't the enum uppercase, like the things you're going to lookup?)
When I shared computers with my parents I had to switch languages back-and-forth all the time. This helped me learn English rather quickly but, I find it a huge accessibility and software design issue.
If your program depends on letter cases, that is a badly designed program, period. If a language ships toUpper or a toLower function without a mandatory language field, it is badly designed too. The only slightly-better option is making toUpper and toLower ASCII-only and throwing error for any other character set.
While half of the language design of C is questionable and outright dangerous, making its functions locale-sensitive by all popular OSes was an avoidable mistake. Yet everybody did that. Just the existence of this behavior is a reason I would like to get rid of anything GNU-based in the systems I develop today.
I don't care if Unicode releases a conversion map. Natural-language behavior should always require natural language metadata too. Even modern languages like Rust did a crappy job of enforcing it: https://doc.rust-lang.org/std/primitive.char.html#method.to_... . Yes it is significantly safer but converting 'ß' to 'SS' in German definitely has gotchas too.
What were they supposed to do?
According to Wikipedia:
> Traditionally, ⟨ß⟩ did not have a capital form, and was capitalized as ⟨SS⟩. Some type designers introduced capitalized variants. In 2017, the Council for German Orthography officially adopted a capital form ⟨ẞ⟩ as an acceptable variant, ending a long debate.[4] Since 2024 the capital has been preferred over ⟨SS⟩.[5]
Source: https://en.wikipedia.org/wiki/%C3%9F#:~:text=Traditionally%2...
So this has been adopted since 2017. And Rust follows the unicode standard. It's not up to Rust, it's up to Unicode. And if that was what the mapping was in 2017, that's on Unicode, not Rust.
But I'm unsure if we can change the existing Unicode mapping without breaking backwards compatibility? I did learn that ẞ (the uppercase variant) is the same as SS (copy the SS and you'll see the ẞ highlighted, both upper and lowercase).
Also, this is the last remaining major system-dependent default in Java. They made strict floating point the default in 17; UTF-8 the default encoding some versions later (21?); only the locale remains. I hope they make ROOT the default in an upcoming version.
FWIW, in the Scala.js implementation, we've been using UTF-8 and ROOT as the defaults forever.
https://gizmodo.com/a-cellphones-missing-dot-kills-two-peopl...
You can also change the default culture to the invariant culture and save all the headaches. Save the localized number conversion and such for situations where you actually need to interact with localized values.
The bug here was the default Java implementation that Kotlin uses on JVM. On kotlin-js both toLowerCase() and lowercase() do exactly the same thing. Also, the deprecation mechanism in Kotlin is kind of cool. The deprecated implementation is still there and you could use it with a compiler flag to disable the error.
In September 2020, nearly a year after the coroutines bug had been fixed and forgotten
[…]
When they came to fix this issue, the Kotlin team weren’t leaving anything to chance. They scoured the entire compiler codebase for case-conversion operations—calls to capitalize(), decapitalize(), toLowerCase(), and toUpperCase()”
Bloody late, I would say. If something like this happened in OpenBSD, I think they would have done that, and more (the article doesn’t mention tooling to detect the introduction of new similar bugs ofof adding warnings to documentation), at the first spotting of such a bug.
How come no reviewer mentioned such things when the first fix was reviewed?
Also, why are they using Locale.US, and not Locale.ROOT (https://docs.oracle.com/javase/8/docs/api/java/util/Locale.h...)?
In c#, setting every letter to its uppercase form is ToUpper, and I think capitalise is perfectly reasonable for setting the first character. I'm not sure I've ever referred to uppercasing a string as capitalising it
> The code is part of a class named CompilerOutputParser, and is responsible for reading XML files containing messages from the Kotlin compiler. Those files look something like this:
"Oh."
"... Seriously?"
As if I didn't hate XML enough already.