Linux systems are more than just a kernel. Does every program across the system now understand that direct comparison of strings doesn't work on filenames? Does every regex touching or analyzing filenames know that they must now be case insensitive? How long until people stop getting bitten by this issue?
That aside, inferring semantic meaning based on a few cultures at this moment in time is a dumb choice. There are tons of such language-specific semantics that could be implemented and the result is needing to memorize all those rules about when it does and doesn't actually matter (and arguing that only certain cultures and languages should have their own semantics encoded in the OS is its own problem).
This was discussed to death on the kernel mailing lists, you should go read them.
The principal question is whether the tool is more important, or the end user. Why did I pay for this machine if it weren't intended to facilitate me? That's the bottom line with most of these kinds of technical "correctness" arguments
And as for whether userspace should catch up, thanks to OS X for the most part that already happened a long time ago for a ton of open source packages
Could you provide pointers to these discussions? As a technical person, case sensitive filesystems have been a loss both from a programmer's and also from an end user's perspective.
I also side with this view, namely that this is something that would be better placed in the userspace rather than the kernel, which really doesn't need more complexity for things that add so little value (negative value to some).
While Linus kicks off about things, he doesn't tend to outright refuse things. I don't think he sees himself as the gatekeeper of the kernel and that is evident in the way the kernel developed right from the beginning. That has attracted criticism from the likes of Ken Thompson who thought that too much crappy code was allowed into linux.
|I've looked at the source and there are pieces that are good and pieces that are not. A whole bunch of random people have contributed to this source, and the quality varies drastically
> Why did I pay for this machine if it weren't intended to facilitate me?
I happen to agree with the idea that the filename should be a dumb blob of bytes and the kernel should not do case folding, as it is the wrong layer for that, eg. the user can change their language but it won't update what has been written to the disk in thousands or millions of places where you could suddenly have a filename collision somewhere based on those rules changing.
But, I do hope you get that refund for your Linux.
The fact that if one has two files, say “test{invalid bytes}.txt” and test{other invalid bytes}.txt”, both have replacement characters inserted at the same spot and would decode to the same codepoints.
How does the UI framework act when you set a label to such payload? How does your web browser act when it sees it in HTML? I have found working on apps that see a lot of usage in varied markets that as much as we wish to see the best and ideal conditions, malformed utf-8 surfaces in the real world pretty often.
I understand that NTFS has its own case folding table which is written once when the volume is formatted. This does seem to have stood the test of time and enormous usage so maybe it is not such a poor idea.
That doesn't sound great if somebody formats your USB stick in Turkey and suddenly speakers of western European languages can observe 'i' as case sensitive.
Out of interest, what special-case manipulation? I generally treat file paths as opaque `\` separated strings (or even as a single blob if I don't need to parse it). I'm uncertain why I'd want to treat them specially. I'll leave that to the OS.
Take for example a file on NTFS.. the filenames can be UTF-16 (they support 16 bit chars under the hood).. but they also might not be valid UTF-16. When you access the file by the filename, you now have the potential to use all of the wide function calls (e.g. _wfopen) or the the ansi versions (e.g. fopen).
Arguably the biggest difference that I notice when using linux as a desktop environment as an end-user is that it trusts "you", the [sometimes root] user, to a far greater extent than other operating systems. It is for this reason that I enjoy it.
It also means that if you want to do something highly annoying and unusual, you can – and arguably case-insensitive filesystems are a subset of that.
I agree that case-insensitive filesystems should be an esoteric feature, but given that they’re the default on Windows and macOS, it should definitely be a well-supported option on Linux for the sake of compatibility.
I've set the Samba shares on my NAS to be case-sensitive, as making them case-insensitive slows down directory access by orders of magnitude.
I've been running this for years accessing them both from Linux desktops and Windows desktops, and only once have I had an issue that required me to manually rename something on the NAS.
This makes sense as most applications don't care about the filename, and will just use what you supply, or generate one and use that string all over.
> That's the bottom line with most of these kinds of technical "correctness" arguments
The problem is emergent behavior. We can create any number of features, but we have a really hard time testing all the available configurations that result. Engineers rely on simplicity as a way of warding off this particular problem, because the other bottom line is people don't want to pay for a system that loses or invalidates the work they've put into it.
This feature doesn't facilitate you at all. It's a historical mistake that macOS and Windows have preserved, and refined a bit over the years.
The only real purpose of this is for easier/faster compatibility with software developed and tested on macOS and Windows, which has accidental case inconsistency in file name references in the code, which happens to work fine on macOS and Windows.
TFA could have made that argument, but it didn't, it made incorrect arguments instead. For example, a user might type in a lower-case name for a file one time, and a capitalized name for a file another time, and intend to access the same file. But what user is typing in a whole filename the second time? They're picking it from a list, or if they're very advanced, using completion in a terminal. TFA also mentions non-English languages, in the context of unicode normalization ... but non-western-european languages won't be handled correctly by any universal case-folding algorithm anyway, with Turkish being the most common example.
The whole strategy just doesn't work out well. Many low-level filesystem developers have known this for over 20 years. It doesn't work better for anyone, but non-technical people just aren't aware of why or how it increases complexity and costs, and reduces performance and reliability.
Ensuring filenames don't get destroyed by an OS that refuses to understand a given language. Case is a complicated mess once you leave ASCII, and that's partially because ASCII is lying to you about how English case works: Yes, the English language has title case, and ASCII conflating it with upper-case does not negate that.
Move on to most anywhere else and the notion that it's fast, reliable, and safe to convert case gets lost in the realities of human writing systems.
In a world of ASCII, maybe. But fixing one problem for a small group of people is a giant can of worms for the rest of the world. Normalization of compound characters, exotic character sets, emoji, different classes of upper/lower-case letters, normalization of compound what-not.
And even then it still doesn't fix the issue outside of the world of ASCII. A filename written in hiragana and katakana is logically the same to the end-user, but they are still distinct. Simplified and traditional Chinese, Hangul and romaja, pinyin, devanagari, thai, and the list goes on.
Case-insensitive filenames fixes nothing, but breaks everything. There is only one sensible thing to do with arbitrary user-input, and that is to leave it be.
How would one go about implementing case-insensitive path lookup against a SMB share containing a few million files from userspace?
It must be in the kernel. Else implementations of stuff with human-derived semantics should move out of the kernel, but moving SMB into userspace would of course be ridiculous
So it complicates any mapping of filenames to data structures in the kernel (all 3 of them?), big deal. Every popular desktop operating system supports it, and basically the only reason Linux does not is a mixture of FUD and fear that we may need to update the code at some point due to changes in human culture, the horror!
Meanwhile, typing "cd music" in a terminal need not print "Command not found" when there is clearly a folder named "Music", like the $3k worth of gear in front of me had the complexity of some 1950s b-movie scifi robot
> How would one go about implementing case-insensitive path lookup against a SMB share containing a few million files from userspace?
>
> It must be in the kernel. Else implementations of stuff with human-derived semantics should move out of the kernel, but moving SMB into userspace would of course be ridiculous
Case-insensitive path lookups on SMB happens in the server (usually samba), in user-space. The client is also usually in user-space through FUSE or the client libs, but CIFS of course exists as a kernel-mode alternative.
And as I have written elsewhere, sure, "music" vs. "Music" is simple when you live in an ASCII world. Trying to be smart with user input only causes trouble for the rest of us. Hiragana and katakana is also logically the same, and on a kana keyboard a similar typo. Simplified and traditional chinese is also logically the same.
> Every popular desktop operating system supports it
> How would one go about implementing case-insensitive path lookup against a SMB share containing a
> few million files from userspace? It must be in the kernel.
SMB in the kernel is a rather dangerous game. Not saying it can't be done (and I know Samsung is doing it :-), but it's a significantly more complex protocol than even NFSv4 (which also really shouldn't be in the kernel either). For complexities sake, userspace is easier IMHO (much easier to debug).
Also, you seem to be of the opinion that code in the kernel must be "magic", in that it can do things that user-space can't. The "missing case" lookup problem still exists for kernel code as it does for user-space code.
Looking up "foo" in all case variations in a file containing a few million files still has to be done by search in the kernel as it does in user-space. It's going to be slow there too unless you provide a case-insensitive indexing mechanism.
Now the kernel offers easier opportunities to do things like directory content caching, which currently aren't exposed to user space via any API - but once you have to do something like directory content caching it's also possible to expose that feature to userspace via an API.
> Else implementations of stuff with human-derived semantics should move out of the kernel, but moving SMB into userspace would of course be ridiculous
Which has caused countless problems with interpreted languages and cross platform functionality. Things lime ruby in OSX will gladly less you mangle your include strings on OSX, which causes a "works fine in Dev" problem. I'm not a fan of case insensitive filesystems because I have to manage services.
I disagree. The error is to consider paths as a high level information, that the user has to know about, rather of what they really are, a low level information, that potentially the user never sees (for example consider mobile operating systems like Android/iOS).
In practice the case insensitive thing if we want to call it should be implemented more high level, in the file manager, rather than in the filesystem itself. That is even what newer versions of Windows/NTFS do! Recent versions of NTFS are in fact case sensitive, and if you mount a NTFS volume with Linux in fact you can create two file with the name differing only by the case: the whole case-insensitive thing is handled at an higher level in the Windows APIs.
It's not enough to have similar matching at the fs level, applications need to have
the same functionality around, otherwise anything that indexes the fs will have
false negatives before reading the files.
Now, if this needs to be taken care of at the application level, then why have
this misfeature? It'd be better to have a good library for matching this that
could be aware of the language and locale (or maybe multiple lang,locale pairs)
instead of throwing this feature into the fs and call it a day.
Also, if some applications benefit from having insensitive matching, like things
built to run on Windows/Mac, then having a wrapper that fixed the fs access with
this matching library would be enough, no need to force other applications to use
insensitive matching because a single one needs it.
> And as for whether userspace should catch up, thanks to OS X for the most part that already happened a long time ago for a ton of open source packages
That’s a bold claim. Open source developer here (on the fish team): given a user-entered path, how do you make it canonical?
Specifically, without recursively opening directories and listing contents, how can you convert /foo/baR to the correct case as it appears in the filesystem index? Without following any symlinks, should any of the path components be a symlink?
It took me a long time to figure out why "ls" on one system sorted my filenames differently than "ls" on my other system.
(meaning that filename handling with the same program isn't even consistent with itself)
Also, what happens when you backup very-important-file.txt and VeryImportantFile.txt and then restore to a case-insensitive filesystem? I'll tell you - unexpected things.
Backup applications always need to be aware of the filesystem features. Otherwise they won’t be able to store all important information, such as permissions and extended attributes.
By the way, having two files in one directory that would clash on a case insensitive filesystem is a bad idea and pretty unlikely.
Why would "every program" be the threshhold? There is no internal file format that every program understands, why should every program be able to deal with every filename? I could write my program to chunk filenames into words, and choke on any that it can't find in your local dictionary: "Spelling-sensitivity." It would be very bad to put that into the kernel.
It's easier to keep things simple, and rely on the program to discard things that it can't or doesn't want to use. If a program wants to insist that "Bob.txt" and Robert.txt" are the same file, that's its prerogative, although it's a dangerous assumption.
The fundamental question is: can you directly compare bytes?
If you can, then how you choose to interpret those bytes is irrelevant.
If you cannot, then you must learn when multiple bytes have the same meaning or when a byte in a different context should have a different meaning.
The latter case is much more complex for the computer as it requires parsing instead of comparison and is more complex for the programmer as it requires understanding of every supported difference (plus learning new ones as they are added).
How? If your program asks the OS for a handle for “data.db” and it returns one for “Data.db”, does it matter? If your program asked for “data.db”, why should your program care that the only match was for “Data.db”, so it got that?
which one should be opened when you ask for "idata.db"?
What about if only the first file exists, and you open for writing a file called "Idata.db"? Should it overwrite the first file, or create the second?
If it should overwrite the first file, unpacking an archive that contains both will result in a missing file.
What about these two, you might feel differently about because they are visually distinct:
ßdata.db
SSDATA.DB
These are not theoretical questions. On some OSes, the answer changes depending on ephemeral language settings that the current program may not be aware of or expecting.
In some environments, the answer doesn't change but doesn't match what happens in other environments. E.g. many systems implement ASCII case-folding only, others implement more characters but not Unicode, others implement all of Unicode.
Inevitably a program somewhere will break because this wasn't anticipated.
A backup using an archiver will not restore what was there before, and something that worked before the backup will behave differently after the restore.
A script (such as a build script) may inadvertantly create both files when run with one language setting, then break when the language setting is changed, even thought the script doesn't have anything to do with languages.
Your carefully written novel, that has chapter files with names you thought were distinct, will have missing chapters when you copy it across to another system. You discover this months later when you start working again on the novel, long after destroying the original system you thought you had copied from.
Interesting in the sense of "May you live in interesting times", yes.
It introduces near-endless complications, especially in the face of Unicode, and one should think thoroughly about it before implementing it, as you can read in Torvalds' initial reaction linked elsewhere in this discussion. But the way they did it seems relatively sane to me, with an easy way to compartmentalize it to, e.g., a Steam directory.
That comment section is pretty interesting to read...
I don't really have a use case for that feature right now. But that doesn't mean others don't have a use-case for this feature.
As the comment section reads, most feel entitled to NOT HAVE this convenience feature implemented for others, because they do not need it. It feels like a cognitive bias, but does it have a name, besides "get off my lawn?"
I definitely prefer case-senstive. I didn't realize that MacOS had switched to case-INsensitve at some point. So the following drove me crazy for several minutes:
$ ls -l /usr/local/bin/virtualbox
-rwxr-xr-x 1 root wheel 77 Oct 2 2015 /usr/local/bin/virtualbox*
$ ls /usr/local/bin | grep virtualbox
<nothing, WTF??>
$ ls /usr/local/bin | grep -i virtualbox
VirtualBox*
My coworker uses MacOS, I use Linux. Several times, changing the case of a directory or file caused the MacOS to mess up the local git repo so badly that we had to blow it away and refetch from remote.
At least the Mac Finder allows changing the case of a filename. Windows File Explorer simply refuses. I change the case of the file name, add a random character, save the file name. Then edit the file name again, remove the random character, save the file name. It's needlessly annoying.
I think case-INsensitive makes more sense for software developers, since there are many instances where things like "string.h" is not the same as "String.h". But for normal people, case-INsensitve may be more useful. However, even for normal users, I think there are situations where "bob.html" (a noun) is different from "Bob.html" (a person).
>At least the Mac Finder allows changing the case of a filename. Windows File Explorer simply refuses
Just tested on Windows 10 and this is definitely not true. I could have sworn you could always do it, but my memory of Windows < 10 is foggy.
And you can also do the renaming from any console app (cmd, powershell, etc), FYI.
NTFS is case-preserving, but case-insensitive. That, IMO is a good balance. When they were first making NT with the posix subsystem, I suppose they found that the number of cases where two files that only differed in case lived in the same directory were small enough to not matter.
I personally don't think case-sensitivity in file names adds anything useful.
If you enabled the posix subsystem NTFS became case sensitive as well, although most API passed a flag to disable that for Win32 file calls.
It’s interesting that Microsoft effectively added the reverse feature to NTFS [1] to support per-directory case sensitive files to be more compatible with Linux.
But I checked again on my own Windows 10 machine (Windows 10 Pro, Version 2004, build 19041.450) and holy shit, it now works. It must have been fixed in a recent Window 10 update.
Mac has been (at least by default) case-insensitive for a very long time, back to the original HFS at least, and probably its predecessor MFS too, though I'm not sure on that one.
What are the benefits to doing this? From the article I got two, "wine can get the kernel to do case insensitive path stuff instead of emulating it itself", and "users don't need special userspace magic to treat their filenames as strings, not bytes".
I have never seen anyone at all mope over lack of the latter, and the former seems quite specific to get such a big feature landed over. What other use-cases are there that benefit from this being in the FS?
I think it might somewhat be required for interoperability with some other filesystems (exfat under some conditions, hpfs likewise).
Moreover, as complex as the problem can be, it might come in handy to have a unicode normalization subsystem in the kernel for other modules to use. I don't think it's in the FS, but it's now a feature of the kernel that various FS can choose to wire up with, and expose knobs to the user to leverage it.
Besides that, whether they do or not is their choice. But I believe this subsystem is going to get used a lot more than some other, obscure kernel areas ;)
I mope at the lack of the latter, and it kept me off linux for twentysome years. Only now that I do most of my work in a GUI and I just click on filenames rather than typing them, does a case-sensitive filesystem not grind my gears. Every time I have to cd Downloads instead of cd downloads, I wonder who thought that was a good idea.
Case-sensitivity is a classic case of users being forced to comply with the computer's needs, rather than the other way around. I contend that that is Wrong, period.
To put it another way: Unicode is the opposite. It is computers adopting complexity to serve the needs of humans. If we can do unicode, we can do case-insensitive filename matching. If we're going to insist on case-sensitivity and ignore the needs of human language, we should just go back to plain old ASCII and force the humans to comply with that too.
With hidden filenames on mobile, desktop GUIs, terminal CLI completion, and shells with insensitive matching like fish (bash via option), the benefits of insensitive fs are not as high as they used to be.
Meanwhile, the complexity of matching modern Unicode causes performance degradation and exposes many edge cases.
In short, the window of time where insensitive made sense has largely closed.
> With hidden filenames on mobile, desktop GUIs, terminal CLI completion, and shells with insensitive matching like fish (bash via option), the benefits of insensitive fs are not as high as they used to be.
We've got better at working around case sensitivity, sure. I don't see that as a good reason not to fix the problem properly.
> More recently, f2fs has started to support this feature as well, following the Ext4 implementation and framework, thanks to an effort led by Google. Most, if not all, of the information described here also applies to f2fs, with small changes on the commands used to configure the superblock.
I'd guess that the f2fs changes at least are motivated by sdcard emulation on Android: sdcards were FAT32 in the beginning, so once they started being emulated with a backing store on the regular (ext4, then f2fs on pixels) data partition, case-insensitivity had to be emulated (first via a FUSE daemon in userspace, then via sdcardfs, a stackable FS in the kernel written by Samsung).
(I work at Google on Android, but I don't have any first-hand knowledge about this.)
Does case folding in the kernel pass the Turkey test? I.e., are different locales taken into account in order for the correct string to be matched? As far as I remember this was a big deal for supporting Unicode on case insensitive filesystems, because it than means that a file stops existing depending on the current locale.
For instance, take a file named "ivory.txt": `stat("IVORY.TXT")` on a case-insensitive filesystem would succeed if the locale is en_US but fail on tr_TR due to the uppercase version of 'i' being 'İ' there instead.
> The case insensitivity is just a horribly bad idea, and Applie could have pushed fixing it. They didn't. Instead, they doubled down on a bad idea, and actively extended it - very very badly - to unicode. And it's not even UTF-8, it's UCS2 I think.
> There's some excuse for case insensitivity in a legacy model ("We didn't know better"). But people who think unicode equivalency comparisons are a good idea in a filesystem shouldn't be allowed to play in that space. Give them some paste, and let them sit in a corner eating it. They'll be happy, and they won't be messing up your system.
Yes, and if they don't, introduce them to a 5-year-old.
That doesn't mean they won't still have trouble working with filenames that differ only by case, and it can still be pretty hard to explain why two characters that are visually similar are distinct to the computer.
But the idea that letters that look different are different is not a hard one to understand. It's the default, and one that people have to actively unlearn.
They're different to a human, why would it be any more difficult to understand that they're different for computers too? Trying to make things easier to understand by introducing complicated mechanisms just makes the whole thing harder to use.
The computer should be more complicated to make things easier for humans. The computer is fast and reliable at performing complicated operations. Humans are neither fast nor reliable at computing tasks.
The computer is fast and reliable at performing complicated well-defined operations.
Case insensitive comparison is less well defined than you might think. You need to know what language, because the rules are different in different languages. You need to know what version of unicode, because the rules can change. You need to know what to do if the data on disk doesn't conform to the rules. You need to know what to do when copying data from one filesystem to another when the names can't be stored on the destination.
I feel it's often better to train the user on how to use a simple system effectively, than to build a complex system. You have to put the complexity somewhere, and humans are easier to change than software and hardware.
> You need to know what to do when copying data from one filesystem to another when the names can't be stored on the destination.
You already need to know that, because filesystems are a jungle. This covers everything from special characters, to non-Unicode encodings, to different Unicode-like encodings, to special names, to file name length limits, and path length limits.
Consider that on Linux a filename is just an arbitrary sequence of bytes with some limitations, and on Windows it's an arbitrary sequence of 16-bit integers. You can put unpaired surrogate characters in a Windows path, and you can put 0xff in a Linux path.
> I feel it's often better to train the user on how to use a simple system effectively, than to build a complex system. You have to put the complexity somewhere, and humans are easier to change than software and hardware.
Filenames are something that affects everyone who uses computers for more than just browsing the web. By and large, these people have been using case-insensitive filesystems for the past 40 years, except the people using Unix-derivatives other than macOS--a small slice of people with a high percentage of experts.
So if you have an idea of how to train a billion people to deal with case sensitivity problems that they didn't have to do with before, I'm all ears.
And yes... the case folding and locale problems exist. I think we're already on the way to the right solution there... hide the filesystem completely from most people who use computers. The filesystem is a terrible way to organize data, so we are building more complicated filesystems and more complicated stacks on top of filesystems to make it easier for humans to work with, and rightly so.
> Consider that on Linux a filename is just an arbitrary sequence of bytes with some limitations, and on Windows it's an arbitrary sequence of 16-bit integers. You can put unpaired surrogate characters in a Windows path, and you can put 0xff in a Linux path.
I don't think these are as equivalent as you imply. While it's technically possible for Windows paths to contain unpaired surrogates, in practice it never happens outside of malicious software (and even that's rare),
Whereas on Linux, getting a "random" bag of bytes is relatively common because the convention of UTF-8 is new (and is a userspace convention). Of course it's usually not truly random but you have to play guess the encoding if you want to interpret it.
> So if you have an idea of how to train a billion people to deal with case sensitivity problems that they didn't have to do with before, I'm all ears.
It's a bit harder for the people who had case insensitivity before, I'd imagine. But the basic way to train people for this is a) to work consistently and b) drop helpful examples in places. For a) When someone makes a new directory lowercse dotless i when they already have a folder named I, they might be surprised to have two directories with the same letter, but they'll figure it out. The same way that they figured out news.ycombinator.com/REPLY doesn't work, because the bytes aren't the same.
For b) make some tools expect a file called Makefile and not work if makefile is present. Done!
That's a bit pessimistic--displaying plain, unstyled text onscreen is significantly more difficult, and yet we expect computers to figure it out automatically. You don't have to position each character by hand or select the right glyph. You can just click in a text box and start typing. Case folding and collation are far easier problems by comparison.
I think so, they understand when writing a sentence you don't start it with lower case character. They know to capitalize names. They recognize every time they see it not done its considered a mistake.
I have met very many people who have trouble doing that–and no, it's not because they thought it was a username, or they were too lazy to use the shift key.
If you sent someone a (physical) document and asked them to "put it in the banana folder", they would happily put it in the folder labelled "Banana". It would be seen as a mistake to have separate "banana" and "Banana" folders.
The problem there is that depending on the person both variants can be terrible UX.
For me case insensitive filenames are horrible UX, for my parents case sensitive filenames are horrible UX.
However I wouldn't advice my parents to use case insensitive filenames: there are too much complications that you might get from it in interaction with literally anything. The UX net effect would be negative for them.
Case sensitivity is actually a powerful feature of Unix systems as it allows for multiple valid string variants of a single word.
For example, common Unix convention in the 1980s-1990s was to name user-created directories with a capital to make them easier to see in a regular directory listing without color (e.g. Poems is a directory, while poems is just a file).
I've used it heavily with content revision (e.g. CATHENA02.yaml is the second cathena configuration file in testing while cathena02.yaml is the production version of it).
Plus, making a filename case-insensitive for processing purposes in a scripting language is very easy.
Consequently, I can't imagine a reason why I'd use case-insensitivity in ext4.
Almost 20 years ago, the place I worked decided it wanted to make something like Wine, except going the other way--it would run Linux binaries on Windows.
We got it working fairly well. We expected that we'd have to put in some kind of hack to deal with filename case, and I think we eventually did.
But before that, when it was still just passing Linux case-sensitive filenames through to the case-insensitive Windows filesystem, I tried installing most of whatever was the current release of Red Hat at the time.
It almost all worked fine. The only thing I remember being a problem was that some things, such as some Perl modules from CPAN, had both "makefile" and "Makefile" in the same directory.
They wanted to be able to run binaries that were built for Linux, including some commercial binaries for which we did not have source. Things have to be recompiled for Cygwin.
As someone who grew up on Commodore, DOS/Win, and later Unix, I just never had a problem with case sensitivity or lack of. shrug
99% of the time I use lower-case only filenames, as they are easier to read. The times I don't shell completion and/or GUI selection obviate the need to care anyway. Given the significant complexity of Unicode I'm not sure insensitive is the way of the future.
There was a window of time where insensitive made the most sense, the time of DOS. 8-bit per character filenames, with a very primitive CLI shell, ie. no assistance. Now? In the days when a majority of users don't even see filenames? Meh.
> Very costly, but the only way to give case-insensitivity.
That is very costly — but it's certainly not the only way to provide case-insensitivity, nor is it the recommended way, and I'd be surprised if any implementation of case-insensitivity actually did what you say.
Normally one would lowercase (or uppercase) both strings, and then do the comparison.
The complexity here usually comes when case-folding various tricky locales.
Ah, my bad: In the post I was replying to, I thought you were saying that all of the permutation were enumerated and tested for — which we can all agree is terrible.
I now understand you mean that the directory is in fact scanned (and filenames compared in a case insensitive manner) once a miss occurs. Which is what I was hinting at.
Apologies, I misunderstood your post — and my reply could have been clearer also.
I have always felt putting case-insensitivity at the filesystem level was a bad idea which would have died in the 80s if it was not for some Apple product manager being stubborn.
But making it so that it requires a kernel compile time option to make your filesystem mountable is a level of absurdity that i would not have believed possible before 2020 made unbelievable the new normal.
103 comments
[ 4.4 ms ] story [ 173 ms ] threadThat aside, inferring semantic meaning based on a few cultures at this moment in time is a dumb choice. There are tons of such language-specific semantics that could be implemented and the result is needing to memorize all those rules about when it does and doesn't actually matter (and arguing that only certain cultures and languages should have their own semantics encoded in the OS is its own problem).
The principal question is whether the tool is more important, or the end user. Why did I pay for this machine if it weren't intended to facilitate me? That's the bottom line with most of these kinds of technical "correctness" arguments
And as for whether userspace should catch up, thanks to OS X for the most part that already happened a long time ago for a ton of open source packages
I also side with this view, namely that this is something that would be better placed in the userspace rather than the kernel, which really doesn't need more complexity for things that add so little value (negative value to some).
|I've looked at the source and there are pieces that are good and pieces that are not. A whole bunch of random people have contributed to this source, and the quality varies drastically
I happen to agree with the idea that the filename should be a dumb blob of bytes and the kernel should not do case folding, as it is the wrong layer for that, eg. the user can change their language but it won't update what has been written to the disk in thousands or millions of places where you could suddenly have a filename collision somewhere based on those rules changing.
But, I do hope you get that refund for your Linux.
Well, now your filename is invalid utf8. How should programs display it or even address such a file?
what's wrong with foo����.txt
> or even address such a file? ... by using the array of bytes ?
That's reality. An OS which can't keep up with reality is broken.
This hasn't been true since the days of CP/M
All of the wide/special-case manipulation when writing code on e.g. Windows drove me nuts.
It also means that if you want to do something highly annoying and unusual, you can – and arguably case-insensitive filesystems are a subset of that.
I've been running this for years accessing them both from Linux desktops and Windows desktops, and only once have I had an issue that required me to manually rename something on the NAS.
This makes sense as most applications don't care about the filename, and will just use what you supply, or generate one and use that string all over.
The problem is emergent behavior. We can create any number of features, but we have a really hard time testing all the available configurations that result. Engineers rely on simplicity as a way of warding off this particular problem, because the other bottom line is people don't want to pay for a system that loses or invalidates the work they've put into it.
The only real purpose of this is for easier/faster compatibility with software developed and tested on macOS and Windows, which has accidental case inconsistency in file name references in the code, which happens to work fine on macOS and Windows.
TFA could have made that argument, but it didn't, it made incorrect arguments instead. For example, a user might type in a lower-case name for a file one time, and a capitalized name for a file another time, and intend to access the same file. But what user is typing in a whole filename the second time? They're picking it from a list, or if they're very advanced, using completion in a terminal. TFA also mentions non-English languages, in the context of unicode normalization ... but non-western-european languages won't be handled correctly by any universal case-folding algorithm anyway, with Turkish being the most common example.
The whole strategy just doesn't work out well. Many low-level filesystem developers have known this for over 20 years. It doesn't work better for anyone, but non-technical people just aren't aware of why or how it increases complexity and costs, and reduces performance and reliability.
Move on to most anywhere else and the notion that it's fast, reliable, and safe to convert case gets lost in the realities of human writing systems.
And even then it still doesn't fix the issue outside of the world of ASCII. A filename written in hiragana and katakana is logically the same to the end-user, but they are still distinct. Simplified and traditional Chinese, Hangul and romaja, pinyin, devanagari, thai, and the list goes on.
Case-insensitive filenames fixes nothing, but breaks everything. There is only one sensible thing to do with arbitrary user-input, and that is to leave it be.
I wish programmers would believe this about names and addresses! My wife has a two-word first name, and I have a two-word family name.
It must be in the kernel. Else implementations of stuff with human-derived semantics should move out of the kernel, but moving SMB into userspace would of course be ridiculous
So it complicates any mapping of filenames to data structures in the kernel (all 3 of them?), big deal. Every popular desktop operating system supports it, and basically the only reason Linux does not is a mixture of FUD and fear that we may need to update the code at some point due to changes in human culture, the horror!
Meanwhile, typing "cd music" in a terminal need not print "Command not found" when there is clearly a folder named "Music", like the $3k worth of gear in front of me had the complexity of some 1950s b-movie scifi robot
Case-insensitive path lookups on SMB happens in the server (usually samba), in user-space. The client is also usually in user-space through FUSE or the client libs, but CIFS of course exists as a kernel-mode alternative.
And as I have written elsewhere, sure, "music" vs. "Music" is simple when you live in an ASCII world. Trying to be smart with user input only causes trouble for the rest of us. Hiragana and katakana is also logically the same, and on a kana keyboard a similar typo. Simplified and traditional chinese is also logically the same.
> Every popular desktop operating system supports it
Doesn't mean we should break stuff here as well.
SMB in the kernel is a rather dangerous game. Not saying it can't be done (and I know Samsung is doing it :-), but it's a significantly more complex protocol than even NFSv4 (which also really shouldn't be in the kernel either). For complexities sake, userspace is easier IMHO (much easier to debug).
Also, you seem to be of the opinion that code in the kernel must be "magic", in that it can do things that user-space can't. The "missing case" lookup problem still exists for kernel code as it does for user-space code.
Looking up "foo" in all case variations in a file containing a few million files still has to be done by search in the kernel as it does in user-space. It's going to be slow there too unless you provide a case-insensitive indexing mechanism.
Now the kernel offers easier opportunities to do things like directory content caching, which currently aren't exposed to user space via any API - but once you have to do something like directory content caching it's also possible to expose that feature to userspace via an API.
> Else implementations of stuff with human-derived semantics should move out of the kernel, but moving SMB into userspace would of course be ridiculous
Samba would beg to disagree of course :-).
In practice the case insensitive thing if we want to call it should be implemented more high level, in the file manager, rather than in the filesystem itself. That is even what newer versions of Windows/NTFS do! Recent versions of NTFS are in fact case sensitive, and if you mount a NTFS volume with Linux in fact you can create two file with the name differing only by the case: the whole case-insensitive thing is handled at an higher level in the Windows APIs.
Now, if this needs to be taken care of at the application level, then why have this misfeature? It'd be better to have a good library for matching this that could be aware of the language and locale (or maybe multiple lang,locale pairs) instead of throwing this feature into the fs and call it a day.
Also, if some applications benefit from having insensitive matching, like things built to run on Windows/Mac, then having a wrapper that fixed the fs access with this matching library would be enough, no need to force other applications to use insensitive matching because a single one needs it.
That’s a bold claim. Open source developer here (on the fish team): given a user-entered path, how do you make it canonical?
Specifically, without recursively opening directories and listing contents, how can you convert /foo/baR to the correct case as it appears in the filesystem index? Without following any symlinks, should any of the path components be a symlink?
(meaning that filename handling with the same program isn't even consistent with itself)
Also, what happens when you backup very-important-file.txt and VeryImportantFile.txt and then restore to a case-insensitive filesystem? I'll tell you - unexpected things.
Both files get restored as independent files, as case isnt the only difference in those files.
(Did you mean very-important-file.txt and Very-Important-File.txt?)
oh sorry, ha! yes.
By the way, having two files in one directory that would clash on a case insensitive filesystem is a bad idea and pretty unlikely.
It's easier to keep things simple, and rely on the program to discard things that it can't or doesn't want to use. If a program wants to insist that "Bob.txt" and Robert.txt" are the same file, that's its prerogative, although it's a dangerous assumption.
If you can, then how you choose to interpret those bytes is irrelevant.
If you cannot, then you must learn when multiple bytes have the same meaning or when a byte in a different context should have a different meaning.
The latter case is much more complex for the computer as it requires parsing instead of comparison and is more complex for the programmer as it requires understanding of every supported difference (plus learning new ones as they are added).
Very interesting feature, of course.
What about if only the first file exists, and you open for writing a file called "Idata.db"? Should it overwrite the first file, or create the second?
If it should overwrite the first file, unpacking an archive that contains both will result in a missing file.
What about these two, you might feel differently about because they are visually distinct:
These are not theoretical questions. On some OSes, the answer changes depending on ephemeral language settings that the current program may not be aware of or expecting.In some environments, the answer doesn't change but doesn't match what happens in other environments. E.g. many systems implement ASCII case-folding only, others implement more characters but not Unicode, others implement all of Unicode.
Inevitably a program somewhere will break because this wasn't anticipated.
A backup using an archiver will not restore what was there before, and something that worked before the backup will behave differently after the restore.
A script (such as a build script) may inadvertantly create both files when run with one language setting, then break when the language setting is changed, even thought the script doesn't have anything to do with languages.
Your carefully written novel, that has chapter files with names you thought were distinct, will have missing chapters when you copy it across to another system. You discover this months later when you start working again on the novel, long after destroying the original system you thought you had copied from.
It introduces near-endless complications, especially in the face of Unicode, and one should think thoroughly about it before implementing it, as you can read in Torvalds' initial reaction linked elsewhere in this discussion. But the way they did it seems relatively sane to me, with an easy way to compartmentalize it to, e.g., a Steam directory.
I don't really have a use case for that feature right now. But that doesn't mean others don't have a use-case for this feature.
As the comment section reads, most feel entitled to NOT HAVE this convenience feature implemented for others, because they do not need it. It feels like a cognitive bias, but does it have a name, besides "get off my lawn?"
$ ls -l /usr/local/bin/virtualbox
-rwxr-xr-x 1 root wheel 77 Oct 2 2015 /usr/local/bin/virtualbox*
$ ls /usr/local/bin | grep virtualbox
<nothing, WTF??>
$ ls /usr/local/bin | grep -i virtualbox
VirtualBox*
My coworker uses MacOS, I use Linux. Several times, changing the case of a directory or file caused the MacOS to mess up the local git repo so badly that we had to blow it away and refetch from remote.
At least the Mac Finder allows changing the case of a filename. Windows File Explorer simply refuses. I change the case of the file name, add a random character, save the file name. Then edit the file name again, remove the random character, save the file name. It's needlessly annoying.
I think case-INsensitive makes more sense for software developers, since there are many instances where things like "string.h" is not the same as "String.h". But for normal people, case-INsensitve may be more useful. However, even for normal users, I think there are situations where "bob.html" (a noun) is different from "Bob.html" (a person).
Just tested on Windows 10 and this is definitely not true. I could have sworn you could always do it, but my memory of Windows < 10 is foggy. And you can also do the renaming from any console app (cmd, powershell, etc), FYI.
NTFS is case-preserving, but case-insensitive. That, IMO is a good balance. When they were first making NT with the posix subsystem, I suppose they found that the number of cases where two files that only differed in case lived in the same directory were small enough to not matter.
I personally don't think case-sensitivity in file names adds anything useful.
It’s interesting that Microsoft effectively added the reverse feature to NTFS [1] to support per-directory case sensitive files to be more compatible with Linux.
[1] https://www.tiraniddo.dev/2019/02/ntfs-case-sensitivity-on-w...
Here is a user complaint about this problem from Nov 2019: https://answers.microsoft.com/en-us/windows/forum/all/cant-r...
Here's another post from May 2018, explicitly mentioning Windows 10: https://answers.microsoft.com/en-us/windows/forum/all/why-ca...
But I checked again on my own Windows 10 machine (Windows 10 Pro, Version 2004, build 19041.450) and holy shit, it now works. It must have been fixed in a recent Window 10 update.
I have never seen anyone at all mope over lack of the latter, and the former seems quite specific to get such a big feature landed over. What other use-cases are there that benefit from this being in the FS?
Moreover, as complex as the problem can be, it might come in handy to have a unicode normalization subsystem in the kernel for other modules to use. I don't think it's in the FS, but it's now a feature of the kernel that various FS can choose to wire up with, and expose knobs to the user to leverage it.
Besides that, whether they do or not is their choice. But I believe this subsystem is going to get used a lot more than some other, obscure kernel areas ;)
Case-sensitivity is a classic case of users being forced to comply with the computer's needs, rather than the other way around. I contend that that is Wrong, period.
To put it another way: Unicode is the opposite. It is computers adopting complexity to serve the needs of humans. If we can do unicode, we can do case-insensitive filename matching. If we're going to insist on case-sensitivity and ignore the needs of human language, we should just go back to plain old ASCII and force the humans to comply with that too.
With hidden filenames on mobile, desktop GUIs, terminal CLI completion, and shells with insensitive matching like fish (bash via option), the benefits of insensitive fs are not as high as they used to be.
Meanwhile, the complexity of matching modern Unicode causes performance degradation and exposes many edge cases.
In short, the window of time where insensitive made sense has largely closed.
We've got better at working around case sensitivity, sure. I don't see that as a good reason not to fix the problem properly.
As mentioned, there is no way to "fix the problem properly" and a close-enough solution, rarely needed these days, slows performance.
I'd guess that the f2fs changes at least are motivated by sdcard emulation on Android: sdcards were FAT32 in the beginning, so once they started being emulated with a backing store on the regular (ext4, then f2fs on pixels) data partition, case-insensitivity had to be emulated (first via a FUSE daemon in userspace, then via sdcardfs, a stackable FS in the kernel written by Samsung).
(I work at Google on Android, but I don't have any first-hand knowledge about this.)
For instance, take a file named "ivory.txt": `stat("IVORY.TXT")` on a case-insensitive filesystem would succeed if the locale is en_US but fail on tr_TR due to the uppercase version of 'i' being 'İ' there instead.
> There's some excuse for case insensitivity in a legacy model ("We didn't know better"). But people who think unicode equivalency comparisons are a good idea in a filesystem shouldn't be allowed to play in that space. Give them some paste, and let them sit in a corner eating it. They'll be happy, and they won't be messing up your system.
Linus Torvalds, 2014, https://web.archive.org/web/20150112214037/https://plus.goog...
But that being said, are many nontechnical users typing filenames anyways?
That doesn't mean they won't still have trouble working with filenames that differ only by case, and it can still be pretty hard to explain why two characters that are visually similar are distinct to the computer.
But the idea that letters that look different are different is not a hard one to understand. It's the default, and one that people have to actively unlearn.
“Report (final).docx” vs “Report (Final).docx”
Case insensitive comparison is less well defined than you might think. You need to know what language, because the rules are different in different languages. You need to know what version of unicode, because the rules can change. You need to know what to do if the data on disk doesn't conform to the rules. You need to know what to do when copying data from one filesystem to another when the names can't be stored on the destination.
I feel it's often better to train the user on how to use a simple system effectively, than to build a complex system. You have to put the complexity somewhere, and humans are easier to change than software and hardware.
You already need to know that, because filesystems are a jungle. This covers everything from special characters, to non-Unicode encodings, to different Unicode-like encodings, to special names, to file name length limits, and path length limits.
Consider that on Linux a filename is just an arbitrary sequence of bytes with some limitations, and on Windows it's an arbitrary sequence of 16-bit integers. You can put unpaired surrogate characters in a Windows path, and you can put 0xff in a Linux path.
> I feel it's often better to train the user on how to use a simple system effectively, than to build a complex system. You have to put the complexity somewhere, and humans are easier to change than software and hardware.
Filenames are something that affects everyone who uses computers for more than just browsing the web. By and large, these people have been using case-insensitive filesystems for the past 40 years, except the people using Unix-derivatives other than macOS--a small slice of people with a high percentage of experts.
So if you have an idea of how to train a billion people to deal with case sensitivity problems that they didn't have to do with before, I'm all ears.
And yes... the case folding and locale problems exist. I think we're already on the way to the right solution there... hide the filesystem completely from most people who use computers. The filesystem is a terrible way to organize data, so we are building more complicated filesystems and more complicated stacks on top of filesystems to make it easier for humans to work with, and rightly so.
I don't think these are as equivalent as you imply. While it's technically possible for Windows paths to contain unpaired surrogates, in practice it never happens outside of malicious software (and even that's rare),
Whereas on Linux, getting a "random" bag of bytes is relatively common because the convention of UTF-8 is new (and is a userspace convention). Of course it's usually not truly random but you have to play guess the encoding if you want to interpret it.
It's a bit harder for the people who had case insensitivity before, I'd imagine. But the basic way to train people for this is a) to work consistently and b) drop helpful examples in places. For a) When someone makes a new directory lowercse dotless i when they already have a folder named I, they might be surprised to have two directories with the same letter, but they'll figure it out. The same way that they figured out news.ycombinator.com/REPLY doesn't work, because the bytes aren't the same.
For b) make some tools expect a file called Makefile and not work if makefile is present. Done!
Computers are for end-users. OS and app developers are in the minority, and we should remember that when we write apps or filesystems.
Why is this the exception?
For me case insensitive filenames are horrible UX, for my parents case sensitive filenames are horrible UX.
However I wouldn't advice my parents to use case insensitive filenames: there are too much complications that you might get from it in interaction with literally anything. The UX net effect would be negative for them.
For example, common Unix convention in the 1980s-1990s was to name user-created directories with a capital to make them easier to see in a regular directory listing without color (e.g. Poems is a directory, while poems is just a file).
I've used it heavily with content revision (e.g. CATHENA02.yaml is the second cathena configuration file in testing while cathena02.yaml is the production version of it).
Plus, making a filename case-insensitive for processing purposes in a scripting language is very easy.
Consequently, I can't imagine a reason why I'd use case-insensitivity in ext4.
We got it working fairly well. We expected that we'd have to put in some kind of hack to deal with filename case, and I think we eventually did.
But before that, when it was still just passing Linux case-sensitive filenames through to the case-insensitive Windows filesystem, I tried installing most of whatever was the current release of Red Hat at the time.
It almost all worked fine. The only thing I remember being a problem was that some things, such as some Perl modules from CPAN, had both "makefile" and "Makefile" in the same directory.
99% of the time I use lower-case only filenames, as they are easier to read. The times I don't shell completion and/or GUI selection obviate the need to care anyway. Given the significant complexity of Unicode I'm not sure insensitive is the way of the future.
There was a window of time where insensitive made the most sense, the time of DOS. 8-bit per character filenames, with a very primitive CLI shell, ie. no assistance. Now? In the days when a majority of users don't even see filenames? Meh.
That is very costly — but it's certainly not the only way to provide case-insensitivity, nor is it the recommended way, and I'd be surprised if any implementation of case-insensitivity actually did what you say.
Normally one would lowercase (or uppercase) both strings, and then do the comparison.
The complexity here usually comes when case-folding various tricky locales.
Windows slient sends a name "foo" over SMB2+. Samba does a stat("foo", &st) call, gets ENOENT.
Now, does that name really not exist ? Or is it there in the requested directory under any of the names "FOO"/"FOo"/"Foo"/"fOo"/"fOO"/"FoO" etc. ?
The only way for an application to tell is to do:
opendir() fname = readdir().... check strlower(client_provided_name, fname); closedir().
If the file really doesn't exist you have ended up scanning the whole directory.
Because Linux doesn't provide directory leasing you have to do this for every missed lookup (another process might have created it in the meantime).
There is no POSIX API for "does this file exist under another cased name" ? If there were Samba would use it.
I now understand you mean that the directory is in fact scanned (and filenames compared in a case insensitive manner) once a miss occurs. Which is what I was hinting at.
Apologies, I misunderstood your post — and my reply could have been clearer also.
But making it so that it requires a kernel compile time option to make your filesystem mountable is a level of absurdity that i would not have believed possible before 2020 made unbelievable the new normal.