And I've had the new linux NTFS driver corrupt files more than once.
In fact, it just corrupted the partition table of my external backup drive this year. Which was last time I trusted it for anything but read-only mounting.
This is pretty great. I've been wanting to make a simple utility that can scan an NTFS drive for a filename pattern; scanning the MFT is incredibly fast versus using the actual filesystem API calls to do so.
It’s probably mostly done out of carefulness. I’d do the same if I were to write my own implementation of a file system.
Bugs and mistakes in the code are already not great when implementing the read part. But if you mess up the writing well now you’re gonna cause corruption of existing data. No one likes that :)
Better to stick to just reading for a long while, and weed out most of the bugs and getting an even deeper understanding of the filesystem, before starting the implementation of anything that will modify the data.
> I’d do the same if I were to write my own implementation of a file system.
Exactly. I have my own ext2 driver for a couple of hobby OS projects where the read side is extremely stable, but as the write side involves inode allocation, block/bitmap allocation etc. I don't turn it on on outside of testing since I know it's going to destroy the disk images until I "get it right".
I would expect writing to be much harder on all file-systems. There's only one way to read the data, but there could be any number of strategies for deciding where to write new data.
Reading is always easier to implement than writing. Only writing has the potential to actively corrupt the filesystem or leave it in some inconsistent state.
Generally, the more modern the filesystem, the more complicated writing becomes.
Linux filesystems seem to be evolving all the time (try to install any new distro today and check out how many filesystems you can pick from the dropdown...)
Apple replaced one filesystem with another in just the past few years, with apparently some very advanced new features.
But NTFS is just... NTFS? Does NTFS today look anything like how it looked in 1994? Or 2018? Does it change radically between versions of Windows? Or does the MS creed of compatibility at all costs mean it's basically a filesystem unchanged from the 90s? Thanks!
> Linux filesystems seem to be evolving all the time (try to install any new distro today and check out how many filesystems you can pick from the dropdown...)
This sounds like evolution of options more than evolution of existing file systems
Is ext4 still not most popular for most end users? Curious how much ext4 has changed vs NTFS past... ~20 years (ext4 went stable 2008 apparently)
ext4 remains the default for a lot of distros and probably most end users, but other options are common as well. openSUSE and Fedora now default to btrfs, RHEL defaults to XFS.
> Linux kernel 4.8 in August 2016 added a new feature, "reverse mapping". This is the foundation for a large set of planned features: snapshots, copy-on-write (COW) data, data deduplication, reflink copies, online data and metadata scrubbing, highly accurate reporting of data loss or bad sectors, and significantly improved reconstruction of damaged or corrupted filesystems. This work required changes to XFS's on-disk format.
My bet is that btrfs will replace ext4 in the nearish future. It's the default for Arch, which also comes default with filesystem snapshots before any system updates.
It's pretty nice. I think it also supports some flavor of volumes or virtual partitions, but I don't have any real use for that kind of thing.
The snapshots are really good though. Very fast and don't take up a ton of space, somehow.
Appreciate it. As someone who basically continues to use ZFS happily and didn't kick the tires because of the stability & ergonomics issues in the past, this is concerning to hear...
Looks like kicking the tires will remain somewhat in the future for me.
It's much stable after 6.4 kernel. Last year was not really stable yet.
I faced too, problem is I didn't know we have to schedule a `btrfs scrub`. After kernel update and scrub, everything works smooth.
I recently had a bit of a data hiccup on my desktop laptop, where I'm using Btrfs with LVM encryption. What's weird is that I started getting these random reboots out of the blue, with no error messages to explain what was going on. I was scratching my head for a while, thinking it might be a hardware issue.
Then, in frustration, I rebooted into my Windows partition, which I rarely ever use. to see if I could back up my stuff and figure out why I wasn't having the same reboot problems in Windows.
After trying a bunch of Btrfs repair commands over a bootable usb disk I found online, I hit a wall with some errors that couldn't be fixed. So, I threw in the towel and reinstalled my system with ext4. It's been running smoothly for three weeks now with no more error headaches.
Could not find the logs of the errors anymore unfortunately to report.
Me too, also on two distinct occasions. It kept going read only on a third distinct occasion and i copied my stuff off and switched to zfs and haven't looked back. I am hoping bcachefs provides a compelling option in the next decade to switch off zfs but its a tall hill to climb
yeah, I managed to corrupt my btrfs on my laptop a few months ago (loose memory stick causing crash at bad time, apparently). I was able to recover almost everything with btrfs-restore (my latest backup was several weeks old then) but it was still not a pleasant experience...
Btrfs seems to be the new standard, it might well end up being the one big thing Oracle did for Linux; it definitely evolved significantly over the last 15 years.
ext2/3/4 are the same on-disk filesystem with features added, so could also compare the timeline from the 90's. NTFS and ext2 have the same year (1993) listed on WP as intro date.
NTFS has gained support for a few features over the years, but it wasn't meant to evolve, much. Microsoft knew what they wanted from the filesystem and they wrote it to spec very early in its life.
ReFS on Windows is where the changes are coming to. finally a 1st party filesystem which supports Copy on Write file cloning. a lot of core features of NTFS aren't supported on ReFS, so it's clearly not intended to be a replacement for NTFS.
Windows recently got support for booting from ReFS volumes, so it seems like Microsoft has started working on it again and maybe does think a move away from NTFS is possible in the future.
The recent interview with Dave Cutler mentioned this briefly. When the Cairo team was working on their object file system, they had no requirement to maintain backwards compatibility (which makes sense for new development). But they didn't build in a way to have in-place upgrades after it would have been released, either.
The Tukwila team, still working with the old codebase, had a rule that NTFS could be upgraded, but it had to either accept the old file system structures, or be upgraded in-place. And their code allowed for that. Relevant portion starts at about 2:10
It hasn't changed radically, but it has changed. Wikipedia actually has a nice write-up if you want the details, but support for symbolic links, transactions, and partition resizing are all things that have materially improved my life in the last ~5-10 years.
For example, I have a dual-boot arch/win system with 2 ntfs partitions and a with a recent update the linux kernel ntfs driver can't mount the filesystems, have to fall back to the fuse implementation.
It also didn't seem to be able to do things like set system flags hsra, but I found a way with the fuse implementation.
53 comments
[ 3.4 ms ] story [ 100 ms ] threadIn fact, it just corrupted the partition table of my external backup drive this year. Which was last time I trusted it for anything but read-only mounting.
And I have had /windows/ corrupt NTFS volumes.
Personal anecdotes rock :D
Why go the effort to index the files in a filesystem when the filesystem IS an index in itself? Just query it directly
It's dual-licensed as MIT or Apache 2.0, so it ought to be usable just about anywhere.
That's how Linux NTFS support started out, too.
What is it about NTFS that makes writing hugely more difficult than reading?
Bugs and mistakes in the code are already not great when implementing the read part. But if you mess up the writing well now you’re gonna cause corruption of existing data. No one likes that :)
Better to stick to just reading for a long while, and weed out most of the bugs and getting an even deeper understanding of the filesystem, before starting the implementation of anything that will modify the data.
Exactly. I have my own ext2 driver for a couple of hobby OS projects where the read side is extremely stable, but as the write side involves inode allocation, block/bitmap allocation etc. I don't turn it on on outside of testing since I know it's going to destroy the disk images until I "get it right".
Generally, the more modern the filesystem, the more complicated writing becomes.
Has NTFS changed much in... 20 years?
Linux filesystems seem to be evolving all the time (try to install any new distro today and check out how many filesystems you can pick from the dropdown...)
Apple replaced one filesystem with another in just the past few years, with apparently some very advanced new features.
But NTFS is just... NTFS? Does NTFS today look anything like how it looked in 1994? Or 2018? Does it change radically between versions of Windows? Or does the MS creed of compatibility at all costs mean it's basically a filesystem unchanged from the 90s? Thanks!
This sounds like evolution of options more than evolution of existing file systems
Is ext4 still not most popular for most end users? Curious how much ext4 has changed vs NTFS past... ~20 years (ext4 went stable 2008 apparently)
> Linux kernel 4.8 in August 2016 added a new feature, "reverse mapping". This is the foundation for a large set of planned features: snapshots, copy-on-write (COW) data, data deduplication, reflink copies, online data and metadata scrubbing, highly accurate reporting of data loss or bad sectors, and significantly improved reconstruction of damaged or corrupted filesystems. This work required changes to XFS's on-disk format.
It's pretty nice. I think it also supports some flavor of volumes or virtual partitions, but I don't have any real use for that kind of thing.
The snapshots are really good though. Very fast and don't take up a ton of space, somehow.
Looks like kicking the tires will remain somewhat in the future for me.
Then, in frustration, I rebooted into my Windows partition, which I rarely ever use. to see if I could back up my stuff and figure out why I wasn't having the same reboot problems in Windows.
After trying a bunch of Btrfs repair commands over a bootable usb disk I found online, I hit a wall with some errors that couldn't be fixed. So, I threw in the towel and reinstalled my system with ext4. It's been running smoothly for three weeks now with no more error headaches.
Could not find the logs of the errors anymore unfortunately to report.
Not like, "hehe we need to sync", but "good luck repairing, enjoy probably formatting"
LVM/MD RAID, ZFS, etc all behave fine. Only BTRFS is this fragile.
Source: had to patch yaboot(PowerPC Mac bootloader) to update ext4 to boot with /boot on ext4.
ReFS on Windows is where the changes are coming to. finally a 1st party filesystem which supports Copy on Write file cloning. a lot of core features of NTFS aren't supported on ReFS, so it's clearly not intended to be a replacement for NTFS.
they're gonna live side by side for 30+ years. they are meant for different uses.
The Tukwila team, still working with the old codebase, had a rule that NTFS could be upgraded, but it had to either accept the old file system structures, or be upgraded in-place. And their code allowed for that. Relevant portion starts at about 2:10
https://www.youtube.com/watch?v=xsQBKkPj5Nc
https://learn.microsoft.com/en-us/windows/dev-drive/
Surely you lie..
For example, I have a dual-boot arch/win system with 2 ntfs partitions and a with a recent update the linux kernel ntfs driver can't mount the filesystems, have to fall back to the fuse implementation.
It also didn't seem to be able to do things like set system flags hsra, but I found a way with the fuse implementation.
If you want to learn more about NTFS internals and my crate, check out the talk I gave at FOSDEM 2022: https://archive.fosdem.org/2022/schedule/event/misc_ntfs_rus...
Also I'm open to any questions here.