The paper analyzes how file systems and PostgreSQL, LMDB, LevelDB, SQLite, and Redis react to fsync failures. It shows that although applications use many failure-handling strategies, none are sufficient: fsync failures can cause catastrophic outcomes such as data loss and corruption.
I'd argue some modern hardware e.g. NVME SSDs already expose a hardware interface that would let a new transactional storage API be faster than what we have right now. Unfortunately it is a rare example in a see full of specs that do their best to avoid providing clear semantics and guarantees like atomic sector writes.
I guess the next study would need to look into what manufacturers actually fully adhere to the spec. They've not exactly shown best behaviour in that regard in the past (lying about fsync etc) :(
Without the hardware being open, how do you prove this? In a past life I concluded and argued that there's no point in assuming hardware tells the truth. You have to expect the worst and have a plan for handling it, even if that plan is giving up and failing.
Importantly this means you can't claim stronger semantics around e.g. atomicity. You can certainly work around most if not all issues, using redundancy, verification and distribution. But in isolation you cannot even properly observe extreme failure scenarios, you can only reduce their probability, and even that is limited.
Fully agreed. You can never prevent all issues because there's always a chance for a fire, bitrot due to cosmic rays etc and thus you can only reduce the probability as you correctly concluded.
Unfortunately the world we are in right now just makes the hardware issue even worse with APIs that are prone to introducing bugs in the programs.
Bad hardware should result in the software being extra carefully crafted to balance it out a bit but somehow we ended up with bad hardware and bad software :(
> all three file systems mark pages clean after fsync fails, rendering techniques such as application-level retry ineffective. However, the content in said clean pages varies depending on the file system; ext4 and XFS contain the latest copy in memory while Btrfs reverts to the previous consistent state. Failure reporting is varied across file systems; for example, ext4 data mode does not report an fsync failure immediately in some cases, instead (oddly) failing the subsequent call. Failed updates to some structures (e.g., journal blocks) during fsync reliably lead to file-system unavailability. And finally, other potentially useful behaviors are missing; for example, none of the file systems alert the user to run a file-system checker after the failure.
Surely there's some motivations behind these behaviors and it's not a bug that was implemented in all 3 filesystems, right?
> Surely there's some motivations behind these behaviors
The primary motivation is probably that it's an annoying case to handle, pretty hard to test, and very uncommon.
It's also the original behaviour (IIRC from the fsyncgate reports, freebsd had added keeping the buffers dirty in the early aught but other bsds had inherited the ur-behaviour of marking them clean).
A second motivation I can think of (with more historical relevance) is that I don't think there's a userland API to tell the kernel to discard dirty pages, so if you can't mark the pages clean there's good chances you've leaked them. In our modern world it's common for writeback errors to be transient (e.g. a USB key that's not ready yet or somesuch) but 40 years back I'm not sure it made much sense. Though I guess network drivers were always a thing and could always have issues.
I’d go for historically very uncommon. fsync() meant “write RAM buffers to hard drive,” and if that failed, you were in a world of hurt, and should probably shut down while doing the least amount of additional harm. With NFS, the situation probably changed to “keep trying for a bit, but don’t do anything dramatic.”
One example for marking dirty pages as clean after fsync failure that they mention filesystem developers having given them is a USB stick that has been pulled and keeping dirty pages causing memory leaks in that case. But to me that argument falls flat, they should free the cache if the underlying storage got removed and any subsequent read or write should fail.
> But to me that argument falls flat, they should free the cache if the underlying storage got removed
Except none of that is true, the USB storage could be reattached and the write successful. The system has no way to know whether the failure is transient or permanent.
> any subsequent read or write should fail.
That’s a different solution than keeping the dirty pages. Instead you discard the pages and lock to failure. IIRC that’s what openbsd implemented in the wake of fsyncgate.
> Except none of that is true, the USB storage could be reattached and the write successful. The system has no way to know whether the failure is transient or permanent.
What is not true? I stated an opinion. It is IMHO not the job of the OS to keep data indefinately in RAM just because there might be a chance the storage comes back at some undetermined point in the future. We can fail the write and the application/user can try again from scratch if storage comes back.
> That’s a different solution than keeping the dirty pages. Instead you discard the pages and lock to failure. IIRC that’s what openbsd implemented in the wake of fsyncgate.
Yes it is different, that's why I said keeping the dirty pages clearly as the paper showed causes all kinds of issues leading to data loss or corruption. I don't see this as a good tradeoff. If OpenBSD went that route then that seems like a much safer and saner decision to me.
After re-reading my original comment I see where I wasn't clear enough. Both marking a page as clean and dirty can cause issues if the page is not evicted upon fsync failure. The fact that different filesystems behave differently (some mark clean, some dirty) just makes the problem worse.
> What is not true? I stated an opinion. It is IMHO not the job of the OS to keep data indefinately in RAM just because there might be a chance the storage comes back at some undetermined point in the future. We can fail the write and the application/user can try again from scratch if storage comes back.
Maybe not indefinitely, but USB and some other busses often experience transient resets, but once settled, the devices reappear. An OS that wants to be useful in the face of flakey busses should be able to handle that by not going to failure right away when a device disappears, because it may reappear shortly. Otoh, it's hard to set the appropriate time to wait, which is how nfs mounts to retired servers hang forever.
IIRC Linux itself has only been reporting asynchronous writeback errors via fsync for a few short years, meaning before that basically any database that wasn't using O_DIRECT would miss I/O errors under memory pressure (or from out-of-process writebacks in general, e.g. root invoking sync). I looked into this stuff before postgres's fsyncgate, before "how are I/O errors actually handled in Linux, anyhow?" got attention, and walked away with the notion that anything other than O_DIRECT is best-effort-probably-works-most-of-the-time on a good day, and O_DIRECT's semantics are basically an unknowable opaque mixture of what drivers and hardware do and expect. There were some papers looking at error handling within Linux file systems at the time and they found a large number of issues in pretty much all of them. As far as I know, all efforts in the area of durable I/O are still focused on the notion of synchronizing I/O (fsync/fdatasync and equivalent), while many databases don't actually care about that too much and would rather want barriers instead. The kicker is of course that hardware (when honest) actually uses barriers and not block synchronization, and the databases that are journaling filesystems of course also use barriers and not synchronization to implement journaling. It struck me as a distinctly classic API-to-real-world mismatch.
io_uring already has IOSQE_IO_DRAIN which sounds like it's currently implemented as stalling the IO pipeline, but maybe it could be translated to hardware barriers instead in some circumstances (e.g. when the surrounding IO is all O_DIRECT).
> Our findings show that although applications use many failure-handling strategies, none are sufficient: fsync failures can cause catastrophic outcomes such as data loss and corruption.
That makes it seem like an immediate abort might be the best action in most cases? Handling it wrong and then chugging along might amplify any corruption that has happened.
It might obviously depend on the application and use case, but I'd like to think projects like pgsql put a lot of effort into getting this right after fsyncgate. I've read quite a bit about it after that incident, but ultimately decided I'm too stupid to get that right and roll the "log error and bail out" route ever since.
If you see EIO 99.999 % of the time you want to do just that - stop anything you're doing, don't attempt any further writes. Chances are they'll fail anyway.
The paper explains that even crashing after fsync fails you can end up with lost or corrupted data because after the next start you might get the wrong data from the page cache which survives the crash as it's not part of the programs memory but in the kernel.
I recommend reading the paper or watching the video, it is very interesting.
After decades of issues with the storage layer and even some of the most popular programs written by top notch developers having bugs due to the problematic nature of the APIs and filesystems involved I wish a completely new storage API would emerge. Something that exposes an asynchronous (and synchronous build upon it) API with ACID semantics. Filesystems are nothing more than specialized databases but they don't expose the necessary interface to use them as such.
We need an API that is dead simple and hard to misuse with clearly defined semantics and guarantees but lets seasoned developers still exploit the hardware to its fullest with additional work. Hope dies last I guess :)
Windows used to have an almost unknown "transactional file system API" that sounds similar to what you're asking for. I think it was recently deprecated, but I don't know why, or what the history of this API is. Might be interesting to read up on it!
Nope, I was thinking of transactional NTFS, which AFAIK is still supported, but deprecated.
I'd totally forgotten about WinFS - I remember it being touted years back, and it sounded great... and then it got cut before the next version of Windows was released. I was pretty disappointed, as at the time it really sounded like the future of file systems. Maybe just ahead of it's time.
> Filesystems are nothing more than specialized databases but they don't expose the necessary interface to use them as such.
Most FS are not even transactional, so they can't expose the necessary interfaces (hell, most individual FS calls are not guaranteed to be transactional).
I assume you could build an application-level transactional system on ZFS, but I don't know that it exposes any such APIs either.
My point is that we need a new storage interface that is transactional. It's extremely hard to build a sound interface on the existing ones. And it would not be built on the existing FS because those are already built for the broken interfaces.
And my point is that you’re working from the wrong end: you can’t have a transactional storage interface when you don’t have a widely available and deployed transactional storage.
It’s easy to build a non-transactional interface atop a transactional storage, but if you don’t have the latter the former makes no sense.
When you say transactional storage do you mean hardware support for transactions? I don't think that would be required. As long as the storage hardware interface (e.g. NVME) guarantees certain low level primitives like atomic sector writes.
I guess what you'd want from hardware is similar to what you get with RAM. Atomic reads and writes of certain size plus read and write barriers.
And then ditch the posix FS APIs, replace with a transactional interface that probably uses io_uring under the hood and build an FS that handles it in a correct and performant way.
Right, you're pretty much on the money: you want transactional storage to build upon.
If NVME or flash interfaces worked reliably as you suggest, then you could do this today by just treating them as memory, but right now there is an fsync (or equivalent) somewhere in the call stack. And even then, hardware complexities are wildly abstracted to the point where a lot of storage devices have no guarantees in the face of e.g. power loss.
Older hardware actually excels here! Tape can be used very, very reliably because the interface is simple and obvious in comparison to flash-based storage.
Most CPU architectures provide atomic read, write and barrier primitives. Locking is a higher level interface and usually involves use of those atomic primitives.
The point is that once you have these low level guarantees, you can actually implement a good transactional system on top.
Historically what we got from storage in terms of guarantees was just "once you fsync your data will probably persist if you did it very carefully but not always".
You don't need a new hardware interface. You only need a sensible API for write barriers. In particular, you want "semi-permeable" barriers, which is just another way of saying tagged commands, or grouped writes.
1) An fcntl() to set a numerical tag ID on an fd. All subsequent writes on the fd belong to this ID group.
2) No writes in a tagged group are allowed to proceed until all writes from a preceding group are completed.
3) Any untagged writes can complete at any time in any order, as usual.
Done. That's all you need to implement safe filesystem transactions, while eliminating the bottleneck of fsync().
My vision for the operating system of the future is to build a microkernel that exclusively uses a modified ZFS that has a transactional API, that can run Linux drivers in userspace with zero or few changes, and that uses a uniform event API like Windows' handles or Plan 9's file descriptors.
But that modified ZFS is perhaps the most important part, and I would want to make it so `fsync()` on that platform behaves as people would want it to: transactional such that if it succeeds, the data was written, and if it doesn't, that data was not written.
The ability to run Linux drivers is so there isn't a chicken-and-egg problem with drivers. (It would also be nice to implement the POSIX API to solve that part of the chicken-and-egg problem.) The uniform event API is because the current OS API's are difficult to work with. OS's are based around resources and events. The resource API's are pretty good, but the event API's (select(), poll(), epoll(), io_uring, kqueue, WaitForMultipleObjects(), etc.) are still artificially constrained.
That's one of those things that sounds good in theory, but in practice has more to do with hardware than software semantics.
Any end-to-end guarantees pretty much requires control over hardware, which of course would be very interesting to a lot of people, but isn't really what mainstream operating system concern themselves with.
Already the block device hides too much of hardware, and any VFS layer will be even worse. Just because a write is done doesn't mean it's actually on the spinning rust. And even if it is, it could be done in several ways, maybe it was relocated, or parity hasn't been computed, or it's in the queue. Any attempt of abstracting different types of storage will pretty much have to resort to the least bad common denominator here.
I'm not saying it's a bad idea, just that it's not the same problem that operating systems try to solve. But in the post-Optane world maybe block devices are too low level anyway and we'll finally see higher level storage systems.
You're right, for the most part. We have to have hardware that doesn't lie when the data made it to disk, whatever that disk is. And we also need hardware that can do atomic updates to single sectors.
But once we have those things, and people are demanding them more and more, a copy-on-write filesystem should be perfect for emulating transactions, using that single sector atomic update to "commit" the transaction.
> It would also be nice to implement the POSIX API to solve that part of the chicken-and-egg problem.
If you already have a microkernel capable of running Linux hardware drivers in userspace, it shouldn't be hard to also run userspace ABIs/personalities interfaces as userspace drivers; NetBSD rump drivers more or less officially support what you're doing, and I suspect you could modify user mode linux (UML) to provide ABI compatibility good enough to run unmodified Linux binaries.
There is an impedance mismatch between what is required out of filesystems in terms of backward compatibility and the features/capabilities that would be useful for applications like databases.
If you discard a requirement to look like a standard Linux filesystem to arbitrary applications, writing a purpose-built filesystem for a specific application often requires less code than trying to make off-the-shelf filesystems consistently deliver desired behavior and performance. And even then it is often difficult to guarantee that the off-the-shelf filesystem will really do what you expect in all cases. Installing a custom filesystem isn't always a practical option operationally but it is sometimes done for applications like high-scale data infrastructure because of the headaches it eliminates. I've built a few systems designed to be deployed either way.
I am not sure there is a "one size fits all" solution for this. Alternative filesystem implementations tend to make very application-specific and hardware-aware design choices.
> fsync() causes all modified data and attributes of fildes to be moved to a permanent storage device. This normally results in all in-core modified copies of buffers for the associated file to be written to a disk.
> Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence.
> Specifically, if the drive loses power or the OS crashes, the application may find that only some or none of their data was written. The disk drive may also re-order the data so that later writes may be present, while earlier writes are not.
> This is not a theoretical edge case. This scenario is easily reproduced with real world workloads and drive power failures.
> For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl. The F_FULLFSYNC fcntl asks the drive to flush all buffered data to permanent storage. Applications, such as databases, that require a strict ordering of
> writes should use F_FULLFSYNC to ensure that their data is written in the order they expect. Please see fcntl(2) for more detail.
I said this elsewhere but, in isolation there will always be failure scenarios where recovery is impossible. There are plenty of verification strategies to detect failures, and combined with redundancy, you can reduce the probability of application failure in the face of fsync failures or other similar failures. But you can never eliminate failures. If your storage gives up the ghost, it's game over.
Distributed systems are the closest we've gotten to resilient, durable storage. Redundancy, external verification, quorum. Sometimes the distributed system lives in a single box on your desk.
46 comments
[ 3.0 ms ] story [ 103 ms ] threadI guess the next study would need to look into what manufacturers actually fully adhere to the spec. They've not exactly shown best behaviour in that regard in the past (lying about fsync etc) :(
Importantly this means you can't claim stronger semantics around e.g. atomicity. You can certainly work around most if not all issues, using redundancy, verification and distribution. But in isolation you cannot even properly observe extreme failure scenarios, you can only reduce their probability, and even that is limited.
Unfortunately the world we are in right now just makes the hardware issue even worse with APIs that are prone to introducing bugs in the programs.
Bad hardware should result in the software being extra carefully crafted to balance it out a bit but somehow we ended up with bad hardware and bad software :(
https://news.ycombinator.com/item?id=32462537
Surely there's some motivations behind these behaviors and it's not a bug that was implemented in all 3 filesystems, right?
The primary motivation is probably that it's an annoying case to handle, pretty hard to test, and very uncommon.
It's also the original behaviour (IIRC from the fsyncgate reports, freebsd had added keeping the buffers dirty in the early aught but other bsds had inherited the ur-behaviour of marking them clean).
A second motivation I can think of (with more historical relevance) is that I don't think there's a userland API to tell the kernel to discard dirty pages, so if you can't mark the pages clean there's good chances you've leaked them. In our modern world it's common for writeback errors to be transient (e.g. a USB key that's not ready yet or somesuch) but 40 years back I'm not sure it made much sense. Though I guess network drivers were always a thing and could always have issues.
Except none of that is true, the USB storage could be reattached and the write successful. The system has no way to know whether the failure is transient or permanent.
> any subsequent read or write should fail.
That’s a different solution than keeping the dirty pages. Instead you discard the pages and lock to failure. IIRC that’s what openbsd implemented in the wake of fsyncgate.
What is not true? I stated an opinion. It is IMHO not the job of the OS to keep data indefinately in RAM just because there might be a chance the storage comes back at some undetermined point in the future. We can fail the write and the application/user can try again from scratch if storage comes back.
> That’s a different solution than keeping the dirty pages. Instead you discard the pages and lock to failure. IIRC that’s what openbsd implemented in the wake of fsyncgate.
Yes it is different, that's why I said keeping the dirty pages clearly as the paper showed causes all kinds of issues leading to data loss or corruption. I don't see this as a good tradeoff. If OpenBSD went that route then that seems like a much safer and saner decision to me.
After re-reading my original comment I see where I wasn't clear enough. Both marking a page as clean and dirty can cause issues if the page is not evicted upon fsync failure. The fact that different filesystems behave differently (some mark clean, some dirty) just makes the problem worse.
Maybe not indefinitely, but USB and some other busses often experience transient resets, but once settled, the devices reappear. An OS that wants to be useful in the face of flakey busses should be able to handle that by not going to failure right away when a device disappears, because it may reappear shortly. Otoh, it's hard to set the appropriate time to wait, which is how nfs mounts to retired servers hang forever.
That makes it seem like an immediate abort might be the best action in most cases? Handling it wrong and then chugging along might amplify any corruption that has happened.
It might obviously depend on the application and use case, but I'd like to think projects like pgsql put a lot of effort into getting this right after fsyncgate. I've read quite a bit about it after that incident, but ultimately decided I'm too stupid to get that right and roll the "log error and bail out" route ever since.
That's exactly where postgres ended up: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...
> PANIC on fsync() failure.
I recommend reading the paper or watching the video, it is very interesting.
We need an API that is dead simple and hard to misuse with clearly defined semantics and guarantees but lets seasoned developers still exploit the hardware to its fullest with additional work. Hope dies last I guess :)
> In 2013 Bill Gates cited WinFS as his greatest disappointment at Microsoft and that the idea of WinFS was ahead of its time, which will re-emerge
I'd totally forgotten about WinFS - I remember it being touted years back, and it sounded great... and then it got cut before the next version of Windows was released. I was pretty disappointed, as at the time it really sounded like the future of file systems. Maybe just ahead of it's time.
https://docs.microsoft.com/en-us/windows/win32/fileio/about-...
Most FS are not even transactional, so they can't expose the necessary interfaces (hell, most individual FS calls are not guaranteed to be transactional).
I assume you could build an application-level transactional system on ZFS, but I don't know that it exposes any such APIs either.
It’s easy to build a non-transactional interface atop a transactional storage, but if you don’t have the latter the former makes no sense.
I guess what you'd want from hardware is similar to what you get with RAM. Atomic reads and writes of certain size plus read and write barriers.
And then ditch the posix FS APIs, replace with a transactional interface that probably uses io_uring under the hood and build an FS that handles it in a correct and performant way.
If NVME or flash interfaces worked reliably as you suggest, then you could do this today by just treating them as memory, but right now there is an fsync (or equivalent) somewhere in the call stack. And even then, hardware complexities are wildly abstracted to the point where a lot of storage devices have no guarantees in the face of e.g. power loss.
Older hardware actually excels here! Tape can be used very, very reliably because the interface is simple and obvious in comparison to flash-based storage.
The point is that once you have these low level guarantees, you can actually implement a good transactional system on top.
Historically what we got from storage in terms of guarantees was just "once you fsync your data will probably persist if you did it very carefully but not always".
My vision for the operating system of the future is to build a microkernel that exclusively uses a modified ZFS that has a transactional API, that can run Linux drivers in userspace with zero or few changes, and that uses a uniform event API like Windows' handles or Plan 9's file descriptors.
But that modified ZFS is perhaps the most important part, and I would want to make it so `fsync()` on that platform behaves as people would want it to: transactional such that if it succeeds, the data was written, and if it doesn't, that data was not written.
The ability to run Linux drivers is so there isn't a chicken-and-egg problem with drivers. (It would also be nice to implement the POSIX API to solve that part of the chicken-and-egg problem.) The uniform event API is because the current OS API's are difficult to work with. OS's are based around resources and events. The resource API's are pretty good, but the event API's (select(), poll(), epoll(), io_uring, kqueue, WaitForMultipleObjects(), etc.) are still artificially constrained.
Any end-to-end guarantees pretty much requires control over hardware, which of course would be very interesting to a lot of people, but isn't really what mainstream operating system concern themselves with.
Already the block device hides too much of hardware, and any VFS layer will be even worse. Just because a write is done doesn't mean it's actually on the spinning rust. And even if it is, it could be done in several ways, maybe it was relocated, or parity hasn't been computed, or it's in the queue. Any attempt of abstracting different types of storage will pretty much have to resort to the least bad common denominator here.
I'm not saying it's a bad idea, just that it's not the same problem that operating systems try to solve. But in the post-Optane world maybe block devices are too low level anyway and we'll finally see higher level storage systems.
But once we have those things, and people are demanding them more and more, a copy-on-write filesystem should be perfect for emulating transactions, using that single sector atomic update to "commit" the transaction.
If you already have a microkernel capable of running Linux hardware drivers in userspace, it shouldn't be hard to also run userspace ABIs/personalities interfaces as userspace drivers; NetBSD rump drivers more or less officially support what you're doing, and I suspect you could modify user mode linux (UML) to provide ABI compatibility good enough to run unmodified Linux binaries.
UML sounds interesting.
Other filesystems may also be headed that way.
If you discard a requirement to look like a standard Linux filesystem to arbitrary applications, writing a purpose-built filesystem for a specific application often requires less code than trying to make off-the-shelf filesystems consistently deliver desired behavior and performance. And even then it is often difficult to guarantee that the off-the-shelf filesystem will really do what you expect in all cases. Installing a custom filesystem isn't always a practical option operationally but it is sometimes done for applications like high-scale data infrastructure because of the headaches it eliminates. I've built a few systems designed to be deployed either way.
I am not sure there is a "one size fits all" solution for this. Alternative filesystem implementations tend to make very application-specific and hardware-aware design choices.
from the macOS fsync manpage:
> fsync() causes all modified data and attributes of fildes to be moved to a permanent storage device. This normally results in all in-core modified copies of buffers for the associated file to be written to a disk.
> Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence.
> Specifically, if the drive loses power or the OS crashes, the application may find that only some or none of their data was written. The disk drive may also re-order the data so that later writes may be present, while earlier writes are not.
> This is not a theoretical edge case. This scenario is easily reproduced with real world workloads and drive power failures.
> For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl. The F_FULLFSYNC fcntl asks the drive to flush all buffered data to permanent storage. Applications, such as databases, that require a strict ordering of
> writes should use F_FULLFSYNC to ensure that their data is written in the order they expect. Please see fcntl(2) for more detail.
[0] https://twitter.com/marcan42/status/1494213855387734019
Distributed systems are the closest we've gotten to resilient, durable storage. Redundancy, external verification, quorum. Sometimes the distributed system lives in a single box on your desk.
To assume that any newbie has hit upon a potential failure condition that we didn't already anticipate and account for in LMDB is frankly laughable.