It seems like Time Machine has been steadily declining. I'm not sure why there is no impetus to get it reliable and functioning well. Between sparse bundles becoming corrupt and having to start a new backup and failing functionality I haven't felt like Time Machine is worth setting up anymore. This is in stark contrast to the iOS/iPadOS backups which have worked every time.
I don’t know how many times I’ve had Time Machine decide it didn’t want to work anymore and I had to wipe the backups and start fresh to make it work again, but it’s a much larger number than it should be.
Extrapolate it one more step: Apple is clearly working on an iCloud backup for Macs, cause then that's more services revenue. While they're doing that, why would they fix bugs in Time Machine. People can't surely be using this old thing while we're working on the spiffy new thing!
And it will happily sync corrupt files and does not provide any versioning.
The best TimeMachine is an SSD connected locally to the Mac.
The second best is an SSD running on a Mac setup with TomeMachine Server.
Then you are lucky if backups continue to work. And even luckier if you can sensibly restore anything via the hellscape that is the interstellar wormhole travel interface! ;-)
BackBlaze is very reliable at least!
Not cheap with a house full of computers to backup though :-/
As a non-Mac user, this sounds like a catastrophic and inexcusable bug the likes of which would inspire a dogpile of hatred against desktop operating systems with penguin mascots and/or headquarters in Washington.
Time Machine works fine. Better than anything available on Windows. I’ve used it for more than a decade and a half, with multiple restores, across multiple machines.
My current desktop Mac environment is a direct descendant of my original Mac from 2004 thanks, largely, to Time Machine.
I don't share this experience. I've been running Time Machine for years now on a Samba share for multiple Macs and if anything I've only seen an improvement. Years back I would regularly get a corrupted Time Machine sparse bundle that had to be recreated (or I would restore a previous ZFS snapshot and it would continue of off that), but it also ran over AFP back then I think, not SMB. Lately I've not have had any of these issues on any of the machines. I do have specific flags enabled in the smb.conf file that are recommended for Time Machine backups.
What would the reasoning be for the file system failing this hard? Filling a partition to 99.999999% capacity always produces a nonsensical situation, but it's usually still recoverable without resorting to DBANing it first.
I do not know the details of Apple's file system, but I wouldn't be surprised if it needs to allocate space for the log (journal) and can't do so.
That isn't reasonable in the sense of "this is what the filesystem should do in this situation" but if the log and user data are allocated from the same pool it is quite possible to exhaust both.
I'm so curious about this too. I've done this with ext2 and ext3 in the past and truncating a large file solved the problem if rm wouldn't. It's been long enough I don't remember so specifically though, but certainly wasn't "dd to larger disk and grow the partition" to the rescue.
If it was a journal issue would something akin to using an initramfs (or live environment) and mounting with data=writeback enable removing files? Or maybe APFS doesn't support that?
Time Machine has been consistently unreliable the entirety of the time since it launched well over a decade ago. It should be common knowledge that it sucks.
Use Backblaze if you don’t care about privacy, rsync+ssh to a selfhosted zfs box if you do.
Backblaze Personal does support encryption, but it's always been incomplete. If you supply your own encryption key, it's true that Backblaze can't read your data at rest. But to restore files, you have to send your key to Backblaze's server, which will then decrypt the data so that you can download it. They say that they never store the key and promptly delete the unencrypted files from the server, but to me this is still an unnecessary risk. There's no reason why they couldn't handle decryption locally on the client device, but they justify on-server decryption in the name of convenience -- you can restore files via the web without downloading an app. If you're concerned about this, the solution is to use B2 with a 3rd party app like Arq.
I actually use Arq to send my Time Machine backups and the rest of my NAS to S3 Glacier, in case the house burns down or the drives fail (whichever comes first). It works great and is very cheap!
Expensive external backups if I ever need it is better than none at all. It's a bet, but hey so is insurance.
EDIT: I checked your tool. It's a 1000 bucks to restore 4 TB in 48 hours. If the house burns down, insurance will cover that. I guess now I know I gotta check those drives a bit more.
What? This tool is exceptionally out of date. Retrieval cost is $30/TB at the high end, and for glacier deep archive and a 48 hour window it only costs $2.50/TB. (Plus a few cents per thousand requests, so maybe don't use tiny objects.)
Glacier's percentage-rate-based retrieval pricing was only active from 2012-2016.
The bandwidth charge of $90/TB is still accurate. Though there are ways to reduce it.
I can’t code. What good is it for me if the code is open source ? I can’t vouch for it and I don’t know anyone who vouches for open source code. Also, I have a Mac. Try to find open source software that runs for four years straight without a hitch on Mac. Arq has done that for me.
> If you supply your own encryption key, it's true that Backblaze can't read your data at rest.
It’s worse than this. The private key for data decryption is sent to their server by the installer before you can even set a PEK. Then, setting the PEK sends the password to them too, since that’s where your private key is stored. So you have to take their word not just that they never store the key and promptly delete unencrypted files during restoration, but also that they destroy the unprotected private key and password when you set up PEK. It’s a terrible scheme that seems almost deliberately designed to lull people into a false sense of security.
I had this issue in October 2018 as documented in this Stack Overflow question, whose text I’ll paste below.
I was lucky: I had an additional APFS partition that I could remove, thus freeing up disk space. Took me a while to figure out, during which time I was in a proper panic.
I’m in a pickle here. macOS Mojave, just updated the other day. I managed to fill my disk up while creating a .dmg, and the system froze. I rebooted. Kernel panic.
Boot to Recovery mode. Mount the disk. Open Terminal.
Since the SSD days (I started like 15 years ago with that) I keep a bit of space empty for two reasons: Emergency situations where you may need some extra space and to give SSDs a bit more room to relocate blocks to (they have a certain amount internally reserved already).
I see in that stackexchange post, there's since been another alternative potential solution where you delete you virtual memory partition and if it's large enough it can give you back enough space to allow deleting of files to happen.
Oh! What I would do is take the disk out, mount it on another machine and delete files then put it back........ /s
I think the whole stack of operating systems and tools that assume that this is possible get in trouble when it's not possible. I don't want my computer to become a locked down sandbox but it seems like this is where we are headed.
If you mounted the disk on another machine, I suspect in this situation the filesystem was so full that you'd get the same error when trying to delete, like what the author here encountered when booting into Recovery OS.
The only solution is to copy all the files you want to keep to another (not full) disk, then reformat and copy them back, or if you don't have another disk to copy to, somehow edit the disk directly to "manually" free some space.
I was just joking about how Macbook drives are not replaceable but some replies on this thread have potential real solutions, which is awesome. I forgot you could attach two Macs during the boot process, which is usually used when you are setting up the new computer and copying files but could be used to fix the broken one too. That's another possible fix, buy a new one with twice the disk space and transfer files on setup?
It's true that mounting the affected Data partition on another machine won't help.
And booting Recovery and mounting is equivalent to mounting on another machine.
But it can help, as follows.
Before resorting to a wipe, try booting into Recovery, mount the Data partition, then use rm on a large file. When it fails, unmount the Data partition and run fsck -y on the Data partition at the commandline. If it finds errors and fixes, you'll get free space.
If you can't figure out how to mount the Data partition read/write at the commandline, close Terminal and run Disk Utility, locate the "Data" partition in the sidebar and right-click to Mount. Quit Disk Utility and restart Terminal. You will find user's data files the /Users folder.
You can locate large files on the mounted Data partition with find /Users -size +100M
Use df -h /Users to verify that 100M or more are free. Find and rm large files as needed. Then boot normally and finish cleanup from comfort of normal operation.
Note— Running fsck via Disk First Aid in Disk Utility should be the same as running fsck -y at commandline, but the UI for Disk Utility can be confusing. For example if it can't unmount the drive to perform the repair, DU will misleadingly advise you that the drive has failed and cannot be repaired. DU has some other odd behaviors, so it's more effective to use fsck at the commandline.
Another fine point: It's the specific APFS "Data" partition filesystem that's locked up (user's data), so you need to repair that specific volume, e.g. disk2s2. Look up the Data partition with disutil list
Repairing the drive as a whole (e.g., disk2) is not what you want; this just checks that there's a partition table, which will naturally be OK. Similarly, repairing the other APFS system partitions will not help, nor will repairing the APFS Container disk. Fix the specific "Data" partition.
On Windows it is hellish trying to copy across files like this due to weird permissions issues that crop up. A weird trick that has worked for me is .zip the files you want to copy, copy across the zip, then unzip that.
btrfs used to have this issue, the problem being that the filesystem has to append metadata to do any operation including (ironically) deletion: https://serverfault.com/a/478742.
AFAIK it's fixed now, because btrfs reserves some space and reports "disk full" before it's reached. macOS probably does the same (I'd hope), but it seems in this case the boundary wasn't enforced properly and the background snapshot caused it to write beyond.
Yes, ZFS has the same fundamental issue that all COW file systems have here. We have a reserved pool of space that you can't use except for operations like removing stuff.
The new problem with that reserved pool mechanism is that in 2024 it's probably way too big, because it's essentially a small but fixed percentage of the storage size. Don't let people use thresholds of total size without some kind of absolute cap!
My best guess at what happened (based on a little knowledge of HFS+ disk structures, but not APFS) is that the journal file also filled up, and since deletion requires writing to it and possibly expanding it, you get into the unusual situation where deletion requires, at least temporarily, more space.
macOS continued to write files until there was just 41K free on the drive.
I've (accidentally) ran both NTFS and FAT32 to 0 bytes free, and it was always possible to delete something even in that situation.
Digging around in forums, I found that Sonoma has broken the SMB/Samba-based networking mount procedure for Time Machine restores, and no one had found a solution. This appears to still be the case in 14.4.
In my experience SMB became unreliable and just unacceptably buggy many years ago, starting around the 10.12-10.13 timeframe; and now it looks like Apple doesn't care about whether it works at all anymore.
I hate to think what people without decades of Mac experience do when confronted with systemic, cascading failures like this when I felt helpless despite what I thought I knew and all the answers I searched for and found on forums.
I don't have "decades of Mac experience", but the first thing I'd try is a fsck --- odd not to see that mentioned here.
If I were asked to recover from this situation, and couldn't just copy the necessary contents of the disk to another one before formatting it and then copying back, I'd get the APFS documentation (https://developer.apple.com/support/downloads/Apple-File-Sys...) and figure out what to edit (with dd and a hex editor) to get some free space.
Similarly, Apple has resolved the increasingly-aged stock Bash install, eschewing Bash's subsequent GPLv3 licencing, by making zsh the default login shell.
I'd like to see zsh also adopt GPLv3 to call Apple's bluff.
dash isn't a fully-featured interactive shell, and would be a major step backward from bash. At least with zsh, Apple are moving forward on a feature basis. (And I say this as someone who's still stubbornly set on bash over that newfangled zsh nonsense ;-)
I'm less familiar with fish, but based on a very fuzzy awareness, it's at least fully-featured.
I do encounter dash on a few systems. It's the default shell on my OpenWRT networking kit, for example. I've installed bash where those systems have enough storage to accommodate it.
For a networked Time Machine restore you can reinstall MacOS without restoring first and then use the migration utility to restore from a remote Time Machine. That seems to use a different smb binary which works. Still, I find it infuriating that restoring, one of the most important things you do on a machine, is broken and was not caught by QA.
That's for a journalling filesystem. For CoW filesystems, the issue is that any change to the filesystem is done by making a new file tree containing your change, and then updating the root to point the new tree. Later, garbage collection finds files that are no longer part of an active tree and returns their storage to the pool.
Changes are usually batched to reduce the amount of tree changes to a manageable amount. A bonus of this design is that a filesystem snapshot is just another reference to a particular tree.
This requires space, but CoW filesystems also usually reserve an amount of emergency storage for this reason.
Ah, Apple. SMB has always performed from horribly slowly a few years back, to barely decent recently, but is still way slower than NFS or (oh the irony) Appleshare on the exact same hardware.
Tested a few years ago throughput to a big NAS connected in 10gigEo from a Hackintosh with BlackMagic Disk Speed Test :
* running Windows, SMB achieves 900MB/s
* running MacOS, SMB achieves 200MB/s
* running MacOS, NFS and AFP both achieve 1000MB/s
Anything related to professional work is a sad joke in MacOS, alas.
(People keep repeating that AFP is dead, however it still works fine as a client on my Mac Pro -- and performs so much better than SMB than it's almost comical).
Fun (until you run into it) fact: the same thing is possible with BTRFS and ZFS. If you manage fill it to the brim, you might have a problem. BTRFS tries to become read-only while there is still room for metadata so you can remount it in safe mode and delete something, but no safecguard is perfect.
> ran both NTFS and FAT32 to 0b and was able to delete something.
Was surprised to learn that with Apple silicon-based Macs, not all ports are equal when it comes to external booting:
If you're using a Mac computer with Apple silicon, your Mac has one or more USB or Thunderbolt ports that have a type USB-C connector. While you're installing macOS on your storage device, it matters which of these ports you use. After installation is complete, you can connect your storage device to any of them.
* Mac laptop computer: Use any USB-C port except the leftmost USB-C port when facing the ports on the left side of the Mac.
* iMac: Use any USB-C port except the rightmost USB-C port when facing the back of the Mac.
* Mac mini: Use any USB-C port except the leftmost USB-C port when facing the back of the Mac.
* Mac Studio: Use any USB-C port except the rightmost USB-C port when facing the back of the Mac.
* Mac Pro with desktop enclosure: Use any USB-C port except the one on the top of the Mac that is farthest from the power button.
* Mac Pro with rack enclosure: Use any USB-C port except the one on the front of the Mac that's closest to the power button.
Some filesystems may require allocating metadata to delete a file. AFIK it's a non issue with traditional Berkeley-style systems, since metadata and data come from a separate pools. Notably ZFS has this problem.
This hasn't been a problem you should be able to hit in ZFS in a long time.
It reserves a percent of your pool's total space precisely to avoid having 0 actual free space and only allows using space from that amount if the operation is a net gain on free space.
You're misunderstanding. See the sibling thread where p_l says that this problem has been resolved, and any further occurrence would be treated as a bug. Setting the quota is only done now to reduce fragmentation (ZFS's fragmentation avoidance requires sufficient free space to be effective).
According to rincebrain, the "disk too full to delete files" was fixed "shortly after the fork" which means "shortly after 2012." My information was quite out of date.
> GlobalReserve is an artificial and internal emergency space. It is used e.g. when the filesystem is full. Its total size is dynamic based on the filesystem size, usually not larger than 512MiB, used may fluctuate.
Modern (well, post-ZFS) filesystems operate by moving the filesystem through state changes where data is not (immediately) destroyed, but older versions of the data are still available for various purposes. Similar to an ACID-compliant database, something like a backup or recovery process can still access older snapshots of the filesystem, for various values of "older" that might range from milliseconds to seconds to years.
With that in mind, you can see how we get in a scenario where deleting a file will require a minor bit of storage for recordkeeping the old and new states, before it can actually free up the storage by releasing the old state. There is supposed to be an escape hatch for getting yourself out of a situation where there isn't even enough storage for this little bit of record keeping, but either the author didn't know whatever trick is needed or the filesystem code wasn't well-behaved in this area (it's a corner-case that isn't often tested).
It feels like insanity that the default configuration of any filesystem intended for laymen can fail to delete a file due to anything other than an I/O error. If you want to keep a snapshot, at least bypass it when disk space runs out? How many customers do the vendors think would prefer the alternative?!
It's not really just keeping snapshots that is the issue, usually. It's just normal FS operation, meant to prevent data corruption if any of these actions is interrupted, as well as various space-saving measures. Some FSs link files together when saving mass data so that identical blocks between them are only stored once, which means any of those files can only be fully deleted when all of them are. Some FSs log actions onto disk before and after doing them so that they can be restarted if interrupted. Some FSs do genuinely keep files on disk if they're already referenced in a snapshot even if you delete them – this is one instance where a modal about the issue should probably pop up if disk space is low. And some OSes really really really want to move things to .Trash1000 or something else stupid instead of deleting them.
Pretty much by the time you get to 100% full on ZFS, the latency is going to get atrocious anyway, but from my understanding there are multiple steps (from simplest to worst case) that ZFS permits in case you do hit the error:
1. Just remove some files - ZFS will attempt to do the right thing
2. Remove old snapshots
3. Mount the drive from another system (so nothing tries writing to it), then remove some files, reboot back to normal
4. Use `zfs send` to copy the data you want to keep to another bigger drive temporarily, then either prune the data or if you already filtered out any old snapshots, zero the original pool and reload it by `zfs send` from before.
I don't buy this? What does defragmentation have to do with snapshotting? Defragmentation is just a rearrangement of the underlying blocks. Wouldn't snapshots just get moved around?
The problem is that you have to track down all pointers pointing to specific block.
With snapshotting, especially with filesystems that can only write data through snapshots (like ZFS), blocks can be referred to by many pointers.
It's similar to evaluating liveness of object in a GC, except you're now operating on possibly gigantic heap with very... pointer-ful objects, that you have to rewrite - which goes against core principle of ZFS which is data safety. You're doing essentially a huge history rewrite on something like git repo, with billions of small objects, and doing it safely means you have to rewrite every metadata block that in any way refers to given data block - and rewrite every metadata block pointing to those metadata blocks.
But more pointers is just more cost, not outright inability to do it. The debate wasn't over whether defragmentation itself is costly. The question was whether merely making defragmentation possible would impose a cost on the rest of the system. So far you've only explained why defragmentation on a snapshotting volume would be expensive with typical schemes, which is entirely uncontroversial. But you neither explain why you believe defragmentation would be impossible (no "ability to do it") with your scheme, nor why you believe it's impossible for other schemes to make it possible "for free"?
In fact, the main difficulty with garbage collectors is maintaining real-time performance. Throw that constraint out, and the game changes entirely.
I never claimed it's impossible - I claimed it's expensive. Prohibitively expensive, as the team at Sun found out when they attempted to do so, and offline defrag becomes easy with two-space approach which is essentially "zfs send to separate device".
You can attempt to add an extra indirection layer, but it does not really reduce fragmentation, it just lets you remap existing blocks to another location at a cost of extra lookup. This is in fact implemented in ZFS as solution for erroneous addition of a vdev, allowing device removal though due to performance cost its oriented mostly at "oops, I added the device wrongly, let me quickly revert".
If by "not able to" you meant "prohibitively expensive" - well, I also don't see why it's prohibitively expensive even without indirection. Moving blocks would seem to be a matter of (a) copy the data, (b) back up the old pointers, (c) update the pointers in-place, (d) mark the block move as committed, (e) and delete the old data/backups. If you crash in the middle you have the backup metadata journaled there to restore from. No indirection. What am I missing? I feel like you might have unstated assumptions somewhere?
My bad - I'm a bit too into the topic and sometimes forget what other people might not know ^^;
You're missing the part where (c) is forbidden by design of the filesystem, because ZFS is not just "Copy on Write" by default (like BTRFS, which has in-place rewrite option, IIRC) nor LVM/disk-mapper snapshot which similarly don't have strong invariants on CoW.
ZFS writes data to disk in two ways - a (logically) write-ahead log called ZFS Intent Log (which handles synchronous writes and is read only on pool import), and transaction group sync (txgsync), where all newly written data is linked into new metadata tree, sharing structure with previous TXG metadata tree (so unchanged branches are shared), and the pointer to the head of the tree is committed into on-disk circular buffer of at least 128 pointers.
Every snapshot in ZFS is essentially a pointer to such metadata tree - all writes in ZFS are done by creating a new snapshot. The named snapshots are just rooted in different places in filesystem. This means that sometimes even in case of catastrophic software bug (for example, master branch had for few commits a bug where they accidentally changed on-disk layout of some structures - one person ran master branch and hit that resulting in pool that could not be imported... but the design meant they could tell ZFS import to "rewind" to TXG sync number from before the bug)
Updating the blocks in place violates design invariants - once you violate them, the data safety guarantees are no longer guarantees. And this makes it into minimally offline operation, and at that point the type of client that needs in-place defragmentation can reasonably do the two-space trick (if you're big enough, to make that infeasible, you're probably big enough to easily throw in an extra JBOD at least and relieve fragmentation pressure).
To make latter paragraphs understandable (beware, ZFS internals as I remember them):
ZFS is constructed of multiple layers[1] - from the bottom (somewhat simplified):
1. SPA (Storage Pool Allocator) - what implements "vdevs" - the only layer that actually deals with blocks. It implements access to block devices, mirroring, RAIDz, draid, etc. and exposes single block-oriented interface upwards
2. DMU (Data Management Unit) - An object oriented storage system. Turns bunch of blocks into object-oriented PUT/GET/PATCH/DELETE like setup, with 128bit object IDs. Also handles base metadata - the immutable/write-once trees for turning "here's a 1GB blob of data" into 512b to 1MB portions on disk. For every given metadata tree/snapshot, there is no in-place changes - modifying an object "in place" means that new txgsync has, for given object ID, a new tree of blocks that shares as much structure with previous one as possible.
3. DSL / ZIL / ZAP - provide basic structures on top of the DMU - DSL is what gives you "naming" ability for datasets and snapshots, ZIL handles the write-ahead log for dsync/fsync, ZAP provides a key-value store in DMU objects.
4. ZPL / ZVOL / Lustre / etc - Those are the parts that implement user-visible filesystem. ZPL is ZFS Posix Layer, which is a POSIX-compatible filesystem implemented over object storage. ZVOL does similar but presents emulated block device. Lustre-on-ZFS similarly talks directly to ZFS object layer instead of implementing ODT/OST on top of POSIX files again.
You could, in theory, add an extra indirection layer just for defragmentation, but this in turn makes problematic layering violation (something found at Sun when they tried to implement BPR) - because suddenly SPA layer (the layer that actually handles block-level addressing) needs to understand DMU's internals (or a layer between the two needing bi-directional knowledge). This makes for possibly brittle code, so again - possible but against overarching goals of the project.
I appreciate the long explanation of ZFS, but I don't feel most of it really matters for the discussion here:
> Updating the blocks in place violates design invariants - once you violate them, the data safety guarantees are no longer guarantees.
Again - you can copy blocks prior to deleting anything, and commit them atomically, without losing safety. The fact that you (or ZFS) don't wish to do that doesn't mean it's somehow impossible.
> the type of client that needs in-place defragmentation can reasonably do the two-space trick (if you're big enough, to make that infeasible, you're probably big enough to easily throw in an extra JBOD at least and relieve fragmentation pressure).
You're moving goalposts drastically here. It's quite a leap to go from "has a bit of free space on each drive" to "can throw in more disks at whim", and the discussion wasn't about "only for these types of clients".
And, in any case, this is all pretty irrelevant to whether ZFS could support defragmentation.
> this makes it into minimally offline operation
See, that's your underlying assumption that you never stated. You want defragmentation to happen fully online, while the volume is still in use. What you're really trying to argue is "fully online defragmentation is prohibitive for ZFS", but you instead made the wide-sweeping claim that "defragmentation is prohibitive for snapshotted filesystems in general".
You're hung on the word "impossible" which I never used.
I did say that there are trade offs and that some goals can make things like defragmentation expensive.
ZFS' main design was that it nothing short of (extensive) physical damage should allow destruction of users data. Everything else was secondary. As such, the project was not interested, ever, in supporting in-place updates.
You can design a system with other goals, or ones that are more flexible. But I'd argue that's why BTRFS got undying reputation for data loss - they were more flexible, and that unfortunately also opened way for more data loss bugs.
> You're hung on the word "impossible" which I never used.
That's not true. That was only in the beginning -- "impossible" was only what I originally took (and would still take, but I digress) your initial comment of "ability to defragment is not free" to be saying. It's literally saying that if you don't pay a cost (presumably, performance or reliability), then you become unable to defragment. That sounded like impossibility, hence the initial discussion.
Later you said you actually meant it'd be "prohibitively expensive". Which is fine, but then I argued against that too. So now I'm arguing against 2 things: impossibility and prohibitive-expensiveness, neither of which I'm hung up on.
> ZFS' main design was that it nothing short of (extensive) physical damage should allow destruction of users data. Everything else was secondary.
Tongue only halfway in cheek, but why do you keep referring to ZFS like it's GodFS? The discussion was about "filesystems" but you keep moving the goalposts to "ZFS". Somehow it appears you feel that if ZFS couldn't achieve something then nothing else possibly could?
Analogy: imagine if you'd claimed "button interfaces are prohibitively expensive for electric cars", I had objected to that assertion, and then you kept presenting "but Tesla switched to touchscreens because they turned out cheaper!" as evidence. That's how this conversation feels. Just because Tesla/ZFS has issues with something that doesn't mean it's somehow inherently prohibitive.
> As such, the project was not interested, ever, in supporting in-place updates.
Again: are we talking online-only, or are you allowing offline defrag? You keep avoiding making your assumptions explicit.
If you mean offline: it's completely irrelevant what the project is interested in doing. By analogy, Microsoft was not interested, ever, in allowing NTFS partitions to be moved or split or merged either, yet third-party vendors have supported those operations just fine. And on the same filesystem too, not merely a similar one!
If you mean online: you'd probably be some intrinsic trade-off eventually, but I'm skeptical it's at this particular juncture. Just because ZFS may have made something infeasible with its current implementation, that doesn't mean another implementation couldn't have... done an even better job? e.g., even with the current on-disk structure of ZFS (let alone a better one), even if a defragmentation-supporting implementation might not achieve 100% throughput while a defragmentation is ongoing, surely it could at least get some throughput during a defrag so that it doesn't need to go entirely offline? That would be a strict improvement over the current situation.
> But I'd argue that's why BTRFS got undying reputation for data loss - they were more flexible, and that unfortunately also opened way for more data loss bugs.
Hang on... a bug in the implementation is a whole different beast. We were discussing design features. Implementation bugs are... not in that picture. I'm pretty sure most people reading your earlier comments would get the impression that by "brittleness" you were referring to accidents like I/O failures & user error, not bugs in the implementation!
I'm most surprised by the lack of testing. Macs tend to ship with much smaller SSDs than other computers because that's how Apple makes money ($600 for 1.5TB of flash vs. $100/2TB if you buy an NVMe SSD), so I'd expect that people run out of space pretty frequently.
It saved me just yesterday when I needed to truncate hundreds of gigabytes of Docker logs on a system that had been having some issues for a while but I didn't want to recreate containers.
O_TRUNC
If the file already exists and is a regular file and the
access mode allows writing (i.e., is O_RDWR or O_WRONLY) it
will be truncated to length 0.
...
That seems to be zsh-specific syntax that is like ">" except that overrides a CLOBBER setting[1].
However, it won't work in bash. It will create file named "!" with the same contents as "filename". It is equivalent to "cat /dev/null filename > !". (Bash lets you put the redirection almost anywhere, including between one argument and another.)
Yikes, then I have remembered wrong about bash, thank you.
In that case I'll just always use `truncate -s0` then. Safest option to remember without having to carry around context about which shell is running the script, it seems.
you can't boot the arm Macs into target disk mode, you can only boot to the recovery os and share the drive - it shows up as a network share iirc. I was super annoyed by this a few weeks ago because you can, for example, use spotlight to search for "target disk mode" and it will show up, and looks like it will take you to the reboot in target disk mode option, but once you're there it's just the standard "choose a boot drive" selector.
> That said, no idea why they can’t be used in this case
My intuitive guess here is how the ports are connected to the T2 security chip. One port is as you said a console port that allows access to perform commands to flash/recover/re-provision the T2 chip. Same as an OOB serial port on networking equip.
The rest of the ports the T2 chip has read/write access to devices connected to it. Since this is an OS drive, I'm guessing it needs to be encrypted and the T2 chip handles this function.
Yes, but the decision to use this firmware was made by Apple.
This is like saying my software did not work because it was based on an incompatible version of some library. Maybe so, but that is a bad excuse. Implementing systems is hard, and like the rest of us, Apple should not get away with bad excuses. And this is even more true because they control more of the stack.
OTOH, the current implementation works and is sufficient so Apple could easily decide that it’s not worth modifying firmware that already works to solve an inexistant issue.
We are talking about booting from USB. On a Mac M2. That’s literally the most power user feature of a MacBook.
The 3 users of this feature on this planet are already happy that it’s even possible at all. The only thing Apple could do is to document this clearly like adding a text in the boot drive selector.
Sure, but it also doesn't make it necessary or useful to implement booting from that port - booting from a port IMHO is not a feature that Apple wants to offer to its target audience at all, so it's sufficient if some repair technician can do that according to a manual which says which port to use in which scenario.
The author tried essentially the same thing as what you suggest. He booted into recoveryOS (a separate partition) then from there tried to delete files from the main system partition. But rm failed with the same error "No space left on device". So as others have suggested, truncating a file might have worked "echo -n >file"
The next step I have used and seen recommended after recoveryOS is single user mode, which is what I think I used to solve the same issue on an old mac. I vaguely remember another reason I used single user mode where recovery mode failed but I do not remember any details.
My bet is that you can get nearly the same functionality with single user mode vs booting from external media, but I only have a vague understanding of the limitations of all three modes from 3-5 uses via tutorials.
But charging through many ports requires extra circuitry to support more power on every port, while booting from multiple ports just requires the boot sequence firmware to talk to more than one USB controller (like PC motherboards do, for example)
I had this happen to me, though I can’t recall how I fixed it.
In general I’ve had good success with Time Machine. I, too, have lost TM volumes. I just erased them and started again. Annoying to be sure but 99.99% of the time don’t need a years worth of backups.
The author mentioned copying the Time Machine drive. I have never been able to successfully do that. Last time I tried I quit after 3 days. As I understand it, only Finder can copy a Time Machine drive. Terrible experience.
That said, I’d rather cope with TM. It’s saved me more than it’s hurt me, and even an idiot like me can get it to work.
I did have my machine just complain about one of my partitions being irreparable, but it mounted read only so I was able to copy it, and am currently copying it back.
I don’t know if this is random bit rot, or if something is going wrong with the drive. That would be Bad, it’s a 3TB spinning drive. Backed up with BackBlaze (knock on wood), but I’d rather not have to go through the recovery process if I could avoid it.
Problem is I don’t know how to prevent it. It’s been suggested that SSDs are potentially less susceptible to bit rot, so maybe switching to one of those is a wise plan. But I don’t know.
> The author mentioned copying the Time Machine drive. I have never been able to successfully do that. Last time I tried I quit after 3 days. As I understand it, only Finder can copy a Time Machine drive. Terrible experience.
rsync -av $SOURCE $DEST has never let me down. Copy or delete on Time Machine files using Finder never worked for me.
> Problem is I don’t know how to prevent it. It’s been suggested that SSDs are potentially less susceptible to bit rot, so maybe switching to one of those is a wise plan. But I don’t know.
OpenZFS with two drives should protect you from bit rot. ZFS almost became the Mac file system in Snow Leopard.
I have notes somewhere on roundtripping Time Machine backups between USB drives and network shares. (It's non-trivial, and it's not supported, but it worked.) It was with HFS+ backups, and there were various bits that were "Here Be Dragons", so I never posted them.
I tried, accidentally, to over fill the HDD on a Windows Vista machine. Vista popped up a box telling me I couldn't do it. Unfortunately, in my panic, I didn't take a picture of the warning for posterity.
My work demands that I generate large amounts of data and I don’t know how much I’ll have to generate up-front. So I run out of disk space a lot.
My experience is that Windows and many of its programs will become very unstable with 0 b on the system drive. And about 3 times out of maybe 50, the system also became unbootable. I’ve learned to do whatever I can to free up space before restarting for stability.
The last time I’d regularly run out of space on Win was around Windows 98 times. I never had a problem then. Now in Windows 11 times, it’s a real headache.
I manage to get to 0 or close to that sometimes, usually through uncontorlles pagefile expansion. Some apps may misbehave, but explorer is stable enough to let me delete something.
I’ve done the same on my Mac perhaps twice in the last few years. Like you, I encountered no crash or any other obvious consequences… I just deleted data and moved on. Though I didn’t try leaving it with zero bytes free for an extended amount of time, or rebooting. Who knows what would happen then.
Still, whatever this APFS bug is, the conditions to trigger it are more specific than just filling up the disk.
Impressive. I've never dealt with a situation where even `rm` failed, but I have had the displeasure of using and managing modern Macs with 256 GB (or less) of internal storage. I like to keep a "spaceholder" file of around 16GB so when things inevitably fill up and prevent an update or something else, I can nuke the placeholder without having to surgically prune things with `ncdu`
I find that one of the main benefits of a space holding file is that when it's needed, freeing up that space provides a window of time where you can implement a long-term solution (like buying a new drive with quadruple the storage space of the original for the cost of an hour of that employee/machine's time).
Safe Boot is your magical way to have the computer delete purgeable and temporary files, like boot caches. Hold shift down and once it gets to the login window, restart again.
Otherwise, go to Recovery mode, mount the disk in Disk Utility, and then open Terminal and rm some shit.
I ran into an issue like this in my first ever job! I accidentally filled up a cluster with junk files and the sysadmin started sending me emails saying I needed to fix it ASAP but rm wouldn’t work. He taught me that file truncation usually works when deletion doesn’t, so you can usually do “cat /dev/null > foo” when “rm foo” doesn’t work.
Although note that several comments here report situations where truncation doesn't work either. 21st century filesystem formats are a lot more complex than UFS, and with things like snapshotting and journalling there are new ways for a filesystem to deadlock itself.
However sometimes filesystems can't do that. For those cases, hopefully the filesystem supports: resize-grow, resize-shrink, and either additional temporary storage or is on top of an underlying system which can add/remove backing storage. You may also need to use custom commands to restore the filesystem's structure to one intended for a single block device (btrfs comes to mind here).
I was once in a situation years ago where a critical piece of infrastructure could brick itself irreparably with a deadlock unless it was always able to write to the file system, so I had a backup process just periodically send garbage directly to dev null and as far as I know that dirty hack is still running years later.
I accidentally filled a ZFS root SSD with a massive samba log file (samba log level set way high to debug a problem, and then forgot to reset it), and had to use truncate to get it back.
I knew that ZFS was better about this, but even so I still got that "oh... hell" sinking feeling when you really bork something.
If you can truncate() an existing file (via 'echo > big_file.img' or similar), I would hope the filesystem could deallocate the relevant extents without requiring more space. Seems a bit like a filesystem defect not to reserve enough space to recover from this condition with unlink().
I had an external hard drive that I overfilled by accident while making a manual backup of media files, and after that I couldn't even mount the APFS volume. Apparently it's something that can happen.
People find it a confusing idea to grasp that deleting things actually requires more space, either temporarily or permanently. Other comments here have gone into the details of why some modern filesystems with snapshotting and journalling and so forth actually end up needing to allocate from free space in order to delete stuff.
In a different field: In the early decade of Wikipedia it often had to be explained to people that (at least from roughly 2004 onwards) deleting pages with the intention of saving space on the Wikipedia servers actually did the opposite, since deletion added records to the underlying database.
Related situations:
* In Rahul Dhesi's ZOO archive file format, deleting an archive entry just sets a flag on the entry's header record. ZOO also did VMS-like file versioning, where adding a new version of a file to an archive did not overwrite the old one.
* Back in the days of MS/DR/PC-DOS and FAT, with (sometimes) add-on undeletion utilities installed, deleting a file would need more space to store a new entry into the database that held the restore information for the undeletion utility.
* Back in the days of MS/DR/PC-DOS and FAT, some of the old disc compression utilities compressed metadata as well, leading to (rare but possible) situations where metadata changes could affect compressibility and actually increase the (from the outside point of view) volume size.
"I delete XYZ in order to free space." is a pervasive concept, but it isn't strictly a correct one.
Maybe flamebait, but here is my honest opinion, which I believe is aligned with the hacker ethos: maybe if you were using an open-source operating system you could, with a little experience, write a simple tool that deletes a couple files without allocating new metadata. (Or more likely, somebody else would have been there before you and you could just use their tool.)
Most users on open-source operating systems can't code as well. And even if they could, this still requires knowledge or guesswork to find the trick for the deletion. Some people suggest truncation, and that's possible with a shell, but what would you do if it failed as well?
I have an answer, but I'm not sure you'll find it particularly fulfilling because it is quite hypothetical.
The average user just needs to be able to ask the question in a decent place. E.g., Hacker News or a fitting Stack Exchange site. Some developers not afraid of touching the kernel will see the question (or one like it), and if no workaround (e.g. truncation) is found to be acceptable, someone may decide to look into the kernel source to see if it's feasible at all. They may find a lower level function that deletes without writing metadata. Or they may find the function in the filesystem driver's source code where the metadata is written first, and if that was successful, the actual data is written. In the easiest case, you could create a copy of the function with the calls swapped, and a live CD with the modified driver could be created. (Course, this solution is quite unsafe, as writing the metadata could still fail for some other or related reason, so it's a bit of an emergency solution.)
There are two other filesystems that were mentioned in the discussion here, btrfs and ZFS.
ZFS solved the problem by reserving space, so creating such a tool isn't needed. (However, ZFS is not part of Linux, so I'm not too interested in digging into the details.)
> maybe if you were using an open-source operating system you could, with a little experience, write a simple tool that deletes a couple files without allocating new metadata
For a filesystem where this happens, it would not be simple and it would require a lot of experience to get right.
> Or more likely, somebody else would have been there before you and you could just use their tool.
I don't think open-source makes such a tool much more likely to exist.
I know someone who ran out of disk space on her iPhone and then tried to fix some issues she didn't realize were being caused by it by upgrading the device to a new version of iOS; but then it failed the upgrade as it couldn't resize the disk but had already committed to doing such (I later laboriously figured this out by debugging the process using idevicerestore). I feel like this was a bug in its "how much space will I need to have to install successfully, let's verify I have enough before I begin" calculation, and maybe later versions of iOS have fixed the issue, but sadly they are all even larger and the fixed version would just prevent it from trying to upgrade in the first place, not fix it once it got to this point.
By contrast, ZFS has "slop space" to avoid this very problem (wedging the filesystem by running out of space during a large operation). By default it reserves 3.2% of your volume's space for this, up to 128GB.
So by adjusting the Linux kernel tunable "spa_slop_shift" to shrink the slop space, you can regain up to 128GB of bonus space to successfully complete your file deletion operations:
>By contrast, ZFS has "slop space" to avoid this very problem
As does ext4 (although they call the space "reserved blocks"). 'man tune2fs' for details. As well as most other modern (and not so modern[0]) filesystems.
[0] As I recall, the same was true for SunOS'[1] UFS back in the 1980s.
In ext[234]fs the reserved blocks is something else though: they are reserved to a specific user, by default root. So if normal users fill out the filesystem, the root user still has some space to write. Sort of a simple quota system.
I believe this problem in is only relevant to CoW filesystems. With ext[234]fs you can set the reserved blocks to 0, fill the fs, and always remove files to fix the situation.
Yes - and reserving a percentage of disk space (for this reason) was a routine feature of "real" filesystems decades before ZFS (or Linux) even existed.
It's kinda like how almost any 1980's MS-DOS shareware terminal program was really good at downloading files over a limited-bandwidth connection, but current versions of MS Windows are utter crap at that should-be-trivial task.
239 comments
[ 2.9 ms ] story [ 239 ms ] threadBecause they dont sell Time Capsule anymore. And they want you to backup everything to iCloud to grow their Services Revenue.
But we shall see!
Hopefully they will provide the ability to backup and restore file versions!
But iCloud isn’t a backup.
Its sync.
And it will happily sync corrupt files and does not provide any versioning.
The best TimeMachine is an SSD connected locally to the Mac.
The second best is an SSD running on a Mac setup with TomeMachine Server.
Then you are lucky if backups continue to work. And even luckier if you can sensibly restore anything via the hellscape that is the interstellar wormhole travel interface! ;-)
BackBlaze is very reliable at least! Not cheap with a house full of computers to backup though :-/
It does not even provide a basic progress bar when used on the phone.
My current desktop Mac environment is a direct descendant of my original Mac from 2004 thanks, largely, to Time Machine.
But hey, we get new emojis and moving desktop wallpapers…
Both machines on wired gigabit ethernet, yet the restore took more than 24 hours. And that was for just a 1TB disk.
That isn't reasonable in the sense of "this is what the filesystem should do in this situation" but if the log and user data are allocated from the same pool it is quite possible to exhaust both.
If it was a journal issue would something akin to using an initramfs (or live environment) and mounting with data=writeback enable removing files? Or maybe APFS doesn't support that?
Use Backblaze if you don’t care about privacy, rsync+ssh to a selfhosted zfs box if you do.
EDIT: I checked your tool. It's a 1000 bucks to restore 4 TB in 48 hours. If the house burns down, insurance will cover that. I guess now I know I gotta check those drives a bit more.
What? This tool is exceptionally out of date. Retrieval cost is $30/TB at the high end, and for glacier deep archive and a 48 hour window it only costs $2.50/TB. (Plus a few cents per thousand requests, so maybe don't use tiny objects.)
Glacier's percentage-rate-based retrieval pricing was only active from 2012-2016.
The bandwidth charge of $90/TB is still accurate. Though there are ways to reduce it.
Why use closed source crypto for money when free software that can be reviewed is available gratis? There are much better options.
It’s worse than this. The private key for data decryption is sent to their server by the installer before you can even set a PEK. Then, setting the PEK sends the password to them too, since that’s where your private key is stored. So you have to take their word not just that they never store the key and promptly delete unencrypted files during restoration, but also that they destroy the unprotected private key and password when you set up PEK. It’s a terrible scheme that seems almost deliberately designed to lull people into a false sense of security.
https://support.apple.com/en-gb/guide/disk-utility/dskuf8235...
I was lucky: I had an additional APFS partition that I could remove, thus freeing up disk space. Took me a while to figure out, during which time I was in a proper panic.
---
https://apple.stackexchange.com/questions/338721/disk-full-t...
I’m in a pickle here. macOS Mojave, just updated the other day. I managed to fill my disk up while creating a .dmg, and the system froze. I rebooted. Kernel panic.
Boot to Recovery mode. Mount the disk. Open Terminal.
–bash–3.2# rm /path/to/large/file
rm: /path/to/large/file: No space left on device
Essentially the same issue as this Unix thread from ‘08! https://www.unix.com/linux/69889-unable-remove-file-using-rm...
I’ve tried echo x > /path/to/large/file, no good.
It’s borked. Does anyone have any suggestions that aren’t “wipe the drive and restore from your backup”?
Kind of like the old Unix file systems that would reserve 5% for root.
I think the whole stack of operating systems and tools that assume that this is possible get in trouble when it's not possible. I don't want my computer to become a locked down sandbox but it seems like this is where we are headed.
The only solution is to copy all the files you want to keep to another (not full) disk, then reformat and copy them back, or if you don't have another disk to copy to, somehow edit the disk directly to "manually" free some space.
It's true that mounting the affected Data partition on another machine won't help.
And booting Recovery and mounting is equivalent to mounting on another machine.
But it can help, as follows.
Before resorting to a wipe, try booting into Recovery, mount the Data partition, then use rm on a large file. When it fails, unmount the Data partition and run fsck -y on the Data partition at the commandline. If it finds errors and fixes, you'll get free space.
If you can't figure out how to mount the Data partition read/write at the commandline, close Terminal and run Disk Utility, locate the "Data" partition in the sidebar and right-click to Mount. Quit Disk Utility and restart Terminal. You will find user's data files the /Users folder.
You can locate large files on the mounted Data partition with find /Users -size +100M
Use df -h /Users to verify that 100M or more are free. Find and rm large files as needed. Then boot normally and finish cleanup from comfort of normal operation.
Note— Running fsck via Disk First Aid in Disk Utility should be the same as running fsck -y at commandline, but the UI for Disk Utility can be confusing. For example if it can't unmount the drive to perform the repair, DU will misleadingly advise you that the drive has failed and cannot be repaired. DU has some other odd behaviors, so it's more effective to use fsck at the commandline.
Another fine point: It's the specific APFS "Data" partition filesystem that's locked up (user's data), so you need to repair that specific volume, e.g. disk2s2. Look up the Data partition with disutil list Repairing the drive as a whole (e.g., disk2) is not what you want; this just checks that there's a partition table, which will naturally be OK. Similarly, repairing the other APFS system partitions will not help, nor will repairing the APFS Container disk. Fix the specific "Data" partition.
AFAIK it's fixed now, because btrfs reserves some space and reports "disk full" before it's reached. macOS probably does the same (I'd hope), but it seems in this case the boundary wasn't enforced properly and the background snapshot caused it to write beyond.
https://zfs-discuss.opensolaris.narkive.com/BQ7RMcjo/cannot-...
The new problem with that reserved pool mechanism is that in 2024 it's probably way too big, because it's essentially a small but fixed percentage of the storage size. Don't let people use thresholds of total size without some kind of absolute cap!
macOS continued to write files until there was just 41K free on the drive.
I've (accidentally) ran both NTFS and FAT32 to 0 bytes free, and it was always possible to delete something even in that situation.
Digging around in forums, I found that Sonoma has broken the SMB/Samba-based networking mount procedure for Time Machine restores, and no one had found a solution. This appears to still be the case in 14.4.
In my experience SMB became unreliable and just unacceptably buggy many years ago, starting around the 10.12-10.13 timeframe; and now it looks like Apple doesn't care about whether it works at all anymore.
I hate to think what people without decades of Mac experience do when confronted with systemic, cascading failures like this when I felt helpless despite what I thought I knew and all the answers I searched for and found on forums.
I don't have "decades of Mac experience", but the first thing I'd try is a fsck --- odd not to see that mentioned here.
If I were asked to recover from this situation, and couldn't just copy the necessary contents of the disk to another one before formatting it and then copying back, I'd get the APFS documentation (https://developer.apple.com/support/downloads/Apple-File-Sys...) and figure out what to edit (with dd and a hex editor) to get some free space.
Apple dropped Samba in favor of their own implementation a long time ago after Samba adopted GPLv3:
https://lists.samba.org/archive/samba-announce/2007/000122.h...
https://www.engadget.com/2011-03-24-apple-to-drop-samba-netw...
I'd like to see zsh also adopt GPLv3 to call Apple's bluff.
https://github.com/fish-shell/fish-shell/blob/master/COPYING
https://git.kernel.org/pub/scm/utils/dash/dash.git/tree/COPY... (Default Ubuntu shell since 6.10)
I'm less familiar with fish, but based on a very fuzzy awareness, it's at least fully-featured.
I do encounter dash on a few systems. It's the default shell on my OpenWRT networking kit, for example. I've installed bash where those systems have enough storage to accommodate it.
Changes are usually batched to reduce the amount of tree changes to a manageable amount. A bonus of this design is that a filesystem snapshot is just another reference to a particular tree.
This requires space, but CoW filesystems also usually reserve an amount of emergency storage for this reason.
s/unusual/usual/ surely.
Tested a few years ago throughput to a big NAS connected in 10gigEo from a Hackintosh with BlackMagic Disk Speed Test :
* running Windows, SMB achieves 900MB/s
* running MacOS, SMB achieves 200MB/s
* running MacOS, NFS and AFP both achieve 1000MB/s
Anything related to professional work is a sad joke in MacOS, alas.
(People keep repeating that AFP is dead, however it still works fine as a client on my Mac Pro -- and performs so much better than SMB than it's almost comical).
> ran both NTFS and FAT32 to 0b and was able to delete something.
AFAIK those aren’t journaled, no?
Use an external storage device as a Mac startup disk https://support.apple.com/en-us/111336
Was surprised to learn that with Apple silicon-based Macs, not all ports are equal when it comes to external booting:
If you're using a Mac computer with Apple silicon, your Mac has one or more USB or Thunderbolt ports that have a type USB-C connector. While you're installing macOS on your storage device, it matters which of these ports you use. After installation is complete, you can connect your storage device to any of them.
* Mac laptop computer: Use any USB-C port except the leftmost USB-C port when facing the ports on the left side of the Mac.
* iMac: Use any USB-C port except the rightmost USB-C port when facing the back of the Mac.
* Mac mini: Use any USB-C port except the leftmost USB-C port when facing the back of the Mac.
* Mac Studio: Use any USB-C port except the rightmost USB-C port when facing the back of the Mac.
* Mac Pro with desktop enclosure: Use any USB-C port except the one on the top of the Mac that is farthest from the power button.
* Mac Pro with rack enclosure: Use any USB-C port except the one on the front of the Mac that's closest to the power button.
Hilariously this failure case doesn't seem to be listed in the docs. https://developer.apple.com/library/archive/documentation/Sy...
i find it really frustrating though. why not just reserve some space?
It reserves a percent of your pool's total space precisely to avoid having 0 actual free space and only allows using space from that amount if the operation is a net gain on free space.
https://github.com/openzfs/zfs/blob/99741bde59d1d1df0963009b...
This is a brokwn implementation.
They also said it was mainly used for other issues, such as fragmentation. In other words, this was stated as a fix for the file delete issue.
How does this invalidate my comment, that this was a broken implementation?
It doesn't matter if it will be fixed in the future, or was just fixed.
https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html
> GlobalReserve is an artificial and internal emergency space. It is used e.g. when the filesystem is full. Its total size is dynamic based on the filesystem size, usually not larger than 512MiB, used may fluctuate.
With that in mind, you can see how we get in a scenario where deleting a file will require a minor bit of storage for recordkeeping the old and new states, before it can actually free up the storage by releasing the old state. There is supposed to be an escape hatch for getting yourself out of a situation where there isn't even enough storage for this little bit of record keeping, but either the author didn't know whatever trick is needed or the filesystem code wasn't well-behaved in this area (it's a corner-case that isn't often tested).
1. Just remove some files - ZFS will attempt to do the right thing
2. Remove old snapshots
3. Mount the drive from another system (so nothing tries writing to it), then remove some files, reboot back to normal
4. Use `zfs send` to copy the data you want to keep to another bigger drive temporarily, then either prune the data or if you already filtered out any old snapshots, zero the original pool and reload it by `zfs send` from before.
You can have cheap defrag but comparatively brittle filesystems by making things modifiable in place.
You can have filesystem that has as its primary value "never lose your data", but in exchange defragmentation is expensive.
With snapshotting, especially with filesystems that can only write data through snapshots (like ZFS), blocks can be referred to by many pointers.
It's similar to evaluating liveness of object in a GC, except you're now operating on possibly gigantic heap with very... pointer-ful objects, that you have to rewrite - which goes against core principle of ZFS which is data safety. You're doing essentially a huge history rewrite on something like git repo, with billions of small objects, and doing it safely means you have to rewrite every metadata block that in any way refers to given data block - and rewrite every metadata block pointing to those metadata blocks.
In fact, the main difficulty with garbage collectors is maintaining real-time performance. Throw that constraint out, and the game changes entirely.
You can attempt to add an extra indirection layer, but it does not really reduce fragmentation, it just lets you remap existing blocks to another location at a cost of extra lookup. This is in fact implemented in ZFS as solution for erroneous addition of a vdev, allowing device removal though due to performance cost its oriented mostly at "oops, I added the device wrongly, let me quickly revert".
You're missing the part where (c) is forbidden by design of the filesystem, because ZFS is not just "Copy on Write" by default (like BTRFS, which has in-place rewrite option, IIRC) nor LVM/disk-mapper snapshot which similarly don't have strong invariants on CoW.
ZFS writes data to disk in two ways - a (logically) write-ahead log called ZFS Intent Log (which handles synchronous writes and is read only on pool import), and transaction group sync (txgsync), where all newly written data is linked into new metadata tree, sharing structure with previous TXG metadata tree (so unchanged branches are shared), and the pointer to the head of the tree is committed into on-disk circular buffer of at least 128 pointers.
Every snapshot in ZFS is essentially a pointer to such metadata tree - all writes in ZFS are done by creating a new snapshot. The named snapshots are just rooted in different places in filesystem. This means that sometimes even in case of catastrophic software bug (for example, master branch had for few commits a bug where they accidentally changed on-disk layout of some structures - one person ran master branch and hit that resulting in pool that could not be imported... but the design meant they could tell ZFS import to "rewind" to TXG sync number from before the bug)
Updating the blocks in place violates design invariants - once you violate them, the data safety guarantees are no longer guarantees. And this makes it into minimally offline operation, and at that point the type of client that needs in-place defragmentation can reasonably do the two-space trick (if you're big enough, to make that infeasible, you're probably big enough to easily throw in an extra JBOD at least and relieve fragmentation pressure).
To make latter paragraphs understandable (beware, ZFS internals as I remember them):
ZFS is constructed of multiple layers[1] - from the bottom (somewhat simplified):
1. SPA (Storage Pool Allocator) - what implements "vdevs" - the only layer that actually deals with blocks. It implements access to block devices, mirroring, RAIDz, draid, etc. and exposes single block-oriented interface upwards
2. DMU (Data Management Unit) - An object oriented storage system. Turns bunch of blocks into object-oriented PUT/GET/PATCH/DELETE like setup, with 128bit object IDs. Also handles base metadata - the immutable/write-once trees for turning "here's a 1GB blob of data" into 512b to 1MB portions on disk. For every given metadata tree/snapshot, there is no in-place changes - modifying an object "in place" means that new txgsync has, for given object ID, a new tree of blocks that shares as much structure with previous one as possible.
3. DSL / ZIL / ZAP - provide basic structures on top of the DMU - DSL is what gives you "naming" ability for datasets and snapshots, ZIL handles the write-ahead log for dsync/fsync, ZAP provides a key-value store in DMU objects.
4. ZPL / ZVOL / Lustre / etc - Those are the parts that implement user-visible filesystem. ZPL is ZFS Posix Layer, which is a POSIX-compatible filesystem implemented over object storage. ZVOL does similar but presents emulated block device. Lustre-on-ZFS similarly talks directly to ZFS object layer instead of implementing ODT/OST on top of POSIX files again.
You could, in theory, add an extra indirection layer just for defragmentation, but this in turn makes problematic layering violation (something found at Sun when they tried to implement BPR) - because suddenly SPA layer (the layer that actually handles block-level addressing) needs to understand DMU's internals (or a layer between the two needing bi-directional knowledge). This makes for possibly brittle code, so again - possible but against overarching goals of the project.
The "vdev removal indirection" works because...
> Updating the blocks in place violates design invariants - once you violate them, the data safety guarantees are no longer guarantees.
Again - you can copy blocks prior to deleting anything, and commit them atomically, without losing safety. The fact that you (or ZFS) don't wish to do that doesn't mean it's somehow impossible.
> the type of client that needs in-place defragmentation can reasonably do the two-space trick (if you're big enough, to make that infeasible, you're probably big enough to easily throw in an extra JBOD at least and relieve fragmentation pressure).
You're moving goalposts drastically here. It's quite a leap to go from "has a bit of free space on each drive" to "can throw in more disks at whim", and the discussion wasn't about "only for these types of clients".
And, in any case, this is all pretty irrelevant to whether ZFS could support defragmentation.
> this makes it into minimally offline operation
See, that's your underlying assumption that you never stated. You want defragmentation to happen fully online, while the volume is still in use. What you're really trying to argue is "fully online defragmentation is prohibitive for ZFS", but you instead made the wide-sweeping claim that "defragmentation is prohibitive for snapshotted filesystems in general".
I did say that there are trade offs and that some goals can make things like defragmentation expensive.
ZFS' main design was that it nothing short of (extensive) physical damage should allow destruction of users data. Everything else was secondary. As such, the project was not interested, ever, in supporting in-place updates.
You can design a system with other goals, or ones that are more flexible. But I'd argue that's why BTRFS got undying reputation for data loss - they were more flexible, and that unfortunately also opened way for more data loss bugs.
That's not true. That was only in the beginning -- "impossible" was only what I originally took (and would still take, but I digress) your initial comment of "ability to defragment is not free" to be saying. It's literally saying that if you don't pay a cost (presumably, performance or reliability), then you become unable to defragment. That sounded like impossibility, hence the initial discussion.
Later you said you actually meant it'd be "prohibitively expensive". Which is fine, but then I argued against that too. So now I'm arguing against 2 things: impossibility and prohibitive-expensiveness, neither of which I'm hung up on.
> ZFS' main design was that it nothing short of (extensive) physical damage should allow destruction of users data. Everything else was secondary.
Tongue only halfway in cheek, but why do you keep referring to ZFS like it's GodFS? The discussion was about "filesystems" but you keep moving the goalposts to "ZFS". Somehow it appears you feel that if ZFS couldn't achieve something then nothing else possibly could?
Analogy: imagine if you'd claimed "button interfaces are prohibitively expensive for electric cars", I had objected to that assertion, and then you kept presenting "but Tesla switched to touchscreens because they turned out cheaper!" as evidence. That's how this conversation feels. Just because Tesla/ZFS has issues with something that doesn't mean it's somehow inherently prohibitive.
> As such, the project was not interested, ever, in supporting in-place updates.
Again: are we talking online-only, or are you allowing offline defrag? You keep avoiding making your assumptions explicit.
If you mean offline: it's completely irrelevant what the project is interested in doing. By analogy, Microsoft was not interested, ever, in allowing NTFS partitions to be moved or split or merged either, yet third-party vendors have supported those operations just fine. And on the same filesystem too, not merely a similar one!
If you mean online: you'd probably be some intrinsic trade-off eventually, but I'm skeptical it's at this particular juncture. Just because ZFS may have made something infeasible with its current implementation, that doesn't mean another implementation couldn't have... done an even better job? e.g., even with the current on-disk structure of ZFS (let alone a better one), even if a defragmentation-supporting implementation might not achieve 100% throughput while a defragmentation is ongoing, surely it could at least get some throughput during a defrag so that it doesn't need to go entirely offline? That would be a strict improvement over the current situation.
> But I'd argue that's why BTRFS got undying reputation for data loss - they were more flexible, and that unfortunately also opened way for more data loss bugs.
Hang on... a bug in the implementation is a whole different beast. We were discussing design features. Implementation bugs are... not in that picture. I'm pretty sure most people reading your earlier comments would get the impression that by "brittleness" you were referring to accidents like I/O failures & user error, not bugs in the implementation!
Finally... you might enjoy [1]. ;)
[1] https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_bl...
Rinse and repeat.
the trick is to truncate a large enough files, or enough small files, to zero.
not sure if this is a universal shell trick, but worked on those i tried: "> filename"
Since then I memorized this: `cat /dev/null >! filename`, and it has worked on systems with zsh and bash.
I believe "> filename" only works correctly if you're root (at least in my experience, if I remember correctly).
EDIT: To remove <> from filename placeholder which might be confusing, and to put commands in quotes.
It saved me just yesterday when I needed to truncate hundreds of gigabytes of Docker logs on a system that had been having some issues for a while but I didn't want to recreate containers.
"truncate -s 0 /var/lib/docker/containers/**/*-json.log"
Will truncate all of the json logs for all of the containers on the host to 0 bytes.
Of course the system should have had logging configured better (rotation, limits, remote log) in the first place, but it isn't my system.
EDIT: Missing double-star.*
However, it won't work in bash. It will create file named "!" with the same contents as "filename". It is equivalent to "cat /dev/null filename > !". (Bash lets you put the redirection almost anywhere, including between one argument and another.)
---
[1] See https://zsh.sourceforge.io/Doc/Release/Redirection.html
In that case I'll just always use `truncate -s0` then. Safest option to remember without having to carry around context about which shell is running the script, it seems.
: is a shell built-in for most shells that does nothing.
https://man7.org/linux/man-pages/man1/truncate.1.html
https://support.apple.com/en-us/108900
That said, no idea why they can’t be used in this case
My intuitive guess here is how the ports are connected to the T2 security chip. One port is as you said a console port that allows access to perform commands to flash/recover/re-provision the T2 chip. Same as an OOB serial port on networking equip.
The rest of the ports the T2 chip has read/write access to devices connected to it. Since this is an OS drive, I'm guessing it needs to be encrypted and the T2 chip handles this function.
The rest of the code necessary to boot from external sources is located on main flash
This is like saying my software did not work because it was based on an incompatible version of some library. Maybe so, but that is a bad excuse. Implementing systems is hard, and like the rest of us, Apple should not get away with bad excuses. And this is even more true because they control more of the stack.
The 3 users of this feature on this planet are already happy that it’s even possible at all. The only thing Apple could do is to document this clearly like adding a text in the boot drive selector.
Also on my mbpro at least the mentioned port is the one closest to the magsafe connector and may have funny electrical connections to it, perhaps.
My bet is that you can get nearly the same functionality with single user mode vs booting from external media, but I only have a vague understanding of the limitations of all three modes from 3-5 uses via tutorials.
iirc, not all ports were equal when it came to charging with the m1 macs, so this is actually not so surprising.
In general I’ve had good success with Time Machine. I, too, have lost TM volumes. I just erased them and started again. Annoying to be sure but 99.99% of the time don’t need a years worth of backups.
The author mentioned copying the Time Machine drive. I have never been able to successfully do that. Last time I tried I quit after 3 days. As I understand it, only Finder can copy a Time Machine drive. Terrible experience.
That said, I’d rather cope with TM. It’s saved me more than it’s hurt me, and even an idiot like me can get it to work.
I did have my machine just complain about one of my partitions being irreparable, but it mounted read only so I was able to copy it, and am currently copying it back.
I don’t know if this is random bit rot, or if something is going wrong with the drive. That would be Bad, it’s a 3TB spinning drive. Backed up with BackBlaze (knock on wood), but I’d rather not have to go through the recovery process if I could avoid it.
Problem is I don’t know how to prevent it. It’s been suggested that SSDs are potentially less susceptible to bit rot, so maybe switching to one of those is a wise plan. But I don’t know.
rsync -av $SOURCE $DEST has never let me down. Copy or delete on Time Machine files using Finder never worked for me.
> Problem is I don’t know how to prevent it. It’s been suggested that SSDs are potentially less susceptible to bit rot, so maybe switching to one of those is a wise plan. But I don’t know.
OpenZFS with two drives should protect you from bit rot. ZFS almost became the Mac file system in Snow Leopard.
There was a time when OSs could deal with the system disk being full quite well. And not so long ago.
I'm always amazed at how the file system survives just fine, but the machine doesn't even crash!
I'm not sure where you got your experience from.
My experience is that Windows and many of its programs will become very unstable with 0 b on the system drive. And about 3 times out of maybe 50, the system also became unbootable. I’ve learned to do whatever I can to free up space before restarting for stability.
The last time I’d regularly run out of space on Win was around Windows 98 times. I never had a problem then. Now in Windows 11 times, it’s a real headache.
Not sure how you’re so lucky.
Still, whatever this APFS bug is, the conditions to trigger it are more specific than just filling up the disk.
I do not know how to fix it without reinstalling
Does it actually do that? I.e. stop the user from writing new data when storage space is extremely low?
[0] https://www.cockroachlabs.com/docs/v23.2/cluster-setup-troub...
Otherwise, go to Recovery mode, mount the disk in Disk Utility, and then open Terminal and rm some shit.
However sometimes filesystems can't do that. For those cases, hopefully the filesystem supports: resize-grow, resize-shrink, and either additional temporary storage or is on top of an underlying system which can add/remove backing storage. You may also need to use custom commands to restore the filesystem's structure to one intended for a single block device (btrfs comes to mind here).
/dev/null is magical and worth reading into
I knew that ZFS was better about this, but even so I still got that "oh... hell" sinking feeling when you really bork something.
In the end I was able to mount and rescue the data using https://github.com/libyal/libfsapfs
I followed this guide: https://matt.sh/apfs-object-map-free-recovery
In a different field: In the early decade of Wikipedia it often had to be explained to people that (at least from roughly 2004 onwards) deleting pages with the intention of saving space on the Wikipedia servers actually did the opposite, since deletion added records to the underlying database.
Related situations:
* In Rahul Dhesi's ZOO archive file format, deleting an archive entry just sets a flag on the entry's header record. ZOO also did VMS-like file versioning, where adding a new version of a file to an archive did not overwrite the old one.
* Back in the days of MS/DR/PC-DOS and FAT, with (sometimes) add-on undeletion utilities installed, deleting a file would need more space to store a new entry into the database that held the restore information for the undeletion utility.
* Back in the days of MS/DR/PC-DOS and FAT, some of the old disc compression utilities compressed metadata as well, leading to (rare but possible) situations where metadata changes could affect compressibility and actually increase the (from the outside point of view) volume size.
"I delete XYZ in order to free space." is a pervasive concept, but it isn't strictly a correct one.
The average user just needs to be able to ask the question in a decent place. E.g., Hacker News or a fitting Stack Exchange site. Some developers not afraid of touching the kernel will see the question (or one like it), and if no workaround (e.g. truncation) is found to be acceptable, someone may decide to look into the kernel source to see if it's feasible at all. They may find a lower level function that deletes without writing metadata. Or they may find the function in the filesystem driver's source code where the metadata is written first, and if that was successful, the actual data is written. In the easiest case, you could create a copy of the function with the calls swapped, and a live CD with the modified driver could be created. (Course, this solution is quite unsafe, as writing the metadata could still fail for some other or related reason, so it's a bit of an emergency solution.)
There are two other filesystems that were mentioned in the discussion here, btrfs and ZFS.
ZFS solved the problem by reserving space, so creating such a tool isn't needed. (However, ZFS is not part of Linux, so I'm not too interested in digging into the details.)
btrfs users apparently accept this as a fact-of-life, but have what they consider decent-enough workarounds, see e.g. https://www.reddit.com/r/btrfs/comments/ibjrpm/can_i_somehow....
(I use neither ZFS nor btrfs; I prefer boring filesystems, thank you very much.)
For a filesystem where this happens, it would not be simple and it would require a lot of experience to get right.
> Or more likely, somebody else would have been there before you and you could just use their tool.
I don't think open-source makes such a tool much more likely to exist.
So by adjusting the Linux kernel tunable "spa_slop_shift" to shrink the slop space, you can regain up to 128GB of bonus space to successfully complete your file deletion operations:
https://openzfs.github.io/openzfs-docs/Performance%20and%20T...
As does ext4 (although they call the space "reserved blocks"). 'man tune2fs' for details. As well as most other modern (and not so modern[0]) filesystems.
[0] As I recall, the same was true for SunOS'[1] UFS back in the 1980s.
[1] https://en.wikipedia.org/wiki/SunOS
I believe this problem in is only relevant to CoW filesystems. With ext[234]fs you can set the reserved blocks to 0, fill the fs, and always remove files to fix the situation.
It's kinda like how almost any 1980's MS-DOS shareware terminal program was really good at downloading files over a limited-bandwidth connection, but current versions of MS Windows are utter crap at that should-be-trivial task.