Doesn’t help with the firmware problem though. If your old boot environment’s driver set doesn’t work with the newer flashed firmware things won’t work as before.
That does sound like a problem Apple could solve if they wanted to. I vaguely recall that some PC motherboards came with a duplicate BIOS, so if an update broke it, it could fall back to a clean copy. AFAIK, some smartphones do this as well.
Of course, if Apple manages to not break their OS with updates, that's not a problem in practice. I don't have a lot of experience with macOS, so I have no idea how problem-prone Apple's updates are.
Does macOS support labeled snapshots instead of time-based ones yet? Is there a way to tell the OS that a certain snapshot never expires? Does disabling and then reenabling Time Machine still delete all your snapshots?
Going into Disk Utility, and selecting View > (Show/Hide) APFS Snapshots I got a list. The most recent one on my system was called "com.apple.TimeMachine.2023-07-15-131420.local". I right-clicked on it and changed "2023-07-15-131420.local" to "foo" and, after a warning, it renamed it (only the com.apple seems to be mandatory):
Unpopular opinion:
I haven’t had a bad update in the last 12 years. Yes, there were some bugs here and there, but nothing dramatic that wanted me to switch back.
The worst thing was Intel’s heartbleed issues that killed the CPU, it was around the 8th gen i7 if my memory is correct.
Agreed, if anything minor updates have been either transparent or visibly addressing issues. I have had some problems when updating from major macOs version to another though. Windows unfortunately has the tendency to break even with minor updates regularly. It is so much of an issue that we have a process at work to test those updates on a few machines and roll them widely to the remaining after.
macOS 10.15.6 had a showstopper kernel zpool memory leak if you used virtualization software to run windows VMs, requiring multiple reboots per day in order to stay in place. It was an absolute dealbreaker for productivity as a developer who needed windows vms.
MacOS Mojave (e.g. 10.14 from 2018) ruined font rendering on standard DPI screens, which included every single Macbook Air from before that year (the first 'Retina' Macbook Air was released in 2018.)
I don’t see this mention the most likely reason: Apple doesn’t really design their software to support arbitrary rollbacks in that direction. What should happen if an app writes out a database in a new format the old version can’t read? Clearly things would break so they don’t want you going in that direction. (For RSRs they can simply avoid doing things like that.)
> What should happen if an app writes out a database in a new format the old version can’t read?
This generally shouldn't be happening in minor updates, and even if it does, there are ways of designing apps to handle this. I've done it before. (And in fact, software developers themselves often roll back versions of their software while developing and testing, which you should know.) As the author indicates, rolling back is possible, albeit overly difficult, so the possibility must be accounted for in any case.
Major OS updates are different story. The key there is to avoid being an early adopter. ;-)
I mean they absolutely could do this but as you know they don’t actually care much. Most of the time they don’t do breaking changes like this but sometimes they do and you probably don’t want to be one of the people unlucky enough to be caught by this.
>Apple doesn’t really design their software to support arbitrary rollbacks in that direction
Up until recently they did support rollbacks from a bad update, so I'm not sure the above holds as a reason. They used to, and if it's harder now then that is part of a deliberate design decision.
At my last job we had a huge problem with employees not wanting to update iOS and macOS software because they feared that it would change things for the worse.
In the case of macOS updates, you could/should instead:
- inform users of the difference between feature updates and security updates
- show them how to install security updates rather than major version changes
- give them a short window after which they must at least upgrade to the latest security patch on whatever version of macOS they're running so that they can choose their own update time for minimal disruption
There's no need to silently break anyone's computer while they sleep, or shut it down while they're working. It's shameful to even expect employees to put up with that.
Especially if they have valid concerns of degraded performance from past updates. That’s a recipe for an adversarial user-IT relationship and inevitably drives users to attempt their own, sometimes risky, work arounds.
Like developer experience roles, maybe security departments need specialist 'user experience', teams or ICs. I think there's fertile middle ground here between anarchy and the received wisdom on 'best practices'.
Unfortunately macOS didn't have an MDM profile to mandate installing updates within a certain timeframe. You can delay them up to xx days but not enforce them.
Not sure if this was fixed in the last 3 years because I moved away from Mac management. But it was a problem unless you used something like jamf which roll their own mechanism.
Intune only supports what Apple offers so they're was no options there
That's awful. Maybe I was wrong, and 'the reason' is security and policy tooling where user experience is an afterthought. :\\
I think there's a lot of room for improvement in this area but I don't see a lot of interest on the part of implementers or vendors. I think it's probably the usual incentives problem with enterprise software, which is tough.
You're technically correct, the best kind of correct, but you essentially responded so someone saying "users have fears" with "respond with (maximum) violence!"
Hey, it's "better" to have your entire department/division/company grind to a halt because everyone's machines broke at the same time, rather than pockets of glitches that you can't quite explain, and you have to individually query each device's version number and patch level.
If you haven't read the article: it is not a complaint. It's a relatively detailed explanation including a number of reasons why rollbacks are not supported. It also gives a method to do your own rollback! (I am not knowledgeable enough to verify the practicality of this method.)
In short, the author's argument is:
- no method to roll back firmware updates
- the sealed system volume is huge and rollback capability would require keeping the whole old ~10GB around, which is a lot of wasted space
The storage isn't soldered to the motherboard on a PC typically so you pay commodity prices for storage instead of Apples markup, making this less of a problem.
Many low end laptops, like some chromebook, feature soldered eMMC. For example, the first thing that I found on the google store, a samsung chromebook[0].
Those are quite literally not PCs, though. They are laptops, but they are much closer to a cellphone than to a PC. They're far more comparable to an iPad than to a Mac, if we're still in that comparison.
I have a Lenovo laptop with eMMC. It isn't unusual. Fortunately they had an undocumented M.2 slot inside but that wasn't ever used in a stock configuration.
They said they are laptops, they’re not PCs. “PC” is a Windows term, traditionally. So a Chromebook is quite literally not a PC. You say they can boot Windows, so that makes the argument a bit less solid, but it wasn’t sold as a PC, then. In my opinion, continuing this line of argument is kind of silly, the point is that most decent consumer laptops that come provisioned with Windows still have replaceable storage. Or maybe I’m just very lucky and all the laptops I’ve purchased did. Even my mini-PCs like Intel NUCs have replaceable storage.
2 minutes of searching for various combinations of "windows laptop", "soldered SSD" and "upgradeable SSD" turned up e.g. Dell's XPS 13, which is not an obscure or low-end device by any metric.
You're correct, it seems like from 2022 the base version of the XPS 13 unfortunately can't have it's SSD upgraded. I searched too, but I looked at the XPS 13 for 2023, which only comes in the Plus model, and that one can be upgraded.
Don’t know if you’ve looked at that segment but, maybe it’s not what you loop for, but it started not long after Apple did (HP & Dell’s MacBook Pro design-ripoff series especially)
Also many lower end laptops have done it for a looooong time.
I dunno if I’d call a decade “recent” when it’s like… 1/3 of the lifetime of the form factor!
But, yes. It’s a form factor thing really, as people want smaller, thinner, lighter, laptops. Memory (and battery) removability gets sacrificed as those aren’t a priority for most customers
One of the disks is. Not all of them are. There is this wonderful technology called "external storage".
But even if the backup has to be stored on the on-board SSD for some reason, so what? It's temporary. That storage can be reclaimed when it's needed. MacOS is already using on-board SSD behind the scenes for Time Machine backups when you don't have an external drive attached. I see absolutely no reason why it could not do the same for a rollback image.
The real reason you can't roll back is simply that Apple doesn't want to allow it, because that would make it easier for the market to reject one of their updates.
> The real reason you can't roll back is simply that Apple doesn't want to allow it, because that would make it easier for the market to reject one of their updates.
Bingo. OpenSUSE makes it easy using BTRFS snapshots, enabled by default. Apple and Microsoft could make it easy too but empowering users like this isn't in their interest.
> The real reason you can't roll back is simply that Apple doesn't want to allow it, because that would make it easier for the market to reject one of their updates.
Right. That's why your Time Machine backups stop worki--oh wait, their integrated backup software lets you restore your disk to the state before the update. So your narrative is FUD.
No, it isn't. What you say is true in theory but it doesn't actually work in practice because major upgrades come with non-backwards-compatible firmware changes. Try installing Mavericks on a machine that has been upgraded to Yosemite or later.
(It also puts the burden on the user to actually make a backup before upgrading.)
I don't really see an explanation for the complexity though, this is usually itself very interesting.
After the live CD years it became much easier to abandon the headaches and work with large signed images, but most OS vendors have had at least one pressure to continue with the hellscape of patch permutations and one way configuration adaptions.
They only need to store the changed part, and the firmware thing was always a thing, also back when it was possible to roll back updates.
This reads like Microsoft’s ‘the old new thing’ where they make up excuses for choices made by Microsoft and here for Apple.
I presume the real reason is that Apple would say instead of preparing for the update to fail, it’s better to make sure it doesn’t fail in the first place.
10GB is nothing nowadays. That much storage costs less than $1. That is a ridiculously small price to pay to be able to roll back from a buggy upgrade.
I thought you were joking, but just confirmed: a 13" Macbook Air M1 defaults to 256GB SSD, with a $200 upgrade cost to get to 512GB, +$400 if you want to hit 1TB.
Regardless of what Apple and other manufacturers charge for more storage, 10GB is still lots of storage space. It might be transferred in ~100 seconds with a gigabit connection, but considering I can store my whole paper archive in ~1GB of space, that's a lot.
I'd be removing that snapshot as soon as possible after a successful update.
You can remove a couple of gig of wallpapers and default apps to resolve it, and anyway, you dont need to waste space forever, just until it's clear the system is stable and the space need is critical, so that's not a good reason
Windows 10/11 keeps 20GB of rollback data on the drive for 30 or something days after an in place upgrade. This works fine on machines even with only 128GB disks
They used to do something to this effect with older MacOS X updates by only transferring a bundle of the files, but at some point they just sliced the OS into giant partitions and started sending the full file system image over the wire regardless of the actual changes.
This wouldn't be such a wasteful idea had they used file system snapshot differential block transfer a la ZFS send. All I can guess is that they either think their customers are so rich they must all live in places with gigabit internet available, or they just don't care and everyone suffers. Either way it's a massive waste of time, resources and energy, just not theirs.
> This wouldn't be such a wasteful idea had they used file system snapshot differential block transfer a la ZFS send.
That would have been so epic. They should do that in FreeBSD. Take the macOS idea of having a separate protected partition for the base system, and combine that with the support for ZFS in FreeBSD to use ZFS snapshots to update the FreeBSD base OS, instead of having to run freebsd-update which applies a bunch of patches
I don't really like that, because it would make it impossible to modify stuff that I want to modify.
I moved to FreeBSD precisely because I wanted more control than Apple would give me (which is ever less of course).
And freebsd-update works well, it can even validate files so you can see which have been manually changed. This is perfect for me, it's not a hard lock to what the provider decides but more of a suggestion.
Not so for Apple SSDs, upgrading from a 256GB to 2TB SSD is an extra $720 (US), when in reality top of the line 2TB nvme SSDs cost about $150. It's insane to me that people accept Apple's massive markup.
>- the sealed system volume is huge and rollback capability would require keeping the whole old ~10GB around, which is a lot of wasted space
IIRC that's how Windows handles update rollbacks, by doing this. Though disk space tends to be much cheaper on Windows devices, on Apple devices you pay a premium for extra internal storage.
When Safari 2.0.4 was released many many aeons ago a whole bunch of sites broke. Not because of browser issues, but because many websites were looking for "netscape" and "4" in the user agent to detect netscape 4. Then they'd (try to) us JSS and layers \o/
Apple is in the top #10 profitable companies worldwide, $400B in annual revenue, decades of OS development experience, and tighter control over OS & hardware integration than any other company on earth. In short, they lack neither resources nor expertise.
"It might be hard" is not a good reason when previously they have done so but made a design &/or business &/or UX decision not to. I very much doubt that difficulty was the driving factor in this choice.
I make a full system drive clone with carbon copy cloner before every update. It was handy that time that Apple released a major operating system version that broke all my USB-C hubs for 6 months...
I honestly have not seen any larger application (enterprise/custom) which allowed real rollback. In practice it's always been "reinstall / restore from backup". I don't see how something as complex as an OS could have that.
Maybe it technically falls under 'restore from backup', but OpenSUSE comes configured to create BTRFS snapshots before and after every package manager operation. Since this excludes the user's home, the effect is that you can rollback an update quickly without losing any work. I think this sort of system would work well for MacOS/Windows/etc, but perhaps it would give the users a bit too much power than these companies prefer...
btrfs is awesome. The number of times I bonked my system only to restore it back just like that. Really gives the confidence to experiment and make bold changes to your system.
Very happy NixOS user here. If I have an issue with any upgrade, I just boot my system into the previous version, and then all is well until I find time to fix anything in the new version.
I upgraded from Mohave to Monterey because a speaker was crackling and some posts online said it might be a software issue. Big mistake!
The upgrade removed emacs, the C dev tools, and deleted any top-level directories it didn't know about / approve of, including my backup!
I have Carbon Copy Cloner and was dumb enough to think I didn't need to use it before this upgrade. Now instead of a development system, I have a glorified web terminal. What a pain in the ass.
Actually deleting things doesn’t sound good, I guess they didn’t test things on such big version jumps or something?
Anyway seeing as Apple froze GPL software at the last GPL2 version, surely you’re using Homebrew or MacPorts to get a newer version of emacs anyway?
As for C compilers, I’m not sure if they’ve ever been in the base install, but AFAIK it’s “xcode-select --install” you want if you don’t want to install all of Xcode.
> surely you’re using Homebrew or MacPorts to get a newer version of emacs anyway?
Nope, because I don't care about emacs' latest upgrades. I've been using the terminal version of emacs (emacs-nox) since the 80's and whatever version is installed has always worked fine.
I tried to install xcode tools yesterday. First it has a license agreement window that needs a click, but it doesn't pop-up in the foreground so I didn't even know it was there. The Install button just error beeps. Then it said it would take 59 hours to download. That's nice. I aborted it.
I restarted it today. It says it has 11 minutes to go so maybe things will go smoother this time. Still a complete pain in the ass, and nope, it didn't fix the speaker (I didn't think it would, just a long shot).
Even just using the terminal version there's a LOT of changes for the better than the ancient version Apple shipped (22?). Package management, native compilation, etc.,.
Somehow I prefer that over Windows' infamous installer always flying trough to 99% in 1% of the time, only to waste the next 99% of total install time to finish up that last 1%
Have a look in /System/Volumes/Data. Catalina moved to having a separate system and data volume. The system volume is mounted at /, with directories from the data volume firmlinked[0] in. The data volume by itself is mounted at /System/Volumes/Data.
Do bad macOS upgrades even happen that often? Been using macOS full time since about 2007 and I can’t ever recall having a bad OS upgrade. I usually go entire hardware cycles without ever having to fresh install the OS.
Somehow Ventura changed the way PDFs are handled, and broke three years of my work.
I'm not an OSX developer (I'm just a writer) and still trying to figure out where the change happened and why. The devs upon whom I depend are also mystified and/or not talking.
Being able to roll back updates when so many people depend on your OS seems like a good thing to me.
Evidently changes to the way PDFs are rendered by the OS, I think via Preview? So for example, if you 'print' a PDF from another app, a lot of stuff gets broken. I'm sorry, I wish I had more information.
I have just as much trouble on Linux, but one could assume that we could do it right by first installing to a subfolder and then restarting with that over root using overlayfs and then restarting again on confirm with changes moved into place.
It would be cool for optional dist-upgrades and it would just be so phenomenal to be able to update and then revert - like we can for monitor settings.
You can do that with NixOS, booting back to an earlier version is literally one click in the boot manager and happens instantly, as the OS is just resetting some symlinks under the hood.
For Debian you might need a file system with snapshot support, e.g. apt-btrfs-snapshot is a package that handles it, though I never used that myself.
117 comments
[ 6.4 ms ] story [ 210 ms ] thread* https://wiki.freebsd.org/BootEnvironments
* https://klarasystems.com/articles/managing-boot-environments...
* https://docs.oracle.com/cd/E23824_01/html/821-2726/gkwcz.htm...
Of course, if Apple manages to not break their OS with updates, that's not a problem in practice. I don't have a lot of experience with macOS, so I have no idea how problem-prone Apple's updates are.
That's no longer an issue on Apple Silicon systems. There is no firmware resident in hardware; everything gets loaded on boot.
* https://support.apple.com/en-ca/guide/disk-utility/dskuf8235...
Running "tmutil listlocalsnapshots /" the new "foo" named one shows up.
Various CLI stuff that can be done:
* https://eshop.macsales.com/blog/56681-working-with-macos-sna...
* https://old.reddit.com/r/mac/comments/zymnlz/apfs_time_machi...
Also, apfs.util has a flag for restoring a snapshot (but I haven't been able to figure out the syntax).
The worst thing was Intel’s heartbleed issues that killed the CPU, it was around the 8th gen i7 if my memory is correct.
(Seems like it got better in Big Sur, and haven't had a single one in Monterey)
This generally shouldn't be happening in minor updates, and even if it does, there are ways of designing apps to handle this. I've done it before. (And in fact, software developers themselves often roll back versions of their software while developing and testing, which you should know.) As the author indicates, rolling back is possible, albeit overly difficult, so the possibility must be accounted for in any case.
Major OS updates are different story. The key there is to avoid being an early adopter. ;-)
Up until recently they did support rollbacks from a bad update, so I'm not sure the above holds as a reason. They used to, and if it's harder now then that is part of a deliberate design decision.
Not sure if this was fixed in the last 3 years because I moved away from Mac management. But it was a problem unless you used something like jamf which roll their own mechanism.
Intune only supports what Apple offers so they're was no options there
I think there's a lot of room for improvement in this area but I don't see a lot of interest on the part of implementers or vendors. I think it's probably the usual incentives problem with enterprise software, which is tough.
In short, the author's argument is:
- no method to roll back firmware updates
- the sealed system volume is huge and rollback capability would require keeping the whole old ~10GB around, which is a lot of wasted space
On a PC laptop it often is. Even on many all in ones now.
Compare like for like for goodness sake
[0]: https://www.google.com/chromebook/discover/pdp-samsung-galax...
32GB eMMC, not enough to do a feature update. The Celeron and 4GB RAM were not ideal either.
I think it was a PC. The low end of PCs is amazingly awful.
Don’t know if you’ve looked at that segment but, maybe it’s not what you loop for, but it started not long after Apple did (HP & Dell’s MacBook Pro design-ripoff series especially)
Also many lower end laptops have done it for a looooong time.
s/even/especially/
This used to be unheard of on PC laptops. Soldered storage in PC laptops is a recent development (past 10 years or so)
But, yes. It’s a form factor thing really, as people want smaller, thinner, lighter, laptops. Memory (and battery) removability gets sacrificed as those aren’t a priority for most customers
What difference does that make? The old version just needs to be kept on disk, not in RAM.
But even if the backup has to be stored on the on-board SSD for some reason, so what? It's temporary. That storage can be reclaimed when it's needed. MacOS is already using on-board SSD behind the scenes for Time Machine backups when you don't have an external drive attached. I see absolutely no reason why it could not do the same for a rollback image.
The real reason you can't roll back is simply that Apple doesn't want to allow it, because that would make it easier for the market to reject one of their updates.
Bingo. OpenSUSE makes it easy using BTRFS snapshots, enabled by default. Apple and Microsoft could make it easy too but empowering users like this isn't in their interest.
Right. That's why your Time Machine backups stop worki--oh wait, their integrated backup software lets you restore your disk to the state before the update. So your narrative is FUD.
(It also puts the burden on the user to actually make a backup before upgrading.)
https://coingeek.com/five-years-later-apple-removes-bitcoin-...
After the live CD years it became much easier to abandon the headaches and work with large signed images, but most OS vendors have had at least one pressure to continue with the hellscape of patch permutations and one way configuration adaptions.
This reads like Microsoft’s ‘the old new thing’ where they make up excuses for choices made by Microsoft and here for Apple.
I presume the real reason is that Apple would say instead of preparing for the update to fail, it’s better to make sure it doesn’t fail in the first place.
I'd be removing that snapshot as soon as possible after a successful update.
This wouldn't be such a wasteful idea had they used file system snapshot differential block transfer a la ZFS send. All I can guess is that they either think their customers are so rich they must all live in places with gigabit internet available, or they just don't care and everyone suffers. Either way it's a massive waste of time, resources and energy, just not theirs.
That would have been so epic. They should do that in FreeBSD. Take the macOS idea of having a separate protected partition for the base system, and combine that with the support for ZFS in FreeBSD to use ZFS snapshots to update the FreeBSD base OS, instead of having to run freebsd-update which applies a bunch of patches
I moved to FreeBSD precisely because I wanted more control than Apple would give me (which is ever less of course).
And freebsd-update works well, it can even validate files so you can see which have been manually changed. This is perfect for me, it's not a hard lock to what the provider decides but more of a suggestion.
They resisted any kind of low data mode for a long time because their “vision” was unlimited, unmetered everything.
Which is a nice vision. But not the reality for now.
IIRC that's how Windows handles update rollbacks, by doing this. Though disk space tends to be much cheaper on Windows devices, on Apple devices you pay a premium for extra internal storage.
I know what you mean by this, but isn’t this the case everywhere? Or at least on comparable systems, e.g. tablets?
On my very important Windows machine I use a full mirroring of disk before letting the OS to update. Rolling back means rewiring a boot disk.
https://www.macrumors.com/2023/07/10/apple-pulls-ios-16-5-1-...
> Unfortunately, it appears that the updates changed the Safari user agent to include an (a), leading some websites to break.
Lol...
"It might be hard" is not a good reason when previously they have done so but made a design &/or business &/or UX decision not to. I very much doubt that difficulty was the driving factor in this choice.
People keep making excuses for buying metaphorical "cars with the hood welded shut."
Glad to hear that Linux hit 3% market share.
The upgrade removed emacs, the C dev tools, and deleted any top-level directories it didn't know about / approve of, including my backup!
I have Carbon Copy Cloner and was dumb enough to think I didn't need to use it before this upgrade. Now instead of a development system, I have a glorified web terminal. What a pain in the ass.
Anyway seeing as Apple froze GPL software at the last GPL2 version, surely you’re using Homebrew or MacPorts to get a newer version of emacs anyway?
As for C compilers, I’m not sure if they’ve ever been in the base install, but AFAIK it’s “xcode-select --install” you want if you don’t want to install all of Xcode.
Nope, because I don't care about emacs' latest upgrades. I've been using the terminal version of emacs (emacs-nox) since the 80's and whatever version is installed has always worked fine.
I tried to install xcode tools yesterday. First it has a license agreement window that needs a click, but it doesn't pop-up in the foreground so I didn't even know it was there. The Install button just error beeps. Then it said it would take 59 hours to download. That's nice. I aborted it.
I restarted it today. It says it has 11 minutes to go so maybe things will go smoother this time. Still a complete pain in the ass, and nope, it didn't fix the speaker (I didn't think it would, just a long shot).
Thanks for letting me vent!
[0]: https://eclecticlight.co/2020/01/23/catalina-boot-volumes/
I'm not an OSX developer (I'm just a writer) and still trying to figure out where the change happened and why. The devs upon whom I depend are also mystified and/or not talking.
Being able to roll back updates when so many people depend on your OS seems like a good thing to me.
It would be cool for optional dist-upgrades and it would just be so phenomenal to be able to update and then revert - like we can for monitor settings.
For Debian you might need a file system with snapshot support, e.g. apt-btrfs-snapshot is a package that handles it, though I never used that myself.