How could an OS corrupt a BIOS ? BIOS is in a <strike>higher</strike> lower (see alex_duf comment) level than OS, and then, should no be modified by any program, except maybe by an update program, and only with a firmware signed by lenovo.
You can do a similar thing in any OS - delete the vast majority of your file system by running a command as root/administrator and ignoring warnings. FYI rm -rf no longer recurses upwards (. and ..) and hasn't done so for a very, very long time.
This problem is about breaking the BIOS in some way, something that you should not be able to do.
well windows can get write access to these aswell. (just not by using rm or any other tool to remove files by default).
I think this is a big difference and should probably be addressed. make /sys read-only and allow to write them differently with a interface where I need to call THIS_METHOD_CAN_BREAK_A_SYSTEM_IF_USED_INCORRECT_WRITE_SYS() command. but well, systemd guys will/won't do that.
It wasn't a systemd thing. The people who relate the tale as if it were a systemd thing or something that the systemd people did are ill-informed. Much as in the case of the headlined bug at hand, it was entirely a kernel thing. This was even stated outright at the time by the creator of that particular kernel mechanism.
This is in reference to the issue a few years ago where it was easy to wipe EFI vars via /sys, which would happen if you rm -rf / .[1] Wiping the EFI vars would brick your system.
We usually refer to the BIOS as a lower level than the OS.
It's also possible to bridge from a higher level to a lower level, typically when flashing your BIOS, chances are you're doing it from the OS (it's usually possible to do it at a lower level though).
In this case it looks like a kernel module that does exactly that has been enabled before being ready from prime time. Woups.
Why are you so enamored with giving up control of your hardware to the laptop manufacturer?
From never-fixed bugs in manufacturer's firmware to setting up things to happen on the next boot, there are innumerable reasons to write to the BIOS from a higher level.
This kernel model was written for the explicit purpose of modifying the BIOS[1]. The module writer makes the recommendation of not enabling in KConfig[2]. The question is did Ubuntu enable it by default and if so, why?
The BIOS update process is usually an afterthought held together by string, and appears to the OS as mysterious peripheral. Every now and again some manufacturer releases hardware where Linux simply trying to probe the hardware bricks the machine.
Yes, while Ubuntu has regressed, the root of the problem seems to be a Lenovo logic error where Lenovo resets the BIOS settings if the write protection bit is set to read/write.
OSes tickling the BIOS is nothing new. Vendors have been providing Windows-based flashing utilities for a long time. Even most of the special bootable flashing disks run freedos.
As pointed out several times in the headlined bug report, this is not an EFI problem. Several people reproduced this problem on systems either that were booting using the CSM or that did not have EFI firmwares in the first place.
My XPS 13 Developer edition had something similar happen to it shortly after upgrading to 17.10. However I think this was circumstance, and that the real culprit was a bad BIOS update from Dell which removed all entries from the boot list (or something to that effect).
(not the parent, but I had the same laptop with the same problem) I reinstalled Fedora. I could have probably recovered the grub bootloader manually, but I wasn't really keen on sitting there for a few hours. All my data was backed up anyway.
I had the exact same issue on 17.10 + XPS 13 + Dell firmware update. I booted in the BIOS setup and manually pointed the EFI to ubuntu's grub's EFI file. That immediately fixed the problem.
I upgraded my XPS 15 (9550) from Ubuntu 17.04 to 17.10 about a week ago and have had no issues so far. Actually, I find it a little less buggy on my XPS than 17.04 was - overall I am pretty happy with it. Of course YMMV - my specs are below
It's a bit of a pain, but quite doable on most distros, if you know your way around. I just did it on two distros for my ThinkPad 25: with openSUSE it was part of a weekend of playing around with it and setting it up, and on NixOS it took an hour or two to set up how I like.
The real problem, imo, is that all of the solutions work badly (they either work at a performance penalty (as with Bumblebee) or require a separate X server for each GPU).
This is entirely a result of NVIDIA's refusal to implement decent support. The Nouveau drivers offer proper PRIME support, which yields an experience more like that on Windows in terms of workflow and the amount of setup (virtually none) required, but they have their own performance problems. Hybrid graphics with AMD GPUs apparently works nicely with whatever drivers you like, granted that the software stack is sufficiently up to date on your system.
I didnt bother - I set it up to only use the Intel GPU b/c I wanted longer battery life. Besides I don't game on the laptop anyway. I dual boot Windows (since the license came with the box) and can use that if I really want to game.
However if you really want to, everything pxc says below is spot on...
There were people in the bug report reporting issues on the XPS 13 and many more brands than just lenovo. Try waiting 6 months until Ubuntu can sort their shit.
edit: there is a guy complaining that it screwed his XPS 13 a few messages up.
Yeah, Maybe I got lucky. If you want to totally play it safe wait until 18.04 LTS is released. They should (hopefully) have this worked out by the LTS release.
At the time systemd got a lot of serious heat and poettering got called names for something that was not systemd's fault to begin with.
> To make this very clear: we actually write to the EFI fs in systemd. Specifically, when you issue "systemctl reboot --firmware" we'll set the appropriate EFI variable, to ask for booting into the EFI firmware setup. And because we need it writable we'll mount it writable for that.
It absolutely was systemd's fault -- systemd regularly half-implements or changes default behavior in ways that "should" be okay, but aren't okay, because they don't know why the fence was built (in the Chestertonian sense). Systems programmers should know the history of their discipline.
I wrote the EFI variables filesystem. It was intended to be mounted read/write. Systemd did nothing wrong whatsoever in mounting it read/write, and mounting it read only would have broken all the existing userland that handles EFI variables.
What userland is it ? And why is it a problem for that userland to mount it itself ?
Why would you even write to efi ? Maybe at boot, i don't know, but certainly not while running ? Not all the time ? Oh i hope there isn't a single program that continuously writes to efi.
In my opinion systemd is at fault, as they aim to be the "system" and yet every once in a while prove that they don't know shit.
EDIT: A program that changes the boot order is all i can think of. (I'm not the one who knows best, on this topic)
Various tools like efibootmgr talk to efivarfs for several reasons. These settings allow things like secure boot management or various booting modes e.g. it allows you to configure PXE boots, select your next OS, or other things -- if you have UEFI fast boot directly to Linux you can safely configure your machine to boot back into UEFI next time to set options from there, etc.
Userspace utilities expect it to be mounted r/w, so changing it unilaterally to read-only will break things if it is intended as a "bug fix" -- which is annoying to ship to users if one fix causes another regression. For example, OpenRC ships it as r/o, but various tools like grub-install do not remount on their own, so they fail strangely. It could be fixed to have userspace mount it, but those things require coordination. Remounting is also a race-y condition for various tools to engage in on their own, for obvious reasons. It also does not change the fact those EFI variables are still exposed as writable once mounted r/w, so even closing the race condition, your system can still be bricked due to a bug.
The fix for this in the kernel was to block writes to non-standard UEFI options in efivarfs making them immutable, which prevents accidentally nuking them. This was the actual source of the 'bricking' in the original bug. This is a low-cost fix inside the kernel, which already has to deal with hostile hardware in a million other ways and sanitize various other things to stop all kinds of hardware from crapping itself, just like this. It is also relatively easy for distributions to backport, does not require changing mount semantics for systemd, does not break existing UEFI userspace tools (which would take time to coordinate, and will not manipulate non-standard variables anyway), and stops people from bricking themselves.
Logistically, the kernel is really the best and most low-cost place to fix such an issue for everyone, regardless of mount policies on efivarfs in the init system -- in a way that is actually safe for everyone. It would have taken the same time, or more time, to fix it badly (e.g. break userspace tools and require userspace remounts) as fixing it this way. You could level better complaints at UEFI or efivarfs or systemd in general than this one, all things considered...
> (I'm not the one who knows best, on this topic)
I like how people somehow breathlessly say "systemd developers prove they don't know shit, of course its their fault!!!" and then immediately follow it up with a bunch of fairly basic questions and just admit "oh by the way i don't know what i'm talking about, can anyone tell me otherwise???"
systemd annoys me sometimes and I don't like some of its design decisions, but by far its worst "feature" is it causes people who are very unsure of what they are talking about to suddenly become experts about it.
>I like how people somehow breathlessly say "systemd developers prove they don't know shit, of course its their fault!!!" and then immediately follow it up with a bunch of fairly basic questions and just admit "oh by the way i don't know what i'm talking about, can anyone tell me otherwise???"
I like how stating to a bunch of people who don't know anything about you that you are not all knowledgeable, and asking questions to better inform yourself and to put things into perspective (for everyone involved) is looked at as a weakness/sign of ignorance.
For the record i know a lot about how computers, and linux, work. From the basic transistors and gates, to high level programming constructs. It's just that this particular topic does not interest me, nor will knowing it help me in any way, shape, or form in my life.
As for "systemd developers prove they don't know shit, of course its their fault!!!". It is. Shit has happened and their response was "Not my problem"... numerous times. If the people that declared themselves "the grand arch-mages of.. everything", and are forcing their "arch-mag-iness" as "the standard of all and forever", then they are responsible for the problems of "everything". As they say "with great power comes great force times distance over time", or something like that.
> What userland is it ? And why is it a problem for that userland to mount it itself ?
At the very least, efibootmgr. The problem with userland mounting it itself is that it has no code to mount it itself because even before systemd everybody mounted the filesystem read/write.
> Why would you even write to efi ? Maybe at boot, i don't know, but certainly not while running ?
A whole bunch of reasons. Maybe you want something that's tied to the hardware without relying on the OS (eg, tpmtotp). Maybe you want to be able to differentiate between a clean and unclean system shutdown. Maybe you want to indicate that the system should install a firmware update next time it's in boot services.
> In my opinion systemd is at fault, as they aim to be the "system" and yet every once in a while prove that they don't know shit.
At the time this issue was raised, nobody knew shit. We had no idea that removing certain variables might prevent a machine from booting. If we had done, I'd have written the filesystem differently.
> I'm not the one who knows best, on this topic
I wrote the code in question. I am one of the people who does know best on this topic. Systemd did nothing wrong.
Couldn't the filesystem be mounted as read-only during normal operation, and only remounted temporarily as rw when necessary? Tools that write to this filesystem tend to have the necessary privileges to do so.
Leaving efi writable like this by default strikes me as a very poor and unsubstantiated design decision.
> Couldn't the filesystem be mounted as read-only during normal operation, and only remounted temporarily as rw when necessary?
Yes, except that none of the tools in question have any support for doing that and so mounting the filesystem read-only rather than read/write would break them. And congratulations, now you've still got a window where something can still break everything while the filesystem is mounted read/write (what happens if the tool crashes during that phase?).
> Leaving efi writable like this by default strikes me as a very poor and unsubstantiated design decision.
Yes with hindsight that's entirely accurate and it was also my decision and nothing to do with systemd.
>Yes, except that none of the tools in question have any support for doing that and so mounting the filesystem read-only rather than read/write would break them.
All 4 of the tools (that can even be wrapped in a shell script until they are, as systemd devs say, "fixed").
>And congratulations, now you've still got a window where something can still break everything while the filesystem is mounted read/write (what happens if the tool crashes during that phase?).
Better to open a window in the morning then to tear down the whole wall forever.
> now you've still got a window where something can still break everything while the filesystem is mounted read/write
I agree, but I'm also sure you would agree that a small window were things can go horribly wrong is much more preferable than than a perpetually open one (sudo and friends stand testament to this).
Please bear in mind that my comments are in no way an attack on your person or one directed at systemd. What I want is to find a workable solution or, at the very least, a compromise to prevent this type of issues from occurring. I strongly believe that protecting hardware is a core responsibility of any operating system.
Would it be reasonable for me to ask you to coordinate with all the major tool developers to gradually phasing out this behavior? As in both the filesystem and tooling continue to support both current semantics and a newer version until the older semantics can be safely disabled. This is how support tends to get phased out from the kernel for obsolete hardware.
Issues of this type have reared their heads many times already and I expect they will continue to occur if the underlying reason is not addressed. As the person who has the ability to do something about it, I urge you to reconsider.
> I agree, but I'm also sure you would agree that a small window were things can go horribly wrong is much more preferable than than a perpetually open one (sudo and friends stand testament to this).
What's even more preferable is a situation where the kernel doesn't let you do the damage in the first place, which is what we have now.
So read/write efi is needed to set what happens when the computer boots and to see what state it was/is (relating to boot stuff).
To see the state you don't need write permission, obviously.
When things are set, they are set by a few very specialized tools.
Those are the facts everybody can agree on, right ? Can we also agree that mounting and unmounting the efi filesystem is really easy ? I assume it can be mounted on multiple points.
So is there a problem in re/mounting it read/write only when needed ?
>At the time this issue was raised, nobody knew shit. We had no idea that removing certain variables might prevent a machine from booting. If we had done, I'd have written the filesystem differently.
That's the difference between systemd devs and people capable of critical thinking. You'r a good programmer, if you ask me. (edit: just to be clear, i don't see you as a systemd dev; even if you are (idk), you are not the one making high level design decisions (of which this is, kind of, one))
Not with the interface that exists now, no. And we wouldn't really want to leave it up to userspace anyway - there are various firmware-based issues that we want to work around in kernel, and relying on userspace to adopt policy there is unlikely to end well.
Might be a coincidence, but I haven't been able to get into my Toshiba's BIOS since moving to 17.10. Every time I try and enter the BIOS config menu, it reboots.
For what it's worth, I haven't had this problem with Arch Linux on my ThinkPad T470s or P50 and I did a fresh install on the P50 this week. Arch has really nice ThinkPad support and documentation, too.
17.10 was the absolute most buggy software that I have installed from Canonical. I tried it on Dell hardware (+ Apple Hardware, + A VMware VM on X64 + VirtualBox on OSX, etc) that does just fine with Fedora, Arch, Gentoo, and it was constantly prompting me to send error reports, killing processes, etc. I just flipped from using the "latest" vs the LTS versions at home because of this version, specifically.
Not the worst Ubuntu version, but I had a lot of issues with the default protocol being Wayland [1].
This broke a few programs I use, [2] mainly the screenshot application "Shutter". Most users don't realise it became the default, so their apps just stopped working... bad decision in my opinion.
The main problem is Linux. Linux cannot be tested, it designates hundreds of variable setups of distributions, kernels and tools. It's not possible to test anything against that.
They shouldn’t support RedHat, Ubuntu and Windows. They should provide a solid, compliant UEFI implementation and then Windows, RedHat Linux and Ubuntu Linux will support them.
71 comments
[ 4.7 ms ] story [ 186 ms ] threadI would look at the wonders of UEFI.
https://superuser.com/questions/742334/is-there-a-scenario-w...
https://archive.fo/g9DKx https://screenshots.firefox.com/QiHbjMEHAIrb9Wt3/thenextweb....
Made the news some years back I think.
This problem is about breaking the BIOS in some way, something that you should not be able to do.
[1] https://github.com/systemd/systemd/issues/2402
To learn the real story, go and read https://news.ycombinator.com/item?id=11152880 where you will find Matthew Garrett xyrself participating in the discussion.
[1]: https://www.phoronix.com/scan.php?page=news_item&px=UEFI-rm-...
https://www.phoronix.com/scan.php?page=news_item&px=UEFI-rm-...
It's also possible to bridge from a higher level to a lower level, typically when flashing your BIOS, chances are you're doing it from the OS (it's usually possible to do it at a lower level though).
In this case it looks like a kernel module that does exactly that has been enabled before being ready from prime time. Woups.
The BIOS is not to be messed with by user software or the OS. It should be signed and only changed by the manufacturer tool to update it.
From never-fixed bugs in manufacturer's firmware to setting up things to happen on the next boot, there are innumerable reasons to write to the BIOS from a higher level.
:(
[1] https://github.com/torvalds/linux/blob/master/Documentation/...
[2] https://github.com/torvalds/linux/blob/8afda8b26d01ee26a60ef...
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1734147...
I personally would blame Lenovo for releasing something so brick-able.
I also learned that backup bios chips were a thing but not usually available on a notebook PC
----------------------
Dell BIOS: 1.5.1
i7-6700HQ CPU @ 2.60GHz
16GB RAM
Intel® HD Graphics 530 + NVidia
512GB SSD
The real problem, imo, is that all of the solutions work badly (they either work at a performance penalty (as with Bumblebee) or require a separate X server for each GPU).
This is entirely a result of NVIDIA's refusal to implement decent support. The Nouveau drivers offer proper PRIME support, which yields an experience more like that on Windows in terms of workflow and the amount of setup (virtually none) required, but they have their own performance problems. Hybrid graphics with AMD GPUs apparently works nicely with whatever drivers you like, granted that the software stack is sufficiently up to date on your system.
However if you really want to, everything pxc says below is spot on...
There were people in the bug report reporting issues on the XPS 13 and many more brands than just lenovo. Try waiting 6 months until Ubuntu can sort their shit.
edit: there is a guy complaining that it screwed his XPS 13 a few messages up.
At the time systemd got a lot of serious heat and poettering got called names for something that was not systemd's fault to begin with.
> To make this very clear: we actually write to the EFI fs in systemd. Specifically, when you issue "systemctl reboot --firmware" we'll set the appropriate EFI variable, to ask for booting into the EFI firmware setup. And because we need it writable we'll mount it writable for that.
Why would you even write to efi ? Maybe at boot, i don't know, but certainly not while running ? Not all the time ? Oh i hope there isn't a single program that continuously writes to efi.
In my opinion systemd is at fault, as they aim to be the "system" and yet every once in a while prove that they don't know shit.
EDIT: A program that changes the boot order is all i can think of. (I'm not the one who knows best, on this topic)
Userspace utilities expect it to be mounted r/w, so changing it unilaterally to read-only will break things if it is intended as a "bug fix" -- which is annoying to ship to users if one fix causes another regression. For example, OpenRC ships it as r/o, but various tools like grub-install do not remount on their own, so they fail strangely. It could be fixed to have userspace mount it, but those things require coordination. Remounting is also a race-y condition for various tools to engage in on their own, for obvious reasons. It also does not change the fact those EFI variables are still exposed as writable once mounted r/w, so even closing the race condition, your system can still be bricked due to a bug.
The fix for this in the kernel was to block writes to non-standard UEFI options in efivarfs making them immutable, which prevents accidentally nuking them. This was the actual source of the 'bricking' in the original bug. This is a low-cost fix inside the kernel, which already has to deal with hostile hardware in a million other ways and sanitize various other things to stop all kinds of hardware from crapping itself, just like this. It is also relatively easy for distributions to backport, does not require changing mount semantics for systemd, does not break existing UEFI userspace tools (which would take time to coordinate, and will not manipulate non-standard variables anyway), and stops people from bricking themselves.
Logistically, the kernel is really the best and most low-cost place to fix such an issue for everyone, regardless of mount policies on efivarfs in the init system -- in a way that is actually safe for everyone. It would have taken the same time, or more time, to fix it badly (e.g. break userspace tools and require userspace remounts) as fixing it this way. You could level better complaints at UEFI or efivarfs or systemd in general than this one, all things considered...
> (I'm not the one who knows best, on this topic)
I like how people somehow breathlessly say "systemd developers prove they don't know shit, of course its their fault!!!" and then immediately follow it up with a bunch of fairly basic questions and just admit "oh by the way i don't know what i'm talking about, can anyone tell me otherwise???"
systemd annoys me sometimes and I don't like some of its design decisions, but by far its worst "feature" is it causes people who are very unsure of what they are talking about to suddenly become experts about it.
>I like how people somehow breathlessly say "systemd developers prove they don't know shit, of course its their fault!!!" and then immediately follow it up with a bunch of fairly basic questions and just admit "oh by the way i don't know what i'm talking about, can anyone tell me otherwise???"
I like how stating to a bunch of people who don't know anything about you that you are not all knowledgeable, and asking questions to better inform yourself and to put things into perspective (for everyone involved) is looked at as a weakness/sign of ignorance.
For the record i know a lot about how computers, and linux, work. From the basic transistors and gates, to high level programming constructs. It's just that this particular topic does not interest me, nor will knowing it help me in any way, shape, or form in my life.
As for "systemd developers prove they don't know shit, of course its their fault!!!". It is. Shit has happened and their response was "Not my problem"... numerous times. If the people that declared themselves "the grand arch-mages of.. everything", and are forcing their "arch-mag-iness" as "the standard of all and forever", then they are responsible for the problems of "everything". As they say "with great power comes great force times distance over time", or something like that.
At the very least, efibootmgr. The problem with userland mounting it itself is that it has no code to mount it itself because even before systemd everybody mounted the filesystem read/write.
> Why would you even write to efi ? Maybe at boot, i don't know, but certainly not while running ?
A whole bunch of reasons. Maybe you want something that's tied to the hardware without relying on the OS (eg, tpmtotp). Maybe you want to be able to differentiate between a clean and unclean system shutdown. Maybe you want to indicate that the system should install a firmware update next time it's in boot services.
> In my opinion systemd is at fault, as they aim to be the "system" and yet every once in a while prove that they don't know shit.
At the time this issue was raised, nobody knew shit. We had no idea that removing certain variables might prevent a machine from booting. If we had done, I'd have written the filesystem differently.
> I'm not the one who knows best, on this topic
I wrote the code in question. I am one of the people who does know best on this topic. Systemd did nothing wrong.
Leaving efi writable like this by default strikes me as a very poor and unsubstantiated design decision.
Yes, except that none of the tools in question have any support for doing that and so mounting the filesystem read-only rather than read/write would break them. And congratulations, now you've still got a window where something can still break everything while the filesystem is mounted read/write (what happens if the tool crashes during that phase?).
> Leaving efi writable like this by default strikes me as a very poor and unsubstantiated design decision.
Yes with hindsight that's entirely accurate and it was also my decision and nothing to do with systemd.
All 4 of the tools (that can even be wrapped in a shell script until they are, as systemd devs say, "fixed").
>And congratulations, now you've still got a window where something can still break everything while the filesystem is mounted read/write (what happens if the tool crashes during that phase?).
Better to open a window in the morning then to tear down the whole wall forever.
I agree, but I'm also sure you would agree that a small window were things can go horribly wrong is much more preferable than than a perpetually open one (sudo and friends stand testament to this).
Please bear in mind that my comments are in no way an attack on your person or one directed at systemd. What I want is to find a workable solution or, at the very least, a compromise to prevent this type of issues from occurring. I strongly believe that protecting hardware is a core responsibility of any operating system.
Would it be reasonable for me to ask you to coordinate with all the major tool developers to gradually phasing out this behavior? As in both the filesystem and tooling continue to support both current semantics and a newer version until the older semantics can be safely disabled. This is how support tends to get phased out from the kernel for obsolete hardware.
Issues of this type have reared their heads many times already and I expect they will continue to occur if the underlying reason is not addressed. As the person who has the ability to do something about it, I urge you to reconsider.
What's even more preferable is a situation where the kernel doesn't let you do the damage in the first place, which is what we have now.
To see the state you don't need write permission, obviously.
When things are set, they are set by a few very specialized tools.
Those are the facts everybody can agree on, right ? Can we also agree that mounting and unmounting the efi filesystem is really easy ? I assume it can be mounted on multiple points.
So is there a problem in re/mounting it read/write only when needed ?
>At the time this issue was raised, nobody knew shit. We had no idea that removing certain variables might prevent a machine from booting. If we had done, I'd have written the filesystem differently.
That's the difference between systemd devs and people capable of critical thinking. You'r a good programmer, if you ask me. (edit: just to be clear, i don't see you as a systemd dev; even if you are (idk), you are not the one making high level design decisions (of which this is, kind of, one))
In a race-free way? No. In a way that fails safe? No.
This broke a few programs I use, [2] mainly the screenshot application "Shutter". Most users don't realise it became the default, so their apps just stopped working... bad decision in my opinion.
[1] http://www.omgubuntu.co.uk/2017/08/ubuntu-confirm-wayland-de...
[2] https://askubuntu.com/questions/971273/screenshot-feature-of...
Owner of a $3k Lenovo P50 here that was "Linux compatible" but a stake xeon that had been useless on Linux until the latest Fedora.
The main problem is Windows because it allows manufacturers to get away with the 'it runs Windows so it's good enough to release' mindset.
Lenovo supports redhat and ubuntu, that's already a lot. https://support.lenovo.com/gb/en/solutions/pd031426