As the title doesn't indicate, this is a pretty big deal. Full host compromise from a guest that always works and doesn't leave any traces. Amazon is claiming it doesn't affect them[0], but they don't give any details as to why.
Cloud providers typically have early access to security fixes and would be patched by now. Amazon has the ability to hot-patch Xen, a feature which is now being developed for general use.
I just saw a bunch of our SoftLayer VMs get rebooted (with ahead notification) and they seem to use Xen. They probably got notification too.
Looks like my others (Digital Ocean, Vultr) use KVM (according to virt-what).
Did anyone see Amazon do reboots? I wonder if Amazon has some method of hot-patching kernel and Xen code under the hood? There are such things, and that would be some interesting secret sauce.
> Our team has been working around-the-clock to find ways to minimize the impact for those requiring a reboot. We're happy to share that we'll now be able to live-update the vast majority of our older hardware for this Xen Security Advisory.
There's been a few subsequent articles mentioning Amazon's ability to hot-patch Xen without a reboot. They didn't perform a reboot for this vulnerability.
Out of the 1,000+ instances I had running in AWS during that last major reboot -- there were only a handful of hosts affected. Sadly, some of them were the very primary core DBs -- but in general, it was still only a handful.
Luckily, many people also converted to HVM over PV already. IIRC it was largely PV instances in EC2 classic... but I can't be sure.
Regardless of the virtualization technology, you need to keep on top of disclosures. Any public provider running xen should be on the disclosure list by now http://www.xenproject.org/security-policy.html
In that case Amazon's wording would be pretty disingenuous - "AWS customers' data and instances are not affected by these issues" when the reality is more like "as far as we know nobody exploited this before we patched"?
Good point. In theory, they could have had other mitigations in place for this class of vulnerability, which they chose not to share with the GPL Xen community.
Breaches are a one-way state change, like death. If customer data WAS affected, then it IS affected, until customers take action such as disclosure, resetting passwords/keys, etc. If customer data was affected, AWS would need to advise customers to take remedial steps. Since they have not issued such advice, is it safe to assume that they had mitigation in place for this attack?
It could also be the case that they have monitoring systems on the host machines that would have reliably detected this issue being exploited, even before this specific vulnerability was discovered (or perhaps better to say, given the age of the bug, before being discovered by someone that disclosed it to Xen).
It depends on the version of Xen they are running. XSA-148 only affects Xen 3.4 and newer. If AWS is running an older version of Xen, they may never have been affected.
without any serious infrastructure you can also hot patch single instructions where the bytes you want to change fall in an 8 byte alignment. though, probably this is not useful for this bug but I know other cloud providers do this for security patches.
Same for Rackspace, IBM/Softlayer, and everyone else with a large cloud running Xen.
Linode was actually my failover target for some of those clouds to weather the reboots. They offer both Xen and KVM based instances. If you're still running Xen, you can live migrate your instance to KVM and make all your new Linodes default to KVM via a toggle in your account settings.
Rackspace has been patching their Xen stuff for a little over a week now.
It's nice having a multi-region / multi-provider setup otherwise the typical Rackspace reboot window (much like UPS deliveries or the cable guy) would've been a pain in the ass.
We deploy to a single region. This maintenance was a major pain in the ass. We got a notice 72 hours ahead of time (only 1 business day ahead of time) that around 60% of all of our hosts failed live migration and would be rebooted.
@gibsonje - similar experience here. Except Rackspace gave notice about a week ahead. The day maintenance was scheduled for it was postponed but without notice. Then the reboots happened across 3 days as opposed to 1 day. Glad they are patching but that's not really fanatical support.
The existence of such a catastrophic bug shows that Xen is unfortunately not suitable for use as an hypervisor in a secure system and needs to be replaced.
We really need a properly written open source hypervisor: entities using secure hypervisors commercially like Amazon AWS should fund the development of one.
It's inferior in basically every way to HVM. CPU performance is worse due to missing protection rings on x86_64, PV drivers for disk and network bring HVM performance to PV for those, SR-IOV for networking makes networking faster, other hardware extension make memory access faster...
THere's very very very very very few workloads that perform better on PV than HVM.
PV at least at one time, was more efficient than emulating straight hardware. And PV was the only way you could run visualization on Intel Atom since the first batches did not have the hardware bits needed for KVM.
I could go on and on and on... but the point is that kvm (and xen HVM mode) may have performance advantages in some situations, but xen pv mode has consistently had fewer security vulnerabilities, especially if you use something like pv-grub to load untrusted guest kernels (rather than loading those kernels directly in the dom0)
The biggest problem is the qemu drivers; Honestly, I don't know enough about kvm to know if it was possible, but if you could completely remove the device emulation and force the guest to only use paravirtualized drivers, your security under KVM (or Xen HVM) would be much better, mostly because you'd vastly reduce the amount of hypervisor code the guest interacts with.
But the point here is that all systems have problems, and the xen pv mode has had fewer security holes found it it than most other hypervisors, mostly because there's a lot less code that the guest interfaces with.
>may have performance advantages in some situations
s/may/certainly
s/some/almost all
It's not a small gap, either. You are basically doubling the number of context switches required for system calls with PV, due to AMD removing CPU protection rings from x86_64, forcing this separation to have to happen in software. You lose out on EPT, so your page table performance suffers in almost all workloads. You cannot take advantage of SR-IOV for NICs or nvme.
PVH might be an answer someday, but for now, there is a pretty massive performance loss. Context switches are important. Memory latency is important. (Nearly) direct access to the hardware is important.
I'm not saying you are wrong about performance (other than to point out that actually using the qemu devices is even slower.) Still, if you go through the xsa list, if I was running HVM, I'd have had to reboot pretty often. It's a huge firedrill every time the security list sends something out, for those of us who use (normally lower-stress) local disk. Twice in one year is bad enough.
Well, yeah, software emulated hardware is slow, but no one should be using emulated PCI devices on anything. There are PV or better yet, SR-IOV drivers that can be used for those that are going to be the closest to bare metal performance you'll get in a VM.
As a kinda-sort-maintainer of a bunch of the low-level Linux code that is all tangled up with the Xen PV stuff, I would be shocked if there isn't a PV-specific security bug or two in there.
x86 is a nasty enough architecture on its own. x86 plus hypercalls is even nastier and, even more importantly, x86 plus hypercalls is differently nasty.
> As a kinda-sort-maintainer of a bunch of the low-level Linux code that is all tangled up with the Xen PV stuff, I would be shocked if there isn't a PV-specific security bug or two in there.
The current bug is pv-specific. there was another one that effected me early this year[1]... I'm not saying it's bug free or anything, I'm just saying that xen PV has had fewer exploits over the last year (and over the last decade) than HVM mode or KVM.
[1]xsa123 if I remember right; my blog archive is all messed up at the moment. but, for example, xsa-108, which was a big deal, was a non-issue for me because all my stuff is paravirtualized.
Did you know that AWS uses their own version of Xen?
Xen was actually written with security in mind, they push as much of the features and driver code out of the hypervisor and onto Dom0 as possible to minimize the attack surface in the hypervisor itself.
Is it possible to tell whether you're running under Xen?
Because that should be considered a security hole. Knowing the attack surface is there is, in this context, a security bug in and of itself, because it implies there is already an attack, or an attack is forthcoming.
Unless it's possible to run an unmodified Xen as a guest under Xen, the project is incomplete.
For PV it's quite easy. 'uname -a' is currently straight up telling me i'm on the el5xen PV kernel.
If the kernel itself doesn't just straight up say it's a kernel designed to work under Xen the drivers used to work with the virtualization will give it away. Right down to the version if you care to dig deep enough and compare builds.
I no longer believe it's possible to write code properly in C (not that it's easy in other languages). Which for a hypervisor probably means waiting for Rust.
> The existence of such a catastrophic bug shows that Xen is unfortunately not suitable for use as an hypervisor in a secure system and needs to be replaced.
What makes this bug a huge deal for me, and probably why it's making the news is that it effects fully paravirtualized guests. There have been many, many bugs for HVM guests (and for KVM guests) since the last PV security hole. (March, was it? xsa-123, I think?)
Does this mean it's a good idea to reset all keys/passwords for any service that was using Xen?
I don't know much about Xen, so perhaps someone else can chime in on this. This exploit says it's for PV, and not for HVM.
But Xen can apparently run both at the same time. Does this mean even if you were using HVM, and Xen had other guests using PV, you were still exploitable?
Per AWS, they aren't affected (not shocking, they run very custom Xen), and even so, the info you upload to tarsnap is already encrypted client-side, that being the whole point.
Right, they are not currently affected, because they got advance notice. But it's very likely that Alibaba sat on the vuln for a long time, and Chinese intelligence knew about it.
It's prudent to consider all data on ec2 servers compromised and act accordingly.
so my speculation (and this is pure speculation) is that they're still running something based on the older rhel5-xen on their paravirt guests, on the general "if it's not broken, don't fuck with it" model. I think that the RHEL5 hypervisor is old enough to not be vulnerable to this problem.
Yeah. AHCI is AHCI.... and if you did have drivers for newer RAID cards and the like, those can be backported with a reasonable number of hours of the skill level amazon can reasonably hire.
| The code to validate level 2 page table entries is bypassed when
| certain conditions are satisfied. This means that a PV guest can
| create writeable mappings using super page mappings.
|
| Such writeable mappings can violate Xen intended invariants for pages
| which Xen is supposed to keep read-only.
"The above is a political way of stating the bug is a very critical one. Probably the worst we have seen affecting the Xen hypervisor, ever. Sadly.
Admittedly this is subtle bug, because there is no buggy code that could be spotted immediately ... On the other hand, it is really shocking that such a bug has been lurking in the core of the hypervisor for so many years. In our
opinion the Xen project should rethink their coding guidelines and try to come up with practices and perhaps additional mechanisms that would not let similar flaws to plague the hypervisor ever again (assert-like
mechanisms perhaps?). Otherwise the whole project makes no sense, at least to those who would like to use Xen for security-sensitive work.
Specifically, it worries us that, in the last 7 years (i.e. all the time when the bug was sitting there having a good time) so much engineering and development effort has been put into adding all sorts of new features and whatnots, yet no serious effort to improve Xen security effectively. Because there have been, of course, many more security bugs found in Xen over the last years (as the numbering of this XSA suggests)... the bugs in Xen are being found regularly, and
this is no good news. For a type-1 hypervisor of the age and maturity of Xen, this simply should not be happening. If it does, it suggests the development process is not prioritizing security."
You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes.
The Qubes commentary on this vulnerability is unintentionally amusing too, it basically boils down to them complaining that the Xen developers failed to spot it for seven years even though there's no obvious way to spot it.
Yes, the problem is that this terrible code directly copies the untrusted "nl2e" variable from the VM into the extremely critical hardware page table, only doing a broken unclearly written check.
Instead, "nl2e" should have a data type preventing such a direct copy, and only allowing to test single bits.
The code should then be written to copy bits one by one (for bits where it is appropriate), with a comment for each bit stating why it is safe to copy them.
The fact that this is not the case means that none of the other code in Xen can really be trusted to be bug-free, and there is probably no way to fix that without starting over or doing an equivalent amount of rewriting work.
... I'll say a bit more ... I think it's sad that it's not practical anymore to write really high quality code. And most computer security researchers aren't interested in that anymore either, because it's not possible to force all other developers to write high quality code (and if you do you're called "mean" "alienating" etc.) So all they/we try to do is find new layers to contain all the bad code that has been and will be written.
Can we really be surprised when those layers, particularly if they are popular because they came out first and have lots of features, are also low-quality code? Is there anywhere a foot can be put down?
Also, openbsd is another good example of a project with code with a very very low bug density.
At $job-2 doing agricultural telemetry, our firmware engineer found that one ISP's satellite was randomly dropping the last character in routed traffic. Took a little while for them to believe us, and that satellite had been up there for 30 years...
But it would be especially important to have SOMETHING that can run multiple applications while being 100% sure that it is enforcing security between them.
So far everything has been a total disaster, with browsers getting completely owned every year at Pwn2Own, conventional kernels being effectively expected to be locally exploitable at all times (which means all mobile sandboxes are broken as well), and hypervisors where Xen seems to be the most secure of the ones that are mature, and is still terrible.
No. This is a subtle vulnerability that involves the flags in the x86 page table not matching the hypervisor's view of them - not a mere buffer overflow. Ordinary static analysis couldn't have fixed this. Safe languages couldn't have fixed this. Even a complete formal proof would have missed this without a good model.
Disagree. If those flags were properly typed then whoever added superpages would have had to make a decision about whether that was something guests should be allowed to set. Sure, they could still have made the wrong choice, but having the mask for which bits are allowed defined separately makes it much easier for a programmer to simply forget.
Heck, even without a type system, the problem is that the check is backwards. There shouldn't be a mask of flags that the guest isn't allowed to set, the flags should have been &ed with a mask that says which flags they are allowed to set, that way any new flag would have been disallowed by default.
The mask of flags that the guest isn't allowed to set is created by taking a whitelist of flags that it is allowed to set and inverting it, so that every flag that's not explicitly allowed is denied. They're flags that the guest was intentionally granted the ability to change. The problem is that Xen subtly mishandled the consequences of a guest changing them in a way that wouldn't affect normal guests. I haven't seen a solution to this that doesn't, in the end, basically boil down to not making that mistake in the first place.
This is not the first time they rant about this sort of things. Of course if they can't stand the code quality of Xen they are always welcome to switch to KVM, virtualbox, bhyve or whatever open source hypervisor they think has the best code quality and security practice.
I know it is within their right to write things as they please. But seriously, ranting like this is not very constructive and doesn't move things forward. If Qubes thinks the practices in Xen community are bad, why don't they start a conversation on Xen development mailing list?
86 comments
[ 2.7 ms ] story [ 142 ms ] thread[0]: https://aws.amazon.com/security/security-bulletins/XSAsecuri...
This guest escape vulnerability (http://xenbits.xen.org/xsa/advisory-148.html) has been present since 2008 and was found by Alibaba, which recently joined the Xen advisory board, http://www.xenproject.org/about/in-the-news/192-xen-project-...
Looks like my others (Digital Ocean, Vultr) use KVM (according to virt-what).
Did anyone see Amazon do reboots? I wonder if Amazon has some method of hot-patching kernel and Xen code under the hood? There are such things, and that would be some interesting secret sauce.
> Our team has been working around-the-clock to find ways to minimize the impact for those requiring a reboot. We're happy to share that we'll now be able to live-update the vast majority of our older hardware for this Xen Security Advisory.
https://aws.amazon.com/premiumsupport/maintenance-2015-03/
There's been a few subsequent articles mentioning Amazon's ability to hot-patch Xen without a reboot. They didn't perform a reboot for this vulnerability.
Luckily, many people also converted to HVM over PV already. IIRC it was largely PV instances in EC2 classic... but I can't be sure.
Regardless of the virtualization technology, you need to keep on top of disclosures. Any public provider running xen should be on the disclosure list by now http://www.xenproject.org/security-policy.html
Linode was actually my failover target for some of those clouds to weather the reboots. They offer both Xen and KVM based instances. If you're still running Xen, you can live migrate your instance to KVM and make all your new Linodes default to KVM via a toggle in your account settings.
It's nice having a multi-region / multi-provider setup otherwise the typical Rackspace reboot window (much like UPS deliveries or the cable guy) would've been a pain in the ass.
We really need a properly written open source hypervisor: entities using secure hypervisors commercially like Amazon AWS should fund the development of one.
A PV VM would use hypercalls instead of regular system calls, would have more context switches due to the lack of protection rings, etc.
It's inferior in basically every way to HVM. CPU performance is worse due to missing protection rings on x86_64, PV drivers for disk and network bring HVM performance to PV for those, SR-IOV for networking makes networking faster, other hardware extension make memory access faster...
THere's very very very very very few workloads that perform better on PV than HVM.
https://rhn.redhat.com/errata/RHSA-2015-1896.html
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3456
I could go on and on and on... but the point is that kvm (and xen HVM mode) may have performance advantages in some situations, but xen pv mode has consistently had fewer security vulnerabilities, especially if you use something like pv-grub to load untrusted guest kernels (rather than loading those kernels directly in the dom0)
The biggest problem is the qemu drivers; Honestly, I don't know enough about kvm to know if it was possible, but if you could completely remove the device emulation and force the guest to only use paravirtualized drivers, your security under KVM (or Xen HVM) would be much better, mostly because you'd vastly reduce the amount of hypervisor code the guest interacts with.
But the point here is that all systems have problems, and the xen pv mode has had fewer security holes found it it than most other hypervisors, mostly because there's a lot less code that the guest interfaces with.
s/may/certainly s/some/almost all
It's not a small gap, either. You are basically doubling the number of context switches required for system calls with PV, due to AMD removing CPU protection rings from x86_64, forcing this separation to have to happen in software. You lose out on EPT, so your page table performance suffers in almost all workloads. You cannot take advantage of SR-IOV for NICs or nvme.
PVH might be an answer someday, but for now, there is a pretty massive performance loss. Context switches are important. Memory latency is important. (Nearly) direct access to the hardware is important.
x86 is a nasty enough architecture on its own. x86 plus hypercalls is even nastier and, even more importantly, x86 plus hypercalls is differently nasty.
The current bug is pv-specific. there was another one that effected me early this year[1]... I'm not saying it's bug free or anything, I'm just saying that xen PV has had fewer exploits over the last year (and over the last decade) than HVM mode or KVM.
[1]xsa123 if I remember right; my blog archive is all messed up at the moment. but, for example, xsa-108, which was a big deal, was a non-issue for me because all my stuff is paravirtualized.
Note, the list I'm going off of is here:
http://xenbits.xen.org/xsa/
Xen was actually written with security in mind, they push as much of the features and driver code out of the hypervisor and onto Dom0 as possible to minimize the attack surface in the hypervisor itself.
Source - The Definitive Guide To Xen - 2007.
Because that should be considered a security hole. Knowing the attack surface is there is, in this context, a security bug in and of itself, because it implies there is already an attack, or an attack is forthcoming.
Unless it's possible to run an unmodified Xen as a guest under Xen, the project is incomplete.
If the kernel itself doesn't just straight up say it's a kernel designed to work under Xen the drivers used to work with the virtualization will give it away. Right down to the version if you care to dig deep enough and compare builds.
http://www.brendangregg.com/blog/2014-05-09/xen-feature-dete...
What makes this bug a huge deal for me, and probably why it's making the news is that it effects fully paravirtualized guests. There have been many, many bugs for HVM guests (and for KVM guests) since the last PV security hole. (March, was it? xsa-123, I think?)
I don't know much about Xen, so perhaps someone else can chime in on this. This exploit says it's for PV, and not for HVM.
But Xen can apparently run both at the same time. Does this mean even if you were using HVM, and Xen had other guests using PV, you were still exploitable?
It's prudent to consider all data on ec2 servers compromised and act accordingly.
"The above is a political way of stating the bug is a very critical one. Probably the worst we have seen affecting the Xen hypervisor, ever. Sadly.
Admittedly this is subtle bug, because there is no buggy code that could be spotted immediately ... On the other hand, it is really shocking that such a bug has been lurking in the core of the hypervisor for so many years. In our opinion the Xen project should rethink their coding guidelines and try to come up with practices and perhaps additional mechanisms that would not let similar flaws to plague the hypervisor ever again (assert-like mechanisms perhaps?). Otherwise the whole project makes no sense, at least to those who would like to use Xen for security-sensitive work.
Specifically, it worries us that, in the last 7 years (i.e. all the time when the bug was sitting there having a good time) so much engineering and development effort has been put into adding all sorts of new features and whatnots, yet no serious effort to improve Xen security effectively. Because there have been, of course, many more security bugs found in Xen over the last years (as the numbering of this XSA suggests)... the bugs in Xen are being found regularly, and this is no good news. For a type-1 hypervisor of the age and maturity of Xen, this simply should not be happening. If it does, it suggests the development process is not prioritizing security."
You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes.
Instead, "nl2e" should have a data type preventing such a direct copy, and only allowing to test single bits.
The code should then be written to copy bits one by one (for bits where it is appropriate), with a comment for each bit stating why it is safe to copy them.
The fact that this is not the case means that none of the other code in Xen can really be trusted to be bug-free, and there is probably no way to fix that without starting over or doing an equivalent amount of rewriting work.
... I'll say a bit more ... I think it's sad that it's not practical anymore to write really high quality code. And most computer security researchers aren't interested in that anymore either, because it's not possible to force all other developers to write high quality code (and if you do you're called "mean" "alienating" etc.) So all they/we try to do is find new layers to contain all the bad code that has been and will be written.
Can we really be surprised when those layers, particularly if they are popular because they came out first and have lots of features, are also low-quality code? Is there anywhere a foot can be put down?
Also, openbsd is another good example of a project with code with a very very low bug density.
Not the space shuttle, but if you think things that go to space (and land on the goddamned Moon) are free of bugs then you are very sorely mistaken.
https://www.netjeff.com/humor/item.cgi?file=ApolloComputer
http://klabs.org/DEI/Processor/shuttle/
[1] http://www.fastcompany.com/28121/they-write-right-stuff/
But it would be especially important to have SOMETHING that can run multiple applications while being 100% sure that it is enforcing security between them.
So far everything has been a total disaster, with browsers getting completely owned every year at Pwn2Own, conventional kernels being effectively expected to be locally exploitable at all times (which means all mobile sandboxes are broken as well), and hypervisors where Xen seems to be the most secure of the ones that are mature, and is still terrible.
No, not even if you're counting in binary.
Of course, the velocity of development is very low, but that's because they want developers who can wrap their head around formal proofs.
I suspect hiring this company would make even a large budget wilt a bit. Just a bit :P
Heck, even without a type system, the problem is that the check is backwards. There shouldn't be a mask of flags that the guest isn't allowed to set, the flags should have been &ed with a mask that says which flags they are allowed to set, that way any new flag would have been disallowed by default.
It's not like Xen HVMs have a better security story than PVMs. The paravirtualization code should probably be more heavily audited than it is already.
I know it is within their right to write things as they please. But seriously, ranting like this is not very constructive and doesn't move things forward. If Qubes thinks the practices in Xen community are bad, why don't they start a conversation on Xen development mailing list?
(edited: typo)