In an attempt to go fast and beat benchmarks and other computers, CPUs attempt to speculatively execute code, and then later undo the results of the speculation if it turns out it was wrong. This causes all sorts of security issues (spectre, meltdown, and friends et al.) even when it's done relatively competently.
When it's done incompetently as on this ARM implementation, then you can't even run perfectly good and correct code, because the CPU will attempt speculative execution on a location that you never asked it to execute code at, and then bork itself when it realizes that can't possibly work.
Naturally, this is the sort of problem that requires tedious dissection of what exactly happened, and copious amounts of alcohol.
So in this specific case, if I understood correctly, here’s what should happen:
Interrupt(?) fires to trigger hypervisor, hypervisor figures out what it needs to do, jumps to that code, does its job, returns.
The “figured out what it needs to do” is the issue right? So what was actually happening was:
Same start… CPU predicts what hypervisor will do, speculatively loads instructions from mispredicted branch target, that wrong instruction reads memory(?) against the “no data prefetch” settings for that part of memory, CPU blows up/halts/whatever.
The fix is to mark the area the branch was mispredicted to in such a way that the CPU won’t prefetch instructions. Thus that won’t be run and prefetch data, thus no violation. CPU execution continues taking the correct branch and everything is fine.
No, that instruction does not read memory. That instruction is itself IN the inaccessible memory.
It does not really matter how execution ended up in the HV in the first place. The misprediction happens due to having a branch-to-register instruction.
Well, that was the whole issue. After marking the memory as not accessible via a data load, the CPU was still executing there and borking itself.
He had to also mark the memory as inaccessible for code loads.
Which, honestly is kinda stupid. Because nobody asked the processor to start executing there. It just took it upon itself to try to start executing there, later decided that was a bad thing, and then borked itself.
People talk about how terrible x86 is, but honestly, one of the reasons that x86 won for decades was because of making things that programmers did that might be suboptimal still work, even if a bit slowly (misaligned data accesses, for example). ARM does that now for that specific case, but didn't before 2002.
So there's an implementation tradeoff for whether you decide to spend transistors to reduce the number of sharp edges on the tool. Obviously ARM just doesn't give a shit about this particular sharp edge.
Speculative instruction fetches fetch code from "a location that you never asked it to execute code at" by design. If it already knew you asked it to execute code there, it wouldn't be speculative.
The Armv8/9-A architecture reference manual is clear that speculative instruction fetches are permitted in Device memory unless that memory is also marked NX. But if your hardware has side-effects from a certain address, but it maps it as Device non-NX memory, then your code is not "perfectly good and correct". Assigning correct memory attributes is one of the many things needed for correct code.
Basically, the NX bit prevents CPU behavior (speculative fetches) that had a hand in Spectre-type vulnerabilities. That's surprising because that's not its purpose. This is for ARM CPUs.
NX means not executable, so some bytes are excutable, some are non excutable. so some are instructions for the processor and others are data. Sometimes those are mixed and the cpu pukes out a bunch and restarts interpretation. most of the time those are normal. keep it normal and go fast.
Hot take: NX bit is shit W^X is shit. proper JIT is having objects written as needed, and cache line flushing is full bullshit, we need self modifying code as a first class citizen and with modern techiques we can have it work and not be crazy slow, it is currently cuz shits fucked, but we can do better.
Disallowing smc is a significant perf/power win. For cpus that run a large variety of large code (e.g. a web browser or ux stack), being able to cache a large instruction footprint and fetch/decode it quickly is important. Having to have the icache snoop data writes and entangle the i-fetch with the store buffer machinery would be a huge penalty to pay for a niche use case. Unlike loads, which are a small fraction of instructions to disambiguate with stores, you'd have to disambiguate every single instruction.
JIT is important to Apple platforms, and they seem to manage to make it work well enough even with the need for explicit invalidation.
So that is only true because we do it, there is a world where we optimize differently and that self modifying version works better, reread the synthesis kernel thesis(one, it is super easy, two they did this), we could have hardware that does this. Because we don't have hardware that does this we don't
FWIW if you look at WebKit/JavaScriptCore, the trend has been towards less-frequent code modification, e.g. inline caches are mostly no longer repatched inline. Whole-function reoptimization is still worth the overhead of sys_icache_invalidate() + `ISB`'ing everyone involved, but at smaller granularities not so much.
On this theme of it not being just about security: if you have a bug like a use after free and it happens to cover a function pointer, the nx bit can ensure that when you follow that pointer through a call, you get a clean trap as close to the failure point as possible. If it blindly executed stale bytes as code, maybe the crash and stack trace doesn't look as nice.
But then, a lot of correctness bugs like that are also security problems.
I appreciated this article. I've asked our CSP guy the difference, but this helps me better understand device vs ns. I also appreciate that this doesn't appear AI written.
Honestly feels like a misdesign in ARM. Where does it ever make sense for Device memory to not be data prefetchable, but allow instruction prefetch? It should IMHO disable all prefetch…
(Total guess) this feels like an attempt at restricting what parts of the hardware pipeline need to know what; if prefetches need to key off of both the device bit and the nx bit, then both of those need to be piped into the frontend, whereas if device is only relevant for data prefetches then only the nx bit needs to be available there (while device would be piped into the backend where the data prefetcher lives).
To the front end, one needs to pipe only appropriate logical combinations of the flags that describe the memory properties, not the flags as they are stored in some control register.
So a control register should contain flags that are meaningful for the programmer, while the kind of flags that hardware happens to need can be generated with a few logic gates from them and routed through hardware wherever they are needed.
It makes more sense to classify the memory in a few types, which must be specified by the programmer, including a "device memory" a.k.a. "memory-mapped peripherals" type, instead of having to specify for each memory area a long set of attributes about each kind of access that may be allowed, or not.
Standard UARTs are accessed in the x86 I/O space, which is uncacheable and strongly ordered, not in the normal memory space.
Non-standard UARTs that are on PCIe add-on cards might be mapped into memory, but when the computer boots, the BIOS already maps the PCIe memory as uncacheable, with the Memory Type Range Registers.
So if you do nothing, you get uncacheable memory for your UART, as you should.
To map the UART in normal memory, i.e. write-back cacheable, you must do this explicitly in the UART device driver.
If you really mapped the UART as normal memory, i.e. write-back cacheable, it can work correctly only if you are incredibly lucky, because the writes to the UART control registers will not happen when you do them in your program, so I do not believe that you did that. With normal memory, not only you cannot predict when the UART registers will be written, but if you do multiple writes to the same register, all but the last will be omitted, so there is no way for the UART to work correctly.
On x86-64, any memory-mapped peripheral must be mapped into uncacheable memory, except for memory banks that are located on PCIe cards, like the GPU memories, which should be mapped as write-combining memory.
The problem is that unlike data prefetch, the so-called "instruction prefetch" is not actually prefetch at all.
It's simply speculative execution. Which doesn't look any different to regular execution. The fetcher has no idea that its predicted branch is about to invalidated and flushed, otherwise it would never have issued that fetch.
Actually, on a modern OoO core, it's very rare for the instruction fetcher to not be doing speculative fetches. Even when it's not predicting a branch, the fact that it has "predicted" the lack of a branch is speculative in itself. It assumes it didn't fetch a branch in the last cycle, but it can't be sure until after instruction decoding, which takes at least 2 cycles (more on larger L1i caches).
About the only time the instruction fetcher is not doing speculative fetching is for a single cycle after each miss-predicted branch.
Well yes. That is why the "prefetch" is a problem.
But the original question was asking why disabling data prefetching to a memory region didn't automatically disable instruction prefetching at the same time.
And the answer is that speculative execution is a completely different mechanism that I'm not even sure can be disabled, at least not per memory region.
The fact it does speculative execution and dual-issue is well documented. The chipsandcheese article [0] is probably the best overview.
The "how" it does dual-issue is not documented at all, so I'm speculating. There is no smoking gun saying "register renaming". While it would be possible to implement its known capabilities without any kind of renaming, it would be so much simpler to implement it with register renaming.
The thing is.. once your forwarding network and hazard detection gets complicated enough, it basically becomes a janky form of register renaming. So it's cleaner to just implement proper register renaming, and actually saves hardware.
While the A53 might issue in-order, the pipelines have different lengths and they don't finish executing in order.
I find the fact the A53 can speculate a few instructions past a cache miss to be very interesting, along with the fact it can issue two writes to the same logical register in a single cycle.
Also, the smoking gun is that the A510 (same lineage as the A53) is documented to do out-of-order issue (see chipsandcheese again [1]), so it must be doing register renaming. ARM still insist on still calling it an in-order core because it's OoO is so much more limited to modern OoO cores, but it's more OoO than early PowerPC designs (including the G3) that everyone is happy applying the OoO label to.
This is doubly weird because actually purposefully executing from device memory is not allowed: “Trying to execute code from a region marked as Device is UNPREDICTABLE.” So: can’t reliably execute from there but can speculatively instruction fetch from there. Funsies!
> while ARM does provide a reference implementation of the architecture, vendors are free to customize it at will, or even roll their own implementations
A nitpick, but this is only true for some vendors, depending on their license, and is very much company-by-company. Many vendors, even big names like Meta, don't have the ability to roll their own. And even for the ones who do, 'customize at will' is a bit strong, as ARM very much does want to maintain uniformity across userspace implementations. E.g. Nvidia shouldn't add new traps for architecturally-legal behavior, since then code compiled for Apple hardware wouldn't work on Grace. Or worse, not trap for architecturally-illegal behavior, since then code compiled for Grace might not work for anyone else at all!
"Speculative instruction fetch to device memory crashes device" is in my list of favorite bugs ever found. I had ran into it on a Cortex-M4 without an MPU, so I couldn't even fix the memory attributes, the only solution was to gerrymander the compiled code to prevent the speculative fetch.
It's crazy that Armv8/9-A permits speculative instruction fetches to device memory. Crazy enough that I had to look it up to believe it:
"Hardware does not prevent speculative instruction fetches from a memory location with any of the Device memory attributes unless the memory location is also marked as execute-never for all Exception levels." - ARM DDI 0487K.a § B2.15.2
The Armv7-M spec is less clear. It does say: "The architecture does not permit speculative accesses to memory marked as Device," in contrast to Armv8/9-A which qualifies a similar statement with "data accesses". But then it later says "To ensure correctness, read-sensitive locations must be marked as non-executable". (This is all from ARM DDI 0403E.e § A3.5.7)
I don't have v7-A older handy to compare what they say.
Nowadays, it's considered bad form to map anything at 0... you now know one of the reasons why!
In the case that I know that has a 1:1 identity mapping of the address space, the physical address space doesn't actually have anything at 0 either, so it still doesn't need to be mapped.
I wrote an 'unlocking exploit' as a kernel module some years ago, and obfuscated it with a bit of self-modifying code.
It could not make it work reliable. No matter which barries I set, I could not understand why is it not reliable, and had to remove the modifying code in the end.
I guess I hit the issue you've described, and now I know. Thanks!
38 comments
[ 1.8 ms ] story [ 37.3 ms ] threadWhen it's done incompetently as on this ARM implementation, then you can't even run perfectly good and correct code, because the CPU will attempt speculative execution on a location that you never asked it to execute code at, and then bork itself when it realizes that can't possibly work.
Naturally, this is the sort of problem that requires tedious dissection of what exactly happened, and copious amounts of alcohol.
Interrupt(?) fires to trigger hypervisor, hypervisor figures out what it needs to do, jumps to that code, does its job, returns.
The “figured out what it needs to do” is the issue right? So what was actually happening was:
Same start… CPU predicts what hypervisor will do, speculatively loads instructions from mispredicted branch target, that wrong instruction reads memory(?) against the “no data prefetch” settings for that part of memory, CPU blows up/halts/whatever.
The fix is to mark the area the branch was mispredicted to in such a way that the CPU won’t prefetch instructions. Thus that won’t be run and prefetch data, thus no violation. CPU execution continues taking the correct branch and everything is fine.
It does not really matter how execution ended up in the HV in the first place. The misprediction happens due to having a branch-to-register instruction.
He had to also mark the memory as inaccessible for code loads.
Which, honestly is kinda stupid. Because nobody asked the processor to start executing there. It just took it upon itself to try to start executing there, later decided that was a bad thing, and then borked itself.
People talk about how terrible x86 is, but honestly, one of the reasons that x86 won for decades was because of making things that programmers did that might be suboptimal still work, even if a bit slowly (misaligned data accesses, for example). ARM does that now for that specific case, but didn't before 2002.
So there's an implementation tradeoff for whether you decide to spend transistors to reduce the number of sharp edges on the tool. Obviously ARM just doesn't give a shit about this particular sharp edge.
The Armv8/9-A architecture reference manual is clear that speculative instruction fetches are permitted in Device memory unless that memory is also marked NX. But if your hardware has side-effects from a certain address, but it maps it as Device non-NX memory, then your code is not "perfectly good and correct". Assigning correct memory attributes is one of the many things needed for correct code.
(Or did you confuse cache flushing with TLB flushing? The remap does the latter, not the former.)
JIT is important to Apple platforms, and they seem to manage to make it work well enough even with the need for explicit invalidation.
But then, a lot of correctness bugs like that are also security problems.
To the front end, one needs to pipe only appropriate logical combinations of the flags that describe the memory properties, not the flags as they are stored in some control register.
So a control register should contain flags that are meaningful for the programmer, while the kind of flags that hardware happens to need can be generated with a few logic gates from them and routed through hardware wherever they are needed.
It makes more sense to classify the memory in a few types, which must be specified by the programmer, including a "device memory" a.k.a. "memory-mapped peripherals" type, instead of having to specify for each memory area a long set of attributes about each kind of access that may be allowed, or not.
This is how it is done in x86-64.
Standard UARTs are accessed in the x86 I/O space, which is uncacheable and strongly ordered, not in the normal memory space.
Non-standard UARTs that are on PCIe add-on cards might be mapped into memory, but when the computer boots, the BIOS already maps the PCIe memory as uncacheable, with the Memory Type Range Registers.
So if you do nothing, you get uncacheable memory for your UART, as you should.
To map the UART in normal memory, i.e. write-back cacheable, you must do this explicitly in the UART device driver.
If you really mapped the UART as normal memory, i.e. write-back cacheable, it can work correctly only if you are incredibly lucky, because the writes to the UART control registers will not happen when you do them in your program, so I do not believe that you did that. With normal memory, not only you cannot predict when the UART registers will be written, but if you do multiple writes to the same register, all but the last will be omitted, so there is no way for the UART to work correctly.
On x86-64, any memory-mapped peripheral must be mapped into uncacheable memory, except for memory banks that are located on PCIe cards, like the GPU memories, which should be mapped as write-combining memory.
It's simply speculative execution. Which doesn't look any different to regular execution. The fetcher has no idea that its predicted branch is about to invalidated and flushed, otherwise it would never have issued that fetch.
Actually, on a modern OoO core, it's very rare for the instruction fetcher to not be doing speculative fetches. Even when it's not predicting a branch, the fact that it has "predicted" the lack of a branch is speculative in itself. It assumes it didn't fetch a branch in the last cycle, but it can't be sure until after instruction decoding, which takes at least 2 cycles (more on larger L1i caches).
About the only time the instruction fetcher is not doing speculative fetching is for a single cycle after each miss-predicted branch.
But the original question was asking why disabling data prefetching to a memory region didn't automatically disable instruction prefetching at the same time.
And the answer is that speculative execution is a completely different mechanism that I'm not even sure can be disabled, at least not per memory region.
The "how" it does dual-issue is not documented at all, so I'm speculating. There is no smoking gun saying "register renaming". While it would be possible to implement its known capabilities without any kind of renaming, it would be so much simpler to implement it with register renaming.
The thing is.. once your forwarding network and hazard detection gets complicated enough, it basically becomes a janky form of register renaming. So it's cleaner to just implement proper register renaming, and actually saves hardware.
While the A53 might issue in-order, the pipelines have different lengths and they don't finish executing in order.
I find the fact the A53 can speculate a few instructions past a cache miss to be very interesting, along with the fact it can issue two writes to the same logical register in a single cycle.
Also, the smoking gun is that the A510 (same lineage as the A53) is documented to do out-of-order issue (see chipsandcheese again [1]), so it must be doing register renaming. ARM still insist on still calling it an in-order core because it's OoO is so much more limited to modern OoO cores, but it's more OoO than early PowerPC designs (including the G3) that everyone is happy applying the OoO label to.
[0] https://chipsandcheese.com/p/arms-cortex-a53-tiny-but-import... [1] https://chipsandcheese.com/p/arms-cortex-a510-two-kids-in-a-...
https://support.arm.com/documentation/100941/0101/Memory-typ...
A nitpick, but this is only true for some vendors, depending on their license, and is very much company-by-company. Many vendors, even big names like Meta, don't have the ability to roll their own. And even for the ones who do, 'customize at will' is a bit strong, as ARM very much does want to maintain uniformity across userspace implementations. E.g. Nvidia shouldn't add new traps for architecturally-legal behavior, since then code compiled for Apple hardware wouldn't work on Grace. Or worse, not trap for architecturally-illegal behavior, since then code compiled for Grace might not work for anyone else at all!
It's crazy that Armv8/9-A permits speculative instruction fetches to device memory. Crazy enough that I had to look it up to believe it:
"Hardware does not prevent speculative instruction fetches from a memory location with any of the Device memory attributes unless the memory location is also marked as execute-never for all Exception levels." - ARM DDI 0487K.a § B2.15.2
The Armv7-M spec is less clear. It does say: "The architecture does not permit speculative accesses to memory marked as Device," in contrast to Armv8/9-A which qualifies a similar statement with "data accesses". But then it later says "To ensure correctness, read-sensitive locations must be marked as non-executable". (This is all from ARM DDI 0403E.e § A3.5.7)
I don't have v7-A older handy to compare what they say.
In the case that I know that has a 1:1 identity mapping of the address space, the physical address space doesn't actually have anything at 0 either, so it still doesn't need to be mapped.
It could not make it work reliable. No matter which barries I set, I could not understand why is it not reliable, and had to remove the modifying code in the end.
I guess I hit the issue you've described, and now I know. Thanks!