I couldn't edit my above post in time because the server was unreachable. What I wanted to add:
I think I misunderstood the blog author's question. He wasn't asking about the ENTER statement with a non-zero argument, but about the special ABI with arrays of frame pointers.
I love how it was already clear that it would be Raymond Chen's blog just from the title and domain name. I really enjoy The Old New Thing and his coverage of obscure details that might otherwise be long forgotten.
His blog is the best, especially when he talks about ancient versions of Windows. He talk about obscure details, but they're so interesting because they give you insight on how things were made!
Well, I also can't recall what I did last month, but if you put me in front of an Apple II+, I know I can get into the machine language monitor with "CALL -151".
Right on the money. Almost. Measurements suggest the microcode has special cases for 0 and 1 level of displays and these perform better, though not crazy well.
Ah, so it's for constructing displays. A nice feature to have if you expect e.g. Pascal to be the most popular high-level language used on your ISA, I guess. Probably could also be used to implement flat closures when combined with "Cheney on the M.T.A." allocation scheme?
these kinds of details about x64 and x86 make it so hard to build something from intel or amd docs. what do you use, what do you omit, feature and instruction wise. so many ways to do the same things.. it takes a lot of experimentation and measurements to see whats efficient and whats simple to implement etc. you can peek at nix and other osses but i suppose tbat defeats the purpose of the excersize :d. anyway, nice blog all in all =) always happy to see people translate more complex constructs into the basic instructions. helps understand a lot what the cpu is actually doing.
As I'm sure most of us know, the x86 lineage goes back nearly 50 years. Many ideas that went nowhere, were taken as design decisions, and are forever etched in silicon.
One of my favourite examples is the x86 task switching support. The 386 had everything needed to implement context-switching and multitasking -- in hardware. (That notion was in vogue, once. See the ICL 2900.) If you set up the right structure in memory, and turn on the hardware task switching, x86 processors will automatically context switch all of the registers in and out of that in-memory structure, at the appropriate privilege boundaries (interrupt, return from interrupt, etc.) So your kernel can switch between processes like this (more or less):
mov next_proc_ptr, tr
iret
It also provides IO port protection - there's another table that lets you specify which ports the process can access, which the processor will examine, when an IO instruction is executed in user mode, presumably for something microkernelish.
Now that's the good ol' CISC spirit. It's quite a complex feature and yet I'm not sure any OS ever used it.
Even if no OS used it I bet it has been used - the era of the 386 saw a decent amount of bare-metal specialized uses of the hardware. Some would nominally run on DOS, some didn’t bother with it at all.
Originally, probably because it was inflexible. You have to use the in-memory layout required by the hardware, and you can't do anything not supported by the hardware. Later, it didn't get updated (probably because no one used it) so today it won't save things like the MMX and SSE registers, which makes it even less useful.
It's certainly much faster today to use sequences of mov instructions. In practice, it's a block move of multiple registers to memory/cache. That's something any modern pipelined and microcoded processor will have specialized support for. What instruction or instructions generates that internal action, is largely beside the point.
It didn't save the x87 FPU registers either, however a task switch set a flag in MSW / CR0 that caused a trap on the next instruction using these registers. The idea is that they are used infrequently, so it's better to only switch the FPU context when required.
This is no longer so true today, however the Linux kernel for example doesn't use any floating point or vector instructions, in order to avoid having to save these registers on every system call or interrupt.
The real inflexibility was scheduling (see my other comment)
It would only be more efficient if user mode tasks jumped directly to each other without any kernel involvement. This is actually possible, but probably useless except maybe some very specialized embedded applications. If you already have a privilege level change and pushing of registers on the stack, a hardware tasks switch just does useless extra work.
One might think that since interrupts can switch tasks, you could set up a simple round-robin scheduler by hooking this mechanism up to the timer interrupt somehow, or have device drivers in separate tasks that become active when receiving an interrupt.
But a "task state segment" on x86 is actually more like a pre-allocated stack frame, with interrupts creating a linked list of these. A task can't be reentered while it is active, and it is supposed to exit by returning to the previous task that was interrupted. Anything else requires software scheduling.
The only reason to ever use this mechanism is for handling exceptions on a specially reserved stack (for example when the kernel stack overflows).
The way processors work, if the single instruction / automatic on interrupt etc method does the same things as do it yourself in your code version, about the only thing you're saving is instruction decode bandwidth and compiled code size. But that's usually not a large enough savings to matter.
And then you get the inflexibility: There's a limited number of hardware task slots; I think you can pick which registers to save (or rather how many to save), but you always save all of them: a kernel syscall might be more judicious and only save the registers it uses (of course saving all registers if returning to a different task); it's hard to change because software must run on old and new cpus; etc.
There's exceptions to the rule of thumb. Rep mov can be faster than doing it yourself. It's also quite possible for a combined operation to do two things at the same time rather than one after the other.
When nobody uses such a feature is typically either because it has worse performance then alternatives or some missing feature. There have been many such cases in x86.
> It's quite a complex feature and yet I'm not sure any OS ever used it.
They did.
Linux used x86 hardware task switching and TR (task register) for years. See "struct tss_struct" and "switch_to" in older kernels.
Linux also allowed userspace processes to get hardware-managed access to individual I/O ports using the per-task x86 I/O port bitmap, and to local segments using the x86 LDT which were useful for running 16-bit code in a 32-bit task.
The hardware task switching was eventually changed to software task switching. By that time, software switching was faster, and it was always more versatile. This change removed the hardware-imposed limit on the number of processes and threads Linux could support.
i looked so long to implement hardware task switching only to find modern OSes decided it wasnt to be used. still not sure im onboard with that decicion haha. great example!! =)
From everything I've seen, the x86 and x86-64 instruction sets are loaded with "it seemed like a good idea at the time", "if you couldn't count on FPU / MMX / SSE instructions being available...", and similar waste-of-silicon instructions.
Plus, there's a feedback loop: Compilers writers restrict themselves to subset S of the full instruction set for cost-benefit reasons <==> CPU designers devote minimal time / effort / silicon to improving the performance of non-S instructions.
> From everything I've seen, the x86 and x86-64 instruction sets are loaded with "it seemed like a good idea at the time", "if you couldn't count on FPU / MMX / SSE instructions being available...", and similar waste-of-silicon instructions.
They are, but a lot of "instructions" these days don't even map onto native silicon logic anyway like they used to. It's all microcode these days.
I know we're splitting hairs here, but really the only difference between microcoded instructions and "native" instructions is that the instruction decoder directly produces u-ops versus doing a micro code table lookup. The microcode table lookup just ends up producing u-ops, and those all go through the rest of the out-of-order engine just the same. It's really just the frontend that is the difference. For CPUs that have a trace (u-op) cache, I think the results of microcoded instructions can also be stored there.
True, although microcode updates can still alter hardwired logic in a more limited way - it can’t fundamentally alter the logic, but (if designed to permit that) can reconfigure it - such as by disabling certain features. You just need some signals that change how the hardwired logic behaves, and a configuration register to feed them, and then allow the microcode to write to that configuration register.
Yeah, I assume it's similar how ARM errata are often worked around - set some bit in an undocumented MSR to disable some microarchitectural optimization.
A silicon vendor I worked at called them CYA (Cover Your Ass) bits :)
The difference is that you do not necessarily need dedicated opcodes to implement a specific instruction if you can implement it on term of other generi opcodes. With no dedicated hardware, the cost then is only negligible ROM space, and not so neglegible encoding space.
Interesting ... Not sure how many languages other than Pascal support local functions and allow access to (potentially multiple levels of) parent variables in this way. The normal Pascal terminology for this "array of frame pointers" is the "display", and it wouldn't normally be copied during a function/procedure entry - just updated (assuming the function being entered had local functions that accessed parent variables).
Only most of the languages you use every day[1]. Pascal inherited nested functions from ALGOL so everything in that family does it. Lisp dialects with lexical scope will have it. You can even get it in C if you use GCC[2]. (And if you don't mind turning off the NX stack.)
Well, I'm not actually using ALGOL or Lisp every day, but ok ... :)
There's a variety of implementations there though - some seem to be talking about closures, and at least one (Rust) doesn't appear to support parent variable access.
I don't know how compilers typically implement closures, but that does seem to be what some languages on that list (e.g. D) support by way of "nested/local functions", and is obviously a bit different in terms of variable lifetime.
In Pascal local functions are not closures and do not affect variable storage. All variables are still stored in the stack frame (unless optimized to be in registers), and local functions can access their parent's (& grandparent's etc) variables via pointers to their stack frames, which are normally held globally in an array of frame pointers called the display.
For a Pascal depth-N nested local function to access it's parent's variables it would use the depth-(N-1) display entry, for it's grandparent's variables the depth-(N-2) entry etc. When a parent calls a nested function the only thing that needs to change in the display is to set the depth N-1 display pointer to it's own stack frame, and to replace it with the old N-1 pointer when the call is complete. The grandparent/etc entries don't need to change.
This x86 ENTER instruction appears to be designed for a more general case than Pascal, since it's apparently copying the entire display (array of frame pointers), not just doing the more efficient incremental update/restore that is all Pascal needs.
It's interesting to see Raymond Chen writing about this "almost always unused" ENTER parameter since it suggests that compilers are not using it even for those languages that might seem to be able to make use of it. Perhaps, as for Pascal, it's not the most efficient way, or perhaps just because languages such as JavaScript don't normally compile to machine code.
"JPC is a fast modern x86 PC emulator capable of booting Windows up to Windows 95 (and windows 98 in safe mode) and some graphical linuxes. It has a full featured graphical debugger with time travel mode along with standard features like break and watch points."
I second that. I can't count how many times Raymond Chen has helped me understand things while working on Windows polyfills for Cosmopolitan Libc. Microsoft is very fortunate to have him.
Interesting. The TXR Lisp VM uses displays, and has a similar concept with keeping track of levels. We can see that with code. We disable optimization so the compiler doesn't eliminate frames:
Interesting. To my untrained eye, this looks like its only purpose is to produce spaghetti code. What's a scenario where this is actually desirable? (I.e., where you actually want to be able to access the stack frame of a parent function call)
51 comments
[ 3.0 ms ] story [ 115 ms ] threadTFA gives at least a partial answer to my earlier q: https://news.ycombinator.com/item?id=38544640
I think I misunderstood the blog author's question. He wasn't asking about the ENTER statement with a non-zero argument, but about the special ABI with arrays of frame pointers.
I suspect it is completely microcoded and there is no dedicated silicon to implement it (well other than ROM space to store the microcode I guess).
One of my favourite examples is the x86 task switching support. The 386 had everything needed to implement context-switching and multitasking -- in hardware. (That notion was in vogue, once. See the ICL 2900.) If you set up the right structure in memory, and turn on the hardware task switching, x86 processors will automatically context switch all of the registers in and out of that in-memory structure, at the appropriate privilege boundaries (interrupt, return from interrupt, etc.) So your kernel can switch between processes like this (more or less):
It also provides IO port protection - there's another table that lets you specify which ports the process can access, which the processor will examine, when an IO instruction is executed in user mode, presumably for something microkernelish.Now that's the good ol' CISC spirit. It's quite a complex feature and yet I'm not sure any OS ever used it.
It's certainly much faster today to use sequences of mov instructions. In practice, it's a block move of multiple registers to memory/cache. That's something any modern pipelined and microcoded processor will have specialized support for. What instruction or instructions generates that internal action, is largely beside the point.
This is no longer so true today, however the Linux kernel for example doesn't use any floating point or vector instructions, in order to avoid having to save these registers on every system call or interrupt.
The real inflexibility was scheduling (see my other comment)
it doesn't use them by default, but code that needs the FPU can call kernel_fpu_begin() to save those registers if needed. https://yarchive.net/comp/linux/kernel_fp.html One big user is the AMD graphics driver: https://lore.kernel.org/amd-gfx/ee26b06b-f1a0-5a8f-388c-13c1...
One might think that since interrupts can switch tasks, you could set up a simple round-robin scheduler by hooking this mechanism up to the timer interrupt somehow, or have device drivers in separate tasks that become active when receiving an interrupt.
But a "task state segment" on x86 is actually more like a pre-allocated stack frame, with interrupts creating a linked list of these. A task can't be reentered while it is active, and it is supposed to exit by returning to the previous task that was interrupted. Anything else requires software scheduling.
The only reason to ever use this mechanism is for handling exceptions on a specially reserved stack (for example when the kernel stack overflows).
And then you get the inflexibility: There's a limited number of hardware task slots; I think you can pick which registers to save (or rather how many to save), but you always save all of them: a kernel syscall might be more judicious and only save the registers it uses (of course saving all registers if returning to a different task); it's hard to change because software must run on old and new cpus; etc.
There's exceptions to the rule of thumb. Rep mov can be faster than doing it yourself. It's also quite possible for a combined operation to do two things at the same time rather than one after the other.
When nobody uses such a feature is typically either because it has worse performance then alternatives or some missing feature. There have been many such cases in x86.
They did.
Linux used x86 hardware task switching and TR (task register) for years. See "struct tss_struct" and "switch_to" in older kernels.
Linux also allowed userspace processes to get hardware-managed access to individual I/O ports using the per-task x86 I/O port bitmap, and to local segments using the x86 LDT which were useful for running 16-bit code in a 32-bit task.
The hardware task switching was eventually changed to software task switching. By that time, software switching was faster, and it was always more versatile. This change removed the hardware-imposed limit on the number of processes and threads Linux could support.
Here's a nice article about the evolution of task switching in Linux. https://www.maizure.org/projects/evolution_x86_context_switc...
From everything I've seen, the x86 and x86-64 instruction sets are loaded with "it seemed like a good idea at the time", "if you couldn't count on FPU / MMX / SSE instructions being available...", and similar waste-of-silicon instructions.
Plus, there's a feedback loop: Compilers writers restrict themselves to subset S of the full instruction set for cost-benefit reasons <==> CPU designers devote minimal time / effort / silicon to improving the performance of non-S instructions.
They are, but a lot of "instructions" these days don't even map onto native silicon logic anyway like they used to. It's all microcode these days.
I know we're splitting hairs here, but really the only difference between microcoded instructions and "native" instructions is that the instruction decoder directly produces u-ops versus doing a micro code table lookup. The microcode table lookup just ends up producing u-ops, and those all go through the rest of the out-of-order engine just the same. It's really just the frontend that is the difference. For CPUs that have a trace (u-op) cache, I think the results of microcoded instructions can also be stored there.
Of course the ability to modify non-microcoded instructions is significantly smaller and limited to what was planned for at design time.
A silicon vendor I worked at called them CYA (Cover Your Ass) bits :)
This is all the very small cost of backwards compatability.
[1] https://en.wikipedia.org/wiki/Nested_function#Languages
[2] https://www.gnu.org/software/c-intro-and-ref/manual/html_nod...
There's a variety of implementations there though - some seem to be talking about closures, and at least one (Rust) doesn't appear to support parent variable access.
Isn't this just a lexical closure? It's common in scripting languages.
https://wiki.c2.com/?LexicalClosure
In Pascal local functions are not closures and do not affect variable storage. All variables are still stored in the stack frame (unless optimized to be in registers), and local functions can access their parent's (& grandparent's etc) variables via pointers to their stack frames, which are normally held globally in an array of frame pointers called the display.
For a Pascal depth-N nested local function to access it's parent's variables it would use the depth-(N-1) display entry, for it's grandparent's variables the depth-(N-2) entry etc. When a parent calls a nested function the only thing that needs to change in the display is to set the depth N-1 display pointer to it's own stack frame, and to replace it with the old N-1 pointer when the call is complete. The grandparent/etc entries don't need to change.
This x86 ENTER instruction appears to be designed for a more general case than Pascal, since it's apparently copying the entire display (array of frame pointers), not just doing the more efficient incremental update/restore that is all Pascal needs.
It's interesting to see Raymond Chen writing about this "almost always unused" ENTER parameter since it suggests that compilers are not using it even for those languages that might seem to be able to make use of it. Perhaps, as for Pascal, it's not the most efficient way, or perhaps just because languages such as JavaScript don't normally compile to machine code.
https://github.com/ianopolous/JPC/blob/master/src/org/jpc/em...
https://github.com/ianopolous/JPC/blob/master/src/org/jpc/em...
"JPC is a fast modern x86 PC emulator capable of booting Windows up to Windows 95 (and windows 98 in safe mode) and some graphical linuxes. It has a full featured graphical debugger with time travel mode along with standard features like break and watch points."