I hope it's ok to answer here instead of on vcfed.org since I don't want to create yet another account...
Sun workstations did not use the dual 68000 approach to handle VM. The problem with the 68000 was that it didn't save enough information on the stack when an exception occurred (e.g. a page fault), so a faulting instruction could not be recovered/re-executed in all cases.
Sun solved this by using "harmless" instructions such as "tst" that did non have to be recovered in case of a page fault. The OS could then kick in, load the contents to memory and just skip the failed instruction. This required changes to the compiler, e.g. when creating a new stack frame that would have required to extend the stack size. For paging, this approach would cause significant overhead. However, the first versions of SunOS which ran on the Sun 1 workstations were based on 7th Edition Unix and used only swapping.
The 68010, used in the Sun-2 workstation, fixed the restarting problem by saving sufficient state to the stack, so SunOS could switch to an early BSD release as its basis.
AFAIK, the dual 68000 approach was used by early Apollo workstations instead. The way I understand it is that the second processor was idling until a page fault occurred. The 68000 has an asynchronous bus which requires an explicit acknowledge (/DTACK) to indicate that data (e.g. from slow memory) has arrived. This feature was used by the Apollos - the second 68k would kick in while /DTACK on the first 68k was deasserted, handle the required virtual memory operations, and then indicated that the first CPU can continue working. So here, no bus error was indicated (which would have caused the restarting problem), but the virtual memory behaved like a very slow physical memory. Apollos were able to perform remote memory accesses to other machines over the TokenRing interface (which was about as fast as memory accesses - 12 Mbit/s IIRC), so this feature was much more useful for DomainOS than for early Unix systems.
Please note that I can't confirm that Apollo used this approach - the earliest Apollo workstation I know (the DN300) used a 68010 already. I would love to hear more details on this...
I'm also not sure which approach early SGI workstations used (the Iris 1000 series), documentation on those is extremely hard to find.
I've also heard and seen the claim that Apollo workstations before the 68010 used this method, but it's difficult to find a definite source for this (though it's likely true. Every reference etc. to the pre-DN300 models claim 2x68000 CPU)
This is great info. Can you elaborate or point to a link on the distinction between paging and swapping? I’ve always treated them as synonyms but clearly there are some differences on earlier systems.
Swapping (in the context of old Unix) is always replacing (copying) an entire process address space (or at least the non-shareable read-write segments such as .data), out to the swap device, or back, in one go.
Paging is more advanced as it allows to keep only a part of the memory required by a process in RAM and contents can be loaded or moved to secondary storage on the granularity of single pages (typically 4 kB, but larger pages are also common nowadays). So a typical exec(2) system call in Unix could set up a new page table which contains no single byte of the executable in the beginning and then jump to the executable's entry point. This, in turn, results in a page fault, in turn, the OS can then load the requested page into RAM.
Paging is more efficient since control/data flow analyses show that something similar to the Pareto rule (or 80/20 rule) also applies to programs which are executed - 80% of the time, only 20% of the address range (for instructions and/or data) is used. With paging, the OS won't need to load any code or data from disk that is never used (and can load it on demand later).
What's probably confusing is that the area reserved on disk for storing RAM contents (a partition or a file) is still called "swap partition/file". This is an old term that refuses to die, I guess...
So “old” swapping is taking low priority processes and putting all of their memory out to disk, almost (but obviously not quite) as though you’re suspending them. Got it!
The setup was more complex than I described, as the second 68000 did not simply wait for a page fault to happen but was executing regular code (unclear if application only or also OS kernel, but the code was not allowed to raise page faults, user processes usually ran on one CPU, see the last quote below) and only handled the fault on the other CPU when interrupted by the MMU.
Unfortunately, the only versions of Aegis/DomainOS available online seem to be 10.x (on bitsavers), but support for the "sau1" 68000-based systems (DN100/DN400/DN416) ended with version 9.x, so we can't have a look at the code implementing this feature.
Interesting quotes from the thread:
"The two processors were called "A" and "B" and processes could be marked runnable on A or on B or both. It was almost a multiprocessor, except that you couldn't run anything that might take a page fault on both at the same time." (by Jim Rees, who runs the Apollo archive at https://jim.rees.org/apollo-archive/)
The answer to the following question was posted by Dave Tweed, a former Apollo engineer who worked on the MMU design for later Apollo workstations (see his resume at http://www.dtweed.com/resume2.html):
"> Hmm. So how did CPU B know that this was needed? It sounds like they
> weren't running the same code, after all.
No, of course not. The MMU was external to both CPUs, and when one CPU encountered a page fault, the MMU would interrupt the other CPU to handle the exception.
The CPUs shared the main memory, so in that sense they were running the same code -- e.g., there was only one copy of the exception handler that either CPU could run. But they were not lock-stepped the way the OP described. I can't even imagine how you'd accomplish that in general."
"Back in the good old days, the 68000 was the only Motorola processor available. It did not have restartable instructions, so you couldn't do VM with them. Early Suns didn't have VM, they squeezed everything into real memory, like a pdp-11. Early Apollos had an ingenious VM system. There were two processors, the 'A' and the 'B'. One of them would run until it took a page fault, then it would go into a memory wait while the other one brought the page in.
It's not really parallel in the usual sense of the word, but there were certain things that could run on either processor, and in fact you could have both of them running at the same time. Normal user processes only ran
on one."
Another not quite that interesting approach (more of an I/O coprocessor) was used when you put a Z80 CP/M "Softcard" into a (6502-based) Apple II.
Steve Wozniak's floppy controller was very simple on the hardware side (more or less only a state machine implemented in a PROM), so most of the data serializing/deserializing and GCR de/encoding was done in software on the 6502. This was very timing-critical, so implementing this in Z80 code for CP/M instead would have been a challenge.
Instead, the Z80 on the Softcard called 6502 routines to handle disk (and other) I/O. Usually, the 6502 is stopped when running Z80 code, but it had to be woken up frequently not only for I/O accesses, but also to ensure that the 6502 did not lose its register values, which are stored in dynamic memory cells in the (non-CMOS) 6502.
Tandem (now HPE) NonStop do something like this... Run 2 systems in lock-step for super resilient computing. The code has specific instructions for validating operations by comparing the outputs on both CPUs
Also Stratus: Wikipedia says they were originally built around M68Ks. The WP article is a bit thin on Stratus's early products, but my vague recollection is that they used 3 processors in lockstep, with voting to detect a faulty processor.
The DEC J-11 CPU is a single CPU that is implemented on two discrete chips with one taking care of control and the other of data paths, not exactly the same idea as using two identical cpu's to implement a different architecture, but somewhat related.
This sounds like the way the 8087 and 8089 (look that one up!) co-processors worked: they snooped the instructions the 8086/8088 CPU read from memory and then handled their own subset of the instruction set. That did require that they could reconstruct the program flow from instruction fetches, which probably became nigh impossible when CPUs started integrating instruction caches.
Nitpick: Only the x87 line snooped the bus and mimicked prefetch as it was a coprocessor that had its opcodes inside the x86's. Specifically, any opcode beginning with 0xD8 through 0xDF is an x87 opcode. The 8089 is different, and worked more like old IBM mainframes did with I/O channels; The 8089 had a dedicated opcode set (distinct from x86) that it would execute. "Jobs" (of sorts) were created on the 8086 CPU, then handed over to the 8089.
It's interesting that Intel decided to use two different methods for executing coprocessor instructions. Why not just have the 8087 work like the 8089 does and send "jobs" to execute? Embedding the instruction set inside the 8086's meant having to put in work to duplicate the prefetch queue.
I've worked on high availability systems where CPUs were duplicated for reliability. In one product - which I won't name here - the surrounding ASICs were constructed of "self-checking logic" - where every data path had parity and all logic elements were designed such that no single wire failure could go undetected. The idea was that with near 100% reliability, a compute module could flag itself defective before recent work by it could be "committed" into the main memory which was transaction based (every segment of which also had to have at least two redundant copies in different modules).
But who will sell you a self-checking CPU? So use two of them and compare everything they do from reset. This may not be so easy with modern CPUs with huge internal clock rates - who says that they will emerge from asynchronous reset in step, for example? But with 1990s technology it was fine.
The phone switch GTD-5 had 2 separate sets of 2 CPUs for each function and instructions were run through both, then compared: https://en.wikipedia.org/wiki/GTD-5_EAX
"If the results were not identical, the processors were immediately reset, and the pair of processors on the other card were brought online as the active processor complex. The active processor always kept memory up-to-date so that when these forced switches occurred, little data loss was suffered. When the switch was requested as a part of routine maintenance, the switch could be accomplished with no data loss at all."
25 comments
[ 3.2 ms ] story [ 43.2 ms ] threadSun workstations did not use the dual 68000 approach to handle VM. The problem with the 68000 was that it didn't save enough information on the stack when an exception occurred (e.g. a page fault), so a faulting instruction could not be recovered/re-executed in all cases.
Sun solved this by using "harmless" instructions such as "tst" that did non have to be recovered in case of a page fault. The OS could then kick in, load the contents to memory and just skip the failed instruction. This required changes to the compiler, e.g. when creating a new stack frame that would have required to extend the stack size. For paging, this approach would cause significant overhead. However, the first versions of SunOS which ran on the Sun 1 workstations were based on 7th Edition Unix and used only swapping.
The 68010, used in the Sun-2 workstation, fixed the restarting problem by saving sufficient state to the stack, so SunOS could switch to an early BSD release as its basis.
AFAIK, the dual 68000 approach was used by early Apollo workstations instead. The way I understand it is that the second processor was idling until a page fault occurred. The 68000 has an asynchronous bus which requires an explicit acknowledge (/DTACK) to indicate that data (e.g. from slow memory) has arrived. This feature was used by the Apollos - the second 68k would kick in while /DTACK on the first 68k was deasserted, handle the required virtual memory operations, and then indicated that the first CPU can continue working. So here, no bus error was indicated (which would have caused the restarting problem), but the virtual memory behaved like a very slow physical memory. Apollos were able to perform remote memory accesses to other machines over the TokenRing interface (which was about as fast as memory accesses - 12 Mbit/s IIRC), so this feature was much more useful for DomainOS than for early Unix systems.
I'm also not sure which approach early SGI workstations used (the Iris 1000 series), documentation on those is extremely hard to find.
https://ia904706.us.archive.org/5/items/Motorola-Seminarsand... In there they suggest that "another bus master must perform this action" and that this one could be "an MC68000 MPU as well".
I've also heard and seen the claim that Apollo workstations before the 68010 used this method, but it's difficult to find a definite source for this (though it's likely true. Every reference etc. to the pre-DN300 models claim 2x68000 CPU)
Paging is more advanced as it allows to keep only a part of the memory required by a process in RAM and contents can be loaded or moved to secondary storage on the granularity of single pages (typically 4 kB, but larger pages are also common nowadays). So a typical exec(2) system call in Unix could set up a new page table which contains no single byte of the executable in the beginning and then jump to the executable's entry point. This, in turn, results in a page fault, in turn, the OS can then load the requested page into RAM.
Paging is more efficient since control/data flow analyses show that something similar to the Pareto rule (or 80/20 rule) also applies to programs which are executed - 80% of the time, only 20% of the address range (for instructions and/or data) is used. With paging, the OS won't need to load any code or data from disk that is never used (and can load it on demand later).
What's probably confusing is that the area reserved on disk for storing RAM contents (a partition or a file) is still called "swap partition/file". This is an old term that refuses to die, I guess...
https://groups.google.com/g/comp.sys.apollo/c/55dqpf11hEw/m/...
The setup was more complex than I described, as the second 68000 did not simply wait for a page fault to happen but was executing regular code (unclear if application only or also OS kernel, but the code was not allowed to raise page faults, user processes usually ran on one CPU, see the last quote below) and only handled the fault on the other CPU when interrupted by the MMU.
Unfortunately, the only versions of Aegis/DomainOS available online seem to be 10.x (on bitsavers), but support for the "sau1" 68000-based systems (DN100/DN400/DN416) ended with version 9.x, so we can't have a look at the code implementing this feature.
Interesting quotes from the thread:
"The two processors were called "A" and "B" and processes could be marked runnable on A or on B or both. It was almost a multiprocessor, except that you couldn't run anything that might take a page fault on both at the same time." (by Jim Rees, who runs the Apollo archive at https://jim.rees.org/apollo-archive/)
The answer to the following question was posted by Dave Tweed, a former Apollo engineer who worked on the MMU design for later Apollo workstations (see his resume at http://www.dtweed.com/resume2.html):
"> Hmm. So how did CPU B know that this was needed? It sounds like they > weren't running the same code, after all.
No, of course not. The MMU was external to both CPUs, and when one CPU encountered a page fault, the MMU would interrupt the other CPU to handle the exception.
The CPUs shared the main memory, so in that sense they were running the same code -- e.g., there was only one copy of the exception handler that either CPU could run. But they were not lock-stepped the way the OP described. I can't even imagine how you'd accomplish that in general."
However, from another thread, also by Jim Rees (https://groups.google.com/g/comp.sys.apollo/c/GzEtBHVHEvA/m/...):
"Back in the good old days, the 68000 was the only Motorola processor available. It did not have restartable instructions, so you couldn't do VM with them. Early Suns didn't have VM, they squeezed everything into real memory, like a pdp-11. Early Apollos had an ingenious VM system. There were two processors, the 'A' and the 'B'. One of them would run until it took a page fault, then it would go into a memory wait while the other one brought the page in.
It's not really parallel in the usual sense of the word, but there were certain things that could run on either processor, and in fact you could have both of them running at the same time. Normal user processes only ran on one."
What a fascinating rabbit hole...
Steve Wozniak's floppy controller was very simple on the hardware side (more or less only a state machine implemented in a PROM), so most of the data serializing/deserializing and GCR de/encoding was done in software on the 6502. This was very timing-critical, so implementing this in Z80 code for CP/M instead would have been a challenge.
Instead, the Z80 on the Softcard called 6502 routines to handle disk (and other) I/O. Usually, the 6502 is stopped when running Z80 code, but it had to be woken up frequently not only for I/O accesses, but also to ensure that the 6502 did not lose its register values, which are stored in dynamic memory cells in the (non-CMOS) 6502.
https://en.wikipedia.org/wiki/File:Tandem_Mug_with_Redundant...
Indeed, I'd never heard of the 8089. Now I'd like to know more about it.
Edit: More links.
https://retrocomputing.stackexchange.com/a/13815
http://www.bitsavers.org/pdf/intel/ISIS_II/9800938-01_8089_A...
It's interesting that Intel decided to use two different methods for executing coprocessor instructions. Why not just have the 8087 work like the 8089 does and send "jobs" to execute? Embedding the instruction set inside the 8086's meant having to put in work to duplicate the prefetch queue.
But who will sell you a self-checking CPU? So use two of them and compare everything they do from reset. This may not be so easy with modern CPUs with huge internal clock rates - who says that they will emerge from asynchronous reset in step, for example? But with 1990s technology it was fine.
"If the results were not identical, the processors were immediately reset, and the pair of processors on the other card were brought online as the active processor complex. The active processor always kept memory up-to-date so that when these forced switches occurred, little data loss was suffered. When the switch was requested as a part of routine maintenance, the switch could be accomplished with no data loss at all."