watching the talk. does he not compare this to a classic segment/call gate architecture because he doesn't expect it to be a familiar reference? i'm certain he's seen it before :)
edit: i thought they managed to do all of this without segments, but at the end we hear about a special local segment with offset addressing apparently introduced just to handle children of fork()..i lost how cow can be expressed losslessly as local/global
redit: question around 1:00:00 explicitly ask this question, and he said, erroneously i think that while semantically similar this is the first time a direct hardware implementation of a call gate has been proposed
Ivan definitely mentioned capabilities at some point in the evening - "I'd love to build a capability architecture but I wouldn't be able to sell it" (which he's also mentioned in previous talks) but it may have been after the camera stopped rolling.
He does say that, but if you watch previous talks (either that or the material available) he says the primary difference (IIRC) is that you aren't supposed to segment individual data objects, but rather instead coarse grained address space segmentation, so there's really no technical difference, but it's kind of an usage difference.
i think the other (turf-like) segmentation strategy is pretty common in earlier work. i wonder about the relative efficacy from a VM implementation standpoint. particularily wrt grant/revoke on a byte range rather than a segment.
the other thing that struck me as really strange was a bit in the question period where he says that 'smart devices' (DMA?) are different from simple devices (pio?) in that they are expected to be first-class multiprocessing citizens. doesn't that imply that high-performance peripherals need to be specially designed for the Mill? (likely with a Mill core attached)
It is, it's not nearly as radical or innovative as the HN crowd would like to think, probably because it's mostly software people with a limited knowledge of 1990s era computer architecture research.
As for peripherals, I presume that would have to be handled in the drivers.
We have focused on the core and less on the uncore, which is why there have been no talks on I/O. The goal is for a smart peripheral to be indistinguishable from just another regular core; the Mill design is big on regularity. That implies that it has its own PLB and TLB, responds to HEYU, and supports the same IPC mechanisms, both those in the talk and those NYF.
Of course, modern peripherals don't look like that, so there will be adaptors. IBM 360 channels and CDC6600 PPs also haven't been architecturally revisited in a while.
The Mill design is fascinating because it is genuinely very different to anything else. But it seems that the entire team might die from old age before they actually have any working silicon produced. Which would be a shame.
No, they are trying to produce silicon. This is their explicit goal. They're just recently caught up in filing a lot of patents because of changes to patent law.
The goal of their first funding round was to create a patent portfolio; the second is earmarked for an FPGA implementation. Hard to do it the other way around, although to be honest FPGA dev kits are so cheap these days I don't know why this requires a funding round.
My understanding is that the cost of the FPGA implementation is about the cost of modifying the tooling that produces a Mill family member from a spec to produce Verilog as described in the 'Specification' talk https://millcomputing.com/docs/specification/
Depends on the size of the chip you are trying to model, once your chip exceeds a certain level of complexity FPGA modeling can get expensive really quickly.
I don't understand this either, they could cut some corners and implement most promising ideas early. I think they delved too deep and it became too complex and alien. I would be surprised if there will be an actual mill cpu as described in the talks. Maybe eventually something that incorporates some of the ideas will be built, but not the whole enchilada.
While I get the sense that this is Ivan's "baby" I'm pretty sure that he hasn't held anything back from the rest of the team and that they could carry on without him. Although his mind is as sharp as ever and his hair is fabulous so I'm not really worried about his health either.
He ends the talk begging for criticism. Listing issues is appreciated-- other comments do so (see talk about capabilities & how this isn't new since the 80s)
Much of the Mill is not new; we don't bother filing those parts. Much that is not new is very good, and was abandoned for reasons unrelated to the actual merits. The first compiler I ever wrote was for the Burroughs B6500, which in 1970 had better security than any current commercial architecture. That compiler is still in use.
Security in architecture is the history of a race to the bottom, driven by newbie customers not knowing there was such a thing and the economics of chip-making. We may hope that there are fewer newbies now. That leaves economics. To a large extent the Mill has been an effort to make old ideas economically viable to today's customers.
Is this comment your less subtle one? Or are you still working on the less subtle criticism? Because I don't see how the analogy of that story fits, who is the seller and who is the target of flattery and deception?
I can't tell if you think the tech sucks, if you think despite all the presentations and demos there somehow is no tech, if you disagree with something about the practicality... what is your actual argument?
He's implying nobody says the mill design is bollocks because they don't want to be viewed as stupid for not seeing the genius in this obviously dysfunctional design
Overall, this seems like one of the weaker Mill talks. Since they apparently don't yet have a real OS running real software in simulation, they probably haven't had the ability to test the ideas that affect how software is structured at a higher level.
They don't provide nearly enough ways to transitively grant permissions. Using the mechanisms discussed in the talk, it doesn't seem like you can implement a simple asynchronous queue of units of work to perform, each having their own permissions. The belt architecture encourages these sorts of second-class mechanisms that have to be used in a rigid way, because the details can be hidden in the belt and not be exposed architecturally.
Unless there's something else not mentioned in the talk, it seems like you still need to trust the OS, because when the OS is asked to allocate a page for a spillet there is nothing stopping it from creating a virtual alias of that page elsewhere and allowing another thread to read its data.
The mechanism to support fork() is a total kludge. Why have a single address space if you're just going to add segmentation in such an ad-hoc way for a single use case? Just run the original binary in emulation until exec() or something like that.
the fork exposition was weak. admittedly fork() was a mistake and constrains alot of implementations in strange ways. i still dont understand how local/global exactly matches the semantics of cow.
transitive permissions are capabilities.
while i'm sympathetic to the lack of market appeal to a capability based system, doesn't it seem like you could implement posix on top of one by compromising it? fd transfer over unix domain is already halfway there.
They were light on the details, but what stops an OS from mapping the same physical page into two distinct local regions of the address space and implementing copy-on-write as usual?
No, it's aliasing, just as any fork() is. COW lowers the cost, for Mill as for any other. The Mill fork duplicates the address space, not the memory. The memory is duplicated page by page on demand, i.e. copy-on-write. Mill paging is quite conventional; it's the address space (SAS) that is different.
The biggest problem with caps and legacy apps is not the semantics - a caps systems can emulate POSIX with no problem. The problem is the data representation: you can't fit a capability into a pointer, so all the data layouts change. Sadly, there are tons of C programs that make rash assumptions about data layout, and they would all break.
The guys are Cambridge have running caps systems that store the extra info in outboard data structures. We judge that the overhead is too great for commercial success. Customers buy benchmarks, and there are no security benchmarks.
They've repeatedly said they want to run current software well, presumably including software that forks and does not follow up with exec (regardless of how ill-advised that may be; obviously opinions vary on this). Trapping to emulation on fork would seem to fly in the face of this.
That being said I'm still not sure how you can get all of fork's semantics out of this mechanism...
(For that matter various sneaky VM aliasing tricks ["magic" circular buffers mapped twice in a row so that any size and position block is contiguous, same file mapped in multiple processes with different (non-contiguous) memory mappings] are going to fail miserably with the virtual-addressed cache. It may well be worth not being able to do those things for the benefit of moving the TLBs, but it also flies in the face of "will run current software.")
> "magic" circular buffers mapped twice in a row so that any size and position block is contiguous
Yes this exact case is one of those that won't work. There are a handful of other things that don't work, like people who roll their own coroutines using assembler. We promise 'don't rewrite, just recompile' but your codebase has to be 64-bit clean and not use any assembly language for any other ISA and not make other unportable assumptions about the hardware.
I can't think of any software that actually uses these circular buffers that wouldn't be worth adapting to the Mill rather than adapting the Mill to them; can you? ;)
How do you plan on supporting user/process level coroutines and multiple stacks / stack switching within the same turf? Obviously the CPU needs to be involved because you've got all the ancillary belt and spiller state that needs to be switched as well.
For that matter have you guys touched on how stack unwinding is going to be handled yet? How do I implement exception handling, longjmp, CL's non-local GOTO, etc?
The next talk will be on threading, and will address all this. The IPC and threads talks belong together, but are too big to combine in one talk unfortunately.
> They don't provide nearly enough ways to transitively grant permissions.
Portals are synchronous and transient permissions can only be used for the duration of the call and by the same thread. Asynchronous isn't so 'simple' because its about lifetime. With the synchronous portal the caller knows that the callee cannot retain any access to the buffers that were passed, and can reuse them safely. If those buffers were put on an asynchronous queue, when would the memory be safely reused and when would the owner know that? If you want asynchronous queues, you either have to have a buffered model like Unix pipes etc or you have to have a global hardware-implemented GC that somehow spans turfs and becomes part of the Trusted Computing Base (TCB) (shudder).
> it seems like you still need to trust the OS, because when the OS is asked to allocate a page for a spillet
Well there's plenty not mentioned in the talk and you touch on one aspect :) When a spillet overflows the extension cannot be in the reserved space, so space must be 'carved out' of the part of the address space where programs also have their needs carved out. Someone has to do the carving, whether its for spillets or for programs, and that someone has to be trusted. The problem isn't aliasing (we're Single Address Space), its that they can simply give as many permissions to it to as many turfs as they choose. There is a possibility that the carver is in the BIOS, but any which way there has to be a turf that can do this. This turf is obviously part of the TCB.
> Why have a single address space if you're just going to add segmentation in such an ad-hoc way for a single use case?
Unfortunately there isn't much market for general purpose CPUs if they are fundamentally unable to run Linux ;)
As you will almost certainly be running a Unix, and as you almost certainly will be using libraries that may fork, then all your normal heap and data stack pointers are going to be Local. Shared mmap pointers will be Global, as will your code. We can hope there is a flag this Linux lets you set that says that you forego the ability to fork() and in return all your pointers can be Global, because the Local puts some constraints on your address space use which become clearer if I explain them:
The local bit works like this:
If a pointer has the bit set, then before use it is mangled with a special register called the Local Space register. The hardware does this every time it uses a pointer. Each turf has its own Local Space value, and you can think of it as a simple offset, so if the address is 0x1...10 and the local space is 2 then the effective address in the global space is 0x1...100.
Now 64-bit adds are relatively slow for this use-case because we really want the effective address asap in the FU so a probable implementation of the Local Space is XORing it into the pointer instead of adding.
When a process forks the OS has to find a position in the global address space where the allocated ranges used by the program are free for the child and then set the new Local Space register for the child appropriately.
> Just run the original binary in emulation until exec() or something like that.
All modern OS don't actually COW until there's a page fault, and I'd expect them to use that trick on the Mill too. So the local bit makes it possible to fork(), but the hole-searching is lazy and only happens if you actually use it.
Hi, capability-security folk here. Have you considered adopting capability-security language for describing what's going on here? It seems to me that Mill portals are very much like capability-designed syscalls. In particular, I am reminded of the "no stale stack frame" rule from Monte [0], and its interesting justification:
"Since Monte permits mutable state, one author’s code’s behavior could be affected by another author’s code running further up the frame stack. Stale frames make comprehension of code much harder as a result."
I wish that we were having more conversations and classes like [1] to get knowledge of KeyKOS and other capability-safe kernels out to CPU makers. We have ideas to share!
Ah well Norm Hardy (KeyKOS) is on team Mill; the Mill is very much a bunch of caps vets :)
The Mill isn't a Capability-based Addressing machine, but it is very much a machine that hopes to push normal programs and operating systems in the Capability Object Model direction with first-class support.
Now you are using a bugmenot login so I have no idea who you are, but if you're a caps vet we'd love to know your thoughts :)
The Mill grant-based model is semantically quite similar to capabilities, but it associates protection with the accessor (thread/turf) rather than the access (pointer/capability). This lets us preserve the size of a pointer, which no one knows how to do efficiently with capabilities.
The difference between the two models is visible when you pass a graph structure across a protection boundary. With caps is is easy to pass the whole graph, and hard to pass only one node. With grants it is vice versa.
> The mechanism to support fork() is a total kludge.
Fork is a kludge. It also happened to be easy to implement on the hardware available at the time and we've been stuck with it ever since. So I happily forgive the mill team that their fork() implementation looks like it is a total kludge, it would be highly surprising if it were not.
>Unless there's something else not mentioned in the talk, it seems like you still need to trust the OS, because when the OS is asked to allocate a page for a spillet there is nothing stopping it from creating a virtual alias of that page elsewhere and allowing another thread to read its data.
The Mill is hardware and architecture, not policy. If you want to use such an OS then you are free to do so. The Mill is designed to efficiently support micro-kernel OSs. Note: micro-kernel, not no-kernel. There always will be a Resource Service that owns the machine. It will be a couple hundred LOC, small enough to be correct by eyeball or proof. Contrast your choice of monolith.
The OS is not involved in allocating spillets. Spillet space is a large statically-allocated matrix in the address space. It is not allocated in memory, only in the address space. As soon as you allocate a turf id and a thread id you have implicitly allocated a spillet. Only on spillet overflow is allocation necessary. Whether allocating turf or thread ids requires OS involvement depends on the policies and models chosen by the OS designer.
When first created the spillet data lives only in backless cache - no memory is allocated. Only if the spillet lives long enough to get evicted from cache is actual memory allocated, using the Backless Memory mechanism described in our Memory talk. The root spillets of apps will live that long; transient spillets from portal calls will likely live only in cache. Consequently truly secire IPC/RPC using Mill portals has overhead, both app and system combined, of the same magnitude of an ordinary function call.
> They don't provide nearly enough ways to transitively grant permissions. Using the mechanisms discussed in the talk, it doesn't seem like you can implement a simple asynchronous queue of units of work to perform, each having their own permissions.
There is a "session" notion that addresses such things. Unfortunately the talks are far enough into details that they must contain background and introduction slides for the viewers who have not already done (and retailed) all the other talks. This limits the amount of new material that can be covered in a single talk, and sessions didn't make the cut this time. We'll get to them.
> The mechanism to support fork() is a total kludge.
Agreed; there seems to be a Law of Conservation of Kludgery. We had as a minimum requirement that the architecture must support Unix shell. The only real problem is fork(). Would that we could issue an edict banning it.
I wonder how the PLB can be fast. You have a dictionary from byte range to permission. This is harder than TLBs, which map a relatively large granule where you can form a search key by just extracting the top bits from the virtual address.
Intel MPX has a similar protection model, and that introduces a lot of overhead (of course it is bolted onto an existing arch and it wasn't a high priority feature).
The protection entries have ranges. The bounds are in bytes, but the range can be massive.
Imagine you load a 7MP image which takes, say, 21MB of RAM. That would be 5184 4K pages in a classic TLB. In the Mill's PLB, that whole part of the address space can be in a single protection entry.
Then, there's a big difference from how things can be organised in software vs hardware. The hardware PLB has some number of entries, and it will check all those entries in parallel.
Yes, you will have a CAM, but the CAM for a PLB would likely be more expensive than a TLB, right? And today's TLBs are only a few hundred entries for the first level. The PLB would seem to necessitate many cycles for a load, longer cycle times, or a high power CAM. Even if you could make address queries quickly by replicating ranges based on some graularity, invalidating the PLB becomes expensive on a grant revocation.
I understand that Itanium was forced into a low frequency, high power L1 cache by being at a similar in order statically scheduled design point, where you need loads to be fast in order for a compiler to be able to come up with a reasonable static schedule. Unless there's some really nice idea out there that's radically different from TLBs as they are today, I bet PLBs will be the major limiter of performance for general purpose code.
Each protection entry is has a lower and upper bound. The entry can cover something as small as a single byte, or as big as the whole address space or anything inbetween.
It is just a normal bounds compare to see if each entry covers the access, and the PLB has as long as the top-level cache access takes to do the checks.
So the PLB misses far far less often than a conventional TLB.
TLBs index a tag with the higher bits and compare one retrieved value for equality only. A PLB with arbitrary resolution would need to do 2 subtractions for the 2 compares and do that for all active entries in parallel. That is, for every simple load/store in a 16 entry PLB you'd need to do 32 subtractions! Unless you come up with a novel scheme to handle this your chip will get hot. Maybe you can reconstruct the indexing TLB scheme on the fly in hardware or something like that, or have an special purpose embedded Cortex-M3 that recreates an efficient lookup structure for the MILL :-). And no, a 64 bit subtraction is not one clock cycle when you run at 1ghz.
There are dededicated Well-Known-Regions for code, stack, globals and TLS that catch the great majority of memory references. The PLB is only consulted when we miss in the WKRs. What's not in the WKRs? MAP_SHARED mmap()s. How many of those are in your program? How often do you access one for the first time or after long enough that the entry has been evicted from the PLB?
Like any cache, the optimal PLB size is determined by the working set. In the typical code we are seeing, the program has a couple of open files, half a dozen mmaps where the heap grew itself, and portal blocks for assorted libraries. The working sets are much smaller than a conventional TLB, and with SAS we have several cycles available in parallel with the caches.
The upshot is that a PLB can be large, cool, and slow. As for the range compares, the PLB permits the same sore of address sub-setting as is done in mixed size TLBs. Think about how many bits in the typical address range differ between lower and upper bounds.
I work with a lot of OO code (ORM) that regularly contains references on objects to other objects. How would that "security model" behave wrt the map of reachable objects in relation to the object passed... let's assume by reference.
I figure this scenario would be somewhat similar to the problem of "/.." paths in URLs on web servers.
The grant model requires you to grant each object individually that you want to pass. That is annoying if you have many objects. In both the caps and grant models you can cut the overhead by thinking of the whole graph as "the object". A typical approach is to allocate graph nodes in an arena and pass the whole arena.
Fine granularity is expensive, which is why the monoliths have one process-granularity. If you have 100,000 graph nodes and want to pass all of them except this one then you will have to pay for the privilege, in any protection model. The Mill lets you pay less.
Looking around the Mill Computing, Inc website, this feels like an (accidental?) sweat equity scam. I realize that is a very loaded charge, but this is NOT how real companies are run: "In the beginning we were a sweat equity organization; no one received a salary; instead, contributors received units that converted to stock when we incorporated. At incorporation 45 people had worked on the Mill and became shareholders. After incorporation we are still a sweat equity organization; we now use a stock option system for sweat equity, and we still pay no salaries. Reward for work today is comparable to what it was before incorporation."
I was involved in something similar around 10 years ago where we were working on a revolutionary EDA suite for analog and mixed signal circuit design. The owner was quite technically competent but kept upping the ante and theatrics to the point that no customers or suitors took the company seriously, only the desperate employees. They never closed sales nor sold the IP.
I advise extreme caution in dealing with the business side of this.
Caution is always warranted when you aren't getting cash on the barrel. The sweat equity documents are available - ask on the site (and now that I think about it, I suppose we should just put them on the site directly). There's no "owner": we all work on the same deal, me included. As it happens I have the largest chunk of equity. You can call that a scam after you have worked full time for over a decade with no paycheck :-)
And yes, Mill Computing, Inc. is not how real companies are run. Is that a bug or a feature?
On a regular basis Mill pops up here and generates some interest, I just can’t understand why.
They haven’t produced an FPGA proof of concept, after claiming they would have one ready last year. They now say they need investors to finish it, yet they previously claimed to not even be looking for funding.
They claim to have angel investors, but they are all secret ones. Of course it’s an investors right to stay private, but the reason you often see investors and companies shouting from roof tops is because the funding event itself can help a company. Publicizing it generates PR, gives the company credibility in dealing with other companies, and is a signal that can generate demand for more investors.
Even putting that aside, the biggest issue is they haven’t made a compelling case for how their ideas will outperform existing CPUs in practical usage scenarios. Yes a running FPGA would be nice, that’s not the only way to show potential.
They could do quantitative analysis, modeling, or start adding a lot more detail to their talks and papers (which tend to sound about as deep as you get in an undergrad architecture classroom), and argue very specifically and comparatively against today’s standards, even for just a few key scenarios.
Maybe they believe even those ways would still have capital/labor/opportunity costs that are prohibitive or for a startup? Another option could be small meetings with a few well respected hardware architects, who will have the best chance of understanding the potential value. Once convinced, they will probably be glad to write about it or just provide a reference, which will make funding, partnerships, hiring, etc all easier.
I dislike being critical of people swinging for the fences, because it’s what many of us here are trying to do, and it’s important that people keep doing it. However in this case it’s not just about long odds. Because of the reasons above and a few other details, things just don’t add up. I don’t believe the FPGA will ever demonstrate anything compelling, and don’t think any investor in their own backyard on sand hill road will bite.
It’s all conjecture of course, I’d be happy to be proven wrong.
You brought up the question of why people are interested in the mill and then discussed the question of whether the mill is viable. They're not completely unrelated but still distinct.
Being a software guy I can't say much about the viability. Watching Ivan's lectures and thinking it over however tickles the same part of my brain that enjoys learning a new programming language. It is just fun to see how some problem could be solved differently.
Most of what you'd like to see are things we'd like to see too. At the beginning we decided to bootstrap rather than follow the usual funding model, at least to the point at which we could demonstrate what we had to people who would understand it in detail. We choose bootstrap in large part because most of us were old enough to have had actual experience with other business models. Yes, it has taken far longer to get this far than we wanted, but we have gotten this far.
About evaluation: it has been our experience that the more senior/skilled a hardware (and software) guy is the more they fall in love with the Mill. You don't hear much of that - we want the tech to be judged on its merits, not on some luminariy's say-so. And of course those senior guys tend to work for potential competitors and don't want to say much publicly.
But you are right: the proof will be running code, and we're starting to do that. We'll be doing more talks like the switches talk, with actual code comparisons. Eventually we will put our tool chain and sim on the cloud for you to play with. Patience, waiting is.
Unrelated to protection: Ivan mentioned that this is an SSA-like architecture.
How does the compiler implement PHIs connecting expressions with different latencies? Let's say I have:
`if (cond) { x = a + b; } else { x = a * b; }`
The MUL is may take a bit longer than the ADD, but the user needs to accept the argument at a given belt position. How do you avoid having the pay the latency cost for the MUL if `cond` is usually true?
Based on my limited understanding of the Mill I'd say (1) (simple_ PHIs are implemented by the pick-phase, (2) it's an exposed pipeline with known latencies, so you can't really have the "I want to avoid the longer latency inside a single basic block", and it's probably not worth doing that for a single mult vs. a single add, (3) for cases where it matters because the latency difference is much better, you'd use two basic blocks, and rearrange belt positions at the end.
If I remember it right, ADD and MUL take the same time, but when instructions differ in time the compiler is expected to reorder them. If it can't reorder, than yes, it has to wait.
The tool chain does hoisting and if-conversion with wild abandon. That code becomes {x = cond ? a+b : a*b}, and both expressions are evaluated in parallel. The conversion is a heuristic; if you have tracing data for the branch then it might not convert. However, a miss-predict is a lot more expensive than a multiply so the tracing has to be pretty skewed to be worth the branch.
The conversion does increase the latency of getting the value of x. If there's nothing else to do then the tool chain will insert explicit nops to wait for the expression. The same stalls will exist on other architectures for the same code, just not visibly in the code. It happens that making the nops explicit is faster than a stall; you can idle through a nop with no added overhead, but you can't restart a stall instantaneously.
Just wanted to say, it's always a real treat to watch Mill talks, and I thank Ivan for putting in the hard work of making them so good! (The negativity I see here on HN really disappoints me).
Also, is the thread talk close at hand? I feel I learned less from this talk than usual; most of the material was already discussed in the security talk.
Of all of the Mill subjects, the pointer kludge to support fork (itself a kludge, yes) seems to me to be the biggest offender of the "sufficiently smart compiler" red flag.
I just have a sinking feeling about hoping a compiler can correctly identify and track all pointers to know how to flag them. The "pointer is a native-word-sized int" assumption may be so ingrained -- from compilers to stdlibs to the wide variety and age of programs -- that it will be nigh impossible to rid existing codebases of it completely.
But I'm not a compiler guy (or hardware, or assembly, or C for that matter) so I could be quite mistaken. Perhaps it's enough to fix the compiler and make it capable of emitting warnings/errors when it detects a violation.
As far as the talk itself goes, I'm a little sad that there was so little new information though I understand that we're quite deep in the technical details and there's a lot of prerequisite background that you can't reasonably expect from a random tech audience. If there are more than a few more talks you might need to reevaluate this method altogether and use a different format.
I'm very glad that you've decided to change the wording to refer to it as an "SSA machine" as opposed to "belt". I think many more people are familiar with SSA or can be convinced that it works ("your current compiler uses it right now" probably helps) by describing it as "SSA where you can only reference the last N results" as opposed to building a whole model based on a "conceptual giant shift register" from before. I've been following the Mill talks since the first few videos and recently I wonder if even the asm programming model should be writing raw SSA instead of belt numbers, especially since genasm assumes an infinite belt anyways.
The emperor has no clothes. The guy who claims to have written 12 compilers hasn't turned out one in a decade. How are microarchitectural decisions being driven without a compiler?
78 comments
[ 3.4 ms ] story [ 107 ms ] threadedit: i thought they managed to do all of this without segments, but at the end we hear about a special local segment with offset addressing apparently introduced just to handle children of fork()..i lost how cow can be expressed losslessly as local/global
redit: question around 1:00:00 explicitly ask this question, and he said, erroneously i think that while semantically similar this is the first time a direct hardware implementation of a call gate has been proposed
He also doesn't mention the word 'capability' anywhere either - this is all 1980s stuff
the other thing that struck me as really strange was a bit in the question period where he says that 'smart devices' (DMA?) are different from simple devices (pio?) in that they are expected to be first-class multiprocessing citizens. doesn't that imply that high-performance peripherals need to be specially designed for the Mill? (likely with a Mill core attached)
As for peripherals, I presume that would have to be handled in the drivers.
Of course, modern peripherals don't look like that, so there will be adaptors. IBM 360 channels and CDC6600 PPs also haven't been architecturally revisited in a while.
http://chipdesignmag.com/images/idesign/fpga/figure1.jpg
And such don't come cheap.
Security in architecture is the history of a race to the bottom, driven by newbie customers not knowing there was such a thing and the economics of chip-making. We may hope that there are fewer newbies now. That leaves economics. To a large extent the Mill has been an effort to make old ideas economically viable to today's customers.
I can't tell if you think the tech sucks, if you think despite all the presentations and demos there somehow is no tech, if you disagree with something about the practicality... what is your actual argument?
(not to imply I agree)
They don't provide nearly enough ways to transitively grant permissions. Using the mechanisms discussed in the talk, it doesn't seem like you can implement a simple asynchronous queue of units of work to perform, each having their own permissions. The belt architecture encourages these sorts of second-class mechanisms that have to be used in a rigid way, because the details can be hidden in the belt and not be exposed architecturally.
Unless there's something else not mentioned in the talk, it seems like you still need to trust the OS, because when the OS is asked to allocate a page for a spillet there is nothing stopping it from creating a virtual alias of that page elsewhere and allowing another thread to read its data.
The mechanism to support fork() is a total kludge. Why have a single address space if you're just going to add segmentation in such an ad-hoc way for a single use case? Just run the original binary in emulation until exec() or something like that.
transitive permissions are capabilities.
while i'm sympathetic to the lack of market appeal to a capability based system, doesn't it seem like you could implement posix on top of one by compromising it? fd transfer over unix domain is already halfway there.
seems like a better alternative.
It's not, it's "copy on reference".
The guys are Cambridge have running caps systems that store the extra info in outboard data structures. We judge that the overhead is too great for commercial success. Customers buy benchmarks, and there are no security benchmarks.
'Tis true 'tis, 'tis pity. 'Tis pity 'tis, 'tis true.
That being said I'm still not sure how you can get all of fork's semantics out of this mechanism...
(For that matter various sneaky VM aliasing tricks ["magic" circular buffers mapped twice in a row so that any size and position block is contiguous, same file mapped in multiple processes with different (non-contiguous) memory mappings] are going to fail miserably with the virtual-addressed cache. It may well be worth not being able to do those things for the benefit of moving the TLBs, but it also flies in the face of "will run current software.")
> "magic" circular buffers mapped twice in a row so that any size and position block is contiguous
Yes this exact case is one of those that won't work. There are a handful of other things that don't work, like people who roll their own coroutines using assembler. We promise 'don't rewrite, just recompile' but your codebase has to be 64-bit clean and not use any assembly language for any other ISA and not make other unportable assumptions about the hardware.
I can't think of any software that actually uses these circular buffers that wouldn't be worth adapting to the Mill rather than adapting the Mill to them; can you? ;)
For that matter have you guys touched on how stack unwinding is going to be handled yet? How do I implement exception handling, longjmp, CL's non-local GOTO, etc?
> They don't provide nearly enough ways to transitively grant permissions.
Portals are synchronous and transient permissions can only be used for the duration of the call and by the same thread. Asynchronous isn't so 'simple' because its about lifetime. With the synchronous portal the caller knows that the callee cannot retain any access to the buffers that were passed, and can reuse them safely. If those buffers were put on an asynchronous queue, when would the memory be safely reused and when would the owner know that? If you want asynchronous queues, you either have to have a buffered model like Unix pipes etc or you have to have a global hardware-implemented GC that somehow spans turfs and becomes part of the Trusted Computing Base (TCB) (shudder).
> it seems like you still need to trust the OS, because when the OS is asked to allocate a page for a spillet
Well there's plenty not mentioned in the talk and you touch on one aspect :) When a spillet overflows the extension cannot be in the reserved space, so space must be 'carved out' of the part of the address space where programs also have their needs carved out. Someone has to do the carving, whether its for spillets or for programs, and that someone has to be trusted. The problem isn't aliasing (we're Single Address Space), its that they can simply give as many permissions to it to as many turfs as they choose. There is a possibility that the carver is in the BIOS, but any which way there has to be a turf that can do this. This turf is obviously part of the TCB.
> Why have a single address space if you're just going to add segmentation in such an ad-hoc way for a single use case?
Unfortunately there isn't much market for general purpose CPUs if they are fundamentally unable to run Linux ;)
As you will almost certainly be running a Unix, and as you almost certainly will be using libraries that may fork, then all your normal heap and data stack pointers are going to be Local. Shared mmap pointers will be Global, as will your code. We can hope there is a flag this Linux lets you set that says that you forego the ability to fork() and in return all your pointers can be Global, because the Local puts some constraints on your address space use which become clearer if I explain them:
The local bit works like this:
If a pointer has the bit set, then before use it is mangled with a special register called the Local Space register. The hardware does this every time it uses a pointer. Each turf has its own Local Space value, and you can think of it as a simple offset, so if the address is 0x1...10 and the local space is 2 then the effective address in the global space is 0x1...100.
Now 64-bit adds are relatively slow for this use-case because we really want the effective address asap in the FU so a probable implementation of the Local Space is XORing it into the pointer instead of adding.
When a process forks the OS has to find a position in the global address space where the allocated ranges used by the program are free for the child and then set the new Local Space register for the child appropriately.
> Just run the original binary in emulation until exec() or something like that.
All modern OS don't actually COW until there's a page fault, and I'd expect them to use that trick on the Mill too. So the local bit makes it possible to fork(), but the hole-searching is lazy and only happens if you actually use it.
"Since Monte permits mutable state, one author’s code’s behavior could be affected by another author’s code running further up the frame stack. Stale frames make comprehension of code much harder as a result."
I wish that we were having more conversations and classes like [1] to get knowledge of KeyKOS and other capability-safe kernels out to CPU makers. We have ideas to share!
[0] http://monte.readthedocs.io/en/latest/semantics.html#scope-i...
[1] https://pdos.csail.mit.edu/6.828/2009/lec/l-microkernel.html
The Mill isn't a Capability-based Addressing machine, but it is very much a machine that hopes to push normal programs and operating systems in the Capability Object Model direction with first-class support.
Now you are using a bugmenot login so I have no idea who you are, but if you're a caps vet we'd love to know your thoughts :)
My thoughts are mostly that I don't want to wait a decade for a Mill. Find me at SPLASH/OCAP if you want to chat.
The difference between the two models is visible when you pass a graph structure across a protection boundary. With caps is is easy to pass the whole graph, and hard to pass only one node. With grants it is vice versa.
Fork is a kludge. It also happened to be easy to implement on the hardware available at the time and we've been stuck with it ever since. So I happily forgive the mill team that their fork() implementation looks like it is a total kludge, it would be highly surprising if it were not.
The Mill is hardware and architecture, not policy. If you want to use such an OS then you are free to do so. The Mill is designed to efficiently support micro-kernel OSs. Note: micro-kernel, not no-kernel. There always will be a Resource Service that owns the machine. It will be a couple hundred LOC, small enough to be correct by eyeball or proof. Contrast your choice of monolith.
The OS is not involved in allocating spillets. Spillet space is a large statically-allocated matrix in the address space. It is not allocated in memory, only in the address space. As soon as you allocate a turf id and a thread id you have implicitly allocated a spillet. Only on spillet overflow is allocation necessary. Whether allocating turf or thread ids requires OS involvement depends on the policies and models chosen by the OS designer.
When first created the spillet data lives only in backless cache - no memory is allocated. Only if the spillet lives long enough to get evicted from cache is actual memory allocated, using the Backless Memory mechanism described in our Memory talk. The root spillets of apps will live that long; transient spillets from portal calls will likely live only in cache. Consequently truly secire IPC/RPC using Mill portals has overhead, both app and system combined, of the same magnitude of an ordinary function call.
> They don't provide nearly enough ways to transitively grant permissions. Using the mechanisms discussed in the talk, it doesn't seem like you can implement a simple asynchronous queue of units of work to perform, each having their own permissions.
There is a "session" notion that addresses such things. Unfortunately the talks are far enough into details that they must contain background and introduction slides for the viewers who have not already done (and retailed) all the other talks. This limits the amount of new material that can be covered in a single talk, and sessions didn't make the cut this time. We'll get to them.
> The mechanism to support fork() is a total kludge.
Agreed; there seems to be a Law of Conservation of Kludgery. We had as a minimum requirement that the architecture must support Unix shell. The only real problem is fork(). Would that we could issue an edict banning it.
Intel MPX has a similar protection model, and that introduces a lot of overhead (of course it is bolted onto an existing arch and it wasn't a high priority feature).
Imagine you load a 7MP image which takes, say, 21MB of RAM. That would be 5184 4K pages in a classic TLB. In the Mill's PLB, that whole part of the address space can be in a single protection entry.
Then, there's a big difference from how things can be organised in software vs hardware. The hardware PLB has some number of entries, and it will check all those entries in parallel.
I understand that Itanium was forced into a low frequency, high power L1 cache by being at a similar in order statically scheduled design point, where you need loads to be fast in order for a compiler to be able to come up with a reasonable static schedule. Unless there's some really nice idea out there that's radically different from TLBs as they are today, I bet PLBs will be the major limiter of performance for general purpose code.
It is just a normal bounds compare to see if each entry covers the access, and the PLB has as long as the top-level cache access takes to do the checks.
So the PLB misses far far less often than a conventional TLB.
Like any cache, the optimal PLB size is determined by the working set. In the typical code we are seeing, the program has a couple of open files, half a dozen mmaps where the heap grew itself, and portal blocks for assorted libraries. The working sets are much smaller than a conventional TLB, and with SAS we have several cycles available in parallel with the caches.
The upshot is that a PLB can be large, cool, and slow. As for the range compares, the PLB permits the same sore of address sub-setting as is done in mixed size TLBs. Think about how many bits in the typical address range differ between lower and upper bounds.
Fine granularity is expensive, which is why the monoliths have one process-granularity. If you have 100,000 graph nodes and want to pass all of them except this one then you will have to pay for the privilege, in any protection model. The Mill lets you pay less.
I was involved in something similar around 10 years ago where we were working on a revolutionary EDA suite for analog and mixed signal circuit design. The owner was quite technically competent but kept upping the ante and theatrics to the point that no customers or suitors took the company seriously, only the desperate employees. They never closed sales nor sold the IP.
I advise extreme caution in dealing with the business side of this.
And yes, Mill Computing, Inc. is not how real companies are run. Is that a bug or a feature?
They haven’t produced an FPGA proof of concept, after claiming they would have one ready last year. They now say they need investors to finish it, yet they previously claimed to not even be looking for funding.
They claim to have angel investors, but they are all secret ones. Of course it’s an investors right to stay private, but the reason you often see investors and companies shouting from roof tops is because the funding event itself can help a company. Publicizing it generates PR, gives the company credibility in dealing with other companies, and is a signal that can generate demand for more investors.
Even putting that aside, the biggest issue is they haven’t made a compelling case for how their ideas will outperform existing CPUs in practical usage scenarios. Yes a running FPGA would be nice, that’s not the only way to show potential.
They could do quantitative analysis, modeling, or start adding a lot more detail to their talks and papers (which tend to sound about as deep as you get in an undergrad architecture classroom), and argue very specifically and comparatively against today’s standards, even for just a few key scenarios.
Maybe they believe even those ways would still have capital/labor/opportunity costs that are prohibitive or for a startup? Another option could be small meetings with a few well respected hardware architects, who will have the best chance of understanding the potential value. Once convinced, they will probably be glad to write about it or just provide a reference, which will make funding, partnerships, hiring, etc all easier.
I dislike being critical of people swinging for the fences, because it’s what many of us here are trying to do, and it’s important that people keep doing it. However in this case it’s not just about long odds. Because of the reasons above and a few other details, things just don’t add up. I don’t believe the FPGA will ever demonstrate anything compelling, and don’t think any investor in their own backyard on sand hill road will bite.
It’s all conjecture of course, I’d be happy to be proven wrong.
Being a software guy I can't say much about the viability. Watching Ivan's lectures and thinking it over however tickles the same part of my brain that enjoys learning a new programming language. It is just fun to see how some problem could be solved differently.
Most of what you'd like to see are things we'd like to see too. At the beginning we decided to bootstrap rather than follow the usual funding model, at least to the point at which we could demonstrate what we had to people who would understand it in detail. We choose bootstrap in large part because most of us were old enough to have had actual experience with other business models. Yes, it has taken far longer to get this far than we wanted, but we have gotten this far.
About evaluation: it has been our experience that the more senior/skilled a hardware (and software) guy is the more they fall in love with the Mill. You don't hear much of that - we want the tech to be judged on its merits, not on some luminariy's say-so. And of course those senior guys tend to work for potential competitors and don't want to say much publicly.
But you are right: the proof will be running code, and we're starting to do that. We'll be doing more talks like the switches talk, with actual code comparisons. Eventually we will put our tool chain and sim on the cloud for you to play with. Patience, waiting is.
How does the compiler implement PHIs connecting expressions with different latencies? Let's say I have:
`if (cond) { x = a + b; } else { x = a * b; }`
The MUL is may take a bit longer than the ADD, but the user needs to accept the argument at a given belt position. How do you avoid having the pay the latency cost for the MUL if `cond` is usually true?
Based on my limited understanding of the Mill I'd say (1) (simple_ PHIs are implemented by the pick-phase, (2) it's an exposed pipeline with known latencies, so you can't really have the "I want to avoid the longer latency inside a single basic block", and it's probably not worth doing that for a single mult vs. a single add, (3) for cases where it matters because the latency difference is much better, you'd use two basic blocks, and rearrange belt positions at the end.
If I remember it right, ADD and MUL take the same time, but when instructions differ in time the compiler is expected to reorder them. If it can't reorder, than yes, it has to wait.
The conversion does increase the latency of getting the value of x. If there's nothing else to do then the tool chain will insert explicit nops to wait for the expression. The same stalls will exist on other architectures for the same code, just not visibly in the code. It happens that making the nops explicit is faster than a stall; you can idle through a nop with no added overhead, but you can't restart a stall instantaneously.
Also, is the thread talk close at hand? I feel I learned less from this talk than usual; most of the material was already discussed in the security talk.
I just have a sinking feeling about hoping a compiler can correctly identify and track all pointers to know how to flag them. The "pointer is a native-word-sized int" assumption may be so ingrained -- from compilers to stdlibs to the wide variety and age of programs -- that it will be nigh impossible to rid existing codebases of it completely.
But I'm not a compiler guy (or hardware, or assembly, or C for that matter) so I could be quite mistaken. Perhaps it's enough to fix the compiler and make it capable of emitting warnings/errors when it detects a violation.
As far as the talk itself goes, I'm a little sad that there was so little new information though I understand that we're quite deep in the technical details and there's a lot of prerequisite background that you can't reasonably expect from a random tech audience. If there are more than a few more talks you might need to reevaluate this method altogether and use a different format.
I'm very glad that you've decided to change the wording to refer to it as an "SSA machine" as opposed to "belt". I think many more people are familiar with SSA or can be convinced that it works ("your current compiler uses it right now" probably helps) by describing it as "SSA where you can only reference the last N results" as opposed to building a whole model based on a "conceptual giant shift register" from before. I've been following the Mill talks since the first few videos and recently I wonder if even the asm programming model should be writing raw SSA instead of belt numbers, especially since genasm assumes an infinite belt anyways.