This is an excellent write up. I've often wondered why Linus never moved towards a microkernel. I read all of these same debates, and they always seemed short sighted.
Given the recent trend towards containers, it's obvious that people can now see how much better a model this is for micro-services.
It's just a matter of time before we start seeing more microkernels to run things like unikernels and such. Docker is just a stepping stone.
Unikernels don't run on microkernels, they run on hypervisors, AKA exokernels.
The difference is that microkernels move abstractions like the file system into server processes that still have to be trusted, but are at least isolated; exokernels and hypervisors remove the abstractions from the trusted code base entirely, and just multiplex the hardware directly. Unikernels thus simplify the typical virtualization stack so it looks more like MIT's exokernel research projects, with abstractions in untrusted libraries.
I would like to read the article, but my eyes do not appreciate white text and black BG web pages. It is incredibly jarring and almost painful once you go "back" to the "normal" web. I will stick with the comments for now I guess.
In Chrome: right-click something on the page, click "Inspect element," click the link to style.css that shows up, select all and backspace. You actually get some fairly nice colors.
(This comment is not intended as an opinion in either direction on whether the original color scheme is reasonable or whether you doing these steps is reasonable.)
The problem is things like that can't be done on my iPad. In such cases I usually rely on reader mode but for whatever reason it's not available for the page.
White-on-black text isn't normally that bad, I think it's the choice of thin font that makes it especially hard to read in this article.
I've been planning to implement an alternate theme but haven't done it yet. Black on white is particularly irritating for people with astigmatism which appears to be a pretty big part of the population.
People with astigmatism (aproximately 50% of the population) find it harder to read white text on black than black text on white. Part of this has to do with light levels: with a bright display (white background) the iris closes a bit more, decreasing the effect of the “deformed” lens; with a dark display (black background) the iris opens to receive more light and the deformation of the lens creates a much fuzzier focus at the eye.
I believe it is in fact correct that, at least as far as the market was concerned, one of the big reasons for the failure of microkernels in the mid-'90s was RPC overhead.
But it's no longer the mid-'90s, and everyone's been working on super efficient RPC (Cap'n Proto, msgpack, BSON, etc.) for unrelated problems. I have been suspecting for a while that a brand-new microkernel based firmly on mid-'10s technology (and targeting mid-'10s processors) would be perfectly performant.
You're right that early microkernels (Mach) were pretty awfully slow. But that was really more the early 90's. Even before 2000 there were new systems that were already much, much faster than Mach.
If you read the article, you'll see several references to papers that show microkernel performance on par with or better than traditional monolithic Unix systems. For example, "EROS: a fast capability system" by Shapiro, Smith, and Farber, in SOSP 1999. http://dl.acm.org/citation.cfm?doid=319151.319163
I don't have a lot of experience with microkernels, but I would imagine the objections due to communication are not a result of serialization of the communications; they would tend to be C structs, more or less, not a terrible XML document, creating them wouldn't be a significant performance issue, although actually doing the IPC may be a performance problem (or may be perceived to be).
I think the market issue for microkernels is that
a) there aren't a lot of choices for microkernels that are 'production ready', today I could reasonably run a website on Windows NT, or OS X, but the Hurd is perpetually not ready.
b) Monolithic kernels are 'good enough'. The benefits microkernels promise don't really make a huge impact (or aren't perceived to). Yes, it would be great if my filesystem driver didn't crash the whole system when it crashes, but if it crashes in a microkernel, I expect I would still have to restart all the processes that had open files, because the state is lost; and either way, I'd rather not have my filesystem driver crash. Most users of monolithic kernels don't spend a lot of time developing or debugging kernel drivers, so developer efficiency there doesn't really pay off. Most of the problems I track down to a kernel issue are things that could be equally present in either style kernel: bugs/inefficiencies in the tcp stack, stupid disk controller resource leaks, etc. Maybe it would be easier to track them down if the tcp process was clearly spinning on all the cpu, or the disk controller process had a ton of ram/resources assigned to it, but it's not that hard to track things down anyway -- figuring out why they're broken is much harder than figuring out where they're broken.
a) there are microkernel-based operating systems that are available right now. QNX and Green Hills, for example. There aren't any free ones that I know about, so the cost might be hindering adoption.
b) agree with you on this point that monolithic is 'good enough' these days. There are specific benefits to my particular application, but for a general desktop user I don't see the point. Even on a server the benefits are not very visible.
I suspect one of the "toy" operating systems being written in Rust (or some other safe/modern systems language) will end up being the Linux of 20 years from now. So, it seems relevant to see how this particular holy war plays out (once again). Linus nixed the idea of a microkernel for the dominant operating system for building the web while it was still a toy...I don't know that he was wrong to do so, but it might be interesting to try a different path for the next generation.
Another neat idea would be to build on an existing microkernel like seL4, but write the userspace -- all the drivers, memory manager, filesystems, etc -- in Rust. The NICTA team already has proofs of correctness for the microkernel itself, so why reinvent the wheel? But at the same time, those proofs were super expensive and difficult to make. It would be awesome if we could get a reliable userland more easily by using a better language.
I wonder what it would take to get Rust compiling for seL4?
Not much, really. The main thing needed is to port the language runtime to seL4. This should be a relatively small effort, and a great project to both learn some of the underbelly of Rust's ABI and seL4.
You can already use Rust on seL4, today! It's not ready for public consumption yet since there are no docs, but I've been working on it a lot recently and hope to have a release of the foundational libraries sometime in the next two weeks.
I have some ideas about encoding RPC protocols as DFAs in Rust's type system, and then doing verification of the protocols in a more abstract setting (perhaps NuSMV or a similar checker). This is somewhat similar to the (ancient at this point) "protocol compiler" that used to be in Rust, doing a similar thing for channels.
I'm interning at Data61 (née NICTA) on the verification team starting Jan 18, and I'm hoping to transfer some of that experience and knowledge into verifying Rust.
It has crossed my mind to wonder if one could obtain many benefits of microkernels while somehow leaning on something like the Rust type system to prove isolation. It seems to me one of the big practical things about microkernels is forcing you to isolate off the modules. In practice, when you do not force isolation, developers will always find reasons to violate the boundaries. Even very well meaning and very experienced ones will still tend to poke holes in, and this is one of those things where once you poke even tiny holes, the flood tends to come in. Thirty years ago, that level of isolation required OS-level enforcement if it was going to happen.
But today, that's not the only way to guarantee isolation. We may also be able to do it via type systems. If we could create things statically guaranteed to be isolated but not requiring actual context switches, it should be theoretically possible to create things that are structurally microkernels, but do not need internal context switches because isolation has been proved at a higher level. Or, arguably, in some sense this would be something that transcends both microkernels and monolithic kernels to become something not properly called either.
I'm sure there's many an i to be dotted and ts to be crossed before this would work, and you'd still probably want to work out a run-time loadable module which would probably make things interesting, you need a story for how a device driver can individually crash without bringing the whole process down (though since Linux can do it, presumably it is possible anyhow), and a whole bunch of things. But it does seem like an interesting opportunity for someone to make a kernel that is, well, not the same basic kernel the C type system naturally affords.
It's basically relying on the type system & safety mechanisms of the underlying programming language to enforce some of the isolation properties that an operating system is normally used for. The Lisp Machine was probably the most commercially successful example. Since neokernels typically require tight integration between the programming language and the kernel, there are also opportunities for efficiency gains, eg. in Lisp Machines the garbage collector was tightly integrated with virtual memory to prevent page thrashing on collection, or in modern languages you could replace the OS scheduler with promises or green threads.
The biggest downside of it is that it requires tight coupling between the programming language and OS, such that you can only use a single programming language (or a family of related ones that all share a type system) for all programs on the box. Few programming languages are flexible enough for that. With the move to cloud computing and virtualization, this becomes more practical, but then that same trend is also driving interest in the unikernel/exokernel pairing. If you only run a single application on a virtualized box, why not program directly to the hardware?
> The biggest downside of it is that it requires tight coupling between the programming language and OS, such that you can only use a single programming language (or a family of related ones that all share a type system) for all programs on the box.
I would point out (from a relatively lay perspective) that while this isn't strictly the case in current popular kernels (Mach/OSX, Windows, Linux), it's still a pretty strong trend. These kernels (AFAIK) are implemented in C and have large ecosystems of C code orbiting them. As such there are many benefits to having a program written in C or at least speaking C's ABI and memory layout.
Maybe I'm mistaking the causes for this situation, but most developers are effectively using something that is either C itself, written in C, or speaks C in some way (CPython, JVM, MRI/CRuby, Perl, are notables that are written in C as far as I know). I think that if a language is sufficiently high performance and general purpose, it's fine for it to be the "core" language that other languages are expected to interface with.
Apparently a lot of telephone switches used neokernels. Ericsson's AXE series telephone switches ran software written in their proprietary PLEX language, and the PLEX runtime and the AXE OS were basically the same thing. That helped reduce overhead for context switches between tasks: hardware address spaces didn't need to be switched, or even supported; the runtime handled task separation.
Erlang was meant to fill this role in newer switches, but it looks like PLEX is still being developed and used.
Thank you for the term to go Google on. I had not heard it before.
It seems to me that recent advances in this field ought to ease the restrictions of requiring the entire stack to be the same language. It seems like a Rust neokernel ought to be able to run other programs on top much like a conventional kernel does, context switches and all. I'm not too worried about userspace/kernel context switches... there's a security barrier there that still strikes me as a good idea. I'm OK with something that produces an artifact in the end that looks like a monolithic kernel, and can function like one, and is merely internally structured with all the discipline of a microkernel.
If all kernel elements have to be Rust, well, I'm fine with that.
Microkernels seem to be solving all the right problems (namely, abstraction and safety) in all the wrong ways.
When you realize that kernel-mode vs userland is a social construct and not inherent, you see that the microkernel is just attempting to recover safety from the processor's built-in paging and protection capabilities. The correct solution of course is just to use a memory safe language.
Likewise, the microkernel encourages abstraction due to the pain of RPC boilerplate, but modern languages and sound engineering practices should make this easy anyway.
Even if you trust the language to provide the isolation you need, today's requirements can't possibly be met by such a system. Maybe in 50 years once all software has been rewritten.
That doesn't solve the problem of running someone else's arbitrary pre-compiled binary (whether that be an application or a driver) without letting it crash your computer. Which is kind of the one thing you expect an OS to be able to provide you; otherwise there's no reason not to just let every binary execute directly on ring 0 of the CPU.
(Though, I mean, you can sort of get around this requirement if you only accept "binaries" that are actually AST or high-level bytecode, and then finish compiling them in kernel-space at module-load time. OSX and iOS could get away with doing this soon-ish given that all new binaries submitted to the App Store are submitted as LLVM bitcode. Microsoft could probably get away with it for Metro apps as well.)
The main issue I have with microkernels is that they don't solve any problem that I have.
A robust system is designed such that any machine in it can randomly crash. Crashing machines then only represent a risk that needs to be mitigated by having enough spare capacity to handle however many machines are expected to be down at once. Linux is already stable enough that this amount is too small to care about; making it more stable doesn't help me.
Would I expect meaningful performance improvements from microkernels? I'm doubtful that you'd get as much as 10%, but I'd be interested in seeing evidence if anybody has any.
Do microkernels offer any application-visible features that I don't have today with Linux? To the best of my knowledge they do not.
To be fair they did say a problem that they don't have. If you're Google, Netflix, Facebook, Amazon, etc. 10% is probably pretty compelling. It's probably safe to assume that asuffield doesn't work at a company that operates at that scale.
Erm. I suppose I should mention that I'm an SRE at Google, but I'm very carefully not talking about my current job here (which is special in ways I can't discuss). Interpret my perspective as relating to personal projects and a career of working at assorted medium-to-large companies.
Security. You ever run a process in its own VM for security reasons? A hypervisor is a microkernel with bloaty processes and even more hardware protections.
Who is "we" ? I had the impression unikernels were still a highly experimental work in progress you could only use with ocaml and that noone had ever used in production for anything serious.
It's beyond experiments now and there are clear uses for such things. For example, a recent post describes using MirageOS to build a firewall VM [1]. There are also network security/intrusion-detection systems [2].
There are a number of ways to build unikernels and they're not all OCaml (which is just MirageOS) -- http://unikernel.org is trying to bring these together.
Yes, of course. I should also have linked to the Unikernel+Docker demo [1], which is taking legacy software and using Rump Kernels and the Docker toolchain to deploy unikernels (to show how all these things can create a multiplier effect in terms of benefits).
It would be great to get a list of live unikernel deployments listed somewhere to address questions like this when they come up.
Hypervisors are much closer to exokernels. General distinction between kernel types:
Monolithic kernel = all familiar kernel abstractions (processes, IPC, filesystem, virtual memory, device drivers, networking stack) are built into the kernel.
Microkernel = Most kernel abstractions run as separate processes. Kernel communicates with them over IPC.
Exokernel/unikernel = There are no kernel abstractions. Instead, these abstractions are built into runtime libraries that are then linked into applications. The kernel's only job is to securely multiplex the hardware, and it exports an interface that is similar to bare metal. In 2016, the exokernel is called a hypervisor, and the libOS is called a unikernel.
Neokernel = the programming language's runtime library provides the primary kernel abstractions, along with any interprocess security. Machines are limited to a single programming language.
Most people have experience running processes in isolated VMs for security reasons. I was trying to use that as an anchor point to explain why one might want to have components of a single operating system run as separate servers, for much the same reason -- compartmentalized security.
Rapid development. There's probably things you'd get out of the latest Linux kernel's networking stack or video stack or btrfs driver or something, but your upgrade option is all-or-nothing, and you certainly don't to upgrade all of it. A microkernel could let you upgrade just the component you want to try a newer version of; it also means that there's a larger base of people like you testing the latest versions of each component, so the kernel improves more quickly.
Of course, this presupposes things about API compatibility/stability between microkernel components that may not be true in all microkernel designs.
It sounds like a really cool idea, but to some extent we already have this with Linux out-of-tree modules - it doesn't work very well, but it's there, you can build a new version of the module, unload the old one, and load the new one. I suspect the microkernel version of this would be hard to implement for most of the same reasons, and probably fail in most of the same ways, which boil down to "APIs change a lot so things don't stay compatible for long".
I would expect that for some modules (networking, video, and FS can be loaded as modules) you can just put the new version of the module in a somewhat-earlier version of the kernel.
I mean, if the ABI or internal kernel macros change, you're in for some porting work, but most of the kernel stuff can be configured as loadable modules. :)
Recovering from individual components crashing is a nice property, but in practice we don't use it at the kernel layer. But it does show up in other places -- when designing datacenters and web services (Netflix, Chaos Monkey, horizontally scaled web servers behind a pair of load balancers), and, in web browsers (Google Chrome is a bunch of processes; if Flash takes down one tab/process, the others keep running).
And lack of tooling and lack of OS design. With NT they had actual isolation to allow local crashes, and MS has the static checker to ensure a level of quality to reduce said crashes.
Windows 3.1 had a VMM that could have isolated drivers just fine. But the drivers that people needed to run were DOS drivers that were shoddily written with precisely-controllable (i.e. non-virtualizable) interrupts in mind. "Consumer" Windows couldn't get around the fact that they had to support hardware that came with crap drivers. The reason NT worked so well was that it targeted an extremely reduced machine profile in the form of "workstations", where Microsoft could write all the drivers themselves.
It took Microsoft until XP to actually be able to stop supporting the old crap VXDs and just force all the hardware manufacturers to "write a new driver or else". That they then also instituted the WHQL program to certify those drivers—and later added tooling to make that certifying job more rigorous—was just icing. It was moving away from VXD drivers that solved most of the problem.
So because of social pressure, from 3.1 to Me, Windows had to cave down to VxD model shipped by manufacturers even though they had an adequate layer in place ? that is .. something.
Not social pressure; rather, Microsoft's continued revenue stream depending on an implicit guarantee of upgrades not breaking backward compatibility, even when what "broke" was an invalid assumption about undocumented APIs made by a third-party. To consumers, it all looks like "Windows broke the app."
You get something of the same with the "do not break user space" policy that Torvalds is holding the kernel devs to.
Just wish that the user space libs would adopt the same policy, rather than try to paper over it by stuffing everything into containers (liable to turn into the Linux equivalent of DLL hell).
You have to decide for yourself what is relevant to your application-visible needs. If you deem that Linux is better for running your Java app than QNX, no one wants to take that decision away from you. I don't think that's where the real discussion is. The real discussion involving microkernels is about the future of OS development. Should it just be the same as Linux, forever, with no architectural changes? Have we literally written the last OS?
As a layperson, I can be persuaded microkernels are a good idea that could yield some reliability and security improvement. I can be persuaded that the performance issues are avoidable. But I'm also pretty sure that most of the problems we have with current systems are in code that would now be outside the microkernel anyway. We are going to end up with, optimistically, about the same number of lines of code. So to me it seems much more important what the development process looks like. Unfortunately there is not a lot of accessible research about that kind of thing, relative to things like microkernel performance which are at least easy to generate demonstrative benchmarks for.
There's a different way to look at costs and downtime here in certain rather core applications. Think about your average server here for a moment. It's got a motherboard, a CPU, some RAM, a few network ports, and a hard disk. Now, let's take, for example, running a redis-like node. The job you care about -- looking up key/value pairs -- only touches memory. Any hard disk access is a slowdown to some part of the system, as a slowdown implies that there's been a context switch, a travel through a few layers of virtual filesystem, before being sent to or fetching from a pile of spinning rust. Most of those accesses, you don't care about, at all. Your logfiles are all being sent on the network anyway, your executables are already in memory, and your scheduled tasks haven't changed in a month.
For this sort of task, there's a remarkable amount of dead weight that can be cleared out. For starters, because your binaries are so small, serving them over PXE or the like becomes a triviality in regards to bandwidth. Your system will start faster because it doesn't have to do things like initialize the various pseudo-filesystems Linux has, or initialize the virtual file system, or any number of other tasks. This now means you don't need that hot, failure prone hard disk at all, nor its buffers and journals and other context switches that occur because of it. Because you've gotten rid of the hard disk, you've gotten rid of its heat, meaning you can pack servers more densely.
Similarly, you can eliminate a considerable amount of the networking stack. You know what the messages it's going to send and receive look like, and any other messages are going to be dropped anyways, so why have the code to handle them loaded in memory at all?
Because of their minimalism, microkernels allow you to ask a lot more fundamental questions about your applications. This means you can build a system much more oriented around your needs, with minimal superfluous framework. This now means your apps are faster, because they have to fight less OS bureaucracy, more reliable because they're much more simple, and more secure, because the system as a whole is smaller, and thus easier to audit. Sometimes, asking yourself what you don't need is just as important, if not moreso, than what you do need.
So you save a few milliseconds if you don't initialize some filesystems, good but makes very little difference in the end.
Not having a hard disk? Doesn't require a microkernel at all. Booting over LAN has been practical for a long time.
If I glance the closest linux machine I have, it has a 6MB kernel with 10MB initrd. To put that on a server with gigabytes of ram? Trimming an unused networking stack here is not the use case where a microkernel will help performance.
The "few milliseconds" was a response to "Your system will start faster because it doesn't have to do things like initialize the various pseudo-filesystems Linux has, or initialize the virtual file system, or any number of other tasks." Doing that init takes very little time and zero disk accesses. That and a few megabytes of ram are all you save by using a microkernel instead of linux in this situation.
Linux won't access the hard drive if it's unnecessary, same as the microkernel.
> The main issue I have with microkernels is that they don't solve any problem that I have.
>
> […]
>
> Do microkernels offer any application-visible features that I don't have today with Linux? To the best of my knowledge they do not.
Don't you hate having to upgrade your whole kernel (memory manager, scheduler, fs, network stack) just to fix a small-but-annoying problem with your GPU (for example tearing)?
With a microkernel you could update that driver without updating the rest of the kernel, or of the OS. Yes "OS", because an update of the kernel often requires an update of the corresponding user-space programs (lmsensors, libdrm, mesa, X11, libalsa, iptables). If our systems were built upon microkernels and developed with strict boundaries between components it would be much easier to maintain them.
However I think Torvarlds has a point when it says that going down the path of microkernels means turning many low-level problems into IPC problems, most of which are not yet solved or not solved in a way that is compatible (performance-wise) with current hardware. Let's hope research and the brewing libre hardware community will tackle this problem.
You can already switch/upgrade graphics drivers without rebooting, thanks to dynamically loaded modules. Just stop X, rmmod the old driver, modprobe the new one, and start X again.
> A robust system is designed such that any machine in it can randomly crash.
This assumes that every robust system can consist of multiple machines. One thing I've learned from the article, and other comments in favor of microkernels, is that they're successful in applications where a single machine has to be a robust system all by itself.
For consumer applications, you are limited by the ecosystems, such Android or iOS or Windows, so you can't choose kernel anyway.
For cloud applications, what is more important is the system architecture, not the kernel. If we assume docker is the preferred way to deploy an app. We only need very limited features from kernel:
- If I specify the CPU and RAM requirements, the kernel simply allocates them to the container. There is no need for complicated dynamic scheduling and balance.
- Network within the same data center has much lower latency than hard disk (<<1ms vs 10ms). We will be much better of using network storage or database.
- If we assign network address to each container, the kernel can deliver network packets to container, and the container uses its own CPU and RAM to process the network traffic. This will avoid the situation that kernel spend a lot of CPU/RAM to process network traffic for all containers and find the correct way to charge the cost.
If we do all above, there is not much kernel features are needed by cloud applications, as such features will be replaced by cloud services like cloud storage/database. It will make develop and deploy cloud services easier and more productive.
One of the problems with micro-kernels is that everyone re-invents abstractions over the basic kernel services. Micro-kernels might be useful in some subset of cases, but it is still useful to have an OS layer above the micro-kernel at which point maybe the micro-kernel is not so useful after-all as a general purpose OS mechanism.
In the most recent talk I saw with Tanenbaum regarding Minix and performance, he mentioned a 20% decrease compared to monolithic kernel based systems, which he himself was happy to sacrifice for the greater system stability it offers.
Of course, the big standout for me has always been that if micro-kernel based systems of today are just as, or close enough in terms of performance, why aren't we getting any benchmarks showing this off ?
It would be fantastic promotion to show your system running against Linux or the BSD's and performing as well or near as well in heavy workloads while at the same time offering the great stability of typically being able to restart modules should they crash and thus not take down the whole system.
The micro-kernel vendors should be shouting it from the rooftops, 'here are the benchmarks!', but they don't as far as I can tell.
The last time I saw a modern micro-kernel vs modern monolithic comparison was probably this one way back in 2007, and in this Minix 3 did not do very well against Linux.
Well, most microkernel vendors have niche markets that aren't directly competing against Linux. Minix 3 is a bit of an outlier. Although it is a desktop and "server" OS like Linux, it's really meant to be more of a teaching OS than anything else.
If you want to compare microkernel operating systems, look at QNX, L4, Windows CE, etc. These are used in embedded and RTOS spaces where performance is critical. They seem to hold their own pretty well, and running in systems where Linux would not be a good alternative due to size and porting cost.
In a lot of these cases a small foot print and guaranteed latency is vastly more important than raw speed. As in with a task that runs 20 times a second, it's not important that it finishes in 1 ms or 3 ms, but having the system go away for 500ms is not acceptable.
I guess I should have clarified that "reliable and predictable" performance is critical. In practice though, these systems have fast enough context switching.
I agree with you that "fast enough" is the only metric that matters here.
Every now and then there's this thing that happens where somebody runs across some surface-level comment by some fairly authoritative source, arrives at a mistaken interpretation of it, and that interpretation just happens to be the one that gets widely repeated, and thus an error enters into the public awareness on that topic. That seems to be what happened with the MINIX-and-embedded-systems meme that keeps showing up.
MINIX tried to position itself as a good candidate for use on embedded systems—in addition to its then-current primary (read: only viable) use as a PC operating system. It never really ended up happening. The idea seems to have been to use the grant money as a catalyst to get it in shape so that it looks attractive and can get commercial adoption at the cusp of the move to a post-PC world, and then it will be sufficiently dug-in with a few parties, enough so that it would see sustained development in your typical open source fashion.
The reality is, it didn't really pan out (NB: not necessarily as a result of its essential microkernellyness), they burned through several million Euro that all seems to have gone towards paying developers and putting very little towards project management, and today nobody is doing serious development on it, save for something like one grad student who mostly continues to work on it in the course of his research. In its current state, it has rudimentary support for something like two boards (from the BeagleBoard line), and its best supported platform, relatively speaking, is still PCs, although its support there can itself only really be called so-so by a more absolute metric.
(Caveat: MINIX is BSD licensed, so it's possible there's somebody out there getting lots of mileage out of it and/or doing spectacular things.)
To be fair, even as a teaching OS, though, things do look rough nowadays. In the past, the way things worked was, Tanenbaum would take the sources, clean it up and trim some of the parts that wouldn't be covered in the Operating Systems book, and then ship it. With the current state of the code, after the quality of the changes that occurred in the aforementioned process, if he were to try to do the same for a 4th edition of the book, he's going to have a lot of work ahead of him.
All in all, given the current state of the project, the answer to the question, "What is the best use for MINIX today?" is "For research."
I say all this as someone who has got some patches for cleaning up and consolidating a bit of the bootstrap code for ARM and x86, but with nowhere to effectively send them.
Having worked with a modern microkernel, 20% is a fair number for device drivers that are not doing bulk data transfers, if you both avoid shared memory and are on an platform that lacks hardware features for accelerating such things, though the OS's choices of IPC mechanisms become very important, and any one number is always wrong.
A microkernel offers you the flexibility to choose this though. On linux, I write all my programs to run as processes. If I find I'm getting performance bound by system-call overhead, I can convert part of the program to a kernel module, to eliminate that.
Microkernels extend this concept into the OS. If you find a bottleneck that is truely bogging you down, you can fix it. After working with microkernels for a while, the monolith way feels like the equivalent of starting every application as a kernel module rather than an independent process.
Lastly I think a microkernel where a single module crash can hang the entire system is still superior to a monolithic kernel. The worst bugs aren't crashes, they are the ones where a bit of memory got corrupted and odd things just start happening. The more address spaces you have, the more localized the behavior of such bugs will be.
One thing that is much easier to do in a microkernel than in a monolithic kernel is build a machine-checked proof of correctness. This is not to say that this cannot be done in a monolithic kernel, but the proof would have to extend to each device driver and service within the kernel, or at the very least, a large portion of the kernel would need to be disabled to bring the proof down to something that can be accomplished by a team of developers in a reasonable period of time.
A complete proof of correctness in a microkernel based operating system can be done in parts. First, the microkernel itself can be formally verified. Then, the communication mechanism between components can be formally verified. Then, each service or driver can be formally verified with regards to the communication mechanism, and its own contribution to the system as a whole. As long as it can be proved that the system will still operate correctly with one or all of these services or drivers in an invalid (crashed) state, then some level of operation can be guaranteed. In a correct context (fault tolerant hardware, such as a well built embedded system), such a microkernel architecture could be designed, and proven, to be robust.
The formal verification of seL4 is one example of this in practice.
Now, could one do the same proof-by-parts with a well designed monolithic kernel? Probably. However, the stronger the division is between parts, the more like a microkernel the monolithic kernel would become, until it would make sense to take advantage of IPC mechanisms between the parts to simplify the mechanical proof. Hence, we sort of come full circle.
While I appreciate both sides of this debate, I would rather run the risk of a slower but more correct system than a faster system that is harder to formally verify. This is not to say that all microkernel based operating systems are more robust than monolithic kernel based operating systems, but there is a growing subset of microkernel based operating systems that have been formally verified, and I'm quite happy about that.
Formal verification is one thing, but being able to write a test suite for a device driver where you simply mock a small kernel API must be a game-changer. Not to mention that a formal protocol with a small surface area should make it easier to build OS-independent drivers.
Certainly, having a well-defined interface makes both empirical testing and portability easier. Microkernels get this by default. To be fair though, modern monolithic kernels have pretty decent abstraction mechanisms that do make it easy to test drivers. Hypervisors and fast emulators have dramatically changed operating system and device driver development, removing a lot of the pain in this process.
I remember testing device drivers on bare metal because we had no choice. Now, I can fire up a GDB server in KVM and step through code. Monolithic or microkernel, testing and debugging is so much easier now than it was even 15 years ago.
A friend of mine in this discussion (microkernel vs monolithic kernels) discussed it this way;
"CPU architects design for monolithic kernels because they are the only type that have been commercially successful, further in the absence of any barrier to making a choice (micro/mono) the monolithic kernel choice has always prevailed. This may be a consipiracy on a monumental scale but it also may be that microkernels just don't work as well as their proponents would wish, and no amount of engineering investment has changed that in 20 years."
Ok, so that probably isn't word for word, but he said it in 2010, and here it is 2016 and its still true. I love the conceptual simplicity of micro kernels, they are easy to abstract and they are easier to show they have predictable response in a wide variety of situations. I have also met some really strong advocates for them, and listened as the QNX folks went on and on about how much they were investing in making these things better than any alternative, and yet here we are. I haven't attended a SOSP[1] symposium in a while but I do still look through their proceedings when I get a chance at the Stanford Library. And it hasn't felt like microkernels have really made any conceptual improvements in their design. Instead we are seeing more and more work going toward multiprocessing with a monolithic kernel on each processor and the thread separation being an interconnect rather than a memory bus.
TFA specifically lists several successful commercial microkernels (disclaimer: I work for a company that makes one of the kernels listed).
In addition, there are a lot of CPU features that are incredibly useful for microkernels that have been created both in embedded and non-embedded parts.
What's common to Symbian, QNX, car-infotainment and in-flight entertainment (system's I've interacted with)? In all those cases (QNX->blackberry) they're systems with awful input lag from my personal experience. I also noticed that performance was mentioned in the article only in terms of throughput. coincidence?
You have this backwards. QNX is a real-time operating system, and is optimized for response time vs Linux, which is a general purpose operating system typically optimized for throughput, although some work has been done on modifications to allow for real-time OS like scheduling.
It's possible to write a UI on QNX that doesn't have awful input lag. Doing so requires you to write software that prioritizes the UI, and to create a UI that runs in short, deterministic bursts. I have some experience with this because that is exactly what I'm doing in my current job, and with QNX to boot.
That sounds interesting! A lot of UI libraries want bursts of CPU time that are longer than acceptable in many real time systems, despite the amortized cost being very low.
Windows NT started as microkernel but Microsoft found out pretty early that the GUI performance was pretty bad so they had to push the GUI and other subsystems into the kernel. That pretty much killed the microkernel mantra.
The past few versions have actually been swinging back to microkernel. Graphics drivers, for example, exist in user space, as well as audio drivers, etc.
Watching this particular dance has been interesting. And the whole Wayland / X discussion as well.
One of the more interesting thing about Graphics drivers has been that while it used to be the CPU that was doing the rendering, and so having it part of the kernel was critical, modern GPUs are so much more capable in terms of rendering than a typical CPU that you have this network interconnect in the form of PCIe and you more often than not have two very nearly distinct memory spaces, so it again is a triumph of multiprocessing over microkerneling (if that is even a word).
Good IPC requires tight coupling between the IPC mechanism and the scheduling mechanism. In QNX, you call MsgSend, which sends a message and blocks. On the receive side, a process waiting in a MsgReceive gets the message and unblocks. The fast path for this, when there's a thread waiting for a message, does not go through CPU dispatching or CPU switching on a multiprocessor. This is crucial to performance.
You do pay a penalty for copying. For short messages, it's low. For anything that fits in cache, that you just created and will use immediately on the receive side, it's very low. Really big messages may add 10%-20% overhead. This is the main problem with drivers in user space, because they don't do anything with the data, they just move it around.
Incidentally, copying under QNX is preemptable by high-priority tasks. There's a hard upper bound on the longest uninterpretable kernel operation, and it's measured in microseconds. That's why you can use QNX for hard real time.
Contrast what happens on Linux. You have a few choices (System V IPC, pipes, sockets), but they all work like I/O operations. You send something, and you don't block when you do. If there's someone waiting on the queue or pipe, their thread becomes active, which means a trip through the scheduler to get them going.
Since the sending thread isn't blocked, if there's a free CPU, the receiving thread starts up on a different CPU, which means lots of cache misses. The sending thread probably blocks shortly after the send by reading something or waiting for a message, but the OS doesn't know that's going to happen. So when it does, there's another trip through the dispatcher for work to do.
In a compute-bound environment, this may mean that each IPC operation sends you to the end of the line for CPU time. Trying to use IPC operations like subroutine calls is painfully slow in such systems.
Write something that does short IPCs and waits for replies, and benchmark it. Then add some compute-bound tasks, enough to keep all the CPUs busy. If the IPC task performance drops by orders of magnitude, the IPC mechanism is badly designed.
L4 is very impressive, but I think they went a little too far by using shared memory as the main interprocess communication mechanism. This gets copying out of the kernel, but means that one process can screw up the format of the memory region it shares with another process. The OS may be robust and secure, but attacks on the process you're talking to by messing with pointers in the shared area may be possible.
The big problem with QNX is that it costs money. Linux is free, and Windows is basically given away with hardware and trialware.
I hope that someone writes a QNX-like microkernel in Rust. We need that.
I know it makes me heterodox in the microkernel community, but I like the choice of shared-memory there. There are well known data-structures that work with shared memory with well-understood behaviors with a malicious writer. They are tricky to get right, but only need to be implemented once, verified and reused many times
Furthermore, as soon as you are communicating with another process, you have to deal with receiving garbage if that process misbehaves, so it's not all sunshine and roses even if you use a more traditional IPC mechanism.
"There are well known data-structures that work with shared memory with well-understood behaviors with a malicious writer."
I'm very interested in learning about these shared memory data structures, would you mind providing a few links or mention some papers I could read.
We implemented a real-time microkernel in university where QNX has its roots [0]. It used traditional copy based IPC however. I'd like to see how these shared-memory data structures could have been used.
I'm actually struggling to find this in the literature, but I've seen it used more than once.
Here's the most simple version:
Minimum two pages, each readable from both sides and one is writable from each side. You need this even for a single-directional queue. Have a simple post/ack counter that is modulo something much larger than the size of the queue (i.e. an unsigned long on most architectures). If (post - ack) is ever greater than the size of the queue, the queue should be considered no longer safe to use; both readers and writers should check this anytime they access these values. Otherwise there are (post - ack) bytes pending in the queue.
To write, write bytes at (post MOD size) and increment post; to read, read from (ack MOD size) and increment ack.
That is a simple one-to-one byte-stream queue; fixed-size message queues are an obvious change, non-fixed size message queues are significantly more complex, but proceed from the same idea. Many-to-one queues are a lot more complex, and one-to-many is more complex if you want reliable data transfer, but much simpler if you don't (just eliminate the ack completely)
If all you care about is throughput and you have more than 2 cores, then just spinning on post/ack is sufficient. Otherwise some synchronization primitive is needed.
On many real-world architectures, particularly SMP ones there are a lot of fiddly details to get right with the MMU &ct. but the underlying idea is simple.
This is a bizarre, bigoted, and prejudiced statement:
> Sheltered users who have never had any background in much beyond Windows and some flavor of free monolithic Unix, will, despite a general apathy or ignorance in the relevant subjects, have strong opinions on the allegedly dreadful performance and impracticality of “microkernels,”
Everyone knows that Windows NT Architecture (in Windows 10) _is_ a Microkernel architecture. Windows, ever since NT, has been ahead of the technology curve.
I expect micro kernels will be more prevalent in the future.
Hardware platforms will have to evolve to have more physical independence and redundancy (both by replication, and different-vendor components assigned to have same functions).
I am of a view, that micro-kernels are better positioned to manage those types of hardware platforms
This discussion would be more interesting in the context of major new developments in OS kernels.
Microkernels are basically solving an engineering problem. That's exciting when engineering is the bottleneck holding up a bunch of really innovative ideas.
So what are those amazing kernel ideas being held back by engineering difficulties?
I never understood why exokernels stalled out; the idea of a kernel simply being a hardware multiplexer sounds great. IMO the problem that will always plauge microkernels is the overhead of context switching and from a technical standpoint it just seems like a waste of cycles considering the possible alternative. The alternative I would want to see is an exokernel with a data driven api, possible batching, some form of scheduler activations and proof carrying code to allow loading of 'modules' from unprivledged userspace. The issues would be 'syscall' time could vary greatly depending on what was requested and logic to optimize what a call actually has to do would have to be within the kernel.
They didn't, the technology just got renamed "virtualization" when people realized that if all the kernel does is multiplex the hardware, its API might as well look like the hardware too. Modern hypervisors are basically exokernels; it's a multibillion-dollar business that's a lot more successful than any microkernel has ever been.
In addition to nostrademons' point about virtualization, exokernels have influenced monolithic kernels in particular situations. For example, the Linux kernel's graphics stack is moving away from rendering through the X server, towards directly rendering into a buffer using libraries (Wayland is only the most recent step in this direction).
QNX was the core of BlackBerry's OS on the PlayBook and newer phones.
(Also, isn't OS X kinda sorta sitting atop Mach? I think it's impure or not even Mach these days, though. EDIT: Ah, XNU is a hybrid which adds kernel-mode drivers and BSD.)
Wow, so this is what it's like to be misrepresented. Firstly, there is no "microkernel hatred" on my part, just hatred of pop-opinions of them being a 100% superior technology for all cases.
And my second comment follows this theme as it was pointing out a contradiction in the poster's argument of microkernels simply being superior. I fully understand the rationale behind running Linux in a microkernel, but it isn't something that reinforces the argument that monolithic kernels are "just inferior".
In short, I'd be a lot more impressed if this dude spent the effort he used scouring the internet for quotes building something where he could just show us the code of a microkernel outperforming a monolithic kernel. Enough of the dogma.
(Oh, and note that at no point do I claim to be a kernel developer either)
Firstly, there is no "microkernel hatred" on my part, just hatred of pop-opinions of them being a 100% superior technology for all cases.
That's not a "pop-opinion". The popular opinion is exactly the opposite: microkernels are mostly a curiosity at best (if not a travesty) and monolithic kernels do 99.99% of what you want.
You're blinded by your microkernel hatred.
I fully understand the rationale behind running Linux in a microkernel, but it isn't something that reinforces the argument that monolithic kernels are "just inferior".
It isn't meant to reinforce such an argument.
In short, I'd be a lot more impressed if this dude spent the effort he used scouring the internet for quotes building something where he could just show us the code of a microkernel outperforming a monolithic kernel. Enough of the dogma.
Damage control. Those damned quotes from those damned academics, they don't mean a thing! All that matters is my stars on GitHub!
(Oh, and I don't call anyone "fags" either)
It should be obvious I was paraphrasing.
(Anyway, it's unsurprising to see you have no concrete rebuttal whatsoever and have to resort to deflecting.)
"That's not a "pop-opinion". The popular opinion is exactly the opposite: microkernels are mostly a curiosity at best (if not a travesty) and monolithic kernels do 99.99% of what you want."
Nope, I'd say the pop opinion is against anything that is described as "monolithic". The word "monolithic" itself is pretty much a pejorative in most circles these days.
"You're blinded by your microkernel hatred."
You're unbelievable. Clearly you must know me better than myself.
"It isn't meant to reinforce such an argument."
That was the whole point of the thread. That was your original principal statement (or implication anyway) which I took issue with.
"Damage control."
What damage?!
"Those damned quotes from those damned academics, they don't mean a thing!"
Those damned results from damned reality... If there's one thing I would be certain of about microkernels, it's that someone would be able to collect a lot of quotes from academics supporting them.
"(Anyway, it's unsurprising to see you have no concrete rebuttal whatsoever"
No! I am not going to spend hours assembling dogma! I have real things to do!
"have to resort to deflecting"
Look buddy, you're the only one who has made personal attacks and assume all kinds of things about me like this weird "hatred" thing and that I care about GitHub stars (wtf?).
Though I'm glad I seem to have ruined your xmas period with my few comments to the point you've spent time assembling a long page of things that have been said before which completely miss my point.
Wow, both of you need to stop flaming each other and say something valuable. No one should care about your line by line refutations of each other (though I'm sure some will make the mistake of caring).
What matters if what you can teach us about microkernels vs. monolithic kernels. The original "Microkernels are slow and Elvis didn't do drugs" somewhat does this, though the presentation as a series of lengthy quotes and the dismissive presentation of viewpoints (describing opponents as calling people "fags") could use some real work. But all three messages above this one are a waste.
I think there is a more pragmatic way to understand the microkernels issues. Performance is but one and over the decades it has become less of an issue, but hasn't vanished which should be noted, you usually have to out do the technology you want to push out. Have you ever modularized something too much? It is a meme here, abstractfactoryfactorybuildersingleton. You break things down in to simple singular purpose components that are easier to build correctly and reusable, then all the sudden you have your favorite Java or .net framework with abstract factory factories.
Take jboss, for example, great code. High quality code. It really works. Complex as all fuck to start using. Then when you want to understand something like jboss serialization, it's spread among a dozen classes in a way you'd have never imagined when you started thinking about it, you just want some function to look at the implements their algorithm. It didn't get that way by accident or even stupidity, it's too reusable. Or rather, their extreme modularity comes at a cost, maybe it's worth it but I don't see a lot of people bragging about a new jboss based whatever here...
Apply this to you operating system and ask yourself why. Why does netbsd port so well to new hardware? And why do people do that? Why is it more difficult to write a posix service on top on a microkernel than a monolithic one! Surely, it's a smaller chunk of code, it should have less complexity, be easier to write, right? It's not, just look at Hurd. The cost is the price of the modularity, your posix service needs to rely on other services, those dependencies need to be coded for, faults handled, etc. what does posix open do when the filesystem service is not responding? (Hint, it depends on the type of fault and whether or not it's recoverable, or you think it is)
They should logically be easier to build and because of that we should be crapping them out left and right. Plan9 should be micro kernel. With the current leaders in operating system semantics, posix and Windows, the modular implementation is more complex than the monolithic one.
There are a bunch of good ones out there, just pick a microkernel and start building a good BSD or Linux drop in, should be easy, right? Maybe if you use it as a hypervisor...
* http://tech-insider.org/linux/research/acrobat/960112.pdf See the Byte Benchmarks Results on page 12 for a laugh. (I mean, really. That is the funniest performance benchmarking chart I have ever seen, in half of a career that involved a lot of performance work.)
I can personally vouch for the 2x performance slowdown using (a derivative of) the Mach microkernel under IBM's WorkplaceOS project.
* https://www.usenix.org/legacy/events/osdi99/full_papers/spat... figure 8. Note that Scout is a microkernel that does no memory protection, Linux uses two domains (user space/kernel space), and the "Accounting" version of Escort/Scout uses 4 or 5. Performance is roughly proportional to the number of protection domains.
139 comments
[ 4.2 ms ] story [ 174 ms ] threadGiven the recent trend towards containers, it's obvious that people can now see how much better a model this is for micro-services.
It's just a matter of time before we start seeing more microkernels to run things like unikernels and such. Docker is just a stepping stone.
The difference is that microkernels move abstractions like the file system into server processes that still have to be trusted, but are at least isolated; exokernels and hypervisors remove the abstractions from the trusted code base entirely, and just multiplex the hardware directly. Unikernels thus simplify the typical virtualization stack so it looks more like MIT's exokernel research projects, with abstractions in untrusted libraries.
(This comment is not intended as an opinion in either direction on whether the original color scheme is reasonable or whether you doing these steps is reasonable.)
White-on-black text isn't normally that bad, I think it's the choice of thin font that makes it especially hard to read in this article.
Sure enough loading w/o 1Blocker enabled reader mode.
Very interesting.
But nobody uses it.
It turns crappy reading experiences into great ones.
This site is gray on gray, which makes it really freaking hard to read (and I have no issues with my eyes at all).
But it's no longer the mid-'90s, and everyone's been working on super efficient RPC (Cap'n Proto, msgpack, BSON, etc.) for unrelated problems. I have been suspecting for a while that a brand-new microkernel based firmly on mid-'10s technology (and targeting mid-'10s processors) would be perfectly performant.
If you read the article, you'll see several references to papers that show microkernel performance on par with or better than traditional monolithic Unix systems. For example, "EROS: a fast capability system" by Shapiro, Smith, and Farber, in SOSP 1999. http://dl.acm.org/citation.cfm?doid=319151.319163
Perhaps microkernels make more sense for our future hardware. Dunno just shooting from the hip here.
I think the market issue for microkernels is that
a) there aren't a lot of choices for microkernels that are 'production ready', today I could reasonably run a website on Windows NT, or OS X, but the Hurd is perpetually not ready.
b) Monolithic kernels are 'good enough'. The benefits microkernels promise don't really make a huge impact (or aren't perceived to). Yes, it would be great if my filesystem driver didn't crash the whole system when it crashes, but if it crashes in a microkernel, I expect I would still have to restart all the processes that had open files, because the state is lost; and either way, I'd rather not have my filesystem driver crash. Most users of monolithic kernels don't spend a lot of time developing or debugging kernel drivers, so developer efficiency there doesn't really pay off. Most of the problems I track down to a kernel issue are things that could be equally present in either style kernel: bugs/inefficiencies in the tcp stack, stupid disk controller resource leaks, etc. Maybe it would be easier to track them down if the tcp process was clearly spinning on all the cpu, or the disk controller process had a ton of ram/resources assigned to it, but it's not that hard to track things down anyway -- figuring out why they're broken is much harder than figuring out where they're broken.
b) agree with you on this point that monolithic is 'good enough' these days. There are specific benefits to my particular application, but for a general desktop user I don't see the point. Even on a server the benefits are not very visible.
People used to say the same about Perl 6! :)
Another neat idea would be to build on an existing microkernel like seL4, but write the userspace -- all the drivers, memory manager, filesystems, etc -- in Rust. The NICTA team already has proofs of correctness for the microkernel itself, so why reinvent the wheel? But at the same time, those proofs were super expensive and difficult to make. It would be awesome if we could get a reliable userland more easily by using a better language.
I wonder what it would take to get Rust compiling for seL4?
https://gitlab.com/robigalia/sel4-sys
Why not.
There's a mailing list, if you want to watch: https://lists.robigalia.org/listinfo/robigalia-dev
And the (still in-progress) source code: http://gitlab.com/robigalia/sel4-sys.
I have some ideas about encoding RPC protocols as DFAs in Rust's type system, and then doing verification of the protocols in a more abstract setting (perhaps NuSMV or a similar checker). This is somewhat similar to the (ancient at this point) "protocol compiler" that used to be in Rust, doing a similar thing for channels.
I'm interning at Data61 (née NICTA) on the verification team starting Jan 18, and I'm hoping to transfer some of that experience and knowledge into verifying Rust.
I am working on some formal verification myself for some commercial projects, and I enjoy reading about stuff like this.
But today, that's not the only way to guarantee isolation. We may also be able to do it via type systems. If we could create things statically guaranteed to be isolated but not requiring actual context switches, it should be theoretically possible to create things that are structurally microkernels, but do not need internal context switches because isolation has been proved at a higher level. Or, arguably, in some sense this would be something that transcends both microkernels and monolithic kernels to become something not properly called either.
I'm sure there's many an i to be dotted and ts to be crossed before this would work, and you'd still probably want to work out a run-time loadable module which would probably make things interesting, you need a story for how a device driver can individually crash without bringing the whole process down (though since Linux can do it, presumably it is possible anyhow), and a whole bunch of things. But it does seem like an interesting opportunity for someone to make a kernel that is, well, not the same basic kernel the C type system naturally affords.
http://c2.com/cgi/wiki?NeoKernel
It's basically relying on the type system & safety mechanisms of the underlying programming language to enforce some of the isolation properties that an operating system is normally used for. The Lisp Machine was probably the most commercially successful example. Since neokernels typically require tight integration between the programming language and the kernel, there are also opportunities for efficiency gains, eg. in Lisp Machines the garbage collector was tightly integrated with virtual memory to prevent page thrashing on collection, or in modern languages you could replace the OS scheduler with promises or green threads.
The biggest downside of it is that it requires tight coupling between the programming language and OS, such that you can only use a single programming language (or a family of related ones that all share a type system) for all programs on the box. Few programming languages are flexible enough for that. With the move to cloud computing and virtualization, this becomes more practical, but then that same trend is also driving interest in the unikernel/exokernel pairing. If you only run a single application on a virtualized box, why not program directly to the hardware?
I would point out (from a relatively lay perspective) that while this isn't strictly the case in current popular kernels (Mach/OSX, Windows, Linux), it's still a pretty strong trend. These kernels (AFAIK) are implemented in C and have large ecosystems of C code orbiting them. As such there are many benefits to having a program written in C or at least speaking C's ABI and memory layout.
Maybe I'm mistaking the causes for this situation, but most developers are effectively using something that is either C itself, written in C, or speaks C in some way (CPython, JVM, MRI/CRuby, Perl, are notables that are written in C as far as I know). I think that if a language is sufficiently high performance and general purpose, it's fine for it to be the "core" language that other languages are expected to interface with.
Erlang was meant to fill this role in newer switches, but it looks like PLEX is still being developed and used.
It seems to me that recent advances in this field ought to ease the restrictions of requiring the entire stack to be the same language. It seems like a Rust neokernel ought to be able to run other programs on top much like a conventional kernel does, context switches and all. I'm not too worried about userspace/kernel context switches... there's a security barrier there that still strikes me as a good idea. I'm OK with something that produces an artifact in the end that looks like a monolithic kernel, and can function like one, and is merely internally structured with all the discipline of a microkernel.
If all kernel elements have to be Rust, well, I'm fine with that.
When you realize that kernel-mode vs userland is a social construct and not inherent, you see that the microkernel is just attempting to recover safety from the processor's built-in paging and protection capabilities. The correct solution of course is just to use a memory safe language.
Likewise, the microkernel encourages abstraction due to the pain of RPC boilerplate, but modern languages and sound engineering practices should make this easy anyway.
Memory safety isn't enough, you need more. (More reading: https://microkerneldude.wordpress.com/2013/05/03/closing-the...)
Even if you trust the language to provide the isolation you need, today's requirements can't possibly be met by such a system. Maybe in 50 years once all software has been rewritten.
(Though, I mean, you can sort of get around this requirement if you only accept "binaries" that are actually AST or high-level bytecode, and then finish compiling them in kernel-space at module-load time. OSX and iOS could get away with doing this soon-ish given that all new binaries submitted to the App Store are submitted as LLVM bitcode. Microsoft could probably get away with it for Metro apps as well.)
A robust system is designed such that any machine in it can randomly crash. Crashing machines then only represent a risk that needs to be mitigated by having enough spare capacity to handle however many machines are expected to be down at once. Linux is already stable enough that this amount is too small to care about; making it more stable doesn't help me.
Would I expect meaningful performance improvements from microkernels? I'm doubtful that you'd get as much as 10%, but I'd be interested in seeing evidence if anybody has any.
Do microkernels offer any application-visible features that I don't have today with Linux? To the best of my knowledge they do not.
Most people run VMs for hardware utilization purposes.
There are a number of ways to build unikernels and they're not all OCaml (which is just MirageOS) -- http://unikernel.org is trying to bring these together.
[1] http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewal...
[2] https://galois.com/project/cyberchaff/
It would be great to get a list of live unikernel deployments listed somewhere to address questions like this when they come up.
[1] http://unikernel.org/blog/2015/unikernels-meet-docker/
Hipsters.
Monolithic kernel = all familiar kernel abstractions (processes, IPC, filesystem, virtual memory, device drivers, networking stack) are built into the kernel.
Microkernel = Most kernel abstractions run as separate processes. Kernel communicates with them over IPC.
Exokernel/unikernel = There are no kernel abstractions. Instead, these abstractions are built into runtime libraries that are then linked into applications. The kernel's only job is to securely multiplex the hardware, and it exports an interface that is similar to bare metal. In 2016, the exokernel is called a hypervisor, and the libOS is called a unikernel.
Neokernel = the programming language's runtime library provides the primary kernel abstractions, along with any interprocess security. Machines are limited to a single programming language.
Most people have experience running processes in isolated VMs for security reasons. I was trying to use that as an anchor point to explain why one might want to have components of a single operating system run as separate servers, for much the same reason -- compartmentalized security.
Of course, this presupposes things about API compatibility/stability between microkernel components that may not be true in all microkernel designs.
I mean, if the ABI or internal kernel macros change, you're in for some porting work, but most of the kernel stuff can be configured as loadable modules. :)
It took Microsoft until XP to actually be able to stop supporting the old crap VXDs and just force all the hardware manufacturers to "write a new driver or else". That they then also instituted the WHQL program to certify those drivers—and later added tooling to make that certifying job more rigorous—was just icing. It was moving away from VXD drivers that solved most of the problem.
See also, everything Raymond Chen has ever written. For example: https://blogs.msdn.microsoft.com/oldnewthing/20031224-00/?p=...
Just wish that the user space libs would adopt the same policy, rather than try to paper over it by stuffing everything into containers (liable to turn into the Linux equivalent of DLL hell).
As a layperson, I can be persuaded microkernels are a good idea that could yield some reliability and security improvement. I can be persuaded that the performance issues are avoidable. But I'm also pretty sure that most of the problems we have with current systems are in code that would now be outside the microkernel anyway. We are going to end up with, optimistically, about the same number of lines of code. So to me it seems much more important what the development process looks like. Unfortunately there is not a lot of accessible research about that kind of thing, relative to things like microkernel performance which are at least easy to generate demonstrative benchmarks for.
For this sort of task, there's a remarkable amount of dead weight that can be cleared out. For starters, because your binaries are so small, serving them over PXE or the like becomes a triviality in regards to bandwidth. Your system will start faster because it doesn't have to do things like initialize the various pseudo-filesystems Linux has, or initialize the virtual file system, or any number of other tasks. This now means you don't need that hot, failure prone hard disk at all, nor its buffers and journals and other context switches that occur because of it. Because you've gotten rid of the hard disk, you've gotten rid of its heat, meaning you can pack servers more densely.
Similarly, you can eliminate a considerable amount of the networking stack. You know what the messages it's going to send and receive look like, and any other messages are going to be dropped anyways, so why have the code to handle them loaded in memory at all?
Because of their minimalism, microkernels allow you to ask a lot more fundamental questions about your applications. This means you can build a system much more oriented around your needs, with minimal superfluous framework. This now means your apps are faster, because they have to fight less OS bureaucracy, more reliable because they're much more simple, and more secure, because the system as a whole is smaller, and thus easier to audit. Sometimes, asking yourself what you don't need is just as important, if not moreso, than what you do need.
Not having a hard disk? Doesn't require a microkernel at all. Booting over LAN has been practical for a long time.
If I glance the closest linux machine I have, it has a 6MB kernel with 10MB initrd. To put that on a server with gigabytes of ram? Trimming an unused networking stack here is not the use case where a microkernel will help performance.
Perhaps there is a use case that is compelling, but this particular one doesn't do it for me.
Linux won't access the hard drive if it's unnecessary, same as the microkernel.
Don't you hate having to upgrade your whole kernel (memory manager, scheduler, fs, network stack) just to fix a small-but-annoying problem with your GPU (for example tearing)?
With a microkernel you could update that driver without updating the rest of the kernel, or of the OS. Yes "OS", because an update of the kernel often requires an update of the corresponding user-space programs (lmsensors, libdrm, mesa, X11, libalsa, iptables). If our systems were built upon microkernels and developed with strict boundaries between components it would be much easier to maintain them.
However I think Torvarlds has a point when it says that going down the path of microkernels means turning many low-level problems into IPC problems, most of which are not yet solved or not solved in a way that is compatible (performance-wise) with current hardware. Let's hope research and the brewing libre hardware community will tackle this problem.
This assumes that every robust system can consist of multiple machines. One thing I've learned from the article, and other comments in favor of microkernels, is that they're successful in applications where a single machine has to be a robust system all by itself.
For cloud applications, what is more important is the system architecture, not the kernel. If we assume docker is the preferred way to deploy an app. We only need very limited features from kernel:
- If I specify the CPU and RAM requirements, the kernel simply allocates them to the container. There is no need for complicated dynamic scheduling and balance.
- Network within the same data center has much lower latency than hard disk (<<1ms vs 10ms). We will be much better of using network storage or database.
- If we assign network address to each container, the kernel can deliver network packets to container, and the container uses its own CPU and RAM to process the network traffic. This will avoid the situation that kernel spend a lot of CPU/RAM to process network traffic for all containers and find the correct way to charge the cost.
If we do all above, there is not much kernel features are needed by cloud applications, as such features will be replaced by cloud services like cloud storage/database. It will make develop and deploy cloud services easier and more productive.
Of course, the big standout for me has always been that if micro-kernel based systems of today are just as, or close enough in terms of performance, why aren't we getting any benchmarks showing this off ?
It would be fantastic promotion to show your system running against Linux or the BSD's and performing as well or near as well in heavy workloads while at the same time offering the great stability of typically being able to restart modules should they crash and thus not take down the whole system.
The micro-kernel vendors should be shouting it from the rooftops, 'here are the benchmarks!', but they don't as far as I can tell.
The last time I saw a modern micro-kernel vs modern monolithic comparison was probably this one way back in 2007, and in this Minix 3 did not do very well against Linux.
http://lwn.net/Articles/220255/
If you want to compare microkernel operating systems, look at QNX, L4, Windows CE, etc. These are used in embedded and RTOS spaces where performance is critical. They seem to hold their own pretty well, and running in systems where Linux would not be a good alternative due to size and porting cost.
I agree with you that "fast enough" is the only metric that matters here.
Every now and then there's this thing that happens where somebody runs across some surface-level comment by some fairly authoritative source, arrives at a mistaken interpretation of it, and that interpretation just happens to be the one that gets widely repeated, and thus an error enters into the public awareness on that topic. That seems to be what happened with the MINIX-and-embedded-systems meme that keeps showing up.
MINIX tried to position itself as a good candidate for use on embedded systems—in addition to its then-current primary (read: only viable) use as a PC operating system. It never really ended up happening. The idea seems to have been to use the grant money as a catalyst to get it in shape so that it looks attractive and can get commercial adoption at the cusp of the move to a post-PC world, and then it will be sufficiently dug-in with a few parties, enough so that it would see sustained development in your typical open source fashion.
The reality is, it didn't really pan out (NB: not necessarily as a result of its essential microkernellyness), they burned through several million Euro that all seems to have gone towards paying developers and putting very little towards project management, and today nobody is doing serious development on it, save for something like one grad student who mostly continues to work on it in the course of his research. In its current state, it has rudimentary support for something like two boards (from the BeagleBoard line), and its best supported platform, relatively speaking, is still PCs, although its support there can itself only really be called so-so by a more absolute metric.
(Caveat: MINIX is BSD licensed, so it's possible there's somebody out there getting lots of mileage out of it and/or doing spectacular things.)
To be fair, even as a teaching OS, though, things do look rough nowadays. In the past, the way things worked was, Tanenbaum would take the sources, clean it up and trim some of the parts that wouldn't be covered in the Operating Systems book, and then ship it. With the current state of the code, after the quality of the changes that occurred in the aforementioned process, if he were to try to do the same for a 4th edition of the book, he's going to have a lot of work ahead of him.
All in all, given the current state of the project, the answer to the question, "What is the best use for MINIX today?" is "For research."
I say all this as someone who has got some patches for cleaning up and consolidating a bit of the bootstrap code for ARM and x86, but with nowhere to effectively send them.
A microkernel offers you the flexibility to choose this though. On linux, I write all my programs to run as processes. If I find I'm getting performance bound by system-call overhead, I can convert part of the program to a kernel module, to eliminate that.
Microkernels extend this concept into the OS. If you find a bottleneck that is truely bogging you down, you can fix it. After working with microkernels for a while, the monolith way feels like the equivalent of starting every application as a kernel module rather than an independent process.
Lastly I think a microkernel where a single module crash can hang the entire system is still superior to a monolithic kernel. The worst bugs aren't crashes, they are the ones where a bit of memory got corrupted and odd things just start happening. The more address spaces you have, the more localized the behavior of such bugs will be.
A complete proof of correctness in a microkernel based operating system can be done in parts. First, the microkernel itself can be formally verified. Then, the communication mechanism between components can be formally verified. Then, each service or driver can be formally verified with regards to the communication mechanism, and its own contribution to the system as a whole. As long as it can be proved that the system will still operate correctly with one or all of these services or drivers in an invalid (crashed) state, then some level of operation can be guaranteed. In a correct context (fault tolerant hardware, such as a well built embedded system), such a microkernel architecture could be designed, and proven, to be robust.
The formal verification of seL4 is one example of this in practice.
Now, could one do the same proof-by-parts with a well designed monolithic kernel? Probably. However, the stronger the division is between parts, the more like a microkernel the monolithic kernel would become, until it would make sense to take advantage of IPC mechanisms between the parts to simplify the mechanical proof. Hence, we sort of come full circle.
While I appreciate both sides of this debate, I would rather run the risk of a slower but more correct system than a faster system that is harder to formally verify. This is not to say that all microkernel based operating systems are more robust than monolithic kernel based operating systems, but there is a growing subset of microkernel based operating systems that have been formally verified, and I'm quite happy about that.
I remember testing device drivers on bare metal because we had no choice. Now, I can fire up a GDB server in KVM and step through code. Monolithic or microkernel, testing and debugging is so much easier now than it was even 15 years ago.
"CPU architects design for monolithic kernels because they are the only type that have been commercially successful, further in the absence of any barrier to making a choice (micro/mono) the monolithic kernel choice has always prevailed. This may be a consipiracy on a monumental scale but it also may be that microkernels just don't work as well as their proponents would wish, and no amount of engineering investment has changed that in 20 years."
Ok, so that probably isn't word for word, but he said it in 2010, and here it is 2016 and its still true. I love the conceptual simplicity of micro kernels, they are easy to abstract and they are easier to show they have predictable response in a wide variety of situations. I have also met some really strong advocates for them, and listened as the QNX folks went on and on about how much they were investing in making these things better than any alternative, and yet here we are. I haven't attended a SOSP[1] symposium in a while but I do still look through their proceedings when I get a chance at the Stanford Library. And it hasn't felt like microkernels have really made any conceptual improvements in their design. Instead we are seeing more and more work going toward multiprocessing with a monolithic kernel on each processor and the thread separation being an interconnect rather than a memory bus.
[1] http://www.sosp.org
In addition, there are a lot of CPU features that are incredibly useful for microkernels that have been created both in embedded and non-embedded parts.
Both Windows and OSX have (somewhat-compromised) microkernels at their core. As do the operating systems for most game consoles.
One of the more interesting thing about Graphics drivers has been that while it used to be the CPU that was doing the rendering, and so having it part of the kernel was critical, modern GPUs are so much more capable in terms of rendering than a typical CPU that you have this network interconnect in the form of PCIe and you more often than not have two very nearly distinct memory spaces, so it again is a triumph of multiprocessing over microkerneling (if that is even a word).
You do pay a penalty for copying. For short messages, it's low. For anything that fits in cache, that you just created and will use immediately on the receive side, it's very low. Really big messages may add 10%-20% overhead. This is the main problem with drivers in user space, because they don't do anything with the data, they just move it around.
Incidentally, copying under QNX is preemptable by high-priority tasks. There's a hard upper bound on the longest uninterpretable kernel operation, and it's measured in microseconds. That's why you can use QNX for hard real time.
Contrast what happens on Linux. You have a few choices (System V IPC, pipes, sockets), but they all work like I/O operations. You send something, and you don't block when you do. If there's someone waiting on the queue or pipe, their thread becomes active, which means a trip through the scheduler to get them going. Since the sending thread isn't blocked, if there's a free CPU, the receiving thread starts up on a different CPU, which means lots of cache misses. The sending thread probably blocks shortly after the send by reading something or waiting for a message, but the OS doesn't know that's going to happen. So when it does, there's another trip through the dispatcher for work to do.
In a compute-bound environment, this may mean that each IPC operation sends you to the end of the line for CPU time. Trying to use IPC operations like subroutine calls is painfully slow in such systems.
Write something that does short IPCs and waits for replies, and benchmark it. Then add some compute-bound tasks, enough to keep all the CPUs busy. If the IPC task performance drops by orders of magnitude, the IPC mechanism is badly designed.
L4 is very impressive, but I think they went a little too far by using shared memory as the main interprocess communication mechanism. This gets copying out of the kernel, but means that one process can screw up the format of the memory region it shares with another process. The OS may be robust and secure, but attacks on the process you're talking to by messing with pointers in the shared area may be possible.
The big problem with QNX is that it costs money. Linux is free, and Windows is basically given away with hardware and trialware.
I hope that someone writes a QNX-like microkernel in Rust. We need that.
Furthermore, as soon as you are communicating with another process, you have to deal with receiving garbage if that process misbehaves, so it's not all sunshine and roses even if you use a more traditional IPC mechanism.
I'm very interested in learning about these shared memory data structures, would you mind providing a few links or mention some papers I could read.
We implemented a real-time microkernel in university where QNX has its roots [0]. It used traditional copy based IPC however. I'd like to see how these shared-memory data structures could have been used.
Thanks.
[0] https://www.student.cs.uwaterloo.ca/~cs452/
Here's the most simple version:
Minimum two pages, each readable from both sides and one is writable from each side. You need this even for a single-directional queue. Have a simple post/ack counter that is modulo something much larger than the size of the queue (i.e. an unsigned long on most architectures). If (post - ack) is ever greater than the size of the queue, the queue should be considered no longer safe to use; both readers and writers should check this anytime they access these values. Otherwise there are (post - ack) bytes pending in the queue.
To write, write bytes at (post MOD size) and increment post; to read, read from (ack MOD size) and increment ack.
That is a simple one-to-one byte-stream queue; fixed-size message queues are an obvious change, non-fixed size message queues are significantly more complex, but proceed from the same idea. Many-to-one queues are a lot more complex, and one-to-many is more complex if you want reliable data transfer, but much simpler if you don't (just eliminate the ack completely)
If all you care about is throughput and you have more than 2 cores, then just spinning on post/ack is sufficient. Otherwise some synchronization primitive is needed.
On many real-world architectures, particularly SMP ones there are a lot of fiddly details to get right with the MMU &ct. but the underlying idea is simple.
> Sheltered users who have never had any background in much beyond Windows and some flavor of free monolithic Unix, will, despite a general apathy or ignorance in the relevant subjects, have strong opinions on the allegedly dreadful performance and impracticality of “microkernels,”
Everyone knows that Windows NT Architecture (in Windows 10) _is_ a Microkernel architecture. Windows, ever since NT, has been ahead of the technology curve.
Microkernels are basically solving an engineering problem. That's exciting when engineering is the bottleneck holding up a bunch of really innovative ideas.
So what are those amazing kernel ideas being held back by engineering difficulties?
(Also, isn't OS X kinda sorta sitting atop Mach? I think it's impure or not even Mach these days, though. EDIT: Ah, XNU is a hybrid which adds kernel-mode drivers and BSD.)
And my second comment follows this theme as it was pointing out a contradiction in the poster's argument of microkernels simply being superior. I fully understand the rationale behind running Linux in a microkernel, but it isn't something that reinforces the argument that monolithic kernels are "just inferior".
In short, I'd be a lot more impressed if this dude spent the effort he used scouring the internet for quotes building something where he could just show us the code of a microkernel outperforming a monolithic kernel. Enough of the dogma.
(Oh, and note that at no point do I claim to be a kernel developer either)
(Oh, and I don't call anyone "fags" either)
That's not a "pop-opinion". The popular opinion is exactly the opposite: microkernels are mostly a curiosity at best (if not a travesty) and monolithic kernels do 99.99% of what you want.
You're blinded by your microkernel hatred.
I fully understand the rationale behind running Linux in a microkernel, but it isn't something that reinforces the argument that monolithic kernels are "just inferior".
It isn't meant to reinforce such an argument.
In short, I'd be a lot more impressed if this dude spent the effort he used scouring the internet for quotes building something where he could just show us the code of a microkernel outperforming a monolithic kernel. Enough of the dogma.
Damage control. Those damned quotes from those damned academics, they don't mean a thing! All that matters is my stars on GitHub!
(Oh, and I don't call anyone "fags" either)
It should be obvious I was paraphrasing.
(Anyway, it's unsurprising to see you have no concrete rebuttal whatsoever and have to resort to deflecting.)
Nope, I'd say the pop opinion is against anything that is described as "monolithic". The word "monolithic" itself is pretty much a pejorative in most circles these days.
"You're blinded by your microkernel hatred."
You're unbelievable. Clearly you must know me better than myself.
"It isn't meant to reinforce such an argument."
That was the whole point of the thread. That was your original principal statement (or implication anyway) which I took issue with.
"Damage control."
What damage?!
"Those damned quotes from those damned academics, they don't mean a thing!"
Those damned results from damned reality... If there's one thing I would be certain of about microkernels, it's that someone would be able to collect a lot of quotes from academics supporting them.
"(Anyway, it's unsurprising to see you have no concrete rebuttal whatsoever"
No! I am not going to spend hours assembling dogma! I have real things to do!
"have to resort to deflecting"
Look buddy, you're the only one who has made personal attacks and assume all kinds of things about me like this weird "hatred" thing and that I care about GitHub stars (wtf?).
Though I'm glad I seem to have ruined your xmas period with my few comments to the point you've spent time assembling a long page of things that have been said before which completely miss my point.
What matters if what you can teach us about microkernels vs. monolithic kernels. The original "Microkernels are slow and Elvis didn't do drugs" somewhat does this, though the presentation as a series of lengthy quotes and the dismissive presentation of viewpoints (describing opponents as calling people "fags") could use some real work. But all three messages above this one are a waste.
I think there is a more pragmatic way to understand the microkernels issues. Performance is but one and over the decades it has become less of an issue, but hasn't vanished which should be noted, you usually have to out do the technology you want to push out. Have you ever modularized something too much? It is a meme here, abstractfactoryfactorybuildersingleton. You break things down in to simple singular purpose components that are easier to build correctly and reusable, then all the sudden you have your favorite Java or .net framework with abstract factory factories.
Take jboss, for example, great code. High quality code. It really works. Complex as all fuck to start using. Then when you want to understand something like jboss serialization, it's spread among a dozen classes in a way you'd have never imagined when you started thinking about it, you just want some function to look at the implements their algorithm. It didn't get that way by accident or even stupidity, it's too reusable. Or rather, their extreme modularity comes at a cost, maybe it's worth it but I don't see a lot of people bragging about a new jboss based whatever here...
Apply this to you operating system and ask yourself why. Why does netbsd port so well to new hardware? And why do people do that? Why is it more difficult to write a posix service on top on a microkernel than a monolithic one! Surely, it's a smaller chunk of code, it should have less complexity, be easier to write, right? It's not, just look at Hurd. The cost is the price of the modularity, your posix service needs to rely on other services, those dependencies need to be coded for, faults handled, etc. what does posix open do when the filesystem service is not responding? (Hint, it depends on the type of fault and whether or not it's recoverable, or you think it is)
They should logically be easier to build and because of that we should be crapping them out left and right. Plan9 should be micro kernel. With the current leaders in operating system semantics, posix and Windows, the modular implementation is more complex than the monolithic one.
There are a bunch of good ones out there, just pick a microkernel and start building a good BSD or Linux drop in, should be easy, right? Maybe if you use it as a hypervisor...
https://www.kernel.org/doc/ols/2007/ols2007v1-pages-251-262....
I can personally vouch for the 2x performance slowdown using (a derivative of) the Mach microkernel under IBM's WorkplaceOS project.
* https://www.usenix.org/legacy/events/osdi99/full_papers/spat... figure 8. Note that Scout is a microkernel that does no memory protection, Linux uses two domains (user space/kernel space), and the "Accounting" version of Escort/Scout uses 4 or 5. Performance is roughly proportional to the number of protection domains.