The fun thing about this is that you can try it out locally and apparently it took (about) an afternoon to put together (most of the work is in rewriting URLs).
This kind of fast experimentation is going to lead to some really cool things happening with unikernels.
This is better than a "container." A Mirage unikernel literally compiles in the pieces of an "OS" as libraries. There is no monolithic operating system with code sitting around waiting to be exploited. The web server in this example is the kernel.
" I have not been able to find a hosting provider that has a minimum configuration which would be appropriate for large or small unikernels (e.g. 8MB vs 64MB RAM)." This is the biggest problem with unikernels right now.
I've always thought that something really cool could be done with unikernel for SaaS: you let each of your customer have a VM on their own, spinning the unikernel up when the requests came in, and let it process any request come in, after a while, if the specific user stop requesting then you kill the VM. There seems to be a lot of benefits from that: single user authentication, horizontally scaling is simple ("vertical" scaling also get easier as you can just move VM away), app design will easily let it become an on premise installation. Code upgrade is also easier as it won't disturb everyone at once, and you can upgrade when the VM is not running for a specific user (no hotswap needed). There is definitely operational overhead in a DNS server, but besides that it seems like the benefits would outweight the cons.
I know that's what DabbleDB was doing back then (in Squeak Smalltalk, each of their users has a smalltalk VM serving them). Is there anyone else doing anything similar nowadays?
That sort of design is what motivated Magnus' post that has been linked here. We've been laying the uniSaaS groundwork with a new Xen toolstack daemon called Jitsu ("Just in Time Summoning of Unikernels").
I think the obstacle is that most folk are not prepared to give up their languages for this, so containerising PaaSes are still likely to dominate the scene.
Hm. Got me thinking about how to fit unikernels into the CF architecture, which is largely taking containers (Garden or Docker, take your pick) as the unit of currency at the low level.
It looks like the place to make the change would be in Diego in the Executor[0][1] component, which currently implements behaviour in terms of containers (using Linux-Garden as the container system).
So long as you can conceptually fit into the Receptor Actions API[2] it looks as though unikernels would be a plausible addition to Executor. Once you have that, you can either wire into Lattice[3] or a full Cloud Foundry installation as a first-class citizen.
Disclaimer: I'm not as familiar with Diego as other CFers are, so it's possible I'm just making stuff up.
The shiny website was new to me. I wonder if we've officially announced it yet?
In any case, we started the Lattice effort last year to make it easier for people to play with Diego without having to stand up a whole Cloud Foundry installation.
Site is running on such a VM as well, press escape for a console to see stats.
Here is an example output from running their demo:
---
300 sec : is how long it takes to launch a Linux instance to availability on Amazon EC2.
50 sec : is the time between power on and the lock screen for an Android phone.
0.4 sec : ago is when we received your request. Within this time we managed to create a new Xen instance, boot it, and run the application that rendered the page you are viewing. By the time you are done reading this, the instance will be gone.
Note that both of the setups are quite different in how they boot unikernels. The EoX Zerg demo boots one unikernel per connection by having an HTTP proxy in the frontend spin up the VM.
Magnus' version builds a static set of unikernels whose total resource usage is known ahead of time, and boots them based on DNS lookups as necessary. Because there is a race between a DNS response, the first TCP SYN from the client, and the VM booting up to receive it, there is a helper unikernel called "Synjitsu" which terminates TCP connections and replays them to the MirageOS unikernel as soon as it boots up. This is very easy due to the OCaml TCP/IP stack in Synjitsu sharing its state via serialising it into an s-expression. More gory details in http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf
Now, imagine a version of Synjitsu that did this handover with TLS connections rather than just TCP. The recipient unikernel wouldn't ever even have to know the private key, just the ephemeral session key...
That is just a demo to show startup times. Presumably (just by looking at it from a high level) a proxy instance for handling initial connection + a pre-allocated, or maybe booted but paused, set of worker images would work in either architecture.
I seem to remember something similar that was an erlang VM running (I think) directly on Xen. The underlying idea in that case was each request causes a new VM to boot, service that request, then terminate.
I suppose work like this is demonstrating that, contrary to what e.g. the creators of Docker say, a virtual machine image is not too heavyweight to be our industry's equivalent of a shipping container. What is too heavyweight is a VM image of a conventional operating system distribution with all of the baggage that it brings.
Would these fine-grained unikernels also run well on KVM, VMware ESXi, Hyper-V, or bhyve (FreeBSD's new VMM)? Or is Xen uniquely suited to this kind of thing?
> "What is too heavyweight is a VM image of a conventional operating system distribution with all of the baggage that it brings."
Indeed. I didn't realise that Docker/Container folks had ever said VMs were too heavyweight. Is this linked anywhere?
> "Would these fine-grained unikernels also run well on KVM, VMware ESXi, Hyper-V, or bhyve (FreeBSD's new VMM)?"
We've been asked this before and for KVM at least, it just boils down to someone suitably motivated to write the boot and VirtIO code [1, 2]. If anyone's interested in this, we'd love to hear about it [3]!
I'm no DNS or security expert, but couldn't one application be increased security of connections? E.g. it's harder to do that 0day kernel hack with a buffer overflow if every URL request is sandboxed in a VM?
I probably got it wrong but I'd love to hear a more expert opinion on why/why not this could be used for security..
This is an interesting experiment in moving the computation to the endpoints of a namespace. I would suggest a variant where instead of a VM you place a Fossil file
Using that you can store a whole filesystem at the URL instead of one file, and you also get synchronization mechanisms built in, so your endpoints can communicate with each other in a structured way that you control.. vs the undefined communication you will get between VM's if you try this with the VM architecture.
33 comments
[ 2.8 ms ] story [ 84.5 ms ] threadThis kind of fast experimentation is going to lead to some really cool things happening with unikernels.
I know that's what DabbleDB was doing back then (in Squeak Smalltalk, each of their users has a smalltalk VM serving them). Is there anyone else doing anything similar nowadays?
Here's the camera ready paper that we'll be presenting at USENIX NSDI in May: http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf
There's a lot more besides just the spawning to build a robust SaaS of course, but you can see the direction we're heading in...
Disclaimer: I worked on Cloud Foundry.
s/languages/frameworks, and I agree. One step at a time -- this work is a foundational block that we're building on, not the ultimate solution.
It looks like the place to make the change would be in Diego in the Executor[0][1] component, which currently implements behaviour in terms of containers (using Linux-Garden as the container system).
So long as you can conceptually fit into the Receptor Actions API[2] it looks as though unikernels would be a plausible addition to Executor. Once you have that, you can either wire into Lattice[3] or a full Cloud Foundry installation as a first-class citizen.
Disclaimer: I'm not as familiar with Diego as other CFers are, so it's possible I'm just making stuff up.
[0] https://github.com/cloudfoundry-incubator/executor
[1] https://github.com/cloudfoundry-incubator/diego-design-notes
[2] https://github.com/cloudfoundry-incubator/receptor/blob/mast...
[3] http://lattice.cf/
In any case, we started the Lattice effort last year to make it easier for people to play with Diego without having to stand up a whole Cloud Foundry installation.
My only Lattice-related claim to fame is not knowing what is was: https://github.com/pivotal-cf-experimental/lattice-app/issue...
It was published at SYSTOR last year (2014).
http://erlangonxen.org/
Site is running on such a VM as well, press escape for a console to see stats.
Here is an example output from running their demo:
---
300 sec : is how long it takes to launch a Linux instance to availability on Amazon EC2.
50 sec : is the time between power on and the lock screen for an Android phone.
0.4 sec : ago is when we received your request. Within this time we managed to create a new Xen instance, boot it, and run the application that rendered the page you are viewing. By the time you are done reading this, the instance will be gone.
---
Magnus' version builds a static set of unikernels whose total resource usage is known ahead of time, and boots them based on DNS lookups as necessary. Because there is a race between a DNS response, the first TCP SYN from the client, and the VM booting up to receive it, there is a helper unikernel called "Synjitsu" which terminates TCP connections and replays them to the MirageOS unikernel as soon as it boots up. This is very easy due to the OCaml TCP/IP stack in Synjitsu sharing its state via serialising it into an s-expression. More gory details in http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf
Now, imagine a version of Synjitsu that did this handover with TLS connections rather than just TCP. The recipient unikernel wouldn't ever even have to know the private key, just the ephemeral session key...
Would these fine-grained unikernels also run well on KVM, VMware ESXi, Hyper-V, or bhyve (FreeBSD's new VMM)? Or is Xen uniquely suited to this kind of thing?
Indeed. I didn't realise that Docker/Container folks had ever said VMs were too heavyweight. Is this linked anywhere?
> "Would these fine-grained unikernels also run well on KVM, VMware ESXi, Hyper-V, or bhyve (FreeBSD's new VMM)?"
We've been asked this before and for KVM at least, it just boils down to someone suitably motivated to write the boot and VirtIO code [1, 2]. If anyone's interested in this, we'd love to hear about it [3]!
[1] http://lists.xenproject.org/archives/html/mirageos-devel/201...
[2] http://lists.xenproject.org/archives/html/mirageos-devel/201...
[3] http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageo...
https://github.com/docker/docker#user-content-better-than-vm...
I probably got it wrong but I'd love to hear a more expert opinion on why/why not this could be used for security..
[0] http://www.zerovm.org/
https://www.fossil-scm.org/index.html/doc/trunk/www/fileform...
Using that you can store a whole filesystem at the URL instead of one file, and you also get synchronization mechanisms built in, so your endpoints can communicate with each other in a structured way that you control.. vs the undefined communication you will get between VM's if you try this with the VM architecture.