Launch HN: Nestybox (YC S20) – Containers beyond microservices
This is Cesar Talledo and Rodny Molina, co-founders of Nestybox (www.nestybox.com).
Nestybox has developed a new container runtime that sits under Docker/containerd (it's a new type of runc) and enables containers to act as virtual-servers capable of running software such as systemd, Docker, and Kubernetes, easily and with proper isolation.
The motivation came from noticing that containers are great at running microservices but struggle to run system-level software in them such as those mentioned above. That is, in order to run such software in a container, we needed unsecure privileged containers with complex images, custom entrypoints, volume mounts, etc., or alternatively a heavier virtual machine. This did not seem right.
We studied the problem and noticed that the container abstraction was not complete enough, meaning that inside the container a root process lacked capabilities to perform certain low-level operations, the namespacing of procfs and sysfs had a few holes, there are limitations for running overlayfs-on-overlayfs, and more.
To solve this, we decided to create a new container runtime that would set up the container in such a way that it could run system software easily and without resorting to privileged containers. That is, a user should be able to do "docker run -it some-image" and get a container inside of which she can run systemd, dockerd, or even K8s without problem (much as if it were a virtual machine).
After lots of long days, we came up with Sysbox. It's a new type of "runc" and sits below OCI-based container managers (e.g., Docker/containerd). You typically don't interact with Sysbox directly, but rather use Docker (or similar) to launch the containers. Sysbox was forked from the excellent OCI runc in early 2019 and has undergone significant changes since then. It uses OS virtualization techniques such as always enabling the Linux user namespace, uid shifting via shiftfs, partial virtualization of procfs and sysfs, selective syscall trapping in user-space, setting up special mounts into the container, and more. It's written in Go.
Here is a video: https://asciinema.org/a/kkTmOxl8DhEZiM2fLZNFlYzbo?speed=1.75
Today we are happy to announce that we are open-sourcing Sysbox (Apache 2.0). You can find it at https://github.com/nestybox/sysbox . We welcome users and contributors, as it has plenty of room to grow and improve. There are plenty of docs in the repo describing how to use it and how it works.
We think Sysbox is a very useful tool to expand the use cases for containers and provides an alternative to virtual machines in many scenarios, particularly for dev environments, testing, CI/CD, and even running legacy apps in containers.
In order to pay the bills, Nestybox (the company we founded) will sell a version of Sysbox called Sysbox Enterprise Edition (Sysbox-EE). We are using an open-core model, such that Sysbox-EE is based on the open-source Sysbox and adds a layer of proprietary enterprise level features. We think this model will help us strike a healthy balance between creating useful technology that all can benefit from and keeping the lights on.
Thanks for reading and we welcome your feedback.
Best, -Cesar & Rodny
111 comments
[ 2.8 ms ] story [ 209 ms ] threadIs it wise to go through all the effort and risk of starting a business to prove demand for a feature that an existing established product will then add, removing the need for your company?
(Note this question is not a general critique of all startup ideas, it's specific to startups launching to address missing features of established products)
It would be good to be able to assign physical network interfaces into the containers(using network namespaces) and also document how to create virtual network devices shared between containers.
Furthermore it would be awesome to make all this numa-aware. So you could have network interfaces and cpus and memory be assigned to a single container where everything is numa local. Then you could break up a single physical box into high performance independent domains.
The numa-aware idea would take some more research on our part, but at first glance looks like something that we could definitely explore.
- Run a distro including system software, systemd etc. as a container
- Run unprivileged, using uid-mapping to provide root and other system uids inside the container?
I've been using LXC and LXD for years to run system images as containers, even migrated some real machines to containers this way.
Note on modern CPUs, each cpu is further broken down into numa nodes(numactl -H and numa stuff n lscpu)...perf degradation isn't as great going between chiplets, but it's measurable(2x reduction in ram bandwidth?).
Also, I think the goal is different: I understand Firecracker is meant as a way of strengthening the isolation of containers by wrapping them in micro-VMs. Sysbox is meant as way of enabling containers to run system workloads without complex images, entrypoints, volume mounts, etc., and with proper isolation via the Linux user-namespace.
Is there any overlap here even though your project seems to go the path of adding more functionality? Is it fair to think of NestyBox as a type of sandbox?
https://pkg.go.dev/github.com/google/gvisor/runsc?tab=overvi...
if you can do this (and build a great experience around it), you have a winner.
k3d.io does it somewhat...but not all the way. Updates,rollbacks, etc - everything that a sysadmin need.
Here is a demo video: https://asciinema.org/a/V1UFSxz6JHb3rdHpGrnjefFIt?speed=1.75
Having said this, K8s is very complex and while its most common functionality works inside the container, we've not yet tested it all.
Here is a doc that describes this in more detail, including what is supported and not supported at this time: https://github.com/nestybox/sysbox/blob/master/docs/user-gui...
You can find kindbox here if you are curious: https://github.com/nestybox/kindbox
Having said that, we have certain limitations at the moment (e.g. we don't run all cni's), but we are not relying on priv containers as i believe is the case for existing K8s-in-docker solutions (pls correct me if i'm wrong in k3d's case).
Btw, we haven't tested k3d.io & sysbox interaction yet (part of our roadmap), but i believe we should be able to interoperate after making some minor adjustments. That would allow existing k3d users to operate in a more secure environment.
> if you can do this (and build a great experience around it), you have a winner.
Genuine question. I'm not that familiar with Kubernetes. Can you explain the pain this solves? What is this winning at?
Are there any performance implications using this custom runtime?
Have you used this in production systems?
Are there any known limitations of using sysbox?
As far as using this in production, the software is well tested but has not been used in production to the best of our knowledge. It has room to mature still, both in term of functionality and security, but it's in pretty good shape already. We hope open-sourcing it allows it to mature it faster.
And as far as limitations, there are a few, here is a list: https://github.com/nestybox/sysbox/blob/master/docs/user-gui... . We hope to remove some of these as the product matures too.
We are out of the critical path, meaning we only emulate interactions with procfs / sysfs, and we only intercept mount syscalls at the moment, so we don't see any tangible performance hit. Having said that, we haven't done large scaling&perf tests yet.
Thanks for the response!
Docker is missing a bunch of features that make some software work, which is why you can't run Docker inside Docker by default. Instead of dropping from containers all the way down to hardware virtualization, Sysbox is "augmenting" containers with the missing features by simulating them in userland. That gives you all the power of a VM, without any of the downside of slow start-up speed, provisioning blocks of memory, not being able to run them on EC2, etcetc.
It reminds me a bit of user-mode Linux [0], weirdly. There's something kinda interesting about simulating a bunch of the kernel in userland.
[0] https://en.wikipedia.org/wiki/User-mode_Linux
What use case do you envision for running sysbox in sysbox?
This matters because it adds cognitive overhead - I have to keep track of which features are available at which layer.
But i also see 'production' scenarios. Think about having a reverse-proxy system container, inside of which you host the backend applications, which happen to be docker containers.
In essence, Sysbox allows you to create hierarchies of containers in a secured fashion, without the need to abuse 'sidecar' pattern, nor priv containers.
Note that Sysbox currently requires Ubuntu Linux, because the latest Ubuntu releases use pretty new kernels and carry kernel patches that Sysbox relies on in order to perform some of the OS-virtualization in userland. See this doc for the distros supported by Sysbox:
https://github.com/nestybox/sysbox/blob/master/docs/distro-c...
We are actively working on adding support for more distros.
https://github.com/nestybox/sysbox/blob/master/docs/quicksta...
It has an article on Docker sandboxing. Hope that helps!
(If I remember correctly you can run systemd in a podman container and/or run a podman container with systemd.
Sysbox, on the other hand, acts as a low-level runtime (same as runc), so we could potentially integrate with Podman too. In fact, we could _potentially_ integrate with anything that speaks OCI spec.
Having said that, we are not there yet, as for example sysbox wouldn't work with Podman in rootless mode right now; it should work in regular mode though, but we haven't tried it yet. If we accomplish this, it would allow podman to launch a larger set of applications too, same as we are doing for Docker.
I'm trying to get a better sense of how this approach differs technically from rootless Docker / usernetes. I understand that it's not there yet, for many reasons, and I see your FAQ about it, but it's clearly working towards the same goal, right?
I think what's going on is that you depend on shiftfs from Ubuntu, and SECCOMP_RET_USER_NOTIF (or something?), unprivileged user namespaces, cgroup namespaces, etc. from the upstream kernel, but the major missing parts in the upstream kernel are procfs and sysfs virtualization and making shifts feel se amless, and so you've written a syscall trapper and a FUSE filesystem that run on the host and emulate the things you need. Is that approximately right?
If so, I'd be really curious whether you see a path to get onto upstream runc at some future point. It seems like you'd need shiftfs to be upstreamed, but if an unprivileged procfs2 + sysfs2 shows up upstream, I think you can use that? And you'd probably fit in at approximately the place something like vpnkit fits in for managing shiftfs?
I have a use case for this sort of thing at work, and we've been exploring rootless Docker and unprivileged containers a bit. I"m trying to get a sense of why to prefer Sysbox EE instead of waiting for (or, ideally, contributing to) upstream support for namespaced procfs/sysfs, for shiftfs, and for properly teaching Kubernetes about user namespaces. I suppose the answer is that your solution works right now, and upstream support might take years?
I guess that puts you in a position much like OpenVZ and even LXC itself, which both had significant out-of-tree code in years past and seemed to be decently successful businesses as stuff slowly got upstreamed.
It seems like the major benefits of Sysbox EE are paid support and not using the same uid_map for each container?
I don't think rootless approach is fully aligned with what we're doing right now. True, we both rely on user-namespaces, and we both emphasize the security angle, but our goal is to expand the number of applications/functionality that can run in containers, which is something rootless approach may struggle with for some time.
Please see here if you haven't done it yet: https://github.com/nestybox/sysbox#sysbox-is-not-rootless-do...
In regards to our dependencies, we can operate with or without shiftfs. In both cases user-namespaces are always utilized. The rest of your approximation is correct: we need most of what you mentioned in your second paragraph, which btw, is already there (thanks to Canonical/LXD folks) starting in Ubuntu 5.0+ and 5.5+ for other distros. As you know, shiftfs is only present in Ubuntu at the moment, but as i said, we can live without it.
Which leads me to your question: why would you wait if the functionality you're after is already there? If having dockerd running as an unprivileged user is not a real must-have for you, then Sysbox provides a fairly secure solution while giving you all the functionality.
Sorry, i'm not familiarized with vpnkit yet, will take a look.
Correct, those are some of the benefits Sysbox-EE offers at the moment. That, plus efficiency & scalability features and hardened testing.
Thanks a lot for your detailed feedback @geofft. Please ping us on slack anytime.
Will try to remember to join the Slack next week, this is definitely a cool project :)
One thing I've noticed is that in modern IT infrastructure, there is usually two levels of sandboxing going on. At the low level you have VMs (sandboxed OS), and on top of it you have containers (sandboxed applications). Sysbox makes it easy replace that lower level with containers (which naturally leads to docker-in-docker or more accurately containers-in-containers).
To be clear I am not saying that containers are equivalent to VMs or that containers should always replace VMs. They are different beasts with different properties. But I am saying that in many scenarios it does make sense to use containers instead of VMs, particularly if your stack is all Linux, you don't need the isolation strength provided by VMs, and want the higher efficiency of the container.
https://katacontainers.io/
Also, Sysbox is a pure OS-virtualization technology, which means it runs in environments where hardware virtualization is not available (e.g., a cloud VM, since most cloud providers don't allow nested virtualization).
David.
The containers created by Sysbox act like virtual-hosts, so it makes sense to have the ability to expose GPUs / hw-accelerators within them. But container nesting, which comes naturally with Sysbox, would introduce another challenge since the GPUs would not just need to be passed to the outer system container, but also to the inner application containers.