HN Explorer

61 comments

[ 4.9 ms ] story [ 72.9 ms ] thread
staticassertion
For those using firecracker, what's your method for building a rootfs?
lgierth
I build them using Buildah, then mkfs.ext4, mount, cp, umount. It's a little bit annoying that I'm still using root at least for the mount part.

OpenWrt's build system has a method of building rootfs ext4 and squashfs images without any root, it's somewhere in that large Makefile mess.

(sorry, reposting this as I first replied to the wrong parent)

mikepurvis
I similarly have built bootable disk images with various tools including buildah and have never been able to fully get away from needing root for various chrooty/loopbacky parts of the process. In principle, it should be very possible to point grub at a filesystem-in-a-file and be like "install to that", but I could never make it happen; it always wanted to be trying to infer things about how to configure itself from examining the host system.

And yes, I've studied the OpenWRT build to no avail. I would be delighted for someone to dissect whatever it is that goes on in there and write it up.

bminor13
No experience with Firecracker specifically, but if squashfs images are sufficient, one should be able to build a tar archive of the filesystem without root (where all the files have the correct owners, mode, etc.) and then convert it to squashfs using `tar2sqfs` in https://github.com/AgentD/squashfs-tools-ng, also without root - I've done something like this to create squashfs images in constrained build environments, which worked well.
rad_gruchalski
Directly from Docker image: https://github.com/combust-labs/firebuild#high-level-example. I’m the author.
ignoramous
nb, building from docker is also what fly.io does: https://news.ycombinator.com/item?id=22514017
rad_gruchalski
Yes, I can also build from Dockerfile:

    sudo $GOPATH/bin/firebuild rootfs \
      --profile=standard \
      --dockerfile=git+https://github.com/hashicorp/docker-consul.git:/0.X/Dockerfile \
      --cni-network-name=machine-builds \
      --ssh-user=alpine \
      --vmlinux-id=vmlinux-v5.8 \
      --tag=combust-labs/consul:1.9.4
That SSH bit is no longer required as I'm using MMDS instead.
gavinray
This is incredibly interesting. Is it alright if I email you with some questions?
sudhirj
Is there something new that prompted this posting?
capableweb
If I were to guess, I'd guess because Firecracker is something "that good hackers would find interesting" https://news.ycombinator.com/newsguidelines.html
jrk
I think the question is raised because Firecracker was released years ago and has been posted and discussed many times before.
tlrobinson
“Are reposts ok?”

“If a story has not had significant attention in the last year or so, a small number of reposts is ok. Otherwise we bury reposts as duplicates.”

https://news.ycombinator.com/newsfaq.html

weikju
I think the question is more along the lines of - was there another news, recently posted, that prompted this post?

Like for example if fly.io was announced yesterday, and they said they used Firecracker, then someone would submit this link about Firecracker.

So the original poster in this thread was wondering if there was something to that effect. Not decrying something being posted that was not new or already discussed before...

moondev
Would be interesting to see a project that unveils these "chain of influences" for tracking HN engagement on stories or even comments
notatoad
as long as we're quoting the rules, "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

assume that the parent's not saying "you shouldn't have posted this, it's old", but rather "i'm interested in firecracker, was there some new development i missed that prompted this submission"... if the answer is no that's okay, but if the answer is yes i'd love to hear about it.

sudhirj
Yeah, I’m actually curious if there’s a new major release or feature launch that I missed. I keep tabs on the project, didn’t think anything new suddenly happened.
m00dy
anyone knows this can create gpu-enabled microvms ?
lgierth
No support for PCI devices at the moment, but I believe it's planned for some point in the future.
tptacek
This is tricky for Firecracker, both because of security (and the attendant switching costs of keeping exactly one VM connected to a GPU at a time) and because of the Firecracker tenet of supporting oversubscription. They've been talking about it for a couple years. I think they're actually talking about forking off a GPU-capable variant of the project to work around the drama.
sudhirj
Best example of current usage that comes to mind (outside of AWS’s own Lambda & Fargate) is Fly.io - they they use Firecracker for all the containers.
rad_gruchalski
There’s also Koyeb. I’m not affiliated.
slimsag
We also use it at Sourcegraph (to run arbitrary LSIF code indexers in a sandboxed environment)
paxys
Does anyone know of any good orchestrators for Firecracker VMs? While the technology is great, the CLI and API are still a bit too complex and low-level to implement some basic use cases in production.
badLiveware
Kubernetes using Kata containers[1] as a containerd[2] backend

[1] https://aws.amazon.com/blogs/opensource/kata-containers-1-5-...

[2] https://github.com/kata-containers/kata-containers/blob/main...

adolph
How does that compare to firecracker-containerd?

https://github.com/firecracker-microvm/firecracker-container...

This repository enables the use of a container runtime, containerd, to manage Firecracker microVMs. Like traditional containers, Firecracker microVMs offer fast start-up and shut-down and minimal overhead. Unlike traditional containers, however, they can provide an additional layer of isolation via the KVM hypervisor.

NathanKP
Take a look at Weaveworks Ignite for an open source tool for running Firecracker VMs from container images: https://github.com/weaveworks/ignite

And of course AWS Fargate and Lambda use this tech under the hood transparently, so that's always an option if you don't want to host and operate it yourself.

lgierth
Ignite looked intruiging when I checked it out recently - but I need to import rootfs tarballs directly, without going through any registries. Any helpful pointers are appreciated :)
gnufx
Depending on what you want, OpenNebula (opennebula.io) has been making a thing of Firecracker.
unixhero
Aw geez. Seems like we're going to need a Firecracker awesomelist now then!
imachine1980_
what is the advantage of server-less if you have to have to manage the servers?
jjice
This is more meant for providers. Best example being AWS's serverless offerings, as well as fly.io.
staticassertion
If you're hosting someone else's code.
hinkley
If you work at a large enough company it all starts to look like someone else’s code.

The malicious person can always find the worst way to break everything, but dumb luck can get you pretty close.

gizdan
Anyone any thought on using this in a homelab (just random servers such as laptops and microservers) with K8s vs normal containers?
andrewstuart
Note that this Amazon developed technology does not work on Amazon Web Services instances.

It does however work on Google, Azure and Digital Ocean instances where you can configure nested virtualisation.

cptnntsoobv
It works on AWS's metal instances, doesn't it?

That is definitely a constraint though, as the smallest available metal instances are pretty large, hence costly on a per instance basis.

oneearedrabbit
Technically speaking you are right. If you are not limited to AWS, then even a $5 DigitalOcean instance is be able to host 4-5 VMs.
cutemonster
> does not work on Amazon Web Services instances

Why not? And what is Google, Azure and DI doing differently

jsolson
Firecracker relies on KVM in the Linux kernel which relies on VT-x or similar. Offering VT-x inside a VM that's already using VT-x is called nested virtualization. EC2 VMs do not support nested virtualization, while others do.

That said, it works fine on EC2 Bare Metal instances.

(n.b., I work on the virtualization stack on Google Compute Engine)

bushbaba
Wouldn’t you want to run firecracker on bare metal anyways. Nested VMs come with a decent perf hit.
cutemonster
Thanks @jsolson for explaining

@bushbaba I wonder how large the percentage hit could be, if it's like 1% or 10%, if you happen to know

And how that percentage hit compares with gvisor performance wise

paxys
It does work on AWS on the instances that support nested virtualization, same as every other provider.
mastax
I just deployed a WordPress container on the same host that runs everything else. Makes me a bit uncomfortable. Was thinking about running a VM just as a Docker Swarm worker node, but this post reminded me that these virtualization runtimes exist. Looks like there's a drop-in containerd replacement using firecracker. Kata containers look a bit better supported, though?

Or am I too paranoid about container escapes?

throwaway984393
What are you worried about them doing? If they take over the app inside the container, they can access your database and seed malware in your users' browsers.
staticassertion
I don't think you can be too paranoid about container escapes. We actually don't consider containers to be a reliable security boundary - we take it where we can, but if we need real isolation it has to be combined with another approach.