I've been planning to implement NixOS tests based on virtiofs instead of 9p for a while, but haven't got it working yet (at the time I tried, the available uid mapping strategies didn't cover what was needed for NixOS tests; there's a patch which I haven't got around to trying out yet though.) and this big improvement takes the pain away a bit so my motivation to finish that off is lower. I'll be sure to write about it if I do though!
It's clearly a good article otherwise. We've changed the title now—if there's a more representative phrase from the article itself, we can change it again.
This is awesome, thanks for sharing so much detail. Btw one thing I noticed, the link to "Determinate Systems" was broken. I think this is the correct link: https://determinate.systems/posts/qemu-fix
From commit:
> The previous implementation would iterate over the fid table for lookup operations, resulting in an operation with O(n) complexity on the number of open files and poor cache locality -- for every open, stat, read, write, etc operation.
Windows's file handling also is very slow at times (especially Windows Explorer has frustrating UX).
We can ask MS to release all their source and we will then fix all O(n) issues for them.
It doesn't actually go through a "real network" at all, the VM speaks 9p to the host via a virtio pipe (not sure what the technically correct term for this is) so there's none of the overhead that you'd have with emulating a network device and encapsulating it all in Ethernet/IP/TCP.
For those who haven't used patches via email before and want to contribute to Linux/qemu/etc that use that, there is a neat site about how to set it up and do it:
Nice story! It also matches my experience with many open source projects. It is not enough to solve a technical problem, you must convince other people to accept the solution.
Having a nice maintainer to guide you through the process is really encouraging and brings more nice and bright people to the comunity. Kudos
Let me preface this comment by mentioning that I've not written a single line of C in my time as a software engineer.
Why would C cease to be lightweight if it added default libraries with lots of capability?
I think this kind of library should generally be in user space, not standard lib as "standard lib is where libraries go to die" (they can't really be modified after because of backward compatibility) but I'm confused about your argument.
My guess: in C++ it's templated and header-only, which means optimized hash table code gets generated for the types you use it for, and you don't pay for it in binary size when you don't use it. In C you get stuff like qsort which is a single symbol in the libc binary, to which you have to pass a comparison function as a pointer, which is awkward and slow because it isn't inlined (unless you do link-time optimization, do people really?).
I frankly don't understand why projects [*] are still written in pure C. With the exception of a few edge cases, C is pretty much a subset of C++, so you can easily write code that's 100% C, except for when useful data structures from the STL are required. In this specific case, what is the disadvantage of writing C code that pulls in std::unordered_map when a hash table is needed, but otherwise does not use any other C++ language features?
[*] aside from those targeting niche, embedded systems for which C++ is unavailable
FWIW, the contemporary alternative to 9p is virtiofs. It is implemented as a separate service that is running aside the hypervisor. Originally out of qemu, it was rewritten in Rust as a freestanding project that can also be used with cloud-hypervisor and crosvm.
31 comments
[ 4.6 ms ] story [ 73.8 ms ] threadSubmitters: "Please use the original title, unless it is misleading or linkbait;" - this one was linkbait, so should not have been used. https://news.ycombinator.com/newsguidelines.html
For example I went to check and in crosvm we use a BTreeMap already for Fids for our p9 implementation (thankfully): https://github.com/google/crosvm/blob/main/common/p9/src/ser...
On WSL you would be using the “hyperv 9p server”.
Same Linux 9p client in both cases but that’s not where the fix was.
Reasonable question but the answer is still no.
Windows's file handling also is very slow at times (especially Windows Explorer has frustrating UX).
We can ask MS to release all their source and we will then fix all O(n) issues for them.
You have any suggestions for getting a foothold in the QEMU codebase?
https://git-send-email.io/
To each their own, I just don’t understand why anyone would prefer this method.
Having a nice maintainer to guide you through the process is really encouraging and brings more nice and bright people to the comunity. Kudos
I wonder how many C projects use linear scans instead of hash maps simply because C doesn't have it in its standard library.
It almost feels like the people who keep churning out new standards of C just gave up and decided that part of it couldn't be improved any further.
Why would C cease to be lightweight if it added default libraries with lots of capability?
I think this kind of library should generally be in user space, not standard lib as "standard lib is where libraries go to die" (they can't really be modified after because of backward compatibility) but I'm confused about your argument.
[*] aside from those targeting niche, embedded systems for which C++ is unavailable
https://gitlab.com/virtio-fs/virtiofsd
On NixOS: https://github.com/astro/microvm.nix