I've always preferred FreeBSD over Linux, especially for servers, but the lack of hardware support has always been a hurdle. And Linux has, overtime managed to reach a kind of feature parity with it and even outperform it as many of these tests show.
Linux's decentralization is both its major strength and its major weakness. The differences between distros can be so large that you might as well be talking about two entirely different operating system. There is little consistency in design of tools. Many tools used to follow the UNIX Philosophy ("do one thing well"), but that no longer seems to be the case.
We (as in Linux users) used to make fun of Windows for how stupid everything was. You need a GUI for configuration because many options are hidden away in some kind of cryptic registry, deploying services is a PITA etc. Now we have somehow (d)evolved to that same mess. For instance, SystemD and SELinux are such a pain to work with (each for different reasons) that it feels like I have to fight my OS. Wanting an OS that works for me, instead of against me, was one of the major reasons why I switched from Windows to BSD and then Linux back in the 90s.
There's definitely something to be said for *BSD's boring, predictable simplicity. But perhaps there's some kind of force that automatically drives operating systems to bewildering compelxity as their user base grows?
I don't understand the hate systemd gets honestly, except around some very corner concerns. I remember the old init systems and don't want to go back. Piles of shell scripts with absolutely no standardization, each doing its own thing with zero consistency. Systemd has massively cleaned up that mess and yes it's more complex, but also more sane. I don't find the FreeBSD init an improvement.
Nobody is forcing SELinux on you, you can turn it off and that's that.
If you're on a Red Hat based system most of policy is there out of the box, providing you with pretty nice additional protection without too much tinkering.
> I don't understand the hate systemd gets honestly, except around some very corner concerns. I remember the old init systems and don't want to go back.
systemd-as-init is/was not the problem. systemd-as-kitchen-sink is where people started getting annoyed.
It needs journald (which can't ship logs off-host like rsyslog, so you needed to run rsyslog anyway), it pulled in udevd, it's doing DNS, it's doing time syncing, it's doing DHCP/IP and VXLAN(!), it has a boot manager (formerly gummiboot), it's tracking user logins, it's ….
Systemd doesn't feel more sane. To give a silly example: the other day I upgraded a Fedora box, and suddenly DNS wasn't resolving short names (foo, instead of foo.example.com). This is something that has worked Just Fine for 25 years, run an upgrade, and bam, it stops working. I wasted a lot of time figuring out that systemd was to blame in the first place, after all, why the hell would systemd be involved in resolving DNS? Well, apaprently systemd-resolved is now a thing. And as is the case with anything systemd related, configuration is a PITA.
Maybe I'm just getting old and change is starting to grate on me. But systemd does feel like everything-and-the-kitchen-sink.
As for SELinux, sure, you can turn it off. At which point it's useless. The pain begins when you actually want to use it and existing policies aren't enough. It is hugely complex. Tooling (like ausearch and audit2allow) is, erhm, not very good. Want to allow something non standard? Better be prepared to spend a couple of hours. Don't forget to turn your new policy into a module file. And then compile that into a pp file, which is another kind of module? And then don't forget to install that file. I remember administering Windows NT and 2K boxen and really hating how its policies worked, well, SELinux now makes me feel the same.
Both of these tools make a lot assumptions about how things should be done, and they're pushing these assumptions onto the user, whether or not they agree.
And hey, I get it, these are complex domains, and we should be very happy that people are giving us Free (both sense) Software in the first place. But boy, if I could pay €100 to have a sensible operating system, I would. Just not €100 per machine -_-
There are many cases where it follows some course of behavior that doesn't follow typical user patterns like the principle of least surprise. Example is the dns/resolv.conf issue.
I also didn't understand the hate for systemd until I have attempted to use it for the first time !
This was a few years ago, so it is very likely that the particular bug that annoyed me was solved since then. Nevertheless, the bug was so unbelievably stupid and it was not only just a coding error but also a problem caused by the wrong architecture of systemd, so I could not accept that it is possible to trust software developers that can make such mistakes.
After that bad experience, I have never touched systemd again.
Even if this was several years ago, systemd had already been in use for several years, so the bug could not be excused by systemd being experimental or something like that.
I use mostly Gentoo Linux, which fortunately does not use systemd, so I had been shielded from its problems. I had heard about some systemd-related controversies, but I had paid no attention to them. I had seen some systemd presentations, which seemed reasonable enough.
One day I was in a hurry, but I had to install Linux on a new computer. Gentoo is nice, but if you do not have a pre-prepared customized installation image, so that you have to install it from scratch, it may need a few hours for compiling everything from sources and for editing the configuration.
So I decided to install Arch Linux, which used systemd. The installation was fast and everything seemed to work OK.
Systemd was always praised for its supposedly fast boot times, but I have seen no improvement in that, compared to my carefully configured Gentoo systems. On the other hand, the computer with systemd was the first computer that I have ever encountered, among many thousands of computers, from the days of Intel 8080 and until the new Zen 3, which was slow at shutdown.
So while the boot time of systemd was OK, the shutdown time was very long, sometimes more than a minute. What is much worse, sometimes the shutdown failed, which is something that I have never experienced before or after. This was on an Intel NUC with a Skylake CPU, so it was not some esoteric, non-standard hardware.
The shutdown failed sometimes due to a race condition, because the systemd daemons exchanged some messages between themselves through dbus, and, IIRC, sometimes it happened that some daemon died before the sender expected and the sender remained stuck in attempts to send to a non-existing recipient, or maybe the dbus daemon died, so there no longer was who to relay the message.
Race conditions may happen in concurrent programming, but to have a shutdown procedure whose success depends on the success of inter-process communication, that shows that everything was ill-conceived at the highest level.
So then I said farewell to systemd, I wiped Arch and I installed Gentoo, which worked perfectly on that hardware, without systemd, but with its much simpler and more reliable openrc.
Yes, from the perspective of a normie desktop user this was (is?) a common reason for disliking it. And yet every time it's mentioned online the only replies are along the lines of 'it's intentional, of course it makes sense to wait in case the service exits cleanly'. Was no one around before systemd to realise this was not an issue then?
It's infuriating when your system won't shut down due to some unknown service, which systemd gives [30 seconds remaining] to quit gracefully. OK, you think, I guess it's reasonable to wait and see if it will close cleanly, so you wait. As soon as the timer hits 0, it jumps up again: [90 seconds remaining]. How are you supposed to know how long to wait? What are you supposed to do if it continues forever? All keyboard input is disabled so there is nothing you can do but hard reset your PC and hope systemd unmounted the filesystems before getting stuck.
Boring? Come over to plan 9. We have cake, 9p and Doom. You are not alone in your disdain for what has happened to Linux. I even mentioned this the other day to a degree: https://news.ycombinator.com/item?id=25570712
The big problem is simple interfaces require more effort on the users part which will never happen for a majority of users. And lets face it, people like me, you and most others here are the extreme minority in computing. We aren't afraid of $ vi /etc/config or echo killuser >/n/pcturret/ctl. But writing documentation is boring and "hard" so it never gets done. People start complaining about docs and someone comes up with the bright idea to automate vi /etc/config by replacing it with a poorly thought out contraption that needs binaries and bolted on protocols. That's exactly what dbus is and is best described as duck tape engineering.
> And lets face it, people like me, you and most others here are the extreme minority in computing.
Yesterday one of KDE devs posted an article mentioning this sort of phenomenon and linked to an article depicting it in full[1]. And now that you mention Plan 9 it makes me think, because I always dreamt about Plan9 taking over the world and becoming big and everyone using it, but Plan9 is so great because it has never left the 'geek' circle, while others like Linux (and arguably the BSDs, to a lesser extent) already had.
It's all ego stroking and github stars. The psychopaths write code for the sake of building an ego instead of better software. The only people they are helping is themselves.
This is why the 9front/cat-v crowd is cantankerous and the humour as dark and cringy as possible. No likes. No stars. No instagram accounts full of pictures some jerk took in the middle of the sahara to show how shallow they are. We only want well thought out code. Otherwise go away.
Inferno is what modern browsers and mobile OS's wish they were. I'd even venture far and say that If someone were to build a modern browser for plan 9, I'd want it built inside of inferno out of pieces connected and isolated via 9p/namespaces and ran as an application (and bare metal on a tablet). Dis is similar to WASM so you deliver web apps as actual applications. Done.
Too bad it's stuck with hard coded 32 bit dependency. It also cant build on 64 bit systems unless they have 32bit compat (windows/linux win here). There's a fork maintained by some 9front users called Purgatorio. Charles Forsyth also still pokes at it and just added Risc-V support. It's not dead, just resting. Patches welcome.
Boring is a feature. For an OS it is THE feature. An OS is a necessary evil. It doesn't actually complete any user tasks, much like a car's engine doesn't do the actual driving. An OS should take the low-level tasks assigned to it, complete them without issue, not do anything unexpected, and generally stay out of the user's way. A car engine's job is to start and stop when asked, not to organize the driver's inbox. BSD has remained true to this while Linux has moved away in recent years.
On the flip side it windows. That OS doesn't hide in the background, rather it screams for attention 24/7, inserting itself in every little task even when specifically told to back off.
There are some cases where squeezing every ounce of performance out of the hardware matters, but for most use cases there is a reasonable "good enough"; and these results show that BSD and Linux are comparable. I think the killer feature that the BSDs have, which Linux can't match, is how well designed, integrated, predictable, and simple they are. That is why I run more BSD servers than Linux servers. I do run Linux servers for some cases, and they work well for those cases, but I much prefer working with BSD.
I like the BSD simplicity, but it's not just hardware, it's software. I want to spin up an X stack in docker? Can't do that, docker doesn't work. There's jails but without any image repositories I'm trading a lot of convenience for simplicity.
There's also software like dotnetcore which will just not build. The situation in the desktop is even more problematic. I wish it wasn't so, I like the separation of concerns quite a bit with FreeBSD but it's not something that unfortunately fits my use cases.
Pretty sure to run a Linux docker image it uses a Linux kernel; as you know, the current WSL is implemented as a Linux kernel running in Hyper-V, i.e. a Linux VM.
I am not an expert in Windows though; perhaps you are referring to a version of docker that I'm not familiar with which doesn't support Linux services but instead runs Windows services in a native Windows container system? That would be cool and interesting but obviously it wouldn't run most docker images.
There are docker images for Windows software, and those are the ones I care about, which yes are based on Windows containers that are even capable of running directly on top of hyper-v, bypassing most of the kernel fat.
I tried to install Dragonfly BSD in a Hyper-V VM recently - didn't matter what configuration I used, the install wouldn't start due to some kind of hardware issue with the virtual disk. Oh well, I'll stick with Linux.
Btw, DragonFly runs fine under either of ESXi or Linux/KVM/Virtio.
I used it under ESXi with physical disks attached as a backup server with hammer/hammer2 recently. Worked great and it was the only FS that offered effective offline dedup.
+1. It's also my recent experience of running FreeBSD on AMD Ryzen, several features are missing. And so far this is a list of problems I've encountered.
1. No ECC driver for AMD Ryzen, and no ECC support in memory management.
While you don't need any OS support to use ECC, it's much better to have it.
* Linux has an ECC kernel driver for AMD CPUs that accurately reports the ECC status, this is extremely helpful given the uncertainty of ECC in consumer-grade hardware. On FreeBSD, it's not possible to tell whether ECC is enabled due to the lack of driver. Also, if ECC is not initialized properly by the firmware, the Linux ECC driver will also detect it and try reenabling ECC and memory scrubbing. On FreeBSD, better to check whether your BIOS is broken.
* When an ECC error occurs, a Machine Check Exception is generated by the CPU. On Linux, it will be correctly decoded and recorded. But on FreeBSD, so far there's no decoder, leaving you a mysterious MCA error in dmesg. For example, a correctable DRAM ECC error will be reported as "L3 cache error" (which made many people to falsely believe that Ryzen's ECC was not working on FreeBSD).
* On Linux, if an unrecoverable multibit ECC error is detected in userspace memory, the process will be killed by a SIGBUS (my username checks out?). If the error occurs in kernel memory, it triggers a kernel panic. On FreeBSD, as far as I know, none of the features is supported by the kernel, it increases the risk of data corruption, and it's not a Ryzen or MCE-specific problem. Also, Linux also supports memory offlining, which can be used to temporarily remove bad memory pages from the system after an ECC error, which is not found in FreeBSD.
Fortunately the ECC-enabled flag is easy to check (I plan to make a contribution to the FreeBSD kernel), and a MCE decoder in userspace should also be easy. But implementing the rest is not trivial, the lack of ECC support in memory management is a problem. It's unfortunately that these features are not supported in a server-grade OS like FreeBSD (I guess FreeBSD runs on "real" servers, which implements all of those ECC features in the management firmware, so "real" FreeBSD users don't really need them...)
2. No USB serial system console and netconsole support. On FreeBSD, although tty login via USB serial is supported (just run a getty on it), it seems that there's no way to redirect all dmesg to a serial port (perhaps someone can correct me), FreeBSD's comconsole requires a real physical serial port, not USB, on the other hand Linux's CONFIG_USB_SERIAL_CONSOLE supports USB natively. On Linux, there's also a kernel-space netconsole that redirects all demsg via UDP/IP, this feature is not found in FreeBSD although there was an attempt in 2012.
I ran thousands of FreeBSD servers, with ECC, and the memory management layer features would have been nice, but unrecoverable failures were rare enough that it wasn't a big deal; anyway our application ran as a single OS process using almost all the ram, so 95% chance the faulty address was OS or our application. Offlining would have been nice though; had a couple faulty addresses that were busy, so all the MCE interrupts made the system really slow, and caused a ton of application congestion.
For USB serial console, I think part of the issue there is that the serial console can interface with the kernel debugger, which might be tricky if it's USB (but I guess USB keyboards probably work in the kernel debugger, so maybe there's a chance?).
Thanks for working on Ryzen ECC. I don't work with a server farm anymore, and I'm too cheap to buy ECC for home use, but someday I might use that support.
Yes, unfortunately FreeBSD has much better support for Intel CPUs and for Intel networking, than for alternatives.
On servers with Intel CPUs, FreeBSD warned me promptly, by e-mail, about some corrected ECC errors.
The errors appeared suddenly after many years of use and they kept recurring, pointing to a probable failure in the near future, but being promptly notified, I have replaced the offending DIMM and all went back to normal.
So yes, you are right, for FreeBSD it is better to stick to Intel, unless you have time to change yourself the sources, to add the missing hardware support, which is somewhat easier than attempting to make changes in the Linux kernel.
There are also examples of hardware for which FreeBSD has better support than Linux, but those belong to classes of hardware less likely to be encountered by non-professional users.
For example FreeBSD has better support for magnetic tapes.
> So yes, you are right, for FreeBSD it is better to stick to Intel, unless you have time to change yourself the sources, to add the missing hardware support, which is somewhat easier than attempting to make changes in the Linux kernel.
This is not just an AMD problem. For example, as I already said, there is no support of unrecoverable ECC errors handling in the FreeBSD memory management subsystem for any CPU, including Intel. On Linux, the kernel has first-class ECC support via the Error Detection And Correction subsystem (EDAC). EDAC can decide to kill a process, to trigger a kernel panic, or even to isolate the faulty memory based on the location of the memory error. On FreeBSD, the only thing that handles unrecoverable ECC errors is the userspace utility sysutils/mcelog [0], the best thing it could do after an unrecoverable error is sending you an email, at which point data corruption may have already occurred [1]. Usually correctable errors occur before unrecoverable ones, so data corruption could be prevented if it's fixed soon enough, but if you get "lucky"...
And unlike writing device drivers, I don't think most people (including myself) is qualified to mess with the memory management code in the kernel. But I have a plan to do something about it, for example, there's nothing stops me from writing a simple script to decode the MCE log and kill the kernel from the userspace. And given how rare unrecoverable ECC errors are, even without graceful handling (like killing userspace process or runtime memory offlining), it should solve my concern of data integrity on FreeBSD.
[0] But corrections are welcomed. I suspect FreeBSD will also halt the kernel on Intel platforms after a fatal MCE due to ECC error, but I didn't find any information and came to a negative conclusion.
[1] I guess it's not really a problem for "real" FreeBSD users, on "real" servers, BIOS can kill the system after an unrecoverable ECC, which probably explains the lack of any unrecoverable error handling in FreeBSD.
I hear about the elegance of BSD design often but have only worked deeply with Linux.
Does anyone have some good LWN-type articles that goes deep into the differences between BSD vs Linux internels (also common user land traits in downstream distros)?
Yes, it seems that we are finally approaching the time when waiting for multiple events and asynchronous I/O will become simple and efficient on Linux.
However, this has always been a weak spot for Linux until recently and it took too many iterations of various partial solutions of the problem to reach the current state.
This was a case when avoiding the NIH approach and looking carefully at the older Windows or FreeBSD solutions would have resulted in an earlier good solution.
IIRC the official reason for simply not porting kqueue to Linux 20 years ago was: "it's over engineered". Linux has a ton a brilliant developers working on it, they had to know epoll wasn't all there, the epoll man page says as much. I've always suspected perhaps, maybe, potential patent infringement worries might have something to do with the broken design. If that's the case, fair enough, no one wants litigation.
According to the benchmarks when using the same compiler (LLVM) and the same file system (OpenZFS) FreeBSD performance is 98% of Linux performance. 2% performance penalty is a bargain in exchange for Dtrace, kqueue, pf, jails, ports, outstanding documentation, simplicity and cohesiveness of a FreeBSD system.
Looking through the benchmarks, there's a huge variation here. For some tests, FreeBSD was much faster, and for some it was much slower.
So, it depends on what you're doing. A lot of the tests where FreeBSD was much slower were media related; it makes me wonder if some CPU acceleration didn't get enabled; which could be FreeBSD (or Clang) missing a feature, or a benchmark issue or something else.
For media encoding you are essentially benchmarking the compiler. Sometimes makefiles force -O2 on BSDs vs -O3 on Linux, which should explain performance differences when using the same compiler.
55 comments
[ 1.8 ms ] story [ 75.6 ms ] threadWe (as in Linux users) used to make fun of Windows for how stupid everything was. You need a GUI for configuration because many options are hidden away in some kind of cryptic registry, deploying services is a PITA etc. Now we have somehow (d)evolved to that same mess. For instance, SystemD and SELinux are such a pain to work with (each for different reasons) that it feels like I have to fight my OS. Wanting an OS that works for me, instead of against me, was one of the major reasons why I switched from Windows to BSD and then Linux back in the 90s.
There's definitely something to be said for *BSD's boring, predictable simplicity. But perhaps there's some kind of force that automatically drives operating systems to bewildering compelxity as their user base grows?
Nobody is forcing SELinux on you, you can turn it off and that's that.
If you're on a Red Hat based system most of policy is there out of the box, providing you with pretty nice additional protection without too much tinkering.
systemd-as-init is/was not the problem. systemd-as-kitchen-sink is where people started getting annoyed.
It needs journald (which can't ship logs off-host like rsyslog, so you needed to run rsyslog anyway), it pulled in udevd, it's doing DNS, it's doing time syncing, it's doing DHCP/IP and VXLAN(!), it has a boot manager (formerly gummiboot), it's tracking user logins, it's ….
Too. Damn. Big.
See: https://www.freedesktop.org/software/systemd/man/systemd-jou...
* https://tools.ietf.org/html/rfc5424
* https://en.wikipedia.org/wiki/Syslog
How do I send/forward/import it into my SIEM/ELK/Splunk?
I'm already using syslog for all my networking events as well as Windows logging (via a translator).
Maybe I'm just getting old and change is starting to grate on me. But systemd does feel like everything-and-the-kitchen-sink.
As for SELinux, sure, you can turn it off. At which point it's useless. The pain begins when you actually want to use it and existing policies aren't enough. It is hugely complex. Tooling (like ausearch and audit2allow) is, erhm, not very good. Want to allow something non standard? Better be prepared to spend a couple of hours. Don't forget to turn your new policy into a module file. And then compile that into a pp file, which is another kind of module? And then don't forget to install that file. I remember administering Windows NT and 2K boxen and really hating how its policies worked, well, SELinux now makes me feel the same.
Both of these tools make a lot assumptions about how things should be done, and they're pushing these assumptions onto the user, whether or not they agree.
And hey, I get it, these are complex domains, and we should be very happy that people are giving us Free (both sense) Software in the first place. But boy, if I could pay €100 to have a sensible operating system, I would. Just not €100 per machine -_-
People don't hate systemd without reason.
There are many cases where it follows some course of behavior that doesn't follow typical user patterns like the principle of least surprise. Example is the dns/resolv.conf issue.
This was a few years ago, so it is very likely that the particular bug that annoyed me was solved since then. Nevertheless, the bug was so unbelievably stupid and it was not only just a coding error but also a problem caused by the wrong architecture of systemd, so I could not accept that it is possible to trust software developers that can make such mistakes.
After that bad experience, I have never touched systemd again.
Even if this was several years ago, systemd had already been in use for several years, so the bug could not be excused by systemd being experimental or something like that.
I use mostly Gentoo Linux, which fortunately does not use systemd, so I had been shielded from its problems. I had heard about some systemd-related controversies, but I had paid no attention to them. I had seen some systemd presentations, which seemed reasonable enough.
One day I was in a hurry, but I had to install Linux on a new computer. Gentoo is nice, but if you do not have a pre-prepared customized installation image, so that you have to install it from scratch, it may need a few hours for compiling everything from sources and for editing the configuration.
So I decided to install Arch Linux, which used systemd. The installation was fast and everything seemed to work OK.
Systemd was always praised for its supposedly fast boot times, but I have seen no improvement in that, compared to my carefully configured Gentoo systems. On the other hand, the computer with systemd was the first computer that I have ever encountered, among many thousands of computers, from the days of Intel 8080 and until the new Zen 3, which was slow at shutdown.
So while the boot time of systemd was OK, the shutdown time was very long, sometimes more than a minute. What is much worse, sometimes the shutdown failed, which is something that I have never experienced before or after. This was on an Intel NUC with a Skylake CPU, so it was not some esoteric, non-standard hardware.
The shutdown failed sometimes due to a race condition, because the systemd daemons exchanged some messages between themselves through dbus, and, IIRC, sometimes it happened that some daemon died before the sender expected and the sender remained stuck in attempts to send to a non-existing recipient, or maybe the dbus daemon died, so there no longer was who to relay the message.
Race conditions may happen in concurrent programming, but to have a shutdown procedure whose success depends on the success of inter-process communication, that shows that everything was ill-conceived at the highest level.
So then I said farewell to systemd, I wiped Arch and I installed Gentoo, which worked perfectly on that hardware, without systemd, but with its much simpler and more reliable openrc.
It's infuriating when your system won't shut down due to some unknown service, which systemd gives [30 seconds remaining] to quit gracefully. OK, you think, I guess it's reasonable to wait and see if it will close cleanly, so you wait. As soon as the timer hits 0, it jumps up again: [90 seconds remaining]. How are you supposed to know how long to wait? What are you supposed to do if it continues forever? All keyboard input is disabled so there is nothing you can do but hard reset your PC and hope systemd unmounted the filesystems before getting stuck.
It's just an unpleasant software.
The big problem is simple interfaces require more effort on the users part which will never happen for a majority of users. And lets face it, people like me, you and most others here are the extreme minority in computing. We aren't afraid of $ vi /etc/config or echo killuser >/n/pcturret/ctl. But writing documentation is boring and "hard" so it never gets done. People start complaining about docs and someone comes up with the bright idea to automate vi /etc/config by replacing it with a poorly thought out contraption that needs binaries and bolted on protocols. That's exactly what dbus is and is best described as duck tape engineering.
Yesterday one of KDE devs posted an article mentioning this sort of phenomenon and linked to an article depicting it in full[1]. And now that you mention Plan 9 it makes me think, because I always dreamt about Plan9 taking over the world and becoming big and everyone using it, but Plan9 is so great because it has never left the 'geek' circle, while others like Linux (and arguably the BSDs, to a lesser extent) already had.
[1] https://meaningness.com/geeks-mops-sociopaths
This is why the 9front/cat-v crowd is cantankerous and the humour as dark and cringy as possible. No likes. No stars. No instagram accounts full of pictures some jerk took in the middle of the sahara to show how shallow they are. We only want well thought out code. Otherwise go away.
Too bad it's stuck with hard coded 32 bit dependency. It also cant build on 64 bit systems unless they have 32bit compat (windows/linux win here). There's a fork maintained by some 9front users called Purgatorio. Charles Forsyth also still pokes at it and just added Risc-V support. It's not dead, just resting. Patches welcome.
0: https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs
On the flip side it windows. That OS doesn't hide in the background, rather it screams for attention 24/7, inserting itself in every little task even when specifically told to back off.
Yeah, and the good documentation. Wish more open source projects had these.
There's also software like dotnetcore which will just not build. The situation in the desktop is even more problematic. I wish it wasn't so, I like the separation of concerns quite a bit with FreeBSD but it's not something that unfortunately fits my use cases.
Doesn't work on macos or windows either; macos "docker" uses a linux VM.
Personally I prefer jails anyway.
I am not an expert in Windows though; perhaps you are referring to a version of docker that I'm not familiar with which doesn't support Linux services but instead runs Windows services in a native Windows container system? That would be cool and interesting but obviously it wouldn't run most docker images.
I used it under ESXi with physical disks attached as a backup server with hammer/hammer2 recently. Worked great and it was the only FS that offered effective offline dedup.
1. No ECC driver for AMD Ryzen, and no ECC support in memory management.
While you don't need any OS support to use ECC, it's much better to have it.
* Linux has an ECC kernel driver for AMD CPUs that accurately reports the ECC status, this is extremely helpful given the uncertainty of ECC in consumer-grade hardware. On FreeBSD, it's not possible to tell whether ECC is enabled due to the lack of driver. Also, if ECC is not initialized properly by the firmware, the Linux ECC driver will also detect it and try reenabling ECC and memory scrubbing. On FreeBSD, better to check whether your BIOS is broken.
* When an ECC error occurs, a Machine Check Exception is generated by the CPU. On Linux, it will be correctly decoded and recorded. But on FreeBSD, so far there's no decoder, leaving you a mysterious MCA error in dmesg. For example, a correctable DRAM ECC error will be reported as "L3 cache error" (which made many people to falsely believe that Ryzen's ECC was not working on FreeBSD).
* On Linux, if an unrecoverable multibit ECC error is detected in userspace memory, the process will be killed by a SIGBUS (my username checks out?). If the error occurs in kernel memory, it triggers a kernel panic. On FreeBSD, as far as I know, none of the features is supported by the kernel, it increases the risk of data corruption, and it's not a Ryzen or MCE-specific problem. Also, Linux also supports memory offlining, which can be used to temporarily remove bad memory pages from the system after an ECC error, which is not found in FreeBSD.
Fortunately the ECC-enabled flag is easy to check (I plan to make a contribution to the FreeBSD kernel), and a MCE decoder in userspace should also be easy. But implementing the rest is not trivial, the lack of ECC support in memory management is a problem. It's unfortunately that these features are not supported in a server-grade OS like FreeBSD (I guess FreeBSD runs on "real" servers, which implements all of those ECC features in the management firmware, so "real" FreeBSD users don't really need them...)
2. No USB serial system console and netconsole support. On FreeBSD, although tty login via USB serial is supported (just run a getty on it), it seems that there's no way to redirect all dmesg to a serial port (perhaps someone can correct me), FreeBSD's comconsole requires a real physical serial port, not USB, on the other hand Linux's CONFIG_USB_SERIAL_CONSOLE supports USB natively. On Linux, there's also a kernel-space netconsole that redirects all demsg via UDP/IP, this feature is not found in FreeBSD although there was an attempt in 2012.
For USB serial console, I think part of the issue there is that the serial console can interface with the kernel debugger, which might be tricky if it's USB (but I guess USB keyboards probably work in the kernel debugger, so maybe there's a chance?).
Thanks for working on Ryzen ECC. I don't work with a server farm anymore, and I'm too cheap to buy ECC for home use, but someday I might use that support.
On servers with Intel CPUs, FreeBSD warned me promptly, by e-mail, about some corrected ECC errors.
The errors appeared suddenly after many years of use and they kept recurring, pointing to a probable failure in the near future, but being promptly notified, I have replaced the offending DIMM and all went back to normal.
So yes, you are right, for FreeBSD it is better to stick to Intel, unless you have time to change yourself the sources, to add the missing hardware support, which is somewhat easier than attempting to make changes in the Linux kernel.
There are also examples of hardware for which FreeBSD has better support than Linux, but those belong to classes of hardware less likely to be encountered by non-professional users.
For example FreeBSD has better support for magnetic tapes.
This is not just an AMD problem. For example, as I already said, there is no support of unrecoverable ECC errors handling in the FreeBSD memory management subsystem for any CPU, including Intel. On Linux, the kernel has first-class ECC support via the Error Detection And Correction subsystem (EDAC). EDAC can decide to kill a process, to trigger a kernel panic, or even to isolate the faulty memory based on the location of the memory error. On FreeBSD, the only thing that handles unrecoverable ECC errors is the userspace utility sysutils/mcelog [0], the best thing it could do after an unrecoverable error is sending you an email, at which point data corruption may have already occurred [1]. Usually correctable errors occur before unrecoverable ones, so data corruption could be prevented if it's fixed soon enough, but if you get "lucky"...
And unlike writing device drivers, I don't think most people (including myself) is qualified to mess with the memory management code in the kernel. But I have a plan to do something about it, for example, there's nothing stops me from writing a simple script to decode the MCE log and kill the kernel from the userspace. And given how rare unrecoverable ECC errors are, even without graceful handling (like killing userspace process or runtime memory offlining), it should solve my concern of data integrity on FreeBSD.
[0] But corrections are welcomed. I suspect FreeBSD will also halt the kernel on Intel platforms after a fatal MCE due to ECC error, but I didn't find any information and came to a negative conclusion.
[1] I guess it's not really a problem for "real" FreeBSD users, on "real" servers, BIOS can kill the system after an unrecoverable ECC, which probably explains the lack of any unrecoverable error handling in FreeBSD.
Does anyone have some good LWN-type articles that goes deep into the differences between BSD vs Linux internels (also common user land traits in downstream distros)?
https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-...
https://people.freebsd.org/~jlemon/papers/kqueue.pdf
However, this has always been a weak spot for Linux until recently and it took too many iterations of various partial solutions of the problem to reach the current state.
This was a case when avoiding the NIH approach and looking carefully at the older Windows or FreeBSD solutions would have resulted in an earlier good solution.
The tests appear to compare ZFS and ext4 and clang and gcc as much as FreeBSD and Linux.
So, it depends on what you're doing. A lot of the tests where FreeBSD was much slower were media related; it makes me wonder if some CPU acceleration didn't get enabled; which could be FreeBSD (or Clang) missing a feature, or a benchmark issue or something else.