> I'm interested in the project, but I think it's driven by people who are very excited about Rust, and I want to see how it actually then ends up working in practice.
(Deleted because I only meant to bring up the broader phenomenon of what makes a person feel "excited" or "unexcited" about a language; I was specifically trying (unsuccessfully) to avoid the hotbed topic that has arisen below)
It would be a mistake to characterize this push as an "emotional concern." The main driver is security. The folks doing this work are advocates of memory safety first, and Rust second. It just so happens that in this particular context, Rust is the most appropriate choice.
EDIT: it's all good! I didn't think your comments were bad.
I think it depends on what "re-write it in Rust" means. Some people say that it means folks specifically asking for a project to be re-written in Rust. Others seem to imply that any new code written in Rust counts. To illustrate the difference, ripgrep would be the latter, but not the former.
If we're talking about the former, I think the actual instances of this are few and far between, and are complained about far more than actually occurs. At one point I was actually trying to get hard data on this, but there really were only a handful of times I could find this seriously occurring, and they were pretty easily dismissed. The folks who talk about it like a plague of some kind never seem to show their evidence, or link to one or two examples at best.
If we're talking about the latter, it shouldn't be surprising that folks who like a language write software in that language. That's why languages exist. I don't know why some people really want to say that folks cannot write whatever software they want in whatever language they want.
> At one point I was actually trying to get hard data on this, but there really were only a handful of times I could find this seriously occurring
This is probably depending on the environment. Corporate environments would move very slow, open source kind of fast and SMB/SME somewhere in the middle. All of them plagued by cargoculting, but especially average performing SMB/SME are the most guilty, and also where I've seen the most "Hey, we should probably rewrite this service in Rust because it'll be better for sure" since some years back.
I actually stumbled upon[1] a data source for you! Seems there is some activity in the "Rewrite it in Rust" domain, and there is a collection of it being suggested in a lot of places: https://github.com/ansuz/RIIR
Right, but this is what I mean. If this is the best collection of stuff, well, it doesn't make a compelling argument that there is a huge scourge of this around the internet.
Even if we assume these are all legit issues, there have been 66 total in the five years this repository has existed. That is a miniscule number.
If we actually look at the issues, many of them are not instances of an outsider demanding a project being rewritten in Rust. Let's look at the most recent five issues, to pick a random sample:
* https://github.com/ansuz/RIIR/issues/59 This one does appear to be legit. Not a ton of drama here, a polite conversation that was closed without issue, even though its existence is generally a bit annoying, yes.
* https://github.com/ansuz/RIIR/issues/67 This is a person who made this issue, then two minutes later, opened an issue being a jerk on the target repo. Yes, that is annoying for the person who had the issue opened, but is hardly an indication that these are real requests. This person is trolling.
The next seven most recent closed issues are someone who has just opened issues with random technologies they thought they'd be funny. I was only gonna do five, like the opened issues, but it's clear this is a series, so have a bonus two.
So already, we have like, let's round it up: two real issues here, out of fourteen. I am not convinced.
(Your hacker news link seems like nonsense to me?)
I think if security is a main concern of people working on Linux, there are both lower hanging fruit AND bigger fish to fry than getting code into Rust.
Not that it isn't a worthy effort, but it's still an effort by people who are more excited about Rust than they are acting in the interest of security.
Edit: It's almost a meme at this point that you can't even be remotely critical of Rust on this board without kneejerk mass downvoting.
Do you know anything about the people whose motives you are questioning? What work they've done, and why? I think if you did, you may not say what you've said.
I'm well aware of whom I'm speaking to and the efforts made by others.
I'm also well aware of lots of other security problems facing Linux that need love and the steps being taken by other operating systems where Linux lags behind.
So while I'm not saying that there isn't a concern about security by the people involved, I am certainly saying why this as opposed to other things.
I have my pet problems and areas where I prefer to work too.
I don't see how that overlaps with many of us, myself included.
My 19 years of programming have taught me that something that continuously craps the bed (like OpenSSL) needs to be rewritten because from one point and on it's clear that the cost of maintenance exceeds the cost of rewriting.
Furthermore, do consider that some of us were programming C and C++ and found them sub-optimal. But here we would likely venture into language wars because there are a lot of programmers like myself out there who will never subscribe under the notion of "C is fine, you are just doing it wrong".
> My 19 years of programming have taught me that something that continuously craps the bed (like OpenSSL) needs to be rewritten because from one point and on it's clear that the cost of maintenance exceeds the cost of rewriting.
I would agree and it was already done by the OpenBSD project. Google started BoringSSL/Tink. Amazon created s2n. Linux continues to use OpenSSL. It's still probably in the top 3 problems with security in Linux that needs fixing.
The rest of your statement is just making my point for me.
Well, in that case your statement of "If only people were so excited about making OpenSSL not a flaming hot bag of shit" is severely misplaced. I have no influence over Linux governance. 99.9999% of us out there don't have that influence. So what was your point exactly?
Rants are fine and we all do them but I will strongly disagree with any responsibility or guilt transferred to me because I am excited about Rust and not about OpenSSL. As you yourself pointed out, alternatives exist but aren't adopted so the problem is political and not technical. Right?
> The rest of your statement is just making my point for me.
> Furthermore, do consider that some of us were programming C and C++ and found them sub-optimal. But here we would likely venture into language wars because there are a lot of programmers like myself out there who will never subscribe under the notion of "C is fine, you are just doing it wrong".
> Not sure what you mean but okay. ¯\_(ツ)_/¯
Blame the carpenter, not the tools.
C or C++ isn't a real business problem domain that anyone has. Nobody's committing their budget to fix that. That's the kind of problem only held by ideologues and other unreasonable people.
My point was that people who care about fixing problems in the security domain will look to fix the biggest problems they can in the security domain. They won't look at it and say "no, I can't fix this because C/C++ is unacceptable to me."
That's someone who cares more about the tools than the problem domain. Whethor or not you think C/C++ are suitable to the problem domain of writing secure software, software will be written in C/C++ and must be secured.
Turn down work all you want though. More for the rest of us.
> My point was that people who care about fixing problems in the security domain will look to fix the biggest problems they can in the security domain. They won't look at it and say "no, I can't fix this because C/C++ is unacceptable to me."
Sometimes that's true, sometimes it isn't. It's a cost-benefit analysis like almost anything else.
Everyone is free to disagree on this -- or like you, misinterpret my motivation as "caring more about the tools than the problem domain". Quite the contrary, I want the problems solved but I also put deadlines and energy budgets on trying to solve problem X with tool Y and if it doesn't cut it within the deadline and/or the budget then it's out. Sure, that means more C/C++ work for you. I am OK with that.
But let's not pretend that OpenSSL can only be written in C or C++ now. We all know that's not true.
> Blame the carpenter, not the tools.
It's no accident that professional craftsmen utilize innovation in their tooling as well, like Sushi chefs using ceramic knives for some fish and steel knives for others. Ceramic knives weren't that popular some mere decades ago.
Your black and white philosophy makes no favours to your argument.
It seems you take an issue with the rising popularity of Rust. But let me ask you this: if you are so confident in C/C++'s superiority, why are you bothered?
> It seems you take an issue with the rising popularity of Rust. But let me ask you this: if you are so confident in C/C++'s superiority, why are you bothered?
I don't have a problem with Rust at all. It's a fine tool and I'd be happy to use it (and have), just like any other. I work on interesting problems and am tools-agnostic (aside from certain languages being suited or not to specific domains).
I have a problem with people who lose their objectivity as a result of their enthusiasm for Rust and go around shouting in every project like it's the most important thing to happen in the last 30 years of Computer Science. It wouldn't be such a meme if people didn't actually friggin' do this.
> I have a problem with people who lose their objectivity as a result of their enthusiasm for Rust and go around shouting in every project like it's the most important thing to happen in the last 30 years of Computer Science. It's an annoying trend the last 4-5 years.
I hear this a lot and I've never witnessed it once. ;)
It's all filter bubbles, dude. Chill. The world is not ending.
As an example: the comments from both Linus/Greg and the Rust community that relate to the OP are quite balanced and nobody is shouting for revolutions.
---
...That being said, I wouldn't make grandiose claims about Rust myself even if it solved a lot of problems for me, but still: Rust is an important innovation regardless. It seems a lot of the software area has chased its own tail before we had some languages step forward and solve various problems that people kept trying to fix themselves on an ad-hoc basis (and kept failing at it).
Rust is one example, Erlang/Elixir are another, and when OCaml finally gains parallel abilities, it will be an additional (and very fine) example as well. Surely other examples exist.
If we're now discussing just the point you quoted me on -- I on my side take issue with the people who are way too liberal and love to pretend that it doesn't matter what language is being used regardless of the business and technical problems. Because oh my god yes, yes it does matter. A lot.
Modern C++ (>C++17) with RAII and neat libraries is already an elegant, high performance and high-productivity language with good IDE/editor support.
And its not like C++ is standing still. You have standardised concepts, modules and co-routines out this year. Modules will fix several of C++'s shortcomings wrt build tooling and setup. I am personally looking forward to C++ 20 co-routine support in all the major compilers - will enable one to develop graceful-async, portable and stable libraries without all that Rust churn needing re-writes all the time.
Competition is good, but its best to take the Rust blinkers off a bit.
Don't get me wrong -- I agree Rust has some footguns. And I had high hopes for the 2021 Edition to introduce some terser syntax here and there but alas, not happening.
I am not fangirling for Rust at all.
I am saying that it's much easier to make costly mistakes in C and legacy C++. I am aware of the modern efforts in C++ but they don't help with the literal mountains of legacy code that has to be maintained out there...
You have to blame both the carpenter and the tools. Since there is no such a thing as a perfect carpenter, mistake will happen, that why a tool that can prevent a whole category of mistakes is a good thing. It's no mistake good carpenter don't use crappy tools. To be the most efficient, you need the good carpenter with the good tools.
Of course you have to care about the problem domain first, but it's absolutely wrong that nobody commit their budget to fix C++ safety problems. There is a massive amount of static and dynamic tooling to mitigate its problems and a substantial part of the C++ evolution these last years was to improve safety. Rust was able to go further than C++ in this area since it is not yet constrained by decades of history.
To be frank BoringSSL and other derivatives are not "rewrites". They are largely the same as OpenSSL with some parts of the code removed. A rewrite starts from a fresh slate, not slowly modifying a fork.
BTW, rustls exists and is a rewrite of OpenSSL of sorts.
> Edit: It's almost a meme at this point that you can't even be remotely critical of Rust on this board without kneejerk mass downvoting.
I do wonder how it happens as I upvote such people, but I've been posting here for years and still don't have downvote privileges. I do wonder sometimes who controls the downvotes.
I think emotional reaction happens to a lot of things that we like, where we raise up their benefits and downplay their negatives. At least for the first few years, until we're in a different context or something else comes along that solves it even better.
I also think that's why lisp people are so feverish about lisp as well: as you get more involved in the language with its charms, the more excitement you get and you want to see the language in more places/contexts
The fact that he is only getting around $1,000 per month on Patreon points to deep, deep flaws in humanity. Especially when you compare it to other funding amounts for say Linux or popular smut on Patreon. A total lack of foresight and values.
Lol what? Deep flaws in humanity? All because some small time non-visible OS doesn't receive more funding? The only deep flaws I can see are within you generalizing an entire population because you don't get what you want.
Redox OS getting much less than Linux or porn on Patreon points to deep, deep flaws in humanity and a total lack of foresight and values? I think there's probably a reasonable version of this opinion, but yours is intensely, unrealistically judgmental and righteous.
The flaws might not be in humanity but in the RIR = rewrite it in Rust crowd. If you are part of RIR crowd let us know why you would not support this project and which one you are supporting or would support if you had the possibility.
I like Rust, but I don't understand why "OS written in [language]" is a compelling goal within itself. Like an OS absolutely needs goals and a target demographic to be useful, and it doesn't feel like simply existing to show that a language can do it is enough.
Plus if I was going to fund a toy research kernel that aims to increase security, I'd go whole-hog and bring Singularity OS[0] back from the dead, because at least that plays with new ideas and tested the limits of what was thought to be possible.
I am not arguing with your original point about the language. But Singularity is as closed code as it can get and has a bunch of crazy assumptions associated to it (must have .NET installed which rules out most architectures)
Can you explain what you like about it?
Compared to C, Rust avoids a whole category of memory related bugs
Redox OS is "*NIX written in Rust." Singularity was a complete reimagining of OS architecture from the ground up. It isn't simply "the same ideas but now in Sing#/CLR!" it was an answer to "what if we designed a clean-sheet OS today?" There's a long paper on it here[0].
But keep in mind Singularity is still a toy OS. It has no use-case, and doesn't aim for real-world goals that would make it commercially successful. I just think that even as toy OS's go, it was one of the more interesting ones of recent times.
> I like Rust, but I don't understand why "OS written in [language]" is a compelling goal within itself.
There's a few good reasons in my mind:
* proof of concept - if a language claims to be a systems language, being able to point to an OS in it is pretty good proof that it's true.
* dogfooding - if there are problems with how the language works for the OS case, they can be highlighted by the OS. (this is similar to self hosting a compiler)
* community building - some folks want to work on a new OS, and doing it in a specific language helps grow both groups and at least introduces them to each other.
For me the 'logical' thing to do to build a 'safety first' OS in the 21st century would be: use the seL4 microkernel and then Rust to build the full OS kernel.
I'm no expert in this domain but it seems to me that RedLeaf (implemented in Rust and formally verified) is vastly superior to seL4. AFAIU RedLeaf is orders of magnitude cheaper to formally verify to a much higher standard[1] while also being significantly more performant[2] and that Rust's properties are key to both of those advantages[1,2]. (Note: I only skimmed [2].)
Edit: It seems RedLeaf is immune to meltdown due to its design while seL4 has meltdown mitigations[2]. I would assume the same goes for spectre, but that would be an assumption on my part. (Note again, I've only skimmed [2] so I can easily be wrong here.)
While I don't know anything specific about RedLeaf, I highly doubt that it is completely immune to Spectre. Spectre fundamentally stems from how modern CPUs are designed and the current understanding is that there is no way to fix Spectre. If you're curious, see the paper "Spectre is here to stay: An analysis of side-channels and speculative execution" [0]. Even on fully up to date OSs with the latest version of Chrome, Spectre is still exploitable (see [1]).
I must admit that I don't really understand the RedLeaf design if it doesn't use hardware protections you cannot run C or use unsafe Rust "safely"??
Everything in 100% safe Rust isn't going to happen ever so it isn't really a seL4 competitor more a research prototype.
That said I can imagine a two layer system: if you use 100% safe Rust you have a 'fast path' direct access, otherwise you fallback to slower hardware protection.
I don't have any particular feelings about Rust, but proponents of Rust contend it is good for systems work, and that its memory safety checks eliminate memory safety issues arrising from safe code and that limiting code that doesn't pass those safety checks to marked areas makes it easier to audit and reason about memory safety leading to fewer memory safety issues in the final product.
An operating system is the essential piece of systems work, so a Rust OS should serve as a demonstration of the efficacy of Rust in systems work. Is the performance on par with contemporary operating systems? Is the code about as easy to understand as contemporaries? Is there a reduction in memory safety issues vs contemporaries?
Positive results from such a project could inspire others to use Rust for more systems work and perhaps integration into other OS kernels and/or key userspace libraries.
>The kernel is different from userspace projects - more difficult in some respects (we use a lot of very odd header files that pushes the boundary of what can be called "C"), but easier in many other respects (mainly in the sense that the kernel is fairly self-contained, and then doesn't rely on other projects for the final binary).
I'm interested in what Torvalds meant by these odd header files, does anyone know?
/*
* This returns a constant expressionn while determining if an argument is
* a constant expression, most importantly without evaluating the argument.
* Glory to Martin Uecker <Martin.Uecker@med.uni-goettingen.de>
*/
#define __is_constexpr(x) \
(sizeof(int) == sizeof(*(8 ? ((void *)((long)(x) * 0l)) : (int *)8)))
I can try... In short, this is about how a C compiler is supposed to deduce the type of the ternary operation. So, if x is a constant expression, the compiler figures that the second operand must be NULL (because it is allowed to perform the multiplication, by 0l), whose type then is whatever the third operand says (a pointer to int). The comparison succeeds. Otherwise the compiler does not perform the multiplication, takes the type of the second operand at its “face value”, i.e. as a pointer to void, and converts the type of the third operand to it - which, of course, makes the comparison false in this case.
I love the idea of more rust into linux because I think it will make the code more secure and easier to contribute/modify. I am scared of any potential problems an immature ecosystem (relative to C ofcourse, don't jump the gun) can cause to this life-depending tool called the linux kernel. I am glad linus is open to it but careful. Balanced approach
Edit: I am also a proponent of careful re-writes of very old userspace utilities (eg: ping) accompanied by very rigorous tests to ensure there aren't any behavour changes
> I am also a proponent of careful re-writes of very old userspace utilities (eg: ping) accompanied by very rigorous tests to ensure there aren't any behavour changes
That would be great, but a prerequisite for that is that the rust language is stable to ensure that there aren't any behaviour changes in the compiler itself.
Rust doesn't support the same amount of architectures C and Linux do, hopefully either Rust learns to support more architectures, or this stays confined to just driver code because I'd hate to see Linux disappear from some of the more esoteric architectures.
> Another point is taking on drivers first for "any initial trials to drivers is simply the architecture side," said Torvalds. "Lots of drivers are only relevant on a couple of target architectures, so the whole issue with Rust code not being supported on some architectures is less of an issue."
Linux removes platform support from time to time, like any project. IA64 was a recent, notable example.
That being said, it is true that Rust doesn't yet support all the platforms that are important to the kernel. That's one of the reasons this is starting with drivers, and is one of the reasons why some folks are pursuing a Rust frontend for gcc.
It depends entirely on whether there are still developers interested in maintaining support for those architectures. It seems pretty safe to assume that Torvalds isn't going to be in favour of "we drop anything without a rust backend tomorrow"; but on the other hand 8 archs were dropped in 2018 -- https://lwn.net/Articles/749292/ -- essentially for lack of any active interest in them. It's possible that eventually "set of archs with a rust backend" and "set of archs which people care about enough to keep alive in the kernel" converge to the same set. It's even possible that one day "does this even have a rust backend?" becomes a marker of "is this thing really of use to anybody" in the same way that in 2018 "does this thing have support in upstream gcc" was a marker for "maybe we can drop this". That would be years from now, if ever, though, IMHO.
Or maybe this would inspire people to expand LLVM support? There's nothing fundamental that prevents Rust from running on those architectures (as far as I know)
I don't believe there is anything fundamentally preventing any language from running on any architecture other than "we haven't gotten around to it yet". Can't you emulate any Turing machine inside of any other Turing machine? I'd guess you could run the latest version of Python on anything more advanced than a abacus if you had a couple of lifetimes to waste on the project.
My point is that a) half of the work doesn't have to happen because of LLVM's back-/front-end split, and b) if these are architectures that people care about keeping around and investing in, the benefits of adding LLVM support will go beyond just Rust
If you're still running a system with an arcane ISA, support it yourself. 99% of the people running modern Linux are interested in three CPU architectures (four or five once RISC-V gains traction), all of them supported by Rust. Supporting old or obscure ISAs is not a priority at this point compared to bringing kernel memory safety into the 21st century, which Rust helps a lot with.
While I think I agree with this article, that isn't the case here. That article is talking about taking C code that happens to be portable to an obscure architecture, but never officially supported it, then complaining when the never-official support breaks for some reason. We're talking here about the Linux kernel, which (by necessity) was explicitly ported to every architecture it works on, so any lost architecture support would be an explicitly removed feature.
It is true that Rust supports a subset of the architectures LLVM supports, but to be honest, I am not sure if "many" is the right adjective. I mean that in a literal sense, I don't have a good handle on the list. For Rust, the list is https://doc.rust-lang.org/nightly/rustc/platform-support.htm..., but I'm not sure if there's such a list for LLVM, other than looking at something like https://llvm.org/doxygen/Triple_8h_source.html and figuring it out?
We actually ran against this problem when I tried to get a rust toolchain up and running for our ARM v6 based product. I think that support has been added now, but it's a harder sell now because people wonder what other incompatibilities might be there, hidden, and waiting to bite us later.
Can someone with more knowledge than me explain what benefit Rust presents to a low level codebase such as a kernel?
I am a C programmer by trade who has played with Rust. It is a great language but there is very little you can do safely. In higher level projects you can mostly push the un-safety into the standard library.
But when you have to do everything yourself, all of the "buggy" code has to be marked unsafe anyway. Allocating and accessing(!!) memory is unsafe. Interfacing with C or asm is unsafe. Your buffer/vector implementation is unsafe.
Basically, all the common culprits for memory bugs in C would be the code marked unsafe if rewritten in rust. So what is the benefit?
You'd write the allocator or low-level memory access logic once with unsafe code and afterwards all code using the functionality can be written as safe code. That limits the amount of code that can have data races.
In a multi-core system, it may be, since you're not really accessing main memory directly, but you're going through several levels of all kinds of caches, some shared and some not (L1 (instruction, data), L2, L3, TLB, ...), and sometimes competing with devices that access the memory "directly" via DMA.
I work on firmware and kernel-level code, in pure Rust, at my job. We don't do malloc at all.
One of the key features Rust enables is that you can build safe abstractions on top of unsafe things. Another key feature is that those abstractions often mean compile-time, but not run-time, cost.
If you build the right abstractions, you can minimize the amount of unsafe code pretty dramatically. I don't have numbers handy for our codebase (and it's not open source yet, but will be eventually), but as an example, the Redox documentation (a Rust-based kernel that is, uh... linked... below) says: https://doc.redox-os.org/book/ch01-07-why-rust.html#unsafes
> A quick grep gives us some stats: the kernel has about 300 invocations of unsafe in about 16,000 lines of code overall. Every one of these is carefully audited to ensure correctness.
The advantage here is that, if and when there are bugs in this code, you only have to look at the modules containing those unsafe invocations, rather than the entire codebase.
There are also other advantages to the Rust language that make it worthwhile too, regardless of safety percentage, but it is true that this is often people's first concerns. The code that landed in linux-next has a lot of unsafe in it, because it is specifically trying to build out the proper abstractions here. The idea is that drivers built on top of this code will not need nearly as much, generally.
> The advantage here is that, if and when there are bugs in this code, you only have to look at the modules containing those unsafe invocations, rather than the entire codebase.
That is a brilliant feature I had not thought of. Thanks!
No problem! I think the skepticism is very warranted. We know that these systems can be created, but we don't have examples of this working at the same scale as projects like Linux yet. That takes time and effort, exactly as Linus and Greg Kroah-Hartman say. We'll see how it works out in practice. I also think the "de novo system" and "adding Rust into a large existing system" are two related, but different, problems, with their own unique challenges.
One great thing that I see with Rust is that the developers are open to extensions of the language that are provided just to be more compatible to the C memory model (like anonymous unions). I don't know many other languages like that.
"The advantage here is that, if and when there are bugs in this code, you only have to look at the modules containing those unsafe invocations, rather than the entire codebase."
I would be cautious with statements like that. You can limit some kinds of operations to unsafe code, audit the daylights out of those, build abstractions that make it impossible to use that unsafe code from safe code in an unsafe manner, and still not be doing the right thing.
Sure, I thought the context of this discussion was memory safety specifically, and so that's what I was talking about. In a less specific conversation I would have said "memory safety bugs," but thought it would be redundant here. I guess not.
Another advantage, when compared with C, is the package/dependency management and build tooling which makes compiling for targeted architectures a little easier
Ideally, you write a small subset of unsafe primitives and encapsulate those in a safe interface and try to use those to implement your actual kernel logic.
unsafe isn't like a taint, where things that use unsafe code must propagate their unsafety outward. It's more like saying "there are invariants that this function expects, but does not enforce, and the caller may violate those invariants". Obviously, we want to minimize code like that. The best way to do that is to build abstractions around the unsafe code that only interact with the unsafe code in a way that respects those invariants. Often, (though not always) that means giving up some amount of flexibility to the user of your safe abstraction. If you design it right, the caller may never miss that flexibility.
A recent example of using private constructors and zero-sized compile time invariant enforcement is an embedded blog here: https://www.ecorax.net/macro-bunker-1/
The embedded rust community uses this typestate pattern to build safe abstractions around pin interactions. Very similar things can be done to build safe abstractions around the volatile memory accesses etc that the kernel does.
I see "unsafe" more like "this is code that the compiler can't prove it holds its invariants, but in fact it does and I hope the programmer added a comment to explain why".
>It is a great language but there is very little you can do safely.
Even extremely performance-oriented and low level projects such as the standard library or the Redox kernel are less than 5% unsafe code. There's lots you can do safely.
The benefit, and really the point of unsafe, is to encapsulate the unsafe stuff behind safe APIs and to use Rust's other features, like the borrow checker for example, to enforce correct usage of such API.
Ex.:
I was using nanovg (vector graphics library in C) to build a GUI app some day. I was using Rust and a bare-bones FFI library that provided a 1:1 mapping to the C API. On top of this unsafe C-API, I built a wrapper library in safe Rust that doesn't allow me to use nanovg incorrectly. And not in the use-after-free sense of incorrect (that too), but instead in the "you can't call this function between these other two, that's undefined / will produce unexpected graphical results".
So the power of unsafe is to encapsulate "unsafe" behaviour and to enrich it with compile-time information, to build guarantees around it.
If you are looking for a Rust nanovg port, look no further https://github.com/femtovg/femtovg. We are in the process of adding a wgpu backend and we are definitely looking for contributors so contact me if you want to get involved.
> From where Kroah-Hartman sits, "it will all come down to how well the interaction between the kernel core structures and lifetime rules that are written in C can be mapped into Rust structures and lifetime rules for drivers in Rust to be able to use them properly. That's going to take a lot of careful work by the developers wanting to hook this all up and I wish them the best of luck."
It's really interesting that any C code already has to have a memory management ownership model, and the consistency of that model makes Rustification easy / hard.
I am all in for Rust in the kernel (as drivers) and I am really curious of how they will manage sharing memory across boundaries (user <--> kernel spaces).
For example, an H264 encoder/decoder module in the kernel: let's say I have a Freescale/NXP i.MX6 platform with a HW H264 decoder, and I have to create kernel module to talk to it. Then use this module from an user space application to decode a video file.
What I did (modified actually) in the past is a module that allocated a couple of (aligned) buffers in video memory and then passed them around between the HW decoder, kernel module and user space application.
This is usually done to not force a deep copy of >100KB buffer for every decoded frame. This memory is shared between different parts of the implementation, meaning that the same buffer belongs to up to 3 parties at any given time. Also, the allocation is done once and lives for the entire application life, usually allocating like 3 buffer so when one buffer is being decoded I can still use already decoded buffers for writing them into a file.
I know there is an alternative (hack?) using something called Pin (possibly clashing with, like, a GPIO Pin?) but I didn't get so deep already with Rust to really understand it.
Ah! whatever driver/module they do, it should be accessible with C. So Rust must fill the gap between "Some C code in userland" <--> Rust <--> "More C code in the kernel", so I'm really interested to see how ownership works in this case.
Userland C app says: "Hey Rust kernel module, take this struct and do something", the I go and change the contents of that struct from another thread in my userland application. Would this be possible?
Rust can handle these scenarios just fine. Specifically what you'd need to do depends on the exact semantics of the details here, but this big picture isn't a problem.
122 comments
[ 5.1 ms ] story [ 185 ms ] thread> I'm interested in the project, but I think it's driven by people who are very excited about Rust, and I want to see how it actually then ends up working in practice.
(Deleted because I only meant to bring up the broader phenomenon of what makes a person feel "excited" or "unexcited" about a language; I was specifically trying (unsuccessfully) to avoid the hotbed topic that has arisen below)
EDIT: it's all good! I didn't think your comments were bad.
If we're talking about the former, I think the actual instances of this are few and far between, and are complained about far more than actually occurs. At one point I was actually trying to get hard data on this, but there really were only a handful of times I could find this seriously occurring, and they were pretty easily dismissed. The folks who talk about it like a plague of some kind never seem to show their evidence, or link to one or two examples at best.
If we're talking about the latter, it shouldn't be surprising that folks who like a language write software in that language. That's why languages exist. I don't know why some people really want to say that folks cannot write whatever software they want in whatever language they want.
This is probably depending on the environment. Corporate environments would move very slow, open source kind of fast and SMB/SME somewhere in the middle. All of them plagued by cargoculting, but especially average performing SMB/SME are the most guilty, and also where I've seen the most "Hey, we should probably rewrite this service in Rust because it'll be better for sure" since some years back.
- [1] https://news.ycombinator.com/item?id=26561093
Even if we assume these are all legit issues, there have been 66 total in the five years this repository has existed. That is a miniscule number.
If we actually look at the issues, many of them are not instances of an outsider demanding a project being rewritten in Rust. Let's look at the most recent five issues, to pick a random sample:
* https://github.com/ansuz/RIIR/issues/62 This is showing a branch that the team themselves made. It is not an outside demand to re-write it in Rust.
* https://github.com/ansuz/RIIR/issues/59 This one does appear to be legit. Not a ton of drama here, a polite conversation that was closed without issue, even though its existence is generally a bit annoying, yes.
* https://github.com/ansuz/RIIR/issues/58 This again is someone who is doing this for their own project.
* https://github.com/ansuz/RIIR/issues/57 This is a transcription of a joke in a telegram chat.
* https://github.com/ansuz/RIIR/issues/53 This is a person who opened an issue on this project, but does not appear to have asked upstream about it.
* https://github.com/ansuz/RIIR/issues/51 This one is amusing, given the article, but again, isn't an upstream request.
Let's look at recent closed issues:
* https://github.com/ansuz/RIIR/issues/67 This is a person who made this issue, then two minutes later, opened an issue being a jerk on the target repo. Yes, that is annoying for the person who had the issue opened, but is hardly an indication that these are real requests. This person is trolling.
The next seven most recent closed issues are someone who has just opened issues with random technologies they thought they'd be funny. I was only gonna do five, like the opened issues, but it's clear this is a series, so have a bonus two.
So already, we have like, let's round it up: two real issues here, out of fourteen. I am not convinced.
(Your hacker news link seems like nonsense to me?)
Not that it isn't a worthy effort, but it's still an effort by people who are more excited about Rust than they are acting in the interest of security.
Edit: It's almost a meme at this point that you can't even be remotely critical of Rust on this board without kneejerk mass downvoting.
I'm also well aware of lots of other security problems facing Linux that need love and the steps being taken by other operating systems where Linux lags behind.
So while I'm not saying that there isn't a concern about security by the people involved, I am certainly saying why this as opposed to other things.
I have my pet problems and areas where I prefer to work too.
Do tell, I am very interested?
> but it's still an effort by people who are more excited about Rust than they are acting in the interest of security
Needlessly dismissive. What if the people got excited because of the increased security?
If only people were so excited about making OpenSSL not a flaming hot bag of shit.
Can't wait to find out what that patch on Thursday will be for.
My 19 years of programming have taught me that something that continuously craps the bed (like OpenSSL) needs to be rewritten because from one point and on it's clear that the cost of maintenance exceeds the cost of rewriting.
Furthermore, do consider that some of us were programming C and C++ and found them sub-optimal. But here we would likely venture into language wars because there are a lot of programmers like myself out there who will never subscribe under the notion of "C is fine, you are just doing it wrong".
I would agree and it was already done by the OpenBSD project. Google started BoringSSL/Tink. Amazon created s2n. Linux continues to use OpenSSL. It's still probably in the top 3 problems with security in Linux that needs fixing.
The rest of your statement is just making my point for me.
Rants are fine and we all do them but I will strongly disagree with any responsibility or guilt transferred to me because I am excited about Rust and not about OpenSSL. As you yourself pointed out, alternatives exist but aren't adopted so the problem is political and not technical. Right?
> The rest of your statement is just making my point for me.
Not sure what you mean but okay. ¯\_(ツ)_/¯
> Not sure what you mean but okay. ¯\_(ツ)_/¯
Blame the carpenter, not the tools.
C or C++ isn't a real business problem domain that anyone has. Nobody's committing their budget to fix that. That's the kind of problem only held by ideologues and other unreasonable people.
My point was that people who care about fixing problems in the security domain will look to fix the biggest problems they can in the security domain. They won't look at it and say "no, I can't fix this because C/C++ is unacceptable to me."
That's someone who cares more about the tools than the problem domain. Whethor or not you think C/C++ are suitable to the problem domain of writing secure software, software will be written in C/C++ and must be secured.
Turn down work all you want though. More for the rest of us.
Sometimes that's true, sometimes it isn't. It's a cost-benefit analysis like almost anything else.
Everyone is free to disagree on this -- or like you, misinterpret my motivation as "caring more about the tools than the problem domain". Quite the contrary, I want the problems solved but I also put deadlines and energy budgets on trying to solve problem X with tool Y and if it doesn't cut it within the deadline and/or the budget then it's out. Sure, that means more C/C++ work for you. I am OK with that.
But let's not pretend that OpenSSL can only be written in C or C++ now. We all know that's not true.
> Blame the carpenter, not the tools.
It's no accident that professional craftsmen utilize innovation in their tooling as well, like Sushi chefs using ceramic knives for some fish and steel knives for others. Ceramic knives weren't that popular some mere decades ago.
Your black and white philosophy makes no favours to your argument.
It seems you take an issue with the rising popularity of Rust. But let me ask you this: if you are so confident in C/C++'s superiority, why are you bothered?
I don't have a problem with Rust at all. It's a fine tool and I'd be happy to use it (and have), just like any other. I work on interesting problems and am tools-agnostic (aside from certain languages being suited or not to specific domains).
I have a problem with people who lose their objectivity as a result of their enthusiasm for Rust and go around shouting in every project like it's the most important thing to happen in the last 30 years of Computer Science. It wouldn't be such a meme if people didn't actually friggin' do this.
I hear this a lot and I've never witnessed it once. ;)
It's all filter bubbles, dude. Chill. The world is not ending.
As an example: the comments from both Linus/Greg and the Rust community that relate to the OP are quite balanced and nobody is shouting for revolutions.
---
...That being said, I wouldn't make grandiose claims about Rust myself even if it solved a lot of problems for me, but still: Rust is an important innovation regardless. It seems a lot of the software area has chased its own tail before we had some languages step forward and solve various problems that people kept trying to fix themselves on an ad-hoc basis (and kept failing at it).
Rust is one example, Erlang/Elixir are another, and when OCaml finally gains parallel abilities, it will be an additional (and very fine) example as well. Surely other examples exist.
If we're now discussing just the point you quoted me on -- I on my side take issue with the people who are way too liberal and love to pretend that it doesn't matter what language is being used regardless of the business and technical problems. Because oh my god yes, yes it does matter. A lot.
Rust is already getting foot-guns like C++ did.
Modern C++ (>C++17) with RAII and neat libraries is already an elegant, high performance and high-productivity language with good IDE/editor support.
And its not like C++ is standing still. You have standardised concepts, modules and co-routines out this year. Modules will fix several of C++'s shortcomings wrt build tooling and setup. I am personally looking forward to C++ 20 co-routine support in all the major compilers - will enable one to develop graceful-async, portable and stable libraries without all that Rust churn needing re-writes all the time.
Competition is good, but its best to take the Rust blinkers off a bit.
I am not fangirling for Rust at all.
I am saying that it's much easier to make costly mistakes in C and legacy C++. I am aware of the modern efforts in C++ but they don't help with the literal mountains of legacy code that has to be maintained out there...
Of course you have to care about the problem domain first, but it's absolutely wrong that nobody commit their budget to fix C++ safety problems. There is a massive amount of static and dynamic tooling to mitigate its problems and a substantial part of the C++ evolution these last years was to improve safety. Rust was able to go further than C++ in this area since it is not yet constrained by decades of history.
That's a weird way to put it. I would blame the carpenter for picking the wrong tools.
BTW, rustls exists and is a rewrite of OpenSSL of sorts.
I do wonder how it happens as I upvote such people, but I've been posting here for years and still don't have downvote privileges. I do wonder sometimes who controls the downvotes.
I also think that's why lisp people are so feverish about lisp as well: as you get more involved in the language with its charms, the more excitement you get and you want to see the language in more places/contexts
https://www.patreon.com/redox_os
The fact that he is only getting around $1,000 per month on Patreon points to deep, deep flaws in humanity. Especially when you compare it to other funding amounts for say Linux or popular smut on Patreon. A total lack of foresight and values.
The true deep flaw of humanity.
Plus if I was going to fund a toy research kernel that aims to increase security, I'd go whole-hog and bring Singularity OS[0] back from the dead, because at least that plays with new ideas and tested the limits of what was thought to be possible.
[0] https://en.wikipedia.org/wiki/Singularity_(operating_system)
But keep in mind Singularity is still a toy OS. It has no use-case, and doesn't aim for real-world goals that would make it commercially successful. I just think that even as toy OS's go, it was one of the more interesting ones of recent times.
[0] https://www.microsoft.com/en-us/research/wp-content/uploads/...
There's a few good reasons in my mind:
* proof of concept - if a language claims to be a systems language, being able to point to an OS in it is pretty good proof that it's true.
* dogfooding - if there are problems with how the language works for the OS case, they can be highlighted by the OS. (this is similar to self hosting a compiler)
* community building - some folks want to work on a new OS, and doing it in a specific language helps grow both groups and at least introduces them to each other.
For me the 'logical' thing to do to build a 'safety first' OS in the 21st century would be: use the seL4 microkernel and then Rust to build the full OS kernel.
Though I don't know if Spectre changed this.
Edit: It seems RedLeaf is immune to meltdown due to its design while seL4 has meltdown mitigations[2]. I would assume the same goes for spectre, but that would be an assumption on my part. (Note again, I've only skimmed [2] so I can easily be wrong here.)
[1]: https://www.ics.uci.edu/~aburtsev/doc/redleaf-hotos19.pdf
[2]: https://www.usenix.org/system/files/osdi20-narayanan_vikram....
[0]: https://arxiv.org/abs/1902.05178 [1]: https://security.googleblog.com/2021/03/a-spectre-proof-of-c...
That said I can imagine a two layer system: if you use 100% safe Rust you have a 'fast path' direct access, otherwise you fallback to slower hardware protection.
An operating system is the essential piece of systems work, so a Rust OS should serve as a demonstration of the efficacy of Rust in systems work. Is the performance on par with contemporary operating systems? Is the code about as easy to understand as contemporaries? Is there a reduction in memory safety issues vs contemporaries?
Positive results from such a project could inspire others to use Rust for more systems work and perhaps integration into other OS kernels and/or key userspace libraries.
Plus, writing an OS is kind of fun.
I'm interested in what Torvalds meant by these odd header files, does anyone know?
"- this will break the minds of everybody who ever sees that expression."
It indeed broke my mind!
sizeof(*(void*)1)
https://stackoverflow.com/a/49481218
I'm not a C dev BTW.
Edit: I am also a proponent of careful re-writes of very old userspace utilities (eg: ping) accompanied by very rigorous tests to ensure there aren't any behavour changes
That would be great, but a prerequisite for that is that the rust language is stable to ensure that there aren't any behaviour changes in the compiler itself.
That being said, it is true that Rust doesn't yet support all the platforms that are important to the kernel. That's one of the reasons this is starting with drivers, and is one of the reasons why some folks are pursuing a Rust frontend for gcc.
If you're still running a system with an arcane ISA, support it yourself. 99% of the people running modern Linux are interested in three CPU architectures (four or five once RISC-V gains traction), all of them supported by Rust. Supporting old or obscure ISAs is not a priority at this point compared to bringing kernel memory safety into the 21st century, which Rust helps a lot with.
A far as I know, the latter supports many architectures the former does not.
I am a C programmer by trade who has played with Rust. It is a great language but there is very little you can do safely. In higher level projects you can mostly push the un-safety into the standard library.
But when you have to do everything yourself, all of the "buggy" code has to be marked unsafe anyway. Allocating and accessing(!!) memory is unsafe. Interfacing with C or asm is unsafe. Your buffer/vector implementation is unsafe.
Basically, all the common culprits for memory bugs in C would be the code marked unsafe if rewritten in rust. So what is the benefit?
One of the key features Rust enables is that you can build safe abstractions on top of unsafe things. Another key feature is that those abstractions often mean compile-time, but not run-time, cost.
If you build the right abstractions, you can minimize the amount of unsafe code pretty dramatically. I don't have numbers handy for our codebase (and it's not open source yet, but will be eventually), but as an example, the Redox documentation (a Rust-based kernel that is, uh... linked... below) says: https://doc.redox-os.org/book/ch01-07-why-rust.html#unsafes
> A quick grep gives us some stats: the kernel has about 300 invocations of unsafe in about 16,000 lines of code overall. Every one of these is carefully audited to ensure correctness.
The advantage here is that, if and when there are bugs in this code, you only have to look at the modules containing those unsafe invocations, rather than the entire codebase.
There are also other advantages to the Rust language that make it worthwhile too, regardless of safety percentage, but it is true that this is often people's first concerns. The code that landed in linux-next has a lot of unsafe in it, because it is specifically trying to build out the proper abstractions here. The idea is that drivers built on top of this code will not need nearly as much, generally.
That is a brilliant feature I had not thought of. Thanks!
I would be cautious with statements like that. You can limit some kinds of operations to unsafe code, audit the daylights out of those, build abstractions that make it impossible to use that unsafe code from safe code in an unsafe manner, and still not be doing the right thing.
A recent example of using private constructors and zero-sized compile time invariant enforcement is an embedded blog here: https://www.ecorax.net/macro-bunker-1/
The embedded rust community uses this typestate pattern to build safe abstractions around pin interactions. Very similar things can be done to build safe abstractions around the volatile memory accesses etc that the kernel does.
Even extremely performance-oriented and low level projects such as the standard library or the Redox kernel are less than 5% unsafe code. There's lots you can do safely.
The benefit, and really the point of unsafe, is to encapsulate the unsafe stuff behind safe APIs and to use Rust's other features, like the borrow checker for example, to enforce correct usage of such API.
Ex.: I was using nanovg (vector graphics library in C) to build a GUI app some day. I was using Rust and a bare-bones FFI library that provided a 1:1 mapping to the C API. On top of this unsafe C-API, I built a wrapper library in safe Rust that doesn't allow me to use nanovg incorrectly. And not in the use-after-free sense of incorrect (that too), but instead in the "you can't call this function between these other two, that's undefined / will produce unexpected graphical results".
So the power of unsafe is to encapsulate "unsafe" behaviour and to enrich it with compile-time information, to build guarantees around it.
You've answered your own question.
It's really interesting that any C code already has to have a memory management ownership model, and the consistency of that model makes Rustification easy / hard.
For example, an H264 encoder/decoder module in the kernel: let's say I have a Freescale/NXP i.MX6 platform with a HW H264 decoder, and I have to create kernel module to talk to it. Then use this module from an user space application to decode a video file.
What I did (modified actually) in the past is a module that allocated a couple of (aligned) buffers in video memory and then passed them around between the HW decoder, kernel module and user space application.
This is usually done to not force a deep copy of >100KB buffer for every decoded frame. This memory is shared between different parts of the implementation, meaning that the same buffer belongs to up to 3 parties at any given time. Also, the allocation is done once and lives for the entire application life, usually allocating like 3 buffer so when one buffer is being decoded I can still use already decoded buffers for writing them into a file.
I know there is an alternative (hack?) using something called Pin (possibly clashing with, like, a GPIO Pin?) but I didn't get so deep already with Rust to really understand it.
Ah! whatever driver/module they do, it should be accessible with C. So Rust must fill the gap between "Some C code in userland" <--> Rust <--> "More C code in the kernel", so I'm really interested to see how ownership works in this case.
Userland C app says: "Hey Rust kernel module, take this struct and do something", the I go and change the contents of that struct from another thread in my userland application. Would this be possible?