It is exciting to see Rust start to replace legacy C code in the Linux kernel. It is only the beginning, but everything seems to point to the start of a gradual migration.
Reading about Asahi Lina's experience doing GPU driver development was intriguing:
Nothing in rust actually limits or makes it any harder to handle something "more dynamic" like JSON.
In rust you just make a JsonValue enum that represents the values and the data types that you're handling.
In something like Python, you just use the built-in runtime value type to represent the structure.
Either way you still have to plumb up the data ingestation and your codepaths that you want to use the dynamic data in. For example you still have to check if the "name" key's value is a string and not an object, array etc.
Otherwise, you probably could just use some accessors that access the values willy nilly that coerce them to text, return an error or panic, that could also exist in rust (some of these things you would have to wire up manually, some are provided by the language and others are fixed by just annotating structs.)
To summarize, fail to see the point you're trying to make, do you have some specific example you are thinking of that I can take a look at?
You made my point, in that it is a slightly more involved process over say Python. I didn't say that it's a "problem" or "impossible", just that it is making your life slightly harder.
Something that I am interested in doing is generating a json schema at runtime, and then parsing some incoming json to see if that matches the target schema. Do you know of any crate already providing this kind of functionality?
I haven't found dynamism like that to be a problem. It's slightly more verbose, but not by much. Instead of `foo: int = unknown['foo']` like in Python, you do `let foo = unknown.get_u64('foo')?` (or something like that) in Rust with `serde_json`.
Same. I gradually learned Rust while I was working in a job where I wrote C on a day-to-day basis, and after a while I realised that Rust was making me a better C developer by forcing me to think more systematically about ownership and lifetimes than I really had to that point. It significantly changed how I designed and structured my code, even in other languages.
Not to say I couldn't have gotten the same education from another language, but something about Rust clicked for me that others didn't.
Our Rust code is much more concise than the previous C code. It takes about three times as much C code versus Rust code, but the C code feels much more brittle and less portable.
There's something incredible about rust in that once it clicks, it results in profound changes in how you are thinking about your code. I know it has certainly improved mine in numerous ways.
What are you compiling? I've "cargo install"-ed - which means deps had to be built too - two rust tools today on a 4c/8t laptop and they were done in maybe two minutes each. Building AUR packages of some C tools isn't much faster in comparison.
How much longer does `rustc` typically take to compile the same programming tasks compared to `gcc`? Is it just "a bit" e.g. 50% slower or by an order of magnitude?
It's not about the compiler, it's about the language. Rust, and C++ as well, have features that generally speaking require the compiler to do a significant amount of work to compile the program. Plain old C can easily compile more than an order of magnitude faster than an equivalent idiomatic Rust or C++ program, even if compiled with the same compiler suite (gcc).
You can install rust toolchain with https://rustup.rs/ in 1 minute, and git clone some project to see for yourself.
It's impossible to answer your question directly, as it depends a lot on dependencies and language features.
That very much depends on the code you're compiling. The factors that come into play are monomorphization (which means the compiler builds one copy of the code per type it is called for), procedural macros (which need to be fully compiled before being able to expand code using them), whether complex type shenanigans are used, etc. etc. Absent that, Rust will compile roughly as fast as C nowadays.
I thought this was an interesting question, so I just compared GNU coreutils [1] and the uutils/coreutils Rust implementation [2]. I've never built either of these projects before, but I was able to easily build them both within a few minutes. I think they're of comparable size: uutils/coreutils has 102 coreutils programs, whereas GNU coreutils has 109 programs. And I liked your phrasing of testing the "same programming tasks": I'm sure that these projects have different implementation styles and philosophies, but at the end of the day they're both trying to implement a similar set of utilities.
My hardware is an 11th gen Intel i7 (4-core, 8-thread) laptop. I included time to build but did excluded time to fetch dependencies. I measured real (wall) time and user time with time(1). I compiled clean release builds in both cases (I think). Here's what I found:
uutils/coreutils: I fetched dependencies (`cargo fetch`) ahead of time, then just ran `time cargo build --release` to build everything. This took 1m37s wall time, 4m19s user time. It looked like about half of that time was spent compiling the `coreutils` multicall (Busybox-style) binary itself, after compiling all the individual `uu_*` utilities (`uu_cat`, etc.).
GNU coreutils: I fetched dependencies (`git submodule update --init --recursive`) ahead of time, then read `README-hacking`, which indicates a three-step build process:
time ./bootstrap: 2m42s wall time, 4m32s user time
time ./configure: 0m36s wall time, 0m21s user time
time make -j8: 0m21s wall time, 2m24s user time
I don't know exactly what ./bootstrap is doing? It's a 1500-line shell script. It does seem to download pofiles at the start, so knock 10s off its runtime. A lot of its time seems to be spent in gnulib-tool(1); it runs aclocal(1) and m4(1) near the end, but I didn't notice it running cc(1). (I'm just watching top.) Maybe someone who knows this codebase better can comment how to fairly compare these.
So, if we're measuring "time for clean release build on a new dev's machine (not including network fetches)", we're looking at ~200s for GNU coreutils and 97s for uutils/coreutils. If we discount ./bootstrap for whatever reason, that goes down to 57s vs. 97s. Either way, it seems within a factor of roughly 2, with the winner going one way or another depending on how you count.
Compile times have greatly improved over the last few years. And if you aren't importing 100+ dependencies from Cargo.toml or using a lot of templates/traits (which I suspect kernel code won't), then it's even better.
Edit: Also kernel compile times are a problem for the elite few, while kernel bugs are a problem for everyone.
The first time you build, sure but that’s true for C/C++ applications too (boost command_line or applications with lots of protobufs). Overall my experience that build times on Rust are waaaaaay better than C++ all things being equal (i.e. similar level of features). And honestly the developer experience of clap I’ve found to be a lot faster to work with than equivalent libraries in TypeScript which is saying something considering the typing on Rust is a lot more strict and cumbersome to work with.
I haven’t done any Rust-on-Linux development, but it seems like this would be a non-issue there? A lot of the compile time issues for Rust come from cargo packages pulling in tons of monomorphized types, but I assume Rust-on-Linux has few-to-no dependencies (I’m pretty sure they don’t even use the standard library).
I didn't mean to imply that the Rust standard library increases compile times, just that Rust-on-Linux is so dependency-lite (as evidenced by the lack of std) that I expect compile times to be good.
Is there evidence that Rust compiles any slower than C++? Granted, it doesn't compile at the speed of C, but it does a lot more and you can write a lot denser code, so one line of Rust may easily supplant dozens of lines of C.
Compared to C, yes. However the tradeoff you get is tighter iteration, with more errors being found at compile-time rather than having to wait for runtime.
If you have to restart an embedded system or schedule a job on a cluster to properly test, that can work out to huge savings.
It would still be nice to see them substantially improve. I know there’s been effort to get debug mode to build substantially faster with cranelift, but I forget if that’s in stable now.
According to pcwalton appears avoiding monomorphization on its own may be unlikely to yield substantial gains [0]:
> I doubt that a hypothetical version of Rust that avoided monomorphization would compile any faster. I remember doing experiments to that effect in the early days and found that monomorphization wasn't really slower. That's because all the runtime bookkeeping necessary to operate on value types generically adds up to a ton of code that has to be optimized away, and it ends up a wash in the end. As a point of comparison, Swift does all this bookkeeping, and it's not appreciably faster to compile than Rust; Swift goes this route for ABI stability reasons, not for compiler performance.
> What you would need to go faster would be not only a non-monomorphizing compiler but also boxed types. That would be a very different language, one higher-level than even Go (which monomorphizes generics).
It's way faster than C++ - doubly so if you count "time spent grokking a compiler error" - and if that's the other option (it usually is), I'll take Rust
True, but I've been highly bullish on that: if that's a core complaint, it seems one of the easier ones to improve in the medium term.
A host of other issues, which Rust notably doesn't suffer from, are almost "unfixable", even in the long term: think build system, type system, memory safety. Rust is fine in these regards.
> Torvalds answered that, while he used to find problems in the LLVM Clang compiler, now he's more likely to find problems with GCC instead; he now builds with Clang.
As I mentioned on mastodon, there's lots of bugs still to be fixed everywhere, but even if we don't fix them, providing competition in the toolchain space has been worth it to users.
I start taking LLVM serious when they give easy to follow documentation on how to compile LLVM completely GCC/binutils/gnu libc independent. I mean a single go to point with easy instructions.
You have to Google everything and then you find out about EXPLICIT_LIBGCC_OPT_IN and you read unofficial responses to issues that describe how to use libunwind and compilert to replace libgcc_s. But all that stuff is out of date fast and never works. Even Rene Rebe who developed T2 ranted about this in 2019 and he couldn't completely figure it out without doing a shitton of workarounds and patches. You need to be fucking Stallmann or Torvalds or whatever computer scientist genius themselves to achieve this. Or Google who did this for Android and Fuchsia. My point is it is rocket science at this point.
Fucking make that straightforward already dear LLVM project.
Regardless of whether you take LLVM seriously, it's a serious project in serious commercial use in many environments.
> Fucking make that straightforward already dear LLVM project.
Much of what you are asking for can be enabled by not-rocket-science changes to the driver. Give it a try.
The LLVM project is much more "bazaar" than other high profile Open Source projects, so I think you'd have a relatively easy time convincing folks to accept your changes provided they don't break existing tests.
macOS LLVM is a bit of a special distribution. It's nice that there's a mostly well built LLVM distribution there, but the pain in the ass is that they ship a custom linker and _don't ship lld_, which means you can't readily use their distribution for a lot of common cross build targets - you need to go build at least an lld to get over the line.
It's "ok" along these axes, a bootstrap for gcc isn't all that straightforward either, the main difference being that if you've already done "apt install build-essential" or similar it just so happens that "it's done" - albeit entirely non-hermetically and the bounds of "it" is probably entirely unknown to the user. I had this discussion with a kernel engineer once who insisted gcc had almost no external dependencies, and he believed this for this reason.
The parts that make it only "ok" are largely cmake derived problems - yes there's sometimes incomplete docs and so on, but largely the confusing parts come from the multi-project split and cmake reconstitution. I'm particularly excited for a future when llvm-libc has made some more progress, at which point a very hermetic build will get quite a bit easier.
The thing is that if you're cross-building this is really the answer, cross building with the deeply intertwined gcc chains is an equally awful experience. I'll take a static clang tool build I can zip up and give to everyone (except nix, because they decided loader hell should only be played on max difficulty) over a fragile distro intertwined mess any day.
> I start taking LLVM serious when they give easy to follow documentation on how to compile LLVM completely GCC/binutils/gnu libc independent. I mean a single go to point with easy instructions.
That's a very niche use-cases. Most people don't care about being "completely GCC/binutils/gnu libc independent".
It's fine to care about your niche use case, but to claim a project is not "serious" because this one thing isn't supported is ridiculous.
I've been compiling my kernel on Gentoo with LLVM for around a year (since I moved to Linux Desktop) and I've had 0 issues. Granted, I run a very stripped down config, with almost only related to my hardware enabled, but I run the latest unstable (not Git) kernel, 6.6.2 as of today.
I remain unconvinced about the viability of Rust for many kinds of systems programming. I'm not sure how I feel about it in the Linux kernel, although I suppose it doesn't matter since it's likely to not penetrate very deeply so to speak.
If there are any Rust experts around...what am I missing? The way I see it, Rust is still fundamentally designed to work at a higher level of abstraction than C, and is still mostly dependent on C (even C++ really, due to LLVM/Clang). And bare metal or "first compiler" support for bare metal doesn't seem like it's really intended to be a first class use case. At least that's my impression of the ecosystem from the little time I've had to play around with it.
Is this mostly just a thing to get more young people interested in kernel development...allowing them to start out in less important areas and in a language they are passionate about? Or is this a serious proposal about the future of operating systems and other low level infrastructure code? Do you just program everything in unsafe mode? What about runtimes?
It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd. Whereas for myself and most of the C programmers I know, Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does. I could spend hours writing examples of this but it's even apparent in the way the Zig team has handled its relationship with LLVM, where they seem very serious about trying to not accept it as fundamental to their language and eventually even eliminate the dependency on the C++ code...with Rust it doesn't seem like this is even on the minds of most of the users. It's purely a dependency for them and that isn't seen as being fundamentally at odds with the intended use cases. That is totally okay...but it ain't the C culture if we are to accept that such a thing exists.
This was my impression as well. I haven’t spent enough time with Rust to hold a hard stance on it but it definitely seems like it was not designed for “bare-metal” applications.
I hadn’t heard of Zig before this but I’m definitely going to look into it now. Got any recommendations for a good intro to zig?
How is that the case? You have bare pointers and the ability to manipulate them, allocation is explicit and can be disabled, you can remove the stdlib entirely, add your own allocator support, etc etc etc.
Well, sorry, but you're not being very clear. First you said it didn't seem designed for low level and now you're saying you aren't sure what the point is if it can do low level things?
It would help if you clarified. I don't want to try to clarify things and end up putting words into your mouth.
The point is that you limit where you're doing that.
Like just as an example - I can write an allocator and toggle register bits etc. All of that requires unsafe code, raw pointers, etc.
But I can then build on top of that in safe Rust, with all the guarantees that brings. I still have to check that the unsafe allocator or whatever work soundly, but Rust checks the stuff on top of it.
This reads like “I’m not sure what’s the point of power tools if every now and then you need to reach for a plain old hammer or screwdriver”.
What’s the point of high-level languages at all if they’re just going to be implemented on top of a CPU that will happily chase raw pointers and divide by zero if you ask it to?
Because everything else can be expressed safely and more concisely than writing it in C or raw asm. And you have to pay virtually zero abstraction costs for the pleasure: see Rust’s comparative performance in language shootouts. Not infrequently Rust even beats C’s performance due to better visibility for optimizations.
And when you do still have to resort to unsafe low-level bit twiddling, you get to keep all the other benefits of having a modern, powerful language.
Do you genuinely think the Linux kernel is just throwing Rust in to attract kids who just follow the latest programming fads?
It is not designed for, but Rust works well on microcontrollers, such as Arduino. For example, a blink program for Arduino in Rust is less than 300 bytes in size when compiled with "s" optimization level. It's a bit larger than C because of vector of interrupt handlers and error handling.
The design of async/await took "using it in embedded" as a constraint. This is why it doesn't allocate, for example. Even C++ coroutines have one allocation, though they say that it can often be optimized away.
This means RTOS-like projects can use it for tasks, and it works well. Embassy is an example of such a project.
(Yet, I should also point out that you don't have to: at work we keep ours synchronous, for Reasons. Rust lets you do what you want.)
This gives a decent overview of some of the areas where the focus of Zig is different than Rust. Other than that...I'd just start by reading through the standard library code and the repo to get a feel. It's very very similar to C so you will be immediately comfortable, just cleans up some stuff like removing default global allocator (you pick your malloc basically), better macros (comptime), and encouraging use of non-null terminated strings and fat pointers etc.
I use Rust for bare-metal development (no libc, no allocator, no panic, 32k size limit), and it is comparable to C and C++ for that use-case. The project is 99.5% Rust with a bit of assembly glue for startup/interrupt handlers.
You will need to generate code for the target which means LLVM/Clang (a C++ project) must support at the very least some amount of codegen for that target.
You can use the cranelift backend, written in Rust, if you prefer. Nothing about LLVM being the best choice by default has to do with the choice of implementation language.
mrustc is written in C++, so I don't see how it's a counter to what the parent is proposing. I also don't know anything about mrustc's performance characteristics, I didn't think performance was a goal, but it's also been a while since I've checked in on it.
I think the assertion is based on the fact that LLVM is written in C++. Doesn’t directly introduce a C++ dependency to the kernel, but it does introduce it to the toolchain.
But rustc is a cross-compiler, you only need a C compiler for one platform and you get a Rust compiler for all platforms. And it's not like LLVM is doing anything that fundamentally prevents it from being written in Rust (see also the Cranelift backend for Rust, written in Rust). I'm still not quite sure what the concrete concern is. Can you elaborate?
> In August 2012, the GCC steering committee announced that GCC now uses C++ as its implementation language.[57] This means that to build GCC from sources, a C++ compiler is required that understands ISO/IEC C++03 standard.
> On May 18, 2020, GCC moved away from ISO/IEC C++03 standard to ISO/IEC C++11 standard (i.e. needed to compile, bootstrap, the compiler itself; by default it however compiles later versions of C++).[58]
So for the past >10 years, gcc and clang have both had a C++ dependency. What is unique about LLVM/Clang here?
If I understand what you’re considering, it’s that the compiler itself may have been compiled from C++ sources, so dependencies in that accounting scheme derive from transitive closure.
Alternatively i consider a compiler might only be presented as a binary, with no visibility as to its own source code, so i consider in that case just a Rust program and a binary that compiles it to a native binary linking with binary object files also of unspecified source code languages.
Sorry, we’re in agreement. My reply was meant to reply to the parent claiming there’s a C++ dependency in the tool chain when building everything from source - that’s true for the Linux kernel today since building GCC requires a C++ compiler as well.
gcc is programmed in C++ too. The language of the compiler doesn't matter. What matters is C makes it too easy to make memory errors which Rust protects you from. This is why Linux should migrate to Rust and you can see this migration starting with adding support for Rust based drivers.
I disagree that the language of the compiler doesn't matter. It absolutely matters in some cases, all of which are cases where (ANSI) C is pretty much the only option because of the inherent difficulty of C++ compilers.
This still doesn't explain why the language of the compiler matters. I could write a C compiler in Pony-lang targeting a 30-year-old MCU were I so inclined.
The available compilers targeting your microcontroller certainly matter, though. You certainly still find lots of options that aren't Rust-compatible, but a non-trivial number microcontrollers are ARM or RISC-V based now, and can be targeted by LLVM/Rust.
Rust is not fundamentally designed to work at a higher level of abstraction than C. Rust is designed to work at a range of abstraction levels, from as low as the lowest level C code to something fairly high (but still short of a language like python). Rust is designed to make it easy to quickly build abstractions on top of super low level (rust) code, because usually that's the less error prone way to do things - that doesn't prevent it from working at those levels though, and if anything it makes working on systems where you have to work on super low level code much easier to use.
When working with low level rust you don't "program everything in unsafe mode", you program a few really low level bits in unsafe rust, but quickly make abstractions that allow you to avoid making the vast majority of your code unsafe. This is opposed to languages like C/C++/zig/... where there isn't anything but "unsafe".
Rust doesn't really have a runtime by default (beyond libc for non-baremetal code).
There are a few tiny niches where rust isn't designed to compete with C, but they really are tiny niches. For example where chars aren't 8 bits.
I'm honestly just not following what you're saying about C++. Rust code tends not to depend on C++ code apart from maybe some system libraries that everyone depends on. The rust community tends to be overly-keen on rewriting those libraries in rust, not underly-keen. Obviously some people will make code that manages to depend on the language - but rust doesn't make it easy (like zig does...).
As for the relationship with llvm - rust doesn't actually have a hard dependency on it anymore (with support for a compiler backend called cranelift, written in rust). I also just don't see "the compiler depends on another language" as a huge impediment - as others have mentioned C compilers tend to as well...
Having a memory safe language in the kernel is very serious. More and more C programmers I speak to say they realise with Rust they will likely never work with C again full time.
Yes Rust helps bring in the next generation of Linux devs. It needs to to survive. But it's not because young people like Rust, it's because C developers like rust too.
Yes, Rust has unsafe and unsafe rust is trickier to get right, but you use it opt in and it's only a small amount if the code. Zig is still having unsoundness bugs in relatively simple code
I don't have much experience and I see C as something I would never be able to touch as they seem quite high level for me with the manipulation of memory and stuff.
And from what I see, Rust can sort of make it easier by making you think about the lifetime of the memory while coding and that's why I really wanted to learn Rust
"Not having C++ anywhere in the stack" is not a goal of the C language or any major implementation I know of. The two major OSS C compilers, GCC and Clang, are both C++ codebases.
It's basically a part of the larger project to "Rewrite It In Rust" -- to replace C code with Rust code wherever possible. C is dangerous to work with; and with safe Rust (but not with Zig), entire classes of bugs that have dogged the industry for decades simply go away.
>The way I see it, Rust is still fundamentally designed to work at a higher level of abstraction than C, and is still mostly dependent on C (even C++ really, due to LLVM/Clang).
I think this might touch on part of the problem. Rust can work on higher abstractions but rust doesn't have to. It is a matter of the code base. I've worked on some that felt very similar in abstraction to C code bases. One in particular was a high performance nginx module. Most of the rust code interfaced with nginx directly through zero cost FFI interfaces to the nginx structs.
I've also worked on extremely meta, high level code that was written by a former Haskell dev.
Like many languages that can serve dual roles, the styles tend to adapt to it. You see very different styles of Java when looking at spring web apps or embedded android systems. Every language you have to pick and chose parts of it that, build a coding standard and need to enforce it. C is exactly the same and also has several roles people pick it for. The style of code of embedded systems, high performance networking systems and codebases that prioritize highly portability tend to be different and use different idioms. I think any of of us who have worked in a C code base that falls in one of those camps can name banned idioms that are normal in the others.
>> It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd.
My career before Rust was very C heavy. I've work from OS-less embedded systems, to device drivers, to Android porting. I've been doing Rust professionally since 2016. This just isn't my impression. A lot of folks I've known from college and internships who's careers went a similar direction ended up in rust independently. There is value in having a language that let's us get the job done that we used to do with C but with more high quality static code analysis. The business case and use case is very clear.
While I haven't done any bare metal development in Rust, in the areas I have worked I've found compiler, community and crate support to be great. I don't know where you are getting this impression. I think if I only read hackernews articles about Rust I would get the impressions that is Haskell that can work on bare metal. I've noticed articles that tend to get upvoted here tend to be either about 1) how quick adoption is 2) higher levels of abstraction like HKT or GATs 3) WASM 4) writing web services. This article is a great example of 1. 2 is catnip for this crowd. The majority of rust devs never need to know what a GAT is but if you used HN as a metric you'd think it was a daily problem. 4 is something that is easy to write, makes good content, approachable but isn't necessarily Rust's best use case. I'm not saying you've only been reading this site about Rust, just saying that it is easy to get a skewed impression of the language unless you really dive in.
> Is this mostly just a thing to get more young people interested in kernel development...allowing them to start out in less important areas and in a language they are passionate about?
Not likely. At the moment you need to do extra work to get Rust working well. It's not exactly beginner friendly and doing work in the kernel, you'll need to dig into C anyway.
> Or is this a serious proposal about the future of operating systems and other low level infrastructure code?
Serious code already exists, so... Yes?
> Do you just program everything in unsafe mode? What about runtimes?
The few places that need direct access / unsafe are almost all single-line areas with an explanation.
> but it ain't the C culture if we are to accept that such a thing exists.
I don't get that whole part of what you wrote. What seems to be the culture / why does it matter in the technical sense? Some parts of the compiler are written in C++ - what's the specific issue here?
I attempted to port a low latency trading system from C to Rust last year, but ended up abandoning after 6 month of development.
The overall feeling that I got is that Rust made my life a dream for 80% of the codebase, the _non critical, non high performance part_.
There are tons of well written libraries, cargo is awesome, the performance is on par with what C++ would do, etc.
But the last 20% of the code base, the high performance part, was a nightmare to implement. The Rust static safety basically get in your way constantly for any kind of non obvious memory layout (self reference, etc). Dynamic safety (cells, Rcs, etc) just add too many overhead for the the critical path, and the "escape hatch" of "unsafe {}" ended up being 1000x more error prone that C.
I keep some resentment against the Rust community from this experience to be honest. I felt like instead of understanding the constraints & limitations of a fixed, microsecond time budget that I had, and trying to find solutions and be open about possible improvements, the overall trend was more in trying to defend Rust with whatever it takes.
> and the "escape hatch" of "unsafe {}" ended up being 1000x more error prone that C.
IMO this is the biggest issue right now. `unsafe` needs some refinement, certainly.
> the overall trend was more in trying to defend Rust with whatever it takes.
This can sometimes be the case, unfortunately. But fwiw I think a lot of us in the community agree entirely that there's room to improve in the unsafe world.
At the risk of sounding like I’m trying to “defend Rust with whatever it takes”, your argument seems bizarre to me.
Even at the surface, if your desired solution is so antithetical to the type of structure that Rust tries to push you towards, you can quite literally just write this component of your code as if it was C. Just using pointers instead of borrows and tag all your functions as unsafe and… pretend you’re writing C? At that point you’ve, I think, more or less disabled 95%+ of the bits that would prohibit you from writing code exactly as you would in C. You don’t get the benefits of Rust for that part, but at least you get them for the remaining 80%.
Or you can very literally just write the “hard part” in C and call it from Rust. You might have to make sure you can’t panic across language boundaries, but other than that the C interop is just about the best I’ve seen from any language.
I also don’t entirely understand how unsafe can be 1,000x more (or even 1.5x more) error prone than C. But I’d love to hear how. The only “trick” to unsafe is that you should aim for your unsafe blocks to be “unit-safe”. Meaning they might do something unsafe inside, but from the outside looking at it as a black box, the unit functionality of it should be safe. I don’t think the docs do a good enough job here of encouraging that style of design. You can violate this guideline, but doing so without sufficient care is quite likely to result in bugs. But of course if you did the same approach in C, you’d have a similar outcome.
The only real way I can reconcile your points is if the performance-critical bits that seriously impact the design of your program are scattered uniformly and don’t have anything resembling clean boundaries. I suppose that’s a real possibility but it does seem very foreign to me.
Yeah see, that is what I thought at first as well.
I sort of imagined that I could get the best of both world, and just "unsafe { <C style code> }" my way out for performance critical things.
But the thing is, the static safety boundaries of Rust allow the compiler to make much, much tighter assumptions than C & C++, especially around aliasing rules, un initialized memory, and moves.
When you relax these boundaries with "unsafe {}", you don't enter "C world", you enter the litteral gates of hell where any innocent temporary cast can throw you in a random load/store reordering bug.
Thanks for the reply. I think if you combine unsafe with raw pointers instead of borrows you relax the rules enough to avoid this. But in this specific corner of things, I haven’t had direct experience so you may very well be right.
There are still some Rust-specific details you would still need to handle—as you mentioned, uninitialized memory—but for that one specifically I haven’t found MaybeUninit to be particularly cumbersome.
Do you mean innocent temporary cast from a pointer to a reference? Cause yeah, those aren't innocent. And that's a newer realization and there were serious documentation issues around it.
> Dynamic safety (cells, Rcs, etc) just add too many overhead for the the critical path
A low latency trading system can't have Rc, or any C equivalent of it, in the critical path, because it does allocation. More generally, your story doesn't ring true to me, because the performance-critical parts of a low latency trading system have to be so simple that there isn't scope for any of the tricky bits you talk about, they have to be braindead simple loops over simple data structures.
One of my colleagues wrote a low latency trading system in Rust. He had to learn Rust to do it, already knew C, and today thinks this was the right decision.
> your story doesn't ring true to me, because the performance-critical parts of a low latency trading system have to be so simple
Hu, so it doesn't ring true because... you feel like it's too complicated from what you imagined? That's a first.
> that there isn't scope for any of the tricky bits you talk about, they have to be braindead simple loops over simple data structures
That is far from the reality of low latency trading systems. Let's just look at the very first building blocks:
1) You will need some form of userspace packet polling, along with its associated multicast A/B line arbitration. That alone is not "simple braindead loops", and we're just talking about getting 1 UDP datagram here.
2) You will need some form of packet queueing before decoding, so that you do not drop packets under bursts. That most likely means some variant of lock free ring buffer in shared memory. Far from "a bunch of brain dead loops" as well.
3) You will need to decode packets and maintain an order book state. That mostly implies a purpose built b+tree. If you've implemented any of those, you will know it's also far from simple.
4) You will have to compute some indicators based on order book state, which most likely will lead you to some form of linear algebra.
5) You will need to disseminate such indicators to perform some simple close formula variant of portfolio optimization. More linear algebra.
6) etc.
I'm happy your colleague that he managed to do all that without struggling, that was not my experience, that's all. But to say that it's "just braindead loops over simple datastructures" is either a sign he didn't do a _real_ low latency system, or he just was somehow spared from the real underlying complexity.
This is a common thing to say for folks who are really good at low level C/C++. You guys know too much, seen horrors no one should have ever seen and came out alive on the other side.
The problem is, the things you needed to do to stay alive in this environment are antipatterns in Rust and you’ve learned the hard way Rust actively opposes being fed Rust antipatterns.
Self referential data structures are some of the hardest problems in CS to get right. There are patterns that work with Rust which replace pointers with eg generational indexes. You’ll find people do the same thing in C++ once they get burnt one time too many. I think you’d have a very different experience if you approached your system with an ECS framework, either off the shelf or roll your own.
> Whereas for myself and most of the C programmers I know, Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does. I could spend hours writing examples of this but it's even apparent in the way the Zig team has handled its relationship with LLVM, where they seem very serious about trying to not accept it as fundamental to their language and eventually even eliminate the dependency on the C++ code...with Rust it doesn't seem like this is even on the minds of most of the users.
Yes, the Rust language needs a complex optimizing compiler to really be viable. In this respect it is more limiting than ANSI/ISO C. For a lot of contexts, this isn't really a problem though? You aren't going to be able to build the Linux kernel without GCC or Clang anyway…
I've been writing a little hobby operating system for the past 3 years in Rust and my impression is that the language is fine for low level code. The language improved a lot over the past decade and I think it's ready to be used more.
I've heard good things about Zig but I've never tried it so I can't compare unfortunately.
To focus on the C/C++ comparison. Rust very much leans towards C++ side of things if you go the std library route. It does bake in some abstraction which would not be appropriate for low level systems development such as the Linux Kernel, which is also not what they're going for.
Rust gives you the option of opting out of that `#[no_std]`, which removes some basic tools, such as allocations, panics etc (I haven't used it much, so this may not be fully precise). As such with no_std you do move closer to what you'd consider C development.
That said rust will still have its restrictions, so some of the wild things you can do in C, isn't possible in rust, at least not without really violating the inner workings of it.
But with unsafe you can technically do most things on the level of C, without the overhead, that C++ would provide, via. its abstractions.
I'd expect the Linux Kernel to move in the direction of developing or using their own libraries and data structures, and that becoming its own bespoke part of their use of the rust language. Such that rust only really provides its safety guarantees and ergonomics. Compile times are also quite quick with no_std, as rust only really struggle with compile times, if you have lots of dependencies.
Circling back:
Rust is more like C++ with std library enabled (default)
and C with `no_std`
> If there are any Rust experts around...what am I missing?
Probably more good examples and possibly more ecosystem.
A first big decision is if you're going no-std or not, and if you end up in the no-std world the ecosystem shrinks substantially. If you're building a kernel that's probably not that much of a problem - the same shrinkage occurs for C/C++ - many such projects bootstrap with nearly zero ecosystem anyway.
The examples side is a bigger problem - I've recently been able to watch some of my more curmudgeonly C friends give it a good dive, and after an initial hump they're fairly happy with the core language. They still have regular issues with the ecosystem when they run into, in their words "web dev crap", which comes up even in the stdlib sometimes - a bugbear a while ago coming up down some error handling code paths. They attempted to send patches and hit nebulous arguments against the correctness target, which were largely born of misunderstandings of posix. This kind of thing can come up anywhere, if you take a dependency on some fancy IO abstraction that happens to be written in C, and you take it somewhere "novel" like a BSD, you might well run into the same. The point here is that _examples_ and _exercise_ of these tasks are the things that are going to shake more of this kind of thing out. At the same time though, it's important to reiterate that if you're on the nostd path, then largely you're on your own, which is equivalent to just gcc bare, and this kind of thing generally doesn't come up.
> Or is this a serious proposal about the future of operating systems and other low level infrastructure code?
This is a serious proposal. The outcome is really strong along key axes of correctness and safety. Those of us who've done it (e.g. Fuchsia, where I was) have been able to observe these benefits relative to history with the same teams using other languages (C, C++). We're professional engineers, these statements aren't coming from a place of craziness. The Android team have been writing about their journey: https://security.googleblog.com/search/label/rust
> Do you just program everything in unsafe mode?
Absolutely not. A good amount of bootstrapping effort has been going in across the ecosystem to make it ever easier to avoid unsafe. To take one slice of examples, there's crates that are designed to help you avoid copies while also avoiding dropping to unsafe - they provide tools for automatic structural analysis of the mapping boundary to make it easier to assert the relevant guarantees. Examples: https://docs.rs/zerocopy/latest/zerocopy/ (came out of Fuchsia), https://github.com/serde-rs/serde/releases/tag/v1.0.0 (serde is commonly used, but has more constraints here), https://rkyv.org/rkyv.html (not sure of prominence, but I hear people talk about it).
These kinds of tools get you a long way toward substantially safer code, without needing to think or audit nearly as much. We know that's important, we have plenty of data that demonstrates how important it is, and lately now, we have data that shows how effective it is too (see the aforementioned Android posts).
> What about runtimes?
They're out there, it depends what level of abstraction you're looking for, runtimes means different things to different people. For embedded there's typically a lot more focus on providing libraries rather than a whole runtime framework, so there are crates for a number ...
Oh, and if you want to see some real experience writing an OS in Rust, there's someone regularly streaming their experience here: https://www.twitch.tv/sphaerophoria. I've stopped in a few times, they're doing well and the language never impedes them. I gave a couple of pointers in chat with a few points in their initial TCP build, and initial vbe side, again, none of that was language related, more just whoops, a math bug. I just noticed he's streaming and specifically he's porting Doom to his system right now.
> Whereas for myself and most of the C programmers I know, Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does.
I agree. In general I think Zig makes lot more sense as a C replacement for the kernel.
However, Rust is stable, and Zig isn't. And I don't expect it will be for some years. So at this point Zig is basically unusable for kernel work.
Memory safety bugs are currently being introduced to the Linux kernel faster than they are being fixed. I assume this is true for all major projects but for Linux there's hard evidence for this:
> Memory safety bugs are currently being introduced to the Linux kernel faster than they are being fixed.
I don't understand where all these memory bugs are coming from. Are programmers not testing their code? Are they not running sanitizers, fuzzers, and static analysis? Switching programming languages isn't going to fix broken development process.
The entire point of Rust is that even in the most critical projects with the best tools and very experienced programmers, writing memory-safe C code in complex systems is extremely difficult.
Even C++ is at least somewhat better, with smart pointers and RAII.
The Linux kernel development process is indeed broken. Fixing that would reduce the discrepancy between the rate of new bugs and the rate of fixes. This needs to be done, C is not going away any time soon. But while we are writing C we will be creating new bugs faster than we can fix them. We will never get 100% fuzzing coverage. Nor can sanitizers catch every bug.
As you suggested though, static analysis has lots of scope for solutions. The best static analyzer around is a little thing called rustc :)
the median contributor to Linux is an overworked driver developer (who may not have much software development experience at all) who is throwing stuff at the wall to try to get something to work on their machine, often with inaccurate or incomplete documentation (even or especially if they work for the company making the hardware they are writing a driver for). It's a swamp of subpar C code once you get outside the core subsystems. And yes, there are a lot of people throwing various analysis tools at the kernel to try to catch these, but the tools are quite noisy, and proper testing is hard without the relevant hardware.
Bit of an aside, but these are not the only two options.
It's true that Rust is currently the only stable language in this space, so it has some advantage there, but "Rust or forever C, there are no other options" is of course a load of nonsense.
I wish it had. C# makes it effortless to write high-level and low-level code that lives together. Want pointers and manual allocation? It’s yours. Wanna ignore all that and write interfaces and Java-style classes? You can do that too and pretend that pointers don’t exist.
Conversely, Rust never lets you do anything above a mid-level effortlessly. Traits barely get you halfway to typeclasses and you still have to mess with the borrow checker no matter what you’re doing. Add in lifetimes and even simple high-level code gets bogged down with extraneous low-level details.
I do ML professionally, but for performance bottleneck data processing code I’ve converted a lot of scripts to Rust. It’s been a joy even if I can’t say I’m proficient. The combination of GPT4 and the Rust compiler really help to smooth out the learning curve.
Some of my python scripts which used to take 24 hours to process data now can finish in under an hour. It’s like magic. Granted theres a development cost so I don’t convert everything but it almost feels like cheating watching it go.
Another thing, it seems that the strictness of the language really cuts down on nasty runtime issues and silent bugs. Rust forces you to state explicitly what you want to do.
> Linus Torvalds said that he was seeing a divide between the filesystem and driver maintainers. Developers on the filesystem side tend to be more conservative, while the driver world "is the wild west". Driver authors tend not to understand concurrency, he said, and a lot of the code there is broken and unfixable. So it is unsurprising that there is interest in bringing in a language that better supports the writing of correct and safe code.
> Torvalds answered that, while he used to find problems in the LLVM Clang compiler, now he's more likely to find problems with GCC instead; he now builds with Clang.
> At the conclusion, Torvalds pointed out that there have been problems over the years with GCC changes breaking the kernel; the same will surely happen with Rust, but it will be the same thing in the end.
145 comments
[ 2.1 ms ] story [ 183 ms ] threadhttps://threadreaderapp.com/thread/1577667445719912450.html
At $WORK we are converting a graphics processing application written in C to Rust and the experience has been similarly enlightening.
Learning Rust has been a process, but the dividends are real and exciting.
An example of this is generating a JSON and only learning the correct schema at runtime.
Or connecting to an external database with tables of unknown structures that you can’t type check.
You can, of course deal with all these by wrapping the objects in some structure; but it’s just one more thing you need to keep in mind.
In rust you just make a JsonValue enum that represents the values and the data types that you're handling.
In something like Python, you just use the built-in runtime value type to represent the structure.
Either way you still have to plumb up the data ingestation and your codepaths that you want to use the dynamic data in. For example you still have to check if the "name" key's value is a string and not an object, array etc.
Otherwise, you probably could just use some accessors that access the values willy nilly that coerce them to text, return an error or panic, that could also exist in rust (some of these things you would have to wire up manually, some are provided by the language and others are fixed by just annotating structs.)
To summarize, fail to see the point you're trying to make, do you have some specific example you are thinking of that I can take a look at?
Something that I am interested in doing is generating a json schema at runtime, and then parsing some incoming json to see if that matches the target schema. Do you know of any crate already providing this kind of functionality?
When looking for crates lib.rs is an excellent search utility for searching crates that you would need. For example: https://lib.rs/search?q=json
I think it fits the bill. I was doing this on python side and aimed to use Pydantic and this looks close enough.
Not to say I couldn't have gotten the same education from another language, but something about Rust clicked for me that others didn't.
Biggest takeaways:
Rust's ownership rules enforce "multiple readers, one writer" patterns so plan for such patterns in your design.
Embrace Rust's Result and Option types for error-handling and design. They are elegant and powerful when used with 'match'.
Unit tests right next to the code they test is a huge win. Doc-tests and 'cargo doc' is literate programming (https://en.wikipedia.org/wiki/Literate_programming).
The phf crate (https://docs.rs/phf/latest/phf/) is awesome for global constant data tables.
>> Was there anything that surprised you?
Our Rust code is much more concise than the previous C code. It takes about three times as much C code versus Rust code, but the C code feels much more brittle and less portable.
I actively avoid Rust projects because they take too long to compile.
I wish rustc had an option for disabling monomorphization, a true -Os option.
My hardware is an 11th gen Intel i7 (4-core, 8-thread) laptop. I included time to build but did excluded time to fetch dependencies. I measured real (wall) time and user time with time(1). I compiled clean release builds in both cases (I think). Here's what I found:
uutils/coreutils: I fetched dependencies (`cargo fetch`) ahead of time, then just ran `time cargo build --release` to build everything. This took 1m37s wall time, 4m19s user time. It looked like about half of that time was spent compiling the `coreutils` multicall (Busybox-style) binary itself, after compiling all the individual `uu_*` utilities (`uu_cat`, etc.).
GNU coreutils: I fetched dependencies (`git submodule update --init --recursive`) ahead of time, then read `README-hacking`, which indicates a three-step build process:
I don't know exactly what ./bootstrap is doing? It's a 1500-line shell script. It does seem to download pofiles at the start, so knock 10s off its runtime. A lot of its time seems to be spent in gnulib-tool(1); it runs aclocal(1) and m4(1) near the end, but I didn't notice it running cc(1). (I'm just watching top.) Maybe someone who knows this codebase better can comment how to fairly compare these.So, if we're measuring "time for clean release build on a new dev's machine (not including network fetches)", we're looking at ~200s for GNU coreutils and 97s for uutils/coreutils. If we discount ./bootstrap for whatever reason, that goes down to 57s vs. 97s. Either way, it seems within a factor of roughly 2, with the winner going one way or another depending on how you count.
Methodology comments welcome.
[1]: https://www.gnu.org/software/coreutils/#source
[2]: https://github.com/uutils/coreutils
Using a different linker (like mold) might change that, if most of the time is spent linking everything together.
(Edited for new timings with mold 2.3.3 instead of 1.0.3.)
Can this power be learned?
Edit: Also kernel compile times are a problem for the elite few, while kernel bugs are a problem for everyone.
Standard library actually shouldn’t increase compile time at all really, since it’s prebuilt and included for most platforms (excepting -Z build-std).
If you have to restart an embedded system or schedule a job on a cluster to properly test, that can work out to huge savings.
It would still be nice to see them substantially improve. I know there’s been effort to get debug mode to build substantially faster with cranelift, but I forget if that’s in stable now.
> I doubt that a hypothetical version of Rust that avoided monomorphization would compile any faster. I remember doing experiments to that effect in the early days and found that monomorphization wasn't really slower. That's because all the runtime bookkeeping necessary to operate on value types generically adds up to a ton of code that has to be optimized away, and it ends up a wash in the end. As a point of comparison, Swift does all this bookkeeping, and it's not appreciably faster to compile than Rust; Swift goes this route for ABI stability reasons, not for compiler performance.
> What you would need to go faster would be not only a non-monomorphizing compiler but also boxed types. That would be a very different language, one higher-level than even Go (which monomorphizes generics).
[0]: https://news.ycombinator.com/item?id=38224941
Still, I would say Rust is fast enough now that compile time isn't a show stopper.
A host of other issues, which Rust notably doesn't suffer from, are almost "unfixable", even in the long term: think build system, type system, memory safety. Rust is fine in these regards.
https://github.com/ClangBuiltLinux/linux/issues is our bug tracker for known issues (a few are tracked in llvm's issue tracker). Bug reporters and future kernel hackers wanted!
As I mentioned on mastodon, there's lots of bugs still to be fixed everywhere, but even if we don't fix them, providing competition in the toolchain space has been worth it to users.
You have to Google everything and then you find out about EXPLICIT_LIBGCC_OPT_IN and you read unofficial responses to issues that describe how to use libunwind and compilert to replace libgcc_s. But all that stuff is out of date fast and never works. Even Rene Rebe who developed T2 ranted about this in 2019 and he couldn't completely figure it out without doing a shitton of workarounds and patches. You need to be fucking Stallmann or Torvalds or whatever computer scientist genius themselves to achieve this. Or Google who did this for Android and Fuchsia. My point is it is rocket science at this point. Fucking make that straightforward already dear LLVM project.
> Fucking make that straightforward already dear LLVM project.
Much of what you are asking for can be enabled by not-rocket-science changes to the driver. Give it a try.
The LLVM project is much more "bazaar" than other high profile Open Source projects, so I think you'd have a relatively easy time convincing folks to accept your changes provided they don't break existing tests.
The parts that make it only "ok" are largely cmake derived problems - yes there's sometimes incomplete docs and so on, but largely the confusing parts come from the multi-project split and cmake reconstitution. I'm particularly excited for a future when llvm-libc has made some more progress, at which point a very hermetic build will get quite a bit easier.
The thing is that if you're cross-building this is really the answer, cross building with the deeply intertwined gcc chains is an equally awful experience. I'll take a static clang tool build I can zip up and give to everyone (except nix, because they decided loader hell should only be played on max difficulty) over a fragile distro intertwined mess any day.
That's a very niche use-cases. Most people don't care about being "completely GCC/binutils/gnu libc independent".
It's fine to care about your niche use case, but to claim a project is not "serious" because this one thing isn't supported is ridiculous.
Then what is the point of llvm ?
Is this another "make" ("we don't use make for our project but use cmake to generate makefiles and make to build") ?
Edit: forgot to mention, ThinLTO from 6.6.2 :)
If there are any Rust experts around...what am I missing? The way I see it, Rust is still fundamentally designed to work at a higher level of abstraction than C, and is still mostly dependent on C (even C++ really, due to LLVM/Clang). And bare metal or "first compiler" support for bare metal doesn't seem like it's really intended to be a first class use case. At least that's my impression of the ecosystem from the little time I've had to play around with it.
Is this mostly just a thing to get more young people interested in kernel development...allowing them to start out in less important areas and in a language they are passionate about? Or is this a serious proposal about the future of operating systems and other low level infrastructure code? Do you just program everything in unsafe mode? What about runtimes?
It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd. Whereas for myself and most of the C programmers I know, Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does. I could spend hours writing examples of this but it's even apparent in the way the Zig team has handled its relationship with LLVM, where they seem very serious about trying to not accept it as fundamental to their language and eventually even eliminate the dependency on the C++ code...with Rust it doesn't seem like this is even on the minds of most of the users. It's purely a dependency for them and that isn't seen as being fundamentally at odds with the intended use cases. That is totally okay...but it ain't the C culture if we are to accept that such a thing exists.
Like?
What other you guys do in C that isn't just integer oriented programming (using ints for everything)?
It would help if you clarified. I don't want to try to clarify things and end up putting words into your mouth.
Like just as an example - I can write an allocator and toggle register bits etc. All of that requires unsafe code, raw pointers, etc.
But I can then build on top of that in safe Rust, with all the guarantees that brings. I still have to check that the unsafe allocator or whatever work soundly, but Rust checks the stuff on top of it.
What’s the point of high-level languages at all if they’re just going to be implemented on top of a CPU that will happily chase raw pointers and divide by zero if you ask it to?
Because everything else can be expressed safely and more concisely than writing it in C or raw asm. And you have to pay virtually zero abstraction costs for the pleasure: see Rust’s comparative performance in language shootouts. Not infrequently Rust even beats C’s performance due to better visibility for optimizations.
And when you do still have to resort to unsafe low-level bit twiddling, you get to keep all the other benefits of having a modern, powerful language.
Do you genuinely think the Linux kernel is just throwing Rust in to attract kids who just follow the latest programming fads?
Making sure Rust can work in embedded environments is absolutely a design goal, and choices around this are made explicitly.
e.g?
I am curious
How is "the design of async/await." for embedded programming?
This means RTOS-like projects can use it for tasks, and it works well. Embassy is an example of such a project.
(Yet, I should also point out that you don't have to: at work we keep ours synchronous, for Reasons. Rust lets you do what you want.)
https://ziglang.org/learn/why_zig_rust_d_cpp/
I'm not following the assertion that compilation with LLVM/Clang introduces a C++ dependency on the language being compiled (Rust here).
https://github.com/gcc-mirror/gcc/blob/master/gcc/input.cc
> In August 2012, the GCC steering committee announced that GCC now uses C++ as its implementation language.[57] This means that to build GCC from sources, a C++ compiler is required that understands ISO/IEC C++03 standard. > On May 18, 2020, GCC moved away from ISO/IEC C++03 standard to ISO/IEC C++11 standard (i.e. needed to compile, bootstrap, the compiler itself; by default it however compiles later versions of C++).[58]
So for the past >10 years, gcc and clang have both had a C++ dependency. What is unique about LLVM/Clang here?
Alternatively i consider a compiler might only be presented as a binary, with no visibility as to its own source code, so i consider in that case just a Rust program and a binary that compiles it to a native binary linking with binary object files also of unspecified source code languages.
The available compilers targeting your microcontroller certainly matter, though. You certainly still find lots of options that aren't Rust-compatible, but a non-trivial number microcontrollers are ARM or RISC-V based now, and can be targeted by LLVM/Rust.
Rust is not fundamentally designed to work at a higher level of abstraction than C. Rust is designed to work at a range of abstraction levels, from as low as the lowest level C code to something fairly high (but still short of a language like python). Rust is designed to make it easy to quickly build abstractions on top of super low level (rust) code, because usually that's the less error prone way to do things - that doesn't prevent it from working at those levels though, and if anything it makes working on systems where you have to work on super low level code much easier to use.
When working with low level rust you don't "program everything in unsafe mode", you program a few really low level bits in unsafe rust, but quickly make abstractions that allow you to avoid making the vast majority of your code unsafe. This is opposed to languages like C/C++/zig/... where there isn't anything but "unsafe".
Rust doesn't really have a runtime by default (beyond libc for non-baremetal code).
There are a few tiny niches where rust isn't designed to compete with C, but they really are tiny niches. For example where chars aren't 8 bits.
I'm honestly just not following what you're saying about C++. Rust code tends not to depend on C++ code apart from maybe some system libraries that everyone depends on. The rust community tends to be overly-keen on rewriting those libraries in rust, not underly-keen. Obviously some people will make code that manages to depend on the language - but rust doesn't make it easy (like zig does...).
As for the relationship with llvm - rust doesn't actually have a hard dependency on it anymore (with support for a compiler backend called cranelift, written in rust). I also just don't see "the compiler depends on another language" as a huge impediment - as others have mentioned C compilers tend to as well...
Yes Rust helps bring in the next generation of Linux devs. It needs to to survive. But it's not because young people like Rust, it's because C developers like rust too.
Yes, Rust has unsafe and unsafe rust is trickier to get right, but you use it opt in and it's only a small amount if the code. Zig is still having unsoundness bugs in relatively simple code
Speaking as someone going the other way, just started working with C full time.
Yes. Oh Dog, Yes!
Data structures that rely on the memory layout semantics of the C compiler and casting void * is very cool, very 1990s, and does my head in!
And from what I see, Rust can sort of make it easier by making you think about the lifetime of the memory while coding and that's why I really wanted to learn Rust
I think this might touch on part of the problem. Rust can work on higher abstractions but rust doesn't have to. It is a matter of the code base. I've worked on some that felt very similar in abstraction to C code bases. One in particular was a high performance nginx module. Most of the rust code interfaced with nginx directly through zero cost FFI interfaces to the nginx structs.
I've also worked on extremely meta, high level code that was written by a former Haskell dev.
Like many languages that can serve dual roles, the styles tend to adapt to it. You see very different styles of Java when looking at spring web apps or embedded android systems. Every language you have to pick and chose parts of it that, build a coding standard and need to enforce it. C is exactly the same and also has several roles people pick it for. The style of code of embedded systems, high performance networking systems and codebases that prioritize highly portability tend to be different and use different idioms. I think any of of us who have worked in a C code base that falls in one of those camps can name banned idioms that are normal in the others.
>> It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd.
My career before Rust was very C heavy. I've work from OS-less embedded systems, to device drivers, to Android porting. I've been doing Rust professionally since 2016. This just isn't my impression. A lot of folks I've known from college and internships who's careers went a similar direction ended up in rust independently. There is value in having a language that let's us get the job done that we used to do with C but with more high quality static code analysis. The business case and use case is very clear.
While I haven't done any bare metal development in Rust, in the areas I have worked I've found compiler, community and crate support to be great. I don't know where you are getting this impression. I think if I only read hackernews articles about Rust I would get the impressions that is Haskell that can work on bare metal. I've noticed articles that tend to get upvoted here tend to be either about 1) how quick adoption is 2) higher levels of abstraction like HKT or GATs 3) WASM 4) writing web services. This article is a great example of 1. 2 is catnip for this crowd. The majority of rust devs never need to know what a GAT is but if you used HN as a metric you'd think it was a daily problem. 4 is something that is easy to write, makes good content, approachable but isn't necessarily Rust's best use case. I'm not saying you've only been reading this site about Rust, just saying that it is easy to get a skewed impression of the language unless you really dive in.
Yes
That is the point
Those abstractions are available to the programmer and compile down as tight and efficient as C
Not likely. At the moment you need to do extra work to get Rust working well. It's not exactly beginner friendly and doing work in the kernel, you'll need to dig into C anyway.
> Or is this a serious proposal about the future of operating systems and other low level infrastructure code?
Serious code already exists, so... Yes?
> Do you just program everything in unsafe mode? What about runtimes?
Why would you? You need that only when interfacing with something that can't hold the Rust compiler assumptions. See for example https://github.com/AsahiLinux/linux/blob/gpu/rebase-6.4/driv...
The few places that need direct access / unsafe are almost all single-line areas with an explanation.
> but it ain't the C culture if we are to accept that such a thing exists.
I don't get that whole part of what you wrote. What seems to be the culture / why does it matter in the technical sense? Some parts of the compiler are written in C++ - what's the specific issue here?
I attempted to port a low latency trading system from C to Rust last year, but ended up abandoning after 6 month of development.
The overall feeling that I got is that Rust made my life a dream for 80% of the codebase, the _non critical, non high performance part_.
There are tons of well written libraries, cargo is awesome, the performance is on par with what C++ would do, etc.
But the last 20% of the code base, the high performance part, was a nightmare to implement. The Rust static safety basically get in your way constantly for any kind of non obvious memory layout (self reference, etc). Dynamic safety (cells, Rcs, etc) just add too many overhead for the the critical path, and the "escape hatch" of "unsafe {}" ended up being 1000x more error prone that C.
I keep some resentment against the Rust community from this experience to be honest. I felt like instead of understanding the constraints & limitations of a fixed, microsecond time budget that I had, and trying to find solutions and be open about possible improvements, the overall trend was more in trying to defend Rust with whatever it takes.
IMO this is the biggest issue right now. `unsafe` needs some refinement, certainly.
> the overall trend was more in trying to defend Rust with whatever it takes.
This can sometimes be the case, unfortunately. But fwiw I think a lot of us in the community agree entirely that there's room to improve in the unsafe world.
Even at the surface, if your desired solution is so antithetical to the type of structure that Rust tries to push you towards, you can quite literally just write this component of your code as if it was C. Just using pointers instead of borrows and tag all your functions as unsafe and… pretend you’re writing C? At that point you’ve, I think, more or less disabled 95%+ of the bits that would prohibit you from writing code exactly as you would in C. You don’t get the benefits of Rust for that part, but at least you get them for the remaining 80%.
Or you can very literally just write the “hard part” in C and call it from Rust. You might have to make sure you can’t panic across language boundaries, but other than that the C interop is just about the best I’ve seen from any language.
I also don’t entirely understand how unsafe can be 1,000x more (or even 1.5x more) error prone than C. But I’d love to hear how. The only “trick” to unsafe is that you should aim for your unsafe blocks to be “unit-safe”. Meaning they might do something unsafe inside, but from the outside looking at it as a black box, the unit functionality of it should be safe. I don’t think the docs do a good enough job here of encouraging that style of design. You can violate this guideline, but doing so without sufficient care is quite likely to result in bugs. But of course if you did the same approach in C, you’d have a similar outcome.
The only real way I can reconcile your points is if the performance-critical bits that seriously impact the design of your program are scattered uniformly and don’t have anything resembling clean boundaries. I suppose that’s a real possibility but it does seem very foreign to me.
I sort of imagined that I could get the best of both world, and just "unsafe { <C style code> }" my way out for performance critical things.
But the thing is, the static safety boundaries of Rust allow the compiler to make much, much tighter assumptions than C & C++, especially around aliasing rules, un initialized memory, and moves.
When you relax these boundaries with "unsafe {}", you don't enter "C world", you enter the litteral gates of hell where any innocent temporary cast can throw you in a random load/store reordering bug.
There are still some Rust-specific details you would still need to handle—as you mentioned, uninitialized memory—but for that one specifically I haven’t found MaybeUninit to be particularly cumbersome.
A low latency trading system can't have Rc, or any C equivalent of it, in the critical path, because it does allocation. More generally, your story doesn't ring true to me, because the performance-critical parts of a low latency trading system have to be so simple that there isn't scope for any of the tricky bits you talk about, they have to be braindead simple loops over simple data structures.
One of my colleagues wrote a low latency trading system in Rust. He had to learn Rust to do it, already knew C, and today thinks this was the right decision.
Hu, so it doesn't ring true because... you feel like it's too complicated from what you imagined? That's a first.
> that there isn't scope for any of the tricky bits you talk about, they have to be braindead simple loops over simple data structures
That is far from the reality of low latency trading systems. Let's just look at the very first building blocks:
1) You will need some form of userspace packet polling, along with its associated multicast A/B line arbitration. That alone is not "simple braindead loops", and we're just talking about getting 1 UDP datagram here.
2) You will need some form of packet queueing before decoding, so that you do not drop packets under bursts. That most likely means some variant of lock free ring buffer in shared memory. Far from "a bunch of brain dead loops" as well.
3) You will need to decode packets and maintain an order book state. That mostly implies a purpose built b+tree. If you've implemented any of those, you will know it's also far from simple.
4) You will have to compute some indicators based on order book state, which most likely will lead you to some form of linear algebra.
5) You will need to disseminate such indicators to perform some simple close formula variant of portfolio optimization. More linear algebra.
6) etc.
I'm happy your colleague that he managed to do all that without struggling, that was not my experience, that's all. But to say that it's "just braindead loops over simple datastructures" is either a sign he didn't do a _real_ low latency system, or he just was somehow spared from the real underlying complexity.
The problem is, the things you needed to do to stay alive in this environment are antipatterns in Rust and you’ve learned the hard way Rust actively opposes being fed Rust antipatterns.
Self referential data structures are some of the hardest problems in CS to get right. There are patterns that work with Rust which replace pointers with eg generational indexes. You’ll find people do the same thing in C++ once they get burnt one time too many. I think you’d have a very different experience if you approached your system with an ECS framework, either off the shelf or roll your own.
Yes, the Rust language needs a complex optimizing compiler to really be viable. In this respect it is more limiting than ANSI/ISO C. For a lot of contexts, this isn't really a problem though? You aren't going to be able to build the Linux kernel without GCC or Clang anyway…
I've heard good things about Zig but I've never tried it so I can't compare unfortunately.
Rust gives you the option of opting out of that `#[no_std]`, which removes some basic tools, such as allocations, panics etc (I haven't used it much, so this may not be fully precise). As such with no_std you do move closer to what you'd consider C development.
That said rust will still have its restrictions, so some of the wild things you can do in C, isn't possible in rust, at least not without really violating the inner workings of it.
But with unsafe you can technically do most things on the level of C, without the overhead, that C++ would provide, via. its abstractions.
I'd expect the Linux Kernel to move in the direction of developing or using their own libraries and data structures, and that becoming its own bespoke part of their use of the rust language. Such that rust only really provides its safety guarantees and ergonomics. Compile times are also quite quick with no_std, as rust only really struggle with compile times, if you have lots of dependencies.
Circling back:
Rust is more like C++ with std library enabled (default) and C with `no_std`
Probably more good examples and possibly more ecosystem.
A first big decision is if you're going no-std or not, and if you end up in the no-std world the ecosystem shrinks substantially. If you're building a kernel that's probably not that much of a problem - the same shrinkage occurs for C/C++ - many such projects bootstrap with nearly zero ecosystem anyway.
The examples side is a bigger problem - I've recently been able to watch some of my more curmudgeonly C friends give it a good dive, and after an initial hump they're fairly happy with the core language. They still have regular issues with the ecosystem when they run into, in their words "web dev crap", which comes up even in the stdlib sometimes - a bugbear a while ago coming up down some error handling code paths. They attempted to send patches and hit nebulous arguments against the correctness target, which were largely born of misunderstandings of posix. This kind of thing can come up anywhere, if you take a dependency on some fancy IO abstraction that happens to be written in C, and you take it somewhere "novel" like a BSD, you might well run into the same. The point here is that _examples_ and _exercise_ of these tasks are the things that are going to shake more of this kind of thing out. At the same time though, it's important to reiterate that if you're on the nostd path, then largely you're on your own, which is equivalent to just gcc bare, and this kind of thing generally doesn't come up.
> Or is this a serious proposal about the future of operating systems and other low level infrastructure code?
This is a serious proposal. The outcome is really strong along key axes of correctness and safety. Those of us who've done it (e.g. Fuchsia, where I was) have been able to observe these benefits relative to history with the same teams using other languages (C, C++). We're professional engineers, these statements aren't coming from a place of craziness. The Android team have been writing about their journey: https://security.googleblog.com/search/label/rust
> Do you just program everything in unsafe mode?
Absolutely not. A good amount of bootstrapping effort has been going in across the ecosystem to make it ever easier to avoid unsafe. To take one slice of examples, there's crates that are designed to help you avoid copies while also avoiding dropping to unsafe - they provide tools for automatic structural analysis of the mapping boundary to make it easier to assert the relevant guarantees. Examples: https://docs.rs/zerocopy/latest/zerocopy/ (came out of Fuchsia), https://github.com/serde-rs/serde/releases/tag/v1.0.0 (serde is commonly used, but has more constraints here), https://rkyv.org/rkyv.html (not sure of prominence, but I hear people talk about it).
These kinds of tools get you a long way toward substantially safer code, without needing to think or audit nearly as much. We know that's important, we have plenty of data that demonstrates how important it is, and lately now, we have data that shows how effective it is too (see the aforementioned Android posts).
> What about runtimes?
They're out there, it depends what level of abstraction you're looking for, runtimes means different things to different people. For embedded there's typically a lot more focus on providing libraries rather than a whole runtime framework, so there are crates for a number ...
Oxide Computer has written an embedded bare-metal OS in Rust, to serve their hardware product: https://hubris.oxide.computer/
I agree. In general I think Zig makes lot more sense as a C replacement for the kernel.
However, Rust is stable, and Zig isn't. And I don't expect it will be for some years. So at this point Zig is basically unusable for kernel work.
That doesn't mean it can't be added in the future though, and perhaps even overtake Rust. See: https://news.ycombinator.com/item?id=26815950
1. Sticking with C for the rest of eternity for the linux kernel and accepting the baggage which comes with that
2. Migrating to rust and accepting the regressions which will inevitably appear as code is rewritten
Bugs can be fixed.
https://syzkaller.appspot.com/upstream/graph/bugs
So, yes, all bugs can be fixed. And yes, the world produces enough food to feed the entire population. But people are still going hungry.
I don't understand where all these memory bugs are coming from. Are programmers not testing their code? Are they not running sanitizers, fuzzers, and static analysis? Switching programming languages isn't going to fix broken development process.
Even C++ is at least somewhat better, with smart pointers and RAII.
As you suggested though, static analysis has lots of scope for solutions. The best static analyzer around is a little thing called rustc :)
It's true that Rust is currently the only stable language in this space, so it has some advantage there, but "Rust or forever C, there are no other options" is of course a load of nonsense.
Hope to contribute patches in rust to kernel in the near future!
Conversely, Rust never lets you do anything above a mid-level effortlessly. Traits barely get you halfway to typeclasses and you still have to mess with the borrow checker no matter what you’re doing. Add in lifetimes and even simple high-level code gets bogged down with extraneous low-level details.
Some of my python scripts which used to take 24 hours to process data now can finish in under an hour. It’s like magic. Granted theres a development cost so I don’t convert everything but it almost feels like cheating watching it go.
Another thing, it seems that the strictness of the language really cuts down on nasty runtime issues and silent bugs. Rust forces you to state explicitly what you want to do.
> Torvalds answered that, while he used to find problems in the LLVM Clang compiler, now he's more likely to find problems with GCC instead; he now builds with Clang.
> At the conclusion, Torvalds pointed out that there have been problems over the years with GCC changes breaking the kernel; the same will surely happen with Rust, but it will be the same thing in the end.
Based Linus Torvalds.