In Rust it's pretty limited. Here is the full list, from the Rustonomicron:
Dereferencing null or dangling pointers
Reading uninitialized memory
Breaking the pointer aliasing rules
Producing invalid primitive values:
dangling/null references
a bool that isn't 0 or 1
an undefined enum discriminant
a char outside the ranges [0x0, 0xD7FF] and [0xE000, 0x10FFFF]
a non-utf8 str
Unwinding into another language
Causing a data race
Note that all of these are inside of unsafe blocks. Besides unsafe blocks, Rust has no undefined behavior, and the compiler will prevent you from doing any of these things.
> Note that all of these are inside of unsafe blocks. Besides unsafe blocks, Rust has no undefined behavior, and the compiler will prevent you from doing any of these things.
Your own code need not use "unsafe" at all, but the program may still crash in your code if you called some function that internally does unsafe things to mess up your memory.
EDIT: I should say that the linked code does not crash, it only reads uninitialized memory. It seems to me like the same hole could be used to make things crash, but I don't have a ready-made example.
The important quote from the article you linked is: "But what happens when we throw some unsafe code at the issue?", so the claim that safe Rust doesn't have UB (that isn't considered a bug in the compiler) still stands. If you mix in unsafe code, bad things may happen, regardless of whether you wrote the unsafe yourself or rely on a library.
> the claim that safe Rust doesn't have UB (that isn't considered a bug in the compiler) still stands
Yes!
> If you mix in unsafe code, bad things may happen
Yes!
My point was only that those bad things may also happen after the unsafe block that is their root cause. This is in reply to parent's "all of these [UBs] are inside of unsafe blocks". They aren't. They are caused by something inside unsafe blocks, but they may also materialize after.
> unsafe does more than pollute a whole function: it pollutes a whole module. Generally, the only bullet-proof way to limit the scope of unsafe code is at the module boundary with privacy.
> the only bullet-proof way to limit the scope of unsafe code is at the module boundary with privacy
I understand how this is meant in the context of that page, and it is true that modules protect against users messing with modules' internal invariants.
But does that also work the other way around? In the example in https://gankro.github.io/blah/only-in-rust/#unbound-lifetime... it's not the caller messing with the callee's state, it's the callee messing up the caller's state. Do modules help at all here? That is, would the function
I would argue that this is not the caller making the mistake; it's this function. That is, since this function is safe, any safe code should be able to call it and not generate UB. It's not the caller's fault here, it's the incorrect implementation.
> It's not the caller's fault here, it's the incorrect implementation.
I agree. But then this shows that it's possible to write "safe" Rust code that only calls "safe" external code and still (to a first approximation) have the possibility of undefined behavior showing up at any point. In other words, Rust's famous static, compiler-enforced guarantees are not guarantees at all, more like firm promises.
Nothing wrong with that; it's easy to write memory-corrupting code in other safe-by-default languages like Haskell or OCaml as well. But it seems like Rust's marketing materials do try to suggest otherwise, and many people get wrong impressions (look at the first post in this thread, and the other comments on this article saying that Rust's "safe" code is 100% free from undefined behavior).
I mean, safe code is. Again, it's the unsafe that's at fault here.
Your point about other languages is exactly what I was going to say; unsafe is like an FFI layer. Nobody says that Ruby isn't memory safe because it can call into C code, and if someone messes up the C, well, it's at fault. "It's memory safe except for FFI" is a mouthful, and so people generally let the exceptions slide. Same with Rust.
Correct me if I'm wrong but I don't think such a strong universal statement can be true (even outside unsafe blocks) because LLVM has corner cases of "undefined behavior". (And since Rust is relies on LLVM, ...)
Maybe it's more accurate to say that Rust minimizes undefined behavior as a design goal, or it doesn't have intentional undefined behavior.
If safe Rust code does not invoke any of the UB corners of LLVM then Rust can claim to be free from UB. I don't know enough to guarantee or verify it, but it's my current understanding that this is the case.
>If safe Rust code does not invoke any of the UB corners of LLVM then Rust can claim to be free from UB.
Sure and I believe that adding a conditional qualification such as "if one does not invoke UB of LLVM" restates my point: one can't make a universal statement that "safe Rust has zero undefined behavior."
E.g., as of this writing, the following "safe Rust" UB issue (3+ years ago) last had comments 21 days ago and I believe it's still open:
>a conditional qualification such as "if one does not invoke UB of LLVM"
A conditional qualification which is intended to be unconditionally true of safe Rust code, outside bugs in the compiler. The universal statement is totally possible, because your conditional is equivalent to saying "if you write valid code".
If somebody asks you what "cat" does, do you say "It copies its input to its output, unless there's a bug in cat or the C compiler that compiled it or cosmic rays hit the program on disk"?
Yes I get what you're saying but I'll try to emphasize again that I'm not trying to play semantic games to irritate everyone. (Yes, we can play word games such as "a tank is an armored military vehicle -- unless it is just a cardboard facade to fool Germans that the Allies are invading a different a part of France's coastline or acting as a movie prop for special effects work.") Every "thing" can be defined with endless cumbersome qualifiers that nobody actually says in real life.
That said, I felt the context in this thread warranted a different threshold to qualify Rust's UB because one example of John Regehr 200 UB bullet points is:
- Demotion of one real floating type to another produces a value outside the range that can be represented (6.3.1.5).
The Rust UB github issue is not exactly the same cast but similar in spirit. Therefore, justinpombrio's comment that "Besides unsafe blocks, Rust has no undefined behavior," doesn't look accurate to me in the context of this UB thread rather than just casual speech about Rust. I can't read the mind of the poster asking the question (chrisdew) to know exactly what his scope of "UB" included but I think the reality of unintentional UB in Rust is relevant in this particular conversation.
I agree. This comes up a lot when discussion C/C++ - is it the compiler's fault, the developers, etc? The reality is it's irrelevant. Rust-the-language is safe but no one uses rust-the-language they use rustc. The end result is that it is possible to have memory unsafe rust code without unsafe blocks.
Rust developers should be aware of this - they're almost always incredibly trivial patterns to avoid, but only if you know about them.
LuaJIT is another modern low-level language. It has some remarkable undefined behavior like the evaluation order for function arguments. Scares me a bit.
The same is true in C/C++: Function argument evaluation order is unspecified. BUT: that doesn't mean this is undefined behavior, it is just unspecified.
This would not be what is called undefined behavior per the C standard, but unspecified or implementation-defined behavior. The bad thing about "undefined behavior" is that the implementation is basically allowed to do whatever it wants, while unspecified and implementation-defined behavior still has sane semantics. Unspecified behavior allows the language implementation to choose any one from several possible implementations, implementation-defined behavior requires the language implementation to define the semantics itself.
Are you sure that undefined, instead of merely unspecified?
Ie function arguments are still evaluated in some (possibly changing) order. C++ undefined behavior allows to eg format your hard disk or launch the nukes.
A lot of undefined behaviour on this list is quite specific to the C programming language, so it would be like comparing apples to oranges.
Though, one major impression I think you will get after going through this list is that there is no good reason for making those things an undefined behaviour in the first place.
For example, "The result of the preprocessing operator ## is not a valid preprocessing token", really?
Regarding those things that actually matter for programmer, following would be well-behaved in Rust (including unsafe blocks): conversion between types, integer arithmetic, pointer arithmetic (there are generally two variants of operations, one that essentially treats pointers as unsigned integers, and another one that behaves like in C with more opportunities for optimization).
On the other side of the coin, in Rust mutable references cannot be aliased.
D runs in @system by default but there are @safe which enforces safety and @trusted (like unsafe in Rust). Those are all attributes you can write libraries in like @nogc etc.
Why don't compilers have an option to reject UB with an compiler error? Surely the compiler must know when it comes across a piece of source code whose semantic is undefined.
Some UB depends on the input (out of bounds access is a popular example). Some UB is created during optimization passes, for example by inlining. It's hard to produce useful error messages.
But it is easy to give it a defined behavior on a particular architecture. For example, on x86-64, it can be defined to overflow just like unsigned does. The compiler can emit errors about the code if it is compiled to a different architecture where the same defined behavior is too expensive to implement.
It can be, but it isn't. All contemporary ISAs use two's complement arithmetic for signed integers, including x86-64, and so I think signed overflow could be made well defined on all archs without particularly disadvantaging one arch over the other.
The reason that that doesn't happen, here as elsewhere, is that there are a number of specific compiler optimizations that rely on it being undefined. See GCC's -fwrapv flag.
UB is the result of C compiler vendors not wanting to give up on their precious optimizations or hardware configurations back when ANSI C was initially being done.
They could have done like in other languages and have ANSI C without UB, but then they would need to give up on some hardware features or lose in one or other benchmark.
It is very easy to be efficiently determined at compile time, other systems programming languages do it, by not having UB on their language standard to start with.
If people prefer "performance trumps correctness" and "winning compiler benchmarks is what counts" then they should happily keep getting exploited.
However I don't see it ever happening in the UNIX world, only on OSes whose architecture is not bound to the traditional UNIX/C culture.
Which is why see as positive that Apple, Google and Microsoft are pushing C and C++ down into the very lowest layers of their OS stacks, mostly visible on Google's case.
Even straight pure Assembly programming has less UB surprises.
> It is very easy to be efficiently determined at compile time
for many cases it is in fact equivalent to the halting problem. What you can do is check at runtime, which has performance implications (from trivial to expensive depending on the check).
Not if the language standard doesn't allow for UB to begin with, then there isn't any halting problem to worry about.
This is what the alternatives in the memory safe systems programming language since Algol days do.
Surely they still allow for developers to disable some of those checks explicitly, but then it is the programmers fault to choose "performance trumps correctness" instead of "correctness trumps performance".
It doesn't matter if the language defines the behaviour of, say, offsetting an array beyond its declared bounds. That doesn't change the general undecidability of statically proving the offset will always be within the bounds at compile-time - in other words, you need runtime checks in some cases (it's just a matter of whether the compiler or the programmer supplies them).
Nah, with some help from the programmer, you can produce those proofs. It's not too hard.
The halting problem is impossible to solve for arbitrarily evil programs, but that doesn't mean that it's impossible to write programs that are easy to analyse.
I think that statically guaranteeing something like in-bound indexing or absence of division by zero you need depended types, which is not exactly easy nor mainstream. And even with dependent types, many non-evil, otherwise correct programs will be rejected.
Aren't dependent types able to express any statement formalizable in predicate logic? This should include most things you might want to say about the correctness of non-evil programs. I think the real problem with them is that type-checking itself becomes undecidable, so you get the additional problem of proving that your program is indeed correctly typed.
Yes, for the general problem you want dependent types (or similar).
There are middle grounds, of course: you can let the compiler figure out on its own what it can, and let it insert runtime checks for the other cases.
And add some way for the programmer to declare that eg "warn me if this can't be optimized" or "trust me, this is safe" or even "here's a proof for you".
But in any case, dependent types are possible---undecidability is not a hard barrier here.
"Very easy" - well, if you think garbage collection is easy...
And yes, if you want to allow manual allocation of objects and setting pointers to arbitrary objects, then your two choices are:
1) garbage collection
2) undefined behavior (use-after-free, double-free, dangling pointers, etc.)
Well, I don't know about half of these programming languages. However:
- ADA requires garbage collection
- Modula-2 and Object pascal have manual memory management, i.e.: "new" and "delete" - together with double-free, use-after-free, and dangling pointers...
I agree that C is by far worse, but please recognize that at least some of the undefined behavior problems are really hard to solve.
EDIT: Oh, and we haven't started talking about multi threading yet, have we?
Of course there are still use cases where Algol derived languages are still unsafe, however those use cases are a tiny portion of what happens in C land.
> Of course there are still use cases where Algol derived languages are still unsafe, however those use cases are a tiny portion of what happens in C land.
On that I agree - there are many problems we wouldn't have if we were using, say, pascal or modula.
> Also you don't seem to know much about Ada given your GC remark, but here is some learning.
I'm sorry, I don't have time to watch that. However, I know that ADA has either GC or manual memory management (with the deallocation method explicitly marked as unsafe). Thus, as far as i know, constructing "use-after-free" in Ada is possible - which is undefined behavior (and that was my point).
But Rust is where it's taken to the extreme (use on memory resource), and used to prevent a class of UBs (use-after-free, double-free, dangling pointer...)
Not quite. Destructors aren't guaranteed to run, so you can't actually rely on them for memory safety guarantees.
For example, leaking data in Rust is a perfectly safe thing to do. But while destructors help to prevent it, they cannot guarantee that nothing ever leaks. (And some of this depends on what you mean by "leaking" and so on.)
use-after-free, double-free and dangling pointers are all handled by Rust's ownership and borrowing systems.
They are called undefined behaviors because the compiler developers are free to do whatever they want when they come across a code that instigates an undefined behavior.
Mostly new compilers now a days do handle such behavior with a compiler error or warning. So if you try to do an out of bound shift a new compiler might throw a warning, but according to the C standard it does not have to. In fact instead of throwing a warning it can wipe your complete hardware and that too is allowed according to the C standard...
In contrast there is implementation defined behavior, and unspecified behavior. Where the compiler developers have to document and take care of certain things.
I wrote a detailed article about the subtle difference you can check that out....
Yes they don't, I assumed they would, but they dont. I just wanted to emphasize the fact that they don't have to. Perhaps an unbound shift was a wrong example.. thanks for catching that...
Because compiler don't generally look for UB then gleefully cackle as they fuck up your code.
Rather they are written under the assumption that there are no UB in the program, and the chips fall where they may as these assumptions are used and propagated.
The argument I've seen is "macros often expand into unintuitive code, especially when multiple macros are involved, so these kinds of warnings would be confusing if there were such macros."
It might be possible to do the same thing if you didn't check any code generated from macros.
You just destroyed all C code in the world. At that level of paranoia, you've basically defined a brand new language, and if you're going to do that, why make it C at all?
Smart people with a ton of experience in C have already examined the possibility of defining a C dialect with much less undefined behavior in it. In fact, I believe it's been seriously tried more than once by independent groups. And the result has been the same each time; initial excitement and optimism gives way to total failure. It's simply way too deeply ingrained in the definition of C, backed by all the code in the world written in C. If you want less undefined behavior, your only choice is to leave C.
You're changing your plan here. Maybe you could at least concede that the issue of UB is difficult and is not going to be solved by saying "why don't they just" on HN?
* It's not UB in Rust, it's guaranteed to be two's compliment overflow.
* It's still a "program error" and debug builds are required to panic.
* Non-debug builds are allowed to either panic or do the overflow, for performance reasons, as you mention. If it was feasible to always panic we'd do that; the current wording allows us to change it to do so in the future if it becomes feasible.
* If you want to do a checked add, you can, it's just not the default.
I wish there was a GCC switch that made the optimizer go "This is clearly undefined behaviour. I'll stop this compile right here", as opposed to "This is clearly undefined behaviour. Sweet jackpot, it means I can do whatever the heck I want. Drop those NULL checks below, I've got the golden UDB ticket now"
I think we deserve a compiler warning whenever a source line of code (not a line generated by the preprocessor) gets removed due to UB exploitation. All such cases are worth the programmer's attention, because they are either useless lines or bugs.
It's more complicated than this, unfortunately. Imagine an int variable x and two expressions f(x) and g(x) of x. Now imagine an if statement with condition f(x) < g(x). Precisely because incrementing the largest int is undefined behavior, the compiler may be able to simplify the if condition because, for example, it can assume x+1 > x is always true. So it's not necessary for the line to be useless or wrong for undefined behavior rules to trigger compiler optimizations.
You must also account for the case where the source line removed was only useless because of the specific context it had been inlined to by an earlier stage of the optimiser.
I wonder why he mentions this TIS interpreter but not the development of complete semantics for C and specifically the k framework (https://github.com/kframework/c-semantics).
AFAIR from the last presentation on the K framework I attended (a year ago), the publicly available version of their C interpreter is incomplete: It only runs programs that don't need the standard library. That's pretty great for many things! But it's not a turnkey solution for analyzing real applications. For that they had a proprietary commercial version. (Again, this is from memory, and it may have changed.)
Another reason to mention TIS-interpreter specially is that Pascal Cuoq (the co-author of this blog post) is one of its developers, and his livelihood depends on it being used.
Original developer of the C semantics in K here! I just wanted to add a pedantic clarification (since pedantry is what semantics is all about).
The semantics of the language itself is more or less complete. On top of that, a sizable portion of the standard library is given semantics directly (e.g, malloc, setjmp, stdargs, I/O including input/output to FILEs, much of printf, etc. are all formalized in the semantics). There's even basic C11 thread support. All of this runs and can be reasoned about formally. I tried to give direct semantics to the things that couldn't be (easily) written in C, because much of the standard library CAN be written in C and so it isn't so critical that it's formalized.
To round out the library you can provide provide C implementations of any missing library functions at compile time (we provide some by default), and it will KCC those just like any other code. The resulting system is pretty good at catching undefined behavior, but it is super slow.
I'm not involved with the commercial version, but they offer a trade off: don't do any semantic checking inside standard library calls in exchange for faster execution times. My understanding is that instead of chugging through the semantics of printf for every bottle of beer on the wall, they just call out to a native of printf. This makes things a lot faster, but it's possible you won't catch as many problems. In general, the commercial version is trying to make using the semantics practical instead of "academically slow", which was all I managed :)
> Be knowledgeable about what’s actually in the C and C++ standards since these are what compiler writers are going by. Avoid repeating tired maxims like “C is a portable assembly language” and “trust the programmer.”
> Unfortunately, C and C++ are mostly taught the old way, as if programming in them isn’t like walking in a minefield. Nor have the books about C and C++ caught up with the current reality. These things must change.
I think it's the compilers that have perverted the language to such an extent that it's become ridiculously difficult to understand, and so it's the compilers that must change back to being less obtuse and adversarial.
The C standard even suggests, when defining undefined behaviour, that one of the possible options is "behaving during translation or program execution in a documented manner characteristic of the environment". If signed int arithmetic on the hardware wraps around upon overflow, then take that into account when optimising and don't assume it can't overflow. This is how C programmers want the language to work; compiler writers and standards be damned.
I've written plenty of Asm and also looked at as much if not more compiler output; and while the latter sometimes pleasingly surprises me, it's usually pretty evident that it was not generated by an intelligent entity. (And countless times, I've obtained size and/or speed savings by replacing the latter with what I would otherwise write.) My general impression is that there aren't enough highly-skilled Asm programmers working on compiler development --- there are many reasons I can think of for that, but one of the things I've always wished to exist is a simple, straightforward C compiler with "understandable" output and optimisation that creates results closer to what a human Asm programmer might write; with the associated optimisation advantages and predictability. This means not assuming anything just because the standard says so, but taking a more holistic (for lack of a better word) approach to compilation and optimisation.
> If signed int arithmetic on the hardware wraps around upon overflow, then take that into account when optimising and don't assume it can't overflow. This is how C programmers want the language to work;
Don't speak for all of us.
Some of us may take the language for what it is, instead of assuming it to be a "high level portable assembler" which it is not, but which a particular implementation of it may be, thanks to the standard leaving much up to the implementation.
It is this same leeway that (thankfully) allows smart optimizing compilers that can generate performant code for programs that are not omgwtf-optimized at the source level.
And it is this same leeway that allows very simplistic, naive implementations that do not go out of their way to perform crazy analysis to try and warn you about some corner case of undesirable behavior your code might hit.
You're right that it is the compilers can and need to change to accommodate different types of users. But I don't think every compiler should just go back to being a 1991 compiler; not every compiler needs to accommodate every user.
They CAN accommodate all users, just have compile time options you can enter in your make file/on the command line/etc to say what level of bounds checking/etc you want compiled into your code. Hell that way even within a given project you can choose "this library I'm nervous about so I'll eat the hit on bounds checking".
The performance culture isn't just C. Far from it.
Rust was susceptible to Stack Clash because they ripped out their existing stack probing to gain a few percent increase in performance.
The article for this thread literally says, "For many use cases ASan is the better choice because it has much less overhead." IME Valgrind does a _much_ better job of detecting memory issues, partly because ASan only detects issues directly triggered in the code that you compile, whereas Valgrind can detect memory issues directly triggered by external code, but indirectly caused by your code.
Rust used guard pages, but was still vulnerable to attacks based on jumping over the guard page, which is the same issue that Stack Clash exploited in C programs. The ideal fix for that is stack probing, which Rust is indeed moving towards. However, in the past, Rust had a different fix: it would add a bit to all function prologues that explicitly checked for running out of stack space - based on LLVM split stack support, and a remnant of when (even further in the past) Rust actually used split stacks. The explicit check also protected against jumping over the guard page, but was much more expensive than stack probing, and also caused issues with embedded use cases. The plan was to replace it with stack probing, but things dragged on and it was removed before the replacement was ready.
> The performance culture isn't just C. Far from it
Sure, but it is where it is more visible, specially micro-optimizing code as it is being written, without validating if it really matters to the application's use case with a profiler.
The school of systems programming languages from Algol side, was that correctness was much more relevant than pure performance, Algol dialects for systems programming already had Rust like unsafe blocks in the mid-60's.
In Burroughs B5500 the administrator could enable which applications with unsafe modules were allowed at all to be executed, 10 years before C was invented.
I've never met a C programmer or C compiler author who felt correctness should take a backseat to performance. And as I demonstrated, the Rust designers are hardly immune to poor decision making on this score.
Someone else corrected me regarding the stack probes--Rust didn't have probes but actually checked the size of the stack. It's also worth pointing out that often times decisions are premised on maintenance burdens. Performance was one excuse for removing the stack checking in Rust; the other was the maintenance burden of maintaining a feature outside of LLVM.
That goes directly to the heart of the undefinedness issue in C. Doing the "obviously correct" thing wrt to choosing between possible undefined- or implementation-defined behaviors isn't as easy as we think. Good compilers are built on abstractions and code reuse, like any other piece of well-written software. Doing the correct thing in some obvious cases can result in having to make non-obvious tradeoffs elsewhere, possibly having to choose between adding (or maintaining) complexity, or removing a powerful feature, such as deterministic stack overflow prevention.
The history of C (indeed, Unix and the whole "worse is better" school of design) is about favoring less complexity in the tooling while shifting more of the burden onto the user. That's a judgment not about runtime performance, but a very unintuitive judgment about the performance and efficiency of programmer time; that you quickly reach a point of diminishing returns, much sooner than most people believe, putting complexity into, and relying on intrinsic features of, the tooling; that it's often better to make it easier to implement and integrate specialized tools, and easier to reuse existing tools as part of a more focused solution. From that perspective the design of C makes a ton more sense, even if reasonable people can still disagree about the results.
The notion that C--the language design, the ecosystem--is principally about performance is something mostly held by inexperienced programmers. It's a strawman of critics. There were plenty of other languages from the 1970s that were capable of generating faster code than C. Certainly performance was and continues to be of significant concern, but C is hardly unique in that sense. If the debate about how to balance performance with other issues stands-out, I'd argue that's primarily because of C's ubiquitous position and how it and the computing industry (including hardware industry) has co-evolved.
When something seems so "obvious" (security vs performance), yet everybody continues to seemingly repeat the same dumb mistakes, often times what we're missing is that our premises are flawed. Maybe C didn't become ubiquitous because of it's performance, but more so because it was easy to port and easy to build upon. After several years Rust still has a single implementation. Even Go has two production-quality implementations. That's not a criticism of Rust, just an observation that the "security vs performance" axis is hardly the only relevant dimension for judgment and explanation.
That's totally fair. Flip side arguing "we shouldn't do this because people who only care about immediate performance don't want them" feels like it is ignoring some percent of people who may not feel that way, or even the possibility of using it as a way to bring more people to systems level programming but are not up to making the leap Rust currently requires (which will hopefully become less of an issue as they work on this year's roadmap, but I would certainly argue still exists right now and we won't know how successful their efforts REALLY are until they are all in place).
When IBM researched RISC in the mid-70s, on the IBM 801 minicomputer, they decided to use a subset of PL/I as their mostly safe system programming language.
They implemented a botstraped optimizing compiler for it, making use of an intermediate language with multiple optimization passes.
The original mandate of X3J11 was to “codify common existing practice”. The published Rationale goes on to describe a number of relevant criteria, including
• Existing code is important, existing implementations are not.
• Avoid “quiet changes.”
• Trust the programmer.
• Keep the language small and simple.
• Make it fast, even if it is not guaranteed to be portable.
In pre-ANSI C, and C89 C interpreted in line with the Rationale, if you wrote ‘+’, the expectation was that you'd get the target machine's ‘add’ instruction, no more and no less. It might overflow, it might not; it might crash or hang your machine on overflow or trap values — but that is all your problem, not the language's or compiler's. The ANSI invention of ‘undefined behaviour’ can be considered a good-faith attempt to codify things like this, but botching it was one of the two great failures of C89.
I worked on a commercial compiler at the time, and I'm confident no one thought of C89 as a license to screw the customer.
They revised that criteria list for C11, given the current state of C induced security exploits.
"12. Trust the programmer, as a goal, is outdated in respect to the security and safety programming communities. While it should not be totally disregarded as a facet of the spirit of C, the C11 version of the C Standard should take into account that programmers need the ability to check their work."
> I think it's the compilers that have perverted the language to such an extent that it's become ridiculously difficult to understand, and so it's the compilers that must change back to being less obtuse and adversarial.
I disagree. Undefined behavior is very important for optimization. See John Regehr's blog post on the topic [1], or any of DannyBee's numerous posts here explaining why GCC and Clang can and must exploit UB, etc. etc.
> Summary: This UB is probably not a problem in practice (even if it is moderately displeasing to some of us).
Since LLVM is mostly based around C/C++ semantics this actually turned out to be a problem for rust, because rust can encode diverging functions in its type system and assume certain code sections to be unreachable. If the compiler then optimizes out code and the unreachable (thus not-compiled) section is reached things explode.
I think that's a great example of how optimisers have gone off the deep end: no reasonable human programmer would think "I can't figure out if this loop terminates, so let's just leave it out", but that appears to be the default behaviour in this case --- when what should happen when the optimiser "gives up" is that it should just translate the code verbatim.
But the optimizer doesn't give up in this case. It successfully figures out that the loop has no side effect except heating up your processor and eliminates it.
119 comments
[ 2.8 ms ] story [ 196 ms ] threadhttps://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html
It's true that unsafe is needed to get these problems, but they can also occur outside of unsafe blocks. See an example here: https://gankro.github.io/blah/only-in-rust/#unbound-lifetime...
Your own code need not use "unsafe" at all, but the program may still crash in your code if you called some function that internally does unsafe things to mess up your memory.
EDIT: I should say that the linked code does not crash, it only reads uninitialized memory. It seems to me like the same hole could be used to make things crash, but I don't have a ready-made example.
Yes!
> If you mix in unsafe code, bad things may happen
Yes!
My point was only that those bad things may also happen after the unsafe block that is their root cause. This is in reply to parent's "all of these [UBs] are inside of unsafe blocks". They aren't. They are caused by something inside unsafe blocks, but they may also materialize after.
> unsafe does more than pollute a whole function: it pollutes a whole module. Generally, the only bullet-proof way to limit the scope of unsafe code is at the module boundary with privacy.
> the only bullet-proof way to limit the scope of unsafe code is at the module boundary with privacy
I understand how this is meant in the context of that page, and it is true that modules protect against users messing with modules' internal invariants.
But does that also work the other way around? In the example in https://gankro.github.io/blah/only-in-rust/#unbound-lifetime... it's not the caller messing with the callee's state, it's the callee messing up the caller's state. Do modules help at all here? That is, would the function
become less dangerous, or maybe impossible to call, when put into a module?I agree. But then this shows that it's possible to write "safe" Rust code that only calls "safe" external code and still (to a first approximation) have the possibility of undefined behavior showing up at any point. In other words, Rust's famous static, compiler-enforced guarantees are not guarantees at all, more like firm promises.
Nothing wrong with that; it's easy to write memory-corrupting code in other safe-by-default languages like Haskell or OCaml as well. But it seems like Rust's marketing materials do try to suggest otherwise, and many people get wrong impressions (look at the first post in this thread, and the other comments on this article saying that Rust's "safe" code is 100% free from undefined behavior).
Your point about other languages is exactly what I was going to say; unsafe is like an FFI layer. Nobody says that Ruby isn't memory safe because it can call into C code, and if someone messes up the C, well, it's at fault. "It's memory safe except for FFI" is a mouthful, and so people generally let the exceptions slide. Same with Rust.
Correct me if I'm wrong but I don't think such a strong universal statement can be true (even outside unsafe blocks) because LLVM has corner cases of "undefined behavior". (And since Rust is relies on LLVM, ...)
Maybe it's more accurate to say that Rust minimizes undefined behavior as a design goal, or it doesn't have intentional undefined behavior.
Sure and I believe that adding a conditional qualification such as "if one does not invoke UB of LLVM" restates my point: one can't make a universal statement that "safe Rust has zero undefined behavior."
E.g., as of this writing, the following "safe Rust" UB issue (3+ years ago) last had comments 21 days ago and I believe it's still open:
https://github.com/rust-lang/rust/issues/10184
A conditional qualification which is intended to be unconditionally true of safe Rust code, outside bugs in the compiler. The universal statement is totally possible, because your conditional is equivalent to saying "if you write valid code".
I emphasized "intended" because it seems like we're talking past each other.
You: re-emphasizing Rust's specified design goal.
Me: emphasizing the current state of Rust compiler as reality which makes the statement "safe Rust has no undefined behavior" as not true.
(In other words, I emphasize the unintentional UB whereas you do not.)
>, because your conditional is equivalent to saying "if you write valid code".
If you look at the github issue, "1.04E+17 as u8" is valid safe Rust code which invokes UB.
Yes I get what you're saying but I'll try to emphasize again that I'm not trying to play semantic games to irritate everyone. (Yes, we can play word games such as "a tank is an armored military vehicle -- unless it is just a cardboard facade to fool Germans that the Allies are invading a different a part of France's coastline or acting as a movie prop for special effects work.") Every "thing" can be defined with endless cumbersome qualifiers that nobody actually says in real life.
That said, I felt the context in this thread warranted a different threshold to qualify Rust's UB because one example of John Regehr 200 UB bullet points is:
The Rust UB github issue is not exactly the same cast but similar in spirit. Therefore, justinpombrio's comment that "Besides unsafe blocks, Rust has no undefined behavior," doesn't look accurate to me in the context of this UB thread rather than just casual speech about Rust. I can't read the mind of the poster asking the question (chrisdew) to know exactly what his scope of "UB" included but I think the reality of unintentional UB in Rust is relevant in this particular conversation.Rust developers should be aware of this - they're almost always incredibly trivial patterns to avoid, but only if you know about them.
And I am only mentioning the most known ones during the 10 years before C was born until early 80's.
Sure some of them also had safety issues like use-after-free, but not in the same amount as C.
https://github.com/LuaJIT/LuaJIT/issues/238
unspecified != undefined behavior
See this Stack overflow answer for an explanation: https://stackoverflow.com/questions/2397984/undefined-unspec...
Ie function arguments are still evaluated in some (possibly changing) order. C++ undefined behavior allows to eg format your hard disk or launch the nukes.
Regarding those things that actually matter for programmer, following would be well-behaved in Rust (including unsafe blocks): conversion between types, integer arithmetic, pointer arithmetic (there are generally two variants of operations, one that essentially treats pointers as unsigned integers, and another one that behaves like in C with more opportunities for optimization). On the other side of the coin, in Rust mutable references cannot be aliased.
This article is about attacking the last category.
Unsigned overflow is far less of a problem than signed overflow because it is defined.
It is very cheap to make signed overflow defined. It is more expensive to detect it and fail at runtime.
The reason that that doesn't happen, here as elsewhere, is that there are a number of specific compiler optimizations that rely on it being undefined. See GCC's -fwrapv flag.
Absolutely not. Many instances of UB depend on runtime values. (E.g. overflows, out-of-bounds shifts, etc. etc.)
UB is the result of C compiler vendors not wanting to give up on their precious optimizations or hardware configurations back when ANSI C was initially being done.
They could have done like in other languages and have ANSI C without UB, but then they would need to give up on some hardware features or lose in one or other benchmark.
If people prefer "performance trumps correctness" and "winning compiler benchmarks is what counts" then they should happily keep getting exploited.
However I don't see it ever happening in the UNIX world, only on OSes whose architecture is not bound to the traditional UNIX/C culture.
Which is why see as positive that Apple, Google and Microsoft are pushing C and C++ down into the very lowest layers of their OS stacks, mostly visible on Google's case.
Even straight pure Assembly programming has less UB surprises.
for many cases it is in fact equivalent to the halting problem. What you can do is check at runtime, which has performance implications (from trivial to expensive depending on the check).
This is what the alternatives in the memory safe systems programming language since Algol days do.
Surely they still allow for developers to disable some of those checks explicitly, but then it is the programmers fault to choose "performance trumps correctness" instead of "correctness trumps performance".
The halting problem is impossible to solve for arbitrarily evil programs, but that doesn't mean that it's impossible to write programs that are easy to analyse.
There are middle grounds, of course: you can let the compiler figure out on its own what it can, and let it insert runtime checks for the other cases.
And add some way for the programmer to declare that eg "warn me if this can't be optimized" or "trust me, this is safe" or even "here's a proof for you".
But in any case, dependent types are possible---undecidability is not a hard barrier here.
Possible? Yes. Too time-consuming and fiddly for the economic realities of 99.9% of all programmers? Yes, that too.
At least that how bank regulation works out in practice.
And yes, if you want to allow manual allocation of objects and setting pointers to arbitrary objects, then your two choices are: 1) garbage collection 2) undefined behavior (use-after-free, double-free, dangling pointers, etc.)
Garbage collection isn't the only way of writing mostly safe systems programs.
ESPOL, NEWP are 10 year older than C.
PL/8 was the systems programming language used by IBM for their first RISC mainframe, followed by PL/S.
Mesa used at Xerox PARC.
Modula-2 used at ETHZ and many 16 bit systems.
Object Pascal used to write the first versions of Mac OS and Lisa.
Ada and SPARK used by high integrity systems where human lives can be at risk.
All of them with not even a quarter of C's UB collection.
- ADA requires garbage collection
- Modula-2 and Object pascal have manual memory management, i.e.: "new" and "delete" - together with double-free, use-after-free, and dangling pointers...
I agree that C is by far worse, but please recognize that at least some of the undefined behavior problems are really hard to solve.
EDIT: Oh, and we haven't started talking about multi threading yet, have we?
"writing mostly safe systems programs."
Do you understand what mostly means?
Ada and Modula-2 have multi-threading as part of their ISO/ANSI language standard.
Also you don't seem to know much about Ada given your GC remark, but here is some learning.
https://archive.fosdem.org/2016/schedule/event/ada_memory/
Of course there are still use cases where Algol derived languages are still unsafe, however those use cases are a tiny portion of what happens in C land.
That's uncalled for, man...
> Of course there are still use cases where Algol derived languages are still unsafe, however those use cases are a tiny portion of what happens in C land.
On that I agree - there are many problems we wouldn't have if we were using, say, pascal or modula.
> Also you don't seem to know much about Ada given your GC remark, but here is some learning.
I'm sorry, I don't have time to watch that. However, I know that ADA has either GC or manual memory management (with the deallocation method explicitly marked as unsafe). Thus, as far as i know, constructing "use-after-free" in Ada is possible - which is undefined behavior (and that was my point).
For example, leaking data in Rust is a perfectly safe thing to do. But while destructors help to prevent it, they cannot guarantee that nothing ever leaks. (And some of this depends on what you mean by "leaking" and so on.)
use-after-free, double-free and dangling pointers are all handled by Rust's ownership and borrowing systems.
Mostly new compilers now a days do handle such behavior with a compiler error or warning. So if you try to do an out of bound shift a new compiler might throw a warning, but according to the C standard it does not have to. In fact instead of throwing a warning it can wipe your complete hardware and that too is allowed according to the C standard...
In contrast there is implementation defined behavior, and unspecified behavior. Where the compiler developers have to document and take care of certain things.
I wrote a detailed article about the subtle difference you can check that out....
https://medium.com/@faizan10114/understanding-c-terminologie...
Right, for doing so needs to determine that this is happening, which neither GCC and Clang can do for fairly trivial cases like
Rather they are written under the assumption that there are no UB in the program, and the chips fall where they may as these assumptions are used and propagated.
It might be possible to do the same thing if you didn't check any code generated from macros.
int res(int a, int b) { return a + b; }
since it can be UB, for instance if a + b > INT_MAX.
Smart people with a ton of experience in C have already examined the possibility of defining a C dialect with much less undefined behavior in it. In fact, I believe it's been seriously tried more than once by independent groups. And the result has been the same each time; initial excitement and optimism gives way to total failure. It's simply way too deeply ingrained in the definition of C, backed by all the code in the world written in C. If you want less undefined behavior, your only choice is to leave C.
As for choice, I'll take "here's what happens" vs "your code is in the hands of fate now" any day. Glad we do have choice.
* It's not UB in Rust, it's guaranteed to be two's compliment overflow.
* It's still a "program error" and debug builds are required to panic.
* Non-debug builds are allowed to either panic or do the overflow, for performance reasons, as you mention. If it was feasible to always panic we'd do that; the current wording allows us to change it to do so in the future if it becomes feasible.
* If you want to do a checked add, you can, it's just not the default.
That's different from "This is UB, and now I do ...".
In C `x + 1` is an undefined behavior if `x` is signed and overflows. Optimizer takes advantage of that for loop optimizations.
Afaik, kcc is also really good at finding undefined behavior. Regehr even wrote about it in the past (https://blog.regehr.org/archives/523).
Another reason to mention TIS-interpreter specially is that Pascal Cuoq (the co-author of this blog post) is one of its developers, and his livelihood depends on it being used.
The semantics of the language itself is more or less complete. On top of that, a sizable portion of the standard library is given semantics directly (e.g, malloc, setjmp, stdargs, I/O including input/output to FILEs, much of printf, etc. are all formalized in the semantics). There's even basic C11 thread support. All of this runs and can be reasoned about formally. I tried to give direct semantics to the things that couldn't be (easily) written in C, because much of the standard library CAN be written in C and so it isn't so critical that it's formalized.
To round out the library you can provide provide C implementations of any missing library functions at compile time (we provide some by default), and it will KCC those just like any other code. The resulting system is pretty good at catching undefined behavior, but it is super slow.
I'm not involved with the commercial version, but they offer a trade off: don't do any semantic checking inside standard library calls in exchange for faster execution times. My understanding is that instead of chugging through the semantics of printf for every bottle of beer on the wall, they just call out to a native of printf. This makes things a lot faster, but it's possible you won't catch as many problems. In general, the commercial version is trying to make using the semantics practical instead of "academically slow", which was all I managed :)
Any questions welcome.
> Be knowledgeable about what’s actually in the C and C++ standards since these are what compiler writers are going by. Avoid repeating tired maxims like “C is a portable assembly language” and “trust the programmer.”
> Unfortunately, C and C++ are mostly taught the old way, as if programming in them isn’t like walking in a minefield. Nor have the books about C and C++ caught up with the current reality. These things must change.
The C standard even suggests, when defining undefined behaviour, that one of the possible options is "behaving during translation or program execution in a documented manner characteristic of the environment". If signed int arithmetic on the hardware wraps around upon overflow, then take that into account when optimising and don't assume it can't overflow. This is how C programmers want the language to work; compiler writers and standards be damned.
I've written plenty of Asm and also looked at as much if not more compiler output; and while the latter sometimes pleasingly surprises me, it's usually pretty evident that it was not generated by an intelligent entity. (And countless times, I've obtained size and/or speed savings by replacing the latter with what I would otherwise write.) My general impression is that there aren't enough highly-skilled Asm programmers working on compiler development --- there are many reasons I can think of for that, but one of the things I've always wished to exist is a simple, straightforward C compiler with "understandable" output and optimisation that creates results closer to what a human Asm programmer might write; with the associated optimisation advantages and predictability. This means not assuming anything just because the standard says so, but taking a more holistic (for lack of a better word) approach to compilation and optimisation.
A long-time favourite post on this: http://blog.metaobject.com/2014/04/cc-osmartass.html
Don't speak for all of us.
Some of us may take the language for what it is, instead of assuming it to be a "high level portable assembler" which it is not, but which a particular implementation of it may be, thanks to the standard leaving much up to the implementation.
It is this same leeway that (thankfully) allows smart optimizing compilers that can generate performant code for programs that are not omgwtf-optimized at the source level.
And it is this same leeway that allows very simplistic, naive implementations that do not go out of their way to perform crazy analysis to try and warn you about some corner case of undesirable behavior your code might hit.
You're right that it is the compilers can and need to change to accommodate different types of users. But I don't think every compiler should just go back to being a 1991 compiler; not every compiler needs to accommodate every user.
https://gcc.gnu.org/onlinedocs/gcc-7.1.0/gcc/Object-Size-Che...
https://gcc.gnu.org/onlinedocs/gcc-7.1.0/gcc/Pointer-Bounds-...
I bet not much UNIX software compiled with gcc enables them.
Android is probably the only OS that does make use of them.
https://android-developers.googleblog.com/2017/04/fortify-in...
Rust was susceptible to Stack Clash because they ripped out their existing stack probing to gain a few percent increase in performance.
The article for this thread literally says, "For many use cases ASan is the better choice because it has much less overhead." IME Valgrind does a _much_ better job of detecting memory issues, partly because ASan only detects issues directly triggered in the code that you compile, whereas Valgrind can detect memory issues directly triggered by external code, but indirectly caused by your code.
Citation please?
I thought Rust used guard pages, and that stack probing was something that it was moving towards, not something that was being ripped out.
Sure, but it is where it is more visible, specially micro-optimizing code as it is being written, without validating if it really matters to the application's use case with a profiler.
The school of systems programming languages from Algol side, was that correctness was much more relevant than pure performance, Algol dialects for systems programming already had Rust like unsafe blocks in the mid-60's.
In Burroughs B5500 the administrator could enable which applications with unsafe modules were allowed at all to be executed, 10 years before C was invented.
Someone else corrected me regarding the stack probes--Rust didn't have probes but actually checked the size of the stack. It's also worth pointing out that often times decisions are premised on maintenance burdens. Performance was one excuse for removing the stack checking in Rust; the other was the maintenance burden of maintaining a feature outside of LLVM.
That goes directly to the heart of the undefinedness issue in C. Doing the "obviously correct" thing wrt to choosing between possible undefined- or implementation-defined behaviors isn't as easy as we think. Good compilers are built on abstractions and code reuse, like any other piece of well-written software. Doing the correct thing in some obvious cases can result in having to make non-obvious tradeoffs elsewhere, possibly having to choose between adding (or maintaining) complexity, or removing a powerful feature, such as deterministic stack overflow prevention.
The history of C (indeed, Unix and the whole "worse is better" school of design) is about favoring less complexity in the tooling while shifting more of the burden onto the user. That's a judgment not about runtime performance, but a very unintuitive judgment about the performance and efficiency of programmer time; that you quickly reach a point of diminishing returns, much sooner than most people believe, putting complexity into, and relying on intrinsic features of, the tooling; that it's often better to make it easier to implement and integrate specialized tools, and easier to reuse existing tools as part of a more focused solution. From that perspective the design of C makes a ton more sense, even if reasonable people can still disagree about the results.
The notion that C--the language design, the ecosystem--is principally about performance is something mostly held by inexperienced programmers. It's a strawman of critics. There were plenty of other languages from the 1970s that were capable of generating faster code than C. Certainly performance was and continues to be of significant concern, but C is hardly unique in that sense. If the debate about how to balance performance with other issues stands-out, I'd argue that's primarily because of C's ubiquitous position and how it and the computing industry (including hardware industry) has co-evolved.
When something seems so "obvious" (security vs performance), yet everybody continues to seemingly repeat the same dumb mistakes, often times what we're missing is that our premises are flawed. Maybe C didn't become ubiquitous because of it's performance, but more so because it was easy to port and easy to build upon. After several years Rust still has a single implementation. Even Go has two production-quality implementations. That's not a criticism of Rust, just an observation that the "security vs performance" axis is hardly the only relevant dimension for judgment and explanation.
They implemented a botstraped optimizing compiler for it, making use of an intermediate language with multiple optimization passes.
http://rsim.cs.illinois.edu/arch/qual_papers/compilers/ausla...
It was like having LLVM in the 70's and is only one example of the research on optimizing compilers since FORTRAN was created.
• Existing code is important, existing implementations are not.
• Avoid “quiet changes.”
• Trust the programmer.
• Keep the language small and simple.
• Make it fast, even if it is not guaranteed to be portable.
In pre-ANSI C, and C89 C interpreted in line with the Rationale, if you wrote ‘+’, the expectation was that you'd get the target machine's ‘add’ instruction, no more and no less. It might overflow, it might not; it might crash or hang your machine on overflow or trap values — but that is all your problem, not the language's or compiler's. The ANSI invention of ‘undefined behaviour’ can be considered a good-faith attempt to codify things like this, but botching it was one of the two great failures of C89.
I worked on a commercial compiler at the time, and I'm confident no one thought of C89 as a license to screw the customer.
"12. Trust the programmer, as a goal, is outdated in respect to the security and safety programming communities. While it should not be totally disregarded as a facet of the spirit of C, the C11 version of the C Standard should take into account that programmers need the ability to check their work."
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2021.htm
I disagree. Undefined behavior is very important for optimization. See John Regehr's blog post on the topic [1], or any of DannyBee's numerous posts here explaining why GCC and Clang can and must exploit UB, etc. etc.
[1]: https://blog.regehr.org/archives/1467
> Summary: This UB is probably not a problem in practice (even if it is moderately displeasing to some of us).
Since LLVM is mostly based around C/C++ semantics this actually turned out to be a problem for rust, because rust can encode diverging functions in its type system and assume certain code sections to be unreachable. If the compiler then optimizes out code and the unreachable (thus not-compiled) section is reached things explode.
https://github.com/rust-lang/rust/issues/28728
If the compiler is able to do that analysis, why not flag the endless spin loop as a compilation error?
Function is used in multiple places and inlined.
With inlined context, compiler can eliminate loop sometimes.
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1528.htm