For the uninitiated: Rust differentiates between safe and unsafe code (safe being the default and unsafe needing the unsafe keyword). The unsafe code is like C/C++: it's more powerful than safe code, allowing you to do more things, including allowing you to trigger undefined behavour. Any mistake can possibly mean UB with all the consequences of crashes, wrong behaviour and nasal demons. Safe Rust on the other hand, is not supposed to be allowed to trigger such UB, no matter what you write. If you are able to trigger UB in 100% safe code, it's considered a bug in the language, an unsoundness bug.
Note that it was known that Pin/Generators had soundness problems when you turned on noalias optimization flags [1]. They have however decided to stabilize it without fixing those problems as they could be fixed afterwards as well. I'm not entirely sure yet about the impact of this one, will have to read the thread I guess.
Simon Peyton Jones initial intuition [0] was that Rust would be full of soundness holes due to not starting with a full formalization. Derek Dryer, Ralf Jung and others in the RustBelt project [1] did amazing work in formalizing Rust's safety guarantees, showing that Peyton Jones' guess was not accurate.
There is nothing wrong about having bugs, of course, but the reaction to the bug in this thread shows that mathematical correctness is not as universally valued as I thought it would be. I agree with the sentiment that this is a bigger issue that shouldn't be discussed in a technical thread about a specific bug. However, after reading this, it is unclear to me whether mathematical correctness is regarded in the Rust project as an explicit goal, an explicit non-goal or an unessential nice-to-have. (I do not mean to insinuate anything with the word "unclear", as I believe all three options are valid and appeal to different use cases. Almost all of the popular languages don't care about this, for instance.)
> However, after reading this, it is unclear to me whether mathematical correctness is regarded in the Rust project as an explicit goal, an explicit non-goal or an unessential nice-to-have. (I do not mean to insinuate anything with the word "unclear", as I believe all three options are valid and appeal to different use cases.)
It's a balance. We cannot drop everything, nor stop all development, in order to get proofs. However, we are actively working toward formalizing the language itself, because we do see value in it.
Totally reasonable stance, in my opinion, since Rust moves quickly and formalizing things takes time. But what is the stance on soundness bugs? Are they sometimes acceptable as permanent parts of the language, or is it a goal to eventually fix any soundness bugs that show up?
Soundness bugs are the only thing we make an exception in our stability policy for. In general, unsoundness is treated as a serious issue. However, "unsoundness" can mean a few different things; in the "semantics of the language" sense, they're taken very seriously. In the "there's a compiler bug" sense, they're taken seriously, but in accordance with how likely they are to affect end users, and how much work they are to fix. For example, an incorrect type signature in the standard library was fixed one day after release by changing it to the correct one and re-releasing; an issue where casting between floating points and integers causes some problems has still not been fixed, because it partially has to do with LLVM's semantics combined with our semantics, and it's not clear how to do the right thing while not seriously regressing performance. (That being said that particular issue has had a flurry of activity in the past week...)
Compiler bugs are not what I had in mind, so this answer is very reassuring! The original thread led me to believe this was not an absolute, official policy of the Rust project, but I am happy to hear it is.
Please note that threads on internals are able to be posted to by anyone; things expressed there may not represent the way that the team feels about things.
Not sure about the exact official stance, but can't imagine it's anything other than eventually fixing all soundness bugs, at least as long as someone is willing to do the work.
However, an important thing to note is that Rust strives to be safe only against unintentional bugs, and not against intentional malicious code (against which you are supposed to use OS/CPU-level sandboxing), so unsoundness issues that are very unlikely to be accidentally triggered aren't necessarily a very high priority.
This is a reasonable policy, since current Rust does not have a certified provably correct compiler and Rust programs in current practice don't come with formal proofs, so full provable soundness in type system doesn't really translate to any useful properties, since programs can intentionally fail to do what they are supposed to do in other ways (just do something safe but malicious or incorrect or exploit bugs in rustc or LLVM).
For instance, this issue is very unlikely to be triggered by non-malicious code, so emergency action to fix it is not being taken, and instead it will probably be fixed only once there is clear consensus on the best fix.
The line between "unintentional" bugs and "intentional/malicious" escape hatches is quite fuzzy given the extent that most Rust projects rely on outside code. Soundness bugs, at least in stable rust, are serious and should be addressed as such, or Rust developers will end up with a leftpad situation as soon as a popular third-party crate 'unintentionally' introduces some sort of soundness concern.
Third party crates can include malicious safe code that doesn't exploit any soundness bugs, so I'm not sure how exploiting soundness bugs could make the situation worse.
It's only a problem if you try to restrict I/O APIs and use the Rust type system as a security boundary, which is something that I don't think anyone does and that should not be done since the surface area is too vast to secure without formal proofs of the whole Rust stack, which are too time-consuming to produce with current technologies.
> A capsule is a Rust struct and associated functions. Capsules interact with each other directly, accessing exposed fields and calling functions in other capsules. Trusted platform configuration code initializes them, giving them access to any other capsules or kernel resources they need. Capsules can protect internal state by not exporting certain functions or fields.
> ...
> Rust’s language protection offers strong safety guarantees. Unless a capsule is able to subvert the Rust type system, it can only access resources explicitly granted to it, and only in ways permitted by the interfaces those resources expose. However, because capsules are cooperatively scheduled in the same single-threaded event loop as the kernel, they must be trusted for system liveness. If a capsule panics, or does not yield back to the event handler, the system can only recover by restarting.
I take it from the present discussion that that might not be as good of an idea as they think.
Worth noting that they also have process isolation on top, but it doesn’t seem to be motivated by any potential insecurity of the type system.
I don't get how that's ever possible, regardless of soundness issues. Can you not simply use unsafe in a capsule, and violate this constraint? The only way isolates in V8 manage to provide any isolation is because the runtime/ language forbid such things entirely.
I suspect it is not so simple as "Rust's safety ensures malicious rust code can't access data".
Stealing credit card numbers, displaying adware and launching missiles are all safe Rust.
Rust does not offer any protection against malicious code and it does not claim to. It offers protection against malicious input to non-malicious code, but no protection against malicious code itself.
In theory, if there were incredibly massive holes that affected most Rust code and could not be fixed without backwards-incompatible changes, I guess so. Details matter in cases like these. It would just depend.
This was something that kept me up at night in the run-up to Rust 1.0 and beyond (dreadful memories of the leakpocalypse), but so far we seem to have been quite lucky in that nobody has yet discovered a dramatic and fundamentally damning soundness flaw (and the progress of formal verification since 1.0 has increased confidence somewhat in the absence of such a flaw, but `Pin` represents a rather hairier increase in surface area than is usual... fingers crossed :P ).
Where it is essentially argued that "all development should be stopped until we have formal proofs".
I guess it depends on which parts of the language things touch. `Ordering::Unordered` touches fundamentals parts of the memory model, and well, `Pin` touches the fundamental kinds of values that Rust has: owned, shared, mut shared, and with pin, also pinned.
When people complain that things are stabilized too quickly, I don't think they complain about async / await taking years. The `Pin` API was unstable for like less than half a year in its final form before being stabilized, that's quite a short time for people to try to find holes in it, given how fundamental it is.
Ralf is the one writing the soundness proofs (and bless him for doing so), but I don't believe that he's actually on the language team, so I wouldn't interpret that as an official stance.
Team members can have their own opinion, of course. That is exactly what is happening in that thread, though Mazdak is the only language team member arguing that position. I personally wouldn't characterize this as "all development should be stopped until we have formal proofs," though, this argument is "if we commit to this API we will be locked out of one particular model and I'm not sure that we should be making that kind of decision yet."
Empirically it's pretty clear that the language is being developed. If that were the decision of the team, it would pretty much have to go through an RFC, and so we'd be able to point to that.
I would say that SPJ was quite right, though. Compare and contrast with Haskell. In Haskell, there are broadly only three cases of unsoundness:
* Infinite loops
* `undefined`
* Abuse of unsafe IO behavior, including monads built on or reducible to IO
Rust's unsoundness bugs are all over [0], and come from so many different angles that they feel indistinct from any other low-level C-like language.
I looked a little into RustBelt; how would it have helped this `Pin` issue? I browsed [1] a bit and it seems like `Pin` would have to have been proven on its own in an ad-hoc way not covered by RustBelt.
You are being very disingenuous. Rust does not have unsound bugs all over stable. Linking to an issue tracker for bugs in development branch proves nothing.
As a Rust user, I think it's fine for the grandparent commenter to link to the I-unsound tag; even if some of those bugs are only accessible on the unstable nightly release, there are plenty of soundness bugs to choose from on the stable releases. The important thing is Rust's philosophy that soundness issues are to be taken seriously, even going so far as to allow backwards-incompatible changes if necessary to address them (though obviously this is a situation of last resort).
There are several people working on formalizing aspects of Rust, and though it will take a long time to get there such efforts have already found and fixed bugs in Rust and its standard library. It's fair to take SPJ's critique on the chin for now; Rust deliberately chose the path of eventual soundness in order to produce a usable language in less time, and as long as we keep working towards that goal there's nothing to be ashamed about.
> I looked a little into RustBelt; how would it have helped this `Pin` issue? I browsed [1] a bit and it seems like `Pin` would have to have been proven on its own in an ad-hoc way not covered by RustBelt.
To clarify, we do have a work-in-progress formalization of pinning as part of RustBelt [0] (I worked on it over the summer). We did not discover this issue because RustBelt does not currently model traits, and the problem here is very much about the interaction between traits and pinning. Ralf touches on this in the linked discussion:
> Actually trying to prove things about Pin could have helped, but one of our team actually did that over the summer and he did not find this issue. The problem is that our formalization does not actually model traits. So while we do have a formal version of the contract that Deref/DerefMut need to satisfy, we cannot even state things like "for any Pin<T> where T: Deref, this impl satisfies the contract".
Not sure if you caught it, but Ralf does address this somewhat in the thread:
> Good question. We even had a sketch of a formal model for pinning, but that model was always about the three separate pointer types (PinBox<T>/PinMut<T>/Pin<T>). The switch to the generic Pin was done without formal modelling.
> Actually trying to prove things about Pin could have helped, but one of our team actually did that over the summer and he did not find this issue. The problem is that our formalization does not actually model traits. So while we do have a formal version of the contract that Deref/DerefMut need to satisfy, we cannot even state things like "for any Pin<T> where T: Deref, this impl satisfies the contract"
That does sound scary, but I think the point is that this is a known blind spot in the formal model. It also has a known solution; the trait ought to be unsafe, so that the implementor takes responsibility for upholding the contract. The issue with Pin is it’s using an existing, safe trait, on the informal assumption that it couldn’t be unsoundly aliased by users. (Of course that turns out to be false.)
So you can look at that and say “an unsound feature got into the stdlib without passing formal verification”, and that’s true, speaking to a lack of concern for formalization.
On the other hand, you could look at it and say “the unsoundness of this feature is well isolated by formal methods and leakage at this interface is not the end of the world”, which is also true, and speaks to the utility of the formal work that has been done.
It seems clear to me that the answer for Rust is “balance”. Formal verification is important not not to have, and things that can’t be verified shouldn’t be allowed. But formal verification takes up time when everything could be working 100% in production, and the alternative is likely even less sound formally, so it’s better to build an API now with some known risks than to wait for something known to be risk-free.
Whatever your personal philosophy is, it seems clear that toeing that line has been instrumental to Rust’s success.
* Hack without fear (which can be interpreted as soundness)
* Practical programming language (a language you can use to get stuff done)
A full operational semantics of Rust with soundness proofs is probably at least a decade away, blocking the language on that would mean that one can't get anything done until that happens. And that is at tension with Rust being a practical language.
So if you had a spectrum of languages, and you were to classify them along these two axes, Rust would be at the very safe and very practical end of the spectrum. But safety isn't an absolute, it's a spectrum. Rust won't protect you form, e.g., race conditions, dead locks, memory leaks, or somebody breaking into your house and hitting your computer with a baseball bat, etc. You need to have a threat model, and then deduce from it how much safety is enough. Rust aims to only give you memory and thread safety, so there is definitely room for much safer languages than Rust down the road, that protect you from these other things. The art is increasing safety, without sacrificing practicality, performance, etc.
For those who haven't heard of `Pin` before (which is reasonable even for Rust users, for whom this is a rather obscure corner of the standard library): it's essentially a trait that indicates that a value cannot be moved from its current location in memory.
Here's an analogy: Rust already differs from most languages in that the behavior that you get "by default" when defining a new type is very limited. For example, if I define a type `struct Foo`, then by default this type can't be copied around: `let x = Foo; let y = x;` is a simple operation that in most languages would involve copying memory, but in Rust involves a move instead. You can think of a move as a copy where the original value is no longer accessible (whether or not any bytes in memory actually get copied as a result of a move is an implementation detail, and is (hopefully) often optimized away). In order to make our type copyable, we would add an implementation of the `Copy` trait.
The observation is that moving is a more fundamental operation than copying, therefore, it is easier for types to opt in to copying than to opt out.
When Rust was released in 2015 this was a relatively radical concept for a language targeting a mainstream audience (we're essentially describing Rust's entire concept of ownership here, after all). But it turns out that it's possible that Rust may not have been radical enough! Consider: what if you want a type that not only can't be copied, but can't be moved as well?
Before Rust 1.0 it wasn't clear that such a concept of "unmoveability" would be generally useful. There were certainly cases, such as self-referential structs, that could have benefited from such a concept, however making moveability opt-in rather than opt-out would have required all types that do want to be moveable to explicitly announce that fact (via something like `#[derive(Move)]`), which is an annotation burden on all other code that must be considered.
It wasn't until the async/await work that another, more critical need for unmoveability was found: if the generators interally produced by `async` are capable of moving, then that means that generators are incapable of containing references, which means that async/await would become drastically less useful; there would be an entire fundamental part of the language that simply couldn't be used with it. Thus `Pin` was born in order to denote things that cannot be moved (and whose design is itself a very, very long discussion).
For this reason I'm somewhat amused by one of the comments in the OP asking something like "if `Pin` had remained unstable for an additional year, would this particular instance of unsoundness have been caught?" Because, conversely, we could ask whether Rust itself could have remained unstable for another five years and found a design that would have obviated `Pin` entirely by making moveability opt-in. However, of course, such things are easy to ask in hindsight, and stability is a prerequisite for having a broad base of adoption. Finding a balance between immediate stability and eventual perfection is the holy grail of industrial language design.
(Regarding the bug in the OP itself, I think it's unfortunate but I'm not especially worried by it at this juncture. The fact that Ralf Jung's team is looking into it gives me confidence that formal methods will eventually be applied here to more thoroughly explore the soundness of `Pin` in general (Ralf being one of the people who shaped `Pin` originally), and in the meantime I wouldn't be opposed to a band-aid fix, given that working extensively with `Pin` in the way shown in the OP is rare for normal users.)
> But it turns out that it's possible that Rust may not have been radical enough! Consider: what if you want a type that not only can't be copied, but can't be moved as well?
When talking about Pin, I think it is important to go back to a more fundamental level.
There are types that are not movable without extra processing, like a `struct` that has fields that reference other fields. When moving, those references become invalid.
Rust chose by allowing types to become unmovable so the `struct` can't go into an invalid state.
While C++ doesn't have the same safety guarantees, they did need to solve this problem. C++ chose to solve this with move constructors. I know move constructors will have their own challenges in Rust but I hope we do get them one day.
There is an unofficial 'transfer' crate that builds on Pin<> and on a further unofficial "stackpin" feature (pinning objects on the stack, which is currently not provided for in standard Rust) to provide an equivalent to C++ move constructors.
> Before Rust 1.0 it wasn't clear that such a concept of "unmoveability" would be generally useful. [...]
Hm, I ran into quite a lot of cases where I wanted Pin or something similar when porting a C program to Rust, and if I recall correctly also the creator of vulkano also ran into plenty of cases when writing it, so I am a bit surprised people did not realize this. My case was a program which in its C version was based around an epoll event loop, so maybe that is a bit similar to the async/generators issue.
I could have perhaps written "would be generally useful enough to justify the additional annotation burden" (alluded to later in the paragraph). Also, in what year were you discovering these use cases? The discussions in question would have had to have happened in 2014 or earlier; for people who began using Rust post-1.0, it may very well be that these use cases were encountered because the language was stable by then, and that people might not have bothered enough with an unstable language to bother pushing the language to its limits like this. Though I guess we can take solace in the idea that the language that comes after Rust someday might benefit from these case studies. :)
You are very correct in your guess that it was post-1.0, it is like you can read my mind. :) I would indeed not have bothered to port my production code to a non-stable language. But once I started doing that I ran into these issues quite quickly.
We can go back to the reason that Pin even exists in the first place. With Rust, you create a computation, and it implements the Future trait. When you want it to execute, you give it to an executor. This often moves the data structure that represents the computation to the heap, so that it has a stable address, and can continue running after the current stack frame is over. So, before the future starts executing, it can be moved around in memory (and probably will at least one time), but after it starts executing, it must never move again.
So the main use is convenience to seamlessly make stack variables permanent? Intuitively that doesn't sound worth the language design troubles compared to making the programmer declare these things on the heap manually. I'm not that familiar with Rust though.
Ideally it should be possible to avoid the heap entirely; futures in Rust should be capable of operating even in contexts that don't come with a memory allocator, such as memory-constrained embedded devices.
Rust’s ownership model is used to provide memory safety. Entities have only one owner so that the compiler can reason about when they go out of scope and can be safely dropped. If x didn’t get invalidated by the move then both x and y would own the entity. You could then do something like pass y into a function that would drop y at the end of the function scope, and then cause a use-after-free by using x. With the ownership model the compiler can guarantee that use-after-free is always a syntax error. The ownership model is a fair bit more complicated than this, but move semantics are at the core of it.
I don't get why we don't just use the same magic that `?Sized` uses, to make it opt-out in terms of mental overhead for the developer, but opt-in in terms of what's going on under the hood:
The syntax is different but Rust has this feature- its called auto traits, and its how we handle traits related to threadsafety. Its not a solution for what Pin solves because the whole notion that a type might be moveable or not moveable is actually a misstatement of the problem that Pin is solving.
> [...] would have required all types that do want to be moveable to explicitly announce that fact (via something like `#[derive(Move)]`) [...]
Isn't `Pin` even crazier, in that the same type can be moveable (when used outside of Pin) or not moveable (when used inside Pin)? A simple derive wouldn't be enough to capture that.
I wouldn't say it's crazy, or even unprecedented. Consider the reference-counted `Rc` smart pointer type, which allows a given type to act as though it were cloneable (when used inside of `Rc`) or not cloneable (when used without `Rc`).
The hypothetical opt-in moveability proposal is difficult to compare directly to `Pin` because they take such different approaches to the problem, but they would both address the problem (at least, as far as I have considered it).
Rather, Rust pushes soundness concerns into specific abstractions which have a hope of ever being formally verified (there are at least three teams working on this), and whose essential soundness invariants can be documented. Coupled with a philosophy that soundness bugs must be fixed even if it means breaking language backwards-compatibility and a community that takes soundness very seriously, I'd contend that this is a dramatic step forward for language security (especially if it can continue to light a fire under C++ to inspire it to evolve towards a more statically-verifiable design).
The evidence so for does not really support your conclusion. Even with some worse soundness bugs than this (a few were actually triggered commonly enough the fix only emitted a warning to give enough time for users to fix their code), bug density in safe rust code seems to be way down compared to C and C++.
Can you stop editing your comments to mean something entirely different after people respond to you? Here’s what I have from a couple of minutes ago:
> Basically, Rust just pushes the subtle bugs down a few layers, making them more subtle and likely to bite you at the worst possible time.
> And false confidence in the borrow checker and the type checker will make it harder to find these problems.
I'll pass.
Here is what you have now:
> Rust is different from any other language, considering the care and deftness that its creators go to bring advanced computer science and language concepts to the masses.
This isn’t the first time you’ve done this and it makes it very difficult to interact with you in good faith.
Personally I am not convinced Pin is a good idea as it works now. Not because of this soundness bug but because it is too hard to understand. Just look at how much explanation is required in the manual and how large the code examples are.
Agreed that `Pin` is imposing by dint of trying to provide "more fundamental" guarantees than Rust was originally designed for (see my sibling comment on this topic). Though ideally in the long run I'd hope that no regular user would ever have to encounter it (right now I think it may show up in some contexts related to futures, but as Rust's async story matures I would hope to see that need diminish).
Ideally the basic building blocks of Rust should be so easy to understand that any experienced Rust developer could use them too and not just the core team.
The cases where I personally wanted Pin were cases that I think will be solved in the future as async improves, but there are probably other more fringe cases which wont.
Absolutely, I don't think there should be arbitrary barriers to entry for people who wish to dive into the deepest parts of Rust's type system. Finding a suitable replacement for `Pin` would be such a big task that I'd say it's not worth considering for anything short of a hypothetical "Rust 2", but hopefully in the meantime we'll soldier on and get a better idea of the problem space in the meantime.
How on earth do you write your own `Future` type without understanding pinning ? Or your own combinator? Or executor ? Or how do you understand the error messages when trying to Box or Rc a future? Or write an `async` abstraction over some C FFI component that you need to use? Or...
If the goal was for users to just use high-level web frameworks, and when there is something that the framework cannot do (which is currently a lot), the expectation is that users should just shrug their shoulders and tell their bosses "Sorry boss, Rust can't do that", then sure, there is no need to understand pinning. But such users do not need to use Rust either, an pretty much any other programming language like Go or Python would have been a much better fit for them.
There is a huge gradient of use cases between "high level web frameworks" and "writing everything from scratch" and we've thought about the impact on users at every step of that gradient. But I'm not about to explain it to you, because you are being disingenuous and aggressive.
And even for the user that only uses high level web frameworks, the moment that user needs to debug its program (which might have a bug in its executor), it needs to understand pinning.
I don't see any use case in which understanding pinning is not necessary.
There's miles between understanding pinning well enough to call `Box::pin` (a safe API) and understanding pinning well enough to call `Pin::get_mut_unchecked` (an unsafe API). When we talk about understanding pinning, we mean the latter.
If they use async code as just a more office synchronous/threaded code, with purely linear control flow, then no knowledge of Pin is required. The users just sprinkle awaits and are good with it.
If users also explore the areas of which are more unique to async code (e.g. real concurrency, select!, custom combinators, cancellation mechanisms, etc) than they will indeed run fairly early into Pin.
I guess the majority of users who anticipated async/await fall into the first category. Most projects and requests around async code seem to be "just" webservers which handle some APIs. That will likely be the domain where one can live fairly well without Pin - at least if you do not implement the underlying webserver.
Same here. It was a clever hack to move Futures forward without (it seemed) needing changes to the language. But for the sake of simplicity, I'd prefer Rust to have some first-class language construct for self-referential structs.
No one has ever proposed a solution - first class or not - which wouldn't be significantly more invasive and infectious. Does it sound fun in code that doesn't have anything to do with futures to spend time worrying if this generic should be `T: ?Move` or not?
One thing I find deeply unintuitive about Pin: Pin means “this object will never ever be moved,” not “this object will not be moved while it is pinned.”
I can imagine an easier-to-understand world in which there is a reference type from which one cannot move during the lifetime of the reference, and in which one only ever gets this type of reference to immovable closures and such. I don’t know if the details would work out, but it would at least seem less weird to me.
The name for that reference type is a shared reference. It's literally the default reference type. It's not a sufficient guarantee for the use cases Pin supports.
I think the `!` part might be a typo, and that pseudocode needs more context to be valid, but the individual elements are themselves valid even if you'd never see them composed in this way, and I wouldn't expect any regular user to have to ever read or write anything like this. The leading `for<>` indicates a higher-ranked lifetime (which I've never once needed in eight years of writing Rust (though to be fair, I think it's only existed for like six of those years...), the `!` is the "Never" type indicating a statically-impossible state (honestly it really should just be called `Never` rather than `!`, this is a leftover artifact of prehistoric versions of Rust that predate even me. Maybe I'll write an RFC to propose that), the `?Sized` is a one-of-a-kind trait bound that users might need rarely (but I don't bother with, generally), and the `'a` is a lifetime type parameter (which is common for anyone working with references, although it's such an important part of the language that you don't bat an eye at it after a while).
Yes, it’s a type signature. It looks typical of Rust internals, but it would be very verbose for user code.
You’re not wrong to find it illegible; apart from the syntax, that phrase invokes the concepts of lifetimes, mutability, borrowing, and traits, none of which have straightforward semantic analogues in other common languages.
But FWIW they’re primitive concepts in Rust. A typical Rust user wouldn’t read or write code that looks like that very much, but they would know what it means, if that makes any sense.
Edit: Seeing sibling comments about whether this snippet actually parses, I guess I should amend this to say that a typical Rust user has an idea of what it’s supposed to mean :)
I think I originally had “piece of a type signature”, but I didn’t actually want to figure out which piece, so I fudged it. The point is well taken, I oughtn’t pretend I have a Rust parser in my head :)
First of all, it’s not actually valid syntax because I believe Rust doesn’t have negative trait bounds. But it’s a trait bound, so it’s (part) of a type signature. The type here is generic over both lifetimes and types (I think). My vague understanding is that for<‘a, T: ?Sized> means “in any lifetime, any type that is a member of ‘?Sized’”. And then the rest of the signature goes on to say (in made-up syntax) “does not implement DerefMut”. I think ?Sized means “size not known at compile time.”
No, it's bits of valid syntax thrown together informally to express an idea. There is no `!:` in Rust, and that `for` quantification never applies to types or bounds that way.
I've been writing rust since just before 1.0 and I've never written a type signature like that. I understand it just fine (even though it is invalid, I get what they're saying), but yeah, never needed it.
It's both a good and bad thing that rust has these sort of deep areas where 95% of users won't ever have to go - Pin being another example. I think it's much more good than bad though.
I didn't follow all of it, but it sounds not so bad.
1 & 2 both involve unsafe code that, while possibly reasonable in a complex application, is obviously wrong in the simple case. Of course turning a reference into a mutable reference will cause trouble. Was Pin SUPPOSED to be resilient to unsafe code? In any case, seems like a bug in DerefMut and Clone, not Pin.
The others are a bit more esoteric and do seem potentially concerning, but I'm not sure.
But still, I'm left with my earlier question: just how resilient is Pin supposed to be?
> Was Pin SUPPOSED to be resilient to unsafe code?
Regardless, for a user to be able to trigger undefined behavior without using the `unsafe` keyword indicates a bug here that must be fixed at the level of the language or standard library.
> Was Pin SUPPOSED to be resilient to unsafe code?
Remember, unsafe does not mean "I can do whatever I want," it means "I am promising to uphold some guarantee on my own." That is, let's assume we have a function:
/// Makes a new Foo.
///
/// # Safety
///
/// x must never be greater than 5
unsafe fn new(x: i32) -> Foo {
and you write
let x = 6;
unsafe {
new(x);
}
You have a bug, even though new is marked unsafe.
In my understanding of the unsafe examples above, the unsafe code was upholding the invariants that it was required to uphold, and so would be more like having written `let x = 2;` in the above example, if that caused an issue, clearly there's a bug.
I think this is a little too dumbed down for me to see the relevance to the actual problem.
Also, I believ 1&2 both require unsafe code to implement the trait that allows the unsoundness.
For what &T can I safely implement DerefMut? Is it the fault of Pin that a type has DerefMut implemented that the unsafe block of that implementation doesn't uphold safety?
The relevance to the problem is that even if these examples needed unsafe code, that doesn't mean that there's no problem. "resilient to unsafe code" is not the right way to think about unsafe.
> For what &T can I safely implement DerefMut? Is it the fault of Pin that a type has DerefMut implemented that the unsafe block of that implementation doesn't uphold safety?
It is the fault of Pin's safety guidelines, which suggest a contract that is not actually enough to uphold safety. That's why this is considered an unsoundness in Pin, and not a problem with the code written with unsafe. That is:
> that the unsafe block of that implementation doesn't uphold safety?
is not correct, the unsafe block does everything it's supposed to to uphold safety.
The 'unsafe' keyword in the internals post is referring to the stdlib implementation of Pin. The stdlib's use of unsafe is supposed to uphold safety guarantees for all functions that aren't exposed as unsafe, and the stdlib failed the guarantee here, thus allowing a soundness bug without the user using 'unsafe' anywhere.
Unsoundness in Pin aside, I think the conversation between Boats and Ralph is more interesting - are soundness issues going to be so strange and edgecasey that they can be addressed as one-offs, or does rust need a more formal model to ensure edge cases are covered?
I have no answer, it's just the only interesting part of that discussion imo. Everything else is just details of an issue that's too weird to really fully understand for me, and not novel or interesting enough to invest into understanding, anyway.
Since (to a first approximation) every individual who has the expertise and contextual knowledge to really evaluate this issue is a poster on internals.rust-lang.org, its pretty surprising to find this thread on the front page of Hacker News. I imagine some Hacker News users who upvoted this link did so out of technical interest, but I suspect a large portion of the attention comes from some combination of these misconceptions:
- The misconception that this could have a practical impact on users (the code being discussed on the thread is all obviously pathological & contrived).
- The misconception that Rust's type system and standard library never contain soundness issues and that this is an exceptional event (in fact we have a number of longstanding soundness issues).
We have a policy of fixing all soundness issues, so this issue will be fixed. In the meantime, while we decide the best solution, it will have no practical impact on Rust users. And none of the solutions we are considering would involve significant breakage to users, or invalidate any real code.
At a high level: the soundness issue occurs because the Pin API was designed based on certain reasoning about the behavior of pointers. This reasoning would be sound but for the fact that we have allowed certain exceptions in relationship to pointers to what are called the "orphan rules" (which usually enable local reasoning like this). These exceptions allow users to introduce code which, while contrived, allows them to violate the guarantees of the Pin API. Such is life.
I think this is really the bottom line here. This "Pin soundness" matter really seems analogous to the process that any compiler project would go through to identify and fix obscure compiler bugs. I don't think it's quite so deserving of the amount of attention it's getting.
> The misconception that this could have a practical impact on users (the code being discussed on the thread is all obviously pathological & contrived).
Famous last words?
I mean I'm not an expert in Rust and even less in Pin, but I've seen my share of theoretical bugs thought of not possibly having any impact in the real world because of too theoretical. In other areas, when you debug a triple segfault and you understand the crazy conditions that lead to it, or when you render a piece of C++ code conforming instead of technically UB and it then starts to crash when in its UB form it worked perfectly, you start to consider that everything is possible :)
This is exactly the kind of comment that makes me dislike Hacker News so strongly, because I find it endemic here.
The idea that someone could think an issue is theoretical and then discover it is practically significant is obvious and no insight at all - it reduces to "people are sometimes wrong." I am declaring based on my significant relevant expertise that this issue is not practically important. Your comment contributes nothing but baseless contradiction.
98 comments
[ 3.1 ms ] story [ 150 ms ] threadNote that it was known that Pin/Generators had soundness problems when you turned on noalias optimization flags [1]. They have however decided to stabilize it without fixing those problems as they could be fixed afterwards as well. I'm not entirely sure yet about the impact of this one, will have to read the thread I guess.
[1]: https://github.com/rust-lang/rust/issues/62149#issuecomment-...
There is nothing wrong about having bugs, of course, but the reaction to the bug in this thread shows that mathematical correctness is not as universally valued as I thought it would be. I agree with the sentiment that this is a bigger issue that shouldn't be discussed in a technical thread about a specific bug. However, after reading this, it is unclear to me whether mathematical correctness is regarded in the Rust project as an explicit goal, an explicit non-goal or an unessential nice-to-have. (I do not mean to insinuate anything with the word "unclear", as I believe all three options are valid and appeal to different use cases. Almost all of the popular languages don't care about this, for instance.)
[0] https://youtu.be/t0mhvd3-60Y?t=130 [1] https://plv.mpi-sws.org/rustbelt/
It's a balance. We cannot drop everything, nor stop all development, in order to get proofs. However, we are actively working toward formalizing the language itself, because we do see value in it.
The GitHub label for these bugs is A-Unsound, which has the description "A soundness hole (worst kind of bug), see: https://en.wikipedia.org/wiki/Soundness".
Please note that threads on internals are able to be posted to by anyone; things expressed there may not represent the way that the team feels about things.
However, an important thing to note is that Rust strives to be safe only against unintentional bugs, and not against intentional malicious code (against which you are supposed to use OS/CPU-level sandboxing), so unsoundness issues that are very unlikely to be accidentally triggered aren't necessarily a very high priority.
This is a reasonable policy, since current Rust does not have a certified provably correct compiler and Rust programs in current practice don't come with formal proofs, so full provable soundness in type system doesn't really translate to any useful properties, since programs can intentionally fail to do what they are supposed to do in other ways (just do something safe but malicious or incorrect or exploit bugs in rustc or LLVM).
For instance, this issue is very unlikely to be triggered by non-malicious code, so emergency action to fix it is not being taken, and instead it will probably be fixed only once there is clear consensus on the best fix.
It's only a problem if you try to restrict I/O APIs and use the Rust type system as a security boundary, which is something that I don't think anyone does and that should not be done since the surface area is too vast to secure without formal proofs of the whole Rust stack, which are too time-consuming to produce with current technologies.
> A capsule is a Rust struct and associated functions. Capsules interact with each other directly, accessing exposed fields and calling functions in other capsules. Trusted platform configuration code initializes them, giving them access to any other capsules or kernel resources they need. Capsules can protect internal state by not exporting certain functions or fields.
> ...
> Rust’s language protection offers strong safety guarantees. Unless a capsule is able to subvert the Rust type system, it can only access resources explicitly granted to it, and only in ways permitted by the interfaces those resources expose. However, because capsules are cooperatively scheduled in the same single-threaded event loop as the kernel, they must be trusted for system liveness. If a capsule panics, or does not yield back to the event handler, the system can only recover by restarting.
I take it from the present discussion that that might not be as good of an idea as they think.
Worth noting that they also have process isolation on top, but it doesn’t seem to be motivated by any potential insecurity of the type system.
I suspect it is not so simple as "Rust's safety ensures malicious rust code can't access data".
Lots of example code https://github.com/tock/tock/tree/master/capsules
Rust does not offer any protection against malicious code and it does not claim to. It offers protection against malicious input to non-malicious code, but no protection against malicious code itself.
That kind of contradict some of the reactions, e.g., in this thread about adding an `Ordering::Unordered` to the language: https://internals.rust-lang.org/t/unordered-as-a-solution-to...
Where it is essentially argued that "all development should be stopped until we have formal proofs".
I guess it depends on which parts of the language things touch. `Ordering::Unordered` touches fundamentals parts of the memory model, and well, `Pin` touches the fundamental kinds of values that Rust has: owned, shared, mut shared, and with pin, also pinned.
When people complain that things are stabilized too quickly, I don't think they complain about async / await taking years. The `Pin` API was unstable for like less than half a year in its final form before being stabilized, that's quite a short time for people to try to find holes in it, given how fundamental it is.
Empirically it's pretty clear that the language is being developed. If that were the decision of the team, it would pretty much have to go through an RFC, and so we'd be able to point to that.
* Infinite loops
* `undefined`
* Abuse of unsafe IO behavior, including monads built on or reducible to IO
Rust's unsoundness bugs are all over [0], and come from so many different angles that they feel indistinct from any other low-level C-like language.
I looked a little into RustBelt; how would it have helped this `Pin` issue? I browsed [1] a bit and it seems like `Pin` would have to have been proven on its own in an ad-hoc way not covered by RustBelt.
[0] https://github.com/rust-lang/rust/labels/I-unsound%20%F0%9F%...
[1] https://gitlab.mpi-sws.org/iris/lambda-rust/tree/popl18
There are several people working on formalizing aspects of Rust, and though it will take a long time to get there such efforts have already found and fixed bugs in Rust and its standard library. It's fair to take SPJ's critique on the chin for now; Rust deliberately chose the path of eventual soundness in order to produce a usable language in less time, and as long as we keep working towards that goal there's nothing to be ashamed about.
To clarify, we do have a work-in-progress formalization of pinning as part of RustBelt [0] (I worked on it over the summer). We did not discover this issue because RustBelt does not currently model traits, and the problem here is very much about the interaction between traits and pinning. Ralf touches on this in the linked discussion:
> Actually trying to prove things about Pin could have helped, but one of our team actually did that over the summer and he did not find this issue. The problem is that our formalization does not actually model traits. So while we do have a formal version of the contract that Deref/DerefMut need to satisfy, we cannot even state things like "for any Pin<T> where T: Deref, this impl satisfies the contract".
[0] - https://gitlab.mpi-sws.org/iris/lambda-rust/tree/gpirlea/pin...
> Good question. We even had a sketch of a formal model for pinning, but that model was always about the three separate pointer types (PinBox<T>/PinMut<T>/Pin<T>). The switch to the generic Pin was done without formal modelling.
> Actually trying to prove things about Pin could have helped, but one of our team actually did that over the summer and he did not find this issue. The problem is that our formalization does not actually model traits. So while we do have a formal version of the contract that Deref/DerefMut need to satisfy, we cannot even state things like "for any Pin<T> where T: Deref, this impl satisfies the contract"
That does sound scary, but I think the point is that this is a known blind spot in the formal model. It also has a known solution; the trait ought to be unsafe, so that the implementor takes responsibility for upholding the contract. The issue with Pin is it’s using an existing, safe trait, on the informal assumption that it couldn’t be unsoundly aliased by users. (Of course that turns out to be false.)
So you can look at that and say “an unsound feature got into the stdlib without passing formal verification”, and that’s true, speaking to a lack of concern for formalization.
On the other hand, you could look at it and say “the unsoundness of this feature is well isolated by formal methods and leakage at this interface is not the end of the world”, which is also true, and speaks to the utility of the formal work that has been done.
It seems clear to me that the answer for Rust is “balance”. Formal verification is important not not to have, and things that can’t be verified shouldn’t be allowed. But formal verification takes up time when everything could be working 100% in production, and the alternative is likely even less sound formally, so it’s better to build an API now with some known risks than to wait for something known to be risk-free.
Whatever your personal philosophy is, it seems clear that toeing that line has been instrumental to Rust’s success.
* Hack without fear (which can be interpreted as soundness) * Practical programming language (a language you can use to get stuff done)
A full operational semantics of Rust with soundness proofs is probably at least a decade away, blocking the language on that would mean that one can't get anything done until that happens. And that is at tension with Rust being a practical language.
So if you had a spectrum of languages, and you were to classify them along these two axes, Rust would be at the very safe and very practical end of the spectrum. But safety isn't an absolute, it's a spectrum. Rust won't protect you form, e.g., race conditions, dead locks, memory leaks, or somebody breaking into your house and hitting your computer with a baseball bat, etc. You need to have a threat model, and then deduce from it how much safety is enough. Rust aims to only give you memory and thread safety, so there is definitely room for much safer languages than Rust down the road, that protect you from these other things. The art is increasing safety, without sacrificing practicality, performance, etc.
Here's an analogy: Rust already differs from most languages in that the behavior that you get "by default" when defining a new type is very limited. For example, if I define a type `struct Foo`, then by default this type can't be copied around: `let x = Foo; let y = x;` is a simple operation that in most languages would involve copying memory, but in Rust involves a move instead. You can think of a move as a copy where the original value is no longer accessible (whether or not any bytes in memory actually get copied as a result of a move is an implementation detail, and is (hopefully) often optimized away). In order to make our type copyable, we would add an implementation of the `Copy` trait.
The observation is that moving is a more fundamental operation than copying, therefore, it is easier for types to opt in to copying than to opt out.
When Rust was released in 2015 this was a relatively radical concept for a language targeting a mainstream audience (we're essentially describing Rust's entire concept of ownership here, after all). But it turns out that it's possible that Rust may not have been radical enough! Consider: what if you want a type that not only can't be copied, but can't be moved as well?
Before Rust 1.0 it wasn't clear that such a concept of "unmoveability" would be generally useful. There were certainly cases, such as self-referential structs, that could have benefited from such a concept, however making moveability opt-in rather than opt-out would have required all types that do want to be moveable to explicitly announce that fact (via something like `#[derive(Move)]`), which is an annotation burden on all other code that must be considered.
It wasn't until the async/await work that another, more critical need for unmoveability was found: if the generators interally produced by `async` are capable of moving, then that means that generators are incapable of containing references, which means that async/await would become drastically less useful; there would be an entire fundamental part of the language that simply couldn't be used with it. Thus `Pin` was born in order to denote things that cannot be moved (and whose design is itself a very, very long discussion).
For this reason I'm somewhat amused by one of the comments in the OP asking something like "if `Pin` had remained unstable for an additional year, would this particular instance of unsoundness have been caught?" Because, conversely, we could ask whether Rust itself could have remained unstable for another five years and found a design that would have obviated `Pin` entirely by making moveability opt-in. However, of course, such things are easy to ask in hindsight, and stability is a prerequisite for having a broad base of adoption. Finding a balance between immediate stability and eventual perfection is the holy grail of industrial language design.
(Regarding the bug in the OP itself, I think it's unfortunate but I'm not especially worried by it at this juncture. The fact that Ralf Jung's team is looking into it gives me confidence that formal methods will eventually be applied here to more thoroughly explore the soundness of `Pin` in general (Ralf being one of the people who shaped `Pin` originally), and in the meantime I wouldn't be opposed to a band-aid fix, given that working extensively with `Pin` in the way shown in the OP is rare for normal users.)
When talking about Pin, I think it is important to go back to a more fundamental level.
There are types that are not movable without extra processing, like a `struct` that has fields that reference other fields. When moving, those references become invalid.
Rust chose by allowing types to become unmovable so the `struct` can't go into an invalid state.
While C++ doesn't have the same safety guarantees, they did need to solve this problem. C++ chose to solve this with move constructors. I know move constructors will have their own challenges in Rust but I hope we do get them one day.
Hm, I ran into quite a lot of cases where I wanted Pin or something similar when porting a C program to Rust, and if I recall correctly also the creator of vulkano also ran into plenty of cases when writing it, so I am a bit surprised people did not realize this. My case was a program which in its C version was based around an epoll event loop, so maybe that is a bit similar to the async/generators issue.
I'd like to ask, why is moving generally useful? Why would I want to move x to y and invalidate x?
Does that make sense?
Isn't `Pin` even crazier, in that the same type can be moveable (when used outside of Pin) or not moveable (when used inside Pin)? A simple derive wouldn't be enough to capture that.
The hypothetical opt-in moveability proposal is difficult to compare directly to `Pin` because they take such different approaches to the problem, but they would both address the problem (at least, as far as I have considered it).
> Basically, Rust just pushes the subtle bugs down a few layers, making them more subtle and likely to bite you at the worst possible time.
> And false confidence in the borrow checker and the type checker will make it harder to find these problems. I'll pass.
Here is what you have now:
> Rust is different from any other language, considering the care and deftness that its creators go to bring advanced computer science and language concepts to the masses.
This isn’t the first time you’ve done this and it makes it very difficult to interact with you in good faith.
https://doc.rust-lang.org/beta/std/pin/index.html
The cases where I personally wanted Pin were cases that I think will be solved in the future as async improves, but there are probably other more fringe cases which wont.
Otherwise it is impossible to become productive with that part of the language.
If the goal was for users to just use high-level web frameworks, and when there is something that the framework cannot do (which is currently a lot), the expectation is that users should just shrug their shoulders and tell their bosses "Sorry boss, Rust can't do that", then sure, there is no need to understand pinning. But such users do not need to use Rust either, an pretty much any other programming language like Go or Python would have been a much better fit for them.
I don't see any use case in which understanding pinning is not necessary.
If they use async code as just a more office synchronous/threaded code, with purely linear control flow, then no knowledge of Pin is required. The users just sprinkle awaits and are good with it.
If users also explore the areas of which are more unique to async code (e.g. real concurrency, select!, custom combinators, cancellation mechanisms, etc) than they will indeed run fairly early into Pin.
I guess the majority of users who anticipated async/await fall into the first category. Most projects and requests around async code seem to be "just" webservers which handle some APIs. That will likely be the domain where one can live fairly well without Pin - at least if you do not implement the underlying webserver.
At best, when using futures, I randomly wrap things in `Pin<Box<..>>`, hoping that the compiler might stop complaining.
I can imagine an easier-to-understand world in which there is a reference type from which one cannot move during the lifetime of the reference, and in which one only ever gets this type of reference to immovable closures and such. I don’t know if the details would work out, but it would at least seem less weird to me.
https://internals.rust-lang.org/t/unsoundness-in-pin/11311/3...
You’re not wrong to find it illegible; apart from the syntax, that phrase invokes the concepts of lifetimes, mutability, borrowing, and traits, none of which have straightforward semantic analogues in other common languages.
But FWIW they’re primitive concepts in Rust. A typical Rust user wouldn’t read or write code that looks like that very much, but they would know what it means, if that makes any sense.
Edit: Seeing sibling comments about whether this snippet actually parses, I guess I should amend this to say that a typical Rust user has an idea of what it’s supposed to mean :)
https://doc.rust-lang.org/beta/nomicon/hrtb.html
It's both a good and bad thing that rust has these sort of deep areas where 95% of users won't ever have to go - Pin being another example. I think it's much more good than bad though.
1 & 2 both involve unsafe code that, while possibly reasonable in a complex application, is obviously wrong in the simple case. Of course turning a reference into a mutable reference will cause trouble. Was Pin SUPPOSED to be resilient to unsafe code? In any case, seems like a bug in DerefMut and Clone, not Pin.
The others are a bit more esoteric and do seem potentially concerning, but I'm not sure.
But still, I'm left with my earlier question: just how resilient is Pin supposed to be?
Regardless, for a user to be able to trigger undefined behavior without using the `unsafe` keyword indicates a bug here that must be fixed at the level of the language or standard library.
Remember, unsafe does not mean "I can do whatever I want," it means "I am promising to uphold some guarantee on my own." That is, let's assume we have a function:
and you write You have a bug, even though new is marked unsafe.In my understanding of the unsafe examples above, the unsafe code was upholding the invariants that it was required to uphold, and so would be more like having written `let x = 2;` in the above example, if that caused an issue, clearly there's a bug.
Also, I believ 1&2 both require unsafe code to implement the trait that allows the unsoundness.
For what &T can I safely implement DerefMut? Is it the fault of Pin that a type has DerefMut implemented that the unsafe block of that implementation doesn't uphold safety?
> For what &T can I safely implement DerefMut? Is it the fault of Pin that a type has DerefMut implemented that the unsafe block of that implementation doesn't uphold safety?
It is the fault of Pin's safety guidelines, which suggest a contract that is not actually enough to uphold safety. That's why this is considered an unsoundness in Pin, and not a problem with the code written with unsafe. That is:
> that the unsafe block of that implementation doesn't uphold safety?
is not correct, the unsafe block does everything it's supposed to to uphold safety.
See the playground link containing a demonstration of the issue: https://play.rust-lang.org/?version=stable&mode=debug&editio...
The 'unsafe' keyword in the internals post is referring to the stdlib implementation of Pin. The stdlib's use of unsafe is supposed to uphold safety guarantees for all functions that aren't exposed as unsafe, and the stdlib failed the guarantee here, thus allowing a soundness bug without the user using 'unsafe' anywhere.
I have no answer, it's just the only interesting part of that discussion imo. Everything else is just details of an issue that's too weird to really fully understand for me, and not novel or interesting enough to invest into understanding, anyway.
Since (to a first approximation) every individual who has the expertise and contextual knowledge to really evaluate this issue is a poster on internals.rust-lang.org, its pretty surprising to find this thread on the front page of Hacker News. I imagine some Hacker News users who upvoted this link did so out of technical interest, but I suspect a large portion of the attention comes from some combination of these misconceptions:
- The misconception that this could have a practical impact on users (the code being discussed on the thread is all obviously pathological & contrived).
- The misconception that Rust's type system and standard library never contain soundness issues and that this is an exceptional event (in fact we have a number of longstanding soundness issues).
We have a policy of fixing all soundness issues, so this issue will be fixed. In the meantime, while we decide the best solution, it will have no practical impact on Rust users. And none of the solutions we are considering would involve significant breakage to users, or invalidate any real code.
At a high level: the soundness issue occurs because the Pin API was designed based on certain reasoning about the behavior of pointers. This reasoning would be sound but for the fact that we have allowed certain exceptions in relationship to pointers to what are called the "orphan rules" (which usually enable local reasoning like this). These exceptions allow users to introduce code which, while contrived, allows them to violate the guarantees of the Pin API. Such is life.
Famous last words?
I mean I'm not an expert in Rust and even less in Pin, but I've seen my share of theoretical bugs thought of not possibly having any impact in the real world because of too theoretical. In other areas, when you debug a triple segfault and you understand the crazy conditions that lead to it, or when you render a piece of C++ code conforming instead of technically UB and it then starts to crash when in its UB form it worked perfectly, you start to consider that everything is possible :)
The idea that someone could think an issue is theoretical and then discover it is practically significant is obvious and no insight at all - it reduces to "people are sometimes wrong." I am declaring based on my significant relevant expertise that this issue is not practically important. Your comment contributes nothing but baseless contradiction.