This is our first-ever z=2 release. Even though it's only 16 days until 1.27.0, we decided to go ahead and issue it, given the unsoundness. Happy as always to answer questions.
I remember reading a few weeks ago on Hacker News about a team of people who were working to mathematically prove that some of these issues are impossible in existing programming constructs. I can't entirely remember whether it was a thing people wanted to do or a thing that people were already doing, would you happen to know?
Participants range from "I'm interested in formal methods" to "my company uses formal methods and we're starting to apply them to Rust" to "The EU has given almost two million Euro for a university grant to prove Rust code" and everywhere in between :)
> Participants range from "I'm interested in formal methods" to "my company uses formal methods and we're starting to apply them to Rust" to "The EU has given almost two million Euro for a university grant to prove Rust code" and everywhere in between :)
Steve - can you give me a source for the latter? I'd like to read more into that, specifically.
The language currently knows nothing of coroutines. So, as you say, you chain Futures together, which is basically a monad. You submit that futures chain to an executor, like Tokio, and it basically moves the chain's stack to the heap, and then executes it. So, in some sense, Tokio is a stack-ful co-routine. It just so happens we know the stack's exact size at compile time, which has some nice properties compared to other stack-ful implementations.
Additionally, we have, in nightly, "generators." These are stack-less coroutines. On top of that, we have accepted (and there's a PR open in the compiler to implement) async/await. Async/await de-sugars to the whole Futures-chain shenanigans, but does it via generators. You then take that chain, and submit it to something like Tokio, making it stack-ful. Writing code with async/await is significantly more ergonomic than chaining futures together by hand, and feels more like goroutines, though suspension points are explicit rather than implicit.
Generators are likely to eventually be stable as well, so you can also write your own stack-less stuff if you want. But for now, they're an implementation detail of async/await, which has much less of a design surface area, and is what most people will want to do with this stuff, and so has priority.
I disagree. For the use case of single thread concurrency, async/await is easier to reason about, since you don't have the added abstraction of a separate execution thread (that isn't even actually a different thread). With coroutines comes the added work of deciding when to "fork", and the added work of stringing everything together with channels and mutuxes.
I myself learned to program on node.js 6 years ago, when everything required piles of callbacks (the least ergonomic form of async/await style concurrency). It's strange to see a bunch of experienced programmers saying that nobody could possibly use this style, as it is too hard, while millions of beginning programmers master it easily. In fact I remember that the people who usually had a problem with callbacks were those who had prior experience with synchronous languages.
Async/await reveals the beauty of this pattern of async, by getting rid of the syntactical cruft of promises and callbacks. It's already awesome in C# and JS, and will be landing in Rust this year.
IMO using "thread" patterns should only be done when you actually have something to gain, with real multi core parallelism. The big advantage of coroutines is that they scale seamlessly across cores. By putting the bad ergonomics of threads everywhere, you make your code ready to scale onto actual threads.
I don't read your parent as saying that async/await is bad compared to futures, but that futures code right now is tough to write. This is why so many people are psyched about async/await in Rust!
One area in which async/await significantly improves over manual futures is borrows between individual futures in the chain. In languages with a GC, this isn't an issue, but it comes up in Rust a bunch.
The user specifically asked about stackfull coroutines, which is not the direction we're going. But I think the pain point the user talked about (futures right now are difficult to deal with) will be resolved by our stackless coroutine approach, and more in line with Rust's values around "zero cost abstractions."
That's fair. This space is quite complex, with so so many options. I guess we'll know if and when the OP elaborates with more :) I very well could be wrong.
You are both right. I was specifically asking about "stackfull coroutines" as tatterdemalion says, but I stackless coroutines with await/async would be as good for code readability.
The difference between stackful and stackless is whether you can yield execution across a function that doesn't know about async/await.
The problem with stackless concerns code reuseability. With stackless you have to duplicate all your intermediate functions: once for code that calls functions or methods which can yield, and again for code that takes functions or methods which won't yield. Or you simply don't reuse code at all, bifurcating the entire ecosystem.
The stackful vs stackless effectively refers to whether the implementation uses the same stack discipline for calls that can yield vs those that cannot. In either case you're always going to have to construct some kind of stack to support nested function invocation, the question is whether you're going to duplicate all that infrastructure.
Also, it helps if you don't conflate asynchronous I/O with coroutines. Coroutines are a meta abstraction over functions (an abstraction over call chains) that can be used to create ergonomic async I/O, but have other uses, like inverting producer/consumer caller/callee relationships (e.g. converting a push parser into a pull parser with a couple of lines of wrapper code). Stackful coroutines reuse the normal call stack discipline; stackless coroutines require function annotations and compiler rewriting and lead to the code reuse problems mentioned above.
Stackful coroutines are actually a perfect fit for Rust's ownership model, and would simplify much of the work, or obviate it altogether. But for other reasons--C compatibility, poor OS constructs for minimizing memory use, and an unfortunate early conflation of coroutines with async I/O--Rust has chosen the path of stackless coroutines a la async/await as the official model.
>I myself learned to program on node.js 6 years ago, when everything required piles of callbacks (the least ergonomic form of async/await style concurrency). It's strange to see a bunch of experienced programmers saying that nobody could possibly use this style, as it is too hard, while millions of beginning programmers master it easily.
It's easy for beginners in the way GOTO spaghetti comes easy to beginners. That's all they know, after all. It however leads to bad designs and complex program flow.
let line = io::read_until(reader, b'\n', Vec::new());
let line = line.and_then(|(reader, vec)| {
if vec.len() == 0 {
Err(Error::new(ErrorKind::BrokenPipe, "broken pipe"))
} else {
Ok((reader, vec))
}
});
// Convert the bytes we read into a string, and then send that
// string to all other connected clients.
let line = line.map(|(reader, vec)| {
(reader, String::from_utf8(vec))
});
...
With monads we end up with much boilerplate that has nothing to do with the actual "business" logic.
Maybe you prefer that, but I prefer the code to describe just the business logic.
This is unrelated, but do you know if Rust's omission of exceptions is a consequence of the borrow checker, or an explicit design choice? I get that most people come to Rust from C++, but as someone coming from a higher level down to Rust, things like the complicated error handling and lack of keyword arguments bother me.
Keyword arguments tend to lead to bloated functions that try to account for every possible situation. So they're unidiomatic when compared to C-style functions which tend to always do the same thing in the same way every time you call them.
Of course, I'm assuming you're imagining an implementation where you are allowed to omit keywords to set default values. Naturally if every argument is always required, you don't have this problem.
Treating errors as values might be the best shift I've done as a programmer in later years. In fact, after learning it in rust, I took it with me back to typescript.
28 comments
[ 0.23 ms ] story [ 66.4 ms ] threadParticipants range from "I'm interested in formal methods" to "my company uses formal methods and we're starting to apply them to Rust" to "The EU has given almost two million Euro for a university grant to prove Rust code" and everywhere in between :)
Steve - can you give me a source for the latter? I'd like to read more into that, specifically.
edit: this? https://joinup.ec.europa.eu/news/eu-funds-open-source-progra...
It's already borne some wonderful fruit. See https://www.ralfj.de/blog/2017/06/09/mutexguard-sync.html for an area where we were unsound, and https://github.com/rust-lang/rust/pull/45267#issuecomment-33... for an area where we were overly restrictive, and could remove some bounds without affecting soundness.
IMHO the current “Futures” monad style solution makes the code look a bit confusing.
The language currently knows nothing of coroutines. So, as you say, you chain Futures together, which is basically a monad. You submit that futures chain to an executor, like Tokio, and it basically moves the chain's stack to the heap, and then executes it. So, in some sense, Tokio is a stack-ful co-routine. It just so happens we know the stack's exact size at compile time, which has some nice properties compared to other stack-ful implementations.
Additionally, we have, in nightly, "generators." These are stack-less coroutines. On top of that, we have accepted (and there's a PR open in the compiler to implement) async/await. Async/await de-sugars to the whole Futures-chain shenanigans, but does it via generators. You then take that chain, and submit it to something like Tokio, making it stack-ful. Writing code with async/await is significantly more ergonomic than chaining futures together by hand, and feels more like goroutines, though suspension points are explicit rather than implicit.
Generators are likely to eventually be stable as well, so you can also write your own stack-less stuff if you want. But for now, they're an implementation detail of async/await, which has much less of a design surface area, and is what most people will want to do with this stuff, and so has priority.
Make sense?
I myself learned to program on node.js 6 years ago, when everything required piles of callbacks (the least ergonomic form of async/await style concurrency). It's strange to see a bunch of experienced programmers saying that nobody could possibly use this style, as it is too hard, while millions of beginning programmers master it easily. In fact I remember that the people who usually had a problem with callbacks were those who had prior experience with synchronous languages.
Async/await reveals the beauty of this pattern of async, by getting rid of the syntactical cruft of promises and callbacks. It's already awesome in C# and JS, and will be landing in Rust this year.
IMO using "thread" patterns should only be done when you actually have something to gain, with real multi core parallelism. The big advantage of coroutines is that they scale seamlessly across cores. By putting the bad ergonomics of threads everywhere, you make your code ready to scale onto actual threads.
One area in which async/await significantly improves over manual futures is borrows between individual futures in the chain. In languages with a GC, this isn't an issue, but it comes up in Rust a bunch.
See this for a great explanation: http://aturon.github.io/2018/04/24/async-borrowing/
Thanks for your work btw.
The problem with stackless concerns code reuseability. With stackless you have to duplicate all your intermediate functions: once for code that calls functions or methods which can yield, and again for code that takes functions or methods which won't yield. Or you simply don't reuse code at all, bifurcating the entire ecosystem.
The stackful vs stackless effectively refers to whether the implementation uses the same stack discipline for calls that can yield vs those that cannot. In either case you're always going to have to construct some kind of stack to support nested function invocation, the question is whether you're going to duplicate all that infrastructure.
Also, it helps if you don't conflate asynchronous I/O with coroutines. Coroutines are a meta abstraction over functions (an abstraction over call chains) that can be used to create ergonomic async I/O, but have other uses, like inverting producer/consumer caller/callee relationships (e.g. converting a push parser into a pull parser with a couple of lines of wrapper code). Stackful coroutines reuse the normal call stack discipline; stackless coroutines require function annotations and compiler rewriting and lead to the code reuse problems mentioned above.
Stackful coroutines are actually a perfect fit for Rust's ownership model, and would simplify much of the work, or obviate it altogether. But for other reasons--C compatibility, poor OS constructs for minimizing memory use, and an unfortunate early conflation of coroutines with async I/O--Rust has chosen the path of stackless coroutines a la async/await as the official model.
It's easy for beginners in the way GOTO spaghetti comes easy to beginners. That's all they know, after all. It however leads to bad designs and complex program flow.
Maybe you prefer that, but I prefer the code to describe just the business logic.
Keyword arguments are something that we haven’t totally rejected, but many are skeptical. We’ll see!
Thanks for your answer!
Of course, I'm assuming you're imagining an implementation where you are allowed to omit keywords to set default values. Naturally if every argument is always required, you don't have this problem.
I can only recommend trying it. It's magical!