70 comments

[ 2.8 ms ] story [ 128 ms ] thread
I've often wondered why the extra scope is needed to borrow the same variable twice in a function. It seems like the borrow checker is being overly pedantic in this case by preventing users from writing a valid program. Rather than training new users to pacify the borrow checker, perhaps the borrow checker could be made to permit valid programs so as to reduce the learning curve?
This is because the borrow checker is based on lexical scope at the moment. It can (and almost assuredly will) be based on non-lexical scope in the future, but enabling that has taken years of work to refactor the compiler in a way to let it happen. We're close, but not there yet.

See also threads like this: https://internals.rust-lang.org/t/accepting-nested-method-ca...

Could you please elaborate a little on the difference here between lexical and non-lexical scope, Steve?

    {
        let mut x = 1;
        let y = &x;
        do_stuff_with(y);
        x += 1 // not allowed right now because y is still active
        // no use of y here. possible uses of x
    }
The lexical scope of y is the entire block (well, starting at the declaration of y). The actual scope of liveness ends after do_stuff_with -- y is no longer used at this point and x could be mutated. Right now you have to introduce an explicit scope block to make this work, but it's not a fundamental restriction in the language design.
The borrow checker drawbacks was actually one of my biggest pet peeves with Rust. If you guys can solve that problem in a meaningful way it'll go a long way towards making me a lot more positive about using Rust.

As it was, I felt as though I were manipulating a child in order to get it to do what I wanted.

FWIW a lot of the issues that newcomers come across are just due to lack of familiarity. The feeling of manipulating a child is common initially, but soon you'll learn how Rust expects you to design your code, and you'll be able to write code that passes the borrow checker without much issue. This is true when learning most languages; it takes some time to write code idiomatically, and till then you'll be stumbling over many things.

But yeah, we should improve this.

The learning curve of a program is relevant. Even if it just takes a weekend of fiddling around with Rust to learn how to satisfy it, it will hinder adoption.
Of course. And we're always looking to improve that.

Just pointing out that this isn't as major a problem when actually using Rust once you've picked it up.

I'd concur, in fact I don't know if I'd like to see the borrow checker get more complex because then it becomes harder to keep a mental model in your head of what it's doing.

I found once I grasped the model that the borrow checker uses that I rarely fight it and instead find it catching many things that I miss.

Yes, I've said this as well. Non-lexical lifetimes will make the borrow checker more complex; but if it tends to work closer to the way that you intuit things, as a developer, it will _feel_ simpler.

Right now, the rules are very simple to explain. But that means your code gets a little more complex at times.

> Right now, the rules are very simple to explain. But that means your code gets a little more complex at times.

And more awkward in places due to convincing the checker that what you're doing is ok.

I think a good example of this is how value categories work in C++ (even pre-move-semantics).

It's initially quite intuitive and makes sense. But as you use the language more you start questioning why it doesn't make sense and you eventually have to learn how rvalues work.

It has nothing to do with familiarity, it's due to the checker not being able to identify safe usages that are obviously safe to a human.

Atleast steveklabnik had the good sense to acknowledge the issue rather than blame it on the developer.

I'm not blaming it on the developer. I acknowledge that we should improve here.

I'm merely pointing out that this is not always an issue when programming in Rust, just mostly for newcomers, which means that if you're willing to bear with it for a while whilst learning it won't be a problem later. This isn't an ideal situation and is one we should fix. But it's not as bad as a problem that plagues you whenever you are writing Rust code.

It's not a problem in that you can make your code more awkward to work around it, but it's still a problem.
No, you misunderstand what I'm saying, I'm saying that a lot of these issues don't crop up in the first place -- because you've structured your code a different way -- not an awkward way, just different (more idiomatic) -- once you have some experience with Rust.

Some of these problems persist as awkwardness in even idiomatic code. Most go away. I write Rust code every day, and I've seen or written borrowck-induced awkward Rust code maybe twice this month.

All of this is not to say that it isn't a problem we should be fixing -- it is and we are. But it's less of an annoyance in everyday Rust programming than one initially gets the impression it will be.

> I write Rust code every day, and I've seen or written borrowck-induced awkward Rust code maybe twice this month.

Which is exactly what I was describing, why are you continuing to argue against the point?

Twice amongst the thousands of lines of code I've read this month is nothing. There are many other sources of awkwardness that trump this, much like any any other code in any other language will have awkwardness in places when you scale up to that level.
So just admit that it should be better and will be once the checker is improved, stop arguing about it. And please stop trying so hard to imply the problem is with the developer.
Are you serious? I have said that it should be improved from the get-go. Every comment of mine above says that.

I'm not implying that the problem is with the developer. I strongly feel that this is something we should fix. I strongly feel that programmers -- new or otherwise -- shouldn't have to deal with this.

I've said all this multiple times, go back and read my comments.. You're the one arguing the same point over and over.

I'm just pointing out that with some experience it ceases to be a problem. My point is simply this: This is not a problem that will plague you once you've been programming Rust for a while, so it's not one of those persistent annoyances that will stick with you no matter how much you program in a language. I'm only making that distinction. That it's not that bad. It's bad. We're fixing it. It's not the developer's fault. But neither is it a problem that will always plague you. It goes away as you start writing more idiomatic code. That should not be necessary, agreed, and we should fix that, but it is true.

I said as much in https://news.ycombinator.com/item?id=13415585

I'm starting to feel you're not commenting in good faith here. I'm not going to repeat the same thing over and over.

> I'm starting to feel you're not commenting in good faith here. I'm not going to repeat the same thing over and over.

You claim it's not the developers issue all the while claiming the problem goes away once you get more experienced with Rust (hint: that means it's a developer issue).

What you really mean is that this issue affects people's non-local code design and that people will eventually get enough experience to head off the issues before they crop up.

What I'm saying, and what others have said, is that's a problem with rust and it needs to be fixed so it stops affecting such design.

I have found myself rewriting entire functions to get rid of awkwardness due to this issue, and having the experience to write it that way the first time doesn't mean it isn't an issue, and it doesn't mean that it should be idiomatic rust.

You keep trying to push this onto the developer, even while claiming you're not doing that.

And I'm accepting that it needs to be fixed. I'm accepting that it is an issue. I've said that multiple times.

This is _not_ shifting the blame to the developer. This is accepting the blame in the right place. The blame has been accepted. The rest of the point is not about blame (which has already been attributed), it's about hope.

Having accepted the blame, I'm saying that one should not feel too discouraged by this issue because most people eventually pick up a programming style that's less prone to this.

I'm not saying developers should have to. I'm not saying it's their fault that they haven't. It's totally our fault, and we should fix it. I'm just pointing out what usually happens for most folks learning Rust, so one should be optimistic about this problem and hopefully not be too discouraged by it :)

You're interpreting "The developer can avoid this by improving X" as a way of shifting blame to the developer. That is not the intention. It is merely pointing out that the developer can avoid the problem; the ability to avoid the problem does not imply that they are to be blamed for it. It's a comment to instil optimism about the issue. The blame for the issue has already been attributed.

I don't disagree that this is a problem with Rust. I don't disagree that Rust should fix it. It's being worked on. In the meantime, I'm advocating optimism for newcomers hitting this issue. One can do both.

> You're interpreting "The developer can avoid this by improving X" as a way of shifting blame to the developer. That is not the intention.

Regardless of intention, it's a reasonable interpretation. Your initial response came across as defensive.

This particular issue was the final straw that made me decide to stop using Rust and wait. It makes the language feel sophomoric and not ready. I'm glad you guys are fixing the issue and maybe at some point I'll try Rust again but probably not before the borrow checker gets more sophisticated.

The ones who are not emotionally involved in Rust are not going to simply be hopeful that they'll eventually learn how to work around the deficiencies in the borrow checker.

Either way, we're beating a dead horse at this point so we may as well drop it.

Glad we're on the same page here now. Sorry if I came off as blaming the developer earlier.

Hope you find the time/inclination to pick up rust again! I'm sure these issues will be fixed eventually (hoping that is soon)

Usually when other languages expect something of their users, its for good reason--there's some tangible benefit in exchange. When they don't, we call it a wart and count it against the language. This seems like a wart.
It's a mixture of the two. Some of the borrowck errors are due to warts which we hope to eventually fix.

Others are due to not-immediately-obvious issues with code. One of the most common borrowck complaints I've seen is about the fact that you can't have mutable and immutable aliases operating simultaneously, even if you don't have threads involved. There's a reason behind that (http://manishearth.github.io/blog/2015/05/17/the-problem-wit...).

I also think this is an issue for Rust. The borrow checker needs to understand more, safe programs. Some are trivially safe but the borrow checker can't verify that. That is a problem. I don't even think the borrow checker needs to become super clever, but it is too rudimentary right now.
Specifics here would be very useful; sometimes you _think_ something is safe, but it's not, because you forgot something the compiler didn't.

Take http://manishearth.github.io/blog/2015/05/17/the-problem-wit... for example...

How about the example given below by Manish? The following program will never fail.

  fn main() {
      let mut x = 0;
      let y = &x;
      println!("{}", y);
      x += 1;
      println!("{}", x);
  }
It still doesn't pass the borrow checker.
I mean, yes, I'm not saying that there aren't safe programs that the borrow checker can't know about. This is an inherent aspect of how safety works; for soundness reasons, you want to be conservative.

What I do mean is, many times we have people drop by IRC and say "the borrow checker is dumb, I want to do X, and it won't let me" and the reply is something like "have you considered threading" and they go "ohhhhh I see." There was a good quote of the week about this at some point, but I can't find it right now.

I don't happen to run into patterns like the above code in real Rust code, personally. That doesn't mean that they don't happen, but I'm interested in _real world_ cases where this becomes an issue. We've got a bunch of them, which is why we're pursuing non-lexical lifetimes, but it seems like for many, this is a _constant_ struggle, and I run into a NLL situation maybe once every two months. Figuring out that disconnect is very interesting to me. It matters because I'm going to be responsible for teaching it, so real examples are way better than contrived ones. Moving that focus is the point of my current re-write of these docs :)

I'm well aware that it is unreasonable to expect the compiler to accept all safe programs, hence it needs to be conservative. It must only accept code that it can prove to be safe or that is marked as unsafe (funny, maybe the keyword here should be safe, as the user vouches for the safety of the code) by the user.

Your argument that for real world programs the borrow checkers conservatism does not create technical problems is completely plausible to me. The issue lies somewhere else.

There is a psychological component to this. People struggle with the borrow checker, or at least they feel like they did. This is negative for rust.

My point is that it would be better if it weren't trivial to write short, safe programs that are not accepted by the compiler.

Where can one expect the line to be drawn? I don't know. I worked with Isabelle/HOL to formalize mathematical proofs and there I saw that we can indeed expect a computer to be reasonably clever in abstract reasoning. I'm not saying that the rust compiler is dumb, however, there is room for improvement.

Thanks for your(plural) work on rust, btw.

Absolutely, agree 100%.

(I even tried to get unsafe renamed to "trusted"... that didn't work out.)

That's really unfortunate because I think too many people misunderstand safety in rust. The word "trust" absolutely describes safe blocks better than the word "safe" does.
The first time I wanted a NLL, I did a hash lookup, matched on the option that came back, and wanted to insert into the hash in the None match arm. Since then I think they've added convenience methods for getting or inserting. I still hope that IndexAddAssign (which would have addressed my use case) becomes an overloadable operator someday.

Recently I was parsing a line-oriented, whitespace delimited file and had to introduce a temporary ("line", below):

    let f = File::open(path)?;
    let reader = BufReader::new(f);
    for line_result in reader.lines() {
        let line = line_result?;
        let mut word_iter = line.split_whitespace();
because I think something didn't live long enough when I did line_result?.split_whitespace(). I forget exactly what the code was like before I got it past the borrow checker, though.

For me, it seems like every time I dip a toe into rust I run into borrow/NLL issues right off the bat.

Thank you! this is helpful.

Ah, see, I'd go straight for Entry in the hash case. Which might be what you're referring to.

yeah, that later one isn't NLL: it's that

  line_result?.split_whitespace()
             ^                   ^
the ? will return your String, but then split_whitespace returns &strs to that String. Since it's not bound anywhere, the String would be deallocated at the end of the line, which would make the iterator dangling. NLL won't fix that.
Thanks for following up. I found the hashmap example; it was from April 2013. I think that might have predated the Entry API. The code had ~ sigils! After failing to insert in the None arm, I tried rewriting it with find_or_insert and insert_or_update_with. Entry seems cleaner though.

Thanks for the clarification about the second case not being solved by NLL. Those examples were the ones I had in my mental "code I would like the borrow checker to accept" bucket.

Would Niko's proposals in that internals thread you linked ("Accepting nested method calls with an `&mut self` receiver") eliminate the need for the temporary in that second example?

If only manipulating my child were as easy as solving borrow checker problems. At least the borrow checker tells you what you're doing wrong.
This is something that will be done as soon as MIR - a new internal layer in the Rust compiler which is easier to operate on - is stable. Right now, it's difficult to make the borrow checker much smarter and the work would have to be thrown away and reimplemented once MIR is stable.
MIR is already in the compiler and used for everything; it's not that MIR needs to become stable (it's unclear if it ever will be), the next step is porting borrowck to be a MIR pass rather than (I believe currently) a HIR pass.
It certainly seems incongruous to require explicit scoping on usage when identifiers are reused inline. Conceptually they have the same principle where it shouldn't matter if you don't cross the streams.
This is important for soundness; it's shadowing, not replacing

  let s = String::from("hello");
  let s = &s[..];
Here, the second s refers to the first, and if it were go to out of scope right away, would be dangling.

  let s = String::from("hello");
  {
       let s = &s[..3];
       // here, s is "hell"
  }
  // here, it's the full string again
This falls out of the usual scoping rules, basically. You can think of any variable introducing a new scope where it's valid; if you don't close it yourself, it's closed at the end of the enclosing scope.
Why can't the rust compiler introduce a hidden temporary variable for the let name = jill.name().first_name() example instead of requiring the developer to explicitly declare and use one?

Edit: To clarify, I am a Rust neophyte and have only begun to scratch the surface of the language.

Edit2: Also, "There is no reason a person should know which class they are in, nor should a class know which people are enrolled in it. "

I know what the author is trying to say but when you read that at face value it comes across as a little humorous.

Please see the link to Rust's internals forum I liked below, it goes over this in great detail. (Along with ways we might fix this in the future.)
Indeed, this is the approach taken in many languages with reference counted types.

One problem is that the user would have no way of naming the thing that the lifetime is attached to. Diagnostic messages when trying to hand off 'name' to other code could be perplexing, as the delta between broken code and working code would be larger.

Given their current error messages, that doesn't seem like a problem. If you must name it, call it `name'` or something, but the output could easily be something like

    28 |     let name = jill.name().first_name();
       |                ----------- value returned here
    
    28 |     let name = jill.name().first_name();
       |                           ^^^^^^^^^^^^^ leaves scope before use,
       |                                         and is unavailable here
> Why can't the rust compiler introduce a hidden temporary variable for the let name = jill.name().first_name() example instead of requiring the developer to explicitly declare and use one?

It could, but it doesn't yet. This kind of analysis needs porting borrowck to MIR, which still has to happen. It's too complicated given the current way it works.

Though it is a matter of explicit over implicit too. If `name()` had a destructor you may want it to be explicitly tied to a scope by dint of being declared in one instead of floating around in a magical hidden variable that's tied to the scope of a different one. So there are reasons for why it may never be made to work that way.

> Though it is a matter of explicit over implicit too. If `name()` had a destructor you may want it to be explicitly tied to a scope by dint of being declared in one instead of floating around in a magical hidden variable that's tied to the scope of a different one. So there are reasons for why it may never be made to work that way.

I'm not sure I see what the problem is with this. It seems like it is very intuitive behaviour in this case and exactly what you would expect.

Rust is more explicit about moving and scopes. It's currently very easy to see how long a value will live and where the destructor will run. This somewhat complicates that.

This is similar to vvanders' and steve's comments in https://news.ycombinator.com/item?id=13413084 ; such things lead to a more complicated mental model even if on the surface it looks simpler.

> "There is no reason a person should know which class they are in, nor should a class know which people are enrolled in it. "

Also replacing a O(1) chasing a pointer into a O(n) array search

Or with a hash-table lookup, left as an exercise to the reader.
I really hate the phrase "fighting". Calling it a fight doesn't do justice to the conversations you have with the borrow checker when you use Rust every day. You don't fight with the borrow checker, because there isn't a fight to win. It's far more elegant, more precise. It's fencing; you fence with the borrow checker, with ripostes and parries and well-aimed thrusts. And sometimes, you get to the end and you realize you lose anyway because the thing you were trying to do was fundamentally wrong. And it's okay, because it's just fencing, and you're a little wiser, a little better-honed, a little more practiced for your next bout.
For a Rust practitioner, there is no fight against the borrow checker. It's like saying a Java developer fights the type system.

Is the borrow checker painful to grasp at first? A little. But the safety and assurance it gives you is immeasurable. Write enough Rust, and before long, you'll know when your code won't compile as written.

Just because something is worth the hazzle doesn't mean people do it. Of course, some people are willing to study and learn, but I think you also need to cater to those who are not if you want widespread adoption.
If I understand correctly, the borrow checker is part of how Rust prevents dangerous code. Rust is not in the business of catering to those who want to write dangerous code, even if that means Rust will not achieve widespread adoption. There are already languages is wide use that allow writing dangerous code; for Rust to become another such language would not be a step forward.
Rust does let you write dangerous code; it just makes you explicitly annotate it.

As a systems language, Rust _must_ give you total control when you need it. It's just not the default.

As I understand it, the issues that people have with the borrow checker in Rust are the same ones they'd have in another language like C++. Only there, the language doesn't help you become aware of issues, like memory leaks or other use-after-free type bugs.

If you really don't want a hassle, then maybe you want to program in a garbage-collected language instead. But then you're potentially giving up some performance and/or memory usage savings.

To be clear, while this is true in the general case, there are specific moments where it does break down, due to the conservative nature of static analysis. This thread has some good examples, but in the interest of transparency, I'll provide another:

    struct Foo {
        s1: String, // some sort of String data
        s2: &str, // a slice of said String
    }
What Rust doesn't understand here is that the String's data is on the heap, and therefore, its address is stable. So if you try to construct this, Rust won't let you, because it thinks that the borrow of s1 by s2 would be invalidated by the move. This is fixed by https://crates.io/crates/owning-ref, but it'd be nice if it understood it by default.

In general, these cases are small, but they can be annoying when you run into them. As I said downthread, I very rarely run into them personally, but some others seem to constantly. YMMV.

Is this actually safe?

What happens if you have ownership, then you can mutate the struct Foo like this:

    foo.s1 = "World"
Which destroys the original heap data. Then s2 is a dangling reference. Since references cannot implement a drop method, you need a wrapping datastructure WITH reference counting, like you said.
> then you can mutate the struct Foo like this:

You couldn't because it'd be immutably borrowed.

> It's like saying a Java developer fights the type system.

Plenty of people have fought the type system in Java. Happens all the time with Generics and Type eraser. How many projects have work around for that Scala, Flink, etc...

And it is a fight with the borrow checker because there a plenty valid and correct code fragments that the borrow checker rejects.

I've said in a few conference talks that I see it more like this: your compiler is a good friend who is telling you that you're about to do something stupid. Relying on your friends' advice is often a good idea.

And sometimes your friends are wrong because you know things they want, so you do what you were doing anyway.

Beautifully written.

I've not yet had the pleasure of using Rust, but you've described my experience of recapitulating ML-style functional programming in terms of the C# type system.

That might be true if the borrow checker were sufficiently advanced to always know better than you.

But in reality it is overly conservative and you have to expend effort just to get things to work.

When that happens it feels a lot more like fighting a war, a waste of resources that ideally would have been avoided.

Of course this may improve in the future, but right now I would leave the rose-tinted glasses in the drawer.

I really feel like I'm going lengths in order to satisfy the compiler. The compiler should work for me, not the other way around.

Why do I need to manually create a scope when the compiler can see that variable is never used again afterwards?

In Jill.name().first_name(). Why can't the compiler extend the lifetime of the return value until the end of the statement?

Having a borrow check to ensure memory safety is good. But those are just unnecessary and pointless. These are not pitfalls in user code, but pitfalls in language design

> when the compiler can see

If you look at the rest of this thread, you'll see that the compiler _can't_ currently see that; this is the whole issue.

I wasn't talking about this particular compiler. I meant "in theory, a compiler would be able to see".
That's fair; as I said elsewhere, we are very interested in making your theory a reality :)
(comment deleted)
In the example given here, how would one answer the following questions:

1. For a given school, return all the students

2. For a given school, return all the classes

In the last data model given (the correct one), the above 2 things can't be done (I think) without doing something funky like enumerating all the Enrollments and finding unique students/classes. In order to modify the data model to provide the needed information, wouldn't you run into the same borrow checker problems?