Using automated resource management to drive socket (and handle, in general) lifetime is a terrible idea. Sometimes you really, really want to close a handle at a given time. The disposal of associated resources is another matter, and automating that is fine, but leaving a socket open 5 seconds too long is simultaneously 'correct' from a garbage collection perspective and totally, totally not the right thing to do.
The key observation here is that things like closing sockets are actual information channels, so tying them to implementation details like object lifetime is a terrible idea. This, concidentally, is why some attempts to tie the lifetime of long-running tasks/threads to object lifetime are a bad idea.
There is no garbage collection in rust. There is no possible "five second gap".
Wherever you would have written "socket.close()", in rust you can just say "drop(socket)". The difference is, it takes advantage of the ownership system to statically prove that you won't try to use the socket after close.
Or just let it go out scope and it will be dropped immediately. NOT eventually garbage collected.
> Wherever you would have written "socket.close()", in rust you can just say "drop(socket)". The difference is, it takes advantage of the ownership system to statically prove that you won't try to use the socket after close.
Does a drop somehow say something to the type system as well, disallowing reads and writes afterwards? I'm new to rust, sorry if that's a stupid question.
In order to do anything to the object, you give up ownership to the function that does it (which can then give it back, if it doesn't need ownership of the value any more and doesn't want to free it itself). This even applies to the `drop` function, which just takes ownership and then does nothing whatsoever to the passed-in value, allowing it to go out of scope immediately. So once you've called `drop(foo)`, you no longer own the value and so can't give it away to any reading or writing function, and you can't drop it for the very same reason.
Freeing resources as soon as humanly (computerly?) possible will optimize your memory usage, but can actually be worse for execution speed. If you're willing to use slightly more memory, it's generally faster to free memory in coarse chunks than one-thing-at-a-time. So even though Rust can conceivably free memory sooner, it might as well defer freeing until the end of the scope, since it can prove that that's safe.
To take this to the logical extreme, see this awesome post from Walter Bright, where he doubles the speed of the D compiler by providing a custom malloc implementation that never frees, leaking all memory until the end of the process: http://www.drdobbs.com/cpp/increasing-compiler-speed-by-over...
Heaps aren't bounded in systems contexts; a process is free to gobble up the entire address space if left unchecked, which would (hopefully) result in your process being killed by the OS.
I believe we have discussed your first bullet point there, with regard to attempting to free waiting-for-the-scope-to-end-but-not-being-used memory in the event that allocation fails. But this is a hairy subject, and is beyond my expertise.
Note that your third bullet point does cause memory to be freed immediately, because the `drop` function is a scope all its own.
I understand that the drop function is simply taking ownership of the passed value so Rust knows that once "drop" finishes, the _x can be 'dropped'.
But I though that the T had to be bound to have "Drop" trait (i.e. T: Drop) so that Rust knows it's possible to insert the call to 'drop' like _x.drop()?
If dropping was really implemented that way, every type would need to implement the Drop trait. Because otherwise there would be no way to free any memory at all.
Instead, Rust knows how to free any object. Implementing Drop is optional, and you only need to do it if you want some custom behavior at destruction.
The names are a bit confusing here, and I might petition to change them.
The `drop` here is just a simple library function and isn't technically related to the `Drop` trait, which is implemented with magical compiler pixie dust and allows you to define a destructor via a `.drop` method. Anything that goes out of scope has this `.drop` method called automagically.
You're correct in that if you were calling `x.drop()` explicitly, you would need to have a `Drop` bound on `T`.
In fact, the `drop` function is totally trivial. It's just a function that takes ownership of a thing, then does nothing with it whatsoever. In other words, the function body is entirely empty! :)
I have not used Rust before; does working with statistical analysis of runtime environments hinder debugging? In a multithreaded environment - does it pause the world for its runtime analysis or are there locks on shared resources representing ownership of individual objects, or does it do copies on writes of resources for idempotent operations as the references are moved around environment for lockless operations?
To be fair, resource disposal is sometimes "delegated" to runtime. But it's always a local concern, like a reference counter for a shared resource, or a mutable `Option<R>` value that can have the R resource taken out of it or put back into it dynamically. There's no creepy overarching runtime system that tracks individual objects from a distance.
Yes, it's hard to figure out which way to talk about it. Rc<T> isn't really part of the borrow checker... it technically subverts it to do its own thing internally.
I wouldn't really consider the whole ownership thing or specifically the mechanism that inserts destructor calls at the logical end of a value's lifetime part of the borrow checker either, but I dunno how the implementation is laid out.
Just to be clear (and to check this myself -- I'm just starting to learn Rust), a five second gap is actually possible right? Say I use a socket, stop using it, don't call drop(socket), and I don't leave the scope for 5 seconds because I'm busy doing something.
What I was trying to explain is that the timing is deterministic and immediate. People are used to unpredictable GC and rightly don't want to trust it with thing like this.
I agree. Having files and sockets close automatically is a bad pattern. Ideally, in a language with linear types, the compiler would notify you of all resources that you didn't properly dispose of when they go out of scope - so, unless you return the handle or call `close(f)` at the end of the function, you'd get a compiler error. The compiler could also auto-insert `close()`/`dispose()` methods when they go out of scope, as is implemented in Rust using the Drop trait.
I don't know if Rust supports forcing explicit deallocation, but even if it doesn't, it's still an improvement compared to most GC-d languages.
If I understand the borrowing and lifetime mechanisms of rust it would be trivial to have it say anything that has a lifetime that is active at the end of a scope is a compiler error.
It would almost be like how in c++ you can disable copying by setting all the copy constructors to delete and then the compiler will tell you if your accidentally making copy's of objects.
Right. C++ used to have auto_ptr, which did roughly the same thing, but didn't quite get it right. Transferring ownership was always iffy. There were, I think, three standardized versions of auto_ptr, all bad. Now there's unique_ptr, which is supposed to be better.
The lesson of C++ is that you can't bolt on safety by using templates to paper over the underlying mess. The mold always seeps through the wallpaper. The usual problem is that at some point you need raw pointers to call some API. If you can get raw pointers, the abstraction has to have a leak.
std::vector<std::string> v;
std::string str = "example";
v.push_back(std::move(str)); // str is now valid but unspecified
str[0]; // undefined behavior: operator[](size_t n) has a precondition size() > n
There's an other fun UB because std::move doesn't check for aliasing, so self-assignment:
v = std::move(v);
is also an UB.
And of course even if it were null., pcwalton talks about use after null. Deref'ing a null pointer (to use it) is still an UB.
OP is correct. You've quoted the general language around move semantics but unique_ptr itself has a stronger gaurantee.
From 20.7.1
Additionally, u can, upon request, transfer ownership to
another unique pointer u2. Upon completion of such a
transfer, the following postconditions hold:
— u.p is equal to nullptr,
unique_ptr isn't what's unsafe here. Null pointers are. The difference, admittedly, is mostly semantic. There is nothing undefined about using a unique_ptr after moving unless you use the internal pointer in undefined ways.
> There is nothing undefined about using a unique_ptr after moving unless you use the internal pointer in undefined ways.
If you dereference (i.e. use) a moved unique pointer you get undefined behavior.
When talking about "using a pointer" most people mean dereferencing it, not for example comparing against null. It may be somewhat imprecise language, but it's what security people mean when they talk about "use after free", for example.
This is definitely outside the scope of my knowledge, but if you dereference a null pointer, won't you get an illegal memory access?
Undefined behavior, at least colloquially, is about dereferencing a pointer to memory that may or may not be accessed and may or may not work/crash-the-program. That's what makes it undefined and an absolute disaster to track down.
I'm pretty sure that if you try to deference memory address zero the OS will bark at you.
Again, it's not my area of expertise, so please correct me if I'm wrong
Dereferencing null is not guaranteed to do any particular behaviour; on many systems, getting the machine to actually dereference 0 will give a segfault/illegal memory access, but the compiler optimises assuming this never happens, and so can break a program that accidentally "relies" on it.
So in the first example, "from_file" is never used after the first call to "io::util::copy". Is Rust smart enough to realize that this is the last use of "from_file" and release it right then, or does it wait until it goes out of scope at the end of the function?
To be more specific, automatic dropping will likely always be scope-based. In the rare case where you need to drop something before its owning scope completes, you can explicitly use `drop` as people have said downstream.
Borrows are a different story. Especially with mutable borrows (which are basically a static lock on the value), the current lexical restriction is too coarse-grained. There are always workarounds, but it will be nice when it's fixed :)
Is there a specific reason for automatic dropping to be scope-based? Is it because dropping things could have side effects (e.g. closing a filehandle) and scope-based dropping makes it easy for someone looking at the code to determine when these side effects will occur?
For files python[0] does not need an explicit with to close the file. It gets closed when references reach zero.
[0] For the pedants repeating each other in the replies... Where by python, I mean the implementation that 98% of people use, called python, which the competition calls CPython. Yes it is a great feature of the implementation, like the deterministic memory management through reference counting.
This is currently true in CPython, but is not necessarily true in other implementations. It's good hygiene in python to use a `with` statement for this reason, as well as helping to enforce that you don't accidentally hold on to a reference, and keep the file open.
I think that context managers in python are generally a very nice abstraction over these sorts of concepts, and it's cool to see how Rust is handling these problems very cleanly.
What happens if there's a cycle? It will get closed when the cycle is finally collected.
You can rely on a GC (and refcounting + a cycle collector is absolutely a GC) to finalize your resources for you, but it is quite difficult to reason about when the file will eventually be closed.
Also, the reference counting semantics of CPython are implementation-specific. From the PEP that introduced `with`:
> Note that we're not guaranteeing that the finally-clause is executed immediately after the generator object becomes unused, even though this is how it will work in CPython. This is similar to auto-closing files: while a reference-counting implementation like CPython deallocates an object as soon as the last reference to it goes away, implementations that use other GC algorithms do not make the same guarantee. This applies to Jython, IronPython, and probably to Python running on Parrot.
Still, do you want to bake assumptions based on CPython's implementation into your codebase, then have those assumptions invalidated when you later switch to PyPy to take advantage of the JIT compiler?
If the garbage collector were turned off and a cycle were to be present, the file would simply never be closed. Though I suppose that would, indeed, be deterministic.
Props to Rust for tackling this head on, more languages should provide resource allocation that's deterministic, predictable and syntactically convenient.
CPython's behavior is nice, but it seems to me it came about by accident. Big heavy resources use refcounting because everything uses refcounting. Plus, if CPython had true concurrency, across-the-board refcounting probably wouldn't have lasted nearly as long (reason being: multithreaded refcounting requires atomic ops which make them much more expensive).
But in the general language design world, IMHO, refcounting is actually a pretty good compromise for resource management. Because:
1) The efficiency lost compared to Rust's approach is probably immaterial (since the Big Expensive Object that you're mananaging dwarfs the refcounting -- syscalls and file descriptors are expensive). So I don't think zero overhead is critical.
2) Cycles are pretty easy to avoid when you're only refcounting Big Expensive Objects (FDs, mmapped buffers, etc. etc.) For a general runtime, refcounting is tricky. But if it's a special part of the language that's small, simple, and only for resource management, it's pretty convenient. I think it's worlds better than the approach of mixing GC and resource management -- language designers should admit that's a terrible idea.
I agree that refcounting is a great and usable compromise in this space, but your own comment flirts with why I don't think modern languages are chomping at the bit to base resource management on it: concurrency.
Granted, I have no idea how much overhead is imposed by atomic operations vs. a stop-the-world or concurrent GC (if anyone has some data, I'd love to see it!). But given how it's become de rigeur for new languages to come with a baked-in concurrency story and emphasize concurrent applications, I don't blame them for not wanting to tie themselves to the RC cart.
"Granted, I have no idea how much overhead is imposed by atomic operations vs. a stop-the-world or concurrent GC."
An atomic read/write/increment/decrement is a non-blocking (from the point of the view of a user code running on the CPU) CPU operation. The CAS operations required to do atomic inc/dec are relatively expensive compared to normal memory read/writes of course but nowhere near the scale of stop-the-world GC (where all threads in the runtime have to be blocked for a potentially large amount of time, sometimes measured in whole seconds or even 10's of seconds for large heaps).
Anyone who has had to troubleshoot JVM's handling large heaps (in the ~10GB range) quickly learns to love deterministic costs of refcounted resource management.
Hah.. Implementation issue. That's a good one. It's just like an SUV not being landmine proof is an implementation issue. Just because somewhere a landmine proof humvee exists doesnt mean its attainable or practical for an ordinary user.
Hit one of my hot-buttons: Threads have got to be one of the most massive WOMBAT boondoggles of the last few decades.
Pro) none of this slow inter-process-communication stuff!
Con) EVERYTHING is now dangerous. (not quite, but close)
But the industry (?) decided "TOTALLY worth it!"
I forgot to mention that threads start a little faster than processes (well, quite a bit faster on a particular shitty OS in widespread use), but not so much faster that you won't need to pool and re-use the threads anyway. I think this is one area where Erlang (actor pattern), for instance, gets it right: pretend tasks run in their own process, and use an IPC proxy to send messages between the tasks/sub-programs, even if implemented with threads instead of OS level processes.
Why? Wouldn't any garbage collector collect the file object when it goes out of scope?
def foolish():
f = open('filename.txt')
Once `foolish` returns, `f` goes out of scope, refcounting or not. Once `f` goes out of scope, its destructor closes the file. In CPython, the destructor is called when the refcount reaches zero. In Pypy, the destructor is called when there are no more references to `f`. Both happen when `foolish` returns, in both Pypy and CPython.
This is an implementation detail of file, not CPython.
Or did I fundamentally misunderstand something here?
Yes, you misunderstood how tracing garbage collection works. Cleanup of data allocated on the heap occurs during tracing phases, not when a reference goes out of scope. Otherwise, imagine if before it returned f copied the object into some global structure g--how would the garbage collector know it was safe to close?
(Some garbage collected languages can do automated "escape analysis" to figure out whether an object "escapes" the scope, and if it doesn't place it on the stack instead of the heap and deterministically free it when it goes out of scope. But that's surprisingly limited in real world programs and is generally speaking an implementation / optimization detail, not a core guarantee of the language).
The garbage collector would still close the file eventually, right? Unless of course it was copied somewhere else in the meantime, but in the code example, that is clearly not the case.
Eventually, yes. But if you're relying on the file being deterministically closed (e.g. if you're using the POSIX file locking mechanism) in CPython, it won't necessarily be correct in non-reference counting garbage-collected languages, and the same is true if you're relying on any other sort of deterministic resource cleanup (e.g. in memory-constrained environments).
Anyone has good learning resources about rust? I find it's a very interesting language but the document on its web site is so limited. Many topics are not covered or explained in vague terms. The new guide is a good starting point but still needs some improvement.
For newcomers I think the documentation is quite good. But where I think there is currently a big problem it is for intermediate / advanced level documentations, it is very limited. Sometimes you'll read some code and say to yourself "I didn't know I could do that, cool". In order to grasp more knowledge you'll have to read RFC, commit logs, issues, and source codes, but because it certainly is not your main job it's really time consuming.
Exactly! I think there's lack of document which quickly goes through basics and spends more time on rust-specific features. (Or I'm too lazy to find them?) For developers programming for decades, we do not need to repeat too much about integers, characters, etc.
I personally like to use slides to learn things. I'm not sure if this is typical, but slides are usually concise and easy to read.
1. Familiarity to C++ programmers, for whom taking a reference via `&` is a familiar operation.
2. There are two different kinds of references: `&` is immutable and `&mut` is mutable. One way or the other you'd need some sort of differentiation.
3. Borrowing, though crucial to the idea of Rust, is still less fundamental than ownership. You can have a language with ownership and without borrowing (indeed, this is how most prior languages with ownership work). But you can't have borrowing without tracking ownership.
Also, there are a whole host of very common types (integers, etc.) that actually want to be passed by value and copied.
There's no point in making a pointer to an integer and then dereferencing the pointer in the callee when an integer fits into the same amount of space as a pointer in the first place.
Very simple types (without heap pointers or destructors) can be passed by value and efficiently copied. You end up with a bit of an intuition for "value types" (simple types whose identity is bound up in the shallow memory that they contain).
In other GC'ed languages, there is a fixed set of these kinds of values (often called "primitives"). In Rust, any simple user-specified type can be a "primitive".
If something is immutable (and doesn't own something else), it doesn't matter whether it's passed by value or reference. That's a decision the compiler can, and should, make, as an optimization decision. In Modula, where pass-as-read-only was the default, small items were passed by value and large ones by reference. "Small" is a CPU-dependent optimization and should be left to the compiler, not "user intuition".
The default should be pass as read-only reference. If you want a copy of something, you should have to explicitly make one. If you want write access, you should have to say so. Rust has all the right concepts, but the defaults aren't in the safest direction. The experience of "const" in C/C++ teaches us that they should be.
There is an important distinction in Rust: ownership.
Copyable objects are (by definition) simple enough that copying them does not introduce aliasing. Objects with heap pointers or destructors cannot be copied, because now there would be two "owners" of those values.
For what it's worth, I've found that a lot of these assumptions ("it doesn't matter, let the compile optimize it") really remove the programmer from important power.
And before you find yourself saying "are you saying everyone should write everything in assembler?":
1. There is a distinction between writing code in assembler and having to use a `&` sigil for "by-reference" in a language that has lambdas, traits and many other high-level features.
2. You should generally use Rust for cases where performance characteristics are important, and where control over memory is important. If you're sure it's not important, by all means go with a higher-level language that abstracts it away.
Doesn't #1 matter less in Rust? C++ has to deal with backward compatibility in syntax, Rust doesn't.
Based on this article, immutable references are the most common kind of declarations. It seems to me that rust code would look much cleaner if they had the shortest syntax (including in pattern matching etc)
Rust's primary intended audience is C++ programmers, and familiarity is a powerful asset. It's not the sole determinant of syntax concerns, but it's an important consideration.
Well, really the default is passing trivial implicitly-copyable values like integers. You probably don't want to write a & every time you pass an i32 to a function, so that sort of thing gets the default just-passing-bytes-around syntax from other C-like languages.
For resources or generally things that require cleaning up when they disappear, that just extends into the transfer, logically a shallow copy with the original variable flagged as dead in the type system.
The & symbol denotes a reference type, and it's an explicit thing in the type system and not just an annotation as part of the function call process. It can show up anywhere you'd write a type, things like `Option<&T>` or other types with `&T` fields aren't too uncommon, and `&&T` is also a type.
I haven't really thought it through, I guess, but modifying a type `T` with a `&` to make it a pointer type `&T` seems to compose a lot better than making all types `T` implicitly pointers and requiring `&T` whenever you want the value inline and not behind a pointer indirection, whether in a local variable or as a field in a struct.
Awesome post. Best explanation I've seen so far how Rust's resource management. Always impressed with Yehuda's ability to explain complex techniques and abstractions clearly.
Having the compiler eliminate resource leaks is really, really handy. This is one thing that's very easy to get wrong in go. Will be interesting to see how much of a cost the upfront typing has during quick prototyping or explorative programming, but its great to see this kind of type system make it into something that's well on track to becoming a mainstream programming language.
"Rust achieves both of these features without runtime costs (garbage collection or reference counting), and without sacrificing safety."
If it is freeing everything at the scope boundary it is incurring a huge runtime cost relative to garbage collected languages. GC scales with the number of live objects during GC while this scheme scales with the number of allocated objects. It is far more predictable than GC but I don't believe that it isn't more expensive.
It sounds like they are talking about a semi-space collector, which I can imagine might in theory be faster than scoped allocation since allocation is a simple pointer bump (just like stack allocation) but when garbage collecting you only have to traverse the live set. However, semi-space collectors can't collect non-memory resources without additional metadata, and they have a memory overhead of at least 2x the maximum live set. I have a hard time imagining even a semispace collector outperforming stack allocation.
Of course if your program is short-lived you can just allocate a sufficiently large heap, pointer bump the whole way and free all non-memory resources on program termination, which ought to be cheaper (but probably not that much in practice due to cache effects).
In Rust, you often use stack allocation, which, as you pointed out, is much more efficient. There's some heap allocation in Rust, but in my experience it's actually a minority of all allocations.
Rust also has an Arena library (http://doc.rust-lang.org/0.11.0/arena/index.html), which can be used for cases where you know that you have enough memory for a given operation and are willing to throw out the entire thing when the operation (or program ;) ) is done.
Not necessarily the same. For example for arena allocation, you can drop lots of related objects at the same time. If you don't have to run destructors on them, the cost will be much lower than any other way of managing them.
> It is far more predictable than GC but I don't believe that it isn't more expensive.
A GC has to determine liveness of objects, so it has to traverse the object graph (or at least the most recent generation if the GC is generational) to know which objects are still alive and which aren't. That's what "GC scaling" refers to. Then it still has to deallocate dead objects, and pay the same deallocation cost as scope-bound free.
All GCs have to deallocate objects. It may be very cheap to do so, for example when you have a semispace collector and no finalizers, but switching the spaces is still a form of deallocation. Production-quality GCs don't use semispace for everything because of the huge memory use overhead, and all practical GC'd languages I know of need finalizers in some form.
Finalizers are certainly the exception rather than the rule for objects. Deallocation to me represents some action taken for a particular object and scales with the number of objects deallocated. Semi-space copying collectors do not have this property. I think we only disagree on the semantics of "deallocation" — which is why I specified my argument in terms of how things scale. Similarly compacting collectors like Java's CMS also have this scaling property without semi-spaces.
But it has to scan/mark all the objects, not just the the unused ones. This has large overhead in big-mem apps. Not to mention that usually this forces the whole app into RAM (makes swap useless) and also potentialy drains cpu caches.
And to amortize this it often has more memory overhead (up to 2x) than typical fragmentation in a manually managed app.
Good generational GC doesn't need to deallocate objects one by one, instead of that it can deallocate objects in bulk (if there is no finalizers). It can allocate memory faster because it doesn't need to use memory allocator to create every object, it just increment pointer that points to the beginning of the free space. It's true that modern GC has greater throughput than simple allocation/deallocation schemes. Problem is predictability (as been mentioned).
I use and love Rust, but I must admit this is true. GC has some advantages over the current options. Fear not however as post 1.0, optional GC will be most likely added to Rust. See https://github.com/rust-lang/rfcs/pull/244.
In general, idiomatic Rust puts most objects that would be in a semispace nursery on the stack instead, which has all the advantages of a semispace collector, plus improved locality and the lack of a mark phase. The lifetime system means that the traditional limitations of escape analysis (higher order functions, separately compiled functions, virtual functions, returning objects on the stack) largely don't apply.
If you can get all the objects for a full request on the stack, that would be pretty great. Is that true for the current HTTP servers / frameworks written in Rust?
I don't do a lot of server Web development in Rust, but I see no reason why it wouldn't be possible, at least for common cases with small vector optimization.
I agree that automatic, deterministic resource management (as opposed to GC plus a dispose pattern of some kind) is highly desirable. But I think for the vast majority of projects, I would prefer automatic reference counting with the occasional weak reference to break cycles, particularly if that were the default for the language. I can appreciate that being explicit about resource ownership is an advantage in situations where one needs to absolutely minimize performance overhead. But for most applications, I think a system of ownership and borrowing like Rust's would just add extra cognitive load for the programmer for no appreciable gain. Of course, Rust isn't a language for most applications AFAIK; it's a systems language for performance-critical infrastructure like Servo. And Rust is probably not a language for blub programmers. But I think this blog post is absolutely right about the problems with GC plus a system for disposing of resources. It sucks that mainstream, blub-friendly language have mostly chosen GC over reference counting; I think automatic reference counting provides the right set of trade-offs for most applications.
I know Rust has RC pointers. But I'm guessing a Rust programmer would still have to understand ownership and borrowing, since the standard library and third-party libraries use those concepts.
I think what I'd really like is something like C# and the .NET base class library, but AOT compiled to native code and using automatic reference counting, with some way to indicate weak references. I know that neither GC nor reference counting quite reaches the level of "don't make me think" when it comes to resource management, but given a choice between explicit resource disposal and the occasional weak reference annotation, I think I'd choose the latter.
That's true, you can't totally avoid knowing about these things. But if you don't want to think about them in your code, and are okay with the price, you can.
It sounds like Swift may be close to what you want, yeah?
First there will be need of a standard library. For the moment to do anything sensible you are importing Apple's Foundation Library (probably in one year time it will be better).
Then I see no problem to have it server-side (both as a scripting or compiled language).
I agree that Rust isn't the be-all, end-all language. But I think this line is mistaken:
> I think a system of ownership and borrowing like Rust's
> would just add extra cognitive load for the programmer
> for no appreciable gain.
In my experience, the fact that the compiler can check this stuff ends up subtracting cognitive load. Of any language I've used, Rust code is the code that I worry least about.
Is there syntactic overhead? Yep. Does it impose constraints on design? Yep. But when it comes to resource management, everything that Rust does is just something that you'd need to use a pen and paper to keep track of in other languages (including pervasively reference-counted languages, since AFAIK there's no way to automatically enforce the proper use of weak pointers).
> In my experience, the fact that the compiler can check this stuff ends up subtracting cognitive load. Of any language I've used, Rust code is the code that I worry least about.
That's interesting. So this holds for languages and applications where you could get away with not knowing the automatic memory management implementation and adjusting your design based on that? For example, that you don't have to worry yourself with making sure to not allocate too many values on the heap, at least in most of the code?
Would you prefer using Rust for problems and domains where you strictly don't need the efficiency and determinism (like memory usage) you get from using Rust?
> Would you prefer using Rust for problems and domains
> where you strictly don't need the efficiency and
> determinism (like memory usage) you get from using Rust?
It would highly depend on circumstance. How often is my code going to be run? How long will it be maintained for, and how often will it be updated, and by whom?
I think we all have a hierarchy of sorts: bash or Perl for one-off, disposable scripting tasks; Python or some other light dynamic language for when things start getting more serious, but are still personal projects; C# or some other relatively heavyweight hammer for when we're writing code that will live beyond our control, maintained by others and run for years and years.
I don't write production code in Rust yet (heavens no, not until 1.0 at least... godspeed to wycats :P ) but right now it's in that third category of languages where I trust that it can be maintained by a team and trusted to exist for years. However, I would consider Rust for that second category of somewhat-serious personal projects, because I know too well how such "personal projects" can vault unexpectedly into that third category of "mission-critical team projects". But before that, I'd need to wait for the library ecosystem to mature.
EDIT: I should clarify too that resource management is only part of the reason why I don't worry about Rust code. There's also stuff like the ability to know that a given global variable is immutable, and can't be changed out from under me. Or knowing that an innocent-looking block of code can't unexpectedly kill the process (divide-by-zero notwithstanding).
In garbage collected languages with macros and an "unwind protect" operator, we avoid explicitly closing files and other resources using with scoped binding constructs implemented by macros.
(with-locked mutex
;; ... critical section
)
;; mutex is released here
Good question. I wonder if there was some Lisp with linear logic that would allow lexical and ~borrowed (sorry if I misuse the term) resources management.
It shouldn't be very hard to create a basic ownership system. Here is a version I just wrote, which works by defining a new version of "defun" that uses dynamic variables to keep track of ownership:
(defvar *transferring* '() "Objects which are going to be transferred from one owner to another.")
(defvar *resources* '() "The resources which need to be freed when the current procedure is done.")
(defmacro new-defun (name args &body body)
"Define a procedure which will have ownership over the resources it creates using any of the 'new' procedures.
`(defun ,name ,args
(let ((*resources* *transferring*))
(setf *transferring* '())
(unwind-protect (progn ,@body)
(mapcar #'free *resources*)
(mapcar #'free *transferred*))))) ; These are resources which were supposed to be transferred by weren't due to an error.
(defun transfer (resource)
"Transfer ownership of a resource from the current one to the next procedure called which can accept ownership."
(push resource *transferring*)
(setf *resources* (remove resource *resources*))
resource)
(defun new-open (&rest args)
"Open a file whose owner is the first procedure up the stack which can accept ownership."
(let ((file (apply #'open args)))
(push file *resources*)
file))
All that is needed now is a method "free" which will free any given resource. With this, any resource allocated with a "new" procedure (the only one I wrote was "new-open") will by put under ownership of the last procedure on the stack which was defined with "new-defun". Transfer between owners can take place by using "transfer". The best part is that more facilities are easy to build on top of this. The "drop" procedure would be trivial to add. The only problem I can see here is that there may be some issues if "free" throws an error, since "unwind-protect" doesn't protect the cleanup-forms. This should be fixable by just wrapping every call to "free" with "ignore-errors".
The 'with' idioms I've seen in the past do, which, combined with yield operators, can make for some unexpected 'fun' at runtime, but it remains extremely convenient most of the time.
Right. One case in which I've come across this problem in Python is writing unit tests for code using Twisted (and Trial, Twisted's unit test framework) with @inlineCallbacks, and trying to also use this style of resource management for mocks (or, essentially the same, the @patch decorator to set up the mocks).
Then you run into the problem that @patch has cleaned up all of your mocks by the time the yield returns.
Now, I haven't tried to do anything equivalent in Rust; I'm not sure what the idiomatic equivalent would be. But having more explicit control of lifetime, rather than it always depending specifically on that stack frame, seems like it might help here.
Yes, I know it is working for quite some time, I was more wondering if there are some hidden inherent problems lurking below the surface or if the theory behind rust is sound, e.g. some corner cases where it breaks down or result in undefined behaviour.
This doesn't mean that Rust is absolutely perfect. There's tons of stuff that is awkward to express, and tons of more features that would be nice to have. But the core all works, and works well.
Great post. I think it's the first I've ever read about Rust clearly clarifies the owner of a ownership is scope. It may sound ridiculous but all other talking about ownership without mentioning who is exactly the owner.
Slightly off topic, how do I go about doing any meaningful GUI programming in Rust? Do Qt bindings exist for Rust? How easy or difficult is it to create such bindings? It's a great language but if it doesn't have a bridge of some sort to the existing legacy code (C++ included), then it's no good to me.
As someone who's been using (and really liking) rust for about 6 months, there are two things in this article which are, I'd say, 'common misconceptions' about rust:
One of the coolest features of Rust is how it automatically manages
resources for you, while still guaranteeing both safety
(no segfaults) and high performance.
In rust there are two types of code. 'safe' and 'unsafe'. The above is only true if your entire code path contains only safe code.
NOT, if you only write safe code.
That is, if you write entirely 100% safe code, and use some library that uses 'unsafe' code at a low level (this is very common; for example, an C binding or memory optimization), then rust does NOT guarantee safety.
Furthermore, there is no way to know if your code path includes unsafe code.
Rust is relatively safe; it's just not totally safe. Using rust does a reasonable job of protecting you from segfaults most of the time; but it's not a silver bullet.
As soon as the program stops using the resource, its cleanup logic gets
invoked.
...
Because only one scope owns an object at a time, you can tell just by
looking at it which objects will be destroyed when it's done executing.
This isn't always necessarily true.
Rust uses 'drop flags', which are extra bytes at the end of struct instances in memroy, to track objects and keep meta information about them to determine if they should be dropped.
At the end of scope when the drop checker runs it checks against these drop flags to determine if an object should be dropped or not.
(For example, if a closure captures a variable it may no be dropped at the end of the scope it would normally have been dropped in).
There seems to be this myth that resource deallocation in rust is magically determined at compile time and the has 'zero cost'; it's simply not true.
There is a runtime cost to the drop checker; it's not particularly large, but it's certainly not just a sequence of drop operations.
Don't get me wrong; I really enjoy rust, and I like it a lot. ...but those two ideas pop up repeated in discussions about it, and they're not exactly correct.
> In rust there are two types of code. 'safe' and 'unsafe'. The above is only true if your entire code path contains only safe code.
I think this is actually a misconception. In every high-level language (Ruby, Python, etc.) there are APIs for interacting with low-level code. These APIs (C bindings and FFI) are unsafe, and are in the path of virtually all high-level code. It is the responsibility of somebody implementing a C binding for Ruby to ensure that the publicly exposed API is safe.
If you use `nokogiri` in Ruby and get a segfault, that is a bug in nokogiri. It is not an indication that Ruby is unsafe. Similarly, the "safe" dialect of Rust is Rust. Rust provides a low-level binding to unsafe code via the unsafe dialect that is philosophically equivalent to the FFI API or C bindings in Ruby. If somebody uses the low-level binding and exposes a Rust library that segfaults, there is a bug in that binding, full stop.
I think this is very important. If you use regular Rust code with the expectation that "anyone in your path may cause you to segfault", something has gone very, very awry in the ecosystem. This is not an expectation in other high-level languages when people write low-level libraries with unsafe code in them, and it should not be an expectation in Rust.
> Rust uses 'drop flags', which are extra bytes at the end of struct instances in memroy, to track objects and keep meta information about them to determine if they should be dropped
This is in the process of being eliminated (see https://github.com/rust-lang/rfcs/pull/320). I didn't mention it because it's runtime overhead that will not be present shortly, and certainly by the time 1.0 is reached later this year.
> If you use `nokogiri` in Ruby and get a segfault, that is a bug in nokogiri. It is not an indication that Ruby is unsafe.
Rust has a strict definition of safety; not all rust code for fills that guarantee.
We're not talking fast and loose here. These are absolute definitions.
You can write safe code that crashes in rust. That absolutely violates the safety guarantee whatever the reason for it.
> This is in the process of being eliminated
No it's not.
Keep dynamic drop semantics, by having each function maintain a (potentially empty)
set of auto-injected boolean flags for the drop obligations for the function that need
to be tracked dynamically (which we will call "dynamic drop obligations").
What's happening is some of the run time overhead is being reduced.
> You can write safe code that crashes in rust. That absolutely violates the safety guarantee whatever the reason for it.
You can write Ruby code that calls into unsafe C code and SEGVs. This is a bug in the C code you called. It should have exposed a safe interface.
You can write Rust code that calls into clearly delineated unsafe code and SEGVs. This is a bug in the unsafe code you called. It should have exposed a safe interface.
Anything to the contrary is a misunderstanding of the purpose and role of unsafe code in Rust, and endangers the entire ecosystem, just as it would endanger the Ruby ecosystem if it were seen as normal for C extensions to SEGV the safe Ruby process.
'Unsafe' is a bad term, effect is much better and it is most definitely a rust issue. If the language doesn't have a way of dealing with effects it won't be able to handle them. Rust is honestly a more safe c++ and in my opinion not an alternative for c.
I'm just saying there are these two misconceptions:
- if you write safe rust code, your code can't segfault
This isn't true if your code path contains any unsafe code.
- Rust code can't segfault
This isn't true at all.
Sure, if it does segfault its a bug in the c code / unsafe code.
You're absolutely correct.
...but bugs aren't deliberate. Obviously no one is going to deliberately expose a c api that has segfaults in it.
The misconception is that if you're using rust you're safe from the bugs that other programmers may have made... which is not true.
There's a trust chain involved; if you depend on a library you trust it's bug free and won't delete your harddisk and crash your program, but there's a difference between trusting a library is not malicious, and trusting a library is bug free.
If you have to trust the library is bug free, you're really no better off than using C++ are you?
> The misconception is that if you're using rust you're safe from the bugs that other programmers may have made... which is not true.
Of course it's not true, but the libraries you're using will (hopefully, for the most part) be written in Rust, which increases the level of trust.
> There's a trust chain involved; if you depend on a library you trust it's bug free and won't delete your harddisk and crash your program, but there's a difference between trusting a library is not malicious, and trusting a library is bug free.
> If you have to trust the library is bug free, you're really no better off than using C++ are you?
The libraries are written in Rust, so have all the benefits of that. It is nonsense to say that "libraries can have bugs so you might as well use C++".
> It is nonsense to say that "libraries can have bugs so you might as well use C++".
My argument is that if you assert 'rust is 100% safe' then you must assert, all your dependencies are bug free.
If you can somehow magically assert 'all my code is bug free', that would be great... but if so, you wouldn't need the borrow checker at all would you?
(I'm certainly not endorsing using C++ :P I'm specifically talking to the assertion that rust code can be safe, as per the definition of safety in the guide, that certain operations are impossible; certainly, and absolutely, rust code is safer and more trust worthy (imo) than c/c++ code... but you've got to be reasonable about what you're arguing. 'rust is magical and can never crash' turns up in blog posts a lot)
> If you have to trust the library is bug free, you're
> really no better off than using C++ are you?
This is a pretty far-out argument. For any programming language and environment, unless every dependency of your program including the compiler and runtime can be proven correct via formal verification, you will be susceptible to bugs. That doesn't imply that all code in the world is topologically as unsafe as C++. I can segfault Python if I try hard enough, and that's a bug in Python. Likewise I can segfault safe Rust code if I try hard enough, and that's a bug in Rust. I don't believe that anyone is really of the misconception that the mere use of Rust absolves you from the bugs of others, because that same person would have to believe that Rust somehow makes all bugs impossible, including bugs in C code that Rust calls into.
So let me give everyone a source to cite on Wikipedia:
"Rust does not keep C code from segfaulting." ~ kibwen
I don't understand -1s to the parent post; it's important to remember and be aware of limitations and dangerous areas in a language (or any other tool). And shadowmint tries to describe them in very civilized and calmly assertive way. I find his posts, as well as the counterarguments (like those from pcwalton and wycats), massively educative and informative. I absolutely can't understand why someone would like to see them fade out and disappear. I see the thread as exactly a kind of educated, professional argument that is so valuable on HN. And in such, being partially wrong or incorrect in some aspect is not something "bad" and "requiring censorship", but rather opportunity for straightening by others with a side effect of educating the larger community, isn't it?
> What's happening is some of the run time overhead is being reduced.
I believe wycats was responding specifically to
> Rust uses 'drop flags', which are extra bytes at the end of struct instances in memroy, to track objects and keep meta information about them to determine if they should be dropped.
And these are being eliminated. That is the drop flag will be entirely separate to the struct instance, and will only be injected into stack frames where there is a conditional branch in which a value is moved out in one, but not the other, e.g.
let x = some_value;
if condition { drop(x); } else { /* do nothing */ }
then `x` will get a drop flag as a single bit (or byte) on the stack. If the code is changed to
if condition { drop(x); } else { drop(x) }
there will be no drop flag, and no overhead.
Furthermore, placing the value on to the heap, e.g. `box some_value`, will not have an associated drop flag for `some_value`, that is, eliminating the byte that is currently stored with it.
The behaviour is determined at compile time, with runtime cost only introduced if it is absolutely needed (when one branch drops, and another doesn't), and, even then, it can be avoided, by manually forcing a drop of the value in every branch.
> That is, if you write entirely 100% safe code, and use some library that uses 'unsafe' code at a low level (this is very common; for example, an C binding or memory optimization), then rust does NOT guarantee safety.
What do you consider a "guarantee"? Do you consider pure JavaScript to "guarantee" safety, even though there have been exploitable bugs in the JIT and interpreter? Do you consider pure-computation, non-allocating Rust to be safe, even though LLVM may contain codegen bugs that allow for undefined behavior? Do you consider seccomp-isolated code to be safe, even though there have been, and will likely be in the future, kernel bugs that allow sandboxed process to escape?
The point is that instead of talking about "100% safe", which is never the case in the real world, I think it's better to talk about trusted computing bases. That is, instead of saying "X is safe" or "X is unsafe", we talk about what effect X has on the memory-unsafe surface area of a program. Rust's goal is to reduce the trusted computing base of an application written in it to the hardware, compiler, unsafe blocks, and type system. This gives us essentially the safety properties of "managed languages" (except I don't like that term because it tends to imply garbage collection and lack of fine-grained control over memory, which don't apply to Rust).
A specific example I like to give: Sure, we could hardwire (for instance) Vec into the language, and therefore reduce the number of lines of unsafe code in the libraries. But doing so would just be moving the unsafe code from the unsafe blocks in the libraries to the compiler itself. There would be no net gain in safety from it—compiler code can have bugs just as library code can—and there would be a decrease in maintainability and flexibility.
You can think of Rust's safety properties as establishing a type of sandbox if you'd like. All sandboxes have trusted computing bases, and Rust's compile-time sandbox is no exception. But "safety" as applied to a sandbox still has an important meaning. Chromium/Firefox OS's sandbox, for example, is a safe sandbox, even though its security depends on strong assumptions about the trustworthiness of the OS kernel and the IPC layer between the trusted and untrusted processes.
> There is a runtime cost to the drop checker; it's not particularly large, but it's certainly not just a sequence of drop operations.
After the drop optimizations are done, this will only be true if you conditionally move an object on one or more branches. It's already true today if the object gets SROA'd on the stack, as LLVM can then optimize out the drop flags. We've talked about adding a lint so the compiler can optionally warn and allow you to fix it if this performance cost is a concern to you. (It's a very minor cost—one test and branch on a stack byte, and again only if the object is conditionally moved.)
Note that C++ move semantics incurs essentially the same cost.
"We've talked about adding a lint so the compiler can optionally warn and allow you to fix it if this performance cost is a concern to you. (It's a very minor cost—one test and branch on a stack byte, and again only if the object is conditionally moved.)"
Might I just make a meta comment that while that may not be a huge use case, the ability to assert that some particular optimization is firing and get warnings if I've done something to break it is something I've wanted out of a language for a long time. I know it's easier said than done, but I'd love it.
There's a couple bugs in the sample code: `is_thirties` won't compile because it's missing a type declaration, and the `person` function should be called `first_name`.
I love Rust, but really can't use it at work until it has a fully-featured AWS API. I had similar issues trying to use D and Dart in the past. Go doesn't have an official SDK either and although goamz isn't complete, the situation there is much better.
Maybe in a parallel universe where I have time for these kind of things... It shouldn't be too hard to reverse-engineer botocore, use the these JSONs here [1], and generate code based on them.
In Rust, unlike in garbage collected languages, you never1 explicitly close or release resources like files, sockets and locks
GC languages is trying to emulate machine with infinite memory so you don't need to bother yourself with _memory_ management. It's not compatible with rust's resource management system that manages resources and treats memory as another type of resource.
The same is true for C++ BTW, in good C++ code you never release resources by hand and use ownership for this (RAII).
164 comments
[ 3.6 ms ] story [ 201 ms ] threadThe key observation here is that things like closing sockets are actual information channels, so tying them to implementation details like object lifetime is a terrible idea. This, concidentally, is why some attempts to tie the lifetime of long-running tasks/threads to object lifetime are a bad idea.
Wherever you would have written "socket.close()", in rust you can just say "drop(socket)". The difference is, it takes advantage of the ownership system to statically prove that you won't try to use the socket after close.
Or just let it go out scope and it will be dropped immediately. NOT eventually garbage collected.
Does a drop somehow say something to the type system as well, disallowing reads and writes afterwards? I'm new to rust, sorry if that's a stupid question.
Yes.
1. Give ownership away to something else.
2. Free the object at the end of the scope.
In order to do anything to the object, you give up ownership to the function that does it (which can then give it back, if it doesn't need ownership of the value any more and doesn't want to free it itself). This even applies to the `drop` function, which just takes ownership and then does nothing whatsoever to the passed-in value, allowing it to go out of scope immediately. So once you've called `drop(foo)`, you no longer own the value and so can't give it away to any reading or writing function, and you can't drop it for the very same reason.
Is there a trade-off involved here?
To take this to the logical extreme, see this awesome post from Walter Bright, where he doubles the speed of the D compiler by providing a custom malloc implementation that never frees, leaking all memory until the end of the process: http://www.drdobbs.com/cpp/increasing-compiler-speed-by-over...
I think one middle-path could be to mark the resource as released as soon as compiler detects it, and then release it
1. when the allocator is starved or
2. when the resource goes out of scope
3. when drop() is explicitly called
I believe we have discussed your first bullet point there, with regard to attempting to free waiting-for-the-scope-to-end-but-not-being-used memory in the event that allocation fails. But this is a hairy subject, and is beyond my expertise.
Note that your third bullet point does cause memory to be freed immediately, because the `drop` function is a scope all its own.
pub fn drop<T>(_x: T) { }
I understand that the drop function is simply taking ownership of the passed value so Rust knows that once "drop" finishes, the _x can be 'dropped'.
But I though that the T had to be bound to have "Drop" trait (i.e. T: Drop) so that Rust knows it's possible to insert the call to 'drop' like _x.drop()?
If dropping was really implemented that way, every type would need to implement the Drop trait. Because otherwise there would be no way to free any memory at all.
Instead, Rust knows how to free any object. Implementing Drop is optional, and you only need to do it if you want some custom behavior at destruction.
The `drop` here is just a simple library function and isn't technically related to the `Drop` trait, which is implemented with magical compiler pixie dust and allows you to define a destructor via a `.drop` method. Anything that goes out of scope has this `.drop` method called automagically.
You're correct in that if you were calling `x.drop()` explicitly, you would need to have a `Drop` bound on `T`.
https://github.com/rust-lang/rust/blob/master/src/libcore/me...
The borrow checker is _entirely_ at compile time.
What I was trying to explain is that the timing is deterministic and immediate. People are used to unpredictable GC and rightly don't want to trust it with thing like this.
I don't know if Rust supports forcing explicit deallocation, but even if it doesn't, it's still an improvement compared to most GC-d languages.
It would almost be like how in c++ you can disable copying by setting all the copy constructors to delete and then the compiler will tell you if your accidentally making copy's of objects.
Sure, other than not actually being safe (UB resulting from use after move of unique_ptr being an obvious one; there are many others).
The lesson of C++ is that you can't bolt on safety by using templates to paper over the underlying mess. The mold always seeps through the wallpaper. The usual problem is that at some point you need raw pointers to call some API. If you can get raw pointers, the abstraction has to have a leak.
Nope, the argument is "valid but unspecified" not necessarily null. And http://en.cppreference.com/w/cpp/utility/move specifically has this as an example of UB:
There's an other fun UB because std::move doesn't check for aliasing, so self-assignment: is also an UB.And of course even if it were null., pcwalton talks about use after null. Deref'ing a null pointer (to use it) is still an UB.
From 20.7.1
unique_ptr isn't what's unsafe here. Null pointers are. The difference, admittedly, is mostly semantic. There is nothing undefined about using a unique_ptr after moving unless you use the internal pointer in undefined ways.If you dereference (i.e. use) a moved unique pointer you get undefined behavior.
When talking about "using a pointer" most people mean dereferencing it, not for example comparing against null. It may be somewhat imprecise language, but it's what security people mean when they talk about "use after free", for example.
Undefined behavior, at least colloquially, is about dereferencing a pointer to memory that may or may not be accessed and may or may not work/crash-the-program. That's what makes it undefined and an absolute disaster to track down.
I'm pretty sure that if you try to deference memory address zero the OS will bark at you.
Again, it's not my area of expertise, so please correct me if I'm wrong
e.g.
- http://stackoverflow.com/q/6793262/1256624
- http://blog.llvm.org/2011/05/what-every-c-programmer-should-...
Borrows are a different story. Especially with mutable borrows (which are basically a static lock on the value), the current lexical restriction is too coarse-grained. There are always workarounds, but it will be nice when it's fixed :)
[0] For the pedants repeating each other in the replies... Where by python, I mean the implementation that 98% of people use, called python, which the competition calls CPython. Yes it is a great feature of the implementation, like the deterministic memory management through reference counting.
I think that context managers in python are generally a very nice abstraction over these sorts of concepts, and it's cool to see how Rust is handling these problems very cleanly.
You can rely on a GC (and refcounting + a cycle collector is absolutely a GC) to finalize your resources for you, but it is quite difficult to reason about when the file will eventually be closed.
Also, the reference counting semantics of CPython are implementation-specific. From the PEP that introduced `with`:
> Note that we're not guaranteeing that the finally-clause is executed immediately after the generator object becomes unused, even though this is how it will work in CPython. This is similar to auto-closing files: while a reference-counting implementation like CPython deallocates an object as soon as the last reference to it goes away, implementations that use other GC algorithms do not make the same guarantee. This applies to Jython, IronPython, and probably to Python running on Parrot.
You are also wrong for CPython which does NOT require the garbage collector to be on, and DOES release the file in a deterministic way.
Simply, the article is misleading with regards to python.
CPython's behavior is nice, but it seems to me it came about by accident. Big heavy resources use refcounting because everything uses refcounting. Plus, if CPython had true concurrency, across-the-board refcounting probably wouldn't have lasted nearly as long (reason being: multithreaded refcounting requires atomic ops which make them much more expensive).
But in the general language design world, IMHO, refcounting is actually a pretty good compromise for resource management. Because:
1) The efficiency lost compared to Rust's approach is probably immaterial (since the Big Expensive Object that you're mananaging dwarfs the refcounting -- syscalls and file descriptors are expensive). So I don't think zero overhead is critical.
2) Cycles are pretty easy to avoid when you're only refcounting Big Expensive Objects (FDs, mmapped buffers, etc. etc.) For a general runtime, refcounting is tricky. But if it's a special part of the language that's small, simple, and only for resource management, it's pretty convenient. I think it's worlds better than the approach of mixing GC and resource management -- language designers should admit that's a terrible idea.
Granted, I have no idea how much overhead is imposed by atomic operations vs. a stop-the-world or concurrent GC (if anyone has some data, I'd love to see it!). But given how it's become de rigeur for new languages to come with a baked-in concurrency story and emphasize concurrent applications, I don't blame them for not wanting to tie themselves to the RC cart.
An atomic read/write/increment/decrement is a non-blocking (from the point of the view of a user code running on the CPU) CPU operation. The CAS operations required to do atomic inc/dec are relatively expensive compared to normal memory read/writes of course but nowhere near the scale of stop-the-world GC (where all threads in the runtime have to be blocked for a potentially large amount of time, sometimes measured in whole seconds or even 10's of seconds for large heaps).
Anyone who has had to troubleshoot JVM's handling large heaps (in the ~10GB range) quickly learns to love deterministic costs of refcounted resource management.
There are plenty of JVMs to choose from, even pauseless ones.
Hit one of my hot-buttons: Threads have got to be one of the most massive WOMBAT boondoggles of the last few decades.
Pro) none of this slow inter-process-communication stuff!
Con) EVERYTHING is now dangerous. (not quite, but close)
But the industry (?) decided "TOTALLY worth it!"
I forgot to mention that threads start a little faster than processes (well, quite a bit faster on a particular shitty OS in widespread use), but not so much faster that you won't need to pool and re-use the threads anyway. I think this is one area where Erlang (actor pattern), for instance, gets it right: pretend tasks run in their own process, and use an IPC proxy to send messages between the tasks/sub-programs, even if implemented with threads instead of OS level processes.
* on pypy
* on jython
* on ironpython
That is, it's broken on Python, and it's not correct Python code. It's only correct CPython code.
This is an implementation detail of file, not CPython.
Or did I fundamentally misunderstand something here?
(Some garbage collected languages can do automated "escape analysis" to figure out whether an object "escapes" the scope, and if it doesn't place it on the stack instead of the heap and deterministically free it when it goes out of scope. But that's surprisingly limited in real world programs and is generally speaking an implementation / optimization detail, not a core guarantee of the language).
The garbage collector would still close the file eventually, right? Unless of course it was copied somewhere else in the meantime, but in the code example, that is clearly not the case.
I personally like to use slides to learn things. I'm not sure if this is typical, but slides are usually concise and easy to read.
http://rustbyexample.com
2. There are two different kinds of references: `&` is immutable and `&mut` is mutable. One way or the other you'd need some sort of differentiation.
3. Borrowing, though crucial to the idea of Rust, is still less fundamental than ownership. You can have a language with ownership and without borrowing (indeed, this is how most prior languages with ownership work). But you can't have borrowing without tracking ownership.
There's no point in making a pointer to an integer and then dereferencing the pointer in the callee when an integer fits into the same amount of space as a pointer in the first place.
Very simple types (without heap pointers or destructors) can be passed by value and efficiently copied. You end up with a bit of an intuition for "value types" (simple types whose identity is bound up in the shallow memory that they contain).
In other GC'ed languages, there is a fixed set of these kinds of values (often called "primitives"). In Rust, any simple user-specified type can be a "primitive".
The default should be pass as read-only reference. If you want a copy of something, you should have to explicitly make one. If you want write access, you should have to say so. Rust has all the right concepts, but the defaults aren't in the safest direction. The experience of "const" in C/C++ teaches us that they should be.
Copyable objects are (by definition) simple enough that copying them does not introduce aliasing. Objects with heap pointers or destructors cannot be copied, because now there would be two "owners" of those values.
For what it's worth, I've found that a lot of these assumptions ("it doesn't matter, let the compile optimize it") really remove the programmer from important power.
And before you find yourself saying "are you saying everyone should write everything in assembler?":
1. There is a distinction between writing code in assembler and having to use a `&` sigil for "by-reference" in a language that has lambdas, traits and many other high-level features.
2. You should generally use Rust for cases where performance characteristics are important, and where control over memory is important. If you're sure it's not important, by all means go with a higher-level language that abstracts it away.
I made this point (perhaps more clearly) in my talk at GoGaRuCo: http://www.youtube.com/watch?v=ySW6Yk_DerY
Based on this article, immutable references are the most common kind of declarations. It seems to me that rust code would look much cleaner if they had the shortest syntax (including in pattern matching etc)
For resources or generally things that require cleaning up when they disappear, that just extends into the transfer, logically a shallow copy with the original variable flagged as dead in the type system.
The & symbol denotes a reference type, and it's an explicit thing in the type system and not just an annotation as part of the function call process. It can show up anywhere you'd write a type, things like `Option<&T>` or other types with `&T` fields aren't too uncommon, and `&&T` is also a type.
I haven't really thought it through, I guess, but modifying a type `T` with a `&` to make it a pointer type `&T` seems to compose a lot better than making all types `T` implicitly pointers and requiring `&T` whenever you want the value inline and not behind a pointer indirection, whether in a local variable or as a field in a struct.
Having the compiler eliminate resource leaks is really, really handy. This is one thing that's very easy to get wrong in go. Will be interesting to see how much of a cost the upfront typing has during quick prototyping or explorative programming, but its great to see this kind of type system make it into something that's well on track to becoming a mainstream programming language.
fn person(person: &Person) -> &str { ... }
should be:
fn first_name(person: &Person) -> &str { ... }
If it is freeing everything at the scope boundary it is incurring a huge runtime cost relative to garbage collected languages. GC scales with the number of live objects during GC while this scheme scales with the number of allocated objects. It is far more predictable than GC but I don't believe that it isn't more expensive.
Are you talking about an additional cost beyond "freeing memory that was allocated"?
Of course if your program is short-lived you can just allocate a sufficiently large heap, pointer bump the whole way and free all non-memory resources on program termination, which ought to be cheaper (but probably not that much in practice due to cache effects).
Rust also has an Arena library (http://doc.rust-lang.org/0.11.0/arena/index.html), which can be used for cases where you know that you have enough memory for a given operation and are willing to throw out the entire thing when the operation (or program ;) ) is done.
A GC has to determine liveness of objects, so it has to traverse the object graph (or at least the most recent generation if the GC is generational) to know which objects are still alive and which aren't. That's what "GC scaling" refers to. Then it still has to deallocate dead objects, and pay the same deallocation cost as scope-bound free.
That said, you're right: if you don't need Rust's strengths, the tradeoff may not be worth it.
I think what I'd really like is something like C# and the .NET base class library, but AOT compiled to native code and using automatic reference counting, with some way to indicate weak references. I know that neither GC nor reference counting quite reaches the level of "don't make me think" when it comes to resource management, but given a choice between explicit resource disposal and the occasional weak reference annotation, I think I'd choose the latter.
Anyway, enough ranting from me for now.
It sounds like Swift may be close to what you want, yeah?
First there will be need of a standard library. For the moment to do anything sensible you are importing Apple's Foundation Library (probably in one year time it will be better).
Then I see no problem to have it server-side (both as a scripting or compiled language).
Anyone familiar to swift internals here to say whether server-side swift is at least theorically feasible ?
Mono -aot and .NET Native.
Just not with RC, though.
Or D, if you with to have both GC and RC.
Is there syntactic overhead? Yep. Does it impose constraints on design? Yep. But when it comes to resource management, everything that Rust does is just something that you'd need to use a pen and paper to keep track of in other languages (including pervasively reference-counted languages, since AFAIK there's no way to automatically enforce the proper use of weak pointers).
That's interesting. So this holds for languages and applications where you could get away with not knowing the automatic memory management implementation and adjusting your design based on that? For example, that you don't have to worry yourself with making sure to not allocate too many values on the heap, at least in most of the code?
Would you prefer using Rust for problems and domains where you strictly don't need the efficiency and determinism (like memory usage) you get from using Rust?
I think we all have a hierarchy of sorts: bash or Perl for one-off, disposable scripting tasks; Python or some other light dynamic language for when things start getting more serious, but are still personal projects; C# or some other relatively heavyweight hammer for when we're writing code that will live beyond our control, maintained by others and run for years and years.
I don't write production code in Rust yet (heavens no, not until 1.0 at least... godspeed to wycats :P ) but right now it's in that third category of languages where I trust that it can be maintained by a team and trusted to exist for years. However, I would consider Rust for that second category of somewhat-serious personal projects, because I know too well how such "personal projects" can vault unexpectedly into that third category of "mission-critical team projects". But before that, I'd need to wait for the library ecosystem to mature.
EDIT: I should clarify too that resource management is only part of the reason why I don't worry about Rust code. There's also stuff like the ability to know that a given global variable is immutable, and can't be changed out from under me. Or knowing that an innocent-looking block of code can't unexpectedly kill the process (divide-by-zero notwithstanding).
If you have something like:
Then you run into the problem that @patch has cleaned up all of your mocks by the time the yield returns.Now, I haven't tried to do anything equivalent in Rust; I'm not sure what the idiomatic equivalent would be. But having more explicit control of lifetime, rather than it always depending specifically on that stack frame, seems like it might help here.
This doesn't mean that Rust is absolutely perfect. There's tons of stuff that is awkward to express, and tons of more features that would be nice to have. But the core all works, and works well.
NOT, if you only write safe code.
That is, if you write entirely 100% safe code, and use some library that uses 'unsafe' code at a low level (this is very common; for example, an C binding or memory optimization), then rust does NOT guarantee safety.
Furthermore, there is no way to know if your code path includes unsafe code.
Rust is relatively safe; it's just not totally safe. Using rust does a reasonable job of protecting you from segfaults most of the time; but it's not a silver bullet.
This isn't always necessarily true.Rust uses 'drop flags', which are extra bytes at the end of struct instances in memroy, to track objects and keep meta information about them to determine if they should be dropped.
At the end of scope when the drop checker runs it checks against these drop flags to determine if an object should be dropped or not.
(For example, if a closure captures a variable it may no be dropped at the end of the scope it would normally have been dropped in).
There seems to be this myth that resource deallocation in rust is magically determined at compile time and the has 'zero cost'; it's simply not true.
There is a runtime cost to the drop checker; it's not particularly large, but it's certainly not just a sequence of drop operations.
Don't get me wrong; I really enjoy rust, and I like it a lot. ...but those two ideas pop up repeated in discussions about it, and they're not exactly correct.
I think this is actually a misconception. In every high-level language (Ruby, Python, etc.) there are APIs for interacting with low-level code. These APIs (C bindings and FFI) are unsafe, and are in the path of virtually all high-level code. It is the responsibility of somebody implementing a C binding for Ruby to ensure that the publicly exposed API is safe.
If you use `nokogiri` in Ruby and get a segfault, that is a bug in nokogiri. It is not an indication that Ruby is unsafe. Similarly, the "safe" dialect of Rust is Rust. Rust provides a low-level binding to unsafe code via the unsafe dialect that is philosophically equivalent to the FFI API or C bindings in Ruby. If somebody uses the low-level binding and exposes a Rust library that segfaults, there is a bug in that binding, full stop.
I think this is very important. If you use regular Rust code with the expectation that "anyone in your path may cause you to segfault", something has gone very, very awry in the ecosystem. This is not an expectation in other high-level languages when people write low-level libraries with unsafe code in them, and it should not be an expectation in Rust.
> Rust uses 'drop flags', which are extra bytes at the end of struct instances in memroy, to track objects and keep meta information about them to determine if they should be dropped
This is in the process of being eliminated (see https://github.com/rust-lang/rfcs/pull/320). I didn't mention it because it's runtime overhead that will not be present shortly, and certainly by the time 1.0 is reached later this year.
Rust has a strict definition of safety; not all rust code for fills that guarantee.
We're not talking fast and loose here. These are absolute definitions.
You can write safe code that crashes in rust. That absolutely violates the safety guarantee whatever the reason for it.
> This is in the process of being eliminated
No it's not.
What's happening is some of the run time overhead is being reduced.You can write Ruby code that calls into unsafe C code and SEGVs. This is a bug in the C code you called. It should have exposed a safe interface.
You can write Rust code that calls into clearly delineated unsafe code and SEGVs. This is a bug in the unsafe code you called. It should have exposed a safe interface.
Anything to the contrary is a misunderstanding of the purpose and role of unsafe code in Rust, and endangers the entire ecosystem, just as it would endanger the Ruby ecosystem if it were seen as normal for C extensions to SEGV the safe Ruby process.
I'm just saying there are these two misconceptions:
This isn't true if your code path contains any unsafe code. This isn't true at all.Sure, if it does segfault its a bug in the c code / unsafe code.
You're absolutely correct.
...but bugs aren't deliberate. Obviously no one is going to deliberately expose a c api that has segfaults in it.
The misconception is that if you're using rust you're safe from the bugs that other programmers may have made... which is not true.
There's a trust chain involved; if you depend on a library you trust it's bug free and won't delete your harddisk and crash your program, but there's a difference between trusting a library is not malicious, and trusting a library is bug free.
If you have to trust the library is bug free, you're really no better off than using C++ are you?
...but people keep saying that if you use rust then there are certain types of safety that are guaranteed by the language: http://doc.rust-lang.org/reference.html#behavior-considered-..., regardless of programmer bugs.
It's just not true.
Once again, I'm not arguing that safety is good or bad or whatever. My assertion is very very specific:
Rust code written 100% safely can crash if it has a dependency that uses unsafe code.
The assertion that 100% safely written rust code with arbitrary dependencies can't crash is false.
Of course it's not true, but the libraries you're using will (hopefully, for the most part) be written in Rust, which increases the level of trust.
> There's a trust chain involved; if you depend on a library you trust it's bug free and won't delete your harddisk and crash your program, but there's a difference between trusting a library is not malicious, and trusting a library is bug free.
> If you have to trust the library is bug free, you're really no better off than using C++ are you?
The libraries are written in Rust, so have all the benefits of that. It is nonsense to say that "libraries can have bugs so you might as well use C++".
My argument is that if you assert 'rust is 100% safe' then you must assert, all your dependencies are bug free.
If you can somehow magically assert 'all my code is bug free', that would be great... but if so, you wouldn't need the borrow checker at all would you?
(I'm certainly not endorsing using C++ :P I'm specifically talking to the assertion that rust code can be safe, as per the definition of safety in the guide, that certain operations are impossible; certainly, and absolutely, rust code is safer and more trust worthy (imo) than c/c++ code... but you've got to be reasonable about what you're arguing. 'rust is magical and can never crash' turns up in blog posts a lot)
...there are two things in this article which are, I'd say, 'common misconceptions' about rust.
So let me give everyone a source to cite on Wikipedia:
"Rust does not keep C code from segfaulting." ~ kibwen
> What's happening is some of the run time overhead is being reduced.
I believe wycats was responding specifically to
> Rust uses 'drop flags', which are extra bytes at the end of struct instances in memroy, to track objects and keep meta information about them to determine if they should be dropped.
And these are being eliminated. That is the drop flag will be entirely separate to the struct instance, and will only be injected into stack frames where there is a conditional branch in which a value is moved out in one, but not the other, e.g.
then `x` will get a drop flag as a single bit (or byte) on the stack. If the code is changed to there will be no drop flag, and no overhead.Furthermore, placing the value on to the heap, e.g. `box some_value`, will not have an associated drop flag for `some_value`, that is, eliminating the byte that is currently stored with it.
The points I was making is that 'zero cost' memory management isn't 100% zero cost. The drop checker does incur some basic dynamic runtime costs.
To be 'zero cost' drop behaviour would be determined at compile time.
What do you consider a "guarantee"? Do you consider pure JavaScript to "guarantee" safety, even though there have been exploitable bugs in the JIT and interpreter? Do you consider pure-computation, non-allocating Rust to be safe, even though LLVM may contain codegen bugs that allow for undefined behavior? Do you consider seccomp-isolated code to be safe, even though there have been, and will likely be in the future, kernel bugs that allow sandboxed process to escape?
The point is that instead of talking about "100% safe", which is never the case in the real world, I think it's better to talk about trusted computing bases. That is, instead of saying "X is safe" or "X is unsafe", we talk about what effect X has on the memory-unsafe surface area of a program. Rust's goal is to reduce the trusted computing base of an application written in it to the hardware, compiler, unsafe blocks, and type system. This gives us essentially the safety properties of "managed languages" (except I don't like that term because it tends to imply garbage collection and lack of fine-grained control over memory, which don't apply to Rust).
A specific example I like to give: Sure, we could hardwire (for instance) Vec into the language, and therefore reduce the number of lines of unsafe code in the libraries. But doing so would just be moving the unsafe code from the unsafe blocks in the libraries to the compiler itself. There would be no net gain in safety from it—compiler code can have bugs just as library code can—and there would be a decrease in maintainability and flexibility.
You can think of Rust's safety properties as establishing a type of sandbox if you'd like. All sandboxes have trusted computing bases, and Rust's compile-time sandbox is no exception. But "safety" as applied to a sandbox still has an important meaning. Chromium/Firefox OS's sandbox, for example, is a safe sandbox, even though its security depends on strong assumptions about the trustworthiness of the OS kernel and the IPC layer between the trusted and untrusted processes.
> There is a runtime cost to the drop checker; it's not particularly large, but it's certainly not just a sequence of drop operations.
After the drop optimizations are done, this will only be true if you conditionally move an object on one or more branches. It's already true today if the object gets SROA'd on the stack, as LLVM can then optimize out the drop flags. We've talked about adding a lint so the compiler can optionally warn and allow you to fix it if this performance cost is a concern to you. (It's a very minor cost—one test and branch on a stack byte, and again only if the object is conditionally moved.)
Note that C++ move semantics incurs essentially the same cost.
Might I just make a meta comment that while that may not be a huge use case, the ability to assert that some particular optimization is firing and get warnings if I've done something to break it is something I've wanted out of a language for a long time. I know it's easier said than done, but I'd love it.
[1] https://github.com/boto/botocore/tree/develop/services
GC languages is trying to emulate machine with infinite memory so you don't need to bother yourself with _memory_ management. It's not compatible with rust's resource management system that manages resources and treats memory as another type of resource. The same is true for C++ BTW, in good C++ code you never release resources by hand and use ownership for this (RAII).