Perhaps my biggest critique is that crates.io has no namespacing. Anyone can just claim a global and generic package name and we mostly have to deal with it (unless you avoid using the crates.io repository, but then you'll probably have more problems...). Some of these globally-claimed generic packages are not really the best package to use.
Maybe it was a reaction against the Java-style reverse DNS notation, which is verbose and annoying, but a more GitHub-style user/group namespace prefixing package names would have been the nice middle ground.
And the horrible decision to not make library-level ("module") and code-unit-level ("package") namespacing orthogonal. The former was an afterthought tacked on since the package system was designed to be used only within Google's monorepo and little care was paid to how it would work when it was released to the public and used more generally.
I want to understand what you just said, but I fear watering your language down a bit might be a tall ask with some people. Would you be willing to eli5 what you believe Go did that was a horrible decision with regards to module/package namespacing?
I think what they mean is: if you see a line like `import git.example.com/foo/bar/baz`, that could be package `baz` inside module `git.example.com/foo/bar`, or it could be package `bar/baz` inside module `git.example.com/foo`.
Also, even if you know it's the latter, package namespacing isn't strictly related to directory structure, so `bar/baz` has no specific meaning outside of the context of a go import. They could have used any other separator for package components - `git.example.com/foo:bar:baz` - but instead they chose the slash, making the scheme both technically ambiguous and easy to confuse for an HTTP URL.
Ah that makes sense. I think Go did somewhat stumble a bit in the early days due to this, especially with repositories in GitLab, where GitLab allows essentially a directory tree where your repository can be nested indefinitely in directories like `https://gitlab.com/mygroup/subgroup1/subgroup2/repository`.
I still don't think this is a huge issue, to be honest. Not one big enough for me to complain about, for sure. But it's definitely not ideal.
URLs for packages makes a lot of sense. It works well in the land of Go. It also conveniently eliminates the need for the language to have a global packages database. Upload your package to example.com/your-thing and it's released! (You can, of course, still offer a cache and search engine if you want to.)
Cargo does support URLs to git repos for dependencies. But crates.io is the official platform and almost every search I do on it returns at least one generically named entry with an empty repository that someone snatched away and never used.
No, URL's don't make sense because your application shouldn't care where on the internet your dependency happened to be hosted when you integrated it. It's location has nothing to do with what it is.
By the time you're going to production, your vetted and locked dependency should be living in your own cache/mirror/vendored-repo/whatever so that you know exactly what code you built your project around and know exactly what the availability will be when you build/instantiate your project.
Your project shouldn't need to care whether GitHub fell out of fashion and the project moved to GitLab, and definitely shouldn't be relying on GitHub being available when you need to build, test, deploy, or scale. That's a completely unnecessary failure point for you to introduce.
Systems that use URL-identified packages can work around some of this, but just reinforce terrible habits.
Isn’t that why GOPROXY exists though? Not sure why you would need an internet connection. URLs don’t necessarily equate to the internet. Our internal and external packages are all locally hosted and work regardless of the internet being available.
> By the time you're going to production, your vetted and locked dependency should be living in your own cache/mirror/vendored-repo/whatever so that you know exactly what code you built your project around and know exactly what the availability will be when you build/instantiate your project.
In the Go world this would be "vendored" dependencies, that is, the dependencies are within your source tree, and your CICD can build to its hearts content with no care in the world about the internet because it has the deps.
The URL is useful for determining which version of a specific project is being used - "Oh we switched to the one hosted on gitlab because the github one went stale"
The advantage of using gitlab, or github, or whatever public code repository is that you get to piggy back off their naming policies which ensure uniqueness.
But, at the same time, there's no reason that the repo being referred to cannot be in house (bob.local) or private.
Having said all of that, the Go module system is a massive improvement on what they did have originally (nothing) and the 3rd party attempts to solve the problem (dep, glide, and the prototype for modules, vgo), but it's not without its edge cases.
URLs are well structured and unique, with a sensible default - sourcing the file from the internet - and ubiquitous processes for easily mapping the URL to an alternative location.
I.e., when you're going to run the production build, the URLs are mapped to fetch from the vetted cache and not the internet.
I don't see any downsides to allowing them as a source, or making them the default approach
> and ubiquitous processes for easily mapping the URL to an alternative location.
This seems strange to me because the whole point of a Uniform Resource Locator is to specify where a resource can be located.
It's a bit like saying "My project depends on the binder on shelf 7 in Room 42, sixth binder from the left. Except when I go into production, then use...." Don't tell me what binder it's in, tell me what it is.
I can see a case made for URIs, which is basically what Java did.
This was a big annoyance for me back in the day when I was dealing with XML namespaces. URLs never made sense for that use case and too many tools tried to pull XSDs from the literal URL which was always generally out of date, some projects switch to URIs like tag uris or URNs and it was much better, imo.
Fully qualified domain names (java/maven) aren't URIs. The latter are far more transient. Maybe a form of permalink could work, but that likely places too great a burden on package maintainers. I don't see that working out honestly.
Isn't that just delegating the problem? URL dependencies do not replace what crates.io does, and a modern language will still want something like it. You'd just end up with most every dependency being declared as crates.io/foo.
It doesn't help with the failure mode of dependencies disappearing, which forces people that care about it to vendor, which in turn brings its own set of issues.
[registries]
maven = { index = "https://rust.maven.org/git/index" }
[dependencies]
some-package = { index = "maven", version = "1.1" }
Obviously Maven doesn't host any Rust crates (yet?), this is just a theoretical example. Very few projects bother to host their own registry, partially because crates.io doesn't allow packages that load dependencies from other indices (for obvious security reasons). The registry definition can also be done globally through environment variables: CARGO_REGISTRIES_MAVEN="https://rust.maven.org/git/index". Furthermore, the default registry can be set in a global config file.
In theory, all you need to do is publish a crate is to `git push upstream master`, and your package will become available on https://github.com/username/crate-name (or example.com/your-package if you choose to host your git repo on there).
Personally, I don't like using other people's URL packages, because your website can disappear any moment for any reason. Maybe you decide to call it quits, maybe you get hit by a car, whatever the reason, my build is broken all of the sudden. The probability of crates.io going down is a lot lower than the probability of packages-of-some-random-guy-in-nebraska.ddns.net disappearing
Agreed. The other thing I don’t really like is that you can’t split up things in the rust namespace hierarchy between crates (something that’s natural with jars in JVM). I would have liked to have defined things so that I could have the unicode handlers for finl live in finl::unicode, the parser in finl::parser, etc. but because they’re in separate crates rust gets upset about finl being defined twice and there’s no workaround for it. There are likely pitfalls I don’t see in what I want, so I live with it.
As I recall, I could do something where I could have a common root crate that would import and re-export the other crates with the namespaces modified on the exports and then control which crates are exported through feature gates, but it just seemed more hassle than it was worth.
> Maybe it was a reaction against the Java-style reverse DNS notation
I suspect it was less a reaction against anything and more just following the norms established by most other package managers. NPM, PyPI, RubyGems, Elixir's Hex, Haskell's Cabal... I'm having a hard time thinking of a non-Java package manager that was around at the time Rust came out that didn't have a single, global namespace. Some have tried to fix that since then, but it was just the way package managers worked in 2014/2015.
Just some random Cargo security-related issues I noticed:
- No strong link between the repo and the published code.
- Many crates were spammed that were just a wrapper around a popular C/C++ library. There's no indication of this, so... "surprise!"... your compiled app is now more unsafe C/C++ code than Rust.
- Extensive name squatting, to the point that virtual no library uses the obvious name, because someone else got to it first. The aformentioned C/C++ libraries were easy to spit out, so they often grabbed the name before a Rust rewrite could be completed and published. So you now go to Cargo to find a Rust library for 'X' and you instead have to use 'X-rs' because... ha-ha, it's actually a C/C++ package manager with some Rust crates in there also.
- Transitive dependencies aren't shown in the web page.
- No enforcement or indication of safe/unsafe libs, nostd, etc...
- No requirement for MFA, which was a successful attack vector on multiple package managers in the past.
DISCLAIMER: Some of the above may have been resolved since I last looked. Other package managers also do these things (but that's not a good thing).
In my opinion, any package manager that just lets any random person upload "whatever" is outright dangerous and useless to the larger ecosystem of developers in a hurry who don't have the time to vet every single transitive dependency every month.
Package managers need to grow up and start requiring a direct reference to a specific Git commit -- that they store themselves -- and compile from scratch with an instrumented compiler that spits out metadata such as "connects to the Internet, did you know?" or "is actually 99% C++ code, by the way".
Sure, but all the drawbacks you enumerate are also advantages for gaining critical mass. A free-for-all package repository is attractive to early adopters because they can become the ones to plug the obvious holes in the standard library. Having N developers each trying to make THE datetime/logging/webframework/parsing library for Rust is good for gaining traction. You end up with a lot of bad packages with good names though.
> Extensive name squatting, to the point that virtual no library uses the obvious name, because someone else got to it first.
Maybe the obvious names should have been pre banned. But I don't see the issue with non-obvious names either way you're going to have to get community recommendation/popularity to determine if
In ASP.NET land, I regularly work on projects where there is an informal rule that only Microsoft-published packages can be used, unless there's good reason.
You don't want to be using Ivan Vladimir's OAUTH package to sign in to Microsoft Entra ID. That probably has an FSB backdoor ready to activate. Why use that, when there's an equivalent Microsoft package?
When any random Chinese, Russian, or Israeli national can publish "microsoftauth", you just know that some idiot will use it. That idiot may be me. Or a coworker. Or a transitive dependency. Or a transitive dependency introduced after a critical update to the immediate dependency. Or an external EXE tool deployed as a part of a third-party ISV binary product. Or...
Make the only path lead to the pit of success, because the alternative is to let people wander around and fall into the pit of failure.
Crates.io has publisher information-- namespacing is not required for that. For example, here are all the crates owned by the `azure` GitHub organization and published by the `azure-sdk-publish-rust` team: https://crates.io/teams/github:azure:azure-sdk-publish-rust
> > "We're pretending security is not an issue." has been the feedback every time this is raised with the Cargo team.
> Do you have a specific link where I can read this response, because this is not at all the responses I have read.
Those aren't people saying security isn't an issue but examples of concerns you have which is different.
For some of those, there are reasonable improvements that can be made but will take someone having the time to do so. While the crates.io team might not be working on those specific features, I do know they are prioritizing some security related work. For instance, they recently added scoped tokens.
For some, there are trade offs to be discussed and figured out.
The people that are working on the project haven't implemented namespaces, or any other security feature really, so what they say is immaterial. What they do is the only thing that matters.
The point is that it's much easier to make a mistake typing "requests" than "
org.kennethreitz:requests" (as a pure hypothetical.)
It also means that more than one project can have a module called "utils" or "common", which once again reduces the risk of people accidentally downloading the wrong thing.
> I'm having a hard time thinking of a non-Java package manager that was around at the time Rust came out that didn't have a single, global namespace
The implication here is that namespaces in package managers weren't a known concept. Outside Java, NPM - probably the biggest at the time - not only supported them but was actively encouraging them due to collective regret around going single-global in the beginning. Composer is another popular example that actually enforced them.
Not only was namespacing a known widespread option, with well documented benefits, it was one that was enthusiastically argued for within the Rust community, and rejected.
NPM added namespaces in version 2, which was released in Sep 2014, just 2 months before cargo was announced. I don't remember anyone making a big deal about using scopes in NPM for several years after that, it was just there as an option. The announcement blog post of v2 only gives two paragraphs to scoped packages and explicitly frames the feature as being for enterprises and private modules [0]:
> The most prominent feature driving the release of npm 2 didn’t actually need to be in a new major version at all: scoped packages. npm Enterprise is built around them, and they’ll also play a major role when private modules come to the public npm registry.
My memory is that the industry as a whole didn't really start paying attention to the risks posed by dependencies in package managers until the left pad incident.
To be clear, I'm not saying that it was a good idea to not have a better namespace system or that they were completely ignorant of better options, just that they were very much following the norms at the time.
The left pad issue was kind of wild coming from the enterprise Java space. Supply chain attacks against open source software were already being taken pretty seriously, my last company had it's own Maven repository manager running that was used to stage and vet packages before they could be used in production.
Yeah it's all a bit of revisionist history here, or I guess a bit ignorant. I had a friend who worked at Sonatype from pretty early days and they were, as I understand it, specifically working in this area of infrastructure for vetting, signing, license checking, etc. for corporate environments that needed to be extra careful about this stuff.
That crates.io launched without explicitly acknowledging this whole problem is either naivety or worse: already by then Java wasn't "cool" and the "cool kids" were not paying attention to what happened over there.
It's not that the industry wasn't paying attention until the 'left pad incident' -- that only holds if one's definition of "the industry" is "full stack developers" under the age of 30; I remember when that happened and I was working in a shop full of Java devs and we all laughed at it...
Maven's biggest problem was being caked in XML. In other respects it was very well thought out. That and it arrived at the tail-end of the period in which Java was "cool" to work in.
It's not revisionist history, the wording I chose was meant to acknowledge that there were segments of the industry that did take dependencies seriously. I'm very much aware that the Java world had a much more robust approach to dependencies before this, but "the industry as a whole" includes all the Node shops that were hit by leftpad as well as all the Python and Ruby shops that were using equally lousy dependency management techniques.
Rust chose to follow the majority of languages at the time. Again, as I noted in my previous comment, I'm not defending that decision, just pointing out that most of the widely-used languages in 2014 had a similar setup with similar weaknesses.
I don't think the left-pad problem wasn't about package namespacing it was about the ability to unpublish packages as well the prevalence of micropackages caused by lack of a decent standard library.
Also npm's bad policy/decision to transfer control of package in the name of predictability(this should probably be avoided for packages that aren't malicious. You could argue for seizing broken/trivial and unmaintained packages that have a good name but even then it might be best to leave well enough alone).
I suppose you're talking about the original dispute which led the developer to unpublish his libraries (which npm stupidly allowed, and cargo didn't). There's a smaller chance of a company wanting a random package namespace then a package name but its not impossible (think Mike Rowe Soft vs Microsoft)
> I don't think the left-pad problem wasn't about package namespacing it was about the ability to unpublish packages as well the prevalence of micropackages caused by lack of a decent standard library.
It was "about" cavalier approach to the dependency supply chain. A dependency disappearing outright is just one of many failure modes it has.
> The left pad issue was kind of wild coming from the enterprise Java space.
This may be a little off topic for this comment thread but this is a little misrepresentative. Hosted private repos for enterprise weren't exclusive to Java at the time of left pad - anyone doing enterprise node likely had one for npm too & were probably well prepared for the attack. Such enterprise setups are expensive though (or at least take a level of internal resources many companies don't invest in) leaving the vast majority exposed to both js & java supply chain attacks even today.
At the time Nexus was free to self host, and that is what many smaller teams did just that to archive known good packages for the CI pipeline, I'm not in the Java space anymore so I don't know if that's still the case.
mainly, you can trust that anything under the foo/ namespace is controlled by only a smaller group of people, as opposed to the current situation on cargo where people pseudo-namespace by making a bunch of packages called foo-bar, foo-baz, and you can't trust that foo-bin wasn't just inserted by someone else attempting to appear to be part of the foo project. It also helps substantially with naming collisions, especially squatting.
If you want to check your dependency tree for the number of maintainers you're dealing with instead of the number of dependencies, this can be done with cargo tree + checking the Cargo.toml/crates.io ownership info for each of the found packages. I don't know if there's a command written to do that already, but I've done that with a small script in the past.
Sorta—it looks like they were mostly just using that system by convention until May 2015, when they finally become enforced [0]. Still, that's a good one that I hadn't thought of, and they at least had the convention in place.
Fair enough. As I noted to another commenter, I'm not trying to say there was no prior art (if nothing else there was Maven), just that they were following the overwhelming majority of mainstream languages at the time.
> just that they were following the overwhelming majority of mainstream languages at the time.
They were trying to do better than mainstream languages in other areas and succeeded. IIRC on this front they just decided Ruby's bundler was the bee's knees.
The same developer who worked on bundler also worked on the initial version of cargo. That’s why they’re similar.
And at that time, it was a good idea. Ruby was popular and bundler + gem ecosystem was a big reason for its popularity. No one was worried that Rust might become so popular that it might outgrow the bundler model. That was only a remote possibility to begin with.
A mistake that many programmers make, as if baking one more feature on top would have made any difference that wouldn't be amortized in just a few weeks... Sigh.
I'm honestly astounded at how badly many languages have implemented dependency management, particularly when Java basically got this right almost 20 years ago (Maven) and others have made the mistakes that Java fixed. With Maven you get:
1. Flexible version (of requirements) specification;
2. Yes, source code had domain names in packages but that came from Java and you can technically separate that in the dependency declaration;
3. You can run local repos, which is useful for corporate environments so you can deploy your own internal packages; and
4. Source could be included or not, as desired.
Yes, it was XML and verbose. Gradle basically fixed that if you really cared (personally, I didn't).
Later comes along Go. No dependency management at the start. There ended up being two ways of specifying dependencies. At least one included putting github.io/username/package into your code. That username changes and all your code has to change. Awful design.
At least domains forced basically agreed upon namespacing.
Yes, #3 in particular is important for many large corps where one team develops a library that may be pulled in by literally thousands of other developers.
> Later comes along Go. No dependency management at the start. There ended up being two ways of specifying dependencies. At least one included putting github.io/username/package into your code. That username changes and all your code has to change. Awful design.
"github.io/username/package" is using a domain name, just like Java. Changing the username part is like changing the domain name--I don't see how this is any worse in Go than in Java.
If you don't like that there's a username in there, then don't put one in there to begin with. Requiring a username has nothing to do with Go vs. Java, but rather is because the package's canonical location is hosted on Github (which requires a username).
I don't know why so many programmer's use a domain they don't control as the official home of their projects--it seems silly to me (especially for bigger, more serious projects).
Slight difference is that it wouldn't break existing builds if you changed namespaces in Java. The maven central repo does not allow packages to be rescinded once they are published.
So that old version of package xyz will still resolve in your tagged build years from now even if the project rebrands/changes namespaces.
> I don't know why so many programmer's use a domain they don't control as the official home of their projects
Not only that, but a commercial, for-profit domain that actively reads all the code stored on it to train an AI. Owned and run by one of the worst opponents of the OS community in the history of computing.
At least move to Gitlab if you must store your project on someone else's domain.
Note that in Java it is merely a convention to use domain names as packages. There is no technical requirement to do so. So moving to a different domain has no impact whatsoever on dependency resolution. Many people use non-existent domain names.
To be honest I really like how Java advocated for verbose namespaces. Library authors have this awful idea that their library is a special little snowflake that deserves the shortest name possible, like "http" or "math" (or "_"...).
It may be a convention but in practice if you want to publish your package to Maven Central you need to prove ownership of your group ID domain. (Or ownership of your SCM account, which is in essence another domain).
Java did a lot of things right beyond the language and VM runtime, both of which were "sturdy" by the standards of the early 1990s. Using domain names for namespaces was one nice touch. Having built-in collections with complete built-in documentation was another excellent feature that contributed to productivity.
The dependency management side of Maven is great. OTOH, I was astounded to learn today that Maven recompiles everything if you touch any source file: https://stackoverflow.com/a/49700942
This was solved for C programs since whenever makedepend came out! (I'm guessing the 80s.)
(Bonus grief: Maven's useIncrementalCompilation boolean setting does the opposite of what it says in the tin.)
This needs to be resolved by every damned language.
Just make signed dependencies a universal default, point to an https page for the package vendor and use the signing key from there.
Neither node nor maven ever bothered to solve this, so we end up wandering the Wild West wondering when it will be that HR, or legal, or architecture comes knocking on the door to ask what we were thinking having a dependency on a dynamic version of Left Pad.
I'd kinda like to see what Cloudflare and Let's Encrypt could come up with if they worked together on at very least a white paper and an MVP POC.
Pay somebody either internally or externally to maintain a repo of all your dependencies and point your code at that. You won't get a left-pad incident. You won't get a malicious .so incident (unless you mirror binaries instead of source code).
Like if you ran out of screws to make your product with do you walk around the street and scrounge up some? No, you go to a trusted vendor and buy the screws.
I'm suggesting that the guys who've repeatedly proven themselves technically competent around security at scale might have a couple of useful ideas regarding how the industry might go about crawling its way out of this little security clusterfuck.
And perhaps even stop treating something as simple as a BOM as an enterprise feature, given that the overhead on such things is damned near zilch and the security implications are staggering.
There's reasons why Google projects don't go out on the internet to get their 3rd party deps.
They're all checked into Google3 (or chromium, etc.). One version only. With internal maintainers responsible for bringing it in and multiple people vetting it, and clear ownership over its management. E.g. you don't just get to willy nilly depend on a new version -- if you want to upgrade what's there, you gotta put a ring on it. If you upgrade it, you're likely going to be upgrading it for everyone, and the build system will run through the dependent tests for them all, etc.
And the consequence is more responsible use of third party deps and less sprawling dependency trees and less complexity.
And additional less security concerns as the code is checked in, its license vetted, and build systems are hunting around on the Internet for artifacts.
This is a common critique, and although I don't have insight into why the original decision to not have namespaces was made, the current outlook is that until issues related to continuity are resolved, it's a no go:
That article starts with the premise that “it’s a feature, not a bug” then goes on to describe a whole bunch of things I consider to be anti-features of a packaging system that has a flat namespace.
The first section says it discourages forking. I consider this to be bad. Nobody’s code should be more important purely because it squatted a better name.
The Identity section actually makes the case that flat registries make naming harder.
The section on Continuity is “we’ve tried nothing and we’re all out of ideas”. Make up an org name and grandfather all packages in the flat namespace into that special org. Also this is already a problem because packages in the flat namespace do get abandoned, then forked, and then we have the associated issues.
The section on Stability seems to take it as a given that crates.io should be the only registry. I don’t. It also seems to conflate cargo with rustc for the benefit of the argument.
The squatting section describes only anti-features and I don’t consider the author’s legitimate use cases to be legitimate reasons to squat.
I think the only legitimate problems that need addressing are the ergonomics of accessing namespaced packages throughout transient dependencies and backwards compatibility with non-namespaced code. But the fact that these are real problems does not, to me, make a flat namespace a “feature”. It’s just easier to implement.
It’s okay for it to be a mistake that takes effort and time to fix.
Another option would be to grandfather all packages into their own org. So serde becomes serde/serde. This way you don't need to manage permission rules in the legacy "all" namespace.
You get some oddities such as serde-derive/serde-derive but the package owners can choose if they want to move to serde/derive or leave it in a separate namespace.
I did some analysis on crates.io to find the top name squatters. Then I did some calculations and found that the top name squatter created their crates at a rate of about one ever 30 seconds for a period of a week straight.
I send the analysis to the crates.io team and pointed that they have a no-automation policy.
They told me that it was not sufficient proof that someone was squatting those names. That's my problem with crates.io is that they have a clear policy and they don't enforce it so all the short/easy to remember names for crates are already taken and there is nothing you can do to get it.
There's a secret effort in the Rust community to supplant Crates.io and create an entirely new package ecosystem with proper namespacing, security, and much better community.
Not naming names, but I know several people working to put Crates.io out to pasture.
There's a level of playing nice with them for the time being (eg. build reproducibility), but it's only KTLO.
Crates.io needs to die for Rust to thrive. They're a bungled, mismanaged liability. New code, new leadership.
Drama avoidance and avoiding bikeshedding seem obvious. Much easier to present a working system than a design that will get nitpicked into irrelevancy.
That quite funny. Just like when some people formed a violent militant group to take down a violent tyranical dictatorship. Of course they would promise that they would absolutely de-arm right after the dictatorship has been overthrown, and immediately establish a peaceful democratic government with fair election. They absolutely would, would they? They would never turn into what they were formed to replace, would they?
Crates.io doesn't need to die necessarily. It needs some competition as a wake-up call.
Once a better alternative is out there, crates.io will either wither and die or improve. If it matches its competition in terms of quality and reliability, everyone is better off. If not, the alternative solution will take over.
I'm eager for this crates.io alternative to land, assuming they don't break too many projects in their improvements.
This was my first thought too. And there are a lot of questions that will get asked, like, will all crate library names start being prefixed as well? So you end up with
use::bar; // changing to
use::foo::bar;
I assume the library names that can be overridden in cargo would still be accepted, and then it all gets a little messy. The transition would be very messy.
Write some automated analysis that looks up popular packages on npm, pub.dev, rubygems, nuget. "Rustify" the package names. Add to it frequently used words, maybe popular names, etc. Then, write a script that creates an empty package, registers a name on crates.io every thirty seconds, and then you have about 20k package names after a week that nobody can use.
> Using an automated tool to claim ownership of a large number of package names is not permitted.
And
- Hey, I found that someone created crates at a rate of about one every 30 seconds for a period of a week straight.
- That's not sufficient proof of squatting.
Whoever answered that, was either supporting the squatter or explicitly in favor of the practice. I cannot conceive that someone would get that evidence in their hands, and in their right mind think that the claim is bogus. Hell, I'd even be willing to suppress the squatter with evidence of one new crate created every 30 seconds for one hour!
The only reasonable conclusion to make is that they didn't really care. But then don't save face and claim that you do. That's hypocrisy.
If you have a namespace, can't people just globally-claim namespaces instead? like serde/serde or something similar. I feel if you really don't want people claim whatever they want, you have to do the Java package style where namespaces are tied to domain names.
The inverse-style domain name thing does a really good job of removing the whole issue of squatting from the ecosystem. You have to demonstrate some level of commitment and identity through control of a domain name in order to publish.
I would also say that this puts just enough friction so that people don't publish dogshit.
crates.io demonstrates quite clearly that you either have to go all the way and take responsibility for curation or you have to get completely out of the way. There is no inbetween.
and i dont particularly think that using xml is that bad. The schema is well defined, and gives you good autocompletion in any competent IDE (such as intellij).
It took some iterations before maven 3 became "good", so people forget that it wasn't as nice before now! Unfortunately, it seems that the lessons learned there is never really disseminated to other ecosystems - perhaps due to prejudice against "enterprisey" java. Yet, these package managers are now facing the sorts of problems solved in java.
I have no problem with XML in general and even think it's still the better format for many things. But it's not really appropriate for a build config. Thankfully Maven now offers polyglot but I've seen no use of it in the wild.
100% agree. It's unbelievable what a PITA it is dealing with pip or npm compared to Maven even 10 years ago. The descriptors could get convoluted but you could also edit them in an IDE that knew the expected tokens to make things happen.
No you see Java devs have stockholm-syndromed themselves into believe that a giant stack of XML, or some unhinged mini-language are actually good, and much better than something the humans involved can actually read and parse easily and now to compensate with other ecosystems providing 85% of the functionality, with 5% of the pain, they’ve got to find some reason to complain about them.
Is this a joke? XML is horrible to work with, more boilerplate than information. Compare your average maven file to a cargo.toml and tell me which is easier to work with...
"XML is more verbose" is a lazy criticism in the same veign as "Python is better than Java because you can do 'Hello World' in one line".
Maven files have a simple conventional structure and a defined schema. Once you learn it, it's a breeze to work with. It's not like you need to write hundreds of lines of SOAP or XLST — which is actually why people started to dislike XML, and not because XML inherently bad.
Edit: I'd also take XML over TOML any day, especially when it comes to nested objects and arrays.
For a descriptor verbose is superior. It's way clearer what you're looking at. Matching a named end tag is much easier than matching a }. Also, XSD means you can strictly type and enumerate valid values and you will instantly see if you've written something invalid.
Maven stores every version of every library you've ever needed in a central location. It doesn't pollute your working directory and it caches between projects. And this is more of a Java thing than a Maven, thing, but backwards compatibility between versions is way easier to manage. There's no incompatible binaries because you changed the node runtime between npm install and running your project.
I love this lack of namespacing personally, because it means that whatever crate you see in a project is going to be the same as the crate your see in another one. Never need to alias crate names. It happens in Golang all the time and I really think namespacing packages was a mistake there.
Golang's problems aren't due to using namespaces, they're due to delaying too many decisions until too late.
Go has namespacing mostly because for a long time it didn't have a package manager at all, so people just used a bunch of ad hoc URL-based solutions mostly revolving around GitHub, which happens to have namespaces and also happened to lend itself to aliasing (because a whole GitHub URL is too long).
If you want to look at an actual example of namespacing done well, Maven/Java is the place to look. There is no aliasing—the same imports always work across projects.
Yet to see any proof that namespacing has made things better in other ecosystems, are go style links or other types of namespaced imports any less prone to supply chain risks?
It's definitely a good thing that people choose new unique names for crates rather than dijan/base64 vs dljan/base64
Do understand the desire of having a crate for audio manipulation called "audio" but at the same time how often do we end up with "audio2" anyway? It's an imperfect solution for an imperfect world and I personally think the crates team got it right on this one.
> Yet to see any proof that namespacing has made things better in other ecosystems
It's really as simple as this: many libraries are generic enough implementing something that already exists. Let's say you want a library to manage the SMTP protocol. On crates.io, of course someone has already taken the "smtp" crate (ironically, this one is abandoned, but has the highest download counts, because it's the most obvious name). Let's say you disagree with the direction this smtp crate has gone, and you make your own. What do you call it?
Namespaces solve this problem. You'd instead of have user1/smtp and user2/smtp competing in feature sets. You can even be user3/smtp if you don't like the first two.
This is precisely what Java enables too. The standard library is in com.java.*; if you don't like how the standard library does something, you can make com.grimburger.smtp and do it yourself. If you choose to publish to the world, all the more power to you. It doesn't conflict with the standard library's smtp implementation.
I don’t work with rust on the regular, but this is so annoying with package repositories in general. No don’t use http-server, it’s bad, instead you have to use MuffinTop, it’s better. And then you just have to know that. The concept of sanctioned package names would be interesting, but probably chaotic in practice as the underlying code behind this sort of alias changes over time. This will remain a part of being a domain expert in any given ecosystem forever I think, hooray!
Crates.io names are human-meaningful and everyone sees the same names, but it's vulnerable to squatting, spamming, and Sybil attacks.
You could tie a name to a public key, like onion addresses do, but it's unwieldy for humans. (NB, nothing stops you from doing this inside of crates.io if you really wanted)
You could use pet names where "http-server" and "http-client" locally map to "hyper" and "reqwest", but nobody likes those, because they don't solve the bootstrap problem.
It's a problem with all repos because when you say "http-server should simply be the best server that everyone likes right now", you have to decide who is the authority of "best", and "everyone", and "now". Don't forget how much useless crap is left in the Python stdlib marked as "don't use this as of 2018, use the 3rd-party lib that does it way better."
So yeah... probably will be a problem forever. As a bit of fun here are some un-intuitive names, and my proposed improvements:
- Rename Apache to "http-server"
- Rename Nginx to "http-server-2"
- Rename Caddy to "http-server-2-golang"
- Rename libcurl to "http-client"
- Rename GTK+ to "linux-gui"
- Rename Qt to "linux-gui-2"
- Rename xfce4 to "linux-desktop-3"
Then you only need to remember which numbers are good and which numbers are bad! Like how IPv4 is the best, IPv6 is okay, but IPV5 isn't real, and HTTP 1.1 and 3 are great but 2 kinda sucked.
Very simple. If a company as big as Apple can have simple names like "WebKit", "CoreGraphics", and "CoreAudio" then surely a million hackers competing in a free marketplace can do the same thing.
CPAN has the best model IMHO. Hierarchy that starts with categories. You build on top of the base stuff and extend it, rather than reinvent/fork something with a random name. Result is a lot more improvement and reuse, and more functionality with less duplication. Plus it's easy to find stuff.
Perl's whole ecosystem is amazing compared to other languages. It's a shame nobody knows it.
I don't think a lack of namespace is that much problem. Sure, it is often annoying, but people are creative enough to create a short enough and still available crate name for most cases. Namespacing only makes sense for a large group of related crates---and it wouldn't give much benefit over a flat namespace.
As other mentioned though, a typosquatting is a much bigger problem and namespacing offers only a partial solution. (You can still create a lookalike organization name, both in npm and in Github.)
I don't see the problem. Even with namespaces you'll have
brandonq/xml vs parsers/xml with no clue if one is better then another.
Also possibly with some confusion over whether things with the same name are forks or not. May make it a little more difficult to Google. Why not just have brandon_xml vs xml-parser and have a community list of best and most popular libraries?
I guess the only issue is that some generic/obvious package names are bad packages. That can have been avoided if they banned/self-squated most of the names. I suppose if you use dns namespaces and actually tie it to ownership of the domain name it might make sense but that would also cause issues(what if you forgot to renew the domain?).
The advantage is one of trust. If the `abc` developers build well known library `abc.pqr` are well trusted then I know I can use `abc.xyz` and everything else under the same namespace without (much) vetting.
We could even have `rust.xyz` for crates that are decoupled from `std` but still maintained by rust core devs such as `regex`.
> brandonq/xml vs parsers/xml with no clue if one is better then another.
You will have this problem only once. After an initial research you settle on one and move on with life.
I don't get how having to vet a dependency is going to be more difficult than before. The process is 99% the same, you still have to do the research work initially in both cases.
While Java made that notation famous, it was already used in NeXTSTEP and Objective-C, hence why you will see this all over the place in macOS and derived platforms, on configuration files and services.
> Perhaps my biggest critique is that crates.io has no namespacing. Anyone can just claim a global and generic package name and we mostly have to deal with it (unless you avoid using the crates.io repository, but then you'll probably have more problems...). Some of these globally-claimed generic packages are not really the best package to use.
This is true that with no namespace anyone can end up squatting a cool name, but with namespace you end up in an even worse place: no-one end up with the cool names, and it makes discoverability miserable, because instead of having people come up with unique names like serde, everybody just name their json serialization/parsing library json and user now needs to remember if they should use "dtolnay/json" or "google/json" (and remember not to use "json/json" because indeed namespace squatting is now a thing), and of course this makes it completely ungoogleable.
We've had the namespace discussion for hundreds of time in the various Rust town squares, and the main reason why we still don't have namespace is because it doesn't actually answer the problem it's supposed to address and if you dig a little bit you realize that it even makes them worse.
Having a centralized public and permissionless repository opens tons of tricky questions, but namespaces are a solution to none of them.
> everybody just name their json serialization/parsing library json and user now needs to remember if they should use "dtolnay/json" or "google/json"
What's the problem with that? You will have to explicitly state intention which is always a good thing.
> and of course this makes it completely ungoogleable
You are aware that just pasting username/projectname gives you exactly what you are looking for in the several top search engines, correct?
> We've had the namespace discussion for hundreds of time in the various Rust town squares, and the main reason why we still don't have namespace is because it doesn't actually answer the problem it's supposed to address and if you dig a little bit you realize that it even makes them worse.
OK, if you say so. Is this worse state of things documented somewhere?
Can’t really accuse programmers of being creative with words and phrases. Most of them will be rehashes of blogs and memes from the last couple of decades.
I feel like Rust finally broke the idea that programmers should be in complete control and completely conscious of everything the compiler is doing. It hasn't been that way in decades, compilers are freaking magic. But Rust undid a lot of that with borrowing. People became comfortable with the compiler knowing better than them. I just wish we could relax further: We should never be explicitly iterating forward over a collection unless we need this behavior for the algorithm. Things should be implicitly parallel. Etc. Give me rusty bash.
> We should never be explicitly iterating forward over a collection unless we need this behavior for the algorithm. Things should be implicitly parallel
There is already a crate providing parallel iterators. You just rename the iter() call and that's it. I don't agree it should be implicit though.
I think "not in control of what the compiler is doing" is overstating things a little bit. In some ways, Rust gives the programmer more control than C does. For example, Rust has standardized support for inline assembly, but inline assembly in C relies on vendor-specific extensions.
But to your point, the convenient defaults are very different. Unsafe typecasts require a lot of ceremony and careful thought in Rust, and they have to follow more rules than in C. In particular, references are all effectively "restrict" in Rust, and it's really easy to screw that up when you do unsafe cast from raw pointers to safe references, which is a big incentive not to write code like that if you have a choice.
I should've said "stabilized" instead of "standardized" to avoid stepping on this conversational landmine. But an important practical difference is that Rust supports inline assembly on Windows. (Correct me if I'm wrong, but I think MSVC mostly does not support inline asm.)
As someone who does a lot of unsafe rust, including tagged pointer foo, I strongly disagree.
If anything I want more explicit control, alleviated with an even more expressive type system. Ideally rusts type system would just be a prolog variant imho.
You're skipping the most needed improvement of all: fixing the issue where scripts with unquoted variables will seem to work until they contain a space. Almost every single bash script of more than a dozen lines I've seen outside of large open-source projects fails if a user puts a space in a file path, because the programmer didn't understand the insanity of variable quoting rules! I don't know of any programming language with a more common major footgun.
Now you have me curious - where do you see the need for floats in shell? You are more creative than myself, because I am struggling to come up with a situation where I would lean on this. A "path" type would be by far more useful to my work.
I have a script which uses wmctrl to tile windows when I press a shortcut key (for when I'm using an editor and want the editor window to take up most of the screen).
I guess I could avoid floats using multiplication, but I would rather keep the code as simple as possible.
We use a Kotlin scripting variant for our own shell scripting needs at Hydraulic, it's pretty nice because we've joined it with a lot of internal APIs that make working with the filing system and network easy. It fits all those requirements and more (you can declare flags easily at the top level, it has built in progress tracking for slow operations etc).
The Kotlin type system is comparable to Rust without the borrow checker. It has non-null types, generics, etc.
It's not really a "product" per se but there's an old version for download and some lightweight docs here:
We've never worked out what to do with it. Ideas and feedback welcome. It's pretty nice to use, albeit you need to use IntelliJ to edit scripts if you want IDE features.
That seems entirely opposite to my Rust experience.
Rust is quite transparent in what it does, and is very conservative with compiler magic. The language doesn’t do heap allocations, doesn’t do reference counting, it doesn’t even have implicit numeric type conversion. It won’t implicitly copy types that did not ask to be implicitly copyable, and even that is only legal for types that can be copied with a simple shallow memcpy.
Rust uses zero-cost abstractions all over the place, which means it’s predictable what code they will compile to, and that will be typically something simple. Std types have well-known basic layout, so you know that e.g. iteration over a Vec is going to be a loop incrementing a pointer, and there’s no implicit parallelism.
Describing borrowing as “compiler knowing better than the programmer” is a weird way of looking at it. Borrowing is like type checking. You declare a type to be temporary, and try to use it as long-lived, you get an error. It’s the same as if you declare function to return struct Foo, but returned struct Bar instead. Yes, compiler “knows better than you”, because you just wrote a bug.
Borrowing still compiles to direct pointer usage without a GC (it’s literally guaranteed to be identical to a C pointer in C ABI structs and functions), and you can override lifetimes with unsafe if you think know better than the compiler.
It has unpredictable performance because of lazy evaluation. Other high-level functional programming languages like OCaml, Standard ML, and Scheme can be compiled and achieve fairly high performance.
In what way? Rust (cargo technically) might have the best support for offline compilation of any language I use. `cargo vendor` does pretty much everything I want it to.
Also you can download documentation for offline using `cargo doc` so you can easily lookup stuff during a flight. Although you do need to figure out all your dependencies ahead of time so that you download their documentation before you leave.
Just FYI, `cargo doc` doesn't download the documentation. It re-generates the documentation from source. You only need the source code to generate documentation.
This is written by someone who has obviously written a lot of Rust code. I like its balance. It feels fair and not fanboyish like these write ups often do.
Learning when to invest in type constraints and when not to is an important lesson. It’s not unique to rust, though it might express a little differently. I’ve dealt with excessively typed c++ and excessively abstracted and typed Java and they have the same class of refactoring problems. I’ve also dealt with plenty of undertyped and under documented go, where there are specific values all over the place which turn into runtime footguns - and these can be truly dire to refactor as well, you get an earlier sense of progress but you ship bugs to users most of the time. There’s no magic answer to this set of trade offs. Rust gives you tools to mostly pick your place, on this specific axis it provides an unusually broad choice.
I never programmed in Rust, but I had enough experience in C programming to know that segmentation faults are annoying to debug. I heard Rust prevents memory management problems at compilation level, so it forces you to create safer program. Given this, I want to ask Rust programmers here: For people who have no experience in managing memory at coding stage (basically programmer who have no experience in C-like language), can such people appreciate what Rust aims to deliver?
disclaimer: I'm not a big rust programmer, but I appreciate it.
I think it's a matter of perspective/background. Depending what language you're coming from you might appreciate a lot of the more modern language features, or binary size, or performance. On the other hand, you might already be working with a pretty cutting edge language with a garbage collector or other and decide that Rust has some cool benefits but not worth the switch.
I heavily appreciate the simplification of the memory model. I was able to go from comfortable with typescript/js to comfortable with rust with essentially zero knowledge of manual memory allocation in just a few months.
In that scenario Rust can be attractive because of its high performance compared to languages with a garbage collector. No matter what Java fans will tell you, any runtime with a GC will have worse overall performance for most large codebases than a compiled language with no GC. Sure, microbenchmarks might not look too bad, but complex software is a different matter.
Rust is great for services such as API endpoints, where low latency and high throughput at a low cost are more important that hot reload during development.
I think so, I wrote an entire video game from scratch in Rust and my experience has been absolutely delightful. I have to manage nulls via Options, cast things explicitly, cannot do dumb stuff like modifying an array while iterating it, etcetera. It’s guiding my code in ways that I wasn’t able to myself, and eventually leading to essentially what is a bug-free game. Imagine!
The experience has been far more than just borrow checking, it’s opened up my mind to a lot of concepts that my previous trials with Java, JavaScript, and Python fully obscured in arcane ways. Rust’s compiler is wonderful, and once you get past the ergonomic struggles (which I did by just going through Advent of Code), you’re set.
For people who've never written C or C++, Rust's biggest selling point is usually performance. Porting code from Python to Rust for example often gives 10-100x speedups right off the bat. You could get the same speedups by porting to C too, but Rust lets you do that without giving up the memory safety and package management convenience that you're used to.
Even when you don't care about performance, another issue that comes up sometimes is keeping track of mutable state. If you've ever relied on bytes instead of bytearray or tuple instead of list* to guarantee that no mutation is happening in some Python code, you know what I'm talking about. Rust can give you a similar level of control over who gets to mutate what, without making you change the type of your data or pay the cost of copies. It's basically the const/non-const distinction from C, but much stricter. Another way of saying the same thing is that Rust gives you a lot of the legibility/correctness benefits that you'd expect from a functional programming language, but you get to code in the usual imperative style.
* And even then, that only prevents assignment to the elements of the tuple. You can still mutate an element internally if it's not also an immutable type.
>> Porting code from Python to Rust for example often gives 10-100x speedups right off the bat.
As long as you don't do a couple of things that, while easy to remember not to do once you know them, a programmer used to languages like Python and Javascript might be oblivious to. Things like:
* Forgetting to use buffered IO wrappers
* Using println!() for high-volume console printing rather than locking stdout and writing to it manually
I’m not sure. Looking through different Rust libraries, I tend to see three different styles, depending on previous programming experience. Each mimics the prior experience, with benefits.
* Everything is a struct with concrete types. This is closest to C. It benefits from the improved memory safety, without sacrificing speed.
* Everything is a Box<Rc<T>>. This is closest to dynamic garbage-collected languages, but with vastly improved performance.
* Everything is an “impl Trait”. This is closest to templated C++, but with much better ergonomics.
So, while I think I agree with your specific statement that the improved memory management may only be fully appreciated by those that cut their teeth on C’s segfaults and dangling pointers, I don’t think that’s the only benefit of Rust as a language.
Many garbage collectors (e.g. CPython’s) will use reference counting as part of the process. If a reference count hits zero, the object can be destructed, prolonging the time between mark-and-sweep passes.
The performance difference, though, is mainly in switching over to a compiled language in the first place.
Mostly to mimic the data structures allowed in languages where objects are held by reference. In Python, I could write a tree structure as `namedtuple(“node”, [“lhs”, “rhs”])`. If I tried to write a similar structure in Rust as `enum Node{Leaf, Branch(Node, Node)}`, the compiler rightfully complains that it would have an infinite size. But the indirection introduced by Box would let it be stored as `enum Node{Leaf, Branch(Box<Node>, Box<Node>)}`.
In my opinion, until you have spent a lot of time debugging C/C++/assembly issues (memory corruption, null pointer crashes, segfaults, build system problems, exception inception, etc), Rust would probably seem like a total waste of time.
If you're used to e.g. OCaml, and the problem you're solving does not require avoiding GC, then Rust is a more cumbersome language for no real benefit.
If you've never used an ML family language then Rust might still be a breath of fresh air even if you don't care about memory management.
I dunno if it's just me but coming from typescript and c# background it honestly wasn't that hard to grasp things, Maybe it's just me but it seems so much easier to grasp than C.
It really depends on what your previous programming experience is, if you can state which languages you know now, someone can give a better response to this.
> I started writing tests in Rust as I would in any other language but found that I was writing tests couldn’t fail.
This is a common refrain in C++ testing: if it compiles then it's probably correct.
> Rust has accounted for so many errors that many common test cases become irrelevant
In practice, if you think this way I think it's a sign that you aren't testing the right things in those other languages. You should be testing business logic, not language stuff.
If you look at your test code and think "I would test this in JavaScript, but I don't need to do that in Rust" then just delete the test.
I’ve definitely experienced it in Haskell and Rust. I can believe some C++ could be that way, but I’ve never experienced it, but then again those projects didn’t have useful tests either. I think with C++ a lot of this depends on domain and the quality of the code and libraries.
A null pointer exception is a bug that breaks business logic. There's no "business logic instead of language stuff" because the language stuff is the foundation that business logic rests on. If you don't test against failure modes, what's even the point in testing?
To close the loop, Rust doesn't include a `null` type and you wouldn't encounter something comparable in idiomatic Rust (because you'd be using eg Option::map to handle None cases gracefully), so this is a class of test that would be common in Java and C that is close to irrelevant in Rust.
To be more specific, Rust does have among other things:
std::ptr::null() - an actual null pointer, probably the zero address on your hardware, and this isn't even an unsafe function. On the other hand, you won't find many uses for a null pointer so, I mean, congrats on obtaining one and good luck with that.
std::ptr::null_mut() - a mutable null pointer, similarly unlikely to be of any use to you in safe Rust, but also not an unsafe thing to ask for.
But, these are pointers, so they're not values that say, a String could take, or a Vec<String> or whatever, only actual raw pointers can be null.
Sure, you can panic unwrapping a None, but there are two important distinctions.
First, this is a controlled panic, not a segmentation fault. The language is ensuring that we don't access the null pointer, or an offset from the null pointer. Null pointer access can be exploitable in certain circumstances (eg, a firmware or kernel). Your use of "exception" suggests you're thinking about it in Java terms however, and Java is equivalent here.
Second, you can only encounter this in explicit circumstances, when you have an Option<T>. Wheras in languages with a null type, any variable can be null implicitly, regardless of it's type.
>First, this is a controlled panic, not a segmentation fault.
But a segmentation fault is also controlled
>Your use of "exception" suggests you're thinking about it in Java terms however, and Java is equivalent here.
No, I am thinking in Delphi terms. It is overspecialized to Windows userspace. Windows gives an Access Violation, and that can be caught, and Delphi throws it as exception
>Second, you can only encounter this in explicit circumstances, when you have an Option<T>. Wheras in languages with a null type, any variable can be null implicitly, regardless of it's type.
Delphi has both nullable types and null-safe types
A segmentation fault may not happen (which is to say, we may corrupt memory or worse) if the null pointer is mapped into memory (as in an embedded or kernel context) or if it's accessed at a large offset, which results in a pointer that's mapped into memory. This may be rotten luck or it may be exploited by an attacker.
> No, I am thinking in Delphi terms.
Fair enough, I don't know Delphi. I'll take what you're saying about it as read.
A null pointer exception is a free runtime check. I really don't understand the fuss about null. The "most expansive design mistake in computer science" and whatever.
Free runtime check of what? The point is that if nulls don't exist, there is no need for a "check". Runtime or otherwise. If I write a type that models Foo, I want it to model Foo and not "Foo or null". If I want to model "Foo or no data" then I use a separate type that makes my intention clear (in Rust spelled `Option<Foo>`). Languages where nothing is precisely Foo and everything is "oh by the way this is only possibly Foo" are deeply, deeply flawed.
If an exception breaks business logic, then you can test the code by testing business logic.
How do null pointer exceptions arise?
Inconsistent data: you have code paths that implicitly assume invariants. Trivial cases like “this field is not provided” and more complex ones like “these fields have a specific relationship”.
You can often move those assumptions into the data structure in any language.
Then your tests become a matter of generating and transforming data from a holistic perspective instead of micromanaging individual code paths.
There is surely something wonderful about the kind of C++ programmer who figures that, since their unusable broken garbage compiled it's probably correct.
Remember unlike most languages you'd be familiar with C++ has IFNDR, which has been jokingly referred to as "False positives for the question: Is this a C++ program?". A conforming C++ compiler is forbidden from telling you in some† unknown number of cases that it suspects what you've written is nonsense, it just has to press on and output... something. Is it a working executable? Could be. Or maybe it's exactly like a working executable except it explodes catastrophically on Fridays. No way to know.
† The ISO standard does identify these cases, but they're so vague that it's hard to pin down everything which is covered. My guess is that all or most non-trivial C++ software is actually IFNDR these days. Just say No to the entire language.
> A conforming C++ compiler is forbidden from telling you in some† unknown number of cases that it suspects what you've written is nonsense, it just has to press on and output... something.
It's not quite that bad - a conforming C++ compiler is permitted to error out and not compile the program. It just doesn't have to.
Many of the cases of IFNDR are semantic constraints, especially in C++ 20 and beyond. As a result of being semantic constraints it's generally impossible to diagnose this with no false positives. The ISO standard forbids such false positives so...
Can you give a more concrete example of the kind of thing you're talking about? Like, if you try to sort something and your comparator implementation for that type is not transitive, the compiler can silently produce a broken binary?
Surely in the undecidable cases the compiler is allowed to produce a binary that errors cleanly at runtime if you did in fact violate the semantic constraint, and any sane implementor would do that. (Not that any sane person would ever write a C++ compiler...)
> Like, if you try to sort something and your comparator implementation for that type is not transitive, the compiler can silently produce a broken binary?
It's not merely about whether your comparisons are transitive, the type must exhibit a total ordering or your sort may do anything, including buffer overflow.
> Surely in the undecidable cases the compiler is allowed to produce a binary that errors cleanly at runtime if you did in fact violate the semantic constraint,
I don't think I know how to prove it, but I'm pretty sure it's going to be Undecidable at runtime too in many of these cases. Rice reduced these problems to Halting, which I'd guess means you end up potentially at runtime trying to decide if some arbitrary piece of code will halt eventually, and yeah, that's not helpful.
I've written about it before, but I should spell it out: The only working alternative is to reject programs when we aren't sure they meet our constraints. This means sometimes we reject a program that actually does meet the constraints but the compiler couldn't see it.
I believe this route is superior because the incentive becomes to make that "Should work but doesn't" set smaller so as to avoid annoying programmers, whereas the C++ incentive is to make the "Compiles but doesn't work" set larger since, hey, it compiles, and I see Rust's Non-Lexical Lifetimes and Polonius as evidence for this on one side, with C++ 20 Concepts and the growing number of IFNDR mentions in the ISO standard on the other side.
> the type must exhibit a total ordering or your sort may do anything, including buffer overflow.
Sure. But there's no requirement for the compiler to be a dick about it, and hopefully most won't.
> I don't think I know how to prove it, but I'm pretty sure it's going to be Undecidable at runtime too in many of these cases. Rice reduced these problems to Halting, which I'd guess means you end up potentially at runtime trying to decide if some arbitrary piece of code will halt eventually, and yeah, that's not helpful.
At runtime can't you just run the code and let it halt or not? Having your program go into an infinite loop because the thing you implemented didn't meet the requirements is not unreasonable.
I'm sympathetic to the idea that there could be a problem in this space, but without a real example of a case where it's hard for a compiler to do something reasonable I'm not convinced.
> I've written about it before, but I should spell it out: The only working alternative is to reject programs when we aren't sure they meet our constraints. This means sometimes we reject a program that actually does meet the constraints but the compiler couldn't see it.
> I believe this route is superior because the incentive becomes to make that "Should work but doesn't" set smaller so as to avoid annoying programmers, whereas the C++ incentive is to make the "Compiles but doesn't work" set larger since, hey, it compiles, and I see Rust's Non-Lexical Lifetimes and Polonius as evidence for this on one side, with C++ 20 Concepts and the growing number of IFNDR mentions in the ISO standard on the other side.
Meh. I'm no fan of the C++ approach, but I'd still rather see C++ follow through on its strategy than half-assing it and becoming a watered-down copy of Rust. People who want Rust know where to find it.
Historically correctness wasn't seen as an important goal in C++ and so no, I don't think any of the three popular C++ stdlib implementations will do something vaguely reasonable for nonsense sort input. It's potentially faster (though bad for correctness) to just trust that this case can't happen since the programmer was required to use types with total ordering. So yes, I'd expect it to result in bounds misses in real software.
I wouldn't think you could get bounds misses without doing extra comparisons, so I'd expect real-world sort implementations to just fail to sort, which doesn't seem particularly unreasonable. But in any case, it's a huge leap from "existing C++ stdlib implementations behave badly in this case" to "the C++ standard requires every implementation to behave badly in this case".
Can you give an example of non-C++ code that a modern compiler (MSVC, clang, g++ or something) successfully compiles with no diagnostics? I’m genuinely curious. If not, this just sounds like more C++ FUD because the spec doesn’t define everything under the sun and allows a certain amount of leeway to compilers for things like emitting different error diagnostics.
Per the C++ standard ([lex.name]/3), this program is ill-formed:
> In addition, some identifiers appearing as a token or preprocessing-token are reserved for use by C++ implementations and shall not be used otherwise; no diagnostic is required. [...] Each identifier that contains a double underscore __ or begins with an underscore followed by an uppercase letter is reserved to the implementation for any use.
Thus, the compiler theoretically has the liberty to emit whatever it wants for this program.
Neither GCC nor Clang produces a warning under -std=c++20 -Wall -Wextra (Clang only produces a -Wreserved-macro-identifier under -Weverything), and MSVC doesn't produce a warning under /std:c++20 /Wall.
In practice, most examples of ill-formed programs where compilers issue no warnings occur with discrepancies between different source files that are linked together; e.g., declaring a function as inline in one file but non-inline in another, or declaring a function with two different sets of default arguments in different files, or defining the same non-inline variable or function in different files, or defining the same inline function differently in different files.
Don't forget since C++ 20 all programs which rely on a concept but don't fulfill the (unchecked) semantic constraints of that concept are ill-formed. This allows C++ to truthfully say that C++ programs have the same properties as Rust programs in this regard, because although most real world C++ may fail those constraints and they aren't checked, as a result those aren't technically C++ programs at all so the fact they're broken and don't do what their programmers expected is magically not the fault of C++ even though the compiler seemed fine with it.
> Thus, the compiler theoretically has the liberty to emit whatever it wants for this program.
In theory, sure. In practice, what does it do? We can look and see.[0][1][2] (I don't know of any compiler that emits garbage in the example you listed). For some reason, there's this sentiment that's arisen that treats undefined behavior as some sort of bugaboo that's capable of anything and everything (including summoning nasal demons).
Here's a question that I can't find an answer to. Is machine code well-defined (as in, can it contain undefined behavior)? If the answer is yes, then all Rust programs can also contain undefined behavior, because they eventually turn into machine code after all. If the answer is no, then that means once a C++ program is compiled into machine code, the executable is a well-defined program.
This whole nasal demon garbage was a good hyperbole at the time to explain that, yes, undefined behavior is bad. But it's been taken to this extreme where people will use arguments like this to try and convince others that if your program has undefined behavior, then it could summon a nasal demon, because who knows what could happen.
In reality, that won't happen. In reality, a signed integer overflow can result in a meaningless program, or a security vulnerability, or many other bad things, but it doesn't mean that the compiled machine code is all of a sudden inventing new instructions at random that the CPU will happily accept. It doesn't mean that your program will turn your OS into a pac-man game. It doesn't mean that it's impossible to find the root cause of the issue and remove the undefined behavior.
In practice, undefined behavior means that your program is broken, but you can look at the compiled program and trace exactly what it does. Computers are machines, they behave predictably (for the most part). They fetch an instruction, execute the instruction, and read/store results to memory. If you look at the instruction and memory (the cache stored on the specific core that the processor is using), you can reliably predict what the CPU will do after executing that instruction.
And yes, multithreading exacerbates the effects undefined behavior and makes it more difficult to debug. But you can debug the issue, even if it means you have to look at the machine code directly.
So while, yes, a compiler can emit garbage when it encounters the example program you gave, using that as an argument for why undefined behavior is bad is dumb. Because in reality, compilers will do something that makes sense, and if it doesn't, you can just look at what it produced. In all the examples you listed, I'm sure that if you created a small example illustrating those situations, the compiled program would do what you expect it to. (Like in the case of having different default args, it will probably just throw an error like this[4]).
Sorry for the rant, it just annoys me that we can't discuss undefined behavior without somebody making an argument that doesn't matter in virtually every case. Undefined behavior is bad! But there are good, real, common examples that show why it's bad! Use those instead instead of talking about how compilers are technically permitted to spew trash when they encounter a program that looks normal, because I'd be very surprised if any modern compiler exists that does spew trash for a normal looking program.
This talk by Chandler Carruth seems pretty good on explaining the nuance of undefined behavior[5].
First of all, and most importantly, we're not talking about Undefined Behaviour, which happens at runtime, but about IFNDR (Ill-formed, No diagnostic required), which means at compile time your program has no meaning whatsoever because it's not a well-formed C++ program after all but your compiler doesn't tell you (and because of Rice's Theorem in many cases cannot possibly do so as it can't determine for sure itself)
This is a conscious choice that C++ made, and my belief is that once you make this choice you have an incentive to make it much worse over time e.g. in C++ 11, C++ 14, C++ 17, C++ 20, and now C++ 23. Specifically, you have an incentive to allow more and more dubious code under the same rule, since best case it works and worst case it's not your fault because if it doesn't work it was never actually a C++ program anyway...
Still, since you decided to talk about Undefined Behaviour, which is merely unconstrained misbehaviour at runtime, let's address that too.
For the concurrency case no, humans can't usefully reason about the behaviour of non-trivial programs which lack Sequential Consistency - which is extremely likely under Undefined Behaviour. It hurts your head to even think about it. Imagine watching an avant garde time travel movie, in which a dozen characters are portrayed by the same actor, the scenes aren't shown in any particular order and it's suggested that some of them might be dreams, or implanted memories. What actually happened? To who? And why? Maybe the screenwriters knew but you've no idea after watching their movie.
Today a huge proportion of software is processing arbitrary outside data, often as a service directly connected to the Internet. As a result, under Undefined Behaviour the unconstrained consequences may be dictated by hostile third parties. UB results in things like Remote Code Execution, and so yes, if they wanted to the people attacking your system certainly could turn it into a Pac man game.
We should strive to engineer correct software. That we're still here in 2023 with people arguing that it's basically fine if their software is not only incorrect, but their tools deliberately can't tell because that was easier is lunacy.
> First of all, and most importantly, we're not talking about Undefined Behaviour, which happens at runtime, but about IFNDR
From the link I posted, from cpp reference, which gives a definition for what constitutes undefined behavior:
>> ill-formed, no diagnostic required - the program has semantic errors which may not be diagnosable in general case… The behavior is undefined if such program is executed
And as for the rest of your argument, you still haven’t answered the one question that matters. Is an executable file with machine code well-defined? If it is, then once you compile a C++ program, the generated binary is well-defined. And once again, all the superfluous arguments about what could happen are irrelevant when there’s no context provided. There’s lots of different types of undefined behavior. An integer overflow by itself does not make your program susceptible to remote code execution. It’s the fact that the integer overflow gets stored in a register that later gets used to write to invalid memory, and then that invalid memory has harmful code injected into it that gets executed.
We should strive to engineer correct software. I agree. It’s all this hand waving about how all undefined behavior is equal that irks me. Because it’s not true, as evidenced by the example listed above about the program that’s “technically” ill-formed C++, but in practice compiles to a harmless program. Engineering requires an understanding of what could go wrong and why. Blindly fretting about all undefined behavior is useless and doesn’t lead to any sort of productive debates.
Engineering is all about understanding tradeoffs. Which is one of the reasons the C++ spec does not define everything in the first place. The fact that in 2023 people don’t understand that all decisions in software come with a tradeoff is lunacy. And once again, I agree that undefined behavior is bad, but to pretend that the existence of IFNDR means the whole language is unusable is silly. People use C++ everyday, and have been for almost 40 years. I’d like more productive conversations on how rust protects the user from the undefined behavior that makes an impact, not about how it doesn’t have IFNDR because that’s irrelevant to the conversation entirely.
> the one question that matters. Is an executable file with machine code well-defined?
While you've insisted that's somehow the one thing which matters I don't agree at all. Are programmers getting paid to produce "any executable file with machine code" ? No. Their employer wanted specific executables which do something in particular.
And there aren't "different types of undefined behavior" there's just one Undefined Behaviour. Maybe you've confused it with unspecified behaviour ?
In the integer overflow case, because that's UB (in C++) it's very common for the result to be machine code which entirely elides parts which could only happen under overflow conditions. Because overflow is UB that elision didn't change the program meaning and yet it makes the code smaller so that's a win - it didn't mean anything before in this case and it still doesn't - but of course the effect may be very surprising to someone like you.
The excuse that "We've done it for decades therefore it can't be a bad idea" shouldn't pass the laugh test. Did you notice you can't buy new lead paint any more? Asbestos pads for ironing boards? Radium dial wristwatches? "That's a bad idea, we shouldn't do that any more" is normal and the reluctance from C++ programmers in particular shows that they're badly out of touch.
As an outsider, I often hear about async Rust being less than ideal. Perhaps I don't understand, because I haven't dipped my toes in the water yet... but I do most of my work in Kotlin with Coroutines, and concurrency is everywhere in the UI. I can't imagine working in a language having a major deficit in this space.
Are there any efforts to overhaul or completely rethink this?
I mean... that's not really the pain that people are referring to when they are struggling with async Rust, especially when you compare it other languages like JavaScript that also have a difference between async and regular functions.
It’s not just that. There’s a few big problems with Rust async aside from the normal coloring problem that is inherent to async and not worth talking about.
* The async runtime and async functions are decomposed but tightly coupled. That means while you could swap out runtimes, a crate built against one runtime can’t generally be used with another unless explicitly designed to support multiple. I believe C++ has a similar problem but no other major language I’m aware of has this problem - that’s typically because there’s only a single runtime and it’s embedded in the language. Things like timers and I/O are not interoperable because the API you use has to be for the runtime you’re running under. I believe there’s work ongoing to try to remedy this although in practice I think that’s difficult (eg what if you’re using a crate relying on epoll-based APIs but the runtime is io_uring).
* async in traits. I believe that’s coming this year although the extra boxing it forces to make that work makes that not something 0-cost you can adopt in a super hot path.
* async requires pinned types which makes things very complex to manage and is a uniquely Rust concept (in fact I read a conceptually better alternative proposal for how to have solved the pin/unpin problem on HN not too long ago, but that ship has long sailed I fear).
* The borrow checker doesn’t know if your async function is running on a work stealing runtime or not which means there’s a lot more hoop jumping via unsafe if you want optimal performance.
* async functions are lazy and require polling before they do anything. That can be a bit surprising and require weird patterns.
Don’t get me wrong. The effing-mad crate is a fantastic demonstration of the power of algebraic effects to comprehensively solve coloring issues (async, failability, etc). But I think there’s stuff with runtime interop that’s also important. I don’t think anyone is yet seriously tackling improving the borrow checker for thread per core async.
Unison [1] has algebraic effects as a first-class feature. They call them "abilities" there. You can make async about as transparent/opaque as you want it. Docs on abilities in [2].
I don't know that anybody is running a company on it yet, no. They've got a couple big milestones coming down the pike, which might make it more attractive to people.
There's efforts to overhaul it, basically there's a lot of stuff that has to be more complicated (as you'd expect) if you want to add support for async somewhere, and understanding the error messages can be pretty mentally taxing when you get it wrong; and then there's a lot of cases where the async version of some pattern just doesn't work right now due to limitations in the compiler or the language itself, and hopefully it'll work someday, but in the meantime you need to use workarounds.
For me I am working on a side project in Rust and had very few issues, Rust felt quite straight forward up until the point where I had to mess around with async. For my domain I need to use a framework that expects you to set up a shared mutable state for the connection pooling in an opinionated way. Suddenly I encountered arc and mutex and very cryptic trait error messages for code doing way too much magic. All I wanted was to share a SQLite connection in an ugly way to get my mvp out but I was dealing with absolutely incomprehensible trait issues.
In general I’ve noticed that there are a lot of code generation modules that can generate very cryptic error messages if you stay clear of their happy paths.
You would run into the same issues if you tried to do it with threads and rust too.
Any kind of shared resource is a tedious experience, that's basically it's selling point.
It is frustrating becasue if you go look for help, people are always condescending about global shared resources like that but there really isn't a better way.
Here's a better explanation by somebody smarter than me about why Rust chooses what it does, what else you could choose and what the price is: https://without.boats/blog/why-async-rust/
tl;dr: You can have different (nicer to program) abstractions, but you can't have them in the same language you use to write firmware for a $10 electronic device, or Linux drivers, and we already have languages like Go and Javascript whereas we did not have a safer alternative to C++.
It feels like most of the hatred for rust async is for people that try to act like Tokio isn't async rust and that for some reason you should try to randomly avoid tokio for some reason.
Maybe because you want to keep the dependency tree under control and bringing in tokio suddenly adds fifty crates you never asked for. Every crate is a potential liability!
Side comment but I looked at the documentation, github repo and even code for wick and I still have zero idea what it does. And I am rust developper full time...
"Programming in Rust is like being in an emotionally abusive relationship. Rust screams at you all day, every day, often about things that you would have considered perfectly normal in another life. Eventually, you get used to the tantrums. They become routine. You learn to walk the tightrope to avoid triggering the compiler’s temper. And just like in real life, those behavior changes stick with you forever."
This is a pretty negative and unfortunate take on rust that I cannot recognize after having spent the past 2 years professionally writing backend rust.
I did write c++ before that so it's likely that it takes some experience with the pain of an unhelpful (and in some cases downright hostile) to fully understand and appreciate the so called "yelling" rust does.
From my perspective every lifetime complaint the compiler has is a deeply appreciated hint that not only saves you countless hours of debugging down the line but also makes me feel confident/safe in the code which you never could achieve in a c++ codebase without making defensive copies everywhere.
Being able to lend/borrow data without fear truely is rust's greatest strength and something people coming from GC languages have a hard time appreciating.
It's a bit of an oversimplification, but a little quip I've used when talking about developing in Rust vs. other languages is, it's a question of where/when do you want the pain. In Rust, it's at development time (and hiring and ramp-up time); in C++ it's at runtime; and with GC languages it's at billing time when you have to pay for that extra compute and RAM. There's no way to get rid of the pain entirely.
Yes but there’s a difference in who feels the pain. Compile time pain is felt by devs, who have the constitution and ability to fix such issues. Runtime pain is felt by users and can result in data loss and security issues in the wild, and is harder to debug.
One kind of pain has a bigger blast radius than the other. I prefer compile time pain to runtime pain.
Sometimes billing time is your customer's patience, their phone battery, or other things like that which make your product worse and give your competitors an edge.
>...which make your product worse and give your competitors an edge
I would find that argument passable if some of the biggest products from the largest companies functioned better. Performant software seems like a distant memory.
And yet the big tech companies are perpetually re-writing their whole stack, or going out of their way to create new compilers for their language in order to lower their billing time.
Salaries are dominant in early stage startups, and infrastructure costs becomes dominant as you scale (also probably a SaaS-centric oversimplification). You can get into trouble in the middle, where you have enough scale to be paying jaw dropping cloud bills, but you don't have the staff to move to a cheaper architecture and you may still need to focus on getting new features out to drive growth (or to attract investment).
In startups that chasm is generally filled with VC money paid to cloud vendors, but if you're bootstrapping or not looking to be a unicorn, you probably will want to drive down infrastructure costs much earlier.
I tend to agree with you, but your aside about it being Saas-centric is really important.
One need only open Microsoft Teams to see the “billing time” costs. If the billing time cost is not paid by the developers or their company, then it stop being a cost to them and it becomes an unpriced externality. Who cares, who even knows, that this app we’re writing runs slowly on regular people’s old hardware.
Indeed, I’ve been writing rust professionally since 2015 (yes before 1.0) and I would consider the first encounter with the compiler as maybe it yelling at me, because I was used to C++. But it actually taught me a lot about why the code I was writing may have seemed okay at first glance, but actually introduced a tricky corner case bug that I hadn’t considered. The borrow checker got it tho, and I had to rearchitect my code to avoid that edge case. Now the code is better, free of that bug, and I won’t make that mistake in the future.
Was that the result of a process in which the compiler “abused” me? I don’t think so.
>From my perspective every lifetime complaint the compiler has is a deeply appreciated hint that not only saves you countless hours of debugging down the line but also makes me feel confident/safe in the code
Meh, the fact that Rust's compiler is overly strict is just a truth (All compiling programs are valid, but not all valid programs are compiling), and I find it lowkey annoying that every time someone has a problem with it, rustceans jump to the defense to the point of almost denying the OPs experience.
I mean on a case by case basis it might simply be a difference of experiences and both are valid, but on a larger scale it does look like a bit of a pattern, at least to me.
Ofcourse, there restrictions aren't just a whim of a Rust core team (usually ;p ), and do come from practical limitations but regardless, it's fair to be frustrated at them.
Did you mean non-GC languages? GC languages by default have at least memory safety because the GC frees for you when an object is detected to no longer be accessible.
Its wild how different the perspective is depending on your background
The author also wrote
>[Rust's] dev tooling leaves much to be desired
I can't even begin to comprehend such a statement. Rust's tooling ecosystem is the number one reason why I want to convince my colleagues to give it a shot. Its literal heaven coming from C++ with cmake...
In general, the error messages are also nice and offer suggestions on how to fix the problem. Far from being yelled at, it's more akin to pair programming. "Try doing this..." "Did you mean to do this?"
Do you need memory safety? Then why use Rust when you could use Java, JS, Python, etc? You can't "disable" memory safety in those languages.
Do you need bare metal performance? Then why use Rust when you could use C or C++, which have much larger ecosystems, platform support, more mature tooling, etc.
Do you need BOTH memory safety and baremetal performance at the same time? Then there really aren't many other options besides Rust.
But what I've started wondering lately is: are there really that many situations where you actually need both of those things at the same time? C++ tends to get misused a lot, but I feel like the same thing is happening with Rust.
If you're remotely sane, you'll always want memory safety. That's not really an optional property, because even in C/C++ compromising memory safety throws you straight into the land of nasal demons. The question is whether you want the compiler to ensure memory safety, or to do it completely on your own without language support.
Personally, I've never met anyone who can ensure memory safety in their C/C++ without onerous restrictions even more severe than those rust imposes, but maybe you're superhuman.
And saying c++ has more mature tooling seems insane too. Anyone who has had to mess with depedency trees using git submodules and cmake, conan, bazel, vcpkg, and meson would not call the tooling, oh and hunter too!, mature.
I'm willing to admit that is likely, could you enlighten me?
I've done professional c++ for a couple decades. My pipelines usually were conan with some dependencies that were not. The used clang tidy and format, san, cppcheck, coverity and tested release and debug builds with clang and gcc. Coverage was sometimes just clang. But sometimes gcc as well. Tests were usually google test. The team standard IDE was vscode.
Rust has rustfmt which is one standard. It has clippy which in addition to actual defects it forces commin idioms so code looks familiar. It has san and llvm coverage. It has miri. It supports aflop for fuzzing. And it has a single tool for package management and build. Edit: IDE is still vscode and it works great.
The only tooling missing for rust would be formal analysis, which is in work (and which isn't that great a story on c++ either) and gcc. A gcc front end is in work, and there is mrustc to generate c from rust, but yes, gcc front end support would be nice to get all those extra targets without the extra work.
A full 70% of security vulnerabilities are caused by memory safety issues. As professionals we need to get serious and have memory safety as a baseline requirement.
So we shouldn't use Rust at all, since Rust is not completely memory safe since it let's you disable the borrow checker. We should be serious as professionals and use safe languages like Java, JS, Python, etc.
But of course there are use cases where you need memory safety guarantees and bare metal performance. In those cases, sacrificing some memory safety by using Rust is an acceptable tradeoff I think.
All memory-safe languages are built on an unsafe foundation. The Java HotSpot VM is very unsafe C++. That's just how computers work.
You aren't meaningfully sacrificing memory safety by using Rust because the unsafe sections are clearly marked out. That's similar to unsafe C#, Python with ctypes, and many other memory-safe languages.
> Rust is not completely memory safe since it let's you disable the borrow checker.
No, it doesn't. What's interesting isn't so much that random HN posters believe this sort of thing, because hey, who needs to know anything about a topic to post their opinion on a forum right? No, what's fascinating is that this applies to people like Herb Sutter in his "cpp2" language, here's Herb:
> I don’t like monolithic "unsafe" that turns off checking for all rules
Nobody does that. It's possible Herb knows that and is being deceitful but honestly I think it's more likely that without investigating at all Herb has just decided everybody else is an idiot...
I don't consider myself a Rust expert, but this feels like a semantic argument? You can use `unsafe` to erase lifetimes, right? I'm not saying it's a good practice, and it's not globally disabling the borrow checker or anything like that, but in theory it's possible to produce unbounded lifetimes that are incorrect. In practice it probably doesn't happen very often. Certainly less often than UB in C or C++
The exact same code, without the transmute (just returning x directly from an unsafe block), fails a type check related to borrowing. Specifically, the compiler can't see why (without a transmute) it should believe that the reference x now has a different lifetime 'b instead of 'a. Which is fair because it doesn't.
The transmute is you claiming it does, and since transmute is an unsafe function that's entirely on you to make sure you're right about that. So, lying has the expected effect.
The borrow checks aren't magic, they can't look into your soul - but they are running inside unsafe code too.
* Edited to make explicit that the alternative is still marked unsafe and yet doesn't work
Marking code unsafe lets you write memory-safe code that the borrow checker can't verify. That increases the risk of bugs but it is not the same as disabling memory safety. It is a subtle distinction.
Don't forget safety from data races. Rust's borrow checking provides much stronger protection against those than java, go, c# etc. So we shouldn't use them either.
Beyond memory safety, Rust's other advantages over C++ are simply that it has a far better type system (by nature of ADTs), cleaner and more consistent syntax, cleaner tooling overall, and a far better module/modularity story.
That, and the borrowing/safety aspects of Rust also shake out into threading/concurrency, where the compiler does a pretty good job of forcing you to adopt better practices in regards to state sharing, locking, concurrency.
Cargo, though... I am coming around to thinking isn't great -- workspace support remains half-assed, especially... and crates.io is amateur-hour for reasons many people have pointed out elsewhere on this post. I have been tempted to switch my personal projects to bazel, with 3rd-party deps checked in/submoduled.
In addition to memory safety and performance, Rust offers something that, as far as I'm aware, no other mainstream languages offer: protection from data races.
And it offers those things along with a fantastic type system, and great tooling.
I continue to be delighted by indications that real high level languages compiling to Webassembly are supplanting the godforsaken nightmare of JavaScript. I think anyone focusing solely on JS is overdue for looking at those languages and getting up to speed on the huge differences before they wake up to find that the market for making and fixing kerosene lamps has been replaced by fusion reactors.
It's only mentioned off hand in one sentence at the end, but have people found that Rust is hard to hire for? In my experience it's relatively easy to filter for good candidates when you're hiring for Rust, whereas, just as a point of contrast, I've noticed it's more difficult to find frontend developers who are good at TypeScript (since a lot of frontend developers just use plain JavaScript).
I don't think it is unusually difficult to find Rust programmers. The challenge is finding Rust programmers who also have expertise in systems software development. Rust was designed to be a systems language but ironically it primarily seems to attract developers that do not have expertise in systems software.
This isn't necessarily a problem. C++ and Rust can coexist pretty well in practice, and Rust is a good entry point for learning how to write systems software.
Sadly I have found the flip side, too: as someone with a systems software interest/background (I wouldn't say expert), the # of jobs in the Rust ecosystem that are not glorified webdev/microservicing (or worse, crypto) is actually quite small.
So I guess I wouldn't be surprised that the applications are trending that way, too, as that is where growth is happening right now it seems.
That's the flip side of the same coin as your parent comment, I guess - because Rust is more accessible than C or C++, a lot of people have started using it for stuff that's not hardcore systems software, and as a result it's not as good a filter for that stuff as C and C++ are.
On the other hand, we use Rust on the backend of our web based saas product, and it serves as a great filter for quality developers for us.
One thing I don't appreciate is nonorthogonality of certain uses and control structures that cannot be refactored into a function despite structural equivalence without introducing a multiple borrow conflict.
"Rust screams at you all day, every day, often about things that you would have considered perfectly normal in another life."
A good C compiler does this when you turn on all the flags. I like languages/compilers that let you selectively disable the screaming and let you write bad code on purpose. Bad code that works but can be written fast is often better than perfect code that takes forever to write. Once you have a bad but working POC, you can make it less bad.
"It’s got no problem attracting new users, but it’s not resulting in dramatically improved libraries or tools. It’s resulting in one-off forks that handle specific use cases."
Age has nothing to do with that. Attracting core devs is hard, and putting lots of effort into making it attractive is necessary. On top of that, cultural conventions are set by the early adopters, and a lack of convention is often just as sinful as a bad existing convention.
Take Python for example. They took a lackadaisical approach to development and runtime environments, and as a result there's 50 competing ways to develop or run a Python program. Their most-used package repository, PyPI, has been a mess for years. Nobody builds on top of existing packages, names make as much sense as a random word generator, the ecosystem is rife with malware, you can't even search for a package on the command line, etc etc. None of that is the language's fault, it's the community and core team's fault for sitting on the sidelines rather than leading. Culture matters more than the tech it's centered around.
(I'm not trying to pick on them, I just know their problems better. C has been around for a half century and its community never really put together half the solutions more modern languages did)
> A good C compiler does this when you turn on all the flags. I like languages/compilers that let you selectively disable the screaming and let you write bad code on purpose. Bad code that works but can be written fast is often better than perfect code that takes forever to write. Once you have a bad but working POC, you can make it less bad.
That doesn't really work. All unsafe lets you do is dereference pointers or call unsafe functions. That's not gonna speed your development up during prototyping.
You can instead wrap everything in Arc<Mutex<T>> and .clone() liberally, though.
Always find it funny that people think unsafe {} means that rust ignores everything in the block, it just enables 4-5 additional abilities that are well documented, it's still doing most of its safety checks.
It will work if you only use raw pointers everywhere, like it's C. Don't use slices or strings; just pass a pointer to the first element of an array, and either pass its length separately or provide a sentinel value (like C's null terminated strings). Navigate balanced search trees using aliasing, raw, mutable pointers. Etc. This person compares the translation of C to Rust, literally versus idiomatically: https://cliffle.com/p/dangerust/
There will probably be weird behavior, though, because Rust optimizes based on assumptions about boxes and references. For example, if you have 3 raw pointers to some object live at once, and you give some library function a mutable reference made from one of those pointers, the compiler will optimize assuming it has exclusive access to that object, and it may make incorrect assumptions.
> All unsafe lets you do is dereference pointers or call unsafe functions.
And all those let you do is drop lifetimes and wave goodbye to the borrow checker. You just have to be explicit about it.
(What I mean is, the borrow checker checks borrows, it checks references. In unsafe you can make a reference forget what it's referring to. Just change &'a T to &'static T, and then the borrow checker is doing nothing.)
"Just mark everything unsafe" seems to be the motto of many (most?) crate developers. There's so much "unsafe" in dependencies used by so many Rust programs. It's a timebomb waiting to go off.
No, 1/5 of Rust code contains unsafe directly. Also, "the" point (in the area of safe/unsafe) is to manage unsafety and provide safe abstractions from unsafe foundations. If you go deep enough there's always something which will be unsafe (the type system would have to able to proof the whole universe otherwise), but most programmers will not have to write such code themselves (no, most of use do not write double-linked lists each day) - they can just use it (e.g. by using the standard library). And if they have to write unsafe code the unsafe parts are restricted to certain areas of their code. Areas they can then invest extra time and care to make certain their assumptions hold true.
I checked the six most recently published crates on crates.io (blablabla, nutp, tord, g2d, testpublishtesttest, hellochi, at the time of writing). Three of those (blablabla, testpublishtesttest, and hellochi) did some variation on printing `hello world`. g2d seems like an interesting graphics library. tord provides a data structure for transitive relations, which is also neat. No crate contained over 250 lines of code. Unsurprisingly, none of them contained unsafe code.
Elsewhere in this thread, it's been pointed out that as a consequence of crates.io having a global namespace, plus lax enforcement of an anti-squatting policy, there are a lot of namesquatting packages. Those presumably contain no unsafe code.
tokio contains unsafe code. rand contains unsafe code. regex contains unsafe code. time contains unsafe code. (method: a smattering of packages chosen from blessed.rs; result: every one that I checked except serde containing unsafe code; epistemic status: eh -- I grepped the codebases, ignoring things that were pretty clearly tests, but might have accidentally included some example code or something that's not part of the core library? Please let me know if I've misattributed unsafe usage to one of these projects, or if I've managed to select a biased sample!)
I'd certainly believe a straightforward reading of the claim "80% of crates have no unsafe code"...but that seems almost meaningless, given that a not-insignificant portion of crates contain basically no code at all? I'd be much more interested in a weighted percentage by downloads: I'd be wildly impressed if 80% of crate _downloads_ contained no unsafe code, and would be somewhat unsurprised if the number was well below 50% -- crates with more functionality would be more useful and therefore more download, but also more likely to use unsafe code, I'd imagine.
Edit: I just noticed crates.io has a most-downloaded list[0] -- I might end up running some numbers on top packages there tomorrow morning, for some more solid data.
What is the fact that many foundational ecosystem crate contain unsafe code supposed to prove? That's the entire point of the language. That someone writes a really good regex crate once and then the rest of us don't have to write unsafe to use it. It seems like you have a fundamental misunderstanding about the goals and purpose of rust.
Libraries safely wrapping unsafe code in safe interfaces, and everyone reusing those safe interfaces is like, the whole point of…reusable libraries???
Also, you’re replying to someone who I’m fairly sure is on one of the core Rust teams, if not closely involved, I’m somewhat more inclined to trust _them_ when they say 80% of libs don’t contain unsafe (given that it cleanly meshes with my own experience of Rust libraries).
Instead of looking at the crates themselves, you might want to check your (or others') Rust application with https://github.com/rust-secure-code/cargo-geiger to get a sense of effective prevalence. I also dispute that the presence of unsafe somewhere in the dependency tree is an issue in itself, but that's a different discussion that many more had in other sub-threads.
At the risk of getting downvoted into oblivion, that sounds a lot like TypeScript (compiler yelling at you to fix things but you just want to try an idea without perfecting it).
> It also looks like (soon) you’ll finally be able to configure global lints for a project. Until now, you had to hack your solution to keep lints consistent for projects.
> I questioned my sanity every time I circled back around to the Clippy issue above. Surely, I was wrong. There must be a configuration I missed. I couldn’t believe it. I still can’t. Surely there must be a way to configure lints globally. I quadruple-checked when I wrote this to make sure I wasn’t delusional.
You create a .cargo/config.toml in the workspace root so it covers all your crates.
The only limitation is that rustflags are not additive, so if you have other sources of rustflags like the RUSTFLAGS environment variable it will overwrite this setting.
Programming in Rust is really not like being in an abusive relationship. The compiler is trying to help out as much as possible, especially since rustc has the best error messages in the world.
This really is an inappropriate comparison. Can we be serious for a bit? A professional-grade tool providing professional-grade feedback is not remotely like an abusive or even turbulent relationship.
As one of the main people working on Rust compiler diagnostics, I find this comparison beyond distasteful. The tooling is not capricious in its restrictions and we go out of our way to make it communicate to people with as much empathy and support as possible.
I'm sorry you're having a bad experience. I've found changing a couple habits developed in other languages helped me to have a good experience with Rust's diagnostics.
1. Reading the error messages. I was used to error messages verbosely printing a lot of details which were mostly irrelevant and letting the programmer sort it out, and I developed a habit of skimming them. I had a better time with Rust when I realized it was giving me information that was mostly relevant, and that I should actually read them.
2. Interpreting error messages as feedback rather than failure. I was used to error messages meaning I had done something wrong, and getting them all the time was frustrating. Watching Jon Gjengset's coding videos, I was struck that he wrote code to trigger errors deliberately in order to get feedback from the compiler. I now try to work through Rust error messages the same way I would failing unit tests; I make some small changes, I knock out the errors, and then move on to the next subtask. This keeps the number of errors small and manageable, and gets me into a tight feedback loop with the compiler.
Relatedly, I encountered the idea (in a blog post I no longer remember) that the Rust compiler is more like an automated pair programming partner than other compilers. This change in mindset really helped for me; thinking of it as a friendly helper rather than a loud complainer made the work lighter. (That's why I personally don't like the 'abusive partner' metaphor, for me that mindset makes it toilsome and miserable. Also because I've had an abusive partner, and it's nothing like that, and I could do without the reminder.)
3. Setting up my IDE to show me inline type annotations and error messages. This made the feedback loop tighter, I only have to drop into a terminal for complex or unfamiliar error messages. With practice I can fix most errors using a one-line summary.
4. With practice, I've internalized Rust's semantics, and I've stopped painting myself into corners with unnecessarily cyclic data structures and such. I also read a blog article or maybe a tweet (which I've also forgotten) about giving yourself permission to use Box or clone and not sweating every copy or allocation. It pointed out that if you were writing Python, everything would be an Arc<T>, and you'd think nothing of it; in Rust the performance cost is explicit, so you agonize over it, but a lot of the time it's not a big deal to copy some data or to perform an allocation.
Wasn't my intention. I'm sorry if I came off that way, I can see how my comment might be presumptive or pretentious. My bad. You don't have to do any of that. You don't even need to write Rust in my book, it's not the one true language or anything. That's just what's working for me personally.
My first six months with Rust were very painful and I struggled to write the simplest programs. (This was in fact, the first six months of my second attempt - the first time I tried to learn it, I gave up.) Eventually I learned how to work with it, and since then, it's my absolute favorite language to work in (though presently I mostly write Typescript and SQL, because I'm writing webapps). I tried to summarize what happened along the way and what changed.
I would encourage you to file tickets whenever you encounter deficiencies in the tooling, including bad diagnostics. We take them seriously. As for whether our efforts are accomplishing anything, trying out older releases can be eye opening at how much has changed.
The OSes want programmers to handle resources correctly, and the Rust compiler makes that task a breeze. We are in a more abusive relationship with our OSes, than the Rust compiler. How about the hardware? Doesn't that need to run assembly in a correct way? That counts as an abusive relationship as well.
Rust's error messages are one of a kind. There no other compiler which comes even close.
As a side note, i used latex lately, it's error messages are horrendous. What a nightmare to figure out what's wrong by inspecting the error.
> The compiler is trying to help out as much as possible, especially since rustc has the best error messages in the world.
Generally good, but man do I hate how any error in my async function causes every recursive call site to generate an error about how the Future is no longer Send and Sync. Literally an entire console scrollback of errors with the actual syntax error buried somewhere in the middle.
I believe this is in our radar and waiting on the new trait solver, but just in case if you have a repro I would appreciate a ticket to improve the diagnostic.
In this case, the first two errors are clear. The next 3 provide multiple, redundant context blocks. The upshot is that this simple example results in rustc printing 11 lines of useful error messages and 86 lines of useless messages. Add to that the fact that the useful error messages need not be at the top or bottom of the error list, they can be anywhere inside of it, depending on the declaration order.
That is a different problem than the one I thought you were seeing.
We do spend a lot of time trying to silence errors that are irrelevant. We also get a lot of complaints when fixing a single error produces a wave of new errors that were hidden due to failures in earlier stages. It's a balancing act.
Also, specific errors are verbose in order to give people a fighting chance to fix their issue. An error that is too verbose is annoying. An error that is too terse will leave users in the cold as to what the problem was. It's yet another balancing act.
> Could I ask you why you didn't consider doing so when you first encountered this problem?
Thanks for looking into it. I have a filed a bug against Rust before, but that was a clear bug, not a poor error message. Remember, the only time a user sees something like this is when they are trying to do something else. The only reason I looked into it just now is because you explicitly asked.
We consider poor error messages to be bugs. I know most people don't, so I've made it one of my missions to yell it from the mountain tops to encourage people to report more often. We can't fix what we don't know about. The majority of the current good errors were a reaction to things people filed in the past.
Rust's compiler is the first one I've seen to use the word perhaps.
My main gripe is that I still don't fully comprehend lifetimes and the compiler can't really help me every time, because it (understandably) errs on the side of caution.
Whenever you see weasel words like that, it means the compiler knows that the issue you encountered could be what it is saying but the necessary metadata to figure out for sure is inaccessible to it. It's the problem with classical stage-oriented compilers. A compiler designed for diagnostics from the beginning would end up looking like a plate of spaghetti where you can call type checking from the parser, to give an example.
>It's the problem with classical stage-oriented compilers. A compiler designed for diagnostics from the beginning
I'm not sure any compiler, even one "designed for diagnostics," can gather the "necessary metadata" from inside my brain based on my original intentions. If the compiler could unambiguously interpret what I meant to write, it wouldn't have needed to fail with a compilation error in the first place. Whenever I see something like "perhaps" or "maybe" in a compilation error, the information just didn't exist anywhere the compiler could possibly get to, and it's just a suggestion based on common mistakes.
> After two decades of JavaScript and decent experience with Go, this is the most significant source of frustration and friction with Rust. It’s not an insurmountable problem, but you must always be ready to deal with the async monster when it rears its head. In other languages, async is almost invisible.
I am a former C and C++ programmer who lived calling into pthread almost every week for a decade. I use async rust everywhere now.
I don’t get the hate that async gets. In my opinion, everyone should be using async for everything. Including stuff that’s seemingly single threaded “simple” stuff.
The problem that I am running into at the moment is that a few things like the rhai Engine aren't Send and I am trying to use them in an async closure. What GPT-4 suggested was creating a tokio Runtime inside the thread and then block_on(). I will try it tomorrow. (This is the first significant Rust project for me.)
1. Not applicable unless it is absolutely required, and not something I will consider until I have exhausted other options, since there is a reason they have not made it Send already. It will likely be quite complex and enlarge the scope.
2. I am using a normal thread but when I make it async the compiler wants Send.
3. The await point is in code that uses the engine. I am not sure there is another good option, since I need to use an API that has several libraries all of which are async.
The block_on is to allow the tokio Runtime created in that thread to execute/poll it
So, thank you for your input, I will test out the suggestion that I mentioned above, and then maybe look into spawn_blocking if that doesn't work.
That ChatGPT suggestion is dodgy. Tokio is going to complain when you create a runtime while in async runtime, and refuse block_on in an async context.
You can have multiple runtimes, but create them ahead of time, in synchronous main, and keep a Handle to them.
> 2. I am using a normal thread but when I make it async the compiler wants Send.
This is likely because you are using the multi-threaded scheduler, which requires futures to be Send, even if you’re running only a single thread. This is because Tokio is based on a “work stealing” runtime, so in “normal” operations, expects the futures themselves to be able to be shuffled around threads where necessary.
For you use case, try running the single threaded executor, additionally try the “local set” executors. These do not require Send as they are statically guaranteed to be confined to the thread they are spawned on. Block on will also work.
Out of curiosity though, how is the interaction with Rhai performed, are you passing around the engine, and executing the code at certain points where applicable? Do you just need certain results from it sometimes? Etc?
It being !Send means its not safe to be sent lol, that's down to the way they implemented rhai engine not rust. It's just that rust catches that it's not safe to send because of the trait bounds.
Recently tried writing some async Rust to compare the error handling when nested async calls are made to how errors are handled in Go, and it seemed like the trivial example I was trying to write up simply couldn't be done without involving Tokio. That barrier simply doesn't exist in Go, or C#, or Typescript.
For instance, you apparently cannot `await` in the main function without a decorator you import from, you guessed it: Tokio.
You need an async runtime to run async code yes, and Rust's isn't built in. Why does that matter though? Rust has a decent package manager; add the dependency and move on.
Why are you trying to avoid Tokio lol, tokio is the defacto async runtime in rust, saying you're trying to avoid it is like saying you're trying to avoid async while writing async, somehow people act like if they merged tokio into std and instead of #[tokio::main] or whatever you had to do #[async::main] it would somehow be better.
Once you stop fighting the fact that tokio = async rust for 99% of cases, things are quite smooth.
Tokio simply doesn't meet the ise case of some people doing wasm, and many people doing embedded. That isn't a huge deal, just don't use it right? Except many otherwise usable crates seem to adopt tokio unnecessarily.
The way closures made it easy to encapsulate code and state in an object you can run at any time, async encapsulates code and state for ability to run and pause or cancel at any time.
This is handy for I/O that can be interleaved and cancelled. You can (ab)use it for other things like generators or various DIY multitasking operations. It can also be a state machine generator (e.g. AI of actors in a game).
But I think OP just meant async for typical networking and DB interfaces. And yes, this usually implies the Tokio dependency.
I think it is the infectiousness of it. Especially in embedded or wasm contexts, the predominant async may not be the async you want. Wasm being the author's use case would definitely have provided a different perspective.
Similarly, I find tasks that use or reuse large buffers to avoid the performance hits from allocation, often benefit from old fashioned thread pools. Bump or shard allocators can make this work ok, but in the cases where you are cpu bound on tight loops of vectorizable operations, thread pools perform better. Async is a good tool, but there are contexts it isn't optimal for.
Javascript to Rust is a big step for sure. It's probably a less painful journey if you already have a background in Typescript/C++. This article has good feedback though and I hope the language evolves to address some of it in the future.
I'm learning Rust because it seems clear that it's going to be important professionally. I wish I loved it, I really do. I see the benefits. But, at least so far, it's one of the most unpleasant languages I've used. I keep hoping that as I gain proficiency, I'll stop disliking it, but as I climb higher on the learning curve, I'm not really warming to it.
It's fine. It won't be the only language I'll be proficient in while being averse to it at the same time. But I heard so many people proclaiming their love for it that I expected to enjoy it, too.
Good question! There's no single thing, I think, and the things I dislike about it aren't even really technical criticism or the like. They're more... aesthetic? I find the syntax unpleasant, for instance. It's extremely opinionated and a few of those opinions are ones I disagree with.
Also, just generally, it tends to make even simple things pretty complex. I understand why and am not really objecting to that, but it does make using it a bit like running with lead shoes.
I expect that the latter part may get better as I use it more (but perhaps not -- there are other languages I'm fully competent in but dislike for similar reasons).
I share your sentiment about Rust. Fine language for sure, I just don't enjoy programming in it. Simply put, I don't like using the abstractions that the language encourages.
For what it is worth, I found pursuing Zig to be a breath of fresh air. It's a promising alternative in the same space as C (and also Cpp and Rust). Checkout Andrew Kelley's introduction to the language.
There are a few, and they aren't something that I can't live with, of course. A great example of the sorts of things I'm talking about are Rust's insistence on camel case and snake case.
The lack of default/named function arguments is what still gets me. It's such an absolute basic programmer ergonomics feature shared by most popular languages; even C++ has had defaults since forever.
It would have been nice for Rust to have Default/named/optional function arguments because the proliferation of slightly differently named functions that do the same thing would go away.
As a counter example, I love programming in Rust. Fighting the borrow checker ended a long time ago. Even the errors are rare these days, except in cases I trigger them in order to examine the types. Rust compiler also seems to have improved in accepting broader cases that are valid.
For me, the key to understanding the borrow checker was understanding the underlying memory model. Rust memory model is the same as that of C, with some extensions for abstractions like generics. The borrow checker rules seem arbitrary at first. But it's deeply correlated to this memory model. The real value of the borrow checker is when I trigger it unintentionally. Those are bugs that I made due to lapse in attention. What scares me is that another language like C or C++ might simply accept it and proceed.
Yet another pleasant side effect of Rust's strict type system and borrow checker is that they gently nudge you to properly structure your code. I can say for certain that Rust has improved my code design in all the languages that I use.
My experience is that most programs dealing with ordinary problems don't need such complicated data structures. In cases you do, Rust has a few options:
1. Use Rust's runtime safety checks, using Rc, Weak, RefCell, etc. It will be as ergonomic as GC'ed languages (no productivity loss). While this has a runtime performance penalty, it will still be mostly comparable to other languages. This works for most use cases.
2. If you need the last ounce of performance, just drop all automated safety checks and do it manually using unsafe. Even if you make a mistake, your debugging will be limited to the unsafe blocks. This approach isn't unusual in Rust.
3. Use either the standard library or something on crates.io that does what's given in 2. Rust's generics make it easy.
1. Option 1 quickly degrades into unrefactorable mess of TypedArenas, nested Rc<RefCell>>, lifetime annotations everywhere, and the like. It is unmanageable, which is why even libraries like PetGraph do not use it.
2. No, I don't need every last bit of performance. C++ performance is OK. Or any other sane language like Nim, Crystal, etc. I need async, though, which is yet another hell in Rust.
3. This absolutely can't be done with standard library. There are libraries like PetGraph, which solve this particular problem (by some very unobvious approaches and bits of unsafe code), but there are many problems like it which don't have any libraries for it yet.
I do have hard problems. I want a language which makes hard problems easy, and impossible problems hard. Rust is definitely _not_ such a language, and it makes me like 5 times less productive, and sucks all joy out of programming.
Use one of the many libraries that have safe abstractions over unsafe code? I don't know why people think you need to roll your own? I guess that's just what people do in c/c++, doesn't seem very productive...
There are definitely no less libraries for C++. I am a scientific software engineer, and occasionally I do develop new things. And Rust makes writing new system-level software way harder, which I don't understand, as it is a system-level language.
735 comments
[ 4.6 ms ] story [ 354 ms ] threadMaybe it was a reaction against the Java-style reverse DNS notation, which is verbose and annoying, but a more GitHub-style user/group namespace prefixing package names would have been the nice middle ground.
Also, even if you know it's the latter, package namespacing isn't strictly related to directory structure, so `bar/baz` has no specific meaning outside of the context of a go import. They could have used any other separator for package components - `git.example.com/foo:bar:baz` - but instead they chose the slash, making the scheme both technically ambiguous and easy to confuse for an HTTP URL.
I still don't think this is a huge issue, to be honest. Not one big enough for me to complain about, for sure. But it's definitely not ideal.
Incorrect. You're able to use any URL you control, regardless of where your SCM is located.
By the time you're going to production, your vetted and locked dependency should be living in your own cache/mirror/vendored-repo/whatever so that you know exactly what code you built your project around and know exactly what the availability will be when you build/instantiate your project.
Your project shouldn't need to care whether GitHub fell out of fashion and the project moved to GitLab, and definitely shouldn't be relying on GitHub being available when you need to build, test, deploy, or scale. That's a completely unnecessary failure point for you to introduce.
Systems that use URL-identified packages can work around some of this, but just reinforce terrible habits.
E.g. doi's https://en.m.wikipedia.org/wiki/Digital_object_identifier
Basically the URL of a package name should be primarily the ID, not the locator (even if it is used for location by default).
In the Go world this would be "vendored" dependencies, that is, the dependencies are within your source tree, and your CICD can build to its hearts content with no care in the world about the internet because it has the deps.
The URL is useful for determining which version of a specific project is being used - "Oh we switched to the one hosted on gitlab because the github one went stale"
The advantage of using gitlab, or github, or whatever public code repository is that you get to piggy back off their naming policies which ensure uniqueness.
But, at the same time, there's no reason that the repo being referred to cannot be in house (bob.local) or private.
Having said all of that, the Go module system is a massive improvement on what they did have originally (nothing) and the 3rd party attempts to solve the problem (dep, glide, and the prototype for modules, vgo), but it's not without its edge cases.
I.e., when you're going to run the production build, the URLs are mapped to fetch from the vetted cache and not the internet.
I don't see any downsides to allowing them as a source, or making them the default approach
This seems strange to me because the whole point of a Uniform Resource Locator is to specify where a resource can be located.
It's a bit like saying "My project depends on the binder on shelf 7 in Room 42, sixth binder from the left. Except when I go into production, then use...." Don't tell me what binder it's in, tell me what it is.
I can see a case made for URIs, which is basically what Java did.
Alternatively, you can do:
Obviously Maven doesn't host any Rust crates (yet?), this is just a theoretical example. Very few projects bother to host their own registry, partially because crates.io doesn't allow packages that load dependencies from other indices (for obvious security reasons). The registry definition can also be done globally through environment variables: CARGO_REGISTRIES_MAVEN="https://rust.maven.org/git/index". Furthermore, the default registry can be set in a global config file.In theory, all you need to do is publish a crate is to `git push upstream master`, and your package will become available on https://github.com/username/crate-name (or example.com/your-package if you choose to host your git repo on there).
Personally, I don't like using other people's URL packages, because your website can disappear any moment for any reason. Maybe you decide to call it quits, maybe you get hit by a car, whatever the reason, my build is broken all of the sudden. The probability of crates.io going down is a lot lower than the probability of packages-of-some-random-guy-in-nebraska.ddns.net disappearing
https://github.com/rust-lang/rfcs/pull/3243
I suspect it was less a reaction against anything and more just following the norms established by most other package managers. NPM, PyPI, RubyGems, Elixir's Hex, Haskell's Cabal... I'm having a hard time thinking of a non-Java package manager that was around at the time Rust came out that didn't have a single, global namespace. Some have tried to fix that since then, but it was just the way package managers worked in 2014/2015.
"We're pretending security is not an issue." has been the feedback every time this is raised with the Cargo team.
To be honest, it's turned me off Rust a little bit.
The attitude of "Rust is memory-safe, so we don't need any other form of security." is not a good one.
Do you have a specific link where I can read this response, because this is not at all the responses I have read.
- No strong link between the repo and the published code.
- Many crates were spammed that were just a wrapper around a popular C/C++ library. There's no indication of this, so... "surprise!"... your compiled app is now more unsafe C/C++ code than Rust.
- Extensive name squatting, to the point that virtual no library uses the obvious name, because someone else got to it first. The aformentioned C/C++ libraries were easy to spit out, so they often grabbed the name before a Rust rewrite could be completed and published. So you now go to Cargo to find a Rust library for 'X' and you instead have to use 'X-rs' because... ha-ha, it's actually a C/C++ package manager with some Rust crates in there also.
- Transitive dependencies aren't shown in the web page.
- No enforcement or indication of safe/unsafe libs, nostd, etc...
- No requirement for MFA, which was a successful attack vector on multiple package managers in the past.
DISCLAIMER: Some of the above may have been resolved since I last looked. Other package managers also do these things (but that's not a good thing).
In my opinion, any package manager that just lets any random person upload "whatever" is outright dangerous and useless to the larger ecosystem of developers in a hurry who don't have the time to vet every single transitive dependency every month.
Package managers need to grow up and start requiring a direct reference to a specific Git commit -- that they store themselves -- and compile from scratch with an instrumented compiler that spits out metadata such as "connects to the Internet, did you know?" or "is actually 99% C++ code, by the way".
Maybe the obvious names should have been pre banned. But I don't see the issue with non-obvious names either way you're going to have to get community recommendation/popularity to determine if
brandonq/xml is better or worse then parsers/xml
You don't want to be using Ivan Vladimir's OAUTH package to sign in to Microsoft Entra ID. That probably has an FSB backdoor ready to activate. Why use that, when there's an equivalent Microsoft package?
When any random Chinese, Russian, or Israeli national can publish "microsoftauth", you just know that some idiot will use it. That idiot may be me. Or a coworker. Or a transitive dependency. Or a transitive dependency introduced after a critical update to the immediate dependency. Or an external EXE tool deployed as a part of a third-party ISV binary product. Or...
Make the only path lead to the pit of success, because the alternative is to let people wander around and fall into the pit of failure.
I think we need to encourage a culture that package managers are our shared garden and we must all help in the weeding.
> Do you have a specific link where I can read this response, because this is not at all the responses I have read.
Those aren't people saying security isn't an issue but examples of concerns you have which is different.
For some of those, there are reasonable improvements that can be made but will take someone having the time to do so. While the crates.io team might not be working on those specific features, I do know they are prioritizing some security related work. For instance, they recently added scoped tokens.
For some, there are trade offs to be discussed and figured out.
Literally nobody has said this.
> The attitude of "Rust is memory-safe, so we don't need any other form of security." is not a good one.
Fortunately it's an attitude that nobody in the Rust project has!
I know of a few people, personally, who have said this.
Because it seems like the people who are working on the project aren’t saying that.
The point is that it's much easier to make a mistake typing "requests" than " org.kennethreitz:requests" (as a pure hypothetical.)
It also means that more than one project can have a module called "utils" or "common", which once again reduces the risk of people accidentally downloading the wrong thing.
Sorry what? It's strictly the opposite: more character to type equals more risks to make a mistake.
In fact, in the general case, namespace increase the risk of supply chain attacks, because it makes packages names even less discernable.
jiggawatts
The implication here is that namespaces in package managers weren't a known concept. Outside Java, NPM - probably the biggest at the time - not only supported them but was actively encouraging them due to collective regret around going single-global in the beginning. Composer is another popular example that actually enforced them.
Not only was namespacing a known widespread option, with well documented benefits, it was one that was enthusiastically argued for within the Rust community, and rejected.
> The most prominent feature driving the release of npm 2 didn’t actually need to be in a new major version at all: scoped packages. npm Enterprise is built around them, and they’ll also play a major role when private modules come to the public npm registry.
My memory is that the industry as a whole didn't really start paying attention to the risks posed by dependencies in package managers until the left pad incident.
To be clear, I'm not saying that it was a good idea to not have a better namespace system or that they were completely ignorant of better options, just that they were very much following the norms at the time.
[0] https://blog.npmjs.org/post/98131109725/npm-2-0-0.html
That crates.io launched without explicitly acknowledging this whole problem is either naivety or worse: already by then Java wasn't "cool" and the "cool kids" were not paying attention to what happened over there.
It's not that the industry wasn't paying attention until the 'left pad incident' -- that only holds if one's definition of "the industry" is "full stack developers" under the age of 30; I remember when that happened and I was working in a shop full of Java devs and we all laughed at it...
Maven's biggest problem was being caked in XML. In other respects it was very well thought out. That and it arrived at the tail-end of the period in which Java was "cool" to work in.
Rust chose to follow the majority of languages at the time. Again, as I noted in my previous comment, I'm not defending that decision, just pointing out that most of the widely-used languages in 2014 had a similar setup with similar weaknesses.
Also npm's bad policy/decision to transfer control of package in the name of predictability(this should probably be avoided for packages that aren't malicious. You could argue for seizing broken/trivial and unmaintained packages that have a good name but even then it might be best to leave well enough alone).
I suppose you're talking about the original dispute which led the developer to unpublish his libraries (which npm stupidly allowed, and cargo didn't). There's a smaller chance of a company wanting a random package namespace then a package name but its not impossible (think Mike Rowe Soft vs Microsoft)
It was "about" cavalier approach to the dependency supply chain. A dependency disappearing outright is just one of many failure modes it has.
This may be a little off topic for this comment thread but this is a little misrepresentative. Hosted private repos for enterprise weren't exclusive to Java at the time of left pad - anyone doing enterprise node likely had one for npm too & were probably well prepared for the attack. Such enterprise setups are expensive though (or at least take a level of internal resources many companies don't invest in) leaving the vast majority exposed to both js & java supply chain attacks even today.
[0] https://getcomposer.org/
[0] https://github.com/composer/packagist/issues/163#issuecommen...
Technically not in the same category, but Docker Hub (2014) had namespaces.
They were trying to do better than mainstream languages in other areas and succeeded. IIRC on this front they just decided Ruby's bundler was the bee's knees.
And at that time, it was a good idea. Ruby was popular and bundler + gem ecosystem was a big reason for its popularity. No one was worried that Rust might become so popular that it might outgrow the bundler model. That was only a remote possibility to begin with.
1. Flexible version (of requirements) specification;
2. Yes, source code had domain names in packages but that came from Java and you can technically separate that in the dependency declaration;
3. You can run local repos, which is useful for corporate environments so you can deploy your own internal packages; and
4. Source could be included or not, as desired.
Yes, it was XML and verbose. Gradle basically fixed that if you really cared (personally, I didn't).
Later comes along Go. No dependency management at the start. There ended up being two ways of specifying dependencies. At least one included putting github.io/username/package into your code. That username changes and all your code has to change. Awful design.
At least domains forced basically agreed upon namespacing.
"github.io/username/package" is using a domain name, just like Java. Changing the username part is like changing the domain name--I don't see how this is any worse in Go than in Java.
If you don't like that there's a username in there, then don't put one in there to begin with. Requiring a username has nothing to do with Go vs. Java, but rather is because the package's canonical location is hosted on Github (which requires a username).
I don't know why so many programmer's use a domain they don't control as the official home of their projects--it seems silly to me (especially for bigger, more serious projects).
So that old version of package xyz will still resolve in your tagged build years from now even if the project rebrands/changes namespaces.
Not only that, but a commercial, for-profit domain that actively reads all the code stored on it to train an AI. Owned and run by one of the worst opponents of the OS community in the history of computing.
At least move to Gitlab if you must store your project on someone else's domain.
To be honest I really like how Java advocated for verbose namespaces. Library authors have this awful idea that their library is a special little snowflake that deserves the shortest name possible, like "http" or "math" (or "_"...).
This was solved for C programs since whenever makedepend came out! (I'm guessing the 80s.)
(Bonus grief: Maven's useIncrementalCompilation boolean setting does the opposite of what it says in the tin.)
Neither node nor maven ever bothered to solve this, so we end up wandering the Wild West wondering when it will be that HR, or legal, or architecture comes knocking on the door to ask what we were thinking having a dependency on a dynamic version of Left Pad.
I'd kinda like to see what Cloudflare and Let's Encrypt could come up with if they worked together on at very least a white paper and an MVP POC.
Pay somebody either internally or externally to maintain a repo of all your dependencies and point your code at that. You won't get a left-pad incident. You won't get a malicious .so incident (unless you mirror binaries instead of source code).
Like if you ran out of screws to make your product with do you walk around the street and scrounge up some? No, you go to a trusted vendor and buy the screws.
I'm suggesting that the guys who've repeatedly proven themselves technically competent around security at scale might have a couple of useful ideas regarding how the industry might go about crawling its way out of this little security clusterfuck.
And perhaps even stop treating something as simple as a BOM as an enterprise feature, given that the overhead on such things is damned near zilch and the security implications are staggering.
https://www.cisa.gov/sbom
They're all checked into Google3 (or chromium, etc.). One version only. With internal maintainers responsible for bringing it in and multiple people vetting it, and clear ownership over its management. E.g. you don't just get to willy nilly depend on a new version -- if you want to upgrade what's there, you gotta put a ring on it. If you upgrade it, you're likely going to be upgrading it for everyone, and the build system will run through the dependent tests for them all, etc.
And the consequence is more responsible use of third party deps and less sprawling dependency trees and less complexity.
And additional less security concerns as the code is checked in, its license vetted, and build systems are hunting around on the Internet for artifacts.
I'm not a fan of those - if the engine's in the wrong place, then why the hell did you fucking put it there?
Don't hack a patch into place to stop people from holding it wrong; design it in such a way that it's impossible to hold wrong in the first place.
https://samsieber.tech/posts/2020/09/registry-structure-infl...
The first section says it discourages forking. I consider this to be bad. Nobody’s code should be more important purely because it squatted a better name.
The Identity section actually makes the case that flat registries make naming harder.
The section on Continuity is “we’ve tried nothing and we’re all out of ideas”. Make up an org name and grandfather all packages in the flat namespace into that special org. Also this is already a problem because packages in the flat namespace do get abandoned, then forked, and then we have the associated issues.
The section on Stability seems to take it as a given that crates.io should be the only registry. I don’t. It also seems to conflate cargo with rustc for the benefit of the argument.
The squatting section describes only anti-features and I don’t consider the author’s legitimate use cases to be legitimate reasons to squat.
I think the only legitimate problems that need addressing are the ergonomics of accessing namespaced packages throughout transient dependencies and backwards compatibility with non-namespaced code. But the fact that these are real problems does not, to me, make a flat namespace a “feature”. It’s just easier to implement.
It’s okay for it to be a mistake that takes effort and time to fix.
You get some oddities such as serde-derive/serde-derive but the package owners can choose if they want to move to serde/derive or leave it in a separate namespace.
I send the analysis to the crates.io team and pointed that they have a no-automation policy.
They told me that it was not sufficient proof that someone was squatting those names. That's my problem with crates.io is that they have a clear policy and they don't enforce it so all the short/easy to remember names for crates are already taken and there is nothing you can do to get it.
Not naming names, but I know several people working to put Crates.io out to pasture.
There's a level of playing nice with them for the time being (eg. build reproducibility), but it's only KTLO.
Crates.io needs to die for Rust to thrive. They're a bungled, mismanaged liability. New code, new leadership.
Once a better alternative is out there, crates.io will either wither and die or improve. If it matches its competition in terms of quality and reliability, everyone is better off. If not, the alternative solution will take over.
I'm eager for this crates.io alternative to land, assuming they don't break too many projects in their improvements.
******* is my password, but you can't see it. Type your password back, and I won't be able to see it. Try it!
hunter2
use::bar; // changing to
use::foo::bar;
I assume the library names that can be overridden in cargo would still be accepted, and then it all gets a little messy. The transition would be very messy.
Still doesn't solve all of the policy problems with namespacing.
What's that? I have scripts that automate publishing of new release of my crates. And I think many projects have.
What they stated was only regarding claiming new ownership over crate names:
> Using an automated tool to claim ownership of a large number of package names is not permitted.
> Using an automated tool to claim ownership of a large number of package names is not permitted.
And
- Hey, I found that someone created crates at a rate of about one every 30 seconds for a period of a week straight.
- That's not sufficient proof of squatting.
Whoever answered that, was either supporting the squatter or explicitly in favor of the practice. I cannot conceive that someone would get that evidence in their hands, and in their right mind think that the claim is bogus. Hell, I'd even be willing to suppress the squatter with evidence of one new crate created every 30 seconds for one hour!
The only reasonable conclusion to make is that they didn't really care. But then don't save face and claim that you do. That's hypocrisy.
So many inferior dependency management systems for other languages have come along later, and learned nothing from those that came before it.
The inverse-style domain name thing does a really good job of removing the whole issue of squatting from the ecosystem. You have to demonstrate some level of commitment and identity through control of a domain name in order to publish.
I would also say that this puts just enough friction so that people don't publish dogshit.
crates.io demonstrates quite clearly that you either have to go all the way and take responsibility for curation or you have to get completely out of the way. There is no inbetween.
It took some iterations before maven 3 became "good", so people forget that it wasn't as nice before now! Unfortunately, it seems that the lessons learned there is never really disseminated to other ecosystems - perhaps due to prejudice against "enterprisey" java. Yet, these package managers are now facing the sorts of problems solved in java.
https://github.com/takari/polyglot-maven
Whats the diff between changing lib version in xml and json?
Maven files have a simple conventional structure and a defined schema. Once you learn it, it's a breeze to work with. It's not like you need to write hundreds of lines of SOAP or XLST — which is actually why people started to dislike XML, and not because XML inherently bad.
Edit: I'd also take XML over TOML any day, especially when it comes to nested objects and arrays.
Go has namespacing mostly because for a long time it didn't have a package manager at all, so people just used a bunch of ad hoc URL-based solutions mostly revolving around GitHub, which happens to have namespaces and also happened to lend itself to aliasing (because a whole GitHub URL is too long).
If you want to look at an actual example of namespacing done well, Maven/Java is the place to look. There is no aliasing—the same imports always work across projects.
It's definitely a good thing that people choose new unique names for crates rather than dijan/base64 vs dljan/base64
Do understand the desire of having a crate for audio manipulation called "audio" but at the same time how often do we end up with "audio2" anyway? It's an imperfect solution for an imperfect world and I personally think the crates team got it right on this one.
It's really as simple as this: many libraries are generic enough implementing something that already exists. Let's say you want a library to manage the SMTP protocol. On crates.io, of course someone has already taken the "smtp" crate (ironically, this one is abandoned, but has the highest download counts, because it's the most obvious name). Let's say you disagree with the direction this smtp crate has gone, and you make your own. What do you call it?
Namespaces solve this problem. You'd instead of have user1/smtp and user2/smtp competing in feature sets. You can even be user3/smtp if you don't like the first two.
This is precisely what Java enables too. The standard library is in com.java.*; if you don't like how the standard library does something, you can make com.grimburger.smtp and do it yourself. If you choose to publish to the world, all the more power to you. It doesn't conflict with the standard library's smtp implementation.
Naming things really is one of the hardest problems. This crates thing is a special case of Zooko's Triangle: https://en.wikipedia.org/wiki/Zooko%27s_triangle
Crates.io names are human-meaningful and everyone sees the same names, but it's vulnerable to squatting, spamming, and Sybil attacks.
You could tie a name to a public key, like onion addresses do, but it's unwieldy for humans. (NB, nothing stops you from doing this inside of crates.io if you really wanted)
You could use pet names where "http-server" and "http-client" locally map to "hyper" and "reqwest", but nobody likes those, because they don't solve the bootstrap problem.
It's a problem with all repos because when you say "http-server should simply be the best server that everyone likes right now", you have to decide who is the authority of "best", and "everyone", and "now". Don't forget how much useless crap is left in the Python stdlib marked as "don't use this as of 2018, use the 3rd-party lib that does it way better."
So yeah... probably will be a problem forever. As a bit of fun here are some un-intuitive names, and my proposed improvements:
- Rename Apache to "http-server"
- Rename Nginx to "http-server-2"
- Rename Caddy to "http-server-2-golang"
- Rename libcurl to "http-client"
- Rename GTK+ to "linux-gui"
- Rename Qt to "linux-gui-2"
- Rename xfce4 to "linux-desktop-3"
Then you only need to remember which numbers are good and which numbers are bad! Like how IPv4 is the best, IPv6 is okay, but IPV5 isn't real, and HTTP 1.1 and 3 are great but 2 kinda sucked.
Very simple. If a company as big as Apple can have simple names like "WebKit", "CoreGraphics", and "CoreAudio" then surely a million hackers competing in a free marketplace can do the same thing.
Or it is just a placeholder from a squatter.
Perl's whole ecosystem is amazing compared to other languages. It's a shame nobody knows it.
As other mentioned though, a typosquatting is a much bigger problem and namespacing offers only a partial solution. (You can still create a lookalike organization name, both in npm and in Github.)
brandonq/xml vs parsers/xml with no clue if one is better then another.
Also possibly with some confusion over whether things with the same name are forks or not. May make it a little more difficult to Google. Why not just have brandon_xml vs xml-parser and have a community list of best and most popular libraries?
I guess the only issue is that some generic/obvious package names are bad packages. That can have been avoided if they banned/self-squated most of the names. I suppose if you use dns namespaces and actually tie it to ownership of the domain name it might make sense but that would also cause issues(what if you forgot to renew the domain?).
We could even have `rust.xyz` for crates that are decoupled from `std` but still maintained by rust core devs such as `regex`.
You will have this problem only once. After an initial research you settle on one and move on with life.
I don't get how having to vet a dependency is going to be more difficult than before. The process is 99% the same, you still have to do the research work initially in both cases.
This is true that with no namespace anyone can end up squatting a cool name, but with namespace you end up in an even worse place: no-one end up with the cool names, and it makes discoverability miserable, because instead of having people come up with unique names like serde, everybody just name their json serialization/parsing library json and user now needs to remember if they should use "dtolnay/json" or "google/json" (and remember not to use "json/json" because indeed namespace squatting is now a thing), and of course this makes it completely ungoogleable.
We've had the namespace discussion for hundreds of time in the various Rust town squares, and the main reason why we still don't have namespace is because it doesn't actually answer the problem it's supposed to address and if you dig a little bit you realize that it even makes them worse.
Having a centralized public and permissionless repository opens tons of tricky questions, but namespaces are a solution to none of them.
Do you really think nobody has ever been able to google a Go import path? Some of these arguments are ridiculous.
What's the problem with that? You will have to explicitly state intention which is always a good thing.
> and of course this makes it completely ungoogleable
You are aware that just pasting username/projectname gives you exactly what you are looking for in the several top search engines, correct?
> We've had the namespace discussion for hundreds of time in the various Rust town squares, and the main reason why we still don't have namespace is because it doesn't actually answer the problem it's supposed to address and if you dig a little bit you realize that it even makes them worse.
OK, if you say so. Is this worse state of things documented somewhere?
But maybe I’m bikeshedding by saying that.
There is already a crate providing parallel iterators. You just rename the iter() call and that's it. I don't agree it should be implicit though.
But to your point, the convenient defaults are very different. Unsafe typecasts require a lot of ceremony and careful thought in Rust, and they have to follow more rules than in C. In particular, references are all effectively "restrict" in Rust, and it's really easy to screw that up when you do unsafe cast from raw pointers to safe references, which is a big incentive not to write code like that if you have a choice.
Whereas Rust is a standard with multiple implementations?
[1]: https://learn.microsoft.com/en-us/cpp/assembler/inline/inlin...
> Inline assembly is not supported on the ARM and x64 processors.
If anything I want more explicit control, alleviated with an even more expressive type system. Ideally rusts type system would just be a prolog variant imho.
https://en.wikipedia.org/wiki/Category:Compiler_optimization...
Otherwise we should trust compilers to know better.
Bash with types (especially floats), fewer edge cases, functions with explicit parameters, simple command line flags...
edit: Although I can see that if they hadn't made string splitting the default, it would have saved a lot of headaches.
I guess I could avoid floats using multiplication, but I would rather keep the code as simple as possible.
I know it’s still not 1.0. Then just get fish. At least would save some keystrokes.
The Kotlin type system is comparable to Rust without the borrow checker. It has non-null types, generics, etc.
It's not really a "product" per se but there's an old version for download and some lightweight docs here:
https://hshell.hydraulic.dev/
We've never worked out what to do with it. Ideas and feedback welcome. It's pretty nice to use, albeit you need to use IntelliJ to edit scripts if you want IDE features.
Rust is quite transparent in what it does, and is very conservative with compiler magic. The language doesn’t do heap allocations, doesn’t do reference counting, it doesn’t even have implicit numeric type conversion. It won’t implicitly copy types that did not ask to be implicitly copyable, and even that is only legal for types that can be copied with a simple shallow memcpy.
Rust uses zero-cost abstractions all over the place, which means it’s predictable what code they will compile to, and that will be typically something simple. Std types have well-known basic layout, so you know that e.g. iteration over a Vec is going to be a loop incrementing a pointer, and there’s no implicit parallelism.
Describing borrowing as “compiler knowing better than the programmer” is a weird way of looking at it. Borrowing is like type checking. You declare a type to be temporary, and try to use it as long-lived, you get an error. It’s the same as if you declare function to return struct Foo, but returned struct Bar instead. Yes, compiler “knows better than you”, because you just wrote a bug.
Borrowing still compiles to direct pointer usage without a GC (it’s literally guaranteed to be identical to a C pointer in C ABI structs and functions), and you can override lifetimes with unsafe if you think know better than the compiler.
I think it's a matter of perspective/background. Depending what language you're coming from you might appreciate a lot of the more modern language features, or binary size, or performance. On the other hand, you might already be working with a pretty cutting edge language with a garbage collector or other and decide that Rust has some cool benefits but not worth the switch.
Rust is great for services such as API endpoints, where low latency and high throughput at a low cost are more important that hot reload during development.
The experience has been far more than just borrow checking, it’s opened up my mind to a lot of concepts that my previous trials with Java, JavaScript, and Python fully obscured in arcane ways. Rust’s compiler is wonderful, and once you get past the ergonomic struggles (which I did by just going through Advent of Code), you’re set.
Even when you don't care about performance, another issue that comes up sometimes is keeping track of mutable state. If you've ever relied on bytes instead of bytearray or tuple instead of list* to guarantee that no mutation is happening in some Python code, you know what I'm talking about. Rust can give you a similar level of control over who gets to mutate what, without making you change the type of your data or pay the cost of copies. It's basically the const/non-const distinction from C, but much stricter. Another way of saying the same thing is that Rust gives you a lot of the legibility/correctness benefits that you'd expect from a functional programming language, but you get to code in the usual imperative style.
* And even then, that only prevents assignment to the elements of the tuple. You can still mutate an element internally if it's not also an immutable type.
As long as you don't do a couple of things that, while easy to remember not to do once you know them, a programmer used to languages like Python and Javascript might be oblivious to. Things like:
* Forgetting to use buffered IO wrappers
* Using println!() for high-volume console printing rather than locking stdout and writing to it manually
* Everything is a struct with concrete types. This is closest to C. It benefits from the improved memory safety, without sacrificing speed.
* Everything is a Box<Rc<T>>. This is closest to dynamic garbage-collected languages, but with vastly improved performance.
* Everything is an “impl Trait”. This is closest to templated C++, but with much better ergonomics.
So, while I think I agree with your specific statement that the improved memory management may only be fully appreciated by those that cut their teeth on C’s segfaults and dangling pointers, I don’t think that’s the only benefit of Rust as a language.
Wait, reference counting everything all the time is faster than normal garbage collection?
The performance difference, though, is mainly in switching over to a compiled language in the first place.
Why the `Box`?
If you've never used an ML family language then Rust might still be a breath of fresh air even if you don't care about memory management.
This is a common refrain in C++ testing: if it compiles then it's probably correct.
> Rust has accounted for so many errors that many common test cases become irrelevant
In practice, if you think this way I think it's a sign that you aren't testing the right things in those other languages. You should be testing business logic, not language stuff.
If you look at your test code and think "I would test this in JavaScript, but I don't need to do that in Rust" then just delete the test.
I’ve heard this in Haskell and in Rust. I’ve never heard it applied to C++…
std::ptr::null() - an actual null pointer, probably the zero address on your hardware, and this isn't even an unsafe function. On the other hand, you won't find many uses for a null pointer so, I mean, congrats on obtaining one and good luck with that.
std::ptr::null_mut() - a mutable null pointer, similarly unlikely to be of any use to you in safe Rust, but also not an unsafe thing to ask for.
But, these are pointers, so they're not values that say, a String could take, or a Vec<String> or whatever, only actual raw pointers can be null.
First, this is a controlled panic, not a segmentation fault. The language is ensuring that we don't access the null pointer, or an offset from the null pointer. Null pointer access can be exploitable in certain circumstances (eg, a firmware or kernel). Your use of "exception" suggests you're thinking about it in Java terms however, and Java is equivalent here.
Second, you can only encounter this in explicit circumstances, when you have an Option<T>. Wheras in languages with a null type, any variable can be null implicitly, regardless of it's type.
But a segmentation fault is also controlled
>Your use of "exception" suggests you're thinking about it in Java terms however, and Java is equivalent here.
No, I am thinking in Delphi terms. It is overspecialized to Windows userspace. Windows gives an Access Violation, and that can be caught, and Delphi throws it as exception
>Second, you can only encounter this in explicit circumstances, when you have an Option<T>. Wheras in languages with a null type, any variable can be null implicitly, regardless of it's type.
Delphi has both nullable types and null-safe types
A segmentation fault may not happen (which is to say, we may corrupt memory or worse) if the null pointer is mapped into memory (as in an embedded or kernel context) or if it's accessed at a large offset, which results in a pointer that's mapped into memory. This may be rotten luck or it may be exploited by an attacker.
> No, I am thinking in Delphi terms.
Fair enough, I don't know Delphi. I'll take what you're saying about it as read.
How do null pointer exceptions arise?
Inconsistent data: you have code paths that implicitly assume invariants. Trivial cases like “this field is not provided” and more complex ones like “these fields have a specific relationship”.
You can often move those assumptions into the data structure in any language.
Then your tests become a matter of generating and transforming data from a holistic perspective instead of micromanaging individual code paths.
Remember unlike most languages you'd be familiar with C++ has IFNDR, which has been jokingly referred to as "False positives for the question: Is this a C++ program?". A conforming C++ compiler is forbidden from telling you in some† unknown number of cases that it suspects what you've written is nonsense, it just has to press on and output... something. Is it a working executable? Could be. Or maybe it's exactly like a working executable except it explodes catastrophically on Fridays. No way to know.
† The ISO standard does identify these cases, but they're so vague that it's hard to pin down everything which is covered. My guess is that all or most non-trivial C++ software is actually IFNDR these days. Just say No to the entire language.
It's not quite that bad - a conforming C++ compiler is permitted to error out and not compile the program. It just doesn't have to.
Surely in the undecidable cases the compiler is allowed to produce a binary that errors cleanly at runtime if you did in fact violate the semantic constraint, and any sane implementor would do that. (Not that any sane person would ever write a C++ compiler...)
It's not merely about whether your comparisons are transitive, the type must exhibit a total ordering or your sort may do anything, including buffer overflow.
> Surely in the undecidable cases the compiler is allowed to produce a binary that errors cleanly at runtime if you did in fact violate the semantic constraint,
I don't think I know how to prove it, but I'm pretty sure it's going to be Undecidable at runtime too in many of these cases. Rice reduced these problems to Halting, which I'd guess means you end up potentially at runtime trying to decide if some arbitrary piece of code will halt eventually, and yeah, that's not helpful.
I've written about it before, but I should spell it out: The only working alternative is to reject programs when we aren't sure they meet our constraints. This means sometimes we reject a program that actually does meet the constraints but the compiler couldn't see it.
I believe this route is superior because the incentive becomes to make that "Should work but doesn't" set smaller so as to avoid annoying programmers, whereas the C++ incentive is to make the "Compiles but doesn't work" set larger since, hey, it compiles, and I see Rust's Non-Lexical Lifetimes and Polonius as evidence for this on one side, with C++ 20 Concepts and the growing number of IFNDR mentions in the ISO standard on the other side.
Sure. But there's no requirement for the compiler to be a dick about it, and hopefully most won't.
> I don't think I know how to prove it, but I'm pretty sure it's going to be Undecidable at runtime too in many of these cases. Rice reduced these problems to Halting, which I'd guess means you end up potentially at runtime trying to decide if some arbitrary piece of code will halt eventually, and yeah, that's not helpful.
At runtime can't you just run the code and let it halt or not? Having your program go into an infinite loop because the thing you implemented didn't meet the requirements is not unreasonable.
I'm sympathetic to the idea that there could be a problem in this space, but without a real example of a case where it's hard for a compiler to do something reasonable I'm not convinced.
> I've written about it before, but I should spell it out: The only working alternative is to reject programs when we aren't sure they meet our constraints. This means sometimes we reject a program that actually does meet the constraints but the compiler couldn't see it.
> I believe this route is superior because the incentive becomes to make that "Should work but doesn't" set smaller so as to avoid annoying programmers, whereas the C++ incentive is to make the "Compiles but doesn't work" set larger since, hey, it compiles, and I see Rust's Non-Lexical Lifetimes and Polonius as evidence for this on one side, with C++ 20 Concepts and the growing number of IFNDR mentions in the ISO standard on the other side.
Meh. I'm no fan of the C++ approach, but I'd still rather see C++ follow through on its strategy than half-assing it and becoming a watered-down copy of Rust. People who want Rust know where to find it.
> In addition, some identifiers appearing as a token or preprocessing-token are reserved for use by C++ implementations and shall not be used otherwise; no diagnostic is required. [...] Each identifier that contains a double underscore __ or begins with an underscore followed by an uppercase letter is reserved to the implementation for any use.
Thus, the compiler theoretically has the liberty to emit whatever it wants for this program.
Neither GCC nor Clang produces a warning under -std=c++20 -Wall -Wextra (Clang only produces a -Wreserved-macro-identifier under -Weverything), and MSVC doesn't produce a warning under /std:c++20 /Wall.
In practice, most examples of ill-formed programs where compilers issue no warnings occur with discrepancies between different source files that are linked together; e.g., declaring a function as inline in one file but non-inline in another, or declaring a function with two different sets of default arguments in different files, or defining the same non-inline variable or function in different files, or defining the same inline function differently in different files.
In theory, sure. In practice, what does it do? We can look and see.[0][1][2] (I don't know of any compiler that emits garbage in the example you listed). For some reason, there's this sentiment that's arisen that treats undefined behavior as some sort of bugaboo that's capable of anything and everything (including summoning nasal demons).
Here's a question that I can't find an answer to. Is machine code well-defined (as in, can it contain undefined behavior)? If the answer is yes, then all Rust programs can also contain undefined behavior, because they eventually turn into machine code after all. If the answer is no, then that means once a C++ program is compiled into machine code, the executable is a well-defined program.
This whole nasal demon garbage was a good hyperbole at the time to explain that, yes, undefined behavior is bad. But it's been taken to this extreme where people will use arguments like this to try and convince others that if your program has undefined behavior, then it could summon a nasal demon, because who knows what could happen.
In reality, that won't happen. In reality, a signed integer overflow can result in a meaningless program, or a security vulnerability, or many other bad things, but it doesn't mean that the compiled machine code is all of a sudden inventing new instructions at random that the CPU will happily accept. It doesn't mean that your program will turn your OS into a pac-man game. It doesn't mean that it's impossible to find the root cause of the issue and remove the undefined behavior.
In practice, undefined behavior means that your program is broken, but you can look at the compiled program and trace exactly what it does. Computers are machines, they behave predictably (for the most part). They fetch an instruction, execute the instruction, and read/store results to memory. If you look at the instruction and memory (the cache stored on the specific core that the processor is using), you can reliably predict what the CPU will do after executing that instruction.
And yes, multithreading exacerbates the effects undefined behavior and makes it more difficult to debug. But you can debug the issue, even if it means you have to look at the machine code directly.
So while, yes, a compiler can emit garbage when it encounters the example program you gave, using that as an argument for why undefined behavior is bad is dumb. Because in reality, compilers will do something that makes sense, and if it doesn't, you can just look at what it produced. In all the examples you listed, I'm sure that if you created a small example illustrating those situations, the compiled program would do what you expect it to. (Like in the case of having different default args, it will probably just throw an error like this[4]).
Sorry for the rant, it just annoys me that we can't discuss undefined behavior without somebody making an argument that doesn't matter in virtually every case. Undefined behavior is bad! But there are good, real, common examples that show why it's bad! Use those instead instead of talking about how compilers are technically permitted to spew trash when they encounter a program that looks normal, because I'd be very surprised if any modern compiler exists that does spew trash for a normal looking program.
This talk by Chandler Carruth seems pretty good on explaining the nuance of undefined behavior[5].
[0]: https://godbolt.org/z/zv4GhPK5E
[1]: https://godbolt.org/z/6obn7134G
[2]: https://godbolt.org/z/33cKcx5xs
[3]:...
This is a conscious choice that C++ made, and my belief is that once you make this choice you have an incentive to make it much worse over time e.g. in C++ 11, C++ 14, C++ 17, C++ 20, and now C++ 23. Specifically, you have an incentive to allow more and more dubious code under the same rule, since best case it works and worst case it's not your fault because if it doesn't work it was never actually a C++ program anyway...
Still, since you decided to talk about Undefined Behaviour, which is merely unconstrained misbehaviour at runtime, let's address that too.
For the concurrency case no, humans can't usefully reason about the behaviour of non-trivial programs which lack Sequential Consistency - which is extremely likely under Undefined Behaviour. It hurts your head to even think about it. Imagine watching an avant garde time travel movie, in which a dozen characters are portrayed by the same actor, the scenes aren't shown in any particular order and it's suggested that some of them might be dreams, or implanted memories. What actually happened? To who? And why? Maybe the screenwriters knew but you've no idea after watching their movie.
Today a huge proportion of software is processing arbitrary outside data, often as a service directly connected to the Internet. As a result, under Undefined Behaviour the unconstrained consequences may be dictated by hostile third parties. UB results in things like Remote Code Execution, and so yes, if they wanted to the people attacking your system certainly could turn it into a Pac man game.
We should strive to engineer correct software. That we're still here in 2023 with people arguing that it's basically fine if their software is not only incorrect, but their tools deliberately can't tell because that was easier is lunacy.
From the link I posted, from cpp reference, which gives a definition for what constitutes undefined behavior:
>> ill-formed, no diagnostic required - the program has semantic errors which may not be diagnosable in general case… The behavior is undefined if such program is executed
And as for the rest of your argument, you still haven’t answered the one question that matters. Is an executable file with machine code well-defined? If it is, then once you compile a C++ program, the generated binary is well-defined. And once again, all the superfluous arguments about what could happen are irrelevant when there’s no context provided. There’s lots of different types of undefined behavior. An integer overflow by itself does not make your program susceptible to remote code execution. It’s the fact that the integer overflow gets stored in a register that later gets used to write to invalid memory, and then that invalid memory has harmful code injected into it that gets executed.
We should strive to engineer correct software. I agree. It’s all this hand waving about how all undefined behavior is equal that irks me. Because it’s not true, as evidenced by the example listed above about the program that’s “technically” ill-formed C++, but in practice compiles to a harmless program. Engineering requires an understanding of what could go wrong and why. Blindly fretting about all undefined behavior is useless and doesn’t lead to any sort of productive debates.
Engineering is all about understanding tradeoffs. Which is one of the reasons the C++ spec does not define everything in the first place. The fact that in 2023 people don’t understand that all decisions in software come with a tradeoff is lunacy. And once again, I agree that undefined behavior is bad, but to pretend that the existence of IFNDR means the whole language is unusable is silly. People use C++ everyday, and have been for almost 40 years. I’d like more productive conversations on how rust protects the user from the undefined behavior that makes an impact, not about how it doesn’t have IFNDR because that’s irrelevant to the conversation entirely.
[0]: https://en.cppreference.com/w/cpp/language/ub
While you've insisted that's somehow the one thing which matters I don't agree at all. Are programmers getting paid to produce "any executable file with machine code" ? No. Their employer wanted specific executables which do something in particular.
And there aren't "different types of undefined behavior" there's just one Undefined Behaviour. Maybe you've confused it with unspecified behaviour ?
In the integer overflow case, because that's UB (in C++) it's very common for the result to be machine code which entirely elides parts which could only happen under overflow conditions. Because overflow is UB that elision didn't change the program meaning and yet it makes the code smaller so that's a win - it didn't mean anything before in this case and it still doesn't - but of course the effect may be very surprising to someone like you.
The excuse that "We've done it for decades therefore it can't be a bad idea" shouldn't pass the laugh test. Did you notice you can't buy new lead paint any more? Asbestos pads for ironing boards? Radium dial wristwatches? "That's a bad idea, we shouldn't do that any more" is normal and the reluctance from C++ programmers in particular shows that they're badly out of touch.
Are there any efforts to overhaul or completely rethink this?
Algebraic effects would solve this, but I don’t know any language other than OCaml that is working on that approach.
You can convert colors simply but only one way, and practically that works, functional core imperative shell, etc.
* The async runtime and async functions are decomposed but tightly coupled. That means while you could swap out runtimes, a crate built against one runtime can’t generally be used with another unless explicitly designed to support multiple. I believe C++ has a similar problem but no other major language I’m aware of has this problem - that’s typically because there’s only a single runtime and it’s embedded in the language. Things like timers and I/O are not interoperable because the API you use has to be for the runtime you’re running under. I believe there’s work ongoing to try to remedy this although in practice I think that’s difficult (eg what if you’re using a crate relying on epoll-based APIs but the runtime is io_uring).
* async in traits. I believe that’s coming this year although the extra boxing it forces to make that work makes that not something 0-cost you can adopt in a super hot path.
* async requires pinned types which makes things very complex to manage and is a uniquely Rust concept (in fact I read a conceptually better alternative proposal for how to have solved the pin/unpin problem on HN not too long ago, but that ship has long sailed I fear).
* The borrow checker doesn’t know if your async function is running on a work stealing runtime or not which means there’s a lot more hoop jumping via unsafe if you want optimal performance.
* async functions are lazy and require polling before they do anything. That can be a bit surprising and require weird patterns.
Don’t get me wrong. The effing-mad crate is a fantastic demonstration of the power of algebraic effects to comprehensively solve coloring issues (async, failability, etc). But I think there’s stuff with runtime interop that’s also important. I don’t think anyone is yet seriously tackling improving the borrow checker for thread per core async.
[1]: https://www.unison-lang.org/
[2]: https://www.unison-lang.org/learn/language-reference/abiliti...
In general I’ve noticed that there are a lot of code generation modules that can generate very cryptic error messages if you stay clear of their happy paths.
Any kind of shared resource is a tedious experience, that's basically it's selling point.
It is frustrating becasue if you go look for help, people are always condescending about global shared resources like that but there really isn't a better way.
tl;dr: You can have different (nicer to program) abstractions, but you can't have them in the same language you use to write firmware for a $10 electronic device, or Linux drivers, and we already have languages like Go and Javascript whereas we did not have a safer alternative to C++.
Adding it to your project probably increases the average code quality:)
I feel this one, something about spending too much time in JS makes me not want to write clean software..
This is a pretty negative and unfortunate take on rust that I cannot recognize after having spent the past 2 years professionally writing backend rust. I did write c++ before that so it's likely that it takes some experience with the pain of an unhelpful (and in some cases downright hostile) to fully understand and appreciate the so called "yelling" rust does.
From my perspective every lifetime complaint the compiler has is a deeply appreciated hint that not only saves you countless hours of debugging down the line but also makes me feel confident/safe in the code which you never could achieve in a c++ codebase without making defensive copies everywhere.
Being able to lend/borrow data without fear truely is rust's greatest strength and something people coming from GC languages have a hard time appreciating.
One kind of pain has a bigger blast radius than the other. I prefer compile time pain to runtime pain.
I would find that argument passable if some of the biggest products from the largest companies functioned better. Performant software seems like a distant memory.
In startups that chasm is generally filled with VC money paid to cloud vendors, but if you're bootstrapping or not looking to be a unicorn, you probably will want to drive down infrastructure costs much earlier.
One need only open Microsoft Teams to see the “billing time” costs. If the billing time cost is not paid by the developers or their company, then it stop being a cost to them and it becomes an unpriced externality. Who cares, who even knows, that this app we’re writing runs slowly on regular people’s old hardware.
Or if you write mobile app and became slow and laggy, then you lose users.
Was that the result of a process in which the compiler “abused” me? I don’t think so.
Meh, the fact that Rust's compiler is overly strict is just a truth (All compiling programs are valid, but not all valid programs are compiling), and I find it lowkey annoying that every time someone has a problem with it, rustceans jump to the defense to the point of almost denying the OPs experience. I mean on a case by case basis it might simply be a difference of experiences and both are valid, but on a larger scale it does look like a bit of a pattern, at least to me.
Ofcourse, there restrictions aren't just a whim of a Rust core team (usually ;p ), and do come from practical limitations but regardless, it's fair to be frustrated at them.
The author also wrote >[Rust's] dev tooling leaves much to be desired
I can't even begin to comprehend such a statement. Rust's tooling ecosystem is the number one reason why I want to convince my colleagues to give it a shot. Its literal heaven coming from C++ with cmake...
Do you need bare metal performance? Then why use Rust when you could use C or C++, which have much larger ecosystems, platform support, more mature tooling, etc.
Do you need BOTH memory safety and baremetal performance at the same time? Then there really aren't many other options besides Rust.
But what I've started wondering lately is: are there really that many situations where you actually need both of those things at the same time? C++ tends to get misused a lot, but I feel like the same thing is happening with Rust.
I think that’s an unusual way frame that requirement.
All programs need to handle memory _correctly_. Very few seg fault as part of expected operation.
Is that better?
Personally, I've never met anyone who can ensure memory safety in their C/C++ without onerous restrictions even more severe than those rust imposes, but maybe you're superhuman.
I've done professional c++ for a couple decades. My pipelines usually were conan with some dependencies that were not. The used clang tidy and format, san, cppcheck, coverity and tested release and debug builds with clang and gcc. Coverage was sometimes just clang. But sometimes gcc as well. Tests were usually google test. The team standard IDE was vscode.
Rust has rustfmt which is one standard. It has clippy which in addition to actual defects it forces commin idioms so code looks familiar. It has san and llvm coverage. It has miri. It supports aflop for fuzzing. And it has a single tool for package management and build. Edit: IDE is still vscode and it works great.
The only tooling missing for rust would be formal analysis, which is in work (and which isn't that great a story on c++ either) and gcc. A gcc front end is in work, and there is mrustc to generate c from rust, but yes, gcc front end support would be nice to get all those extra targets without the extra work.
But of course there are use cases where you need memory safety guarantees and bare metal performance. In those cases, sacrificing some memory safety by using Rust is an acceptable tradeoff I think.
All memory-safe languages are built on an unsafe foundation. The Java HotSpot VM is very unsafe C++. That's just how computers work.
You aren't meaningfully sacrificing memory safety by using Rust because the unsafe sections are clearly marked out. That's similar to unsafe C#, Python with ctypes, and many other memory-safe languages.
No, it doesn't. What's interesting isn't so much that random HN posters believe this sort of thing, because hey, who needs to know anything about a topic to post their opinion on a forum right? No, what's fascinating is that this applies to people like Herb Sutter in his "cpp2" language, here's Herb:
> I don’t like monolithic "unsafe" that turns off checking for all rules
Nobody does that. It's possible Herb knows that and is being deceitful but honestly I think it's more likely that without investigating at all Herb has just decided everybody else is an idiot...
The exact same code, without the transmute (just returning x directly from an unsafe block), fails a type check related to borrowing. Specifically, the compiler can't see why (without a transmute) it should believe that the reference x now has a different lifetime 'b instead of 'a. Which is fair because it doesn't.
The transmute is you claiming it does, and since transmute is an unsafe function that's entirely on you to make sure you're right about that. So, lying has the expected effect.
The borrow checks aren't magic, they can't look into your soul - but they are running inside unsafe code too.
* Edited to make explicit that the alternative is still marked unsafe and yet doesn't work
That, and the borrowing/safety aspects of Rust also shake out into threading/concurrency, where the compiler does a pretty good job of forcing you to adopt better practices in regards to state sharing, locking, concurrency.
Cargo, though... I am coming around to thinking isn't great -- workspace support remains half-assed, especially... and crates.io is amateur-hour for reasons many people have pointed out elsewhere on this post. I have been tempted to switch my personal projects to bazel, with 3rd-party deps checked in/submoduled.
And it offers those things along with a fantastic type system, and great tooling.
No. This is an exceedingly rational anger.
This isn't necessarily a problem. C++ and Rust can coexist pretty well in practice, and Rust is a good entry point for learning how to write systems software.
So I guess I wouldn't be surprised that the applications are trending that way, too, as that is where growth is happening right now it seems.
On the other hand, we use Rust on the backend of our web based saas product, and it serves as a great filter for quality developers for us.
One thing I don't appreciate is nonorthogonality of certain uses and control structures that cannot be refactored into a function despite structural equivalence without introducing a multiple borrow conflict.
A good C compiler does this when you turn on all the flags. I like languages/compilers that let you selectively disable the screaming and let you write bad code on purpose. Bad code that works but can be written fast is often better than perfect code that takes forever to write. Once you have a bad but working POC, you can make it less bad.
"It’s got no problem attracting new users, but it’s not resulting in dramatically improved libraries or tools. It’s resulting in one-off forks that handle specific use cases."
Age has nothing to do with that. Attracting core devs is hard, and putting lots of effort into making it attractive is necessary. On top of that, cultural conventions are set by the early adopters, and a lack of convention is often just as sinful as a bad existing convention.
Take Python for example. They took a lackadaisical approach to development and runtime environments, and as a result there's 50 competing ways to develop or run a Python program. Their most-used package repository, PyPI, has been a mess for years. Nobody builds on top of existing packages, names make as much sense as a random word generator, the ecosystem is rife with malware, you can't even search for a package on the command line, etc etc. None of that is the language's fault, it's the community and core team's fault for sitting on the sidelines rather than leading. Culture matters more than the tech it's centered around.
(I'm not trying to pick on them, I just know their problems better. C has been around for a half century and its community never really put together half the solutions more modern languages did)
Rust supports that. Just mark everything unsafe.
You can instead wrap everything in Arc<Mutex<T>> and .clone() liberally, though.
There will probably be weird behavior, though, because Rust optimizes based on assumptions about boxes and references. For example, if you have 3 raw pointers to some object live at once, and you give some library function a mutable reference made from one of those pointers, the compiler will optimize assuming it has exclusive access to that object, and it may make incorrect assumptions.
And all those let you do is drop lifetimes and wave goodbye to the borrow checker. You just have to be explicit about it.
(What I mean is, the borrow checker checks borrows, it checks references. In unsafe you can make a reference forget what it's referring to. Just change &'a T to &'static T, and then the borrow checker is doing nothing.)
I checked the six most recently published crates on crates.io (blablabla, nutp, tord, g2d, testpublishtesttest, hellochi, at the time of writing). Three of those (blablabla, testpublishtesttest, and hellochi) did some variation on printing `hello world`. g2d seems like an interesting graphics library. tord provides a data structure for transitive relations, which is also neat. No crate contained over 250 lines of code. Unsurprisingly, none of them contained unsafe code.
Elsewhere in this thread, it's been pointed out that as a consequence of crates.io having a global namespace, plus lax enforcement of an anti-squatting policy, there are a lot of namesquatting packages. Those presumably contain no unsafe code.
tokio contains unsafe code. rand contains unsafe code. regex contains unsafe code. time contains unsafe code. (method: a smattering of packages chosen from blessed.rs; result: every one that I checked except serde containing unsafe code; epistemic status: eh -- I grepped the codebases, ignoring things that were pretty clearly tests, but might have accidentally included some example code or something that's not part of the core library? Please let me know if I've misattributed unsafe usage to one of these projects, or if I've managed to select a biased sample!)
I'd certainly believe a straightforward reading of the claim "80% of crates have no unsafe code"...but that seems almost meaningless, given that a not-insignificant portion of crates contain basically no code at all? I'd be much more interested in a weighted percentage by downloads: I'd be wildly impressed if 80% of crate _downloads_ contained no unsafe code, and would be somewhat unsurprised if the number was well below 50% -- crates with more functionality would be more useful and therefore more download, but also more likely to use unsafe code, I'd imagine.
Edit: I just noticed crates.io has a most-downloaded list[0] -- I might end up running some numbers on top packages there tomorrow morning, for some more solid data.
[0]: https://crates.io/crates?sort=downloads
Also, you’re replying to someone who I’m fairly sure is on one of the core Rust teams, if not closely involved, I’m somewhat more inclined to trust _them_ when they say 80% of libs don’t contain unsafe (given that it cleanly meshes with my own experience of Rust libraries).
> I questioned my sanity every time I circled back around to the Clippy issue above. Surely, I was wrong. There must be a configuration I missed. I couldn’t believe it. I still can’t. Surely there must be a way to configure lints globally. I quadruple-checked when I wrote this to make sure I wasn’t delusional.
You create a .cargo/config.toml in the workspace root so it covers all your crates.
inside the file:
The only limitation is that rustflags are not additive, so if you have other sources of rustflags like the RUSTFLAGS environment variable it will overwrite this setting.People do sick things with lifetimes and generic associated types. Sick, sick things.
Doesn't feel that way to me.
1. Reading the error messages. I was used to error messages verbosely printing a lot of details which were mostly irrelevant and letting the programmer sort it out, and I developed a habit of skimming them. I had a better time with Rust when I realized it was giving me information that was mostly relevant, and that I should actually read them.
2. Interpreting error messages as feedback rather than failure. I was used to error messages meaning I had done something wrong, and getting them all the time was frustrating. Watching Jon Gjengset's coding videos, I was struck that he wrote code to trigger errors deliberately in order to get feedback from the compiler. I now try to work through Rust error messages the same way I would failing unit tests; I make some small changes, I knock out the errors, and then move on to the next subtask. This keeps the number of errors small and manageable, and gets me into a tight feedback loop with the compiler.
Relatedly, I encountered the idea (in a blog post I no longer remember) that the Rust compiler is more like an automated pair programming partner than other compilers. This change in mindset really helped for me; thinking of it as a friendly helper rather than a loud complainer made the work lighter. (That's why I personally don't like the 'abusive partner' metaphor, for me that mindset makes it toilsome and miserable. Also because I've had an abusive partner, and it's nothing like that, and I could do without the reminder.)
3. Setting up my IDE to show me inline type annotations and error messages. This made the feedback loop tighter, I only have to drop into a terminal for complex or unfamiliar error messages. With practice I can fix most errors using a one-line summary.
4. With practice, I've internalized Rust's semantics, and I've stopped painting myself into corners with unnecessarily cyclic data structures and such. I also read a blog article or maybe a tweet (which I've also forgotten) about giving yourself permission to use Box or clone and not sweating every copy or allocation. It pointed out that if you were writing Python, everything would be an Arc<T>, and you'd think nothing of it; in Rust the performance cost is explicit, so you agonize over it, but a lot of the time it's not a big deal to copy some data or to perform an allocation.
My first six months with Rust were very painful and I struggled to write the simplest programs. (This was in fact, the first six months of my second attempt - the first time I tried to learn it, I gave up.) Eventually I learned how to work with it, and since then, it's my absolute favorite language to work in (though presently I mostly write Typescript and SQL, because I'm writing webapps). I tried to summarize what happened along the way and what changed.
If you want sugarcoating or simply rapid prototyping I'd use a dynamic or scripting language instead.
Rust's error messages are one of a kind. There no other compiler which comes even close.
As a side note, i used latex lately, it's error messages are horrendous. What a nightmare to figure out what's wrong by inspecting the error.
Generally good, but man do I hate how any error in my async function causes every recursive call site to generate an error about how the Future is no longer Send and Sync. Literally an entire console scrollback of errors with the actual syntax error buried somewhere in the middle.
In this case, the first two errors are clear. The next 3 provide multiple, redundant context blocks. The upshot is that this simple example results in rustc printing 11 lines of useful error messages and 86 lines of useless messages. Add to that the fact that the useful error messages need not be at the top or bottom of the error list, they can be anywhere inside of it, depending on the declaration order.
We do spend a lot of time trying to silence errors that are irrelevant. We also get a lot of complaints when fixing a single error produces a wave of new errors that were hidden due to failures in earlier stages. It's a balancing act.
Also, specific errors are verbose in order to give people a fighting chance to fix their issue. An error that is too verbose is annoying. An error that is too terse will leave users in the cold as to what the problem was. It's yet another balancing act.
I filed https://github.com/rust-lang/rust/issues/117233 for you. Could I ask you why you didn't consider doing so when you first encountered this problem?
Edit: good news, from the next stable onwards the issue you saw no longer reproduces. You only get the parse errors with your case. https://play.rust-lang.org/?version=beta&mode=debug&edition=...
Thanks for looking into it. I have a filed a bug against Rust before, but that was a clear bug, not a poor error message. Remember, the only time a user sees something like this is when they are trying to do something else. The only reason I looked into it just now is because you explicitly asked.
My main gripe is that I still don't fully comprehend lifetimes and the compiler can't really help me every time, because it (understandably) errs on the side of caution.
I'm not sure any compiler, even one "designed for diagnostics," can gather the "necessary metadata" from inside my brain based on my original intentions. If the compiler could unambiguously interpret what I meant to write, it wouldn't have needed to fail with a compilation error in the first place. Whenever I see something like "perhaps" or "maybe" in a compilation error, the information just didn't exist anywhere the compiler could possibly get to, and it's just a suggestion based on common mistakes.
I am a former C and C++ programmer who lived calling into pthread almost every week for a decade. I use async rust everywhere now.
I don’t get the hate that async gets. In my opinion, everyone should be using async for everything. Including stuff that’s seemingly single threaded “simple” stuff.
1. Fix rhai::Engine so it is Send (if !Send is unintentional)
2. Use tokio::spawn_blocking or normal threads to run the rhai::Engine bits
3. Don't hold the rhai::Engine across an await point.
Which one depends on rhai Engine details and what you want to accomplish.
Doing a block on inside a new thread seems unlikely to do anything useful (unless there's some undisclosed detail that makes it reasonable).
I encourage you to ask about this in the rhai repo in a discussion or issue.
2. I am using a normal thread but when I make it async the compiler wants Send.
3. The await point is in code that uses the engine. I am not sure there is another good option, since I need to use an API that has several libraries all of which are async.
The block_on is to allow the tokio Runtime created in that thread to execute/poll it
So, thank you for your input, I will test out the suggestion that I mentioned above, and then maybe look into spawn_blocking if that doesn't work.
You can have multiple runtimes, but create them ahead of time, in synchronous main, and keep a Handle to them.
But you probably need: https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html
This is likely because you are using the multi-threaded scheduler, which requires futures to be Send, even if you’re running only a single thread. This is because Tokio is based on a “work stealing” runtime, so in “normal” operations, expects the futures themselves to be able to be shuffled around threads where necessary.
For you use case, try running the single threaded executor, additionally try the “local set” executors. These do not require Send as they are statically guaranteed to be confined to the thread they are spawned on. Block on will also work.
Out of curiosity though, how is the interaction with Rhai performed, are you passing around the engine, and executing the code at certain points where applicable? Do you just need certain results from it sometimes? Etc?
I think I have a lot of good advice to go try now. Thanks.
From a quick look at the source, it looks like it switches between using Rc/RefCells and Arc/RwLock.
[0] https://docs.rs/rhai/1.16.2/rhai/#enable-special-functionali...
For instance, you apparently cannot `await` in the main function without a decorator you import from, you guessed it: Tokio.
Once you stop fighting the fact that tokio = async rust for 99% of cases, things are quite smooth.
Does "use async rust everywhere now" imply "use tokio everywhere now?" Honest question.
This is handy for I/O that can be interleaved and cancelled. You can (ab)use it for other things like generators or various DIY multitasking operations. It can also be a state machine generator (e.g. AI of actors in a game).
But I think OP just meant async for typical networking and DB interfaces. And yes, this usually implies the Tokio dependency.
Similarly, I find tasks that use or reuse large buffers to avoid the performance hits from allocation, often benefit from old fashioned thread pools. Bump or shard allocators can make this work ok, but in the cases where you are cpu bound on tight loops of vectorizable operations, thread pools perform better. Async is a good tool, but there are contexts it isn't optimal for.
I learned the opposite, that programmers tend to make everything unnecessarly async when it's not needed causing more complexity and mental load.
It's fine. It won't be the only language I'll be proficient in while being averse to it at the same time. But I heard so many people proclaiming their love for it that I expected to enjoy it, too.
Also, just generally, it tends to make even simple things pretty complex. I understand why and am not really objecting to that, but it does make using it a bit like running with lead shoes.
I expect that the latter part may get better as I use it more (but perhaps not -- there are other languages I'm fully competent in but dislike for similar reasons).
For what it is worth, I found pursuing Zig to be a breath of fresh air. It's a promising alternative in the same space as C (and also Cpp and Rust). Checkout Andrew Kelley's introduction to the language.
For me, the key to understanding the borrow checker was understanding the underlying memory model. Rust memory model is the same as that of C, with some extensions for abstractions like generics. The borrow checker rules seem arbitrary at first. But it's deeply correlated to this memory model. The real value of the borrow checker is when I trigger it unintentionally. Those are bugs that I made due to lapse in attention. What scares me is that another language like C or C++ might simply accept it and proceed.
Yet another pleasant side effect of Rust's strict type system and borrow checker is that they gently nudge you to properly structure your code. I can say for certain that Rust has improved my code design in all the languages that I use.
1. Use Rust's runtime safety checks, using Rc, Weak, RefCell, etc. It will be as ergonomic as GC'ed languages (no productivity loss). While this has a runtime performance penalty, it will still be mostly comparable to other languages. This works for most use cases.
2. If you need the last ounce of performance, just drop all automated safety checks and do it manually using unsafe. Even if you make a mistake, your debugging will be limited to the unsafe blocks. This approach isn't unusual in Rust.
3. Use either the standard library or something on crates.io that does what's given in 2. Rust's generics make it easy.
This resource deals with these topics in-depth: https://rust-unofficial.github.io/too-many-lists/
2. No, I don't need every last bit of performance. C++ performance is OK. Or any other sane language like Nim, Crystal, etc. I need async, though, which is yet another hell in Rust.
3. This absolutely can't be done with standard library. There are libraries like PetGraph, which solve this particular problem (by some very unobvious approaches and bits of unsafe code), but there are many problems like it which don't have any libraries for it yet.
I do have hard problems. I want a language which makes hard problems easy, and impossible problems hard. Rust is definitely _not_ such a language, and it makes me like 5 times less productive, and sucks all joy out of programming.
https://docs.rs/petgraph/latest/petgraph/