I read the article, and the linked GitHub issue, and posted it. I'm curious if people who use this for a living and aren't some reporter pasting snippets agree with this framing. The variety of responses here now suggests many do not.
Ok, well I'm a professional rust developer and I agree with the other Rust developers directly quoted in the article, of which there were many.
But this isn't a Rust developer issue so much as a distro and security issue. Most Rust developers don't have to concern themselves with supply chain attacks or security audibility, so to them this probably seems like a weird thing to focus on.
But as someone who does have to worry about such things, trading off the ability to audit your supply chain for a small, one-time compilation boost on your first compile is a _terrible_ trade-off.
The tinfoil hat explanation is that the developer is breaking all projects that depend on the default package manager cargo, forcing them to move to buck or bazel - projects that the developer of serde-derive has invested work into.
This is not noisy for no reason - the expectation is all dependencies are compiled from source on your computer, and here the developer forces the use of a binary program, with no alternative.
As someone who hasn't used Rust seriously since 2018, I'm tangentially curious what are the improvements (real or perceived) that buck and bazel offer over cargo? I didn't know there were alternative package managers, and cargo always seemed pretty nice to me.
Answering my own question, sorry, looks like I may have got the wrong end of the stick here. Bazel and buck seem like language-agnostic build tools, rather than rust-specific package managers?
> Bazel and buck seem like language-agnostic build tools, rather than rust-specific package managers?
Yeah. Bazel is the sanitized version of Google's internal build system for most of their projects (known internally as blaze). Buck is Facebook's equivalent.
The previous comment is pretty paranoid thinking, so I wouldn't take it seriously.
That said, Bazel and Buck are good for handling projects "at scale" (e.g. company wide monorepos at mid-to-large companies), or with a lot of different needs (e.g. sources in C++, Rust, Python, and Javascript that need to be combined together).
They're not really an alternative to Cargo, and more the next step up for when Cargo isn't sufficient any more. But because they're dealing with much more complicated problems, they're also more complicated to set up and use, so if you're not running into issues with Cargo, you probably don't need to go in that direction (and possibly never will).
Is it worth it? Maybe. Downloading blobs isn't ideal but could be made easier in the future. Ideally a hack like this wouldn't be needed and we'd have comptime serde.
10x sounds great until you look at this as an absolute value. It's 10 seconds.
That would be huge if it were a a runtime cost, but this is a one time cost at compile time, and this build will be cached unless you're upgrading serde (or its dependencies). The value of that 10 seconds compared to the cost is a faustian bargain.
We mainly use high level languages like TypeScript or Python because they work well for us. We do use C/C++ from time to time though, both for high performance requiring bottlenecks and for embedded since we do work with solar plant technology. A couple of years ago we looked into Rust and found it a combination of nice and wanting. It’s sort of great for low level programming and not really worth your time for anything else. Or at least that’s what we decided. We did adopt it and build a few solar inverter applications with Rust where we might’ve used C/C++ before, and it’s a decent language and nice to work with. But then a few months back, or maybe it was longer, some drama hit the ecosystem. Again it was so unimportant to us that I can’t even begin to remember what it was about, but similar to this Serde “drama” it speaks about an immature ecosystem and community. From a purely risk management offset we decided to stop using Rust for the time being. Which probably sounds more dramatic than it is, because we don’t really care about a lot of these issues and we’re probably never even going to be affected by them. But why would we want to spend time and energy on a technology which might change a lot? Well, the quick answer is that we won’t. Take this Serde issue as an example, it’s a very used package. I’m not sure exactly what people use it for, I assume that it’s for things we would likely build in Python, but even with 170 million users it’s main maintainer seems to be an issue. I even agree with the maintainer, but if the Rust ecosystem doesn’t have someone who can fork it and maintain a non-binary version to meet the wants and needs or all these noise makers, well then that’s not exactly a mature ecosystem.
So I think the answer is that it’s mostly just noise, but it’s noise you probably don’t want to have to deal with unless you’re invested in Rust itself.
> I'd also be curious if anyone has any good explanations as to why a choice like this would be made?
It has to do with how Rust builds are structured.
The compilation unit is the crate, and the result of compiling it depends on the source code, the version of hte compiler, the compiler options, the target triple, and the set of features (think #ifdef FEATURE_FLAG).
Since there's exponential blowup of possible combinations of a single crate, the largest package registry (crates.io) does not cache compiled crates. It only caches their metadata, and when Cargo (the package manager) does dependency resolution, it uses that metadata alone - and your build system (also Cargo, most of the time) will need to actually build all those artifacts.
Seems pretty reasonable, right?
There's a special kind of crate called a proc_macro crate (procedural macros). When you use these crates you're not adding their code to your compiled output - you're creating a compiler extension - the compiled crate is loaded by the compiler, and then used to generate code in your crate, which is then compiled by the compiler.
One of the reasons Rust build times (particularly clean builds) is that they first have to compile all these compiler extensions and then use them to generate the code that the compiler compiles. These crates are super useful however, like serde, which generates serializers/deserializers that can be used for a wide variety of encodings (XML, JSON, TOML, various binary encodings, and so on).
It would seem that the same rules apply for normal crates - there's exponential blowup of compiler version, target triple, and features - but that's not really true. proc_macro crates usually don't have features, the compiler ABI is stable enough that they probably won't break across versions, and almost everyone and their mother is compiling on a handful of host systems (x86_64-linux being the most important one).
So while it's true that crates-io probably shouldn't cache every crate for every permutation of compiler/target/feature/etc, a few crates are so pervasive and used in such a sparing manner that it should be possible for the registry to host those
In fact some build systems can do this automatically. But Cargo can't. But it does have a pre-build feature (build.rs) that works like proc macros, and can pull down a pre-compiled proc macro crate, so widely used ones can use this hack to get around limitations of cargo.
My personal take is that the reason that crates-io and Cargo haven't been upgraded to do this, despite years of people complaining, and known solutions existing, is mostly personal/political. There aren't many people willing to put in the work to do it, and the teams haven't been open to contribution of this kind for a long time. For example, they do not believe that Cargo is a build system, and a lot of this is "out of scope" for cargo.
Plenty of projects ship binaries without a problem. Nearly every Linux distribution, for example. What’s special about Rust in that binaries for dependencies are a problem?
Also, for the “why” - is it just the very slow build times for Rust?
> Nearly every Linux distribution, for example. What’s special about Rust in that binaries for dependencies are a problem?
The inability to opt out of running a non-reproducible binary library just to build code. This isn't just devs complaining, it's against several large orgs security policies.
> Also, for the “why” - is it just the very slow build times for Rust?
It's an initial build time thing. It makes no difference except after a full clean or fresh build.
The binary is in the package you would be vendoring. That's half the problem; the maintainer put binaries in something consumers treat as a source archive.
Plenty of projects ship a) reproducible and b) application binaries.
serde-derive is a macro that is used in the compilation of projects. The way the developers have implemented it, when you build a project _from source_, it pulls and executes a binary blob against your code.
The binary in serde-derive is not reproducible (at least, there is no published hash and other users haven't been able to compile the same binary on their computers), and saves a very minor amount of time - serde-derive has to be compiled once per project per compiler version, unless you delete your dependencies.
Yes, binary packages are a bad idea everywhere. Usually, their justification comes from two sources:
1. Users may not have the correct build toolchain installed. (Mostly relevant for end-user applications, or native FFI modules.)
2. Building every package from source is slow.
Neither justification makes sense for serde-derive. For 1, serde-derive is just regular Rust code, built by their regular Rust toolchain. You already need to have the toolchain installed to use it! Hell, you need the same toolchain to build the wrapper library for the binary! For 2, serde-derive has a tiny dependency tree, and all of those are likely to be shared by any other proc macros the user depends on anyway, so the only thing you're skipping is... compiling serde-derive itself.
Yeah, the whole Java ecosystem for example is built on binary dependencies, and is a very healthy one in spite of that. The same holds for C#/.NET and Apple frameworks. Having the source available, or preferably open, and preferably with reproducible builds, is of course desirable.
I dislike some of the framing of the article. In particular:
> Valentini further inquired to the project maintainers, how were these new binaries "actually produced," and if it would be possible for him to recreate the binaries, as opposed to consuming precompiled versions. David Tolnay, who is the primary Serde maintainer, responded with potential workarounds at the time.
The article links to the response [1] as well, which is very good. However, the response doesn't provide workarounds. It provides clear instructions:
>> how is the x86_64-unknown-linux-gnu binary actually produced? Would it be possible for us to re-create the binary ourselves so we can actually ship it?
When the article states that the library author only provided "potential workarounds", it sounded really shady to me, and I believe this is mischaracterizing the real response.
The issue is that build.sh doesn't create reproducible binaries. You can run that, but there's no guarantee you'll build the same binary that ships, so you can't automate proof that there hasn't been a supply chain attack.
But I would assume you would build it in your own hermetic build system and use those artifacts as the reproducible artifacts, since you don’t trust the precompiled distribution. Instead you would start your chain of trust with your build.
Yes thanks as I read the issue and the code I realized the advice was to patch downstream. I know a lot of folks who build hermetic and reproducible builds for large amounts of open source code often have to patch downstream, but its fairly rare to see that requirement from such a basic crate/library. Hopefully this resolved at some point, it seems like a pretty aggressive stance to take that puts people who have a hard job (rigorous chain of trust in builds is hard work and done for very good reasons!!) in a harder spot.
I don't know why a library has to be necessarily small in size. Having micro packages that only do one tiny thing would only lead to a npm-like hellscape, and Serde is the most complete option in the Rust ecosystem.
In my opinion it's better to have simple interfaces for complex features, rather than simple interfaces for simple features. You could code those yourself.
Agreed. I'd even go further and say that apart from humphrey_json, the alternatives GP presented provide complex interfaces for simple features. They provide no easy way to convert to/from a strongly-typed struct, which is the reason for serde_derive.
You linked to serde_json and not serde, which this story is about. And the crate sizes listed include not just the source, but also test code and other project files.
This change happened in a patch release, so the serde author either did not consider it breaking, or doesn't follow semver in the first place.
Most crates do not raise their major version when increasing the minimum supported rust version. So there is precedence for treating which changes can break for some people, but which don't have a breaking API changes, as non-breaking for semver purposes.
I'm not sure if he's made an official statement about his versioning policy, but if you look at https://crates.io/users/dtolnay?sort=downloads, you can see that most of his crates follow the incremental patch versioning style.
People have complained about the build time of proc macros for ages in the community. This might be a misguided hack, but the response to this is bordering on a witch hunt, particularly when there is a glaring security hole (build.rs) that most people likely use without second thought every single day. I simply do not believe that most people commenting on this issue are auditing the builds of all their transitive dependencies.
I'm quite ambivalent on this issue overall, but let me just point out:
build.rs absolutely is a glaring security hole in the sense you say, but compared to that, this is much worse. You can verify the build.rs code that you download (at least in theory, and some people in banks or distro packages probably actually do), but binaries are orders of magnitude more difficult to inspect, and with the current Rust build system pretty much irreproducible.
> build.rs absolutely is a glaring security hole in the sense you say, but compared to that, this is much worse. You can verify the build.rs code that you download
In theory you can compile your own blob, but you'll need musl and whatnot to make a universal Linux build. Code for making the blob is there in the repo.
build.rs is at best equal. It can access your locally available DB, and transmit your data.
The problem here is that nothing in that build is pinned. It builds with nightly, but doesn't define which version / date.
We also don't know how it's build. Ideally there is a Docker container out there that does just an import of source code and then builds. No apk install or apt install (you'd do that in a base published layer). Referenced with an SHA256.
We then use this Docker container to pull in the source code AND its dependencies based on a Cargo.lock. Which... isn't there. So we don't know the exact dependencies that went in.
(Even if there were a Cargo.lock, we need to make sure we actually respect it. I believe cargo install by default ignores the lock file and tries to get the latest version that matches).
Yeah, there are already binaries in the crates.io ecosystem, and I'm certain that almost none of these people have audited a `build.rs` file or a proc macro implementation which effectively runs as you, completely unsandboxed.
EDIT: I was wrong, this is not actually `watt` -- it may have been re-using code from the project.
This one of those pile-ons where everyone gets excited about having a cause-du-jour to feel passionate about, while simultaneously ignoring issues that are far more pressing.
> almost none of these people have audited a `build.rs` file or a proc macro implementation which effectively runs as you, completely unsandboxed
The biggest orgs tend to run all of their builds sandboxed, including what happens with build.rs. It's part of how they enforce dependency management day to day, but also helps protect against supply chain attacks.
So not everyone does, but enough people do that you can rely on their complaints for the more well trodden parts of the ecosystem.
> The biggest orgs tend to run all of their builds sandboxed, including what happens with build.rs. It's part of how they enforce dependency management day to day, but also helps protect against supply chain attacks.
Sandboxing doesn't completely prevent supply-chain attacks. You can avoid getting persistent malware on your CI machines, sure. Exfiltration of tokens and secrets during build? Maybe in a small handful of CI setups where admins have carefully split the source fetch, or limited CI network access to Nexus. Exfiltration of tokens and secrets and/or backdooring after the software has been deployed to production? No, build-time sandboxing doesn't help here at all.
On all developer machines as well? No. Very few big orgs do this and only for mission-critical stuff. Some very important ones have docker-based sandboxed workflows or SSH-to-sandboxed-cluster workflows, or air-gapped laptops but that's very, very rare (I worked in air-gapped environment for a bit and it was a massive pain).
Oh wow. I'd be very interested in hearing how they sandbox rust-analyzer. I found a discussion of supporting the analyzer itself by generating config files [1][2], but not how you can sandbox it.
That would be extremely useful as the analyzer is a pretty juicy target and also runs proc-macros/build.rs scripts.
Yeah everywhere I’ve been at back to 1995 does this for high security environments, and if we detected an issue we had security response teams that worked with maintainers and others to remediate.
For lower environments we generally used 3 month or more embargoes on precompiled stuff we couldn’t easily compile ourselves to mitigate some of the supply chain issues if we weren’t directly managing the chain of trust for the binary.
You keep saying this but I suggest you actually look at the code. The precompiled binary is not a sandboxed WASM binary. Despite the name "watt" it has nothing to do with https://github.com/dtolnay/watt . `watt::bytecode` refers to the serialization protocol used by the proc macro shim and the precompiled binary to transfer the token stream over stdio, not anything related to WASM.
Also it's worth noting that even if it was a sandboxed binary ala https://github.com/dtolnay/watt , it's not obvious that distributions or users would be satisfied with that. For example Zig had this discussion with the own WASM blob compiler that they use as part of bootstrapping. https://news.ycombinator.com/item?id=33915321 . As I suggested there, distributions might be okay with building their own golden blobs that they maintain themselves instead of using upstream's, and that could even work in this Rust case for distributions that only care about a single copy of serde for compiling everything. But it's hard for the average user doing `cargo build` for their own projects with cargo registry in `~/.cargo` to do the same replacement.
A really nice (IMO) solution would be to build a wasm blob reproducibly and to ship the blob’s hash, along with a way to download the blob, as part of a release. Then distros could build the blob, confirm that the hash matches, and ship a package that is built from source and nonetheless bit-for-bit identical to the upstream binary.
It’s probably true that most people commenting don’t audit the builds of transitive deps, but the original issue was a distro that couldn’t distribute precompiled binaries, I’m going to guess this has something to do with their license.
I think having an exit path for those that want to compile from source is important, and I can’t understand the reluctance to provide that.
Well, there is an exit path for those who want to compile from source. If you mean build from source for Cargo users, I believe there's issues with how feature flags interact with transitive dependencies that make this difficult. At least, there's comments on the issue that speak to this. Maybe someone more familiar with Cargo can chime in.
At least Linux, OpenBSD, and (with more annoyance) Windows make it relatively straightforward to run things like build.rs in a sandbox. I wonder why Cargo doesn’t do this.
There was also a project (that dtolnay was involved in, I believe!) a few years ago to compile proc macros to wasm.
build.rs is a source file that you can audit. A binary that has no reproducible build is not auditable even if anyone wanted to.
A single person does not audit all of their dependency tree, but many people do read the source code of some if not many of their dependencies, and as a community we can figure out when something is fishy, like in this case.
But when there are binaries involved, nobody can do anything.
This isn't the same as installing a signed binary from a linux package manager that has a checksum and a verified build system. It's a random binary blob someone made in a way that nobody else can check, and it's just "trust me bro there's nothing bad in it".
> and it's just "trust me bro there's nothing bad in it".
The developer should be very concerned about what happens if his system(s) are compromised and the attacker slips a backdoor into these binaries-- it will be difficult to impossible to convince people that the developer himself didn't do it intentionally. Their opacity and immediacy make them much more interesting targets for attack than the source itself (and its associated build scripts).
Saving a few seconds on the first compile on the some other developers computer hardly seems worth that risk.
And at the meta level, we should probably worry about the security practices of someone who isn't worrying about that risk-- what else aren't they worrying about?
It's all the right of the author. And he's done so before.
Unfortunately, I'm afraid this could lead to a more fragmented ecosystem which is a pity.
It takes a long time to build trust but it can be destroyed quickly. I'm not sure if the author cares about that though. From my reading of the comments he doesn't.
"If there is implementation work needed in some build tools to accommodate it, someone should feel free to do that work (as I have done for Buck and Bazel, which are tools I use and contribute significantly to) or publish your own fork of the source code under a different name."
Locked issues with a comment similar to "works for me"
I'm not complaining. It's his project, it's his right. He will be overwhelmed by notifications and this is probably the only way he can stay on top of things.
I should expand: This happened to me twice where there was still discussion between people to find solutions for them.
In both cases the majority of people accepted the answer of the maintainer but it was still unhelpful for them and they were discussing workarounds.
This discussion was cut short by just locking the issues.
The second time it happened to me I stopped our sponsorhip out of frustration. That also sucks and is a bit petty. I acknowledge that.
> I found the discussion to be mostly very civilized and focused on finding solutions for those affected.
I really disagree with this characterization of the discussion. While there's plenty of more or less dispassionate comments focused on finding a solution, there's a greater number of people from the peanut gallery drowning that out with comments that amount to "I don't like this!" at best and questioning the integrity of dtonlay at worst. I feel like this demonstrates some of the worst features of open source software on github, where a bunch of nobodies feel the need to add their totally useless input rather than participating in good faith to find solutions and make the software better. You even see comments like that here, speculating that dtonlay is trying to break cargo and force people to use bazel, which is just an insanely conspiratorial and bad faith speculative interpretation that is obviously untrue and helps no one.
> While there's plenty of more or less dispassionate comments focused on finding a solution, there's a greater number of people from the peanut gallery drowning that out with comments that amount to "I don't like this!" at best and questioning the integrity of dtonlay at worst.
Only see comments doing that are on here.
The comments on the github issue are all pretty productive and constructive. The worst is the commentator that calls the approach 'despicable', but also provides constructive feedback and helps describe the problem.
Even if it was civil it was going to annoy the hell out of maintainer due to sheer number of notificiations. I was by accident in Chrome WEI commit, simply horrible number of notifications.
Also I'm not lying about (albeit hyperbolic) calls for dtolnay's head. From one of Rust Discords.
you just still don't make sense in your replies to users. literally no one asked about discord or reddit, this conversation is about the GitHub discussion.
I didn't move the goalpost - it's how I interpreted the peanut gallery part.
But ok, let's say limiting yourself only to GitHub comments, you can still find doubt casting and uncharitable interpretations:
> Nice micro-optimization. And if you bury your head deep enough into the sand, doesn't sacrifice anything important either.
> "The only supported way ..." is a polite way of saying "Works for me, WONTFIX".
> Another interpretation of an earlier comment is that he is all-in on bazel/buck and does not care about cargo. Both interpretations have me concerned.
> And ~80 comments with "This doesn't work for me".
Even in this case, the __jem comments make sense. Many people are just writing their version, "I don't like this, " meaning any interesting message is drowned out by noise. THAT will make any maintainer limit to contributors.
Are you just making stuff up? Not one of those quotes are comments are on the github issue that was closed and which was being discussed in this thread: https://github.com/serde-rs/serde/issues/2538
I already referenced the worst comment and it falls closer to brutally honest than actually rude.
> Are you just making stuff up? Not one of those quotes are comments are on the github issue
It's customary to assume other side is arguing in good faith and not making stuff up. I assumed you read all the comments, had access to Github, so you wouldn't have problems finding them. But apparently that wasn't the case.
> I already referenced the worst comment and it falls closer to brutally honest than actually rude.
It's hard to take this at face value when you mistook my quotes for outright fabrication. You either didn't read them all, or you convienently forgot the examples I cited.
Also nice micro-optimization comments is dishonest, apparently dtolnay felt the optimizations provided for his use case were strong enough to warrant a fix.
My remark was an observation based on the obvious public resentment, rather than necessarily an exhortation to do so. I promise you, I will not be so cagey and ambiguous when I actually call for someone's head on the chopping block.
It reduces the build time of a clean build by about 10 seconds. Recompiling after changes to your code is not faster, unless the version of serde the code depends on changes.
Compile times. It's the same as using a binary distribution or a source-based distribution. Only very few people want to "recompile the world" when they install new software. Which is why mainstream distributions pretty much always just distribute binaries via apt, yum, etc. However, for the people that do need source distribution, it's very important for them. And it's almost always required in high-assurance environments where supply-chain attacks are a real concern. And those people can't rely on the upstream security of crates.io, or this developer's laptop.
As far as I'm concerned pre-compiled binaries should only be offered as potential default in scripting language ecosystems - there is an expectation that things should "just work".
I don't mind it for when I work in Node - honestly I prefer it. But in Rust? The entire toolchain is designed to compile from source. It may be a bit slow for certain crates still, but that's much weaker justification to make pre-compiled the default.
Crates should compile from source by default as long as source actually exists (ex. has not been lost somehow, or wide distribution of the crate is unlikely). I don't buy any of the malicious theories since the author is well appreciated and respected. It's likely just a misguided choice in this case.
People have rightly mentioned build.rs flexibility - I think there is nuance here and maybe it's good to discuss what default behavior should be expected by users, since sometimes build.rs does wild and useful things.
On the argument of "you're not reading build.rs anyway": yes, but you can, and that's a critical difference.
"No one" is validating all their transitive dependencies for weird binaries, except a few people, and that's enough to get us all here talking about it. In the internet age it only takes one. But if you make it require more work to inspect a blob, you reduce the chances that you'll get even that one person who sounds the alarm. At minimum malicious code will survive longer that way.
Could be neat if serde (and other libraries with proc macros) were able to ship WebAssembly blobs. Wasm blobs are portable and run with restricted capabilities. Rustc could expose an embedded API to allow the wasm to generate code and expand macros without hitting the network.
The sketchy part is it was changed and shipped silently and in a sneaky way, and if it wasn’t for some developer to notice it, he wouldn’t have mentioned it..
However, no need to pushback or anything, consider the maintainer got hit by a bus, fork it, and take it from there.
Also, the “security” guy claiming no one is going to read it, do you read linux kernel every update? No, not you, but other developers do, or whenever someone is interested/auditing they can, that kind of arguments are stupid.
We must, must, must stop assuming the worst in others and jumping to react accordingly. Please, please don't fork serde. I'm sure this situation can be worked out without resorting to such measures.
I’m just part of the peanut gallery, but how can you work something out when the maintainer can and has already just locked the issue and walked away?
I don’t see a problem with the community forking as motivation to get the maintainer to come back the conversation or to provide a path for the community to resolve the issue for themselves. A fork was literally one of dtolnay’s own suggestions, bluff or not.
Me too. As a Rust user and small-scale evangelist, I find this whole thing a bit interesting and also somewhat sad. At least, it seems like increased awareness, and perhaps sandboxing, of proc macros are good outcomes that may result.
> how can you work something out when the maintainer can and has already just locked the issue and walked away?
That's not what happened here. dtolnay has not walked away, and the discussion is progressing. For instance, this pre-RFC [0].
> I don’t see a problem with the community forking
Serde, to the limited extent that I understand it, works because it is a single shared interface - multiple forks would be counter to the purpose of the project. That said, here we're talking about serde-derive specifically, so it's not quite a fork of the serde interface that we're talking about. But, I can understand why people might be upset that such an important a piece of the Rust ecosystem is such a flashpoint.
Kudos for him for reverting the binary requirements but it was still weeks after the initial objections to foisting binaries onto users unannounced, with not much productive discussion in between that and that pre-RFC. It's not clear to me that anything would have happened without the mob attention, which sucks.
But, I think the attention on proc macro performance and overall weaknesses in security with cargo could be net benefits, even if I don't personally care about a 9 second speedup in clean builds. I was also really dreading dealing with serde forks, selfishly.
I don't understand why this is so complex? Just make it a feature you put in Cargo.toml:
serde = { version = "1.0", features = ["serde_derive", "fast_compile"] }
Either opt-in or opt-out is fine. It's surely useful during development but it shouldn't be on for production builds. Any transitivity issues should be considered problems with the dependency.
Sure features are transitive. A crate’s features at build are the superset of what every package defines. If I define “fast_compile” (opt-in) or “no_binaries” (opt-out) it’ll get applied across the board, will it not?
Rubygems have been doing this for about a decade now to provide precompiled binaries for Windows systems in particular (generally while compiling native extensions on Linux at install time, although sometimes shipping precompiled binaries for most every target for speed + decoupling the build environment).
116 comments
[ 3.6 ms ] story [ 221 ms ] threadI'd also be curious if anyone has any good explanations as to why a choice like this would be made?
But this isn't a Rust developer issue so much as a distro and security issue. Most Rust developers don't have to concern themselves with supply chain attacks or security audibility, so to them this probably seems like a weird thing to focus on.
But as someone who does have to worry about such things, trading off the ability to audit your supply chain for a small, one-time compilation boost on your first compile is a _terrible_ trade-off.
This is not noisy for no reason - the expectation is all dependencies are compiled from source on your computer, and here the developer forces the use of a binary program, with no alternative.
Yeah. Bazel is the sanitized version of Google's internal build system for most of their projects (known internally as blaze). Buck is Facebook's equivalent.
That said, Bazel and Buck are good for handling projects "at scale" (e.g. company wide monorepos at mid-to-large companies), or with a lot of different needs (e.g. sources in C++, Rust, Python, and Javascript that need to be combined together).
They're not really an alternative to Cargo, and more the next step up for when Cargo isn't sufficient any more. But because they're dealing with much more complicated problems, they're also more complicated to set up and use, so if you're not running into issues with Cargo, you probably don't need to go in that direction (and possibly never will).
Is it worth it? Maybe. Downloading blobs isn't ideal but could be made easier in the future. Ideally a hack like this wouldn't be needed and we'd have comptime serde.
That would be huge if it were a a runtime cost, but this is a one time cost at compile time, and this build will be cached unless you're upgrading serde (or its dependencies). The value of that 10 seconds compared to the cost is a faustian bargain.
So I think the answer is that it’s mostly just noise, but it’s noise you probably don’t want to have to deal with unless you’re invested in Rust itself.
It has to do with how Rust builds are structured.
The compilation unit is the crate, and the result of compiling it depends on the source code, the version of hte compiler, the compiler options, the target triple, and the set of features (think #ifdef FEATURE_FLAG).
Since there's exponential blowup of possible combinations of a single crate, the largest package registry (crates.io) does not cache compiled crates. It only caches their metadata, and when Cargo (the package manager) does dependency resolution, it uses that metadata alone - and your build system (also Cargo, most of the time) will need to actually build all those artifacts.
Seems pretty reasonable, right?
There's a special kind of crate called a proc_macro crate (procedural macros). When you use these crates you're not adding their code to your compiled output - you're creating a compiler extension - the compiled crate is loaded by the compiler, and then used to generate code in your crate, which is then compiled by the compiler.
One of the reasons Rust build times (particularly clean builds) is that they first have to compile all these compiler extensions and then use them to generate the code that the compiler compiles. These crates are super useful however, like serde, which generates serializers/deserializers that can be used for a wide variety of encodings (XML, JSON, TOML, various binary encodings, and so on).
It would seem that the same rules apply for normal crates - there's exponential blowup of compiler version, target triple, and features - but that's not really true. proc_macro crates usually don't have features, the compiler ABI is stable enough that they probably won't break across versions, and almost everyone and their mother is compiling on a handful of host systems (x86_64-linux being the most important one).
So while it's true that crates-io probably shouldn't cache every crate for every permutation of compiler/target/feature/etc, a few crates are so pervasive and used in such a sparing manner that it should be possible for the registry to host those
In fact some build systems can do this automatically. But Cargo can't. But it does have a pre-build feature (build.rs) that works like proc macros, and can pull down a pre-compiled proc macro crate, so widely used ones can use this hack to get around limitations of cargo.
My personal take is that the reason that crates-io and Cargo haven't been upgraded to do this, despite years of people complaining, and known solutions existing, is mostly personal/political. There aren't many people willing to put in the work to do it, and the teams haven't been open to contribution of this kind for a long time. For example, they do not believe that Cargo is a build system, and a lot of this is "out of scope" for cargo.
Also, for the “why” - is it just the very slow build times for Rust?
The inability to opt out of running a non-reproducible binary library just to build code. This isn't just devs complaining, it's against several large orgs security policies.
> Also, for the “why” - is it just the very slow build times for Rust?
It's an initial build time thing. It makes no difference except after a full clean or fresh build.
I’m still learning the Rust environment but usually vendoring is what you do when you want full control.
serde-derive is a macro that is used in the compilation of projects. The way the developers have implemented it, when you build a project _from source_, it pulls and executes a binary blob against your code.
The binary in serde-derive is not reproducible (at least, there is no published hash and other users haven't been able to compile the same binary on their computers), and saves a very minor amount of time - serde-derive has to be compiled once per project per compiler version, unless you delete your dependencies.
1. Users may not have the correct build toolchain installed. (Mostly relevant for end-user applications, or native FFI modules.)
2. Building every package from source is slow.
Neither justification makes sense for serde-derive. For 1, serde-derive is just regular Rust code, built by their regular Rust toolchain. You already need to have the toolchain installed to use it! Hell, you need the same toolchain to build the wrapper library for the binary! For 2, serde-derive has a tiny dependency tree, and all of those are likely to be shared by any other proc macros the user depends on anyway, so the only thing you're skipping is... compiling serde-derive itself.
> Valentini further inquired to the project maintainers, how were these new binaries "actually produced," and if it would be possible for him to recreate the binaries, as opposed to consuming precompiled versions. David Tolnay, who is the primary Serde maintainer, responded with potential workarounds at the time.
The article links to the response [1] as well, which is very good. However, the response doesn't provide workarounds. It provides clear instructions:
>> how is the x86_64-unknown-linux-gnu binary actually produced? Would it be possible for us to re-create the binary ourselves so we can actually ship it?
> By https://github.com/serde-rs/serde/blob/v1.0.177/precompiled/.... Yes.
When the article states that the library author only provided "potential workarounds", it sounded really shady to me, and I believe this is mischaracterizing the real response.
[1] https://github.com/serde-rs/serde/issues/2538#issuecomment-1...
Do they provide binaries for other architectures? Aren't new macbooks and some cheap VPSs using ARM now?
This all sounds like a huge mistake, see python/node deps with C bindings.
146 kB: https://crates.io/crates/serde_json
43 kB: https://crates.io/crates/sj
29 kB: https://crates.io/crates/humphrey_json
7 kB: https://crates.io/crates/rust_json
do you know that, or are you just assuming that?
In my opinion it's better to have simple interfaces for complex features, rather than simple interfaces for simple features. You could code those yourself.
Most crates do not raise their major version when increasing the minimum supported rust version. So there is precedence for treating which changes can break for some people, but which don't have a breaking API changes, as non-breaking for semver purposes.
build.rs absolutely is a glaring security hole in the sense you say, but compared to that, this is much worse. You can verify the build.rs code that you download (at least in theory, and some people in banks or distro packages probably actually do), but binaries are orders of magnitude more difficult to inspect, and with the current Rust build system pretty much irreproducible.
In theory you can compile your own blob, but you'll need musl and whatnot to make a universal Linux build. Code for making the blob is there in the repo.
build.rs is at best equal. It can access your locally available DB, and transmit your data.
We also don't know how it's build. Ideally there is a Docker container out there that does just an import of source code and then builds. No apk install or apt install (you'd do that in a base published layer). Referenced with an SHA256.
We then use this Docker container to pull in the source code AND its dependencies based on a Cargo.lock. Which... isn't there. So we don't know the exact dependencies that went in.
(Even if there were a Cargo.lock, we need to make sure we actually respect it. I believe cargo install by default ignores the lock file and tries to get the latest version that matches).
EDIT: I was wrong, this is not actually `watt` -- it may have been re-using code from the project.
This one of those pile-ons where everyone gets excited about having a cause-du-jour to feel passionate about, while simultaneously ignoring issues that are far more pressing.
The biggest orgs tend to run all of their builds sandboxed, including what happens with build.rs. It's part of how they enforce dependency management day to day, but also helps protect against supply chain attacks.
So not everyone does, but enough people do that you can rely on their complaints for the more well trodden parts of the ecosystem.
Sandboxing doesn't completely prevent supply-chain attacks. You can avoid getting persistent malware on your CI machines, sure. Exfiltration of tokens and secrets during build? Maybe in a small handful of CI setups where admins have carefully split the source fetch, or limited CI network access to Nexus. Exfiltration of tokens and secrets and/or backdooring after the software has been deployed to production? No, build-time sandboxing doesn't help here at all.
On all developer machines as well? No. Very few big orgs do this and only for mission-critical stuff. Some very important ones have docker-based sandboxed workflows or SSH-to-sandboxed-cluster workflows, or air-gapped laptops but that's very, very rare (I worked in air-gapped environment for a bit and it was a massive pain).
Correct, it's more a defense in depth technique, not a complete defense.
> On all developer machines as well? No. Very few big orgs do this and only for mission-critical stuff.
All builds at Google for instance use the model I laid out including 'developer builds'.
That would be extremely useful as the analyzer is a pretty juicy target and also runs proc-macros/build.rs scripts.
[1] https://github.com/bazelbuild/rules_rust/pull/384
[2] https://bazelbuild.github.io/rules_rust/rust_analyzer.html
https://www.grepular.com/Sandbox_Rust_Development_with_Rust_...
Well, that's a bit out of date now as I use podman, to get around the sudo issues. But that's the basic idea.
For lower environments we generally used 3 month or more embargoes on precompiled stuff we couldn’t easily compile ourselves to mitigate some of the supply chain issues if we weren’t directly managing the chain of trust for the binary.
Also it's worth noting that even if it was a sandboxed binary ala https://github.com/dtolnay/watt , it's not obvious that distributions or users would be satisfied with that. For example Zig had this discussion with the own WASM blob compiler that they use as part of bootstrapping. https://news.ycombinator.com/item?id=33915321 . As I suggested there, distributions might be okay with building their own golden blobs that they maintain themselves instead of using upstream's, and that could even work in this Rust case for distributions that only care about a single copy of serde for compiling everything. But it's hard for the average user doing `cargo build` for their own projects with cargo registry in `~/.cargo` to do the same replacement.
I think having an exit path for those that want to compile from source is important, and I can’t understand the reluctance to provide that.
There was also a project (that dtolnay was involved in, I believe!) a few years ago to compile proc macros to wasm.
https://github.com/dtolnay/watt
This would be a lot more compelling if it were integrated into rustc and cargo.
A single person does not audit all of their dependency tree, but many people do read the source code of some if not many of their dependencies, and as a community we can figure out when something is fishy, like in this case.
But when there are binaries involved, nobody can do anything.
This isn't the same as installing a signed binary from a linux package manager that has a checksum and a verified build system. It's a random binary blob someone made in a way that nobody else can check, and it's just "trust me bro there's nothing bad in it".
The developer should be very concerned about what happens if his system(s) are compromised and the attacker slips a backdoor into these binaries-- it will be difficult to impossible to convince people that the developer himself didn't do it intentionally. Their opacity and immediacy make them much more interesting targets for attack than the source itself (and its associated build scripts).
Saving a few seconds on the first compile on the some other developers computer hardly seems worth that risk.
And at the meta level, we should probably worry about the security practices of someone who isn't worrying about that risk-- what else aren't they worrying about?
I found the discussion to be mostly very civilized and focused on finding solutions for those affected.
https://github.com/serde-rs/serde/issues/2538
It's all the right of the author. And he's done so before.
Unfortunately, I'm afraid this could lead to a more fragmented ecosystem which is a pity.
It takes a long time to build trust but it can be destroyed quickly. I'm not sure if the author cares about that though. From my reading of the comments he doesn't.
"If there is implementation work needed in some build tools to accommodate it, someone should feel free to do that work (as I have done for Buck and Bazel, which are tools I use and contribute significantly to) or publish your own fork of the source code under a different name."
https://github.com/serde-rs/serde/issues/2538#issuecomment-1...
This is so sad.
What do you mean by "he's done so before"?
I'm not complaining. It's his project, it's his right. He will be overwhelmed by notifications and this is probably the only way he can stay on top of things.
I don't have to like it either though.
This discussion was cut short by just locking the issues.
The second time it happened to me I stopped our sponsorhip out of frustration. That also sucks and is a bit petty. I acknowledge that.
I really disagree with this characterization of the discussion. While there's plenty of more or less dispassionate comments focused on finding a solution, there's a greater number of people from the peanut gallery drowning that out with comments that amount to "I don't like this!" at best and questioning the integrity of dtonlay at worst. I feel like this demonstrates some of the worst features of open source software on github, where a bunch of nobodies feel the need to add their totally useless input rather than participating in good faith to find solutions and make the software better. You even see comments like that here, speculating that dtonlay is trying to break cargo and force people to use bazel, which is just an insanely conspiratorial and bad faith speculative interpretation that is obviously untrue and helps no one.
Only see comments doing that are on here.
The comments on the github issue are all pretty productive and constructive. The worst is the commentator that calls the approach 'despicable', but also provides constructive feedback and helps describe the problem.
Reddit comments went into tinfoil hat territory.
Discord comments went into calling for dtolnay's head.
Also I'm not lying about (albeit hyperbolic) calls for dtolnay's head. From one of Rust Discords.
https://imgur.com/a/ei6wzxW
don't move goalposts
But ok, let's say limiting yourself only to GitHub comments, you can still find doubt casting and uncharitable interpretations:
> Nice micro-optimization. And if you bury your head deep enough into the sand, doesn't sacrifice anything important either.
> "The only supported way ..." is a polite way of saying "Works for me, WONTFIX".
> Another interpretation of an earlier comment is that he is all-in on bazel/buck and does not care about cargo. Both interpretations have me concerned.
> And ~80 comments with "This doesn't work for me".
Even in this case, the __jem comments make sense. Many people are just writing their version, "I don't like this, " meaning any interesting message is drowned out by noise. THAT will make any maintainer limit to contributors.
I already referenced the worst comment and it falls closer to brutally honest than actually rude.
It's customary to assume other side is arguing in good faith and not making stuff up. I assumed you read all the comments, had access to Github, so you wouldn't have problems finding them. But apparently that wasn't the case.
- https://github.com/serde-rs/serde/issues/2538#issuecomment-1... https://archive.is/bHQS6#issuecomment-1684896969
- https://github.com/serde-rs/serde/issues/2538#issuecomment-1... https://archive.is/bHQS6#issuecomment-1682639314
- https://github.com/serde-rs/serde/issues/2538#issuecomment-1... https://archive.is/bHQS6#issuecomment-1684609445
> I already referenced the worst comment and it falls closer to brutally honest than actually rude.
It's hard to take this at face value when you mistook my quotes for outright fabrication. You either didn't read them all, or you convienently forgot the examples I cited.
Also nice micro-optimization comments is dishonest, apparently dtolnay felt the optimizations provided for his use case were strong enough to warrant a fix.
I don't mind it for when I work in Node - honestly I prefer it. But in Rust? The entire toolchain is designed to compile from source. It may be a bit slow for certain crates still, but that's much weaker justification to make pre-compiled the default.
Crates should compile from source by default as long as source actually exists (ex. has not been lost somehow, or wide distribution of the crate is unlikely). I don't buy any of the malicious theories since the author is well appreciated and respected. It's likely just a misguided choice in this case.
People have rightly mentioned build.rs flexibility - I think there is nuance here and maybe it's good to discuss what default behavior should be expected by users, since sometimes build.rs does wild and useful things.
"No one" is validating all their transitive dependencies for weird binaries, except a few people, and that's enough to get us all here talking about it. In the internet age it only takes one. But if you make it require more work to inspect a blob, you reduce the chances that you'll get even that one person who sounds the alarm. At minimum malicious code will survive longer that way.
Also, the “security” guy claiming no one is going to read it, do you read linux kernel every update? No, not you, but other developers do, or whenever someone is interested/auditing they can, that kind of arguments are stupid.
I don’t see a problem with the community forking as motivation to get the maintainer to come back the conversation or to provide a path for the community to resolve the issue for themselves. A fork was literally one of dtolnay’s own suggestions, bluff or not.
Me too. As a Rust user and small-scale evangelist, I find this whole thing a bit interesting and also somewhat sad. At least, it seems like increased awareness, and perhaps sandboxing, of proc macros are good outcomes that may result.
> how can you work something out when the maintainer can and has already just locked the issue and walked away?
That's not what happened here. dtolnay has not walked away, and the discussion is progressing. For instance, this pre-RFC [0].
> I don’t see a problem with the community forking
Serde, to the limited extent that I understand it, works because it is a single shared interface - multiple forks would be counter to the purpose of the project. That said, here we're talking about serde-derive specifically, so it's not quite a fork of the serde interface that we're talking about. But, I can understand why people might be upset that such an important a piece of the Rust ecosystem is such a flashpoint.
[0] https://internals.rust-lang.org/t/pre-rfc-sandboxed-determin...
But, I think the attention on proc macro performance and overall weaknesses in security with cargo could be net benefits, even if I don't personally care about a 9 second speedup in clean builds. I was also really dreading dealing with serde forks, selfishly.
serde = { version = "1.0", features = ["serde_derive", "fast_compile"] }
Either opt-in or opt-out is fine. It's surely useful during development but it shouldn't be on for production builds. Any transitivity issues should be considered problems with the dependency.
There's also the problem that the build may fail because it will still download the binary, which you may want to forbid entirely.
https://doc.rust-lang.org/cargo/reference/resolver.html#feat...
That said I'm not sure how that applies transitively, I could have sworn I've seen conflicting features not cause transitive build failures.