Can I ask, is this a real, actual concern? Why do I need to sign and verify my software is my software? Why is a hash not sufficient integrity verification?
I have never heard of a good argument for this besides the Apple-esque control of remotely disabling the ability for software to run based on certificate authority, which is not a feature I'm interested in.
Further, I'd like to not see this as possible, since year after year more and more software companies seem to think they're entitled to more and more.
This isn't about users' entitlement to run software, but about publishers and users' ability to confirm that everyone has the same version of the software -- to mitigate supply-chain attacks. This is a real, actual concern because governments and probably organized crime are interested in attacking supply chains in order to attack end users, for example by hacking software developers or publishers in order to substitute a slightly modified version of code.
It could be useful for software distributors to have their version history memorialized somewhere outside of their own control, for example to reduce the attractiveness of governments trying to compel them to secretly tamper with some users' versions of a program, or to make it meaningful and more straightforward to check later on whether they published what they thought they published.
As some other replies pointed out, if that list is distributed in the same place as the software itself, an attacker can modify both of them at the same time.
Maybe the list is digitally signed by the publisher, but then often the signing key is also distributed in the same place as the software itself (and also often used on the same infrastructure that the software was compiled on).
Also, if the list of hash values is distributed only by a software publisher, the software publisher will get the ability to secretly backdoor some users, but not others, by creating two or more different versions of that list. Then it can deny the existence of the backdoored version to the public (or to itself, if the backdooring was done by an unauthorized insider!).
It's also a significant step in a direction to where every dependency that these software publishers is also signed, increasing the overall confidence that there's no malicious code in the software they publish.
How do you know which hash is the correct one? you can
a) check a signature on the hash (current method)
b) check with a central authority if the hash is ok (now that's central control)
c) somehow discover the hash from some other source - possible, but not something users will generally manually verify, and the question of how to trust that hash again quickly comes down to signatures or authority.
At least as much as a signature check, and if you send a hash to somewhere to look it up it reveals more about your system than the signature verification does. If you fetch a list of trusted hashes, the source can customize it more precisely to what to tell you is ok than if you verify a signature.
How is telling you "don't trust this hash" any less "Apple-esque control of remotely disabling the ability for softwareto run" than something saying "don't trust this certificate after $date"? (E.g. the recent-ish Apple outrage was that they switched to uploading hashes over just verifying signatures)
Figuring out the correct hash is different from knowing that the software you got is from who you expected it to be from. If you sign 100 versions of something and you verify what you got... you'll know they came from who you expected (rather than a bad actor) but you might not know which version to run from that.
This is where the transparency log comes in. The hash / signature and public key (by way of a signed x509 certificate) are hashed into an tamper resistant immutable merkle tree. This makes it hard to tamper with the hash. However a bad hash could still be put into the tree, but this is sort of a feature not a bug aspect of a transparency log, anyone can audit the log and see those bad entries. You as an individual are not susceptible to a targeted attack, you see what everyone else sees.
This is an idiom borrowed from certificate transparency. You kind of want the badly signed certs to be recorded, as they can be monitored and audited for. Everything is out in the open in the plain light of day.
Yes it is a real concern. If you're running a Linux distribution, chances are you are downloading your packages from a mirror and not the primary mirror for your distribution. This is done to everyone's benefit bandwidth-wise.
However it opens up the possibility that the artefacts can be tampered with on the server. Signing confirms their authenticity. In a (cryptographically secure) hash, there is no 'key' and so anyone can create a valid one for their modified bundle.
Ditto for containers, Android Apps, and just about anything else we use.
> Why is a hash not sufficient integrity verification?
not necessarily endorsing the need for this, but one reason is that if the hash and file are hosted by the same site an attacker who compromised the web server can trivially change the hash to match their payload. With this system, they would have to compromise both the web server and the signing server.
You can also leverage this to trust a binary even from an unknown source because the signature will match and confirm authenticity regardless of how trustworthy the source of the binary is.
That's just an incorrect usage of an integrity hash. If you obtain a resource from another party other than the original vendor, you need to verify integrity.
If the vendor itself is compromised, then I can understand such a system.
Fifteen years ago the principle security worry in running a web application was that some "script kiddy" would break in and deface your homepage.
Today the threats are much more real. Ransomware, cryptocurrency miners, even state actors.
An enormous point of weakness in modern software is the supply chain - many projects now have thousands of nested dependencies.
Most of those dependencies represents at least one human being who can be threatened with a crowbar and forced to ship an exploit, which can then infect vast numbers of production applications.
Ultimately the system will need to support signatures which represent not just "I made this" but "I reviewed this", and people will need to set policies for whose reviews they trust, and how many reviews they require for each component.
If reviewers can build up a reputation anonymously, that will make it harder to find the human who needs to be crowbarred, but I'm not sure how you prove you are a good reviewer in a way which isn't gameable.
Alternatively, the reviewers could be well known teams in multiple jurisdictions, such that an attacker would need to buy multiple crowbars and multiple plane tickets.
Those are interesting points / possible approaches, however is there any indication that this particular project enables any of that?
This seems focused on signing binaries / build artifacts.
IMHO it seems like if you have the threat model of "crowbared maintainer forced to insert backdoor" you probably don't trust sources let alone binaries and need to vet your dependency sources and then compile your own binaries from them.
Many open source dependencies will not have a jurisdictionally diverse review team, or any review team at all (single maintainer).
With reproducible builds, the difference between signing a binary and signing the source code from which it is built should be meaningless.
I agree that the threat model should include the threat of untrustworthy source code, because we want the countermeasures to work equally well against backdoors, "bugdoors", and genuine bugs.
I suspect for a lot of projects reproducible builds are themselves a bit of a hurdle and not being verified in the rarer case that they already exist, but the point of reproducible + signed builds as indirect source-signing stands.
You have to be in person for that attack, which is a much higher cost than taking over someone's account remotely from a different country. It's also a much higher risk of getting caught and going to jail.
Ok but the premise of physical harm in person comes from the parent comment, not mine:
"Most of those dependencies represents at least one human being who can be threatened with a crowbar and forced to ship an exploit, which can then infect vast numbers of production applications."
This is where the transparency log comes in, certificate signing is openly auditable. It's the same as certificate transparency, malicious or mistakenly administered certificates are openly auditable (instead of being a transaction that occurs behind the doors of a commercial CA).
Also, in future implementations it would be possible for security conscious publishers to attach cryptographic attestations produced by trusted third party hardware manufacturers to their signature demonstrating the private key was generated on the hardware device and therefore could not be under the control of the service operators. Security conscious clients could start to require this for dependencies they pull in.
> Can I ask, is this a real, actual concern? Why do I need to sign and verify my software is my software? Why is a hash not sufficient integrity verification?
Security is layered so there is no one thing to just secure it. With that in mind, how do you control which revision of something is deployed? Do you use the hash? What if someone pushed a different version to your container repo? Could that be somehow run somewhere? It may require someone gaining privileges but that is not an uncommon situation. This is why multi-layered security is useful.
If something is signed and verified you can know WHO it came from. Someone with access to sign it. In the situation where a bad actor pushed something to a container repo the verification step would fail and you would catch it.
I hope this feature will be completely optional. Code signing in every major OS (Windows, macOS, Android, iOS) is pure pain, you cannot distribute your own apps properly without obtaining a signature for 100$ (usually per OS, sometimes per year).
sigstore will be a non profit / free to use service. Think Let's Encrypt for software signing.
My hope is that we shift the paradigm so that consuming untrusted software via packages / dependencies etc becomes as unappealing as serving a website over just plain ole HTTP has now become.
In order to make that shift, open source communities require a free and easy to use service and this is what we hope sigstore will become, which is why it's a Linux Foundation project with all code being developed and maintained by a community.
What is your rationale behind this (if I understood it correctly) being a Service rather than a way of doing things ?
It's becoming increasingly difficult for me to run my own code, that I am perfectly happy to sign myself, on my own devices. That might seem like a fringe case. But it shows a deeper problem:
I do not trust you. I don't even know you in the first place.
Yet you (well, as part of a goup) are asking me to give you more power over my devices.
What is this push towards centralizing trust ?
Example: Firefox extensions need to be signed now. I can't send my friends the extensions I wrote myself. There is just no way. I can't go over to their house, sit next to them, and install my self signed-root certificate, and have their version of firefox trust it.
It must be signed by Mozilla: An organization that most people will never ever in their lives's interact with. That makes no sense.
And not even to speak of trying to install private Root CAs into iPhones or Android devices.
How does your solution empower users to own their own devices, and not have them owned by someone they are separated from by several degrees and whom they have never met ?
Did you follow the link to the project list on Github? The actual tool for doing the signing, cosign, is just a binary you can install on your device and generate signatures and keys yourself. The "service" part of it seems to just be having your public certificate vouched for by a trusted code signing CA. I don't see anything in the tooling that requires your users to only trust that CA. If you want to sign your cert with your own CA and tell your users to trust that instead, they seemingly can do that, just as you can do that today in browsers. That you can't do it with Firefox extensions and mobile app stores is a limitation intentionally built into the distribution channel. It's not a limitation of PKI itself. iOS, Android, and Mozilla could have chosen to let users install arbitrary trusted CAs. You shouldn't dismiss all PKI based on the fact that a few vendors have chosen to implement it in a crappy way to make walled gardens.
It doesn't say this on the announcement, but looking at the actual PKI service (https://github.com/sigstore/fulcio), it seems to be entirely possible to self-host the service and roll your own CA.
As a hardening measure for production container or machine images, it would be good if code interpreters, including things like Python and Node.js but also shells, could be restricted to only accept source code that comes from a signed and verifiable bundle. That would mean no interactive mode, no running code supplied on the command line (meaning no shell injection vulns), and no support for eval or equivalent. But we wouldn't have to go to the full trouble of using a distroless, shell-less image. Does anyone know of active work in this area?
I'd even be happy with no SemVer and only commit hashes for version control. This way, you know the version you just tested will always be the same, and nobody could modify it from under you whilst keeping the same SemVer
Good question, cosign is a client that works with containers OCI / registries. We are also develop clients to work with pypi, rust cargo and cases such as helping to protect against curl | bash attacks
This is pretty cool and i think one of the good application areas of distributed ledger technology.
Signing is still a hard problem, even for established projects like Rust. Right now, rustup does not verify signatures in any way or form. The security is solely thanks to https and the S3 bucket not being compromised.
> It's for open source maintainers, by open source maintainers.
> Google
> Red Hat (IBM)
Marketing in this manner is deceptive. Saying it’s “by and for” open source maintainers gives the impression this is a grass roots effort, when in reality this is a corporate initiative.
There are still independent developers who contribute to passion projects in their spare time, of course, but if you've been using open source software because you believe corporations aren't substantial contributors I think you should update your priors.
I don't think saying something is built "by and for open source maintainers" is implying it's a grassroots project. All it means is that it's designed with the intent of supporting the integration of patches from members of the public, which has very little to do with it being run by a corporation or unpaid individuals.
> which has very little to do with it being run by a corporation
On what basis can you say that? It’s clear that a corporation that accepts legal liability for the software they run in production or ship would have a strong incentive to create a process around determining the origin of said software.
I feel like everyone who runs software has a strong incentive to determine the origin of it? Do you not feel that way?
It's true that different open source projects care about different things (and some may genuinely not care about who contributes), but there are plenty of prominent open source software projects (the linux kernel, various programming languages, apache, etc) who deal with people who would like to submit vulnerabilities or simply cannot code well enough to avoid problems. It seems like all projects would like to avoid those outcomes, even if their legal situation is different, and providing a verifiable certificate chain could be a method of achieving that (either in contributions or in distribution).
I guess I was thinking incentive beyond the legal - like, people often do things because they have desires outside of the legal code.
For example - even though the GPL contains, to the extent legally possible, an "as-is" clause like the MIT liscence, it's extremely common for GPL projects to publish information (ex: hashes, known good package repos, etc) that help people ensure they've gotten the software they expect. Sigstore seems like it's driving towards the same goal.
If they are doing this as a side project I’d be more hesitant to adopt it.
The fact that it’s backed by an entity that is more eternal is actually a plus. Any concerns about bait and switch profit are addressed in the OSS license they pick.
I haven’t looked at the license but I’d hope it’s permissive (Apache, MIT, etc) otherwise, wouldn’t touch it.
I seem to recall something on HN a few months back where google wanted to require full names for FOSS code, but can't find it now. (If anyone remembers that I'd appreciate a link.) This smells like it's trying to do the same thing and let google try to attain the control over the foss ecosystem that it has over the web.
One of the founders here, this is completely untrue. It's a project funded under the Linux Foundation and the service will be completely run by the Linux Foundation, the exact same mode as used for Let's Encrypt.
Thanks for the reply. I'm (hopefully understandably) pretty skeptical of anything google puts their name on nowadays.
That said, my speculations were about Google's intentions (not sigstore's) and if you think they are "completely untrue" then you are being misled and should potentially reevaluate the entire project.
Banning anonymous and pseudonymous contributors and requiring real names be provided to a trusted party is an explicitly stated goal of Google's[1] and one of the reasons they contribute to the sigstore project in the first place[2]. If I was going to try to make that happen, sigstore is a great way to lay down the infrastructure to implement something like that at a later point.
Additionally, anything that will centralize FOSS development (as being a root signing key for software would do) is probably not in it's best interests long term, because at that point the difference between a good and bad actor is a matter of policy and not a technical question.
I believe that you have the best of intentions, but others involved in the project clearly don't.
Perhaps this this will put you at rest a little. The project was founded in Red Hat along Purdue University (Santiago, part of Arch Linux security team) and Google joined via GOST (The google open source security team). GOST are funded to help improve Open Source Security, they are not there to create products (they are the team that are funding the rust module work in the linux kernel). They were happy to join sigstore as Red Hat were already there first as stewards of it being open source / community centric. I also know Dan Lorenc who heads up the team and he is a great guy and very knowledgable about FOSS and Open Source.
If there were any evil intent, I am telling you know, I would be kicking up a storm.
Sorry about the "completely untrue" statement, that was not helpful. It's just we face so much unneeded FUD all the time.
Thanks for pointing this out - seems like another step by corporates towards stepping up control over open source software. I'd rather have someone like the Free Software Foundation and EFF to be in charge of these kinds of things.
It's under the Linux Foundation, a non profit org and you also missed the third founder Purdue University for whatever reason. We also have community members from NYU, Arch Linux, Debian an Alpine Linux (plus folks from the rust community, python etc).
Red Hat and Google helps to show that money will be behind running the service (funded through the Linux Foundation)
This is the same set up with Let's Encrypt . Linux Foundation > ISRG > Let's Encrypt with money funded by corporate sponsors.
Money needs to come from somewhere when you run a critical service.
because your boxes and web of charts do little to give this project credibility. Maybe if you're around in 10 years and when it comes to security, that burden of proof is on you.
To me the most fascinating thing here apart from the idea of a software signing standard, was that I’ve never witnessed a remote key signing ceremony before.
I’m used to the root ones where they take laptops out of safes, etc.
81 comments
[ 4.0 ms ] story [ 132 ms ] threadI have never heard of a good argument for this besides the Apple-esque control of remotely disabling the ability for software to run based on certificate authority, which is not a feature I'm interested in.
Further, I'd like to not see this as possible, since year after year more and more software companies seem to think they're entitled to more and more.
It could be useful for software distributors to have their version history memorialized somewhere outside of their own control, for example to reduce the attractiveness of governments trying to compel them to secretly tamper with some users' versions of a program, or to make it meaningful and more straightforward to check later on whether they published what they thought they published.
Edit: I misread the latter portion of your post. I see now. Thanks.
Maybe the list is digitally signed by the publisher, but then often the signing key is also distributed in the same place as the software itself (and also often used on the same infrastructure that the software was compiled on).
Also, if the list of hash values is distributed only by a software publisher, the software publisher will get the ability to secretly backdoor some users, but not others, by creating two or more different versions of that list. Then it can deny the existence of the backdoored version to the public (or to itself, if the backdooring was done by an unauthorized insider!).
a) check a signature on the hash (current method)
b) check with a central authority if the hash is ok (now that's central control)
c) somehow discover the hash from some other source - possible, but not something users will generally manually verify, and the question of how to trust that hash again quickly comes down to signatures or authority.
How is telling you "don't trust this hash" any less "Apple-esque control of remotely disabling the ability for softwareto run" than something saying "don't trust this certificate after $date"? (E.g. the recent-ish Apple outrage was that they switched to uploading hashes over just verifying signatures)
Figuring out the correct hash is different from knowing that the software you got is from who you expected it to be from. If you sign 100 versions of something and you verify what you got... you'll know they came from who you expected (rather than a bad actor) but you might not know which version to run from that.
There's a real vulnerability where users get a compromised package and a matching hash from the same compromised repository.
This is an idiom borrowed from certificate transparency. You kind of want the badly signed certs to be recorded, as they can be monitored and audited for. Everything is out in the open in the plain light of day.
However it opens up the possibility that the artefacts can be tampered with on the server. Signing confirms their authenticity. In a (cryptographically secure) hash, there is no 'key' and so anyone can create a valid one for their modified bundle.
Ditto for containers, Android Apps, and just about anything else we use.
not necessarily endorsing the need for this, but one reason is that if the hash and file are hosted by the same site an attacker who compromised the web server can trivially change the hash to match their payload. With this system, they would have to compromise both the web server and the signing server.
You can also leverage this to trust a binary even from an unknown source because the signature will match and confirm authenticity regardless of how trustworthy the source of the binary is.
If the vendor itself is compromised, then I can understand such a system.
Today the threats are much more real. Ransomware, cryptocurrency miners, even state actors.
An enormous point of weakness in modern software is the supply chain - many projects now have thousands of nested dependencies.
Most of those dependencies represents at least one human being who can be threatened with a crowbar and forced to ship an exploit, which can then infect vast numbers of production applications.
So yes, for me this is a very real concern!
If reviewers can build up a reputation anonymously, that will make it harder to find the human who needs to be crowbarred, but I'm not sure how you prove you are a good reviewer in a way which isn't gameable.
Alternatively, the reviewers could be well known teams in multiple jurisdictions, such that an attacker would need to buy multiple crowbars and multiple plane tickets.
This seems focused on signing binaries / build artifacts.
IMHO it seems like if you have the threat model of "crowbared maintainer forced to insert backdoor" you probably don't trust sources let alone binaries and need to vet your dependency sources and then compile your own binaries from them.
Many open source dependencies will not have a jurisdictionally diverse review team, or any review team at all (single maintainer).
I agree that the threat model should include the threat of untrustworthy source code, because we want the countermeasures to work equally well against backdoors, "bugdoors", and genuine bugs.
I suspect for a lot of projects reproducible builds are themselves a bit of a hurdle and not being verified in the rarer case that they already exist, but the point of reproducible + signed builds as indirect source-signing stands.
[1] https://github.com/crev-dev/cargo-crev
"Most of those dependencies represents at least one human being who can be threatened with a crowbar and forced to ship an exploit, which can then infect vast numbers of production applications."
Security is layered so there is no one thing to just secure it. With that in mind, how do you control which revision of something is deployed? Do you use the hash? What if someone pushed a different version to your container repo? Could that be somehow run somewhere? It may require someone gaining privileges but that is not an uncommon situation. This is why multi-layered security is useful.
If something is signed and verified you can know WHO it came from. Someone with access to sign it. In the situation where a bad actor pushed something to a container repo the verification step would fail and you would catch it.
This is just one example. There are many others.
sigstore will be a non profit / free to use service. Think Let's Encrypt for software signing.
My hope is that we shift the paradigm so that consuming untrusted software via packages / dependencies etc becomes as unappealing as serving a website over just plain ole HTTP has now become.
In order to make that shift, open source communities require a free and easy to use service and this is what we hope sigstore will become, which is why it's a Linux Foundation project with all code being developed and maintained by a community.
It's becoming increasingly difficult for me to run my own code, that I am perfectly happy to sign myself, on my own devices. That might seem like a fringe case. But it shows a deeper problem:
I do not trust you. I don't even know you in the first place.
Yet you (well, as part of a goup) are asking me to give you more power over my devices.
What is this push towards centralizing trust ?
Example: Firefox extensions need to be signed now. I can't send my friends the extensions I wrote myself. There is just no way. I can't go over to their house, sit next to them, and install my self signed-root certificate, and have their version of firefox trust it.
It must be signed by Mozilla: An organization that most people will never ever in their lives's interact with. That makes no sense.
And not even to speak of trying to install private Root CAs into iPhones or Android devices.
How does your solution empower users to own their own devices, and not have them owned by someone they are separated from by several degrees and whom they have never met ?
https://certificate.transparency.dev/
and less like AuthentiCode or app store code signing.
It doesn't say this on the announcement, but looking at the actual PKI service (https://github.com/sigstore/fulcio), it seems to be entirely possible to self-host the service and roll your own CA.
And further to this, anyone could stand up their own sigstore service. In fact we expect some of the bigger oss projects to do just that.
You can install your company’s internal CA certs as part of any MDM package.
And products like HashiCorp’s Vault let you manage the whole CA and signing certs etc
I think that compromised dependency chains are a nightmare, and we're just getting started.
I write Apple apps, and dread the whole provisioning thing (which they seem to change, regularly), so this is not heavy-duty at all.
Signed checksums of binaries baked into the package manager
Reproducible builds made dead-simple stupid-easy
Selinux/sandboxing made more transparent and simple enough to use for mere mortals
Something like tripwire, by default (I think netbsd has this built in, but it's not default)
I'd even be happy with no SemVer and only commit hashes for version control. This way, you know the version you just tested will always be the same, and nobody could modify it from under you whilst keeping the same SemVer
I'm genuinely confused. Does this also apply to "user-facing" software such as CLI tools or GUIs?
Signing is still a hard problem, even for established projects like Rust. Right now, rustup does not verify signatures in any way or form. The security is solely thanks to https and the S3 bucket not being compromised.
https://github.com/rust-lang/rustup/issues/2028
https://github.com/rust-lang/rustup/issues/2027
> Google
> Red Hat (IBM)
Marketing in this manner is deceptive. Saying it’s “by and for” open source maintainers gives the impression this is a grass roots effort, when in reality this is a corporate initiative.
On what basis can you say that? It’s clear that a corporation that accepts legal liability for the software they run in production or ship would have a strong incentive to create a process around determining the origin of said software.
It's true that different open source projects care about different things (and some may genuinely not care about who contributes), but there are plenty of prominent open source software projects (the linux kernel, various programming languages, apache, etc) who deal with people who would like to submit vulnerabilities or simply cannot code well enough to avoid problems. It seems like all projects would like to avoid those outcomes, even if their legal situation is different, and providing a verifiable certificate chain could be a method of achieving that (either in contributions or in distribution).
Strong incentive no. In fact, most volunteer open source developers release their software with no legal liability at all (cf. MIT license).
For example - even though the GPL contains, to the extent legally possible, an "as-is" clause like the MIT liscence, it's extremely common for GPL projects to publish information (ex: hashes, known good package repos, etc) that help people ensure they've gotten the software they expect. Sigstore seems like it's driving towards the same goal.
The fact that it’s backed by an entity that is more eternal is actually a plus. Any concerns about bait and switch profit are addressed in the OSS license they pick.
I haven’t looked at the license but I’d hope it’s permissive (Apache, MIT, etc) otherwise, wouldn’t touch it.
That said, my speculations were about Google's intentions (not sigstore's) and if you think they are "completely untrue" then you are being misled and should potentially reevaluate the entire project.
Banning anonymous and pseudonymous contributors and requiring real names be provided to a trusted party is an explicitly stated goal of Google's[1] and one of the reasons they contribute to the sigstore project in the first place[2]. If I was going to try to make that happen, sigstore is a great way to lay down the infrastructure to implement something like that at a later point.
Additionally, anything that will centralize FOSS development (as being a root signing key for software would do) is probably not in it's best interests long term, because at that point the difference between a good and bad actor is a matter of policy and not a technical question.
I believe that you have the best of intentions, but others involved in the project clearly don't.
[1] https://security.googleblog.com/2021/02/know-prevent-fix-fra...
[2] https://news.ycombinator.com/item?id=26603661
If there were any evil intent, I am telling you know, I would be kicking up a storm.
Sorry about the "completely untrue" statement, that was not helpful. It's just we face so much unneeded FUD all the time.
The legal entity here is the Linux Foundation.
As they’ve said, this seems like LetsEncrypt for software signing.
I found out about this project 30 mins ago and honestly it sounds great. it would be awesome for us to roll this out even for our internal libraries.
Red Hat and Google helps to show that money will be behind running the service (funded through the Linux Foundation)
This is the same set up with Let's Encrypt . Linux Foundation > ISRG > Let's Encrypt with money funded by corporate sponsors.
Money needs to come from somewhere when you run a critical service.
I’m used to the root ones where they take laptops out of safes, etc.
But no reason why this can’t happen right?