> It is IMHO the strongest security guarantee you can offer without forcing each module author to manage keys.
Outsourcing the problem to DNS still forces module authors to manage keys. Not keys to sign code, but keys to sign TLS certs nonetheless. Granted, authors can further delegate managing TLS keys to other people, but you cannot escape the ultimate issue of managing control of the domain (passwords at registries at least).
If we cannot get rid of key management anyway, can we split the problem into 1) Code Signing and 2) Code Distribution? Once we get the Code Signing done right, Code Distribution will be much easier with robust solutions (e.g. content addressable storage and various kinds of proxies/caches for high availability).
Now code signing, lots of schools of thoughts… On the extreme end of decentralization, PGP/GPG people will say Web of Trust and Crypto/web3 people will say hardware wallets; while on the other end of centralization we seem to be getting by just fine with publishing public keys on GitHub and various social networks. Just pick a place where you Feel Good Enough™ :p
This is a largely academic discussion - I assume over 95% of modules are hosted on some kind of "forge" (mostly GitHub nowadays), so the users don't have to worry about keys/certificates, just about the forge's continued existence and their account not being removed. Any attempt to switch to code signing would raise the barriers of contributing to the Go ecosystem quite a bit, so (if at all) it should be introduced as an alternative or supplement to the existing system, not a replacement.
Code signing certs expire though. So you basically limit any code's usefulness to some arbitrary date. Then you need to maintain that code cert, continue to pay for it, renew it, etc, which you can realistically only do until you get hit by a bus, die of old age, or just fall on hard times. This is a worse (very real) problem vs. some domain thing that probably won't ever happen for the majority of modules.
Provenance artifacts via something like SigStore would go a long way, since it solves both key management and the linking of artifacts to their build process.
For Go where you're just publishing sources with a build necessarily, you could opt for SigStore git signing so you can at least validate the author via a 2FA auth provider's OIDC token.
I have a lot of bad things to say about Gos module system, but this ain't one of them. With this system you don't need an account somewhere to publish a package and you still get to use GitHub, Gitlab, etc. without your module URL being dependent on them. If you consider alternative systems with accounts, if you loose access to the underlying email, you may be out of luck, too. Or in the case of Rust/crates.io, if you delete/switch your Github account you can't log in anymore either.
What I like about the go.mod file is that it easily allows to remap go modules to local path folders.
This way I can have a backup of the dependencies wherever I want, even when not using "go mod vendor" (which I like also a lot tbh).
So I don't understand the critic in the article, you cannot solve this problem efficiently other than putting everything on a blockchain, which will waste a lot of hard drive space.
I'm curious what bad things you have to say about it. I haven't actually used it but my impression from reading the design docs was that it's the best module design so far. Better than Rust for example.
My main gripe with it is the missing separation between test and prod dependencies. Let's say you work on program A, which depends on library B. Library B uses library C as a test dependency. It shouldn't have any effect on A whatsoever and should not even be downloaded. Yet it is.
Apart from the obvious bandwidth and storage space wasted, this is also a problem when compiling an SBOM (Software Bill of Materials) and when scanning for vulnerable dependencies. There is simply no way to tell if your Go binary is vulnerable simply by looking at the go.mod/go.sum file, you have to build and analyze the binary and hope that everything went well. Same for compiling NOTICE files for license compliance. (wink wink Apache license)
This isn't an unknown or unsolved problem either, Maven solved this ages ago.
There are also some minor nitpicks, such as their absolute requirement to do path-based semantic versioning from v2, but not from v1 or v0. If you change anything, but absolutely anything that's publicly visible, you must increase the minor version of the library you are publishing or you will have a really bad time. If you want to publish both a library and an application, you have to either version them with ridiculous version numbers, or you have to split the two.
What's the problem that your tests depends on lib C, if it's using it it makes sense that the main program depends on it. Go test are part of the main program, if you want them separated then create another module that import a and c for testing.
It's not _my_ tests that depend on C, but B's tests. I don't maintain B, nor do I care for its tests. They shouldn't be in my dependency tree at all.
This is a problem because every time C has a vulnerability, you get a deluge of vulnerability scanner alerts or Dependabot pull requests to update it, even though your application has nothing to do with C.
If anyone has a solution for this, I'd really love to hear it because it consumes a silly amount of time to deal with bogus vulnerability alerts.
Most of the time this will be a Github or Gitlab URL. (Or similar.) Then domain name expiration isn’t a problem, but you do need to protect your account.
Using a personal domain name should probably be discouraged unless you’re sure you know what you’re doing, and even organizations might want to think hard about whether they’re really want to maintain a website indefinitely.
Unless Github or Gitlab goes out of business. Github might not be much of an issue, but what is some shifty company buys Gitlab during the next financial crisis?
Github has Microsoft money, so I wouldn't be to concerned, but even yanking Gitlab could break an insane amount of modules in one go. Having modules distributed among many domains reduce the chances of a large scale issue, for the cost of many small problems.
The problem here is not about modules breaking. Even if gitlab went away nothing would break because of vendoring and proxies. The problem is if gitlab goes away and someone buys it and starts pushing malware through it, there is no way to distinguish a malicious update from a legitimate one.
If github just falls out of existence as a business with no warning I think go module issues will be just one of an impossibly long list of issues we'll all have
As much as GitHub might try to convince people otherwise, Git is a decentralised version control system. Which means if a cloud git service goes offline tomorrow, anyone who already cloned that repo still has a verbatim copy of the code that can seed a new upstream source.
Not saying this wouldn’t be disruptive. But it’s not a project ending scenario.
As some who writes a lot of Go and thus has actually had to deal with this problem already: the hard part is finding a new source for the module. Updating your project to point to the new module takes around two minutes tops. But finding that module is the hard part.
However this isn’t a uniquely Go problem. You have the same problem when switching to a new module in any language and the reasons you’d struggle is the same: because the existing module has been abandoned. If it hadn’t been abandoned then it’s very easy to find the new git repo and thus update your Go project.
> git decentralization has nothing to do with it, unfortunately.
It has everything to do with it. You have a local copy of the repo on your development machine and thus can throw a new mirror up if you need to. All you lose is any issue tracking and existing maintainers. But if you’re having to resort to this, those maintainers have likely already abandoned that module already.
I've never heard of this being a bad pattern, could you elaborate a bit on that and how we are supposed to do it? If you have some dependencies on gitlab.com and some on github.com, how will your go.mod file look like if you are not supposed to use the full path?
My point was that’s not actually the problem in the grand scheme of things. Thats only an issue if the module is abandoned because then you cannot find the new host for it. And if the module is abandoned then you’ve got much bigger issues already regardless of the language and import semantics.
Plus if you really don’t want a full domain in your projects import paths then you can vendor in those modules, albeit at a cost of higher maintenance for yourself.
The biggest problem with having the domain name included in the import path is just purely visual noise. It’s pretty damn ugly. But that problem is purely aesthetic.
I will grant you that when package versions were encoded in the import path, that was extremely problematic (to say the least). But that’s gone away now we have proper module versioning (and I say that even as someone who doesn’t particularly like the solution that Go finished up with).
> Third-party packages are always known by their fully-qualified names, which include the site that hosts the code (e.g. github.com), the user or organization that develops it (e.g. google), and the base name (e.g. uuid).
"google/uuid" tells you nothing about the owner. It might be Google, or it might be me, who squatted the name early on.
"github.com/google/uuid" would be much difficult to squat. Also, it tells you that this comes from the Google user in GitHub, which is something much easier to verify.
"code.google.com/go/UUID" would be even more of a guarantee. But I guess they decided relying on GitHub to host their Go packages was good enough.
One could argue that if the personal website goes away, you should be taking a hard look at that dependency anyway as it may have become unmaintained. Also, you can use vendoring if this is a concern.
No, the article suggested that when hosting a Go package on your private domain that domain might expire. However, most Go packages are published as the module name being github.com/author/packagename, in which case domain expiry is not a problem, but you need to keep a hold of the account.
I was once unable to build some Go software because some random individual's personal git hosting website was down. It's sadly not only gitlab or gitlab; at least that would've limited the amount of single points of failure.
There's a reason tou need namespaces even without name squatting considerations. At least for some package systems like pip. If you are using a private registry you want to prevent dependency confusion attacks.
I guess Go sidesteps this by not having a registry, and Rust seems to require you to explicitly say which registry a package comes from. But pip is vulnerable to this issue because it still doesn't have namespaces.
Forgive me, but I don't get what the issue is. Surely the "ownership" of the domain is irrelevant? The package-name is just a unique name; the only thing it refers to is the package.
In particular, I don't see any reason why the package-name needs to be related to the mailing-list/bugtracker, or to the location of the canonical code repository.
XML uses URLs as labels for namespaces; they are just labels. Nobody expects to find a document about the namespace, nor a schema, at the namespace URL. There's never a need to resolve a namespace URL. Why should it be necessary to resolve a Go package URL?
[Edit] I honestly don't know, because I've never used in Go.
Go uses the package name to retrieve the pagacke source - a package named github.com/example/whatever will be downloaded by basically cloning it with git & then checking out the specified version as a tag.
That's in itself a really simple system and doesn't rely on a single third party, but obviously comes with the ownership issue (unless when pinning a specific commit) - in my opinion a good compromise.
A Uniform Resource Locator (URL, RFC1738) is a name for localization and access of resources via the Internet. A Uniform Resource Name (URN, RFC8141 & RFC3986) is a name that identifies an abstract or physical resource.
Your argument just favors distorting the concept of URL to use URL as URN. But if we were going to use URLs to identify abstract resources, why don't we use URNs instead?
Someone famous once said "Good URLs don't change". Turns out he was pissing in the wind; nobody paid attention. And the market in DNS names guaranteed he'd be wrong.
The linked post (toot?) from Tony Arcieri says this when talking about the matter of avoiding namespace squatting in Cargo:
> (In Rust) there will be namespaces soon anyway, just not the kind that would require a huge paid manual review/support staff to administer everything
Is he talking about Maven Central?
Because using DNS prefixed packages as a means to avoid namesquatting seems to me like the best solution out there. For all its faults, Maven got a lot of package-management things right, and I'd argue DNS prefixes is one of them.
I wish that solution wasn't handwaved so quickly.
You just make an account at Maven Central, prove ownership of your domain name, and can then start publishing packages with your publisher credentials. If you suddenly lose your DNS name (as in the case of .ga), you'll still be able to publish packages for a while, at least until there is a new ownership verification, giving you ample time to publish new versions with deprecation warnings.
But, of course, that requires people on top of the registration system, approving requests, attending issues. Claiming that the need for such staff is something to avoid, sounds to me akin to if we were in the 90s thinking how to distribute software packages in our Linux distro, and someone said the need for package maintainers would be something to avoid.
It's worth noting that the proxy and pinning behavior actually help with scope a fair bit. The threat is still there for new versions, but both visibility and external controls can still help.
I wish GOPROXY was a little more... obvious? The way it's working feels like it's trying to hide something, for whatever reason.
For a long time I did not even realize that all of my Go package installations were going through Google-controlled infrastructure by default; something I'd object to had I known sooner. The cache also tends to serve the last working release, rather than the one that's actually the latest, so you sometimes don't even realize that the tip is broken. I can forgive most things, but not hiding errors, especially as I bash my head against the keyboard. On top of that, proxy.golang.org has historically not been playing nice with upstream repository servers[1]. I tend to keep GOPROXY=direct in my environment for these reasons.
Should IPFS support be built into the Go toolchain? Should the package authors be expected to be running an IPFS node? Should Google operate the infrastructure to access IPFS? This is an interesting proposition but it opens quite a few followup questions.
I know it is not always possible but forking the repos helps mitigate the issue of original repo being "compromised". It also helps reproducible builds...
That is not the point. If I newly checkout my Go project and build it, it will use my forked repos instead of the orginal ones.
If someone use my project, it will also use my forked repos, so it makes the builds more reproducible.
That way, I'm not pushing the dependencies in my project.
Someone help me here, but doesn't every package manager have this problem?
If control of crates.io or npmjs.org change, how many rust/js projects will still build?
The trade-off seems to be between distributed vs centralized. Where in the distributed case, you take on more risk that something will break for a lower chance that everything will break. And in the centralized case you have a lower chance that something will break, but if anything does, you're completely SoL.
Now, I think there's a conversation to be had on that trade-off, but it doesn't feel like the article really covers it. Centralization has some really nice day to day benefits, but it's not exactly a trade-off with zero downside.
Neither does github. Any package at github.com/user/package (which is a lot of them) has this property.
With the bonus that if github does go down, you can redirect it to another mirror of the package if the maintainer puts one up (or you can vendor early, but that's an option for everything).
It's talking about domains going down in general as far as I can tell. People with their own domain are just much more likely for it to happen.
It seems you have two specific options.
Centralize on a big company's url, which you can use in go via sticking to github.com (and a couple other domains) packages as well as js/rust/java/etc.
Decentralize it and have to worry about constant low-level package rot, but not be able to have any individual domain take out all of your dependencies, which you can only really do in go (or languages like C where git submodules are a fairly standard form of dependency management).
There is a tradeoff of having a single point of failure in the first example in exchange for increased reliability during normal operations, and in the second example, you're much more likely to be hit by package rot, but you're significantly less vulnerable to your entire dependency tree being wiped out. This is the tradeoff I'm talking about.
Perhaps users can still install it, but the maintainer has lost control over the package and can't push updates or fixes. Additionally, as the article points out, a new person may now have taken control over the package.
It’s good to think through the risks here for your own use case, but ultimately, as the article points out, there’s no way to entirely avoid the problem for Go or any other language. So you just have to figure out how to best manage your own risks for your own required level of assurance.
Luckily the go ecosystem has a number of tools for dealing with this. You can set up a local trusted proxy of all the code you use internally and point your builds to that instead of Google’s default proxy or just doing direct pulls. Or you can vendor the code you’re using into your own repository. You can also easily alias specific libraries to point elsewhere if specific ones are problematic, without having to change the code directly.
The expiry problem is not much different to if someone deletes the package, so I’m not so concerned about that, but my main pain point is that if you’re developing Go code and your company decides to move Git provider (had this happen multiple times as companies grow, get acquired, etc) then you either have a mess of “replace” all over go.mod, or you have to point everything to a new location. It’s particularly annoying if you want to check out and build an older version of your code, because the changes to go.mod have to be cherry-picked, and by that time you’ve probably updated your dependency’s versions.
60 comments
[ 2.8 ms ] story [ 115 ms ] thread> It is IMHO the strongest security guarantee you can offer without forcing each module author to manage keys.
Outsourcing the problem to DNS still forces module authors to manage keys. Not keys to sign code, but keys to sign TLS certs nonetheless. Granted, authors can further delegate managing TLS keys to other people, but you cannot escape the ultimate issue of managing control of the domain (passwords at registries at least).
If we cannot get rid of key management anyway, can we split the problem into 1) Code Signing and 2) Code Distribution? Once we get the Code Signing done right, Code Distribution will be much easier with robust solutions (e.g. content addressable storage and various kinds of proxies/caches for high availability).
Now code signing, lots of schools of thoughts… On the extreme end of decentralization, PGP/GPG people will say Web of Trust and Crypto/web3 people will say hardware wallets; while on the other end of centralization we seem to be getting by just fine with publishing public keys on GitHub and various social networks. Just pick a place where you Feel Good Enough™ :p
For Go where you're just publishing sources with a build necessarily, you could opt for SigStore git signing so you can at least validate the author via a 2FA auth provider's OIDC token.
This way I can have a backup of the dependencies wherever I want, even when not using "go mod vendor" (which I like also a lot tbh).
So I don't understand the critic in the article, you cannot solve this problem efficiently other than putting everything on a blockchain, which will waste a lot of hard drive space.
Apart from the obvious bandwidth and storage space wasted, this is also a problem when compiling an SBOM (Software Bill of Materials) and when scanning for vulnerable dependencies. There is simply no way to tell if your Go binary is vulnerable simply by looking at the go.mod/go.sum file, you have to build and analyze the binary and hope that everything went well. Same for compiling NOTICE files for license compliance. (wink wink Apache license)
This isn't an unknown or unsolved problem either, Maven solved this ages ago.
There are also some minor nitpicks, such as their absolute requirement to do path-based semantic versioning from v2, but not from v1 or v0. If you change anything, but absolutely anything that's publicly visible, you must increase the minor version of the library you are publishing or you will have a really bad time. If you want to publish both a library and an application, you have to either version them with ridiculous version numbers, or you have to split the two.
This is a problem because every time C has a vulnerability, you get a deluge of vulnerability scanner alerts or Dependabot pull requests to update it, even though your application has nothing to do with C.
If anyone has a solution for this, I'd really love to hear it because it consumes a silly amount of time to deal with bogus vulnerability alerts.
Using a personal domain name should probably be discouraged unless you’re sure you know what you’re doing, and even organizations might want to think hard about whether they’re really want to maintain a website indefinitely.
Unless Github or Gitlab goes out of business. Github might not be much of an issue, but what is some shifty company buys Gitlab during the next financial crisis?
Github has Microsoft money, so I wouldn't be to concerned, but even yanking Gitlab could break an insane amount of modules in one go. Having modules distributed among many domains reduce the chances of a large scale issue, for the cost of many small problems.
Not saying this wouldn’t be disruptive. But it’s not a project ending scenario.
However this isn’t a uniquely Go problem. You have the same problem when switching to a new module in any language and the reasons you’d struggle is the same: because the existing module has been abandoned. If it hadn’t been abandoned then it’s very easy to find the new git repo and thus update your Go project.
> git decentralization has nothing to do with it, unfortunately.
It has everything to do with it. You have a local copy of the repo on your development machine and thus can throw a new mirror up if you need to. All you lose is any issue tracking and existing maintainers. But if you’re having to resort to this, those maintainers have likely already abandoned that module already.
https://www.digitalocean.com/community/tutorials/importing-p...
>import "github.com/google/uuid"
This should be just "google/uuid".
Plus if you really don’t want a full domain in your projects import paths then you can vendor in those modules, albeit at a cost of higher maintenance for yourself.
The biggest problem with having the domain name included in the import path is just purely visual noise. It’s pretty damn ugly. But that problem is purely aesthetic.
I will grant you that when package versions were encoded in the import path, that was extremely problematic (to say the least). But that’s gone away now we have proper module versioning (and I say that even as someone who doesn’t particularly like the solution that Go finished up with).
> Third-party packages are always known by their fully-qualified names, which include the site that hosts the code (e.g. github.com), the user or organization that develops it (e.g. google), and the base name (e.g. uuid).
"google/uuid" tells you nothing about the owner. It might be Google, or it might be me, who squatted the name early on.
"github.com/google/uuid" would be much difficult to squat. Also, it tells you that this comes from the Google user in GitHub, which is something much easier to verify.
"code.google.com/go/UUID" would be even more of a guarantee. But I guess they decided relying on GitHub to host their Go packages was good enough.
https://go.dev/ref/mod#goproxy-protocol
I guess Go sidesteps this by not having a registry, and Rust seems to require you to explicitly say which registry a package comes from. But pip is vulnerable to this issue because it still doesn't have namespaces.
In particular, I don't see any reason why the package-name needs to be related to the mailing-list/bugtracker, or to the location of the canonical code repository.
XML uses URLs as labels for namespaces; they are just labels. Nobody expects to find a document about the namespace, nor a schema, at the namespace URL. There's never a need to resolve a namespace URL. Why should it be necessary to resolve a Go package URL?
[Edit] I honestly don't know, because I've never used in Go.
That's in itself a really simple system and doesn't rely on a single third party, but obviously comes with the ownership issue (unless when pinning a specific commit) - in my opinion a good compromise.
Your argument just favors distorting the concept of URL to use URL as URN. But if we were going to use URLs to identify abstract resources, why don't we use URNs instead?
Not quite.
In go, you import 3rd party packages via a URL where a VCS can find them, eg.:
The package name is thus both, an unique name for the package AND a location where the toolchain can find and fetch it.Someone famous once said "Good URLs don't change". Turns out he was pissing in the wind; nobody paid attention. And the market in DNS names guaranteed he'd be wrong.
> (In Rust) there will be namespaces soon anyway, just not the kind that would require a huge paid manual review/support staff to administer everything
Is he talking about Maven Central?
Because using DNS prefixed packages as a means to avoid namesquatting seems to me like the best solution out there. For all its faults, Maven got a lot of package-management things right, and I'd argue DNS prefixes is one of them.
I wish that solution wasn't handwaved so quickly.
You just make an account at Maven Central, prove ownership of your domain name, and can then start publishing packages with your publisher credentials. If you suddenly lose your DNS name (as in the case of .ga), you'll still be able to publish packages for a while, at least until there is a new ownership verification, giving you ample time to publish new versions with deprecation warnings.
But, of course, that requires people on top of the registration system, approving requests, attending issues. Claiming that the need for such staff is something to avoid, sounds to me akin to if we were in the 90s thinking how to distribute software packages in our Linux distro, and someone said the need for package maintainers would be something to avoid.
For a long time I did not even realize that all of my Go package installations were going through Google-controlled infrastructure by default; something I'd object to had I known sooner. The cache also tends to serve the last working release, rather than the one that's actually the latest, so you sometimes don't even realize that the tip is broken. I can forgive most things, but not hiding errors, especially as I bash my head against the keyboard. On top of that, proxy.golang.org has historically not been playing nice with upstream repository servers[1]. I tend to keep GOPROXY=direct in my environment for these reasons.
[1]: https://sourcehut.org/blog/2023-01-09-gomodulemirror/
https://proxy.golang.org does not save all modules forever. See FAQ on the site.
If control of crates.io or npmjs.org change, how many rust/js projects will still build?
The trade-off seems to be between distributed vs centralized. Where in the distributed case, you take on more risk that something will break for a lower chance that everything will break. And in the centralized case you have a lower chance that something will break, but if anything does, you're completely SoL.
Now, I think there's a conversation to be had on that trade-off, but it doesn't feel like the article really covers it. Centralization has some really nice day to day benefits, but it's not exactly a trade-off with zero downside.
If my package is at example.org/pkg/mypkg and I stop paying for my domain name for any number of reasons then that package is essentially lost.
With the bonus that if github does go down, you can redirect it to another mirror of the package if the maintainer puts one up (or you can vendor early, but that's an option for everything).
It seems you have two specific options.
Centralize on a big company's url, which you can use in go via sticking to github.com (and a couple other domains) packages as well as js/rust/java/etc.
Decentralize it and have to worry about constant low-level package rot, but not be able to have any individual domain take out all of your dependencies, which you can only really do in go (or languages like C where git submodules are a fairly standard form of dependency management).
There is a tradeoff of having a single point of failure in the first example in exchange for increased reliability during normal operations, and in the second example, you're much more likely to be hit by package rot, but you're significantly less vulnerable to your entire dependency tree being wiped out. This is the tradeoff I'm talking about.
By default go mod fetch packages from Google proxy/cache.
https://sum.golang.org/
https://index.golang.org/index
Doesn't sound like a good situation to me.
Luckily the go ecosystem has a number of tools for dealing with this. You can set up a local trusted proxy of all the code you use internally and point your builds to that instead of Google’s default proxy or just doing direct pulls. Or you can vendor the code you’re using into your own repository. You can also easily alias specific libraries to point elsewhere if specific ones are problematic, without having to change the code directly.