132 comments

[ 3.6 ms ] story [ 17.8 ms ] thread
Isn't this going to introduce the dependency of domain management? I understand the positive side of it, but put some infra level manamgment layer to individual Open Source devs, just a thought. But yeah, positives vs negatives weigh and pick.
GitHub is almost forever. Your custom domain disappears when you stop paying the bills, which if you’re an open source developer has a higher likelihood than GitHub disappearing.

One day, we’re all going back to vendoring dependencies.

Why did we stop vendoring dependencies in the first place?
Because its not as convenient
Can you elaborate?
Updates mean you have to copy over all the code into your repo, which creates a large diff, and hope you aren't overwriting any local changes someone might have made.

It bloats your repo, both with the actual code, and the large diffs when you update it.

You have to manually track new versions, without something to tell you if new versions are available, or if your version has known security vulnerabilities.

If the dependency has it's own dependencies, you have to vendor those too recursively. And if multiple dependencies have the same transitive dependency, it is up to you to deduplicate them, and make sure you have a version compatible with all dependents.

Etc.

Disk is cheap.

Recursive dependencies have the same issues whether you vend them or not.

Ensuring that diffs to updated dependencies remain within a vendor folder is trivial.

So, what’s left?

> Disk is cheap

The biggest problem isn't (usually) disk space, or network bandwidth, it is that git operations slow down as the size of the repo grows. And it means that cloning or pulling the repo takes longer, which can be especially problematic for CI.

> Recursive dependencies have the same issues whether you vend them or not.

Package managers usually handle resolving recursive/transitive dependencies for you. Some have support for vendoring dependencies, but not all do. In theory, you could have similar tooling for vendoring dependencies, but in practice that often isn't the case.

These are all solvable problems IME. But it does mean you need people who are experienced at solving them or who care enough to learn.
> But it does mean you need people who are experienced at solving them or who care enough to learn.

That sounds like it's inconvenient to me.

> Disk is cheap

If I want to upgrade the disk on my MacBook Pro, I need to buy a new MacBook Pro with a larger disk. If I want to upgrade the disk on my work laptop, I’m SOL.

> Ensuring that diffs to updated dependencies remain within a vendor folder is trivial.

It’s not obvious to me how putting the dependencies in a vendor folder solves the diff problem. Does every code host allow you to hide diffs to certain directories?

And what’s the advantageous scenario for vendored dependencies? Is it just when the mod proxy and the upstream code host go down at the same time?

Because it sucks. But maybe it should suck? It would make us think twice before adding dependencies.

I use dotnet and I never liked seeing dlls and binary files in my diffs. I would argue if we are adding vendor code to our projects, we should demand the FULL source code instead of dlls. Maybe it is already possible with things like x unit. I have never given it much thought... But then that vendoree code has to come from somewhere as well, right? I mean there is something to be said about provenance or something here?

Sorry if this feels like a stream of consciousness because it is ↔

That’s a bit too judgmental a response for me to evaluate. How about facts?
I think its a good question. The short answer is: because tooling doesn't default to this or make it easy.

The answer to the question of why THAT is the case - is not so easy to answer. In fact I have no answer. I think Go used to work this way, but its the only language I can think of that ever did (in terms of standard tooling).

I vendor all my Go dependencies all the time for every project. It is the way.
Has this ever been necessary/useful? Genuinely curious because I’ve been using Go since 2012 and can’t recall a time when vendoring solved a problem better than “regular” modules. Like have there been times when the source went down and the module proxy went down (or didn’t have your dependency cached)?
In the case of a typical software enterprise having your domain gone means that you probably don't care about the code anymore anyway.
Unless the field of software development somehow totally stagnates, I'm confident that at some point I will move to a different hosting service, as github goes the way of sourceforge, travis-ci, and myspace. So my personal domain will continue to be relevant (it's lasted fine 23 years).

Maybe at that point it won't be using git either, or maybe it will - who knows.

Mine has lasted over 25 at this point, but one day I will die, my domain name will lapse because what I do as a career might as well be witchcraft to anyone in my family and then my modules will be lost forever. Sucks any way you cut it. Hopefully someone who found my modules useful will fork whatever they find and keep them going.
go mod vendor... works for me. Also it makes sense for an ecosystem like Go with strong standard lib and strong backward compatibility commitment.

Also: https://proxy.golang.org/

> In my opinion, every commerical software development team using Go should be using custom domains for namespacing their internal libraries and packages.

I’d remove “go” from the above, i.e. I think same applies to other stacks.

Even using GitHub domain links in code comments gets problematic long term. Ie when a migration happens and those links start pointing nowhere.

I'm confused by the article, and this comment, treating these URLs as difficult to replace.

Why can't you just search-and-replace? Presumably all of them refer to "GitHub.com" and not much else code will, so I'd think this was an exceptionally easy case.

Even easier for comments, since them being obsolete for a few hours during a migration doesn't exactly break anything.

Even easier than that, you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from; you can point to a folder on disk or to another forge, and if you want total control you can indirect everything to your own artifact cache via GOPROXY (which can be something as simple as a static folder of source code).

The "module name is network path" is a convenient convention but not at all some "limitation" of the tooling.

> you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from

Heads up for anyone who doesn't know: this only works at the "top level". Any replace directives in your dependencies will be ignored[1].

So for example if you have a dependency tree like [main -> thirdpartyframework -> golang.org/x/net/http2], and thirdpartyframework uses a vulnerable version of `golang.org/x/net/http2`, you can't just fix it by patching the thirdpartyframework repository with a replace directive; no, because that would be too convenient. Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.

Even though I like Go, it really seems like they don't care about anything other than monorepos. As soon as you need to work with forks, mirrors, or even just private modules[2][3], the tooling actively works against you. Also using your custom module proxy is a pain.

[1] See: https://go.dev/ref/mod#go-mod-file-replace:~:text=replace%20...

[2]: If you've only used private modules hosted on GitHub you might not have noticed too much pain because the Go tooling has hardcoded behavior specifically for GitHub and a few mainstream forges. You don't find out about this until you try to self-host something like Forgejo on your own domain thinking it would Just Work(tm), but it doesn't, and now you're left wondering why tf it works with GitHub but not with your own forge instance.

[3]: I think there's no hardcoded code for SourceHut, so you might be able to experience the inconvenience by hosting private modules in there.

> Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.

Wait, what? If you decide, in your own application, to force all your dependencies to use a specific version of golang.org/x/net/http2, then obviously you'd want to be able to put this directive into your own application's source instead of going around patching 3rd-party repositories (that's just rude).

What about anyone else using your code? You can find and replace your own code, but do you have access to all of the code relying on your go library? Is it all yours? Customers? Other developers?

Even in the case where it’s an internal only Lu array, it can be complicated to refactor a library name.

Thank you for the concise explanation! :)

I did not realize it ran that deep - I'm used to distributing EXE files, not anything that would need to know internal dependencies like that.

I mean yeah, but other languages don't make it quite so easy to make this mistake. Once your Go project gets to a size where it makes sense for different files to live in different folders (which, given Go's folder-based module system, often happens at just a couple hundred lines of code), the most straightforward way which the tooling nudges you towards means putting GitHub URLs (or URLs to whatever git host website you happen to use) in your source code.
Or example.com/yourname if your module is never going to be imported from another project.
to clarify: I mean literally "example.com", not your own domain name
I do something like this in all my projects now. Except that I don't use example.com, I just use the name of the project. So if I'm working on a program called Frobnicator, I'll just have `module Frobnicator;` in my go.mod and do things like `import "Frobnicator/lib/blah"` in my source files. It works really well to be honest for non-library software.

But it's something you have to do as an active choice; the tooling, and your colleagues, will nudge you towards using URLs to your primary git host's web front-end. And it naturally doesn't work for libraries.

If the first part of the module path is without dot, it is assumed to be stdlib.

Many tools will fail if you break this contract.

Oh that's horrible. I've somehow never encountered it but damn.
I think this is one of the biggest missteps made in Go and in all honesty I don't think it was any more intuitive than going through a package manager. Probably less, looking back.

Everything else is fine. Generics, errors, whatever...

I think TFA is specifically targeted at Go because of how their package system works.
Yeah, the only problem is when your whole company decides to move to a different custom domain entirely, e.g. from contoso.com to xtools.cn (which doesn't even resolves if you're not on the corporate VPN), and half of your friendly teams, whose libraries you depend on, drag out their own migration by fiddling with their local /etc/hosts and .gitconfig and .ssh/config (yes, they also wrote bespoke scripts that do those changes to their CI machines; no, trying to run those same scripts on your CI machines breaks your own shims) instead of updating their source code.
oh man deep cut contoso ref (contoso being the fake company used widely in microsoft demos/documentation/examples)
I don't love that either, having all of your stack point to your own domains for how it can be accessed is nice. But in my opinion, it's desirable to be able to build your whole stack from source code. Of course you download external dependencies, but for code you own you should be able to build it from source.

I have seen too many cases of requirements on internal projects that prohibit devs from working because oops the vpn is down now or oops gitlab is under load and the pipeline for your dependency won't finish for the next hour.

I also think that vendoring or at least caching your dependency tree somewhere local, so you can build everything from ground up to a known state.

It's not only an issue of availability or depending on circumstances you can't control, but a matter of reproducibility hence dependability.

I want to be able to compile critical software when a zombie apocalypse hits, and I have a growing discomfort about trusting outside parties for their availability and benevolence.

In the new world of AI and all the supply chain attacks on centralised package managers, I think circling back to vendoring probably makes more sense than depending on the network in your build step. Which in some ways is always great, as you mention, in terms of working offline.

Even if you don't want to go as far as vendoring (which can get a bit out of hand with, say, NPM), a middle ground is using Artifactory or whatever as a proxy.

Then be more deliberate about when you update. AI may even help here where static analysis doesn't, because you might be able to use inference to see if a dep upgrade is even needed. Unless it has a severe vuln you probably don't need to track latest.

Seems like an error to use a URL. This is perfect for a URN or some other form of URI.

It could be a URN that used the DNS as a back end (though that’s a lot like a URL) so better would be something more abstract with multiple possible resolvers and a signature.

The only useful alternative you're talking about would be a content-addressed one. A "URN" that relies on some kind of mutable registry lookup is actually a URL (both are URIs).
Not really convinced. I'd rather it stayed on Github so it doesn't disappear. Particular for businesses whose priorities might change.
braid has been a treat for us to manage external dependencies. Much cleaner than submodules.
[delayed]
You don't need an SBOM or anything fancy to get started

-- Curious how this avoids SBOM need?

[delayed]
Correct, that is what confused me, how SBOM came into the picture, now got it, thanks!
> That is, if you move your git hosting to GitLab then you have to change your code!

You can also just use "replace github.com/example/example => gitlab.com/example/example" in your go.mod file and everything will keep working. That seems like a very pre-mature optimization for something that doesn't really matter.

What about everyone else using your code? You are going to ask them all to update their code as well? Many of them probably won't realize, and will get stuck on an old version that never gets updated, even to fix bugs and security issues.
I think the migration can be made pretty easy for users with deprecation notices and the `go fix` tool. Better explained in [1] (I have not tried it myself).

The idea is: the package maintainer pushes a final version of the old package that is a shim, and which has a deprecation notice. The shim imports the new package, and forwards all calls to the new package (and I suppose type aliases and variable aliases as well). The shim includes `//go:fix inline` annotations on all exported symbols, so that when users run `go fix` it rewrites their code to use the new package, by inlining their usages of the old package (which is now simply a shim that references the new package).

Not perfect. There is still a bit of a discoverability problem. Not all tooling warns on deprecated packages/functions (go toolchain doesn't, but gopls and staticcheck do). And users need to know to run `go fix`.

1: https://go.dev/blog/inliner#example-renaming-ioutilreadfile

As library maintainer, you can mark the last old domain version as deprecated:

    // Deprecated: use example.com/mod/v2 instead.
    module example.com/mod
As library user, it's your responsibility to keep your stuff updated. There may also be utilities in the various automatic dependency updaters that can migrate these things.
why not just use replacements to begin with if you don't want to bother with setting up domains?

"replace example => github.com/example/example" at first, then change to "replace example => gitlab.com/example/example" or similar when there's a migration

(i don't use go, but i hope this is possible and i am confused if it's not)

Yeah, I think putting github.com in your repo is a mistake but it seems massively overstated. Seems trivial to fix all the code, or to just use go.mod, or first use go.mod and then fix your code.
don't do (thing that is extremely standard and yes has some drawbacks). Solution (nginx.conf)
I have not worked with Go, so I thought I'd ask: why wouldn't simple find and replace be able to fix this? And why wouldn't a coding agent be able to do this for you quickly?
Thanks for answering my question! So yes, if I intend to use go and study the packages I'd have to rely on beforehand, if they're all hosted on GitHub should make me steer away from Golang. Or at least plan to cache the packages I need locally so CI always have something to work with when building.

Alternatively, could we ourselves build an automatic mirror so there's a redundant supply provider that doesn't depend on a maintainer's choice of git forge

https://news.ycombinator.com/item?id=49434625

You shouldn't be using fully qualified domains at all. Use relative paths instead, like sibling-hosted git submodules.
That's a beautiful solution. I love it! The alternative work went with was to just build everything ourselves.
This is great and I think for any company/individual who is going to ensure their domain is registered and maintained this makes a lot of sense. It would be catastrophic, but there is no reason Github couldn't disappear or otherwise change some policies that require moving away from it. The way Go works makes committing these package names tied to Github so much more weighty than simply the place you pull from.

The one thing that I was worried about was returning 301 in the example Nginx config. If you ever wanted to change the url that clients are redirected to then any browsers that visited the old url config would be forced to go to the old config's redirect url. For `go ...` and `curl` it wouldn't matter, but Chrome/Firefox will cache that 301 permanently and break the intended redirect. Not sure if this is really an issue in practice though.

True, but beware of the domain name you're using. Because VeriSign may unilaterally decide to delete your domain name along with thousands of others [1] and you're back to square one…

[1] https://neil.fraser.name/news/2026/09/03/

I broadly like Go's "the import is the hosted location (or a pointer to it)" quite a lot, as it largely solves name-squatting and ownership (while allowing major risks with domain sales/abandonment), but yeah - I really do wish they baked a SHA into the go.mod (not just go.sum) so you could find a library and get a known-good download from any proxy with any name. A few languages now have content-addressed imports/packages, instead of just adding hashes as verification, and I hope we see more in the future.
> It's sorta a shame that Go keeps doing such a good job at a minimum-viable wheel-rewrite, but then lets it linger for so long without catching up to the rest of the programming world.

How many mainstream languages have content addressed imports? I can’t think of any, so I assume I’m misunderstanding your meaning of the term because you seem to be suggesting that it is common and Go is the outlier for lacking it?

Go is not really an outlier for not having signed packages (there are a fair number that have it, but far from most)... but definitely stuck behind common accepted practice. By decades, if comparing against some (e.g. Java).

Which keeps happening with stuff they rebuild from scratch - an excellent and somewhat unique first showing, far beyond what most first attempts manage, but followed by near-complete stagnation while issues that everyone familiar with the field predicted from miles away pile up.

Coming from the Java side this is typical of Google libraries. They start off impressive and gain wide adoption but then they stop supporting changes the community wants, missing basic features.
(comment deleted)
Yeah, it still very much tastes like Google in many of the worst ways :/ clearly Google isn't actually running the project, it's far too well run for that, but the same general "why would anyone need [that thing nearly the entire open source world does outside Google's monorepo]?" ignorance pervades a lot of it.

Which is a shame because there is quite a lot to like about it. And in spite of it all I'm thrilled that it is eating into Python's share in a lot of places.

> I really do wish they baked a SHA into the go.mod (not just go.sum) so you could find a library and get a known-good download from any proxy with any name.

You already can use SHA in go.mod in exactly the same way you would use a version string.

I don’t know about using multiple proxies in go mod though.

Kinda. If there's a versioned release at the same SHA, the next `go mod tidy` will replace it with the version.
At least the fallout will be less if you have to resolve your custom domain differently in the intranet due to something like this.

If github is "taken down" you can't just resolve the whole domain differently, you need to only resolve the packages differently. If your "Golang domain" is taken down, it should be a lot easier to hotfix until something proper is implemented (the quickest and dirtiest would be a hostfile-entry).

reading the article confuses me though I don't want to do the context switching by googling what 3rd level domain is about
All you need to know is that for some reason you could buy foo.bar.com without owning bar.com directly from the registrar.

It is being retired because very few people use it. It kind of sucks but this is how domains work, it’s not real estate.

Or if your goal is to open source code, think about the lifetime of the entity owning the domain: the source code might be of interest to users beyond the control of the domain name. Domain controled by an individual or a company? What if the domain is not renewed, or taken over by an hostile entity?

Deepcopy [1] is a Go package that is still heavily used despite its creator has disappeared 9 years ago. At least GitHub is a stable and trusted host as a distribution point and communication point for users.

[1]: https://pkg.go.dev/github.com/mohae/deepcopy

(comment deleted)
> One of the good features of Go is that you namespace your code with the location to fetch the code.

Then the rest of the article explains why this is NOT a good feature in practice.

I don't think it's unworkable either, but this is one of these little thing that Go decided to do different and convinced its fans that this is a great idea and all the other languages where doing it wrong. After a couple of road bumps appeared, instead of admitting there are some advantages to having official package names, we're now told that everybody should just set up their own custom domain with an nginx server or a Go Vanity URLs forwarder to serve traffic for their GitHub-hosted packages.

combine these domain aliases with the caching goproxy you're supposed to use, and they're well on their way to reinventing a traditional package index from first principles
> That is, if you move your git hosting to GitLab then you have to change your code!

Not to be cynical about this but I fail to see how running sed s///g after a very rare event qualifies as a serious problem.

Now, sure, given that the solution is so simple, it's a nice recommendation, but still...

I'm not sure that custom domains are a good idea. It's way too easy to end up with broken links once vanity domains go away.

However, Golang saves the hashes of all the modules in `go.sum` files, so we just need to add a way to do content-addressable fetches. This solves the issue of reproducibility for old builds. As long as you can find the module in some repository somewhere, you'll be able to build it.

The next step is supporting module _evolution_. We need a way to declare: "From this point onward, `github.com/company/someproject` is now `company.com/someproject`", so that all the references are to these packages are identical. This is possible on a per-package basis with `replace` directives, but this doesn't scale.

And this is not easy to solve in general (especially in the age of supply-chain attacks). If the initial project cooperates (or if Github can be convinced to help), perhaps at least a part of this can be solved by adding special "redirecting module" support to Go.

Great when a company goes out of business and the domains are dangling. Next one to scoop it up takes over source code that others new depend on. We've seen this many times already in other ecosystems. It is a very bad situation.

It is bad advice to move your packages under your own domain. You will never be as good as Microsoft to keep paying for the domain. There are no guarantees in life but I do guarantee you that when you go out of business, that domain is the last thing you will think about.

Thinking this through, using my domain might actually be much less risky than using a host's. If Gitlab closes down, and someone else buys the domain, then I'm forced to make a massive refactor with little notice.

If I own the domain, I'm likely to continue to owning it until I shut down my company. At which point, no one is likely to be running my code.

Possibly different if I was developing an open source library or something, but either way there's risk and need to be thoughtful of the risk versus impact.

Well, actually you don't download from GitHub - you download from the central mirror proxy operated by Google, and that will always return the same package given the same versioned reference, even if the original source differs from it.
Here's the primer for those who thing find a replace is the answer. Pretend you have a project with A 1.2.3 depends on B 2.4.6 which depends on C 0.1.1. If you are in github you get module github.com/example/A requires ( github.com/example/B 2.4.6 github.com/example/C 0.1.1 )

You now have many choices on how to proceed, but none of them will include A) being able to build old releases, or B) doing so without making changes to all dependencies.

One choice is go to the leaves, C in our example, and make a release on gitlab. Then go to B, and have B depend on the gitlab C. This is fine for rolling forward, but if you want to roll back you would have to rewrite all of C to use the gitlab url, and find all of the equivalent tags and repush all of them. Also tell your users that any binaries you've released now have new hashes. Then repeat this for every repo. It's not a small task.

Honestly google should have always had a package index even if it points tooling to the repo, but it would maintain the package name with the ability to change where the repo is.
I fear one of these days someone will copy a legit package and make it onto your search results, and then the Go tools pull in something extra.