StableLib https://stablelib.com/ seems to be taking an interesting approach to libraries. Go guarantees backwards compatibility and it's awesome for developers. Why shouldn't libraries have the same guarantees?
The standard library is pretty nice, but one of the reasons for that is that it is a well tested well supported backwards compatible library. It would be nice if there were more of those.
It's pretty much a given that the first few libraries solving problem X will have a sub-optimal API, especially if they aim to be both general (say read and display JPEG) and full-featured (support all the optional bits of the format or work efficiently in low-memory embedded situations, etc.) at the same time.
All of that while preserving the feel and idioms of the host language. This is key since it invalidates a lot of prior experience with similar APIs.
So until the "right" API coalesces from collective experience with the problem domain and APIs for similar libraries (read and display {TIFF, GIF, PNG, etc.} in $LANGUAGE) backward compatibility is not a feature you want.
Wait for the second or third wave of libraries where the problem is solved and understood in a nice idiomatic way for the language and its quirks.
Perfect is the enemy of good I guess. If I'm building a product now, I want a library that works well now and never breaks when I upgrade to the latest version. If someone gets a great idea for a better library, make it a separate library. So backwards compatibility is definitely a feature I want.
Yeah. If you're concerned about limiting the amount of additional code one needs to understand in order to use part of a library of code, then the fix is relatively simple: make each function in your library depend on as little of the rest of the library as is possible.
If the author is legitimately concerned about code bloat from the unused parts of a library, Go's linker needs to get a lot smarter.
It's just against every software engineering best practices. You find a library which implements what you want/need? You don't copy&paste 10% of it into your repository just to make sure it won't ever brake. What if it's broken already? They will fix it and your version won't be updated. This logic is not even ok with your own code, why should it be promoted with somebody else's code?
I think I agree. There are a lot of great things about Go, but (currently) library versioning is not one of them.
This doesn't contradict much of what the author writes: it really is true that two different versions of one library are actually two different libraries; it really is true that sometimes copy-and-paste is better than generalising (when the cost of being general obscures the meaning of the code itself); it really is true that many libraries should just specify and implement an API, then stop.
The issue with libraries in versions in Go could probably be resolved with a more flexible import syntax. It'd be sweet to be able to write 'import foo.invalid/bar@94d0ef312881e694ded9ff38e33a2c07fb398eb8' and know that I'd always either get the exact version that I specified. It'd be even cooler to have some way to specify 'I want the latest version of library X which offers the exact API I'm using'; in principal this would be possible by calculating some deterministic ordering of the used, exposed API and hashing that, but the details would be tricky (as an example, consider a case where Foo() and Bar() still exist, with the same signatures, but in version 2.0 Bar() can only run properly if Baz() has been called previously—there's no way in Go to express that sort of contract to the compiler.
You'll already have to recompile because go is statically linked.
The current state of the art is just vendoring dependencies (see godeps) which means they never get updated ever. Security updates don't happen now, and version pinning by vendoring is a hack, so it would be better to at least have version pinning built in so I can delete a few million lines of (vendored) code from my repository and instead just use a better import path. I assure you that that will be easier than dealing with the hell of vendoring tools and hacks go has now.
And yes, the semver way to do it is great and is actually done well in some ecosystems. npm does a really good job of encouraging users to use semver and as a result you can just say "I want the latest 1.1.x release" or "1.x.x". The hardest problem there is to get people to version things.
Your next point, that you don't want to recompile all go programs, is where I'll again point out go is statically linked. The authors of the language designed it so that no, you can't just update a go library in one place and build once. You have to build everything that depends on it again. You have to hope there were no breaking changes or you have to backport patches. Go does not handle this and is designed so as to be actively hostile to security.
And yeah, the idea that semantic changes are backwards incompatible is a developer call, but you need a programmatic way to express it.... hence semver.
In conclusion, having that sort of import-path thing would be awesome not because it solves any of the things you complain about (it doesn't) but because it's a step in the right direction. Any change is good change when your package management is "eh, fuck it" like Go's is now. If you look at all other good package managers (e.g. npm, rubygems, crate, etc) you can specify a repository and specific commit because there are plenty of valid cases for that, even when you have other options. No one is saying Go shouldn't have those other options in the future too here, just that having this option is a step in the right direction.
> I haven't studied Go. Your telling me every file is statically linked? How does it use the OS APIs?
OS APIs are accessed through syscalls, you don't usually need to link to anything.
The exceptions to static binaries have been to use the host resolver when necessary, the user db on linux, and looking up certificates on osx; none of which are directly available through syscalls.
Honestly, with the release of Go 1.5, I've taken to using "go get" once to download the library, copy it over into the vendor folder, remove its .git folder, and commit the whole thing to source control. It's easier than messing with version numbers or commit hashes, and more stable than always using the latest master.
It puts you in the driver's seat of dependency managing, but now you have to be a lot more careful to actually update them when the time comes. If you tend to put something into production and leave it there for months / years, coming back to do maintenance will be a real chore.
With Ruby I can 'bundle update', run the tests, and have reasonable certainty that everything's working. If I took over a Ruby project with vendored gems I'd mark them all for replacement for standard Bundler-managed gems at first opportunity. Vendored code has a bad tendency to become 'unmaintained code' otherwise.
Just because it works and is stable doesn't mean it doesn't have security vulns.
> The Stockholm syndrome shown by Go users in things like this is just incredible.
Well, the problem is that Go is a bait/switch . It has enough goodness to make someone enthusiastic about it at first place. And then when one finds out that the language has major issues , one has two way to deal with things : denial or giving up on Go. Go is a niche language , its "community" will always be very protective of what Go is no matter how much flaws it has, because this community isn't really in control of the language anyway. This is a top/down governance, with the Google team at the top, and there is no debate what soever. There used to be but as I said earlier people that are not satisfied with Go have been shut down in the mailing list (or by silly processes imposed by the go team when it comes to contribution or feature suggestion ) and moved on.
50 dependencies is a pretty small number. My guess is your main issue is the monolith that has piled up. A pragmatic node.js developer would have been able to greatly reduce the size of your monolith by breaking out independent functionality and managing it with npm.
Oh, I didn't build it that way. We're far down the path of extracting microservices at this point but it was a fully entrenched monolith about 4 years ago.
I am surprised Rubygems has not implemented some kind of semver helper to easily report which libraries have minor updates that can likely be trivially installed...
The ideal utility would look at a Gemfile.lock and show changelog information to help the developer evaluate which libraries can be updated immediately, which require some code/api review, and which will require refactoring.
the thing is it doesn't matter since a gem file and lock file will tell you what is the version of the library you are currently using.
What is the version of the library go get has fetched one month ago ? you don't know ? and you will never know.
The problem therefore is not 3rd party libraries that can break. The problem is the tool go use to get dependencies doesn't help you with something like that.
Therefore all this "a version of a lib is a different lib" is a straw-man masking the real issue with go get. In every other modern language ,there is no such issue.
It matters because of inter-dependencies between all those gems. I'm not saying it's "the way" but Go's popularity is almost entirely due to solving problems that people experience in other modern languages.
Dependency problems are one of those.
Two gem needing a specific version of a common gem immediately bind their upgrade paths together for the life of your application. Without including version numbers in go get, it forces those 3rd party libraries to be backwards compatible. Some people like that.
I would argue that this is only if they do not follow Semantic Versioning, which any self-respecting ruby gem should. (Ignoring the fact that SO MANY production gems still use 0.x.x versions...)
Semantic Versioning is not a magical cure-all. It's still up to the author to make the right choice about which number to bump.
We have a node/gulp-driven build with, ultimately, about 500 dependencies. We don't store node_modules with the code, we run npm install with each build. This leads to periodic failures because some sub-sub-sub-dependency bumped number 3 when it should have bumped number 2, causing unwanted build-time behavior. Even if our package.json lists specific versions of top-level dependencies instead of ranges this will happen. It's aggravating.
Projects exist to bundle up known-good node_modules and restore at build time to avoid this. NPM offers the concepts of shrinkwrapping and peer dependencies so you can tell consumers which specific version of each runtime dependency they have to use as well. IMO if SemVer actually worked in practice, none of this would be necessary.
Some of this is an indictment of node/npm - maybe Ruby is just better - but I still think SemVer is overestimated.
31 comments
[ 2.3 ms ] story [ 126 ms ] threadThe standard library is pretty nice, but one of the reasons for that is that it is a well tested well supported backwards compatible library. It would be nice if there were more of those.
All of that while preserving the feel and idioms of the host language. This is key since it invalidates a lot of prior experience with similar APIs.
So until the "right" API coalesces from collective experience with the problem domain and APIs for similar libraries (read and display {TIFF, GIF, PNG, etc.} in $LANGUAGE) backward compatibility is not a feature you want.
Wait for the second or third wave of libraries where the problem is solved and understood in a nice idiomatic way for the language and its quirks.
If the author is legitimately concerned about code bloat from the unused parts of a library, Go's linker needs to get a lot smarter.
This doesn't contradict much of what the author writes: it really is true that two different versions of one library are actually two different libraries; it really is true that sometimes copy-and-paste is better than generalising (when the cost of being general obscures the meaning of the code itself); it really is true that many libraries should just specify and implement an API, then stop.
The issue with libraries in versions in Go could probably be resolved with a more flexible import syntax. It'd be sweet to be able to write 'import foo.invalid/bar@94d0ef312881e694ded9ff38e33a2c07fb398eb8' and know that I'd always either get the exact version that I specified. It'd be even cooler to have some way to specify 'I want the latest version of library X which offers the exact API I'm using'; in principal this would be possible by calculating some deterministic ordering of the used, exposed API and hashing that, but the details would be tricky (as an example, consider a case where Foo() and Bar() still exist, with the same signatures, but in version 2.0 Bar() can only run properly if Baz() has been called previously—there's no way in Go to express that sort of contract to the compiler.
The current state of the art is just vendoring dependencies (see godeps) which means they never get updated ever. Security updates don't happen now, and version pinning by vendoring is a hack, so it would be better to at least have version pinning built in so I can delete a few million lines of (vendored) code from my repository and instead just use a better import path. I assure you that that will be easier than dealing with the hell of vendoring tools and hacks go has now.
And yes, the semver way to do it is great and is actually done well in some ecosystems. npm does a really good job of encouraging users to use semver and as a result you can just say "I want the latest 1.1.x release" or "1.x.x". The hardest problem there is to get people to version things.
Your next point, that you don't want to recompile all go programs, is where I'll again point out go is statically linked. The authors of the language designed it so that no, you can't just update a go library in one place and build once. You have to build everything that depends on it again. You have to hope there were no breaking changes or you have to backport patches. Go does not handle this and is designed so as to be actively hostile to security.
And yeah, the idea that semantic changes are backwards incompatible is a developer call, but you need a programmatic way to express it.... hence semver.
In conclusion, having that sort of import-path thing would be awesome not because it solves any of the things you complain about (it doesn't) but because it's a step in the right direction. Any change is good change when your package management is "eh, fuck it" like Go's is now. If you look at all other good package managers (e.g. npm, rubygems, crate, etc) you can specify a repository and specific commit because there are plenty of valid cases for that, even when you have other options. No one is saying Go shouldn't have those other options in the future too here, just that having this option is a step in the right direction.
OS APIs are accessed through syscalls, you don't usually need to link to anything.
The exceptions to static binaries have been to use the host resolver when necessary, the user db on linux, and looking up certificates on osx; none of which are directly available through syscalls.
It has a nice effect where every change to a library requires a commit which can trigger a build. There are no hidden updates.
With Ruby I can 'bundle update', run the tests, and have reasonable certainty that everything's working. If I took over a Ruby project with vendored gems I'd mark them all for replacement for standard Bundler-managed gems at first opportunity. Vendored code has a bad tendency to become 'unmaintained code' otherwise.
Just because it works and is stable doesn't mean it doesn't have security vulns.
Well, the problem is that Go is a bait/switch . It has enough goodness to make someone enthusiastic about it at first place. And then when one finds out that the language has major issues , one has two way to deal with things : denial or giving up on Go. Go is a niche language , its "community" will always be very protective of what Go is no matter how much flaws it has, because this community isn't really in control of the language anyway. This is a top/down governance, with the Google team at the top, and there is no debate what soever. There used to be but as I said earlier people that are not satisfied with Go have been shut down in the mailing list (or by silly processes imposed by the go team when it comes to contribution or feature suggestion ) and moved on.
Should be output to the screen in BLINKING ALL CAPS on every invocation of rubygems and npm.
The ideal utility would look at a Gemfile.lock and show changelog information to help the developer evaluate which libraries can be updated immediately, which require some code/api review, and which will require refactoring.
What is the version of the library go get has fetched one month ago ? you don't know ? and you will never know.
The problem therefore is not 3rd party libraries that can break. The problem is the tool go use to get dependencies doesn't help you with something like that.
Therefore all this "a version of a lib is a different lib" is a straw-man masking the real issue with go get. In every other modern language ,there is no such issue.
Dependency problems are one of those.
Two gem needing a specific version of a common gem immediately bind their upgrade paths together for the life of your application. Without including version numbers in go get, it forces those 3rd party libraries to be backwards compatible. Some people like that.
We have a node/gulp-driven build with, ultimately, about 500 dependencies. We don't store node_modules with the code, we run npm install with each build. This leads to periodic failures because some sub-sub-sub-dependency bumped number 3 when it should have bumped number 2, causing unwanted build-time behavior. Even if our package.json lists specific versions of top-level dependencies instead of ranges this will happen. It's aggravating.
Projects exist to bundle up known-good node_modules and restore at build time to avoid this. NPM offers the concepts of shrinkwrapping and peer dependencies so you can tell consumers which specific version of each runtime dependency they have to use as well. IMO if SemVer actually worked in practice, none of this would be necessary.
Some of this is an indictment of node/npm - maybe Ruby is just better - but I still think SemVer is overestimated.