> They keep old releases in their CI configuration and block PR’s from being merged if they don’t work on older release lines.
If you are doing that, shame on you, that's not how stable and long-term-support works at all. If you backport, you backport on a best-effort basis, not on a must-have basis.
There is 0 sense in supporting old versions when you just backport all the new stuff into them anyway.
>You, in your GitHub project, probably don’t have the time or resources to manage more than one release line of development.
That is certainly true. Over the whole essay I don't find much reason not to have a LTS schedule.
I don't have to backport all the stuff to older releases. My own policy is that any previous release will receive security patches as long as I can be made to care (usually about 1 year and in one case I support a 3 year old release of a project with security patches)
It is not that much effort to patch security stuff. It doesn't take you away from your project for days, it costs you a few hours and some users of your project will be happier for it. That's all I want.
Agreed. I'd go a step further and say requiring constant upgrades to get even security patches makes semantic versioning essentially pointless. It doesn't really matter that the version number indicates an API change if my only choice is to upgrade or get completely left behind.
I hit this a lot of with the Rails ecosystem. In many cases, upgrading to the latest version of some gem was more disruptive or riskier than keeping the current version and patching myself. But some devs will also yank old versions making that tricky as well. Rails itself, however, does a remarkable job of providing both patches and backports for security issues.
I'd like to extend a big thanks to anyone that does expend extra effort to provide a solution to existing users. When having to deal with a security issue, nothing's more frustrating than not being able to upgrade immediately because doing so entails API changes or incompatible dependency graph changes.
> I don't have to backport all the stuff to older releases. My own policy is that any previous release will receive security patches
Besides it being easier, isn't not porting feature patches the entire point of Long Term Support releases? People use LTS because they don't want any unexpected behaviour changes, only security updates.
Well said. Redis is almost 100 percent backward compatible back to the first release, so I never found a good reason to support old released. Want support? Upgrade... Of course this does not stop an infinite stream of bug reports about several years old releases.
I believe that the Angular team is also taking this approach. [1]
[1] At ng-conf 2017, the core team was asked whether they'd consider designating LTS releases, and the answer was essentially what you're saying here: They work very hard to be backward compatible, and encourage everyone to move _forward_ instead of getting stuck in the past.
They did, but quite honestly they should have just renamed the framework; Angular 1->2 is less of an upgrade and more of a completely new framework that happens to share the same name. It was a mistake imo, but one they made because Angular is a household name amongst web devs.
From what I understand, they've done a better job from Angular 2-5.
The destructive practice is not knowing how to write stable things. There is a small period at the beginning of a project where stability doesn't matter. There is a long period where instability will lead folks away.
> The destructive practice is not knowing how to write stable things.
Well, you can do it the Linux way, too: even if it is a bug, as long as fixing it breaks existing userland code, the fix gets reverted and the docs adjusted.
Compare this to anything except plain JS, jQuery and PHP - breakages all the time, everywhere, because no one values backward compatibility any more (python2=>3, for one, or whatever MS is choosing as the current framework to develop Windows applications now). This is the difference that MS took with the win32 layer and what made them successful - until they started ignoring BC in favor of loading new crap on consumers and enterprises alike every patchday.
I think it's dishonest to claim that Python does not value backwards compatibility because of the Python 2->3 update. That's a single point of backwards-incompatibility in a project that has been going on for decades. (If you don't agree, please list all the backwards-incompatible changes that have been introduced in the years upon years since Python 3.0.)
>They keep old releases in their CI configuration and block PR’s from being merged if they don’t work on older release lines.
Wait, people do that? If every PR lands on the next release and on the LTS release, how is the LTS branch different from the latest release? Isn't the point of LTS that it stays the same (except bugs) and isn't evolving?
I think they're talking about code bases that use the software with the LTS, not the code base of the software with LTS itself. They test against latest and LTS, and your PR needs to work on both.
Sounds like the author picked a very specific case and assumed all LTS was that way. I've personally always understood LTS to be a feature freeze with only security patches being back ported.
I've worked at places that did stuff like the author suggests without pinning versions, and it always just ends up with some unlucky person spending their weekend fixing a dumb bug cause by an upgrade.
LTS is the reason we can all sleep soundly knowing our code won't spontaneously break, so I'm extremely grateful to the hard work everyone puts into it.
I generally tow the line when it comes to software, but I do have a couple contrarian opinions and one of them is that LTS is really, really dumb. Not just for maintainers, for everyone.
LTS destroys your codebase. LTS allows mistakes like pushing state into your server that you didn't know was there. LTS makes it much harder to upgrade because by the time you're moving off of, say, Ubuntu 12.04 the whole world has changed. LTS multiplies the number of things that can go wrong. LTS makes people think that the OS is going to be around forever so they don't architect their devices to be OS-updateable. IoT is bad now, but it's slowly getting worse too because half these devices are stuck on some out of date version of linux running some out of date version of Python.
All my servers run Ubuntu 17.10. Rails is on the latest version. Ruby is on the latest version. I don't pin any of my gems and every time I upgrade Ruby I upgrade the latest version of all the gems too.
This is only possible with extensive testing, and mistakes do get made here and there, but overall it's much, much easier handling these issues one at a time. The only thing I don't do is keep current on packages that I install via `apt` because I don't want to risk building from source.
If you need the latest OS version for your software to run, you're doing software design wrong. Except for very special features, software should run all the way to 10-20 years old OS versions. And if it doesn't, you should have abstractions in place such that all you have to do is change a dll and be good to go again.
Why the hell would a high level language like Ruby have such deep OS dependencies? Use the goddamn POSIX standard and you'll be compatible to every single important OS that's come out in the last decades, except for Windows which can be made POSIX compatible with a few layers of abstraction.
> IoT is bad now, but it's slowly getting worse too because half these devices are stuck on some out of date version of linux running some out of date version of Python.
Sure, this is bad - but if they are to be OS updateable then the OS has to guarantee it won't drop support for things, or increase significantly in size because then it won't for in the flash.
Enterprise customers really like LTS as a concept, but they should be made to pay for it like the article says.
I just think you have a choice of one or neither of these, but not both:
1. Internet connected.
2. Code and forget.
Choose one or none, but not both.
Selling medical equipment that gets hacked by wannacry is not acceptable. It isn't even theoretically possible to promise that the OS won't drop support for things since you may rely on something that is later exploitable. Realistically speaking, unless you're writing code for something crazy like VR, there is a great chance that with tests you can update the code to match the new OS in a week or two.
Using Ubuntu has the big advantage that you dont have to maintain the packages yourself, so you just dont notice all the regressions that maintainer have to deal with on a daily basis.
Seeing fundamental things like glibc or gcc breaking almost every release makes you start to value LTS releases a bit more.
Disclaimer: Im maintainer myself and i feel your opinion is disrespectful towards the people who fix regressions for you (for free, even!).
There is no disrespect. I don't disrespect hardworking people trying to make the world better. This is just what I've found after writing software for decades. It's a viewpoint that I'm willing to change, but also one I think others should entertain so that all our software gets better and stays current.
Also, as I said in the original comment, I only lean on apt for a minority of things. Most stuff gets upgraded right away and you know what? I find and fix or communicate bugs because I'm on the bleeding edge. And I frankly do not see how my using apt invalidates my views on LTS. In the ruby world upgrading is almost never a pain because most of us keep current and test our codebases. I shouldn't be maintaining packages myself, and I do think that if the people behind Debian / Ubuntu / rando packages available took my worldview it would be easier to always be on the latest version and upgrading shit wouldn't be this giant headache all the time.
Common, glibc should break or break something every release. This is insanity.
In language package ecosystems (like Ruby's), you decide when to push out a new release. You find that updating your lockfile breaks your code, so you put it off. You push out a few versions without updating the deps. Then, in one release, you update them all, fix all the problems that introduces, and then push out a new version that has the new deps and works with them.
OS package management doesn't work this way, because, for the most part, for almost the entire history of OSes, developers have had no control over the environment their application is running under. In a lot of cases (enterprise IT; academic/HPC mainframes; PaaSes) someone else has an upgrade schedule for the OS packages, and you don't get any say on when it happens. The OS updates, your package breaks, your problem. You can maybe ask the ops department to hold off on updating, or have your PaaS "pin" your stack (until it becomes deprecated.) Either way, you're now vulnerable to security vulnerabilities while (hurriedly) working to push out the new release so that ops can get back to doing its job as normal.
LTS is a guarantee by the OS distributor that your ops staff can update any package at any time (from the LTS repos, at least), and such updates won't break your deployed custom software. So LTS solves the cases above: enterprise/academic/HPC/PaaS ops staff can just upgrade as they will, without worry that anyone will tell them to hold off.
But the value of LTS goes even further: if updates are guaranteed by the distributor to not break your runtime environment, then you can actually allow OS package upgrades to happen completely automatically and asynchronously to your development release schedule. The OS can just update itself. The OS distributor is now, themselves, part of your ops staff, ensuring that these updates aren't bothering anyone downstream of them. Your actual company doesn't have to care about the OS. It's always going to be secure, because it's going to update as soon as vulnerabilities are discovered+patched, and you can just deploy your code at your own pace without worry.
Now, mind you, this all might sound quaint to people who bake containers for everything and treat said containers' host OS as immutable infrastructure. For these people, OS package security vulnerabilities aren't really something you can let "fix itself"; they're something that requires baking a new version of your container-image, just as app library-dependency security vulnerabilities would be. Relying on OS services rather than bundling a copy into your container-image is a trade-off in this case: you don't get to pin the version of a dependency to ensure its exact behaviour, but in exchange, the distro will attempt to ensure that it will be free of security vulnerabilities while being pretty much the same version you started with.
(Oh, and as a side-benefit, usually LTS package updates don't force system reboots. Recently even kernel updates don't, with fancy ksplice-based patch distribution channels. So LTS releases also mean long individual-host uptimes, which is handy if you're running a service on the cheap and can't afford much horizontal scaling to manage load-balancing under rolling updates, but want five or more 9s of availability anyway. Try getting that using immutable-infrastructure reboot-to-update VM images.)
>> I don't pin any of my gems and every time I upgrade Ruby I upgrade the latest version of all the gems too.
How do you deal with this in production? Gems get abandoned. Some gems get upgraded faster than others. Some before you're ready.
You get into a race where you either need to fix your code, or fix a gem, or not upgrade. What do you do then? What if you don't have time to fix code to deal with a gem upgrade that ca happen at any time?
What do you do when a zero day comes out for nginx or openssl?
When things somehow go sideways I do the same thing: Triage the situation, including taking things offline if needed, make choices that solve the short term situation, run the tests, everything ok push live. Otherwise mitigate the issue by partially removing services (no image upload right now) and getting the rest of them live.
On the server itself I have a second copy of the repo. I install the new version of ruby in that repo and install the gems in that repo. I run the tests there and I check to make sure that Gemfile.lock hasn't changed. All good? Time to go live. Otherwise? Fix things locally first and re-push.
That said, doesn't work at all levels, but in general it's not nearly common enough in the ecosystem to keep compat consistent. When I think of libraries I love to use, I think mostly of libraries where compat is very well maintained.
Package management systems tend to be working against you here. Bumping major versions for "good" reasons makes it so that your biggest library users just get stuck, because they have so much code in the old paradigm. Real compat changes deserve new names/namespaces, so that you can progressively upgrade.
Yeah, for sure. I think the best takeaway for me from that talk was that major versioning is actually a kinda crummy part of dependency managers, because the way we structure them / code tends to allow you to only use one version of some library at a time. I've definitely been bit on many occasions by libraries doing entire restructures of the whole thing and then pushing under the same namespace.
A better strategy for namespacing would go a long way.
The clojure.set thing is crummy but a hard call. That's a super fundamental piece of the library puzzle, and write once, run forever is a really very expected thing there. Those are something that a wrapper can overcome pretty simply though. In a lot of those cases the approach they take does seem reasonable though, namely that e.g. clojure.set/union perform exactly as is documented on the documented inputs.
In my systems integration work I would much rather have LTS than not. LTS is a blessing because I would rather front-load upgrade issues all at once than have them spread out over a few years because we shipped products on a bleeding-edge OS.
LTS helps administrators and integrators maintain reliability, which is very important in the perception of one's (at least our) users. More important than keeping on the bleeding edge.
I think a reasonable business model would be to have original dev(s) push new versions and outsource LTS to some commercial organization, even with dual free/open source license for new releases and commercial one for support. If users don't want to pay for support, project is going to be dead in a few years anyway unless somebody wins a lottery/mines *coins/gets large inheritance $ etc.
Outsourcing LTS support to (mostly) vendors is a big deal for Linux. When I write Linux code that needs a backport, I'll put a little bit of effort into making it easyish. Maybe I'll answer questions about backporting and try, on a not-very-much-effort basis, to help. This makes it way easier to keep the code clean.
I think there's some ambiguity here around LTS for the project itself vs its dependencies, but as one concrete example, I support node 4 for an open source project I maintain, and I would defend that decision.
In my case, the only additional effort needed for node 4 is to use Array.prototype.indexOf instead of Array.prototype.includes, which is mildly annoying, and sometimes I forget and it causes a failure of the node 4 CI build, but really it's not that big of a deal. TypeScript handles the rest of the JS stuff, and the project doesn't use enough of the node API for those differences to matter. Other projects may different, but I think the right call is to drop support for old node versions when maintaining them is hard or because nobody is using them, not simply because they're old.
I want to use open source projects at work, not just in side projects, and at work we're on node 6 and always lag behind the latest node release by some amount, partly because upgrading node is a real project. We're a startup, not a massive enterprise, but it's still pragmatic to focus on real work instead of upgrading our dependencies all the time. If all of the open source projects out there only supported node 8, I think it would make them much less useful in the real world.
Why not just polyfill `Array.prototype.includes`? That's what my project does when the only features we care about missing from an old release are polyfillable.
That's what I'd do in an application, but it's bad manners to modify the global Array.prototype from a library, or to expect your parent application to provide such a polyfill, so usually it's best to just avoid polyfills in open source libraries.
I’ve noticed this problem in the npm ecosystem pertaining to typescript typings. Typescript is evolving really quickly, each new release allows you to be more expressive, and there are stricter and stricter settings to take advantage of. It’s fairly common for a third party library’s typings to be incompatible with new typescript settings. The problem is that maintainers are reluctant to add support in the name of backwards compatibility. The only solution I’ve found in some cases is to completely override the library’s typings myself locally.
35 comments
[ 0.14 ms ] story [ 90.0 ms ] threadIf you are doing that, shame on you, that's not how stable and long-term-support works at all. If you backport, you backport on a best-effort basis, not on a must-have basis.
There is 0 sense in supporting old versions when you just backport all the new stuff into them anyway.
>You, in your GitHub project, probably don’t have the time or resources to manage more than one release line of development.
That is certainly true. Over the whole essay I don't find much reason not to have a LTS schedule.
I don't have to backport all the stuff to older releases. My own policy is that any previous release will receive security patches as long as I can be made to care (usually about 1 year and in one case I support a 3 year old release of a project with security patches)
It is not that much effort to patch security stuff. It doesn't take you away from your project for days, it costs you a few hours and some users of your project will be happier for it. That's all I want.
I hit this a lot of with the Rails ecosystem. In many cases, upgrading to the latest version of some gem was more disruptive or riskier than keeping the current version and patching myself. But some devs will also yank old versions making that tricky as well. Rails itself, however, does a remarkable job of providing both patches and backports for security issues.
I'd like to extend a big thanks to anyone that does expend extra effort to provide a solution to existing users. When having to deal with a security issue, nothing's more frustrating than not being able to upgrade immediately because doing so entails API changes or incompatible dependency graph changes.
Besides it being easier, isn't not porting feature patches the entire point of Long Term Support releases? People use LTS because they don't want any unexpected behaviour changes, only security updates.
[1] At ng-conf 2017, the core team was asked whether they'd consider designating LTS releases, and the answer was essentially what you're saying here: They work very hard to be backward compatible, and encourage everyone to move _forward_ instead of getting stuck in the past.
From what I understand, they've done a better job from Angular 2-5.
Well, you can do it the Linux way, too: even if it is a bug, as long as fixing it breaks existing userland code, the fix gets reverted and the docs adjusted.
Compare this to anything except plain JS, jQuery and PHP - breakages all the time, everywhere, because no one values backward compatibility any more (python2=>3, for one, or whatever MS is choosing as the current framework to develop Windows applications now). This is the difference that MS took with the win32 layer and what made them successful - until they started ignoring BC in favor of loading new crap on consumers and enterprises alike every patchday.
Wait, people do that? If every PR lands on the next release and on the LTS release, how is the LTS branch different from the latest release? Isn't the point of LTS that it stays the same (except bugs) and isn't evolving?
I must be missing something obvious here.
I've worked at places that did stuff like the author suggests without pinning versions, and it always just ends up with some unlucky person spending their weekend fixing a dumb bug cause by an upgrade.
LTS is the reason we can all sleep soundly knowing our code won't spontaneously break, so I'm extremely grateful to the hard work everyone puts into it.
LTS destroys your codebase. LTS allows mistakes like pushing state into your server that you didn't know was there. LTS makes it much harder to upgrade because by the time you're moving off of, say, Ubuntu 12.04 the whole world has changed. LTS multiplies the number of things that can go wrong. LTS makes people think that the OS is going to be around forever so they don't architect their devices to be OS-updateable. IoT is bad now, but it's slowly getting worse too because half these devices are stuck on some out of date version of linux running some out of date version of Python.
All my servers run Ubuntu 17.10. Rails is on the latest version. Ruby is on the latest version. I don't pin any of my gems and every time I upgrade Ruby I upgrade the latest version of all the gems too.
This is only possible with extensive testing, and mistakes do get made here and there, but overall it's much, much easier handling these issues one at a time. The only thing I don't do is keep current on packages that I install via `apt` because I don't want to risk building from source.
Why the hell would a high level language like Ruby have such deep OS dependencies? Use the goddamn POSIX standard and you'll be compatible to every single important OS that's come out in the last decades, except for Windows which can be made POSIX compatible with a few layers of abstraction.
Sure, this is bad - but if they are to be OS updateable then the OS has to guarantee it won't drop support for things, or increase significantly in size because then it won't for in the flash.
Enterprise customers really like LTS as a concept, but they should be made to pay for it like the article says.
1. Internet connected. 2. Code and forget.
Choose one or none, but not both.
Selling medical equipment that gets hacked by wannacry is not acceptable. It isn't even theoretically possible to promise that the OS won't drop support for things since you may rely on something that is later exploitable. Realistically speaking, unless you're writing code for something crazy like VR, there is a great chance that with tests you can update the code to match the new OS in a week or two.
Seeing fundamental things like glibc or gcc breaking almost every release makes you start to value LTS releases a bit more.
Disclaimer: Im maintainer myself and i feel your opinion is disrespectful towards the people who fix regressions for you (for free, even!).
Also, as I said in the original comment, I only lean on apt for a minority of things. Most stuff gets upgraded right away and you know what? I find and fix or communicate bugs because I'm on the bleeding edge. And I frankly do not see how my using apt invalidates my views on LTS. In the ruby world upgrading is almost never a pain because most of us keep current and test our codebases. I shouldn't be maintaining packages myself, and I do think that if the people behind Debian / Ubuntu / rando packages available took my worldview it would be easier to always be on the latest version and upgrading shit wouldn't be this giant headache all the time.
Common, glibc should break or break something every release. This is insanity.
OS package management doesn't work this way, because, for the most part, for almost the entire history of OSes, developers have had no control over the environment their application is running under. In a lot of cases (enterprise IT; academic/HPC mainframes; PaaSes) someone else has an upgrade schedule for the OS packages, and you don't get any say on when it happens. The OS updates, your package breaks, your problem. You can maybe ask the ops department to hold off on updating, or have your PaaS "pin" your stack (until it becomes deprecated.) Either way, you're now vulnerable to security vulnerabilities while (hurriedly) working to push out the new release so that ops can get back to doing its job as normal.
LTS is a guarantee by the OS distributor that your ops staff can update any package at any time (from the LTS repos, at least), and such updates won't break your deployed custom software. So LTS solves the cases above: enterprise/academic/HPC/PaaS ops staff can just upgrade as they will, without worry that anyone will tell them to hold off.
But the value of LTS goes even further: if updates are guaranteed by the distributor to not break your runtime environment, then you can actually allow OS package upgrades to happen completely automatically and asynchronously to your development release schedule. The OS can just update itself. The OS distributor is now, themselves, part of your ops staff, ensuring that these updates aren't bothering anyone downstream of them. Your actual company doesn't have to care about the OS. It's always going to be secure, because it's going to update as soon as vulnerabilities are discovered+patched, and you can just deploy your code at your own pace without worry.
Now, mind you, this all might sound quaint to people who bake containers for everything and treat said containers' host OS as immutable infrastructure. For these people, OS package security vulnerabilities aren't really something you can let "fix itself"; they're something that requires baking a new version of your container-image, just as app library-dependency security vulnerabilities would be. Relying on OS services rather than bundling a copy into your container-image is a trade-off in this case: you don't get to pin the version of a dependency to ensure its exact behaviour, but in exchange, the distro will attempt to ensure that it will be free of security vulnerabilities while being pretty much the same version you started with.
(Oh, and as a side-benefit, usually LTS package updates don't force system reboots. Recently even kernel updates don't, with fancy ksplice-based patch distribution channels. So LTS releases also mean long individual-host uptimes, which is handy if you're running a service on the cheap and can't afford much horizontal scaling to manage load-balancing under rolling updates, but want five or more 9s of availability anyway. Try getting that using immutable-infrastructure reboot-to-update VM images.)
How do you deal with this in production? Gems get abandoned. Some gems get upgraded faster than others. Some before you're ready.
You get into a race where you either need to fix your code, or fix a gem, or not upgrade. What do you do then? What if you don't have time to fix code to deal with a gem upgrade that ca happen at any time?
When things somehow go sideways I do the same thing: Triage the situation, including taking things offline if needed, make choices that solve the short term situation, run the tests, everything ok push live. Otherwise mitigate the issue by partially removing services (no image upload right now) and getting the rest of them live.
On the server itself I have a second copy of the repo. I install the new version of ruby in that repo and install the gems in that repo. I run the tests there and I check to make sure that Gemfile.lock hasn't changed. All good? Time to go live. Otherwise? Fix things locally first and re-push.
Rich Hickey has a great talk here https://www.youtube.com/watch?v=oyLBGkS5ICk
That said, doesn't work at all levels, but in general it's not nearly common enough in the ecosystem to keep compat consistent. When I think of libraries I love to use, I think mostly of libraries where compat is very well maintained.
Package management systems tend to be working against you here. Bumping major versions for "good" reasons makes it so that your biggest library users just get stuck, because they have so much code in the old paradigm. Real compat changes deserve new names/namespaces, so that you can progressively upgrade.
It’s not just a cost for maintainers.
A better strategy for namespacing would go a long way.
The clojure.set thing is crummy but a hard call. That's a super fundamental piece of the library puzzle, and write once, run forever is a really very expected thing there. Those are something that a wrapper can overcome pretty simply though. In a lot of those cases the approach they take does seem reasonable though, namely that e.g. clojure.set/union perform exactly as is documented on the documented inputs.
LTS helps administrators and integrators maintain reliability, which is very important in the perception of one's (at least our) users. More important than keeping on the bleeding edge.
In my case, the only additional effort needed for node 4 is to use Array.prototype.indexOf instead of Array.prototype.includes, which is mildly annoying, and sometimes I forget and it causes a failure of the node 4 CI build, but really it's not that big of a deal. TypeScript handles the rest of the JS stuff, and the project doesn't use enough of the node API for those differences to matter. Other projects may different, but I think the right call is to drop support for old node versions when maintaining them is hard or because nobody is using them, not simply because they're old.
I want to use open source projects at work, not just in side projects, and at work we're on node 6 and always lag behind the latest node release by some amount, partly because upgrading node is a real project. We're a startup, not a massive enterprise, but it's still pragmatic to focus on real work instead of upgrading our dependencies all the time. If all of the open source projects out there only supported node 8, I think it would make them much less useful in the real world.