Yes, nowadays it is very common for software to pin its version. This practice strikes me as questionable. One danger is that dependencies are not getting updated regularly and one day one find oneself depending on a very old version of a dependency. The other is that it may not always be possible/practical for other people to be at the that exact version. Some libraries can depend on system libraries that cannot be easily upgraded. E.g., numpy depends on underlying system libraries and it is not always possible to choose whatever version. GUI-libraries ultimateley depend on system GUI-libraries which also, practically speaking, cannot be updated to any random version.
One danger is that dependencies are not getting updated regularly and one day one find oneself depending on a very old version of a dependency.
That is only a problem if the old dependency no longer provides the required functionality, though. In many cases, that will never happen.
The other is that it may not always be possible/practical for other people to be at the that exact version. Some libraries can depend on system libraries that cannot be easily upgraded. E.g., numpy depends on underlying system libraries and it is not always possible to choose whatever version. GUI-libraries ultimateley depend on system GUI-libraries which also, practically speaking, cannot be updated to any random version.
Ironically, the problem you've described here originates with dependencies not having stable dependencies themselves. This is why it's so important for foundations like operating system or other platform APIs to offer excellent stability and longevity, and more generally for libraries that want to be relied upon to offer clear version information and backward-compatible updates.
But what if you never have to experience that pain at all? Or if you can experience it only once, much later, when it really does become unavoidable, instead of experiencing a similar amount of pain multiple times along the way?
Look at the sketch graph in that article, the one with time between actions and pain as the axes. Tell me what it means.
Nothing. It means nothing. It conveys literally no evidence or data of any kind. And the entire argument rests on that non-existent foundation.
Yes, and keeping up with these changes is a treadmill, for developers.
They also statically link binaries, though, and an old binary could be in production for a while if the product isn’t being worked on.
The popularity of Docker files shows another way of freezing dependencies.
The general principle here is that changes get tested during the release process. You’re not creating new combinations of dependencies and running them in production without testing.
(Well, within the same binary, at least. Servers talking to each other over the network may be unexpected combinations.)
But why should we take Google seriously as an example of good software development practice? For an organisation with effectively unlimited dev resources, it breaks things pretty often and kills off a lot of products outside of the star performers that support everything else. And thanks to the dominance of Chrome and Android, when Google does break things, the pain is shared by countless other developers as well. No-one can deny that Google has been a wildly successful business, but whether its current software development practices have anything to do with that or make it a worthy role model to emulate is a different matter.
It means that as time passes by, the things that need to be done accumulate and hence the pain of doing them. You don't need evidence for that. Its a common sense.
Otherwise, give one example when waiting for action (and in absence of divine/external intervention) makes it easier. Via this logic we can deduce that there must exist 100% self-fixing entities. I am not aware of any such thing. Many entities have some autocorrection (including us) but up to a certain point and usually (only?) when life is involved.
It means that as time passes by, the things that need to be done accumulate and hence the pain of doing them. You don't need evidence for that. Its a common sense.
I couldn't disagree more. It is not self-evident either that updates will inevitably need to be done or that the pain of doing them if they are needed will increase (never mind increase exponentially, as the article arbitrarily claimed) with time.
You should introduce an external dependency into your program if and only if it provides useful extra functionality and the benefits of using it outweigh the costs. You should update an existing dependency to a new version under exactly the same conditions.
Otherwise, give one example when waiting for action (and in absence of divine/external intervention) makes it easier.
When you never need to take that action at all, for example if you include a library in your program and the library does what you need and that's that.
OK, how about this: any work done has non 0 probability for mistake (now we can use math to deduce the same)
> You should introduce an external dependency into your program if and only if it provides useful extra functionality and the benefits of using it outweigh the costs. You should update an existing dependency to a new version under exactly the same conditions.
People are bad in judging such stuff hence this is tautology and provide no actionable advice.
> When you never need to take that action at all, for example if you include a library in your program and the library does what you need and that's that.
This doesn't qualify as you already said that "you dont need to take action". I said "when waiting for action" which implies action must be done at some point.
Distro package managers implicitly pin versions too - at distribution granularity - updated randomly by a bunch of strangers downstream with limited testing. The result is incredibly poor.
I deploy to consoles. I deploy to mobile. I deploy to the web via WASM. I deploy to windows as EXEs, MSIs, and APPXes. Your distro's linux package manager won't help me keep my dependencies up to date in any of these places. They won't even help me keep my dependencies up to date on linux, when the packages lag behind upstream - or in disconnected applications with no web access to update the linux distro with. How do I get security updates onto my OpenDingux RG350M? Through new images and newly built and pushed OPKs, not through fetched APT packages.
With versions pinned upstream at the app source code level, I can actually see what versions I've shipped had a vulnerable version, and push updates / advise my users appropriately. Stale dependencies are admittedly a concern, but they're not a concern that's even remotely fixed by having an unpinned, floating version. There's a reason software started bundling auto-updaters, and it's not because linux distros were handling everything perfectly fine already.
The article defends, among others, the Rust approach. Apps are pinned to exact versions via lockfiles. Libs are pinned to approximate verisons by semver, with CI typically catching when that breaks. This is a nice balance, and applies social pressure that encourages Rust projects to properly adhere to semver. Dependabot can flag dependencies with security issues, dependencies without security issues that are just old, auto-PR version updates to see if they pass CI or need manual action, and even auto-merge changes if you let it when the builds go green.
I've used this article as an excuse to double check my settings. I didn't have dependabot enabled for a couple of private repositories. I have a few flagged issues in an offline, user-inputless, electron clock widget I keep on my windows desktop. Harmless... but if they hadn't been? This would let me know I need to rebuild, retest, and redeploy. Via push notifcation. Despite the bundling approximating static linking. Despite the lack of linux. Despite the complete lack of an OS package manager, or even proper CI scripts. It's a much more versatile and portable approach.
It largely doesn’t matter because the future is sandboxed flatpak apps. So even if your notes app has an old dependency, it doesn’t matter because it has access to nothing but your notes.
The recipe for secure software is people doing the work, force-multiplied by cooperation and tooling. Safe languages, sane APIs, tests, CI, code reviews, audits, static analysis, dependency tooling, ease of forking when upstream is abandoned or unresponsible, easy build setups, good cross-platform support... the list goes on.
To hack on the average abandoned Rust codebase, "git clone ..." + "cargo build" alone probably just works, and I can start making, testing, and upstreaming incremental improvements one out-of-date or similarly abandoned dependency, bugfix, or feature branch at a time.
To hack on the average abandoned C codebase, I probably need new build rules and an undocumented list of extra build tools unique to this project (many of which almost certainly lack official windows binaries), and I'm probably left with a completely broken build that I can't even test my new build rules against - due to the unpinned versions causing multiple dependencies to be several major versions full of breaking changes out of sync with what the code expected. If I can be bothered to fix the build, the resulting megacommit will make any reviewer's eyes glaze over.
You can guess which of the two I'll bother trying to patch.
The "balkanized" downstream has been working quite well, unlike malicious packages on PyPI.
The only major issue was the OpenSSL fiasco that was due to overpatching upstream. Overpatching indeed should stop but is not a flaw of the package manager itself.
> The "balkanized" downstream has been working quite well
Disagreed.
> The only major issue was the OpenSSL fiasco that was due to overpatching upstream
There's a reason OpenSSL got forked as LibreSSL/BoringSSL - even downstream realized their approach wasn't cutting it, and they needed to go upstream and start burning everything down with fire. Granted, OpenSSL has been getting their act together - so perhaps unforking might be warranted?
Meanwhile, the CVE database continues to be flooded with memory vulnerabilities. Distros dutifully push out the patches when they get them, but that's the bare minimum.
> Distro package managers implicitly pin versions too - at distribution granularity - updated randomly by a bunch of strangers downstream with limited testing. The result is incredibly poor.
I think that points needs to be made more often.
People focus a lot on active decisions that have the potential to introduce security vulnerabilities, but they underestimate how similar problems may exist in the status quo.
Having a single, only-one-canonical version dependency system is a recipe for package managers that update slowly because any dependency change may break an unrelated package.
Frankly, being able to say "just install every dependency and be done with it" and have Cargo/npm/etc handle all the conflicts implicitly is a huge time-saver. I think people underestimate how valuable that saved time is, both in terms of adding features and maintaining security.
Downstream projects that only consume old versions of upstreams projects are deadweights to the upstream. They don't encourage, support, find bugs in, or send patches to the upstreams.
And they live is some delusion that this creaking garbage pile of mummified, unmaintained, full-of-known-vulns old versions they are building on are "stable".
We use a decade+ old version of log4net. It suits us just fine and I’m not even sure “vulns” are something we would need to care about in debug logging software.
I think “vulnerabilities” are always worth thinking through like this. If a YAML library you only use to read in a config file at startup has some buffer overflow bug...who cares? It’s usually painless to just bump the version, but just because a bug was fixed doesn’t mean you’re now insecure.
so says the library publisher. The product publisher says in counter:
> I don't care about your issues in the way you care about them, this lib does what my customers need it to and I'm happy about that. When it stops doing what my customers need, then I'll up it (read: probably never).
Its a discord. I don't think either party is wrong or either party is entirely right. They just have competing interests.
I think the point is that the build is “stable” not that the dependency is. The last thing I want when I’m implementing a feature is to have to deal with some unwanted, unrelated API change because a dependency was updated.
It's always an issue of externalizing costs. Everyone wants to do what benefits them in the short term, and have the environment maintained and supported by someone else.
A: Hey, let's build this cool graphical UI and add lots of features to our car.
B: And the cost of maintaining that OS, libs, dependencies for the next 15 years will be paid by whom, exactly?
A: Naw, we'll support for one year only. Maybe two.
B: And when there is a software vuln found in a 10 year old version that would allow an attacker to disable the steering and breaks while the car is moving, you wont issue a recall?
A: recall? This is software! Everything is provided without warranty or fitness of use, haven't you read the EULA? It's in ALL CAPS
Downstream projects that only consume old versions of upstreams projects are deadweights to the upstream. They don't encourage, support, find bugs in, or send patches to the upstreams.
You're making some big assumptions about the motivations of the upstream developers there.
For example, maybe the upstream developers make a useful library and sell it in some form to downstream developers who find it valuable enough to pay. Clearly your upstream developers have motivation in this case, and that motivation is to provide what the downstream developers actually need and will be willing to pay for.
As another example, maybe the upstream developers provide some platform that is how they make their money, and they offer a useful library that enables downstream developers to write software that runs on the their platform. Again, the upstream developers are motivated to provide what the downstream developers actually need.
If the upstream developers are relying on more social motivations, as in parts of the FOSS community for example, then you have a totally different set of incentives. Relying on libraries maintained by developers with those incentives might not be appropriate for downstream projects that require stability and longevity.
> For example, maybe the upstream developers make a useful library and sell it in some form to downstream developers who find it valuable enough to pay.
The article is saying pin stuff for as long as possible, like the guy replying here who says the version of the logging tool he integrated one time 10 years ago is all he will ever want.
Paid-for or FOSS, after his initial interaction 10 years ago, he is not going to *pay*, encourage, support, find bugs in or send patches to the upstreams. And that's the vast majority of users, me too when I look at the projects I uses and never contribute to.
> Relying on libraries maintained by developers with those incentives might not be appropriate for downstream projects that require stability and longevity.
That's a particularly sniffy approach to FOSS. You would have walked away from the now-dying Xorg back in the day? Gcc?
If people don't contribute to the things they are dependent on, there is no reason to expect them to still be around when they do want more from them.
From the FOSS project perspective these users do not directly manifest themselves in any useful way at all.
FOSS is great. A lot of people have contributed a lot of value to society through making code they wrote freely available. The world is a better place for it, and I appreciate their efforts when something they've made helps me.
None of that means it's a smart decision to build something important on a foundation that is likely to shift under you later.
If people don't contribute to the things they are dependent on, there is no reason to expect them to still be around when they do want more from them.
If you're relying on something where the developers' incentives are primarily about personal satisfaction and it's only a small team or solo developer then you have little reason to assume they will still be around anyway.
Unless some software has become big enough to be an established community project or someone is developing it professionally, relying on it as a long-term dependency might be unwise. More specifically, either it already does what you need (or you're confident you can patch it to do so yourself), or it might need ongoing development but you have confidence in the support/maintenance model behind it, or you're taking a risk by relying on it.
Take any engineering analogy you like. At some point you need to restrict changes in foundational components in order to create a robust product. It’s not practical to continually change the foundational components, especially when the builder is left with the responsibility for continuously integrating and testing the compatibility between those components.
Every website has the ground changing under it whenever browsers update to a new version.
This usually doesn’t break much because browser vendors are pretty conservative about changing features that are in wide use. I guess that’s one way of restricting changes to foundational components, but maybe not what you meant?
And as a result, the browser may never break existing APIs so JS is left with multiple ways to do the same things where certain ways are strictly worse than others. Also the web browser apis go under orders of magnitude more scrutiny than your average library.
> Every website has the ground changing under it whenever browsers update to a new version.
Every app has the ground changing under it whenever the kernel updates to a new version.
Yet somehow we can still run binaries from the 90s. The browsers are broken and need to be frozen. Not something that will happen because google, like microsoft before it, benefits from constantly breaking, changing and extending everything to not let anyone catch up to them.
Its not only about practicality or work involved either. You are replacing something that works and is battle tested, with something that is unknown (even if its just minor upgrade). That can lead to bugs that affect end users. Even if you have automatic tests, it still applies, as you can never test everything.
Because dependencies are not perfectly managed / versioned. When you release the end product app, you have 3 options:
- assume that your published version compatibility is correct and let the app fail for some library upgrade in the future
- publish a strict list of dependency versions and keep updating it
- publish a strict list of dependency versions and rely on old versions as long as possible
The first one results in unhappy users, second one in lack of distro packages because you rely on too recent deps, third one either in lack of distro packages because of too old deps or in unhappy users because deps don't work on their new system anymore. There's no way to satisfy everyone unless distributions themselves become aware of multiple levels of package management.
The problem is people depend on you because you provide X. You thought you provided Y. X was just a small side benefit. As a provider, how do you advanced Y without breaking X?
- supply your own dependencies instead of relying on the environment to provide them for you in ways you can't control
The ideal format for distributing software is a single, self-contained executable program far more often than parts of the developer community would like to admit. So many of these potentially serious problems just don't exist if you use a compiled language and you statically link a controlled set of dependencies at build time.
Obviously you still need to monitor your dependencies for essential updates and then issue updates of your own software to incorporate them when necessary, for example in the event of a security vulnerability being identified and fixed.
And obviously you may still depend on services provided by your host environment that you're communicating with but not incorporating directly.
But at least you always know exactly what your user will be running within your own program, you can test it properly before you ship it, and it won't then mysteriously change because of something they or some other software on their system did that you have no knowledge about or control over.
I maintain a batterie of libraries for my company. Managing dependencies is one of the hardest part even with internal only libraries. Our major pain-point is the dependency towards 3rd party software and dealing with semver. Semver is not really hard to understand it’s just hard to execute as the developer is executing the rules. So for instance if I create a library/plugin etc which wraps around a tool, would it be a breaking change when the software stops supporting a specific version of the wrapped tool even though the internal API stayed the same (this happens a lot for me with gradle plugins which suddenly decide to not being compatible with the version of gradle I use). I would say yes but for larger dependency trees this becomes a nightmare. For our internal unity3d libraries we decided to not count the supported Unity versions as part of the semver version. The other option would be to have special library packages for each version of unity similar to how the Spock testing framework[1] deals with groovy. For this option we don’t have enough engineers at my company.
There is also no way to really know what a breaking version is. Every single update must be tested to make sure it really isn’t breaking. I have had cases where a library fixed a bug and it turns out we rely on the existing behaviour so the bug being removed is a breaking change.
Yes behavior is the other huge problem. I cover this mainly with integration tests. And whatever wasn’t covered by these specs is not part of the API. But there lies the issue. What if the developer forgot to add a spec or worth the tests returned false positives because of test setup issues.
Or also one of my favorite issues are when dealing with rest APIs. They get flagged deprecated removed etc without remorse that it means that large parts of old software libraries are practically broken without actually being broken. Catching these means running active nightly builds etc if you don’t actively maintain the library on a regular basis. The switch from GitHub to use main as the default branch name for repositories brought a lot of issues with one gradle plugin I wrote because I hardcoded (yes it was my fault here) the value to be “master” which for years was the norm that every repo had one. In hindsight it was a clear oversight from my side because one could have removed the master branch before obviously.
The subtle nature of this topic drives this rule of thumb for me: Carefully consider each dependency you use, and if you need it. In Python, JS, and to a lesser extent Rust, it's common for articles to recommend dependencies to do things that can be done in a similar fashion without. This seems harmless at first, but leads to the problems described in the article, especially as the number of dependencies increases.
Another guideline: Avoid dependencies that have many dependencies of their own.
> Carefully consider each dependency you use, and if you need it.... Avoid dependencies that have many dependencies of their own.
Good luck with nodejs apps :-)
"Carefully considered EACH dependency recursively" is not actionable advice (and nothing worst then tautology).
I rely on automatic tests and penetration tests.
The actionable advice is more like: Carefully consider if you need dependency. When you choose to use them, don't spend more then a dozen of minutes analyzing them internally but rely on automatic tests.
Yeah, they are. Recursive checking of libs as suggested doesn't take 30 minutes, or otherwise nobody would complain. And what if you missed something because nobody checks thoroughly the library code in 30 minutes - one will certainly obfuscate any malicious procedure.
Also you can outsource the tests and its easy to do. Depending on the contract, they can even be held liable if you come to get owned via the method that should have been tested.
It's interesting that managed services in cloud environments are always latest version. Either these services will become outmoded or software that depends on them will have to be maintained in prod.
I don't know, as a software author myself I don't like the "modern" way of thinking. I like to write an application once, and then be able to update the dependencies and it will still work later, because the dependencies use a stable and sensible interface.
I remember writing a couple of apps in the early GTK 2 days, in PyGtk, and they would still run with the very latest GTK 2 version. Due to pythons dynamic nature, I could even get them to run on older GTK versions (when I needed to run on CentOS) by providing polyfills for the one or two missing methods.
I get that compatibility shifts the maintenance burden to the library developers but that complexity has to go somewhere, and I'd prefer it centralized.
> I like to write an application once, and then be able to update the dependencies and it will still work later, because the dependencies use a stable and sensible interface.
as a software user I really hate using an app and having its GUI randomly change even if the app wasn't updated, just because the linux distro changed some lib underneath
The reason one updates the distribution is to get new features or fixes. It would be a friggin disaster if only this program got the fixes, but not that program, so now they can't agree on the clipboard format, for example.
A lot of users don't want updates. They just want their software to work and do whatever they're using it for.
As a practical matter, software that is network-accessible or deals with data from outside sources may well need a security patch from time to time. Those updates are good for almost everyone, but by their nature, they should also be transparent to users.
It's a different story when you're talking about changing functionality that is visible to the user. Almost everyone hates it when UI elements like menus or toolbars or dialog layouts or shortcut keys get changed from what they're used to, particularly if there is no apparent reason for the change.
> A lot of users don't want updates. They just want their software to work and do whatever they're using it for.
>As a practical matter, software that is network-accessible or deals with data from outside sources may well need a security patch from time to time. Those updates are good for almost everyone, but by their nature, they should also be transparent to users.
My business servers typically run Debian Stable for exactly this reason. Most platforms, sadly, are not as reliable in this respect as Debian's.
In particular, big consumer platforms like Windows, iOS and Android have all been horrendous when it comes to conflating essential updates for things like security patches or catastrophic data loss bugs with arbitrary updates that change behaviour and move UIs around.
> A lot of users don't want updates. They just want their software to work and do whatever they're using it for.
True. But these updates often are what keeps the software working, as software doesn't usually live in complete isolation from the rest of the system. Not having to rely on the system dependecies helps in decoupling and being able to update in a more controlled fashion.
If you have to fear with every update that something is taken away from you, then maybe you are just getting your software from the wrong people?
I'm not sure I follow. You have to update to keep the software working when you update? Why not just not update by not updating and just keep using the software that works?
The point they are making is that the software generally don't run in isolation. It might be true in some rare cases, but often computer are connected to network and need to read files that weren't created on the same computer.
You can probably use a computer with software from 20 years ago, but don't expect the web to work well, and to be able to open files made with recent software for example.
>You can probably use a computer with software from 20 years ago, but don't expect the web to work well, and to be able to open files made with recent software for example.
There's no real reason why this can't be the case, or at least graceful degradation. The sole reason is because it's all ultimately a battle for control of your computer. Thanks, capitalism
Because ultimately you have no idea what you are doing, unless you spend your whole day tracking what is going on with the (often hundred) pieces that make up the software you are using.
There might be an error in the software you are using that malware or hackers can use to run abitrary code on your machine. Unless you keep your software up to date or manually make sure the updates are not for security purposes, those who not update increase their exposed surface.
Then imagine you don't update your software, but (because of security reasons) you update your operating system. Suddenly your (back then working) software finds itself in a changed environment. If you are lucky it works, but just as well it can stop working (or create new security holes because the software relied on something in the OS that changed).
I know people who e.g. run 10 year old Audio programs or stuff. But they do so on a computer with a 10 year old OS that is not allowed to connect to the internet. This is a good strategy.
For security patches - that is by far the most important issue, followed by (in order) non-security bug fixes and finally non-breaking enhancements. What is definitely not wanted are regressions, which, in my book, includes cryptic changes to the way one does a task, even if it is still possible.
And the first of these is, of course, a strong argument against bundling, while the last of these is the thing that most motivates bundling.
It is unfortunate that this is being presented in an us-versus-them way, when cooperation for the common good is needed.
Yeah but isn't one point of updating to get UI improvements?
If you install a new theme, I expect the UI of apps to change. I recall there were updates that allowed you to drag a window from the titlebar, add overlay scrollbars, get a macOS-like menu and so on.
(I might be showing my age here but I recall how excited everybody was when Windows 95, 98, 2000, XP came out, just because they looked fresher. You didn't need to force updates on people, people were excited to pay $200 to run the same apps on a fancier OS. I wonder when that changed.)
It's kind of astonishing to me that I can run software written 30 years ago in C without changing anything, yet I can't expect a node project written last week with the exact same functionality to work today without a huge test suite.
What amazes me more is that there are people actively arguing that this is a good thing. You see developers spend huge amounts of time chasing down obscure dependency issues and begging on forums for help with how library X works with version Y on platform Z, yet many of these same folks will come online and tell you that this is a great thing.
It's like many of us (and I include myself in this observation) are unable to step outside our own work and see how we actually spend our time. We become so obsessed on the goal that we lose track of how many new obstacles we put in our own way year-after-year.
I’m half convinced that developers self-inflict incidental complexity to justify their job at some level. And also because they have a poor technical aesthetic and don’t know that it can be better.
I caught myself thinking this as well. TBH, I feel the same about most professions, politicians in primis.
But if I have to think rationally "Never attribute to malice that which is adequately explained by stupidity"
Malice implies intent, which absent supporting evidence may be an unwarranted assumption. This is a form of attribution bias.
For that matter, assuming stupidity can be an attribution bias as well, as the action being judged may be random, ill-informed, or otherwise subject to additional context of constraints or circumstances we aren't privy to.
The alternative is asking dependencies to never ever make any breaking changes anywhere.
This is not sustainable long term. It forces library authors to grow APIs purely additively, which eventually makes them overcomplicated and full of traps that can't be fixed, e.g. look at X11 and its recurring drama of screensaver lock security.
Library authors have to eventually clean stuff up. Ossified applications have to eventually update. Semver just adds a way to coordinate it.
This argument is sensible, but given that applications outlive both hardware and operating systems, the alternative is users keeping whole operating systems around forever without updating them.
The Linux kernel is one opposite example. They are pretty hard line about never breaking userspace APIs. It is overcomplicated and full of traps though. For most end users it's still worth it.
The reason you took X11 as an example is because it's still around, all modernized APIs that leveraged broken backwards compatibility are long gone by now.
There is a solution to the additive API problem, but C and C++ do not allow it. The solution is to create separate modules like x11/v1, x11/v2, etc. which live in the same distribution. It is therefore permitted for module A to use x11/v1, module B to use x11/v2, and module C to use both A and B (you may need to also provide a shim between common types like a to_v1 function). In this setup x11_v2 probably exposes many of the same bindings as v1, but that is ok since it is low effort to wrap a function or to re-export a private module with the common bindings.
This doesn't work in C/C++ due to how the C ABI works. There needs to be an easy way to say that even though A and B both have a foo function they are not the same foo and a way for module C to use both versions of foo by renaming one of them in its source. This is the greatest sin of the C ABI in my opinion, not even the lack of overloads.
I think if you use dynamic linking, it is in principle possible for module A to use v1 internally, and for module C to use v2, and neither to know about the other. But you can't use global objects, and you can't share objects from the internal libraries between modules. And the dynamic linker on Linux at least tries to reuse symbols, so it doesn't work by default. This was a big problem in the Gtk2 / Gtk3 transition, where you could not use both in the same process. I think there is a flag to prevent symbol reuse, though.
This is a solved problem with symbol versions and symbol aliases, which are used extensively by libaries like glibc that care about providing a stable ABI.
For C++, GCC and Clang also provide the abi_tag attribute:
> The abi_tag attribute can be applied to a function, variable, or class declaration. It modifies the mangled name of the entity to incorporate the tag name, in order to distinguish the function or class from an earlier version with a different ABI; perhaps the class has changed size, or the function has a different return type that is not encoded in the mangled name.
This was introduced to allow mixing pre-C++11 and post-C++11 std::string and std::list implementations in one libstdc++.so but is more general.
The other solution is of course a new SOVERSION but that does not allow two different versions to be used in the same process as ELF symbols are a process-global namespace.
Coming from node world, global dependencies like in java and python are a total deal breaker.
In node same library in two different versions can coexist in different parts of the dependency tree and that's totally fine (there are some small exceptions). You give up some space but the gains are incredible.
In contrast, in Java I've got runtime errors because a library down the tree was importing some other library with the same package name.
Yeah it's not a "deal breaker" but I agree it's pretty annoying. Venvs in python kinda soften the blow on a global dependency level, but it could happen your project needs two conflicting dependencies
Yeah, maybe "deal breaker" is to strong. But I've had consumers of a library not being able to use it because a critical dependency (html parser) was not the same version as they had
This is one thing that I have always wondered about. Lots of app-development frameworks insist and die by the practice that is trying to be smart about common dependencies or global dependencies like Ruby gems/Bundler. In all of my experience, running "npm install" has always been a much easier affair than bundle install. The amount of time saved by npm's dumber approach has added much greater value than the hard disk space saved.
I'm trying to see what I'm missing about the two dependency packaging philosophies, but in practice npm is by far the best experience for a dev on the frontlines. Is it just hard drive space that is the problem?
In the case of Ruby, it's not really trying to be smart. Ruby loads everything into a single namespace, so loading multiple versions of the same dependency could cause classes, modules, etc. from version 1.1 to overwrite those already loaded by version 1.0. The dependency manager is constrained by the language. The same is true of most languages that I'm aware of, though Java can load multiple versions of the same class.
The biggest problem caused by using multiple versions of the same library is that object representations are not stable across versions. Let's say you use a library "foo" that creates and manipulates Foo objects. You want to serialize them to JSON, so you use the third-party library "foo-json". If you have foo@2.1.0 and foo-json depends on foo@1.9.7 then foo-serializers will likely fail to serialize the Foo objects created by your application. This is why npm supports peer dependency constraints, though usually they're invisible to you as a user.
There are also performance and cost issues associated with using multiple versions of each library:
1. Starting the application requires loading many of those library versions, so you will have higher start-up latency than if you only had one copy of each library.
2. If the target platform uses method-based JIT compilation (e.g. Java), each of those libraries will be compiled independently when it is used. This can cause the application to feel sluggish between the time it initially starts up and the time that all of the copies of the libraries on the hot path have been compiled.
3. The program needs to be loaded into memory to execute it. You're not only paying for more hard disk space, but also more RAM. The additional memory required can start to add up if you're running enough replicas of the application in production.
For many small applications the performance and cost issues aren't major concerns, but they can become problematic as the application grows.
OK, let me be the one to say it: majority of tools don't need any updates if they already work as designed.
You will say "what about security" but majority of apps don't need world level security, or any security at all. In large coorps for which I work exclusively big number of systems are older then decade without any source code change during that time and it is obviously good enough. I am just replacing the government e-banking system that worked 5 years without ANY change (source code seems to be lost). They are replacing it because it has bugs, not because it uses outdated dependencies (and we are talking here about state level service). I implemented babysitting side service that fixes the problems the original one makes as it detects them while we redo the entire stuff and we didn't hear any complaint after that.
People obsessing over security and purity should learn to let it go and get down to Earth - if you don't have a problem, there is no need for fix. We have tones of old inventions people still use unchanged today. Door locks come first to my mind. Boosting up internal security often reminds me of keeping all house rooms locked with the best lock ever, and nobody does that (if you do, your house is exception and has some special use).
Outdated deps are usually a problem in my experience when you need to redeploy some old work manually. That is one reason fat binaries are great.
And this is also the major reason that using cloud services is a bad thing - you can't just let it be for as much time as you want. Vendor will change API and you must adapt or GTFO. I have currently at least 5 services that work in government unattended around decade and are critical.
"If you don't have a problem, there is no need to fix it"
You really need to think a bit deeper about this. You obviously don't have much experience with security and you make a lot of assumptions here.
First of all, you seem to make the assumption that when an organization gets compromised, they will know about it. No one is going to change your desktop wallpaper to say "You have been hacked", like in the movies...
Which leads me to the second point. What valuables does your organization have? Do you have recipes for a special kind of glue that your researchers spent a few million dollars inventing? Would you want a bunch of foreign companies sitting in your network, siphoning all your hard work out for free?
And what about the customers that buy your product? Do they care if all your communication with them, and any issues concerning this product that should only be known internally is widespread?
>You really need to think a bit deeper about this. You obviously don't have much experience with security and you make a lot of assumptions here
No it's you who is making assumptions and operating on "it must be this way" sense.
Simply because a project has not had changes in a few years (or even many years) doesn't make the project insecure or bad. Can there be bugs or security issues? Or course, any software can have those, but merely change rate as you see on the project download page has no bearing on that.
> You obviously don't have much experience with security and you make a lot of assumptions here.
You obviously work in security and want to sell me bs.
> First of all, you seem to make the assumption that when an organization gets compromised, they will know about it.
No, you assume a lot about me while knowing 0. The bigger problem here is that you assume that people in corp care that there are hidden agents taking some data out. Let me open your eyes: they usually don't and ESPECIALLY if its not visible. If you replace their main site front page they will care.
> What valuables does your organization have?
Dude... go read my post again.
> Would you want a bunch of foreign companies sitting in your network, siphoning all your hard work out for free?
I don't care. Its up to my stakeholders, I don't work for my granpas company. WHEN they care, I do it. Mine is to inform. Theirs to decide. Guess what: they always decide against it, naturally, as it costs more otherwise. Security is a feature. It costs money. It can be theoretically done forever and take infinite amount of money. In last decade I had just 1 stakeholder requesting "unbreakable system" and we did the best we could.
Your main mistake seems to be to extrapolate your context to entire world. Context is everything.
"majority of tools don't need any updates if they already work as designed"
This is just about the dumbest thing I've read on HN. It's fascinating that you think of yourself as professional.
Like I said, most organizations have no clue what's actually happening in their networks - mostly due to people like you.
Have fun when the digital forensics team is dispatched to one of your sites that has had all its data lost to some crypto virus (because when that happens, they notice). And the forensics guys will write a report which outlines the reason why it happened. And the reason is nearly always outdated software.
Lots more software today is on the network than many people think. Even if network access is not required for an application’s functionality, the software may still do an update check to a remote server so that it can alert users to a new update available.
But we're talking about software that we are writing. If we have any dependencies that are connecting to the network without us knowing about it, we have a serious problem already!
I was responding to the grandparent post’s statement "A majority of the software I use either for my work or for my leasure time don't even open a socket".
I'm really surprised to see you're being downvoted for this statement.
You don't even need to have anything valuable in your company for security to matter because your customers' personal data is valuable to them, and by law you have to protect it in many places now.
A security breach must be publicized, and that's a black mark on your company whenever anyone is considering using it in the future. It doesn't even really matter how big that breach is, or what data was stolen.
But even worse, an unethical competitor might hack your data and erase it, leading to lengthy downtime or even permanently lost data.
That's most a server-side focus on security, but even client-side there could be things that allow hackers to mislead or damage your customers.
Libraries are updated constantly as old vulnerabilities are discovered and new methods of attack are invented. Leaving your OS or libraries un-updated is virtually guaranteeing that you end up vulnerable somehow.
The idea that if you don't know about a security problem, you don't need to fix it... Wow, it's just mind-boggling to me.
I suspect the difference of opinion is because those two people are making very different assumptions. One comes across as too dismissive of security concerns, suggesting security is not their problem unless management is bothered. The other comes across as assuming most software is used in ways where security problems are a major risk. Neither is realistic. Combine that with an unnecessarily hostile tone and some ad hominem, and it's going to attract downvotes on a forum like HN where civil discussion and logical arguments are both usually appreciated.
Government work is the worst example for quality.
A monopoly without accountability having terrible practices doesn't surprise me.
In all the private companies I worked either software was maintained or the feature wasn't in use anymore or the entire engineering team was about to be fired or the company itself had 2 months of life.
I agree on your point regarding security, closing every vulnerability is impossible or not economically sound for many realities.
> Government work is the worst example for quality. A monopoly without accountability having terrible practices doesn't surprise me.
Exactly, the rules here are different and there is monopoly involved. Accountability is dispersed. However, if you freck up here and it shows, you are in the news tomorrow. Good luck finding job after that. On company level, you can be banned for years in entire gov sector.
The big problem with that stance is however, that government usually doesn't do that work, it typically outsources to private companies that work on behalf of mega corps like Oracle, IMB etc. And they produce that "worst example of quality".
So, its not about the type of company its much more complicated.
> In all the private companies I worked either software was maintained or the feature wasn't in use anymore or the entire engineering team was about to be fired or the company itself had 2 months of life.
I am sure this is not the norm either. You say it like there is some kind of sensor telling you when "feature wasn't in use anymore" - for some domains it can be easy, for majority its not however.
Its understandable that people minimize the complexity of things involved here but that doesn't reflect reality.
> I agree on your point regarding security, closing every vulnerability is impossible or not economically sound for many realities.
Indeed. People act here like I said 0 security but I didn't - every professional worth its salt should do common "best practices", but going more then that needs serious thought, budget, planning etc. and it all depends on type of thing you do.
Or library and programming language authors could not break things. I've co-authored a few C libraries that have never broken their callers, over 15+ years of development. In C you can use symbol versioning for API changes you really must make, and the rest is about choosing a sensible API to start with and sticking with it.
But how can you really be sure you're not going to break something? There might be software that accidentally depends on timing, say, or the order of items in a list or some other undefined behavior that you assume you're free to change but software authors didn't realize was unstable. A lot of programming is done by trial and error rather than reading the manual.
We're careful to specify what is defined behaviour (basically what we document) and the rest is not. Of course bugs exist both in our libraries and our callers, and we fix those, and work with callers if they had depended on the buggy behaviour, but that's a genuine case where the caller must fix their code. Fixing a bug is miles different from unnecessary breaks in APIs.
Nothing wrong with having good intentions but if you fixing a bug breaks someone else's software then they still have to fix their software no matter who's fault it is. Repeat for all their other dependencies and they have an endless stream of annoying little new bugs to fix.
I work on software that has dependencies probably over a decade old. They have that long history of proving themselves to be perfectly good so there's no upside to updating them and only a tedious bug-hunting downside. One of them was already abandoned when I started using it but it still had an active community of users who agreed that it was perfectly usable without any more updates despite having the odd minor bug or limitation.
That's where I hope software can end up - "finished" then sit around being stable and useful until it becomes obsolete. Other than security, I think people have a psychological problem with knowing there won't be any more updates. They always hope the next update might fix the bug that bothers them or add the feature they want. But often that never happens and they're just being strung along on hope and dripfed a few treats every now and then.
The sad thing is that this isn't an either-or choice. Applications could provide a precise list of dependencies and ranges that they need. Distribution package managers could try to respect those as much as possible, but still have the authority to override a library or two due to security issues, in a centralized manner.
Everyone could be reasonably happy if both sides did some work to understand each other and support each other.
Actually, all software packaging problems today stem from software development practices. Software developers don't make languages, build tools, or dynamic libraries, that support compatibility with different versions of dependencies in a way that is widely compatible.
Right now, as a dev, you have to make your code work with one version of a dependency, and hope that the way you did it is widely compatible. But why is this the only way to do it? Because nobody has improved on this model yet.
It's actually very simple to fix. Glibc actually already supports loading a specific version of a specific dependency from a specific shared library version for each and every function. So you can build a single app that works with conflicting library versions. Just write your function against one version, and if there's a breaking change, write another version of the function. Then load the right one at run time. Now you support multiple versions of dependencies.
Once that becomes a common convention, package managers no longer need to restrict systems to only one version of a dependency, and we no longer have to deal with "bundling". Is this more work for the developer? Yes. But it also solves all the problems that affect developers, packagers, and end users, by allowing for a maximum amount of compatibility.
The alternative of course is for devs to just stop writing breaking interfaces, and to handle versions in those interfaces so breaking changes can be handled gracefully.
109 comments
[ 4.4 ms ] story [ 188 ms ] threadIt sure is a pain that occasionally a dependency breaks. One question that one can ask, though, is whether one wants to experience that pain sooner or later. I am thinking of https://martinfowler.com/bliki/FrequencyReducesDifficulty.ht... here.
That is only a problem if the old dependency no longer provides the required functionality, though. In many cases, that will never happen.
The other is that it may not always be possible/practical for other people to be at the that exact version. Some libraries can depend on system libraries that cannot be easily upgraded. E.g., numpy depends on underlying system libraries and it is not always possible to choose whatever version. GUI-libraries ultimateley depend on system GUI-libraries which also, practically speaking, cannot be updated to any random version.
Ironically, the problem you've described here originates with dependencies not having stable dependencies themselves. This is why it's so important for foundations like operating system or other platform APIs to offer excellent stability and longevity, and more generally for libraries that want to be relied upon to offer clear version information and backward-compatible updates.
One question that one can ask, though, is whether one wants to experience that pain sooner or later. I am thinking of https://martinfowler.com/bliki/FrequencyReducesDifficulty.ht... here.
But what if you never have to experience that pain at all? Or if you can experience it only once, much later, when it really does become unavoidable, instead of experiencing a similar amount of pain multiple times along the way?
Look at the sketch graph in that article, the one with time between actions and pain as the axes. Tell me what it means.
Nothing. It means nothing. It conveys literally no evidence or data of any kind. And the entire argument rests on that non-existent foundation.
They also statically link binaries, though, and an old binary could be in production for a while if the product isn’t being worked on.
The popularity of Docker files shows another way of freezing dependencies.
The general principle here is that changes get tested during the release process. You’re not creating new combinations of dependencies and running them in production without testing.
(Well, within the same binary, at least. Servers talking to each other over the network may be unexpected combinations.)
It means that as time passes by, the things that need to be done accumulate and hence the pain of doing them. You don't need evidence for that. Its a common sense.
Otherwise, give one example when waiting for action (and in absence of divine/external intervention) makes it easier. Via this logic we can deduce that there must exist 100% self-fixing entities. I am not aware of any such thing. Many entities have some autocorrection (including us) but up to a certain point and usually (only?) when life is involved.
I couldn't disagree more. It is not self-evident either that updates will inevitably need to be done or that the pain of doing them if they are needed will increase (never mind increase exponentially, as the article arbitrarily claimed) with time.
You should introduce an external dependency into your program if and only if it provides useful extra functionality and the benefits of using it outweigh the costs. You should update an existing dependency to a new version under exactly the same conditions.
Otherwise, give one example when waiting for action (and in absence of divine/external intervention) makes it easier.
When you never need to take that action at all, for example if you include a library in your program and the library does what you need and that's that.
OK, how about this: any work done has non 0 probability for mistake (now we can use math to deduce the same)
> You should introduce an external dependency into your program if and only if it provides useful extra functionality and the benefits of using it outweigh the costs. You should update an existing dependency to a new version under exactly the same conditions.
People are bad in judging such stuff hence this is tautology and provide no actionable advice.
> When you never need to take that action at all, for example if you include a library in your program and the library does what you need and that's that.
This doesn't qualify as you already said that "you dont need to take action". I said "when waiting for action" which implies action must be done at some point.
I deploy to consoles. I deploy to mobile. I deploy to the web via WASM. I deploy to windows as EXEs, MSIs, and APPXes. Your distro's linux package manager won't help me keep my dependencies up to date in any of these places. They won't even help me keep my dependencies up to date on linux, when the packages lag behind upstream - or in disconnected applications with no web access to update the linux distro with. How do I get security updates onto my OpenDingux RG350M? Through new images and newly built and pushed OPKs, not through fetched APT packages.
With versions pinned upstream at the app source code level, I can actually see what versions I've shipped had a vulnerable version, and push updates / advise my users appropriately. Stale dependencies are admittedly a concern, but they're not a concern that's even remotely fixed by having an unpinned, floating version. There's a reason software started bundling auto-updaters, and it's not because linux distros were handling everything perfectly fine already.
The article defends, among others, the Rust approach. Apps are pinned to exact versions via lockfiles. Libs are pinned to approximate verisons by semver, with CI typically catching when that breaks. This is a nice balance, and applies social pressure that encourages Rust projects to properly adhere to semver. Dependabot can flag dependencies with security issues, dependencies without security issues that are just old, auto-PR version updates to see if they pass CI or need manual action, and even auto-merge changes if you let it when the builds go green.
I've used this article as an excuse to double check my settings. I didn't have dependabot enabled for a couple of private repositories. I have a few flagged issues in an offline, user-inputless, electron clock widget I keep on my windows desktop. Harmless... but if they hadn't been? This would let me know I need to rebuild, retest, and redeploy. Via push notifcation. Despite the bundling approximating static linking. Despite the lack of linux. Despite the complete lack of an OS package manager, or even proper CI scripts. It's a much more versatile and portable approach.
For the record, I see both sides as having legitimate concerns.
Agreed.
> I suspect leaving security updates to upstream is not a recipe for secure distributions.
Agreed - but neither is leaving it to a balkanized downstream. Debian's getting picked on in another thread: https://news.ycombinator.com/item?id=26207965 .
The recipe for secure software is people doing the work, force-multiplied by cooperation and tooling. Safe languages, sane APIs, tests, CI, code reviews, audits, static analysis, dependency tooling, ease of forking when upstream is abandoned or unresponsible, easy build setups, good cross-platform support... the list goes on.
To hack on the average abandoned Rust codebase, "git clone ..." + "cargo build" alone probably just works, and I can start making, testing, and upstreaming incremental improvements one out-of-date or similarly abandoned dependency, bugfix, or feature branch at a time.
To hack on the average abandoned C codebase, I probably need new build rules and an undocumented list of extra build tools unique to this project (many of which almost certainly lack official windows binaries), and I'm probably left with a completely broken build that I can't even test my new build rules against - due to the unpinned versions causing multiple dependencies to be several major versions full of breaking changes out of sync with what the code expected. If I can be bothered to fix the build, the resulting megacommit will make any reviewer's eyes glaze over.
You can guess which of the two I'll bother trying to patch.
The only major issue was the OpenSSL fiasco that was due to overpatching upstream. Overpatching indeed should stop but is not a flaw of the package manager itself.
Disagreed.
> The only major issue was the OpenSSL fiasco that was due to overpatching upstream
There's a reason OpenSSL got forked as LibreSSL/BoringSSL - even downstream realized their approach wasn't cutting it, and they needed to go upstream and start burning everything down with fire. Granted, OpenSSL has been getting their act together - so perhaps unforking might be warranted?
Meanwhile, the CVE database continues to be flooded with memory vulnerabilities. Distros dutifully push out the patches when they get them, but that's the bare minimum.
I think that points needs to be made more often.
People focus a lot on active decisions that have the potential to introduce security vulnerabilities, but they underestimate how similar problems may exist in the status quo.
Having a single, only-one-canonical version dependency system is a recipe for package managers that update slowly because any dependency change may break an unrelated package.
Frankly, being able to say "just install every dependency and be done with it" and have Cargo/npm/etc handle all the conflicts implicitly is a huge time-saver. I think people underestimate how valuable that saved time is, both in terms of adding features and maintaining security.
And they live is some delusion that this creaking garbage pile of mummified, unmaintained, full-of-known-vulns old versions they are building on are "stable".
2 - Not everyone who has the bandwidth to write something also has the bandwidth to rewrite very often just to keep using the latest and greatest APIs
> I don't care about your issues in the way you care about them, this lib does what my customers need it to and I'm happy about that. When it stops doing what my customers need, then I'll up it (read: probably never).
Its a discord. I don't think either party is wrong or either party is entirely right. They just have competing interests.
A: Hey, let's build this cool graphical UI and add lots of features to our car.
B: And the cost of maintaining that OS, libs, dependencies for the next 15 years will be paid by whom, exactly?
A: Naw, we'll support for one year only. Maybe two.
B: And when there is a software vuln found in a 10 year old version that would allow an attacker to disable the steering and breaks while the car is moving, you wont issue a recall?
A: recall? This is software! Everything is provided without warranty or fitness of use, haven't you read the EULA? It's in ALL CAPS
You're making some big assumptions about the motivations of the upstream developers there.
For example, maybe the upstream developers make a useful library and sell it in some form to downstream developers who find it valuable enough to pay. Clearly your upstream developers have motivation in this case, and that motivation is to provide what the downstream developers actually need and will be willing to pay for.
As another example, maybe the upstream developers provide some platform that is how they make their money, and they offer a useful library that enables downstream developers to write software that runs on the their platform. Again, the upstream developers are motivated to provide what the downstream developers actually need.
If the upstream developers are relying on more social motivations, as in parts of the FOSS community for example, then you have a totally different set of incentives. Relying on libraries maintained by developers with those incentives might not be appropriate for downstream projects that require stability and longevity.
The article is saying pin stuff for as long as possible, like the guy replying here who says the version of the logging tool he integrated one time 10 years ago is all he will ever want.
Paid-for or FOSS, after his initial interaction 10 years ago, he is not going to *pay*, encourage, support, find bugs in or send patches to the upstreams. And that's the vast majority of users, me too when I look at the projects I uses and never contribute to.
> Relying on libraries maintained by developers with those incentives might not be appropriate for downstream projects that require stability and longevity.
That's a particularly sniffy approach to FOSS. You would have walked away from the now-dying Xorg back in the day? Gcc?
If people don't contribute to the things they are dependent on, there is no reason to expect them to still be around when they do want more from them.
From the FOSS project perspective these users do not directly manifest themselves in any useful way at all.
FOSS is great. A lot of people have contributed a lot of value to society through making code they wrote freely available. The world is a better place for it, and I appreciate their efforts when something they've made helps me.
None of that means it's a smart decision to build something important on a foundation that is likely to shift under you later.
If people don't contribute to the things they are dependent on, there is no reason to expect them to still be around when they do want more from them.
If you're relying on something where the developers' incentives are primarily about personal satisfaction and it's only a small team or solo developer then you have little reason to assume they will still be around anyway.
Unless some software has become big enough to be an established community project or someone is developing it professionally, relying on it as a long-term dependency might be unwise. More specifically, either it already does what you need (or you're confident you can patch it to do so yourself), or it might need ongoing development but you have confidence in the support/maintenance model behind it, or you're taking a risk by relying on it.
This usually doesn’t break much because browser vendors are pretty conservative about changing features that are in wide use. I guess that’s one way of restricting changes to foundational components, but maybe not what you meant?
Every app has the ground changing under it whenever the kernel updates to a new version.
Yet somehow we can still run binaries from the 90s. The browsers are broken and need to be frozen. Not something that will happen because google, like microsoft before it, benefits from constantly breaking, changing and extending everything to not let anyone catch up to them.
It is almost ironic, as linux is known for not breaking up the abstraction ladder.
- assume that your published version compatibility is correct and let the app fail for some library upgrade in the future
- publish a strict list of dependency versions and keep updating it
- publish a strict list of dependency versions and rely on old versions as long as possible
The first one results in unhappy users, second one in lack of distro packages because you rely on too recent deps, third one either in lack of distro packages because of too old deps or in unhappy users because deps don't work on their new system anymore. There's no way to satisfy everyone unless distributions themselves become aware of multiple levels of package management.
The problem is people depend on you because you provide X. You thought you provided Y. X was just a small side benefit. As a provider, how do you advanced Y without breaking X?
- don't have the dependency in the first place
- supply your own dependencies instead of relying on the environment to provide them for you in ways you can't control
The ideal format for distributing software is a single, self-contained executable program far more often than parts of the developer community would like to admit. So many of these potentially serious problems just don't exist if you use a compiled language and you statically link a controlled set of dependencies at build time.
Obviously you still need to monitor your dependencies for essential updates and then issue updates of your own software to incorporate them when necessary, for example in the event of a security vulnerability being identified and fixed.
And obviously you may still depend on services provided by your host environment that you're communicating with but not incorporating directly.
But at least you always know exactly what your user will be running within your own program, you can test it properly before you ship it, and it won't then mysteriously change because of something they or some other software on their system did that you have no knowledge about or control over.
Your JS fad of the day likely ain't even considering.
[1] https://github.com/spockframework/spock
[edit] spelling
Or also one of my favorite issues are when dealing with rest APIs. They get flagged deprecated removed etc without remorse that it means that large parts of old software libraries are practically broken without actually being broken. Catching these means running active nightly builds etc if you don’t actively maintain the library on a regular basis. The switch from GitHub to use main as the default branch name for repositories brought a lot of issues with one gradle plugin I wrote because I hardcoded (yes it was my fault here) the value to be “master” which for years was the norm that every repo had one. In hindsight it was a clear oversight from my side because one could have removed the master branch before obviously.
Another guideline: Avoid dependencies that have many dependencies of their own.
Good luck with nodejs apps :-)
"Carefully considered EACH dependency recursively" is not actionable advice (and nothing worst then tautology).
I rely on automatic tests and penetration tests.
The actionable advice is more like: Carefully consider if you need dependency. When you choose to use them, don't spend more then a dozen of minutes analyzing them internally but rely on automatic tests.
You mention penetration tests here (?), are those really less time consuming and more effective than taking 30 minutes to review a library?
Also you can outsource the tests and its easy to do. Depending on the contract, they can even be held liable if you come to get owned via the method that should have been tested.
While I agree with you, good luck with that if you are using node/npm.
I remember writing a couple of apps in the early GTK 2 days, in PyGtk, and they would still run with the very latest GTK 2 version. Due to pythons dynamic nature, I could even get them to run on older GTK versions (when I needed to run on CentOS) by providing polyfills for the one or two missing methods.
I get that compatibility shifts the maintenance burden to the library developers but that complexity has to go somewhere, and I'd prefer it centralized.
as a software user I really hate using an app and having its GUI randomly change even if the app wasn't updated, just because the linux distro changed some lib underneath
The reason one updates the distribution is to get new features or fixes. It would be a friggin disaster if only this program got the fixes, but not that program, so now they can't agree on the clipboard format, for example.
As a practical matter, software that is network-accessible or deals with data from outside sources may well need a security patch from time to time. Those updates are good for almost everyone, but by their nature, they should also be transparent to users.
It's a different story when you're talking about changing functionality that is visible to the user. Almost everyone hates it when UI elements like menus or toolbars or dialog layouts or shortcut keys get changed from what they're used to, particularly if there is no apparent reason for the change.
This is as good argument as any for allowing the dependencies to be updated without updating the main software.
>As a practical matter, software that is network-accessible or deals with data from outside sources may well need a security patch from time to time. Those updates are good for almost everyone, but by their nature, they should also be transparent to users.
That's a problem of the system you're using. I use Debian stable for my servers because you can only allow security patches: https://wiki.debian.org/UnattendedUpgrades
In particular, big consumer platforms like Windows, iOS and Android have all been horrendous when it comes to conflating essential updates for things like security patches or catastrophic data loss bugs with arbitrary updates that change behaviour and move UIs around.
True. But these updates often are what keeps the software working, as software doesn't usually live in complete isolation from the rest of the system. Not having to rely on the system dependecies helps in decoupling and being able to update in a more controlled fashion.
If you have to fear with every update that something is taken away from you, then maybe you are just getting your software from the wrong people?
You can probably use a computer with software from 20 years ago, but don't expect the web to work well, and to be able to open files made with recent software for example.
There's no real reason why this can't be the case, or at least graceful degradation. The sole reason is because it's all ultimately a battle for control of your computer. Thanks, capitalism
There might be an error in the software you are using that malware or hackers can use to run abitrary code on your machine. Unless you keep your software up to date or manually make sure the updates are not for security purposes, those who not update increase their exposed surface.
Then imagine you don't update your software, but (because of security reasons) you update your operating system. Suddenly your (back then working) software finds itself in a changed environment. If you are lucky it works, but just as well it can stop working (or create new security holes because the software relied on something in the OS that changed).
I know people who e.g. run 10 year old Audio programs or stuff. But they do so on a computer with a 10 year old OS that is not allowed to connect to the internet. This is a good strategy.
For security patches - that is by far the most important issue, followed by (in order) non-security bug fixes and finally non-breaking enhancements. What is definitely not wanted are regressions, which, in my book, includes cryptic changes to the way one does a task, even if it is still possible.
And the first of these is, of course, a strong argument against bundling, while the last of these is the thing that most motivates bundling.
It is unfortunate that this is being presented in an us-versus-them way, when cooperation for the common good is needed.
If you install a new theme, I expect the UI of apps to change. I recall there were updates that allowed you to drag a window from the titlebar, add overlay scrollbars, get a macOS-like menu and so on.
(I might be showing my age here but I recall how excited everybody was when Windows 95, 98, 2000, XP came out, just because they looked fresher. You didn't need to force updates on people, people were excited to pay $200 to run the same apps on a fancier OS. I wonder when that changed.)
It's like many of us (and I include myself in this observation) are unable to step outside our own work and see how we actually spend our time. We become so obsessed on the goal that we lose track of how many new obstacles we put in our own way year-after-year.
Malice implies intent, which absent supporting evidence may be an unwarranted assumption. This is a form of attribution bias.
For that matter, assuming stupidity can be an attribution bias as well, as the action being judged may be random, ill-informed, or otherwise subject to additional context of constraints or circumstances we aren't privy to.
This is not sustainable long term. It forces library authors to grow APIs purely additively, which eventually makes them overcomplicated and full of traps that can't be fixed, e.g. look at X11 and its recurring drama of screensaver lock security.
Library authors have to eventually clean stuff up. Ossified applications have to eventually update. Semver just adds a way to coordinate it.
The Linux kernel is one opposite example. They are pretty hard line about never breaking userspace APIs. It is overcomplicated and full of traps though. For most end users it's still worth it.
The reason you took X11 as an example is because it's still around, all modernized APIs that leveraged broken backwards compatibility are long gone by now.
This doesn't work in C/C++ due to how the C ABI works. There needs to be an easy way to say that even though A and B both have a foo function they are not the same foo and a way for module C to use both versions of foo by renaming one of them in its source. This is the greatest sin of the C ABI in my opinion, not even the lack of overloads.
For C++, GCC and Clang also provide the abi_tag attribute:
> The abi_tag attribute can be applied to a function, variable, or class declaration. It modifies the mangled name of the entity to incorporate the tag name, in order to distinguish the function or class from an earlier version with a different ABI; perhaps the class has changed size, or the function has a different return type that is not encoded in the mangled name.
This was introduced to allow mixing pre-C++11 and post-C++11 std::string and std::list implementations in one libstdc++.so but is more general.
The other solution is of course a new SOVERSION but that does not allow two different versions to be used in the same process as ELF symbols are a process-global namespace.
In node same library in two different versions can coexist in different parts of the dependency tree and that's totally fine (there are some small exceptions). You give up some space but the gains are incredible.
In contrast, in Java I've got runtime errors because a library down the tree was importing some other library with the same package name.
I'm trying to see what I'm missing about the two dependency packaging philosophies, but in practice npm is by far the best experience for a dev on the frontlines. Is it just hard drive space that is the problem?
The biggest problem caused by using multiple versions of the same library is that object representations are not stable across versions. Let's say you use a library "foo" that creates and manipulates Foo objects. You want to serialize them to JSON, so you use the third-party library "foo-json". If you have foo@2.1.0 and foo-json depends on foo@1.9.7 then foo-serializers will likely fail to serialize the Foo objects created by your application. This is why npm supports peer dependency constraints, though usually they're invisible to you as a user.
There are also performance and cost issues associated with using multiple versions of each library:
1. Starting the application requires loading many of those library versions, so you will have higher start-up latency than if you only had one copy of each library.
2. If the target platform uses method-based JIT compilation (e.g. Java), each of those libraries will be compiled independently when it is used. This can cause the application to feel sluggish between the time it initially starts up and the time that all of the copies of the libraries on the hot path have been compiled.
3. The program needs to be loaded into memory to execute it. You're not only paying for more hard disk space, but also more RAM. The additional memory required can start to add up if you're running enough replicas of the application in production.
For many small applications the performance and cost issues aren't major concerns, but they can become problematic as the application grows.
You will say "what about security" but majority of apps don't need world level security, or any security at all. In large coorps for which I work exclusively big number of systems are older then decade without any source code change during that time and it is obviously good enough. I am just replacing the government e-banking system that worked 5 years without ANY change (source code seems to be lost). They are replacing it because it has bugs, not because it uses outdated dependencies (and we are talking here about state level service). I implemented babysitting side service that fixes the problems the original one makes as it detects them while we redo the entire stuff and we didn't hear any complaint after that.
People obsessing over security and purity should learn to let it go and get down to Earth - if you don't have a problem, there is no need for fix. We have tones of old inventions people still use unchanged today. Door locks come first to my mind. Boosting up internal security often reminds me of keeping all house rooms locked with the best lock ever, and nobody does that (if you do, your house is exception and has some special use).
Outdated deps are usually a problem in my experience when you need to redeploy some old work manually. That is one reason fat binaries are great.
And this is also the major reason that using cloud services is a bad thing - you can't just let it be for as much time as you want. Vendor will change API and you must adapt or GTFO. I have currently at least 5 services that work in government unattended around decade and are critical.
You really need to think a bit deeper about this. You obviously don't have much experience with security and you make a lot of assumptions here.
First of all, you seem to make the assumption that when an organization gets compromised, they will know about it. No one is going to change your desktop wallpaper to say "You have been hacked", like in the movies...
Which leads me to the second point. What valuables does your organization have? Do you have recipes for a special kind of glue that your researchers spent a few million dollars inventing? Would you want a bunch of foreign companies sitting in your network, siphoning all your hard work out for free?
And what about the customers that buy your product? Do they care if all your communication with them, and any issues concerning this product that should only be known internally is widespread?
No it's you who is making assumptions and operating on "it must be this way" sense.
Simply because a project has not had changes in a few years (or even many years) doesn't make the project insecure or bad. Can there be bugs or security issues? Or course, any software can have those, but merely change rate as you see on the project download page has no bearing on that.
You obviously work in security and want to sell me bs.
> First of all, you seem to make the assumption that when an organization gets compromised, they will know about it.
No, you assume a lot about me while knowing 0. The bigger problem here is that you assume that people in corp care that there are hidden agents taking some data out. Let me open your eyes: they usually don't and ESPECIALLY if its not visible. If you replace their main site front page they will care.
> What valuables does your organization have?
Dude... go read my post again.
> Would you want a bunch of foreign companies sitting in your network, siphoning all your hard work out for free?
I don't care. Its up to my stakeholders, I don't work for my granpas company. WHEN they care, I do it. Mine is to inform. Theirs to decide. Guess what: they always decide against it, naturally, as it costs more otherwise. Security is a feature. It costs money. It can be theoretically done forever and take infinite amount of money. In last decade I had just 1 stakeholder requesting "unbreakable system" and we did the best we could.
Your main mistake seems to be to extrapolate your context to entire world. Context is everything.
I don't care. Its up to my stakeholders, I don't work for my granpas company."
Enough said.
What part of "educating stakeholder and let them decide" you didn't understand ?
This is just about the dumbest thing I've read on HN. It's fascinating that you think of yourself as professional.
Like I said, most organizations have no clue what's actually happening in their networks - mostly due to people like you. Have fun when the digital forensics team is dispatched to one of your sites that has had all its data lost to some crypto virus (because when that happens, they notice). And the forensics guys will write a report which outlines the reason why it happened. And the reason is nearly always outdated software.
you act like most software is on the network. But a majority of the software I use either for my work or for my leasure time don't even open a socket.
most of my day-to-day software is linux distro software which definitely does not do that except a few rare exceptions.
You don't even need to have anything valuable in your company for security to matter because your customers' personal data is valuable to them, and by law you have to protect it in many places now.
A security breach must be publicized, and that's a black mark on your company whenever anyone is considering using it in the future. It doesn't even really matter how big that breach is, or what data was stolen.
But even worse, an unethical competitor might hack your data and erase it, leading to lengthy downtime or even permanently lost data.
That's most a server-side focus on security, but even client-side there could be things that allow hackers to mislead or damage your customers.
Libraries are updated constantly as old vulnerabilities are discovered and new methods of attack are invented. Leaving your OS or libraries un-updated is virtually guaranteeing that you end up vulnerable somehow.
The idea that if you don't know about a security problem, you don't need to fix it... Wow, it's just mind-boggling to me.
In all the private companies I worked either software was maintained or the feature wasn't in use anymore or the entire engineering team was about to be fired or the company itself had 2 months of life.
I agree on your point regarding security, closing every vulnerability is impossible or not economically sound for many realities.
Exactly, the rules here are different and there is monopoly involved. Accountability is dispersed. However, if you freck up here and it shows, you are in the news tomorrow. Good luck finding job after that. On company level, you can be banned for years in entire gov sector.
The big problem with that stance is however, that government usually doesn't do that work, it typically outsources to private companies that work on behalf of mega corps like Oracle, IMB etc. And they produce that "worst example of quality".
So, its not about the type of company its much more complicated.
> In all the private companies I worked either software was maintained or the feature wasn't in use anymore or the entire engineering team was about to be fired or the company itself had 2 months of life.
I am sure this is not the norm either. You say it like there is some kind of sensor telling you when "feature wasn't in use anymore" - for some domains it can be easy, for majority its not however.
Its understandable that people minimize the complexity of things involved here but that doesn't reflect reality.
> I agree on your point regarding security, closing every vulnerability is impossible or not economically sound for many realities.
Indeed. People act here like I said 0 security but I didn't - every professional worth its salt should do common "best practices", but going more then that needs serious thought, budget, planning etc. and it all depends on type of thing you do.
I work on software that has dependencies probably over a decade old. They have that long history of proving themselves to be perfectly good so there's no upside to updating them and only a tedious bug-hunting downside. One of them was already abandoned when I started using it but it still had an active community of users who agreed that it was perfectly usable without any more updates despite having the odd minor bug or limitation.
That's where I hope software can end up - "finished" then sit around being stable and useful until it becomes obsolete. Other than security, I think people have a psychological problem with knowing there won't be any more updates. They always hope the next update might fix the bug that bothers them or add the feature they want. But often that never happens and they're just being strung along on hope and dripfed a few treats every now and then.
Everyone could be reasonably happy if both sides did some work to understand each other and support each other.
No wonder it works for developers.
Right now, as a dev, you have to make your code work with one version of a dependency, and hope that the way you did it is widely compatible. But why is this the only way to do it? Because nobody has improved on this model yet.
It's actually very simple to fix. Glibc actually already supports loading a specific version of a specific dependency from a specific shared library version for each and every function. So you can build a single app that works with conflicting library versions. Just write your function against one version, and if there's a breaking change, write another version of the function. Then load the right one at run time. Now you support multiple versions of dependencies.
Once that becomes a common convention, package managers no longer need to restrict systems to only one version of a dependency, and we no longer have to deal with "bundling". Is this more work for the developer? Yes. But it also solves all the problems that affect developers, packagers, and end users, by allowing for a maximum amount of compatibility.
The alternative of course is for devs to just stop writing breaking interfaces, and to handle versions in those interfaces so breaking changes can be handled gracefully.