When you statically link and executable, all the code from the libraries that it links to (at least the code that is directly or indirectly called by the application) is bundled into the binary. If you have 13 applications running that each statically link in 60% of a 2MB library, then you're using 15.6MB of memory where with dynamic linking you only need 2MB.
On the flip side, if you have a small tool that pulls in 1% of that 2MB library you only need 20KB of memory, whereas with dynamic linking you still need the full 2MB.
Ditto, static linking leads to a lot more optimization opportunities, both in terms of performance and in terms of size. I would be very surprised if that 13 x 60% of a 2MB library was actually anywhere near 15.6MB.
a) You are failing to account for the bloat caused by the presence of the dynamic linker, the dynamic loader, all the code that must operate under the assumptions of a dynamically linked environment, and the various auxiliaries used to treat shared library hell issues like WinSxS and libtool.
b) Most modern static linking implementations do things like sharing of text segments across processes and other dedup techniques, so they're not actually that bloated at all.
Are you saying that two Golang projects sharing the same (large, for the sake of the argument) dependency would not take (size of the dependency * 2) on i.e. the Ubuntu live CD (as was the example in the fine article)?
You have program X, and Y, both depending on big library L with 1000 symbols.
X uses 1 symbol l1 from L, Y use another symbol l2 from L.
With dynamic linking you need the whole library, no matter how much of it you use. With static linking (and --gc-sections -fdata-sections) you have parts of L compiled into X and Y binaries, but just the functions used (and their dependencies). It's possible that l1 and l2 share dependencies - and these will be duplicated. But still static linking is likely to be net win.
A statically linked binary can in principle only include code from the libraries it actually needs, so the overhead for some programs can be quite small and not as large as you might imagine. It also reduces the memory footprint at runtime.
I would not be surprised if those two Golang projects take less than the size of the dependency. 2x is a degenerate worst-case.
With static linking, you only need to pull in things you actually need (and you can optimize from there!), whereas with dynamic linking you need to pull in the entire dependency regardless. (You only need to pull in one copy regardless of how many things are using the library, true. But you need to pull in one full copy always.)
There are very few cases where you actually use the entire library in a project - and if you are indeed using an entire library in a project your project is necessarily large enough compared to the library that it's not much overhead percentage-wise.
One of the best features I found in Golang was the statically linkage concept. There is just to much trouble with tools that come either in dozens of versions for different OS versions (Ubuntu, Debian, Mint, CentOS .... all Linux, but no Linux is the same Linux ..., it goes on: Ubuntu 12.04, 12.10, 14.x, ..., Debian Sid, Debian with and without systemd ...)
While statically linking might add some bloat, I don't see, that the real bloat-factor is because of this. The real bloat-factor for example comes from programs that rely to much on template libraries (yes, I mean C++), that just unroll everything into new statements and because many even simple tasks have a multitude of solutions in the code ... each aside each other. When hundreds of developers are developing on one application, many of them invent the wheel again ...
> But doesn't static linking JUST link in the used features of the linked libs?
No, not if it works like it does in C and C++.
Lisp used to do that, back when Symbolics still existed; it was called tree-shaking, as in "shake the call graph tree to see what falls out". The main function calls functions FOO, BAR, BAZ, which call FROTZ and QUUX, which call... and remove everything not in that tree.
Of course, this gets to be equivalent to solving the Halting Problem the moment the EVAL function is called, so if the tree-shaker found any mention of EVAL anywhere in the code (probably after it had removed the provably dead code) it punted and linked in the whole Lisp runtime. It was the only viable solution, really, especially given how creative Lispers can get...
In C and C++ you can conservatively eliminate unused code for statically linked libraries using --gc-sections and -fdata-sections.
Go uses the Plan9 compiler architecture, which means that its linker knows more about the code and is responsible for more optimizations than its Unix equivalent, so it quite effectively eliminates dead code.
Of course, libraries are often highly coupled within themselves, so not much code can be eliminated, and then they might pull in additional dependencies, etc., etc.
The notion of 'premature optimisation is evil' is partly to blame; It's not a terrible rule, but it's much better with conditions applied. i.e. apply some intelligence/judgement to where you use optimisation rather than just lazily following some rule.
Fair point. I suspect that that subtlety to the rule is often lost to the reader. Perhaps a rewording is in order that puts emphasis on the conditional part (i.e. what constitutes premature)
Firstly, lack of optimization for software size or execution time is a form of optimization: it's optimization of someone's time.
Not writing a compiler for your scripting language saves you years.
Not choosing carefully what features to include in a program, or what packages in a system image: design time saved again.
It also saves considerable time if a user wants the program to do something, and by golly, "look, they thought of that already: there is a feature for it!" Seconds later, the user is just doing whatever they need instead of surfing the web for workarounds or calling support.
Having all the features also increases the flexibility of customization: you have more choices about what you can remove to create smaller images which have specific feature sets. You can hardly remove anything from a minimal image to make a specialized custom image.
I.e. by not making a program as minimal as possible, compact as possible or fast as possible, you are making some kind of trade off. Though the program is worse in some regard, there exist parameters (of something, not necessarily the program) which have gotten better: if you think about it a little, you can identify what those parameters are. The worsening in some of those parameters in a "bad trade" is what makes premature optimization "evil".
I agree with the general point, however, I believe it's commonplace for software to be poor on those other metrics you mention too, i.e. the one metric that /has/ been optimised is development cost. All of the optimal outcomes define a Pareto front, i.e. spend time optimising feature A versus speed, cost, etc. But as soon as you bring in low(er) quality engineers you're moving away from the Parento front, probably quite significantly so.
Premature optimization is evil -- when you don't have a design for how your system will work.
The axiom is only valid in those cases because if you've designed your system well, the points for optimization are already obvious and the code can be written mostly optimized the first time round.
I'm not really convinced. Leaving aside the argument about whether or not software is 'bloated' at all, or what 'bloat' actually means, this in particular stands out:
“wouldn’t it have been easier not to create the problem in the first place?” It would seem obvious that the answer to this is indeed “yes”.
It doesn't seem obvious at all, especially considering the implicit assumptions about whether or not bloat is actually a problem. There are three classic examples highlighted in the article of writing about this tradeoff: "Make it work, make it right, make it fast", "Premature optimization is the root of all evil", and "Bloatware and the 80/20 Myth" – but the argument about why these don't apply is a bit weak: "bloated software simply makes me sad".
To be honest, I don't find we've got a problem, generally. My experience with my computer is that I can do many more things much faster than I could in the past. And I don't think that counts as bloat.
> My experience with my computer is that I can do many more things much faster than I could in the past.
Bloat is an invisible problem, but that doesn't mean it doesn't exist. Less bloat is more storage space, more performance, more battery life, etc. And the problem is you can't judge "bloat" standalone in one piece of software, you need to judge it as the sum of all the software the user is running. In the first equation, your numbers will always tell you "it's not worth it", unless things are really bad. In the latter, not so much.
Maybe if the Bonbon Squash Series game I have on my android phone didn't link to liballthesocialthings, I'd shave 1MB off a game which is 50MB large. But maybe if every app on my phone did the same thing, I'd have 200MB more on my phone. My old HTC desire doesn't even have half that much memory total.
I came to believe there are two types of programmers - those who get emotionally upset about ineficciencies, and those who couldn't care less. I'm in the first camp, so I understand the feelings of the author perfectly.
You say your computer can do "more things much faster" than before - just imagine how many more things, how much faster you could do if the software didn't bloat itself up almost at the same pace as the hardware gets faster.
And you see, I can understand abstraction layers. I can excuse coding for function first, speed second. But then there's the kind of bloat that can only be explained by programmers having no clue about what their software is doing. There's literally no reason why an article should lag in your browser, or Spotify should take 10+% of CPU on standby. There seems to be a lot of pointless code that's being executed, and nobody cares, because developers often have powerful machines. Not so the general population who then has to endure such creations.
It's not always a business decision. GNOME, KDE, X, systemd, dbus... The list goes on and on. None of this software is business derived.
Honestly, an engineer that cares about the performance of their code has several options available:
- design the project first, then code later.
- design the code for the architecture (arrays of objects vs. objects of arrays).
- use tools that can optimize out inefficiencies.
- hand optimize after writing a functional slice.
- document the APIs and interfaces to explain the performance expected as part of their contract.
Unfortunately, it seems as though we've gotten into a bit of a rut when it comes to this -- we rely on the awful tooling we have to optimize the code we write (yes, LLVM and friends are light years better at this than GCC et. al., but it's still not enough), we never document performance characteristics of the libraries and APIs we create, and we (at least in the OSS world) almost never design our projects to the level of detail necessary to fully understand it.
Honestly, I'm not sure how we can solve these problems if the engineers have a lack of discipline. It's a hobby, after all, right? We do this in our spare time, and nobody wants to write designs or document because that's the least interesting part of coding. See also GTK's autodocs, which for /years/ lay around as nothing more than a simple enumeration of which functions were in the libraries. Even in self documenting languages we fail at this (I'm looking at you, Common Lisp).
Optimizing afterward is a simple, if lazy, way out but post-facto optimization can only take you so far if the design is simply broken.
Since we never document or design, the problem ends up falling to our tooling. Things like oprofile and so on work, but are usually so coarse grained that digging through the code to instrument it effectively becomes a grind nobody wants to do.
Maybe we need companies to develop free and open tooling for understanding the performance characteristics of the code we write. After all, the "modern" tooling we use for languages like C and C++ hasn't changed since the 80s, and it's often actually hostile to the end user, which just makes the job harder. Even our linkers often fail to do their jobs of optimizing out useless dependencies or symbols unless we explicitly tell them to do so.
Ultimately, the blame lies with us, the engineers who work on this code. New engineers joining our projects need to understand the code itself through and through. Our projects have become so large that without documentation no-one can make a clear, clean decision on what should or shouldn't be written anymore.
What I want to see is tooling that can semantically understand the code I write, and give me characterizations of the function calls I make as I write them. When I need to examine source to better understand it, I want tooling that tells me how often the functions are called, where they are called, and how. Raw text is great for writing prose, but we're not writing prose -- we're writing code. Our tools should reflect this, and give us as much help as possible.
Civilian comment: the Gnome/dbus/logind/systemd situation seems to have arrived as a result of a number of independent projects each on their own random walk but responsing to the earlier states of other projects that each depends on. So it may be overall integration issues (Linux based systems have always seemed to me as a user to be like bags of lego) rather than efficiency of any particular aspect of one project.
While I agree they're all attacking the same problems, ultimately they all have exactly the same problem when it comes to code: no designs, no docs, and no tooling.
The point you make about Linux systems being bags of Lego is interesting. That's exactly how it is supposed to be. Unfortunately, the projects we're talking about don't compose well -- they're more like Duplo than Lego because often times they subsume whole feature sets into one huge monolithic block. Huge lumps of code exist in GNOME and KDE and systemd and dbus that just simply have no business being in there.
Dbus and systemd especially fail at the whole design angle.
Go far back enough and you find corporations everywhere.
At the time of X's creation, DEC and IBM was looking for a GUI for their unix offerings.
And dbus came out of trying to make it easier to get the various Linux GUI toolkits to talk to each other, under the stewardship of Freedesktop (where also Systemd was housed until very recently). Thing is that RH was a big backer of that, and was out to win against the big unixes (A recent video interview with various employees there pretty much proclaim "we won the unix war(s)").
I'm one of these developers with powerful machines and I routinely kill both Spotify and Skype because they use obscene amounts of memory and CPU power for programs that are just idling. Just having a chat program and a music player open should not take over 400Mb of my RAM. I might have plenty of RAM, but I have no problem using it more productively.
>The usual counter to this is that the actual (as opposed to imagined) bottlenecks will only become apparent after intense usage.
The usual counter is that "optimising" takes time. And all of the time that you're spending trying to optimise before release is time that no one is able to use your software at all.
And the reality is that a lot of the bottlenecks can't be predicted in advance - who knows how may files the average user might want to load into your system, or whether your website is going to attract 100 or a million users a month? And many expected bottlenecks become largely irrelevant as technology moves on.
Take his Ubuntu example - for what percentage of users does the fact that it doesn't fit on an actual CD really matter any more? Is it worth spending weeks or months fine-tuning the distro to a point that it can be fitted onto a medium that most people probably aren't even using anymore?
I could edit a doc with a word processor that could fit onto a floppy, and it worked well. Now, why does the same task require a multi GB pos bloatfest to do the same job? And oftentimes, more slowly!
Imagine for a second, the power of today's hardware running optimised lean code. It could be so good, but we accept bloated crappy bug ridden shitfests of OS and application software. We seem to care more about glossy and social then performance.
It is a real problem.
How often do you need to edit a doc using a word processor that's running on a floppy? I'm guessing that it's not very often.
If you really need a lightweight word processor, there's plenty out there. They just probably won't have a lot of the features that the big hitters have now. And if you really want a lightweight OS, there's a fair few Linux distros aimed at that market.
What do you want your code to optimised for? Fast startup times? Functionality? Extensibility? Stability? Compatibility? Security?
None of those things come magically for free, and every day that a developer spends on one is a day that they aren't spending on another. And if they are waiting until any, or all, of those are optimised for any possible scenario, you'll be waiting a long time for any software at all.
As Blaise Pascal (or possibly one of many other possible candidates) once said - "If I had more time, I would have written a shorter letter." The challenge of optimisation v delivery is not a new one.
Because it is _not_ the same program. For one the spell checker is much better, for another it has added a grammar checker, for a third it now has change tracking and comments and lots of other goodies.
Don't complain that you don't need those features: it is on you to choose the right tool for the right job. If all you need is a light spell check and text editing, notepad++, vim or Emacs will do nicely. Plenty of people need the more advanced features.
> Take his Ubuntu example - for what percentage of users does the fact that it doesn't fit on an actual CD really matter any more?
That's only one thing among many mentioned, to prove a point.
The fact that it doesn't run well with only 2GB of RAM certainly affects plenty of actual and potential users. So the point stands. Not about the CD per se, but some optimizations are general and apply everywhere.
And every single Ubuntu user has complained at one point that pressing the Win/Super key does not bring the menu fast enough, or that the search results are not fast enough.
There's less feature-rich versions of Linux out there that will run comfortably in less than 2GB. If that's what you're after, use one of them.
Of course all software could be faster. It could also be more secure, more feature-rich, more user friendly, smaller and "better" in any number of other ways. But developers don't have infinite time to focus on all of them.
So they have to make tradeoffs, and the fact that most people default to the more feature-rich OSs rather than the lightweight ones, suggests that this is the place that the average user would rather the devs spend most of their time.
Which is easier? Re-running make, or having to write and induce another library to read configuration files?
Configuration files introduce a new point of failure (file not present, file corrupted, file not readable), a point of slowdown (how long does it take to read the file from disk, how long does it take to parse and load the file), and a point of complexity (parsing even ini files is pretty complicated, if you want to support the myriad of ways we write them).
All of this for values which change once every... how long? Once a year? Once an install? Or for most configuration values, never?
I still use configuration files, because I'm OK with including third party libraries... but I can certainly understand why some people may not be.
Parsing config files is not hard. A simple key-value flat file is often more than enough. You don't need external libs for it, but it does add additional complexity.
Suckless is about minimalism and speed, and if you want speed, you don't want extra junk like config files clogging up your code's mainline.
> Which is easier? Re-running make, or having to write and induce another library to read configuration files?
It's certainly easier for the programmer to just leave some of the work to the compiler, but it's at the cost of being a complete pain in the ass to packagers and users.
Do you even have to do much work yourself? .Xresources is already parsed and loaded for you by xrdb if you can't afford the cost of an extra 100 microseconds doing it yourself. Is the API to interact with that mindblowing horrid or something?
> All of this for values which change once every... how long?
When I'm configuring software to taste, several times a minute. And considering this is likely to be my first exposure to the software, it better not suck completely.
As a big fan of suckless software I can agree. While I understand the reasoning behind it, that configuration files are awful and introduce needless dependencies and make it too easy to leave a design decision up to the user as a config option rather than just picking the right choice for them, having to recompile software is a problem. The biggest issue is package management. I maintain local forks of some AUR packages for st and surf (the suckless terminal and web browser). It was a lot more work to set up than the single command that installing software should be, and I don't get automatic updates.
But if suckless maybe takes their fight against bloat too far, it's only because hardly anyone else seems to care at all.
However, that bloat is winning on its own merits. For instance, GNU/Linux distributions which are minimal are not popular. The popular ones tend to be the bloated ones.
Vim has gotten a lot bigger in 20 years, yet I'm not going to jump ship to a smaller vi implementation.
FOSS makes us confront the fact that we actually seem to like bloat. We inflict bloat upon ourselves --- there is no monopolistic software vendor there to blame.
Because bloat is like government. Multiple people might agree that there's too much of it, but they don't necessarily agree on which parts should be gotten rid of.
This. As always, there's a relevant xkcd[1]. Features that I might decry as bloat are probably someone else's "necessary, to be preserved at all costs" features. If you want to cover a large number of use cases, you have to include all of them. And with that comes bloat. There's a reason why even a minimalist CLI-only Linux installation requires more than 24MB of RAM when Windows 3.1 runs on 3MB - the Linux kernel is capable of a hell of a lot more.
A minimalist CLI-only Linux installation was still possible in around in 2 megs of RAM in 1994.
A minimalist XWindow installation was possible with a 40 megabyte hard drive and 4 megs of RAM over a 386 CPU. (That's similar specs to what 68K-based Sun workstations had perhaps a decade before that.)
Well, writing a text editor is far more complicated than it seems at first place.In order to support big files (50mb logs for instance), you need to make all your code rely on complex memory management, streaming and caching. Maybe js is not the right tool for the job in order to implement these features. I always thought that atom should be build in Qt with an api that can interact with JS,a bit like Adobe products for instance. That way you still allow JS plugins, just like Photoshop while getting a max of perfs thanks to C++.
Atom is a super weird example, wouldn't it bloat the software to add support for files larger than 2MB? Wouldn't that be feature bloat? I mean, a code editor shouldn't have to deal with gigantic files since code shouldn't be written that way <sup>[citation needed]</sup>.
One obvious use case for opening large files in a code editor are XML files. XML files can be huge but can profit a lot from the features of a good code editor.
Atom and Vim (and imo even Ubuntu) aren't good example of software bloat. While they might be slow and use a lot of memory, they're usually shipped with basic functionality and more functions can be added and removed at any time. Good examples are iTunes, Office, iOS and Android (the latter two are usually shipped with non removable bloatware).
In my (controversial) opinion, Atom is a great example. It includes/requires Nodejs, an embedded web browser, and runs Javascript, to run a rather basic text editor.
And in many cases, it's slow, CPU hungry, and memory intensive. Those three characteristics are the very definition of bloat to me.
I don't think Atom wants to be a basic text editor in the first place. It's shipped as a basic editor but the killer feature is that you can easily hack together extensions to suit your needs. It's a tradeoff: you pay for the html+js engine, but get a lot of potential developers.
That being said, having worked with Atom on a larger project I can't confirm it's slow.
> While they might be slow and use a lot of memory, they're usually shipped with basic functionality and more functions can be added and removed at any time.
Isn't that what bloat is? By definition, something that's shipped with "basic functionality" should not consume so much.
The question I ask is: could it be built more lightweight? While there is certainly room for improvement, I really don't think you can have an editor that can be extended by most web developers with much less resources. You could write your own backend engines for js/html but now your using/wasting manpower instead of machine power. The same goes for Vim or Ubuntu: what could you cut to easily make it slimmer without hurting their purpose? Not much imo.On the other hand, the examples I stated have certainly many parts you could outsource to extensions or cut alltogether. To me, bloat means stuff that doesn't serve the software's purpose, and I still don't think the examples in the article have much of that.
"Convenient though it would be if it were true, Mozilla is not big because it's full of useless crap. Mozilla is big because your needs are big. Your needs are big because the Internet is big. There are lots of small, lean web browsers out there that, incidentally, do almost nothing useful. If that's what you need, you've got options..."
That doesn't excuse resource inefficiency, which is really the main point. The issue of size can be further addressed by separation of mechanism and policy, and building for extensibility.
Are you sure there's inefficiency? Do you have specific points in the code or behavior which are obviously inefficient, or does it just feel "too big"?
The most obvious way it manifests is through progressive performance degradation, delayed response or output delivery times, locking up and an eventual necessity to kill/exec again for usable operation.
It isn't about "feeling big" at all. You can have large, feature-packed programs that still respond quickly and don't fragment.
There's tons of code in Mozilla that isn't necessary to do the one job the browser was built for.
I don't need code that manages bookmarks. At all. I don't need code that stores history for years. Honestly, I don't /want/ cookies persisted /at all/. I don't need themes to browse the web, and a reorganizeable UI is /never/ what I'm looking for in a browser.
I don't need WebGL. I don't need Canvas. Hell, I definitely don't need syncing of all this extra bloated data around to services somewhere in the 'net, either. Frames? What about floating frames?
Half this stuff is better left to tools that are built for it, the other half is junk I don't need, or is junk that actually creates more problems than it solves.
I think you misunderstand. The parent was asking for examples of bloated code. I was providing examples.
I actually /do/ use other browsers like Midori, w3m, and surf. The problem doesn't disappear, though, because it's up to all of us writing code to be better at it.
Not all code bases need to be minimal to be valid. Some people like having the features you class as superfluous. So long as you have the option to choose software that suits your taste there's not really much of a problem. I, for one, am glad that the web continues to evolve beyond its original design, clearly that's not something you're as interested in, both viewpoints are valid.
So my point here isn't that I feel those features are superfluous, but that that including them wholesale into the web browsing application directly is bloat -- it would be better served to produce separate applications specialized for those features.
Bookmarks, for instance, can be handled in a much cleaner and portable way than isolating them to the specific browser implementation. The same can be said for history, authentication, and so on.
Also, to be fair, I wasn't arguing for minimal codebases -- that was your point. I was providing examples for the GP.
I've idly thought of writing a 'manifesto for quality software', and/or trying to start some kind of political-esque movement to that effect.
I get annoyed by having to reboot my TV or radio when they freeze up, or by having delays to my channel changes, or having to navigate complex menus to do things that would have just been a physical button click before.
(this is all part of the same problem as bloat IMO).
This is where the likes of Apple or Dyson do well (or OK) I think. It's not necessarily that they're brilliant at what they do, it's just that the competition is pretty awful.
There is another side of extra resource use that I don't really see addressed except in the mobile space: ecology.
Even though my computer can run all applications without a hitch, it is still very wasteful to constantly use CPU power because of technology choices or plain laziness. As an example, Spotify and Slack are two applications that seem to use most of my CPU after Chrome. Spotify and Slack combined seems to hover around 5-15% of total CPU (a two year old i7). When there is a lot of traffic in Slack I have seen it using 15-20% by itself, with multiple processes running and memory use going above 200 megs.
Both applications work smoothly, but should they really use that much resources? A chat application? A music player? With modern CPU's I would expect them to be at the bottom of the process list when sorted by CPU usage. I used IRC on my Pentium 75Mhz and it ran fine. When simple applications are made so poorly that they use that much resources, what is the worldwide impact of that power use? And what about the users that don't have powerful and expensive CPU's?
A very good point, we should have a metric that establishes how much carbon is released extra into the environment from cpu cycle wasting crap like atom and even more bloated versions of word.
Clippy could pop up: "I can see you would like to release an extra ton of carbon into the atmos, why not upgrade word now!"
Apple has started doing something about energy efficiency, probably because most of their computers are laptops where energy use is quite important. OSX tracks energy efficiency, which somehow calculates how much power a single application uses (CPU + GPU if I remember correctly). But I don't think they do anything with that info at the moment.
I guess it would be a nice incentive for developers if OSX could notify you that an app that is on background is using a lot of energy at the moment, maybe even with a quit-button when on battery power. At least I wouldn't want my app to end up on that kind of a popup.
Yeah, and then we could see how small the differences would actually be.
A 15" Macbook Pro has a 99.5Wh battery and lasts about 8h; that's 12.5W. A program that decreases battery life by 20% would only mean an increase of 2-3W. Even if the machine and program are running 24/7 and all the energy comes from coal, that means an increase of less than 0.03T of CO2 per year. For comparison, the average carbon footprint of an US citizen is 20 Tons per year.
It's not that they have poor algorithms or whatever, it's that they're part native, to handle the desktop interaction and the rest is a bunch off html/css/JavaScript running in an embedded browser. At least that's how Spotify works.
The problem is that our tools for making multiplatform native GUI's suck so bad we'd rather just embed an entire Web browser into everything.
There are some okayish cross platform frameworks, such as QT and even JavaFX. One of the main complaints of cross platform GUI's has been that they don't work like native applications. But for some reason nobody cares when the app works like a single page web app, which in many cases is a lot worse than even plain old Swing apps. Which at least supports right-click properly.
I think the main reason that node-webkit and what-not are popular, is because of web developers moving to native app development. It's really easy to get started that way, and you can even share code with your web app. Where something like QT has a really huge learning curve for programmers transitioning from Javascript.
About poor algorithms. I actually worked on optimizing a well known web browser for a couple of years. And most of the stuff we did, was because of really bad Javascript code. Even though it seems gluttonous to embed a web browser in applications, and even insecure, it doesn't have to be as bad as it is, especially with a simple application like Spotify. This is going on a bit of a tangent, but every frontend programmer should at least learn how the browser actually works, a nice site for that is http://jankfree.org/
> There are some okayish cross platform frameworks, such as QT and even JavaFX. One of the main complaints of cross platform GUI's has been that they don't work like native applications. But for some reason nobody cares when the app works like a single page web app, which in many cases is a lot worse than even plain old Swing apps. Which at least supports right-click properly.
There's more to non-nativeness than just look and feel, which the more mature cross-platform GUI toolkits can ape fairly well. Another concern is accessibility for people with disabilities, e.g. blind people using screen readers. Qt, for example, is kind of accessible on Windows, Linux, and Mac, but not at all on mobile platforms. Not sure about the status of JavaFX. At least a single-page rich web app can be made accessible using the ARIA extensions to HTML, and if you use one of the big four web rendering engines, you can be sure they've done the hard work of implementing the underlying OS accessibility APIs well. Of course, many (most?) web developers don't implement ARIA for their custom UIs.
Accessibility is a point I didn't think about at all. Thanks for reminding me, it's really something that is all too often forgotten. JavaFX supports ARIA and all standard controls have accessibility built-in. But I have no expertise to actually comment on the quality of accessibility features in JavaFX.
> One of the main complaints of cross platform GUI's has been that they don't work like native applications.
wxWidgets applications work like native applications because they are in fact using the native toolkits, and some successful applications, like Audacity, are written using wxWidgets.
And it runs fast, as it is C++, as expected. Why it is not used more fits perfectly with the content of this article.
Hmm. Java went through this with AWT and Swing. You can have "identical on all platforms, nonnative, missing some native features and look&feel" OR "native features, look and feel, but different across platforms". Embedded browsers are closest to the former.
It sounds like solution (for spotify at least) is better methods for each platform of creating a GUI around web services. Microsoft have sort-of had a go at this.
The solution is to promote the browser to a full VM container, with both low level and high level APIs for graphics, sound, networking, and so on.
Browsers are already closer to this than most people realise. By the time you've included websockets, openGL, webaudio, and bunch of other stuff, you're 90% of the way to a useful OS.
The problem is that the current state of the web "OS" is a random collection of ad hoc APIs hacked together for a completely different job, and it's very badly designed for what it's trying to do now.
In a perfect world the FOSS world would collaborate to thrash out a new spec, and also design a new browser to follow the spec which was backwards compatible with standard HTML etc, but also included new and more efficient APIs for much faster performance.
Something like this has already happened on servers - see also, containerised VMs running RoR or Node or whatever you want - and it's only a matter of time before it happens in browsers too.
The problem is that the current plan seems to be to build a VM layer on top of the existing DOM/js/etc layer, which will kill performance even further. It should really replace it and emulate it.
As students we learn of the cpu/memory tradeof. In the real world it is engineering time/cpu/memory that is involved in the tradeof. Given how much money we make the business case often doesn't come close to existing for moving the tradeof to CPU time. Take Slack - they are adding tons of new users and their users love them, the company has a high valuation.
Focusing on getting that percentage down to nothing would have torpedoed their company.
On the one hand, I wonder how much of the wasted CPU cycles can be attributed to carelessness that could easily be avoided, or caught early, if we developers deliberately used underpowered hardware for our own machines. Speaking for myself, my main workstation, where I also do most of the testing on my desktop apps, has a Core i7-4770 processor and 32 GB of RAM. Maybe if I used something more modest, like an ultrabook with only 8 GB of RAM, I'd be more likely to notice when I'm carelessly writing inefficient code.
On the other hand, as others have argued both on this thread and elsewhere, there's a trade-off between machine efficiency and developer productivity. We may argue that it's wrong to waste machine resources when the machines in question don't belong to us, but then again, developer productivity means we can crank out more features that drive sales and make users happy.
I've previously decried bloat too, but I find that my rants against software bloat are based on emotion, not reason.
So, looking at this rationally, consider your favorite lightweight window manager or desktop environment for Linux. Is it fully internationalized, including support for CJK input methods? Is it fully accessible to users with disabilities, e.g. blind people and people with mobility impairments? Does it auto-mount USB thumb drives? Does it do all of the other things I'm not even aware of that are required for a fully usable desktop environment covering a wide variety of machines, users, and use cases? AFAIK, the only Linux desktop environments that come close are GNOME and Unity.
Basically, the world is complicated, so real-world software has to be complicated too.
It sounds like Atom could still use some optimization though.
Is it fully internationalized, including support for CJK input methods?
i18n is an orthogonal service that the DE or any other program may call into.
Is it fully accessible to users with disabilities, e.g. blind people and people with mobility impairments?
Screen readers and screen magnifiers are orthogonal services that the DE may make accommodations for, though some DEs may be lesser fit for users with disabilities in general.
Does it auto-mount USB thumb drives?
This isn't even the job of the desktop environment, it's the job of a dedicated device node manager, which the DE may call into. It's completely orthogonal and can be accomplished even without so much as a display server, because it's a different layer.
Not sure if device node managers are the right place either, as not every mount is related to a local device.
NFS in the classic cross-network *nix mount. And depending on available software (FUSE and related) you can mount the likes of SMB/CIFS shares, FTP servers, even SSH/FISH can be used via a mount.
NFS grossly violates POSIX file system semantics, so it's better to use something like 9P/Styx, instead.
Unix systems really aren't good at abstracting local and remote mounts and binds into a singular resource, but the parent poster's scenario was implied to be a local user hotplug, which is what device managers largely do (if even that, udev these days just maintains the /dev/disk/* symlinks and handles query requests for the hardware database).
it's better to use something like 9P/Styx, instead
Who does this, though? Are there reliable adoption numbers? NFS semantics are wrong in well-known, familiar ways and it ships everywhere, making it still quite popular.
The real problem with NFS is that NFSv3 is easy to implement but insecure and has scalability issues (e.g. users can be in no more than 16 secondary groups) while NFSv4 is complicated and requires a complete Kerberos infrastructure (which is more complicated than rocket science!) to utilize in a way that works around the limitations of NFSv3.
There's much better network file systems but because of many factors (boat/complexity being the biggest) their adoption is minimal.
Once you move beyond single-user/simple read/write permissions there's really no easy to avoid complexity. It's just the nature of the beast that is sharing files over the network in a way that appears native to the user.
The real problem is precious snow flake developers who do not care about bloated code, who think it becomes the dev ops problem to go and get bigger aws instances, or newer macs. Shit attitude to your work will always show. If engineers, plumbers, electricians, doctors, chefs - basically any other field of employment took the same cavalier approach to the end product, we would right castigate them. But somehow, we give devtards a free pass. Why?
> AFAIK, the only Linux desktop environments that come close are GNOME and Unity.
I'm the project lead of LXQt. All those things you listed can be supported without introducing much bloat at all. In DEs, most bloat comes from pure technical debt. Unnecessary libraries, duplicated code/libs, etc.
Feature bloat is another matter and is more subjective. A lot of it is down to UX.
Fun fact: KDE 4.x depends on ruby in most distros. Do you know why? Dolphin ships with a ruby script to update some folders (don't remember what it does). Nobody runs it. That's it.
> Fun fact: KDE 4.x depends on ruby in most distros. Do you know why? Dolphin ships with a ruby script to update some folders (don't remember what it does). Nobody runs it. That's it.
Why did they allow an additional dependency on Ruby in? It seems like perl or python would be the more logical choice, given they'd already be on the system (heck.. maybe bash?)
Bash would have been more appropriate, yes. I don't think the author of the script even thought it was going to be run by users, but some debian package checker probably flagged ruby as a dependency and there it was.
A script that is compatible with POSIX Bourne shell (/bin/sh) would be more appropriate. FreeBSD, for example, does not come pre-installed with bash. If you want cross-platform portability, this is a better default.
Just make sure your use of the light version won't impact anyone else who would mind the missing features.
For example, suppose you're developing an application, and you want to use a lightweight GUI toolkit, or roll your own as Spotify apparently used to do. Unless the toolkit is just a thin wrapper over the underlying OS's native GUi facilities, like wxWidgets or SWT, the resulting application is very likely to have accessibility problems, unless you do the hard work of implementing the accessibility APIs for your target platform(s).
Only experience with the same problem space can make you accurately predict where a bottleneck is going to be. Premature optimization will have you focusing on areas where optimization isn't required. YAGNI applies to all levels of development.
I would rather see bloat and new ideas, than only experts working in a problem domain. My best ideas come from a marriage of understanding a single problem space, and a wonder what I can do in another. Bring on the bloat.
Once it installed from the floppy, why would I need to keep using from the floppy?
You also described caching behaviour of the OS, rather than the huge size of modern day word.
Over time, all software tends to get bigger and not necessarily better.
Windows went from 7MB required (95) to 6GB required (Vista), 1000x bigger, in just 10 years. It's true in file formats as well and verbosity is one of the reasons XML was/is so hated.
There is an irresitable urge to make things bigger bigger bigger, and huge cpu, mem, and disk resources are essentially free.
Feature bloat is inevitable if you write an application in which thousands of users want different features. Most users want just a subset of the features, and think of the rest as "bloat" even though it might be a core feature for another user. The alternative is multiple lighter applications which would just reduce bloat at the cost of integration, a trade off almost no one seems to want.
I am not sure we need to go as lean as hand-rolled asm, but we sure went down a wrong path when we just accept that word will get bigger and bigger with each release. No one asks why.
I think so too. It's not about ASM snippets; it's about people not knowing what their software actually does. Software bloat isn't magic, nor is it static; your CPU is doing something. If with time application gets bigger and slower, and yet no extra features appear to explain it, it means your CPU is running code that shouldn't be run, wasting its cycles on pointless and irrelevant computations. That's laziness and/or stupidity.
It's not laziness or stupidity (well, it might be sometimes, but certainly not always).
It's using your time to do something else, such as improving security, or adding a new feature. Some people seem to be talking as if it automatically takes the same amount of time to create efficient code as inefficient code, and the idiot developer has simply forgotten to add the -run_faster option on the compiler.
I'm involved in a project at the moment where a team of developers and engineers have spent much of the last month trying to get response times in some very specific circumstances in an application down to an acceptable level. That's not because they are either stupid or lazy. It's because these things are hard - there's an awful lot of moving parts, and an awful lot of things that can go wrong in each of them.
When they get it down to an acceptable level, they'll ship it to the users. It won't be blazingly fast, but it'll be good enough. They could definitely spend longer tuning it further and getting it going even faster. But every day they do that will be a day longer until the features that the users want are actually available to them.
Perhaps too many moving parts. Engineers are criticized for 'reinventing the wheel' and writing code when open source is available. But use too much open source, and you have too many moving parts, you don't know how they work, and things like locks and latency are uncontrollable. A familiar storey!
Apart from the OS (which isn't where the problem lies), there's no open source in it.
The reason there's lots of moving parts is because modern computers, OSs, languages, application servers, libraries, databases etc are built to do a lot of very complicated things.
We could have written the entire application from scratch, not using any 3rd party elements so that we had full control over every element. But that would have taken several orders of magnitude longer to deliver actual useful working software to the users.
Or, to be non-Aristotelian about it, some parts could have been dispensed with and appropriate code written for the purpose, and other more robust parts could be kept.
The opposite of that. Using somebody else's code, written for a different app in a different environment, is where most unexpected behavior comes from.
Here's the bits that aren't robust enough: any open source that isn't either used by thousands, or used in exactly the way you will be using it.
I would still be mostly using a 3rd part application framework if I rewrote some bit of it that I found problems with. I wouldn't be rewriting the whole lot - that would take forever. So I'd be left with a 95% of the framework being written by the 3rd party and 5% written by my team.
That 95% would still need to interact with my new 5%, and as it was never designed to do that, it could easily introduce new unexpected behaviour. Imagine trying to rip out (for example) the indexing code out of a database because it's performing badly and then rewriting it.
Imagine having control over the behavior of your code! That's the advantage of writing it. And I thought we were talking about replacing open source packages, not doing open-heart surgery on a faulty package?
Why would you think that? I explicitly said "Apart from the OS (which isn't where the problem lies), there's no open source in it."
It's a 3rd party commercial platform for developing your own applications on. To rewrite individual bits that aren't working efficiently would be a nightmare.
As for imagining having full control over the behaviour of my code - I had that quarter of a century ago or more when I started coding, and spent weeks or months building windowing systems and low-level networking code in assembler that I could achieve in 30 seconds with any off the shelf language these days.
Yes, my code was extremely efficient (it had to be, trying to do graphical comms software on 8086 machines didn't give you much of an option to do otherwise), but the use of my time wasn't.
The third party product that we're using is the result of many years of a large team of dedicated developers. Rewriting that from scratch in order to make sure we've got full knowledge of what's going on is an utter ludicrous idea.
That's probably because you have SSD, but anyway; I could do the exact same process 10 years ago (with "winword" instead of "word"), get similar response times, and yet the software was an order of magnitude smaller and more responsive. And it's not like Word gained many actually useful features during that time.
>> And it's not like Word gained many actually useful features during that time.
Then use the 10 year old version! Unless, of course, one of those few "actually useful features" is something you can't live without.
A lot of people use Word for a lot of use cases. The value of an added feature that someone needs always trumps the performance cost of adding it until the performance becomes so bad that it becomes the reason that other people stop using it.
People who complain about bloat are usually complaining about features they don't use. Of course watch what happens when people start doing metrics and optimizing their UI for "common uses" - turns out you're optimizing for non-existent users (see: the Ribbon).
The ribbon gets knocked a lot; but the median word processor user has zero experience. The ribbon gives them a fighting chance to find what they need. At the expense of 'expert users' with their idiomatic expectations.
Which is a problem because tools should be just that - tools. You don't make welders or soldering irons easy for people with zero experience. You make them effective and efficient tools, and then train people to use them. Heck, show me a single musical instrument giving zero-experienced users "a fighting chance to find what they need".
It's weird how new trends in UX design try to make a first-time user become a genius immediately after double-clicking on the program icon. The only way you can do that is by dumbing down the software to the point it can actually be comprehended this way - which makes it much less usable and effective as a tool.
That's the expert talking. The tool is made for the most common case - a new hire meeting it for the first time. They're not gonna do a good job; but anything that can improve their performance, pays. Follow the money.
As someone who is also saddened by software bloat, I also wondered a bit about where it stems from. There are of course the usual suspects (some of them mentioned in the article), but I think it boils down to another symptom of the modern rat-race culture, and it is not limited to software. When the goal of creating beauty and enjoying (as well as taking pride in) the creation process is replaced by the god of money and getting it out the door as fast as possible, the quality of the thing you're getting out the door has to deteriorate. I'd argue that a side effect of this is that the quality of life of both the creator and the receiver of the product decreases, but that is not immediately apparent when you're too busy trying to win a rat race...
However, I'm optimistic. For one, I think that the tools we have today are superior in absolute terms (convenience, speed, ease of use etc.) than they ever were. Of course they are not exponentially better (like the hardware they run on), but they are better. Also, occasionally I see some gems here and there. There are still many developers who take pride in what they do and I enjoy finding that diamond once in a while when searching for an alternative technology to use in a new project.
I don't care if an app is 20% bigger in file size as long as the user interface makes sense and the product actually works, especially if the bloat is due to a programming language or architecture that makes it easier to add new features or fix bugs.
But what if it's 200% bigger? I remember reading an article that looked at the time it takes to load MS office under Windows. Even though computers over the last 15-20 years or so are much, much faster, the wall-clock time to get Word or Excel up and running has pretty much stayed constant. Granted, disks haven't kept pace with CPUs, but when your computer is 100 times faster and it still takes just as long... Is that a good trade off?
161 comments
[ 2.6 ms ] story [ 232 ms ] threadWhile I really like the language itself, statically linking everything is creating exactly what this article describes.
Burn karma, burn. I don't care.
Ditto, static linking leads to a lot more optimization opportunities, both in terms of performance and in terms of size. I would be very surprised if that 13 x 60% of a 2MB library was actually anywhere near 15.6MB.
a) You are failing to account for the bloat caused by the presence of the dynamic linker, the dynamic loader, all the code that must operate under the assumptions of a dynamically linked environment, and the various auxiliaries used to treat shared library hell issues like WinSxS and libtool.
b) Most modern static linking implementations do things like sharing of text segments across processes and other dedup techniques, so they're not actually that bloated at all.
Are you saying that two Golang projects sharing the same (large, for the sake of the argument) dependency would not take (size of the dependency * 2) on i.e. the Ubuntu live CD (as was the example in the fine article)?
X uses 1 symbol l1 from L, Y use another symbol l2 from L.
With dynamic linking you need the whole library, no matter how much of it you use. With static linking (and --gc-sections -fdata-sections) you have parts of L compiled into X and Y binaries, but just the functions used (and their dependencies). It's possible that l1 and l2 share dependencies - and these will be duplicated. But still static linking is likely to be net win.
Dynamic linking is mostly about maintanance.
I would not be surprised if those two Golang projects take less than the size of the dependency. 2x is a degenerate worst-case.
With static linking, you only need to pull in things you actually need (and you can optimize from there!), whereas with dynamic linking you need to pull in the entire dependency regardless. (You only need to pull in one copy regardless of how many things are using the library, true. But you need to pull in one full copy always.)
There are very few cases where you actually use the entire library in a project - and if you are indeed using an entire library in a project your project is necessarily large enough compared to the library that it's not much overhead percentage-wise.
Thanks for the insight, I now understand why my comment was misguided.
One of the best features I found in Golang was the statically linkage concept. There is just to much trouble with tools that come either in dozens of versions for different OS versions (Ubuntu, Debian, Mint, CentOS .... all Linux, but no Linux is the same Linux ..., it goes on: Ubuntu 12.04, 12.10, 14.x, ..., Debian Sid, Debian with and without systemd ...)
While statically linking might add some bloat, I don't see, that the real bloat-factor is because of this. The real bloat-factor for example comes from programs that rely to much on template libraries (yes, I mean C++), that just unroll everything into new statements and because many even simple tasks have a multitude of solutions in the code ... each aside each other. When hundreds of developers are developing on one application, many of them invent the wheel again ...
Wouldn't that actually be better?
Then I don't have 5 versions of a huge dynamic lib for 5 programs that all use slightly incompatible version and that only use 1 or 2 methods.
No, not if it works like it does in C and C++.
Lisp used to do that, back when Symbolics still existed; it was called tree-shaking, as in "shake the call graph tree to see what falls out". The main function calls functions FOO, BAR, BAZ, which call FROTZ and QUUX, which call... and remove everything not in that tree.
Of course, this gets to be equivalent to solving the Halting Problem the moment the EVAL function is called, so if the tree-shaker found any mention of EVAL anywhere in the code (probably after it had removed the provably dead code) it punted and linked in the whole Lisp runtime. It was the only viable solution, really, especially given how creative Lispers can get...
Go uses the Plan9 compiler architecture, which means that its linker knows more about the code and is responsible for more optimizations than its Unix equivalent, so it quite effectively eliminates dead code.
Of course, libraries are often highly coupled within themselves, so not much code can be eliminated, and then they might pull in additional dependencies, etc., etc.
See for example the LispWorks documentation: 4.6 How Delivery makes an image smaller, http://www.lispworks.com/documentation/lw70/DV/html/delivery...
Not writing a compiler for your scripting language saves you years.
Not choosing carefully what features to include in a program, or what packages in a system image: design time saved again.
It also saves considerable time if a user wants the program to do something, and by golly, "look, they thought of that already: there is a feature for it!" Seconds later, the user is just doing whatever they need instead of surfing the web for workarounds or calling support.
Having all the features also increases the flexibility of customization: you have more choices about what you can remove to create smaller images which have specific feature sets. You can hardly remove anything from a minimal image to make a specialized custom image.
I.e. by not making a program as minimal as possible, compact as possible or fast as possible, you are making some kind of trade off. Though the program is worse in some regard, there exist parameters (of something, not necessarily the program) which have gotten better: if you think about it a little, you can identify what those parameters are. The worsening in some of those parameters in a "bad trade" is what makes premature optimization "evil".
The axiom is only valid in those cases because if you've designed your system well, the points for optimization are already obvious and the code can be written mostly optimized the first time round.
“wouldn’t it have been easier not to create the problem in the first place?” It would seem obvious that the answer to this is indeed “yes”.
It doesn't seem obvious at all, especially considering the implicit assumptions about whether or not bloat is actually a problem. There are three classic examples highlighted in the article of writing about this tradeoff: "Make it work, make it right, make it fast", "Premature optimization is the root of all evil", and "Bloatware and the 80/20 Myth" – but the argument about why these don't apply is a bit weak: "bloated software simply makes me sad".
To be honest, I don't find we've got a problem, generally. My experience with my computer is that I can do many more things much faster than I could in the past. And I don't think that counts as bloat.
Bloat is an invisible problem, but that doesn't mean it doesn't exist. Less bloat is more storage space, more performance, more battery life, etc. And the problem is you can't judge "bloat" standalone in one piece of software, you need to judge it as the sum of all the software the user is running. In the first equation, your numbers will always tell you "it's not worth it", unless things are really bad. In the latter, not so much.
Maybe if the Bonbon Squash Series game I have on my android phone didn't link to liballthesocialthings, I'd shave 1MB off a game which is 50MB large. But maybe if every app on my phone did the same thing, I'd have 200MB more on my phone. My old HTC desire doesn't even have half that much memory total.
You say your computer can do "more things much faster" than before - just imagine how many more things, how much faster you could do if the software didn't bloat itself up almost at the same pace as the hardware gets faster.
And you see, I can understand abstraction layers. I can excuse coding for function first, speed second. But then there's the kind of bloat that can only be explained by programmers having no clue about what their software is doing. There's literally no reason why an article should lag in your browser, or Spotify should take 10+% of CPU on standby. There seems to be a lot of pointless code that's being executed, and nobody cares, because developers often have powerful machines. Not so the general population who then has to endure such creations.
Not everyone is cut out for high performance software development. If you're not happy with what they've produced, don't support them.
Part of it is not code reviewed by senior developers. Another part is but too late in the cycle and goes in anyway because features.
The bloat is technical debt accrued in favor of shipping.
It probably happens to both types you described, but shipping is a business decision.
Honestly, an engineer that cares about the performance of their code has several options available: - design the project first, then code later. - design the code for the architecture (arrays of objects vs. objects of arrays). - use tools that can optimize out inefficiencies. - hand optimize after writing a functional slice. - document the APIs and interfaces to explain the performance expected as part of their contract.
Unfortunately, it seems as though we've gotten into a bit of a rut when it comes to this -- we rely on the awful tooling we have to optimize the code we write (yes, LLVM and friends are light years better at this than GCC et. al., but it's still not enough), we never document performance characteristics of the libraries and APIs we create, and we (at least in the OSS world) almost never design our projects to the level of detail necessary to fully understand it.
Honestly, I'm not sure how we can solve these problems if the engineers have a lack of discipline. It's a hobby, after all, right? We do this in our spare time, and nobody wants to write designs or document because that's the least interesting part of coding. See also GTK's autodocs, which for /years/ lay around as nothing more than a simple enumeration of which functions were in the libraries. Even in self documenting languages we fail at this (I'm looking at you, Common Lisp).
Optimizing afterward is a simple, if lazy, way out but post-facto optimization can only take you so far if the design is simply broken.
Since we never document or design, the problem ends up falling to our tooling. Things like oprofile and so on work, but are usually so coarse grained that digging through the code to instrument it effectively becomes a grind nobody wants to do. Maybe we need companies to develop free and open tooling for understanding the performance characteristics of the code we write. After all, the "modern" tooling we use for languages like C and C++ hasn't changed since the 80s, and it's often actually hostile to the end user, which just makes the job harder. Even our linkers often fail to do their jobs of optimizing out useless dependencies or symbols unless we explicitly tell them to do so.
Ultimately, the blame lies with us, the engineers who work on this code. New engineers joining our projects need to understand the code itself through and through. Our projects have become so large that without documentation no-one can make a clear, clean decision on what should or shouldn't be written anymore.
What I want to see is tooling that can semantically understand the code I write, and give me characterizations of the function calls I make as I write them. When I need to examine source to better understand it, I want tooling that tells me how often the functions are called, where they are called, and how. Raw text is great for writing prose, but we're not writing prose -- we're writing code. Our tools should reflect this, and give us as much help as possible.
The point you make about Linux systems being bags of Lego is interesting. That's exactly how it is supposed to be. Unfortunately, the projects we're talking about don't compose well -- they're more like Duplo than Lego because often times they subsume whole feature sets into one huge monolithic block. Huge lumps of code exist in GNOME and KDE and systemd and dbus that just simply have no business being in there.
Dbus and systemd especially fail at the whole design angle.
At the time of X's creation, DEC and IBM was looking for a GUI for their unix offerings.
And dbus came out of trying to make it easier to get the various Linux GUI toolkits to talk to each other, under the stewardship of Freedesktop (where also Systemd was housed until very recently). Thing is that RH was a big backer of that, and was out to win against the big unixes (A recent video interview with various employees there pretty much proclaim "we won the unix war(s)").
The usual counter is that "optimising" takes time. And all of the time that you're spending trying to optimise before release is time that no one is able to use your software at all.
And the reality is that a lot of the bottlenecks can't be predicted in advance - who knows how may files the average user might want to load into your system, or whether your website is going to attract 100 or a million users a month? And many expected bottlenecks become largely irrelevant as technology moves on.
Take his Ubuntu example - for what percentage of users does the fact that it doesn't fit on an actual CD really matter any more? Is it worth spending weeks or months fine-tuning the distro to a point that it can be fitted onto a medium that most people probably aren't even using anymore?
If you really need a lightweight word processor, there's plenty out there. They just probably won't have a lot of the features that the big hitters have now. And if you really want a lightweight OS, there's a fair few Linux distros aimed at that market.
What do you want your code to optimised for? Fast startup times? Functionality? Extensibility? Stability? Compatibility? Security?
None of those things come magically for free, and every day that a developer spends on one is a day that they aren't spending on another. And if they are waiting until any, or all, of those are optimised for any possible scenario, you'll be waiting a long time for any software at all.
As Blaise Pascal (or possibly one of many other possible candidates) once said - "If I had more time, I would have written a shorter letter." The challenge of optimisation v delivery is not a new one.
Don't complain that you don't need those features: it is on you to choose the right tool for the right job. If all you need is a light spell check and text editing, notepad++, vim or Emacs will do nicely. Plenty of people need the more advanced features.
That's only one thing among many mentioned, to prove a point.
The fact that it doesn't run well with only 2GB of RAM certainly affects plenty of actual and potential users. So the point stands. Not about the CD per se, but some optimizations are general and apply everywhere.
And every single Ubuntu user has complained at one point that pressing the Win/Super key does not bring the menu fast enough, or that the search results are not fast enough.
So, there's bloat.
Of course all software could be faster. It could also be more secure, more feature-rich, more user friendly, smaller and "better" in any number of other ways. But developers don't have infinite time to focus on all of them.
So they have to make tradeoffs, and the fact that most people default to the more feature-rich OSs rather than the lightweight ones, suggests that this is the place that the average user would rather the devs spend most of their time.
Actually there is a whole community around this http://suckless.org/
Configuration files introduce a new point of failure (file not present, file corrupted, file not readable), a point of slowdown (how long does it take to read the file from disk, how long does it take to parse and load the file), and a point of complexity (parsing even ini files is pretty complicated, if you want to support the myriad of ways we write them).
All of this for values which change once every... how long? Once a year? Once an install? Or for most configuration values, never?
I still use configuration files, because I'm OK with including third party libraries... but I can certainly understand why some people may not be.
Suckless is about minimalism and speed, and if you want speed, you don't want extra junk like config files clogging up your code's mainline.
It's certainly easier for the programmer to just leave some of the work to the compiler, but it's at the cost of being a complete pain in the ass to packagers and users.
Do you even have to do much work yourself? .Xresources is already parsed and loaded for you by xrdb if you can't afford the cost of an extra 100 microseconds doing it yourself. Is the API to interact with that mindblowing horrid or something?
> All of this for values which change once every... how long?
When I'm configuring software to taste, several times a minute. And considering this is likely to be my first exposure to the software, it better not suck completely.
But if suckless maybe takes their fight against bloat too far, it's only because hardly anyone else seems to care at all.
Free open source software bloat makes me sad.
However, that bloat is winning on its own merits. For instance, GNU/Linux distributions which are minimal are not popular. The popular ones tend to be the bloated ones.
Vim has gotten a lot bigger in 20 years, yet I'm not going to jump ship to a smaller vi implementation.
FOSS makes us confront the fact that we actually seem to like bloat. We inflict bloat upon ourselves --- there is no monopolistic software vendor there to blame.
[1]https://xkcd.com/1172/
A minimalist XWindow installation was possible with a 40 megabyte hard drive and 4 megs of RAM over a 386 CPU. (That's similar specs to what 68K-based Sun workstations had perhaps a decade before that.)
So ti's a tradeoff. On the margin, many more people are likely to write a halfway decent Atom extension than learn elisp and write one for emacs.
Well, writing a text editor is far more complicated than it seems at first place.In order to support big files (50mb logs for instance), you need to make all your code rely on complex memory management, streaming and caching. Maybe js is not the right tool for the job in order to implement these features. I always thought that atom should be build in Qt with an api that can interact with JS,a bit like Adobe products for instance. That way you still allow JS plugins, just like Photoshop while getting a max of perfs thanks to C++.
And in many cases, it's slow, CPU hungry, and memory intensive. Those three characteristics are the very definition of bloat to me.
Isn't that what bloat is? By definition, something that's shipped with "basic functionality" should not consume so much.
http://www.jwz.org/doc/easter-eggs.html
It isn't about "feeling big" at all. You can have large, feature-packed programs that still respond quickly and don't fragment.
There's tons of code in Mozilla that isn't necessary to do the one job the browser was built for.
I don't need code that manages bookmarks. At all. I don't need code that stores history for years. Honestly, I don't /want/ cookies persisted /at all/. I don't need themes to browse the web, and a reorganizeable UI is /never/ what I'm looking for in a browser.
I don't need WebGL. I don't need Canvas. Hell, I definitely don't need syncing of all this extra bloated data around to services somewhere in the 'net, either. Frames? What about floating frames?
Half this stuff is better left to tools that are built for it, the other half is junk I don't need, or is junk that actually creates more problems than it solves.
https://en.wikipedia.org/wiki/List_of_web_browsers
Based on your stated requirements, perhaps you'll like Dillo.
http://www.dillo.org
I actually /do/ use other browsers like Midori, w3m, and surf. The problem doesn't disappear, though, because it's up to all of us writing code to be better at it.
Bookmarks, for instance, can be handled in a much cleaner and portable way than isolating them to the specific browser implementation. The same can be said for history, authentication, and so on.
Also, to be fair, I wasn't arguing for minimal codebases -- that was your point. I was providing examples for the GP.
I get annoyed by having to reboot my TV or radio when they freeze up, or by having delays to my channel changes, or having to navigate complex menus to do things that would have just been a physical button click before.
(this is all part of the same problem as bloat IMO).
This is where the likes of Apple or Dyson do well (or OK) I think. It's not necessarily that they're brilliant at what they do, it's just that the competition is pretty awful.
Even though my computer can run all applications without a hitch, it is still very wasteful to constantly use CPU power because of technology choices or plain laziness. As an example, Spotify and Slack are two applications that seem to use most of my CPU after Chrome. Spotify and Slack combined seems to hover around 5-15% of total CPU (a two year old i7). When there is a lot of traffic in Slack I have seen it using 15-20% by itself, with multiple processes running and memory use going above 200 megs.
Both applications work smoothly, but should they really use that much resources? A chat application? A music player? With modern CPU's I would expect them to be at the bottom of the process list when sorted by CPU usage. I used IRC on my Pentium 75Mhz and it ran fine. When simple applications are made so poorly that they use that much resources, what is the worldwide impact of that power use? And what about the users that don't have powerful and expensive CPU's?
I guess it would be a nice incentive for developers if OSX could notify you that an app that is on background is using a lot of energy at the moment, maybe even with a quit-button when on battery power. At least I wouldn't want my app to end up on that kind of a popup.
A 15" Macbook Pro has a 99.5Wh battery and lasts about 8h; that's 12.5W. A program that decreases battery life by 20% would only mean an increase of 2-3W. Even if the machine and program are running 24/7 and all the energy comes from coal, that means an increase of less than 0.03T of CO2 per year. For comparison, the average carbon footprint of an US citizen is 20 Tons per year.
The problem is that our tools for making multiplatform native GUI's suck so bad we'd rather just embed an entire Web browser into everything.
I think the main reason that node-webkit and what-not are popular, is because of web developers moving to native app development. It's really easy to get started that way, and you can even share code with your web app. Where something like QT has a really huge learning curve for programmers transitioning from Javascript.
About poor algorithms. I actually worked on optimizing a well known web browser for a couple of years. And most of the stuff we did, was because of really bad Javascript code. Even though it seems gluttonous to embed a web browser in applications, and even insecure, it doesn't have to be as bad as it is, especially with a simple application like Spotify. This is going on a bit of a tangent, but every frontend programmer should at least learn how the browser actually works, a nice site for that is http://jankfree.org/
There's more to non-nativeness than just look and feel, which the more mature cross-platform GUI toolkits can ape fairly well. Another concern is accessibility for people with disabilities, e.g. blind people using screen readers. Qt, for example, is kind of accessible on Windows, Linux, and Mac, but not at all on mobile platforms. Not sure about the status of JavaFX. At least a single-page rich web app can be made accessible using the ARIA extensions to HTML, and if you use one of the big four web rendering engines, you can be sure they've done the hard work of implementing the underlying OS accessibility APIs well. Of course, many (most?) web developers don't implement ARIA for their custom UIs.
wxWidgets applications work like native applications because they are in fact using the native toolkits, and some successful applications, like Audacity, are written using wxWidgets.
And it runs fast, as it is C++, as expected. Why it is not used more fits perfectly with the content of this article.
It sounds like solution (for spotify at least) is better methods for each platform of creating a GUI around web services. Microsoft have sort-of had a go at this.
Browsers are already closer to this than most people realise. By the time you've included websockets, openGL, webaudio, and bunch of other stuff, you're 90% of the way to a useful OS.
The problem is that the current state of the web "OS" is a random collection of ad hoc APIs hacked together for a completely different job, and it's very badly designed for what it's trying to do now.
In a perfect world the FOSS world would collaborate to thrash out a new spec, and also design a new browser to follow the spec which was backwards compatible with standard HTML etc, but also included new and more efficient APIs for much faster performance.
Something like this has already happened on servers - see also, containerised VMs running RoR or Node or whatever you want - and it's only a matter of time before it happens in browsers too.
The problem is that the current plan seems to be to build a VM layer on top of the existing DOM/js/etc layer, which will kill performance even further. It should really replace it and emulate it.
On the one hand, I wonder how much of the wasted CPU cycles can be attributed to carelessness that could easily be avoided, or caught early, if we developers deliberately used underpowered hardware for our own machines. Speaking for myself, my main workstation, where I also do most of the testing on my desktop apps, has a Core i7-4770 processor and 32 GB of RAM. Maybe if I used something more modest, like an ultrabook with only 8 GB of RAM, I'd be more likely to notice when I'm carelessly writing inefficient code.
On the other hand, as others have argued both on this thread and elsewhere, there's a trade-off between machine efficiency and developer productivity. We may argue that it's wrong to waste machine resources when the machines in question don't belong to us, but then again, developer productivity means we can crank out more features that drive sales and make users happy.
https://twitter.com/lix/status/589171043010412544/photo/1
So, looking at this rationally, consider your favorite lightweight window manager or desktop environment for Linux. Is it fully internationalized, including support for CJK input methods? Is it fully accessible to users with disabilities, e.g. blind people and people with mobility impairments? Does it auto-mount USB thumb drives? Does it do all of the other things I'm not even aware of that are required for a fully usable desktop environment covering a wide variety of machines, users, and use cases? AFAIK, the only Linux desktop environments that come close are GNOME and Unity.
Basically, the world is complicated, so real-world software has to be complicated too.
It sounds like Atom could still use some optimization though.
i18n is an orthogonal service that the DE or any other program may call into.
Is it fully accessible to users with disabilities, e.g. blind people and people with mobility impairments?
Screen readers and screen magnifiers are orthogonal services that the DE may make accommodations for, though some DEs may be lesser fit for users with disabilities in general.
Does it auto-mount USB thumb drives?
This isn't even the job of the desktop environment, it's the job of a dedicated device node manager, which the DE may call into. It's completely orthogonal and can be accomplished even without so much as a display server, because it's a different layer.
NFS in the classic cross-network *nix mount. And depending on available software (FUSE and related) you can mount the likes of SMB/CIFS shares, FTP servers, even SSH/FISH can be used via a mount.
Unix systems really aren't good at abstracting local and remote mounts and binds into a singular resource, but the parent poster's scenario was implied to be a local user hotplug, which is what device managers largely do (if even that, udev these days just maintains the /dev/disk/* symlinks and handles query requests for the hardware database).
Who does this, though? Are there reliable adoption numbers? NFS semantics are wrong in well-known, familiar ways and it ships everywhere, making it still quite popular.
There's much better network file systems but because of many factors (boat/complexity being the biggest) their adoption is minimal.
Once you move beyond single-user/simple read/write permissions there's really no easy to avoid complexity. It's just the nature of the beast that is sharing files over the network in a way that appears native to the user.
I'm the project lead of LXQt. All those things you listed can be supported without introducing much bloat at all. In DEs, most bloat comes from pure technical debt. Unnecessary libraries, duplicated code/libs, etc.
Feature bloat is another matter and is more subjective. A lot of it is down to UX.
Fun fact: KDE 4.x depends on ruby in most distros. Do you know why? Dolphin ships with a ruby script to update some folders (don't remember what it does). Nobody runs it. That's it.
Why did they allow an additional dependency on Ruby in? It seems like perl or python would be the more logical choice, given they'd already be on the system (heck.. maybe bash?)
A script that is compatible with POSIX Bourne shell (/bin/sh) would be more appropriate. FreeBSD, for example, does not come pre-installed with bash. If you want cross-platform portability, this is a better default.
For example, suppose you're developing an application, and you want to use a lightweight GUI toolkit, or roll your own as Spotify apparently used to do. Unless the toolkit is just a thin wrapper over the underlying OS's native GUi facilities, like wxWidgets or SWT, the resulting application is very likely to have accessibility problems, unless you do the hard work of implementing the accessibility APIs for your target platform(s).
I would rather see bloat and new ideas, than only experts working in a problem domain. My best ideas come from a marriage of understanding a single problem space, and a wonder what I can do in another. Bring on the bloat.
Windows went from 7MB required (95) to 6GB required (Vista), 1000x bigger, in just 10 years. It's true in file formats as well and verbosity is one of the reasons XML was/is so hated.
There is an irresitable urge to make things bigger bigger bigger, and huge cpu, mem, and disk resources are essentially free.
Not perfect (especially as writing things as plugins tend to introduce a certain amount of bloat regardless), but it can be better regardless.
Or the failure of a large team to properly manage software complexity.
It's using your time to do something else, such as improving security, or adding a new feature. Some people seem to be talking as if it automatically takes the same amount of time to create efficient code as inefficient code, and the idiot developer has simply forgotten to add the -run_faster option on the compiler.
I'm involved in a project at the moment where a team of developers and engineers have spent much of the last month trying to get response times in some very specific circumstances in an application down to an acceptable level. That's not because they are either stupid or lazy. It's because these things are hard - there's an awful lot of moving parts, and an awful lot of things that can go wrong in each of them.
When they get it down to an acceptable level, they'll ship it to the users. It won't be blazingly fast, but it'll be good enough. They could definitely spend longer tuning it further and getting it going even faster. But every day they do that will be a day longer until the features that the users want are actually available to them.
The reason there's lots of moving parts is because modern computers, OSs, languages, application servers, libraries, databases etc are built to do a lot of very complicated things.
We could have written the entire application from scratch, not using any 3rd party elements so that we had full control over every element. But that would have taken several orders of magnitude longer to deliver actual useful working software to the users.
And rewriting them, along with modifying the rest of the application to use them, takes time and can lead to further unexpected behaviour.
Here's the bits that aren't robust enough: any open source that isn't either used by thousands, or used in exactly the way you will be using it.
That 95% would still need to interact with my new 5%, and as it was never designed to do that, it could easily introduce new unexpected behaviour. Imagine trying to rip out (for example) the indexing code out of a database because it's performing badly and then rewriting it.
It's a 3rd party commercial platform for developing your own applications on. To rewrite individual bits that aren't working efficiently would be a nightmare.
As for imagining having full control over the behaviour of my code - I had that quarter of a century ago or more when I started coding, and spent weeks or months building windowing systems and low-level networking code in assembler that I could achieve in 30 seconds with any off the shelf language these days.
Yes, my code was extremely efficient (it had to be, trying to do graphical comms software on 8086 machines didn't give you much of an option to do otherwise), but the use of my time wasn't.
The third party product that we're using is the result of many years of a large team of dedicated developers. Rewriting that from scratch in order to make sure we've got full knowledge of what's going on is an utter ludicrous idea.
Then use the 10 year old version! Unless, of course, one of those few "actually useful features" is something you can't live without.
A lot of people use Word for a lot of use cases. The value of an added feature that someone needs always trumps the performance cost of adding it until the performance becomes so bad that it becomes the reason that other people stop using it.
It's weird how new trends in UX design try to make a first-time user become a genius immediately after double-clicking on the program icon. The only way you can do that is by dumbing down the software to the point it can actually be comprehended this way - which makes it much less usable and effective as a tool.
Maybe yours was faster, but I think a lot of people have rose-tinted memories of the speed of applications in the old days.
However, I'm optimistic. For one, I think that the tools we have today are superior in absolute terms (convenience, speed, ease of use etc.) than they ever were. Of course they are not exponentially better (like the hardware they run on), but they are better. Also, occasionally I see some gems here and there. There are still many developers who take pride in what they do and I enjoy finding that diamond once in a while when searching for an alternative technology to use in a new project.