Things break down and things eventually need to be replaced. This has been a fact of life since time immemorial. I do tire of programmers acting as if the necessity of maintenance is some novel and unexpected problem, unique to software development, presented with hyperbolic titles like "software is drowning the world". Is physical infrastructure "drowning the world" because it needs to be maintained?
The inability to fully understand how a system works due to the interconnecting pieces is not a problem unique to software either! That ship sailed a long time ago. E.g. with cars can the average person now repair it at home and maintain the entire car themselves? It would be foolish to recommend that we "fix" cars because of our inability to personally understand the entire thing from inception to finish, yet some feel we can apply the same to software.
Just look at the Windows Internals books to understand the complexity of just the systems in the operating system. It isn't something that needs to be "fixed" just because something is complex.
Everything is more complex than we think it is, and that's not really a problem.
Car maintenance can be shifted to another party with a different business model, eg leasing your car makes it someone else's problem.
Similarly, "Functions as a Service" makes many problems the hosting provider's problem. All of the low level goop is theirs. You still have OP's problems but the responsibility surface is reduced.
But then you get an entirely new and surprising goop of problems like bursty lag problems (one out of 500 requests finishes in 5s instead of 0.1s), lack of consistency because one of the datacenters is offline, and so many others.
It's more like "you now have to deal with different problems" much more than it is "you have to deal with less problems". Maybe the problem quantity got reduced from 1000+ to 50+ but the latter might swallow much more time to solve so it's really questionable which of both is better.
I don't know whether or not you've had the opportunity/personal experience of owning two cars concurrently for which you are the only party responsible for doing repairs and maintenance, and where one is much newer (say >10 years), putting you in a position to be able to compare and contrast the two.
Cars are not a good example to point to. The thing that would make cars a good example would be if cars were shown to exhibit a steady increase in essential complexity. In that case, yes, we might point to them and say, "Maybe the situation for software is the same, and we're mistaken to have any unreasonable expectations to the contrary?" But that doesn't match reality. Car manufacturers are as guilty as software developers of producing things that have gotten steadily complex without good engineering reasons to justify the complexification.
(Pre-emptive request: please do not respond to this comment as if I'm arguing that cars have not become safer or more fuel efficient.)
Yes I have a 1972 VW Beetle, and a 2016 VW Beetle. I can safely say that the former is far simpler and also doesn't require a computer to diagnose or repair. It also doesn't have any firmwares that I can't update, eg. the bluetooth controller in the modern Bug, or the stereo. I can't enable the telephone button on my steering wheel without paying VW. I can't enable Android Auto without paying VW £200 here in the UK. (They couldn't even get it to work when I asked them to).
As for the mechanical components of the car - the modern Bug is a diesel (non-PD) so has a common rail fuel system. I can read fault codes with the OBD2 mechanism but not everything is accessible.
Both tremendous fun but the modern one is 100 times more complex and I would struggle to maintain it single-handedly. But I don't feel the need to reduce the modern car or "fix" the issue of its complexity simply because it is more complex.
My wife has a Mini that has developed a fault but who knows what sensor has failed according to the OBD2 codes?! I don't feel the need to "fix" that and reduce it to the old 1991 Mini Cooper I had (which had its own set of baffling problems, like in-line fuses for important things like the injector but a fuse box for non-essential things like the interior light..... but that's another story).
I didn't see in the article any claims that the problems outlined are unique to software development, can you point them out?
Hyperbolic titles are indeed a problem of our age though, I agree on that completely. But your prejudice towards programmers who are supposedly thinking their problems are novel and unexpected is not justified.
> That sounds a bit broad. In the general aviation world, engine designs from the 60's are still in widespread use.
As is leaded fuel, to the delight off everyone living within a few miles of airports. Old can be good, until it's not anymore. There's often survivorship bias involved as the worst--or unadaptable--designs have naturally fallen out of use or been supplanted.
He is talking about the second law of thermodynamics in realised designs.
You can use designs from the 60s today and the result will be a brand new system that in time will rot away and need to be replaced by, possibly the same design. That doesn't invalidate what he is talking about.
Even the pyramids of Giza will eventually rot away given time.
The difference in software, in my opinion, is speed in which change occurs.
As a GA pilot I find that a counter-argument: the fact that these engine designs from the 60's are still in widespread use is showing how far behind GA is. These engines work, but they are less performant and economic than car engines of 2020, while costing a lot more (smaller market, regulations, certifications). The old, heavy, underpowered and ubiquitous Cessna 172s are the best example of what is wrong in GA.
I should have better explained my point. Their fuel efficiency is atrocious, their power output is pitiful given the size and mass, and they require leaded fuel, as paulryanrogers mentioned. My point wasn't that they're ahead of their time. Engine technology has of course improved in the last 60 years. My point was that they've stuck around. Of course, there are examples of long-lived software, too.
But I think the initial-to-maintenance ratio of software is both different than for physical infrastructure, but also perhaps more importantly, very different than many decision-makers expect.
To be fair, I don't agree with this point of view at all.
Things in the physical world break down beacuse they are made of matter, and the entire concept of "working" is a very simplified modeling of their behaviour that doesn't include second-order effects like tear,consumption of materials etc.
Code, on the other hand, is a social constrution but entirely confined in an abstract and mathematical world. The _only_ reason we still don't have a reliable and solid code infrastructure (in the sense explained by the article) is our inability as a community to once and for all define a solid foundation for the principles of computing. Hence all the hacked together languages, tools and frameworks that people have piled on each other over the decades.
If one century of efforts hasn't been enough, fine. But we absolutely can work towards that goal.
> Code, on the other hand, is a social constrution but entirely confined in an abstract and mathematical world.
Brian Cantrill had a good observation in a 2007 Google TechTalk introducing Dtrace:
> In software the blueprints are the machine. In software once you design the thing you've built it: the design is the machine. That's why the waterfall model is so fundamentally flawed. This idea that you could design the design before you design it, which is what the waterfall model is essentially saying, is flawed. Software is both, it's both information and machine. (@3m40s)
That is an extremely simplified view of non-software.
Hardware standards shift because of an Apple PR camopaign. We stopped using zeppelins only after one disaster of several dozens made very public by live radio coverage. Non-Sufficiently standardized rockets explode because countries can't pick a measurement system. The auto industry in a country changes overnight because a dictator decides that the roadside to travel on shall be switched on a whim.
Just like software, infrastructure/hardware/whatever is an unfathomably complex web made from science, faith, half-knowledge, geopolitics, coincidences, rules, dick moves, physics and random tweets.
I don't think we are winning anything by trying to lift software onto some kind of pedestal of specialness in every regard...
> is our inability as a community to once and for all define a solid foundation for the principles of computing
Is this possible? If code is a social construction as you say, then wouldn't it be impossible to arrive at this point unless society remains unchanging?
> Hence all the hacked together languages, tools and frameworks that people have piled on each other over the decades.
This hints at the what is IMO the real problem, software is hard to replace with confidence. Sure, it's easy to swap one version for another, but doing so with the full understanding of what will change...is basically impossible for any non-toy project today. And there are exceptions to this, but they're the "exception that proves the rule" type.
I remember reading an article about coding at nasa, like the code that ran the moon landing ship.
I can't remember the link but you can probably find something.
the way they code is very careful and deliberate. everyone has an office, none of this "rockstar" nonsense that you have at startups, they are all serious people, the deadlines are sensible, no one pulls all nighters, they come in at 9 and get out at 5. absolutely nothing gets written without documenting it. no line of gets changed without the reason for the change written in documentation. everything is carefully panned.
needless to say the code got the ship to the moon and back. this is how coding should be done
A lot of programmers I know have no hard feelings about needing to maintain software. My product managers, on the other hand, don't necessarily understand that every line of code has potential long-term cost (and obviously, they ideally also have long-term benefit).
I think a real issue we developers face is the management and client attitude that software does not require maintenance. Time and time again, the fact that our completed projects require maintenance or they simply fail with newer systems is greeted with shock, debate and accusation of deceit.
Yes. IMO that's the root of the issue. If there was an incentive to invest in what the OP advocates for, it would have received a military level of funding and attention decades ago.
Problem is, most people are under the illusion that software is like a woodwork or metalwork gadget: you make it once and it can work happily for 10 years without changes. Which is an extra funny thing to believe in because it doesn't apply to those physical gadgets either.
It seems people just want things to be created once and used infinite amount of times and even if I can relate to that desire, it's simply not the universe we are living in.
And in spite of countless evidence to the contrary, it's still an extremely persistent illusion that creates untold amounts of friction between shareholders and programmers every day.
I think I see it slightly differently. People have a shifting baseline of expectation around software that isn't present in "traditional" tools. The hammer you inherited from your grandfather still does what it did the day it was made, and so do ls and grep. It absolutely does not hammer nails at 10000x the speed while talking to every new kind of hammer made around the world, but somehow software is expected to gain that ability without any additional effort.
I don't really have a solution for this, I think tech consumers/clients are just always going to want what they want and not care so much about what it takes to make it happen. Thus they will always be surprised and grumble about the price. That just seems like human nature to me.
> Thus they will always be surprised and grumble about the price. That just seems like human nature to me.
Absolutely. I see it the same way. After commercially programming for 19 years I've seen exactly zero progress in the human race with regards to that particular problem. It is apparently a built-in bug of our brains and everybody discovers it for themselves down the line (and 99% proceed to never fix it).
I think I recall one quote from a seemingly famous building architect (I think the guy who built the Fallingwater house, so Frank Lloyd Wright?) that was along the lines that you never put all expenses in plain view of the customer at first; you put a reasonable cost plan in front of them and then, after they are heavily invested in you, start giving them the other expenses (which are inevitable and not invented by you, let's accentuate that!). Many people call that a scummy practice but the truth is, EVERYONE is expensive, some are just better at selling themselves. So you have to give the customer the truth spoon by spoon because they prefer lies and truth is too hard to swallow at one bite.
And apparently that's how people tick. I don't like it either. But... blame the game, not the player -- as the saying goes.
Most clients I’ve worked for are fully aware of this and want maintenance contracts in place to support systems before go live.
Of course that doesn’t mean they are happy to spend even more it upgrade systems later.
But what the author is proposing would automate and reduce the cost of the support. A company that could achieve this cost saving and market or successful would have an advantage. Unfortunately it is a very hard problem abs many upgrade transformations can’t be proved to be correct with good automated tests. And then you have the problem of how do you automated the maintenance of the tests.
But that's just the problem: programmers don't want to pay for such a tool. I personally see it as a part of my work and having to pay for that feels like microtransactions in a game to me.
So not sure how such a thing could be monetised properly. Likely by only offering enterprise-grade services: the tool itself is free, the customer's programmers could employ it to achieve migration but the customer prefers to outsource the cost. Sort of how several Linuxes are free but come with paid support contracts.
Have you tried explaining it in a different way to them?
Imagine you brought your car for an oil change and the mechanic says "uhh yeah maybe you can get new oil next year but for the next few months we need to... Uh... Address technical debt in your car... So umm... It's gonna cost you thousands of dollars and you aren't going to get anything you perceive as a change"
Sadly most programmers are introverted and not well-articulated (which is much worse than being introverted). So when it comes to them you are quite right.
However, as a counterpoint to yours, even those of us who can communicate quite clearly get met with very ugly accusations of ripping customers off. People simply come to you with a preconceived notion about how much something must cost (in terms of money, effort and time) and they are absolutely not willing to change that notion -- even if they are on the side of the pure imagination and you are on the side of the facts.
It's basic psychology that many of us had to learn even if we never wanted to.
For concentrating on high level business logic, there are various low code / no code systems. But that still means someone needs to take care of all other dependencies, security fixes, scaling, etc. Someone needs to understand and maintain all pieces for it to work properly, and need to know how to optimize bottlenecks.
Automating fixes is only possible if you understand fully where the problem is. Then, it still needs testing does it break something else. For testing, automatic tests most likely do not test for everything. And someone would need to create those tests.
There is also a lot of existing code for existing software. Would it be possible to analyze or convert that amount of code to be somehow updateable?
Software is usually made with some programming language and related dependencies. Those dependencies don't necessarily exist for other programming languages and web frameworks.
Totally agree. If the new version of a library requires a slightly updated usage; please just give me simple examples of how to update my code. Include a simple regex. And while we’re are it, let’s standardize how we provide those upgrade scripts. So when snyk tells me there’s and issue, it also easy to find and apply the necessary code changes (at least for the simplest 90% of cases).
Like python’s “2to3”.
It’s not a magic bullet; the developer still benefits from thorough testing, but I think changelogs could stand to improve in a lot of projects.
For real. If its not acting the same, then its not the same thing, give it a different name and release your 0.1. I guess maybe we run out of namespace with this methodology? Suddenly we end up with software named axsladkjfaksldjf, and then hackers start camping on axsladejfaksldjf
The name space is not a problem, in my opinion. If you have a library sdaffj, and you want to release a new, incompatible version, release it under the name sdaffj2, next incompatible version under name sdaffj3, etc.
Or you know, library deprecates "easy way of doing something" replacing it with "weird and convoluted" way of doing this again without any guidance of how to do the old thing
And unless the old way was genuinely bad/hard, why deprecate?
Talking about Spring at several places specifically, I assume the author has been bitten by the Spring version treadmill (eg older Spring remoting not being maintained, forcing you to upgrade to a new Spring major version) and the big deprecation coming with Java 9 and the JEE to Jakarta migration. Been there, done that, and it sure sucks to do the month-long blues to bring deps of even a medium-sized app to a new version. But, not picking on Spring specifically, it's implicit in Spring's design as a grand unified wrapper for all things infrastructure. Whereas building on the base technology Spring is wrapping (JEE, mostly) would in principle allow picking versions much more freely. However, at this point, with the partial starvation of former JEE packages, even that isn't a given (had troubles with pointless JAX-RS apps and deprecated maven plugins at a customer of mine last year, whereas Spring migration was relatively painless).
I think computation is still converging. It is too new to be expected to abide to universal standards. Cars, for instance have been around for over a century and they are still subject to change in standards, mainly because of safety and efficiency, just like software, but to a different degree.
We should, however, have these kinds of discussion because I don't think we should outsource this debate to corporations. But this shouldn't take the tone of contempt and resentment towards what we already have.
The problem is financial incentives. I am not paid to devise such universal tooling. It's also a huge undertaking and much harder than all of my everyday's work. It's exploratory work that has the potential to revolutionize huge chunks of the programming area.
And I don't want to only be paid an average salary for it.
But... an excellent salary (above-average, maybe even above-high) would motivate me and no, I won't try to copyright that work and mooch off it for billions until I hit 80.
As I said: incentives. Many companies expect programming to be like installing a kitchen sink yet are unwilling to pay to commoditize the activity.
I think this is part of the problem, yes, but the root is more fundamental in my opinion.
Even if you had a lot of financial support, how would you go about implementing the standards and transformations described in the article? Who would adopt it?
Maybe in the context of a company this could reap benefits and become a case study. Or in a partnership between many companies there could arise such applications. But they would still be locked to a niche. That is why I think even with financial support, the problem is maturity, or lack thereof.
You could argue that once a large enough group adopted practices that proved to be effective in reducing maintenance time and complexity, that others would follow, but that stops being a matter of pure financial support and starts being a matter of adoption and convergence, pointing once again to maturity of the ecosystem.
And another aspect of maturity, standardization and sanitization is that it comes at a cost to diversity, which in turn has benefits of its own, like new ideas and technology. This is kind of the balance between exploration and exploitation we see in training neural networks. I think software is in the transitioning phase from exploration to exploitation and we are living it.
Oh, I am with you on this as well. My point is that we never get to the point where network effects (or lack thereof) is the problem to solve. Since nobody is financing the work we simply never get to find out if a proposed external tool would be accepted by the community or not. That's the disappointing part. We never had the problem to begin with.
IMO if you make such a tool for something extremely popular like Python or Javascript and then start pivoting upwards from there, people will be coming to you. There are a lot of deficiencies in the everyday work of the programmer to this day.
This is not only possible but has been done commercially.
I have a friend that contracted for a company doing y2k work. The main guy have written a system to parse cobol, do pattern matching to find y2k issues, apply transforms to the AST to fix the problems and the serialise back to cobol.
It supported a dozen of so cobol variants. I think it was very successful.
> Every time you decide to solve a problem with code, you are committing part of your future capacity to maintaining and operating that code. Software is never done.
I am clipping this for use in future internal discussions. I have trouble communicating this point, this sentence does so concisely.
It'd be better to start off project with a depreciation schedule on software. Such as discussing plans for ROI after 5 years etc. Past that point, maintenance needs to be considered new work, and billed as a new project.
We've got to stop thinking about software in agile sprints once a product exists. I'm not saying don't use agile for the build. But we've got to learn as software professionals to think like professions that maintain other expensive assets, such as aircraft, ships, or public works. There's a recognition, that a 50 year dam, needs a tear down and replacement in 50 years.
> Every time you decide to solve a problem with code, you are committing part of your future capacity to maintaining and operating that code. Software is never done.
Notice that this includes using code written by other people (e.g., dependencies).
Well, yes, but with possibly different costs. The trick of course is in trying to estimate what they will be.
But using off-the-shelf software that you simply install is certainly one form of "using code written by other people." If it is popular software offered by a stable company, I would always estimate it's likely future cost as less than anything written in-house. (Which doesn't mean it's risk free, the software can become unsupported or change in backwards incompat ways incurring costs. You can get unlucky. But I'm almost always going to advise that it's the safer choice with regard to future costs than in-house development).
If we're talking about in-house software projects to begin with, as you probably intended, but evaluating using a third-party dependency vs writing the functionality in-house, figuring out which of those paths will be likely to end up lower-cost is not at all easy to do, and dependent on a lot of context.
But certainly in all cases, not having any software will be lower-maintenance-cost than having any software. Well, at least lower software maintenance cost. :)
This is one of the reasons why statically typed languages have a big advantage when it comes to maintenance.
Take Scala for example. There is now mature tooling for keeping your dependencies / libraries up-to-date automatically, using automatic migration-scripts (must be provided by the library author of course).
The difficult part here is of course to write the migrations. This works very well in Scala (and can work as well in certain other languages) because the type-system provides enough information to do automatic rewrites and it is easy to _not_ use unsound techniques such as reflection, code generation, macros etc.
The reason why we don't see such tooling in other languages yet is that they are either dynamically typed, which makes it almost impossible to write migration scripts that pretty much always work. Or they are statically typed but the typesystem is so limited that developers have to fall back on mentioned unsound features.
That was basically my main reason to learn Rust (okay, mega-speed is another but it wasn't the first): no matter the syntax, no matter how hard it is to learn in general, no matter how hard to learn and use all the idioms is, you have a very strict language that makes it that much easier to automatically refactor.
...Or at least I hope that will be the case in the future. But like you, I believe it's much better positioned to do so compared to the dynamically-typed languages.
I'd say it's harder to refactor because you can only refactor to correct programs. Other, more lax languages are easier to refactor to a wrong new state.
In a business setting, making technology choices is hard. The only thing that is harder than making a bunch of technology choices one time is making them constantly.
Pick technologies that are stable over time frames measured in years, not months. You will never be able to construct any great works on top of fields of quicksand.
For example - Instead of using a fun & shiny new language you read about on HN to construct your company's new loan processing backend, perhaps consider technologies such as Java or .NET. Instead of using a database engine that was developed this decade, look at something that has already stood the test of 1-2 prior decades.
Now, 100% of everything I have just said is totally irrelevant if we are talking about software engineering just for the sake of it (fun, hobby, exploration, etc). This is the important distinction. If the livelihoods of your coworkers depend on you not making a bad technology choice, try to go with the safe bet. Separation of hobby & work mindset can solve a lot of this mess.
Computation doesn't have the equivalent infrastructure as table lamps yet.
Testing / Certification:
A table lamp will be UL/CE approved. This means it will be tested in such a manner that normal use and abuse will not cause unexpected side effects. Knocking a UL approved lamp off a table will not cause your house to get burned down. A light bulb of rated capacity will not cause a short circuit. Normal wear and tear will not cause it to become unsafe.
Outlets: A table lamp plugs into an outlet, of a standard size and characteristics. The outlets are a standard interface that is often nation or region wide. The voltage and current that can be delivered through an outlet are standardized, as is the nature of the loads which may be plugged into it. The outlet itself, and the housing, and wiring are well regulated.
The outlet is connected to a circuit, and a circuit breaker (or fuse). These devices completely and permanently interrupt the delivery of voltage for a number of standard conditions, including over-current, and also ground faults.
The circuit breaker, or fuse socket, is part of a standard and well understood panel, which is designed to allow a variety of options in deployment, with electricians able to configure them in a wide variety of applications. Yet, the user of a breaker of fuse panel is protected from the voltages and power inside, and given a standard and easy to understand model of how power is distributed, and controlled. In fact, some advanced users can use the panel to disable part of their local power network to allow maintenance and modification in a safe manner.
Breakers and fuse panels allow for lock out, which will physically prevent the reapplication of power, while maintenance is ongoing.
Power feeds include metering, and themselves are driven from a circuit, and the system is designed in much the same way as the residential and business circuits, but on a larger scale.
Not only all of those are true, but there is more
Circuit breakers and fuses on devices, etc.. are all designed to be coordinated. The smallest fuse/breaker in the chain from source to load should always trip first. This prevents a scenario such as the one where a person plugging a defective toaster into an outlet brings down the entire power grid.
Circuits and power routing have well separated areas of concern. There is no way that a power system problem can directly cause problems with the sewers or other infrastructure, other than failing to deliver power.
Our software infrastructure lacks all of this sophistication and standardization. You can plug in a fan made 100 years ago into a modern outlet, and it will work. You can't even run an MS-DOS program from the 1980s without resorting to an emulation layer.
You can know that same fan will not cause Russian bots to be able to control the power grid. Nothing plugged into an outlet can ever give you control over the grid. The same is not true of software, no matter how carefully constructed. We're still at the stage where a bad program can take out its host operating system, and then the network its on.
To say we need more skilled programmers, better aware of the security implications of their work, is to insist that we could run a power grid without circuit breakers in every home.
The operating system has the job of protecting the network and its users and applications from each other. There is no equivalent to circuits and breakers. You can't run a program, and know that it will only consume X amount of CPU, network I/O, or RAM, unlike the outlet limited to 15 amps.
You can't be sure that a program won't have side effects. Plugging in a bad toaster in the kitchen shouldn't effect a circuit in the garage. The damage should be limited by default. None of our Operating Systems do that.
Computation lacks maturity, especially our operating systems. We have no stable standards for anything important.
64 comments
[ 3.8 ms ] story [ 141 ms ] threadJust look at the Windows Internals books to understand the complexity of just the systems in the operating system. It isn't something that needs to be "fixed" just because something is complex.
Everything is more complex than we think it is, and that's not really a problem.
Similarly, "Functions as a Service" makes many problems the hosting provider's problem. All of the low level goop is theirs. You still have OP's problems but the responsibility surface is reduced.
It's more like "you now have to deal with different problems" much more than it is "you have to deal with less problems". Maybe the problem quantity got reduced from 1000+ to 50+ but the latter might swallow much more time to solve so it's really questionable which of both is better.
Cars are not a good example to point to. The thing that would make cars a good example would be if cars were shown to exhibit a steady increase in essential complexity. In that case, yes, we might point to them and say, "Maybe the situation for software is the same, and we're mistaken to have any unreasonable expectations to the contrary?" But that doesn't match reality. Car manufacturers are as guilty as software developers of producing things that have gotten steadily complex without good engineering reasons to justify the complexification.
(Pre-emptive request: please do not respond to this comment as if I'm arguing that cars have not become safer or more fuel efficient.)
As for the mechanical components of the car - the modern Bug is a diesel (non-PD) so has a common rail fuel system. I can read fault codes with the OBD2 mechanism but not everything is accessible.
Both tremendous fun but the modern one is 100 times more complex and I would struggle to maintain it single-handedly. But I don't feel the need to reduce the modern car or "fix" the issue of its complexity simply because it is more complex.
My wife has a Mini that has developed a fault but who knows what sensor has failed according to the OBD2 codes?! I don't feel the need to "fix" that and reduce it to the old 1991 Mini Cooper I had (which had its own set of baffling problems, like in-line fuses for important things like the injector but a fuse box for non-essential things like the interior light..... but that's another story).
Hyperbolic titles are indeed a problem of our age though, I agree on that completely. But your prejudice towards programmers who are supposedly thinking their problems are novel and unexpected is not justified.
That sounds a bit broad. In the general aviation world, engine designs from the 60's are still in widespread use.
As is leaded fuel, to the delight off everyone living within a few miles of airports. Old can be good, until it's not anymore. There's often survivorship bias involved as the worst--or unadaptable--designs have naturally fallen out of use or been supplanted.
You can use designs from the 60s today and the result will be a brand new system that in time will rot away and need to be replaced by, possibly the same design. That doesn't invalidate what he is talking about.
Even the pyramids of Giza will eventually rot away given time.
The difference in software, in my opinion, is speed in which change occurs.
https://www.strongtowns.org/the-growth-ponzi-scheme
But I think the initial-to-maintenance ratio of software is both different than for physical infrastructure, but also perhaps more importantly, very different than many decision-makers expect.
Things in the physical world break down beacuse they are made of matter, and the entire concept of "working" is a very simplified modeling of their behaviour that doesn't include second-order effects like tear,consumption of materials etc.
Code, on the other hand, is a social constrution but entirely confined in an abstract and mathematical world. The _only_ reason we still don't have a reliable and solid code infrastructure (in the sense explained by the article) is our inability as a community to once and for all define a solid foundation for the principles of computing. Hence all the hacked together languages, tools and frameworks that people have piled on each other over the decades.
If one century of efforts hasn't been enough, fine. But we absolutely can work towards that goal.
Brian Cantrill had a good observation in a 2007 Google TechTalk introducing Dtrace:
> In software the blueprints are the machine. In software once you design the thing you've built it: the design is the machine. That's why the waterfall model is so fundamentally flawed. This idea that you could design the design before you design it, which is what the waterfall model is essentially saying, is flawed. Software is both, it's both information and machine. (@3m40s)
* https://www.youtube.com/watch?v=6chLw2aodYQ
I originally ran across it on Google Video:
* http://video.google.com/videoplay?docid=-8002801113289007228
Social constructs are not immutable any more than physical ones. They're more likely to change, from what I've seen.
Making software is about groups of people interacting, side effects of the software, and lossy interfaces more than it is about mathematics.
Hardware standards shift because of an Apple PR camopaign. We stopped using zeppelins only after one disaster of several dozens made very public by live radio coverage. Non-Sufficiently standardized rockets explode because countries can't pick a measurement system. The auto industry in a country changes overnight because a dictator decides that the roadside to travel on shall be switched on a whim.
Just like software, infrastructure/hardware/whatever is an unfathomably complex web made from science, faith, half-knowledge, geopolitics, coincidences, rules, dick moves, physics and random tweets.
I don't think we are winning anything by trying to lift software onto some kind of pedestal of specialness in every regard...
Is this possible? If code is a social construction as you say, then wouldn't it be impossible to arrive at this point unless society remains unchanging?
> Hence all the hacked together languages, tools and frameworks that people have piled on each other over the decades.
This hints at the what is IMO the real problem, software is hard to replace with confidence. Sure, it's easy to swap one version for another, but doing so with the full understanding of what will change...is basically impossible for any non-toy project today. And there are exceptions to this, but they're the "exception that proves the rule" type.
I can't remember the link but you can probably find something. the way they code is very careful and deliberate. everyone has an office, none of this "rockstar" nonsense that you have at startups, they are all serious people, the deadlines are sensible, no one pulls all nighters, they come in at 9 and get out at 5. absolutely nothing gets written without documenting it. no line of gets changed without the reason for the change written in documentation. everything is carefully panned. needless to say the code got the ship to the moon and back. this is how coding should be done
Problem is, most people are under the illusion that software is like a woodwork or metalwork gadget: you make it once and it can work happily for 10 years without changes. Which is an extra funny thing to believe in because it doesn't apply to those physical gadgets either.
It seems people just want things to be created once and used infinite amount of times and even if I can relate to that desire, it's simply not the universe we are living in.
And in spite of countless evidence to the contrary, it's still an extremely persistent illusion that creates untold amounts of friction between shareholders and programmers every day.
I don't really have a solution for this, I think tech consumers/clients are just always going to want what they want and not care so much about what it takes to make it happen. Thus they will always be surprised and grumble about the price. That just seems like human nature to me.
Absolutely. I see it the same way. After commercially programming for 19 years I've seen exactly zero progress in the human race with regards to that particular problem. It is apparently a built-in bug of our brains and everybody discovers it for themselves down the line (and 99% proceed to never fix it).
I think I recall one quote from a seemingly famous building architect (I think the guy who built the Fallingwater house, so Frank Lloyd Wright?) that was along the lines that you never put all expenses in plain view of the customer at first; you put a reasonable cost plan in front of them and then, after they are heavily invested in you, start giving them the other expenses (which are inevitable and not invented by you, let's accentuate that!). Many people call that a scummy practice but the truth is, EVERYONE is expensive, some are just better at selling themselves. So you have to give the customer the truth spoon by spoon because they prefer lies and truth is too hard to swallow at one bite.
And apparently that's how people tick. I don't like it either. But... blame the game, not the player -- as the saying goes.
But what the author is proposing would automate and reduce the cost of the support. A company that could achieve this cost saving and market or successful would have an advantage. Unfortunately it is a very hard problem abs many upgrade transformations can’t be proved to be correct with good automated tests. And then you have the problem of how do you automated the maintenance of the tests.
So not sure how such a thing could be monetised properly. Likely by only offering enterprise-grade services: the tool itself is free, the customer's programmers could employ it to achieve migration but the customer prefers to outsource the cost. Sort of how several Linuxes are free but come with paid support contracts.
Imagine you brought your car for an oil change and the mechanic says "uhh yeah maybe you can get new oil next year but for the next few months we need to... Uh... Address technical debt in your car... So umm... It's gonna cost you thousands of dollars and you aren't going to get anything you perceive as a change"
You'd think they are ripping you off.
However, as a counterpoint to yours, even those of us who can communicate quite clearly get met with very ugly accusations of ripping customers off. People simply come to you with a preconceived notion about how much something must cost (in terms of money, effort and time) and they are absolutely not willing to change that notion -- even if they are on the side of the pure imagination and you are on the side of the facts.
It's basic psychology that many of us had to learn even if we never wanted to.
Automating fixes is only possible if you understand fully where the problem is. Then, it still needs testing does it break something else. For testing, automatic tests most likely do not test for everything. And someone would need to create those tests.
There is also a lot of existing code for existing software. Would it be possible to analyze or convert that amount of code to be somehow updateable?
Software is usually made with some programming language and related dependencies. Those dependencies don't necessarily exist for other programming languages and web frameworks.
Like python’s “2to3”.
It’s not a magic bullet; the developer still benefits from thorough testing, but I think changelogs could stand to improve in a lot of projects.
And unless the old way was genuinely bad/hard, why deprecate?
We should, however, have these kinds of discussion because I don't think we should outsource this debate to corporations. But this shouldn't take the tone of contempt and resentment towards what we already have.
And I don't want to only be paid an average salary for it.
But... an excellent salary (above-average, maybe even above-high) would motivate me and no, I won't try to copyright that work and mooch off it for billions until I hit 80.
As I said: incentives. Many companies expect programming to be like installing a kitchen sink yet are unwilling to pay to commoditize the activity.
Even if you had a lot of financial support, how would you go about implementing the standards and transformations described in the article? Who would adopt it?
Maybe in the context of a company this could reap benefits and become a case study. Or in a partnership between many companies there could arise such applications. But they would still be locked to a niche. That is why I think even with financial support, the problem is maturity, or lack thereof.
You could argue that once a large enough group adopted practices that proved to be effective in reducing maintenance time and complexity, that others would follow, but that stops being a matter of pure financial support and starts being a matter of adoption and convergence, pointing once again to maturity of the ecosystem.
And another aspect of maturity, standardization and sanitization is that it comes at a cost to diversity, which in turn has benefits of its own, like new ideas and technology. This is kind of the balance between exploration and exploitation we see in training neural networks. I think software is in the transitioning phase from exploration to exploitation and we are living it.
IMO if you make such a tool for something extremely popular like Python or Javascript and then start pivoting upwards from there, people will be coming to you. There are a lot of deficiencies in the everyday work of the programmer to this day.
I have a friend that contracted for a company doing y2k work. The main guy have written a system to parse cobol, do pattern matching to find y2k issues, apply transforms to the AST to fix the problems and the serialise back to cobol.
It supported a dozen of so cobol variants. I think it was very successful.
I am clipping this for use in future internal discussions. I have trouble communicating this point, this sentence does so concisely.
We've got to stop thinking about software in agile sprints once a product exists. I'm not saying don't use agile for the build. But we've got to learn as software professionals to think like professions that maintain other expensive assets, such as aircraft, ships, or public works. There's a recognition, that a 50 year dam, needs a tear down and replacement in 50 years.
Notice that this includes using code written by other people (e.g., dependencies).
But using off-the-shelf software that you simply install is certainly one form of "using code written by other people." If it is popular software offered by a stable company, I would always estimate it's likely future cost as less than anything written in-house. (Which doesn't mean it's risk free, the software can become unsupported or change in backwards incompat ways incurring costs. You can get unlucky. But I'm almost always going to advise that it's the safer choice with regard to future costs than in-house development).
If we're talking about in-house software projects to begin with, as you probably intended, but evaluating using a third-party dependency vs writing the functionality in-house, figuring out which of those paths will be likely to end up lower-cost is not at all easy to do, and dependent on a lot of context.
But certainly in all cases, not having any software will be lower-maintenance-cost than having any software. Well, at least lower software maintenance cost. :)
Take Scala for example. There is now mature tooling for keeping your dependencies / libraries up-to-date automatically, using automatic migration-scripts (must be provided by the library author of course).
See here: https://github.com/scala-steward-org/scala-steward/blob/mast...
The difficult part here is of course to write the migrations. This works very well in Scala (and can work as well in certain other languages) because the type-system provides enough information to do automatic rewrites and it is easy to _not_ use unsound techniques such as reflection, code generation, macros etc.
The reason why we don't see such tooling in other languages yet is that they are either dynamically typed, which makes it almost impossible to write migration scripts that pretty much always work. Or they are statically typed but the typesystem is so limited that developers have to fall back on mentioned unsound features.
...Or at least I hope that will be the case in the future. But like you, I believe it's much better positioned to do so compared to the dynamically-typed languages.
(Actually, n log(n) + a constant offset; i.e. partitioning the code has its limits where it becomes detrimental.)
Pick technologies that are stable over time frames measured in years, not months. You will never be able to construct any great works on top of fields of quicksand.
For example - Instead of using a fun & shiny new language you read about on HN to construct your company's new loan processing backend, perhaps consider technologies such as Java or .NET. Instead of using a database engine that was developed this decade, look at something that has already stood the test of 1-2 prior decades.
Now, 100% of everything I have just said is totally irrelevant if we are talking about software engineering just for the sake of it (fun, hobby, exploration, etc). This is the important distinction. If the livelihoods of your coworkers depend on you not making a bad technology choice, try to go with the safe bet. Separation of hobby & work mindset can solve a lot of this mess.
Testing / Certification: A table lamp will be UL/CE approved. This means it will be tested in such a manner that normal use and abuse will not cause unexpected side effects. Knocking a UL approved lamp off a table will not cause your house to get burned down. A light bulb of rated capacity will not cause a short circuit. Normal wear and tear will not cause it to become unsafe.
Outlets: A table lamp plugs into an outlet, of a standard size and characteristics. The outlets are a standard interface that is often nation or region wide. The voltage and current that can be delivered through an outlet are standardized, as is the nature of the loads which may be plugged into it. The outlet itself, and the housing, and wiring are well regulated.
The outlet is connected to a circuit, and a circuit breaker (or fuse). These devices completely and permanently interrupt the delivery of voltage for a number of standard conditions, including over-current, and also ground faults.
The circuit breaker, or fuse socket, is part of a standard and well understood panel, which is designed to allow a variety of options in deployment, with electricians able to configure them in a wide variety of applications. Yet, the user of a breaker of fuse panel is protected from the voltages and power inside, and given a standard and easy to understand model of how power is distributed, and controlled. In fact, some advanced users can use the panel to disable part of their local power network to allow maintenance and modification in a safe manner.
Breakers and fuse panels allow for lock out, which will physically prevent the reapplication of power, while maintenance is ongoing.
Power feeds include metering, and themselves are driven from a circuit, and the system is designed in much the same way as the residential and business circuits, but on a larger scale.
Not only all of those are true, but there is more
Circuit breakers and fuses on devices, etc.. are all designed to be coordinated. The smallest fuse/breaker in the chain from source to load should always trip first. This prevents a scenario such as the one where a person plugging a defective toaster into an outlet brings down the entire power grid.
Circuits and power routing have well separated areas of concern. There is no way that a power system problem can directly cause problems with the sewers or other infrastructure, other than failing to deliver power.
Our software infrastructure lacks all of this sophistication and standardization. You can plug in a fan made 100 years ago into a modern outlet, and it will work. You can't even run an MS-DOS program from the 1980s without resorting to an emulation layer.
You can know that same fan will not cause Russian bots to be able to control the power grid. Nothing plugged into an outlet can ever give you control over the grid. The same is not true of software, no matter how carefully constructed. We're still at the stage where a bad program can take out its host operating system, and then the network its on.
To say we need more skilled programmers, better aware of the security implications of their work, is to insist that we could run a power grid without circuit breakers in every home.
The operating system has the job of protecting the network and its users and applications from each other. There is no equivalent to circuits and breakers. You can't run a program, and know that it will only consume X amount of CPU, network I/O, or RAM, unlike the outlet limited to 15 amps.
You can't be sure that a program won't have side effects. Plugging in a bad toaster in the kitchen shouldn't effect a circuit in the garage. The damage should be limited by default. None of our Operating Systems do that.
Computation lacks maturity, especially our operating systems. We have no stable standards for anything important.
That, in my opinion.. is the real root issue.