Is there any human activity where quality is an attribute successfully taught? In my experience, being able to produce something of quality is gained only through practice, practice, practice.
Good teaching largely consists of setting the learner up in situations where they can practice effectively. To pick just one example many people are taught to improve the quality of their writing. This largely consists of giving guidance on what writing to attempt and (more importantly) guidance how to reflect on the quality of the writing you've just done so you can improve.
> Is there any human activity where quality is an attribute successfully taught?
Every industrial practice.
On the other hand, the title just means that programming is not an industrial practice. What should be obvious to anybody that looked, but some people insist on not seeing it.
For pilots, there are many filters to ensure people that failed to learn can't take many responsibilities. They ensure the pilots study and train, but there isn't any theory making sure the pilots learn and get the best safety practices. (In fact, if you are comparing with CS teaching, pilot teaching will give you a heart attack.)
For engineers, the situation is very similar to software. There are many tools for enforcing quality, but there's no structure for teaching the engineers, and no, there isn't a widely accepted theory for how to teach design quality either.
The one place where people are consistently taught how to build quality is on manufacturing.
Aviation in particular has a very strong culture around (government mandated) checklists and post-crash investigations. This has both pros and cons. The pros is that every airline learns from the mistakes made by every other airline and over time the system becomes really quite safe indeed. The cons are that it is quite expensive and time consuming.
Imagine if every software company was obliged by law to:
- Every single release has to have been signed off by someone who got their "software release engineer" certification at the software equivalent of the FAA.
- This engineer is required by law to not sign off unless every box on a 534 item checklist has been manually verified.
- Any time an unplanned downtime happens at any company, a government team comes in to investigate the root cause and add points nr 535 through 567 to the checklist to make sure it never happens again.
If such a system was mandated for software companies, most of the common bugs would very rapidly become a thing of the past. Development velocity would also fall through the floor though, and most startups would probably die overnight. Only companies that could support the overhead of such a heavyweight process would be viable, and the barrier to entry would massively increase.
I wish someone would create that 500 line checklist. I've seem attempts, but they tend to be either not actionable (is the software high quality - meaningless), or of metrics that are just gamed (is test code coverage > 80%?)
If I understand his theory correctly, in your case there would be a competing metric to the "test coverage" one that said for any changeset, a test cannot itself change by more than 20% in the same changeset as non-test code. So you can change the code such that it still passes the existing tests, or you can change the test to adapt to new requirements, but you cannot rewrite the tests to match your newly changed code
I'm acutely aware this is a terrible example, the devil's in the details, and (in my experience) each company's metrics are designed to drive down their own organizational risk <https://en.wikipedia.org/wiki/Conway%27s_law>, combined with "you're always fighting the last war" :-D
> > Aviation in particular has a very strong culture around (government mandated) checklists and post-crash investigations
That's the reason why aviation can only shine when it becomes a private means of transportation, and I don't mean 70mm private jets but, 150k light helicopters.
When a critical mass is hit then accidents will become no more traumatic to the collective psyche than car accidents, the lighter the aircraft the better because it would seem exactly like a car crash as opposed to leaving a huge burning hole into the ground
Citation needed. The key to the industrial revolution was trivializing the human work so as to take as many human errors out as possible and to systematize everything. I wouldn't call that type of process "teaching quality".
The arts. The further and further you go in instruction, the more it becomes about the little differences and quality. Practice always helps, but quality definitely taught and learned by many as well.
I cannot fly professionally anymore due to health, but this is something we are taught in aviation and something I too find lacking from tech so far.
Like, you’re taught the standards as part of learning to fly, but as time goes on, you’re told to narrow your tolerance of what is acceptable. So if you are learning how to do steep turns, for instance, the altitude standard is +- 100’. You’re taught, “that’s the minimum, you should be trying for 50’” and then 20’, then the absolute best performance would be where you do your turn, the needle doesn’t move, and as you roll out on a calm day you fly through your wake. But the goal is “better, always better, what can I do better?” And flying is not graded on the overall performance, if you don’t do satisfactory everywhere you fail. Culturally satisfactory, isn’t, it’s the starting point.
That encourages a much more collaborative model I feel like. I’ve only worked one or two flying jobs that were not collaborative. In the outside world it sometimes feels the opposite. In flying, you truly want everyone to succeed and do well, and the company does. Even the guys I hated that I flew with, I didn’t want them to fail. If they failed, I was partially responsible for that.
It wasn’t always perfect, and I worked with some legendary assholes while I was flying, but truly, they supported me and I supported them, and if I screwed up (or they screwed up) the culture required that we found a way to minimize the future potential screwups.
You’re actually trained on what quality means in a wide variety of contexts too, and flight operations quality assurance (FOQA) is a big part of many airlines. In smaller bush operations where I worked, it is significantly more informal, but we truly had a “no fault” culture in nearly all the places I worked. It’s not perfect, but that’s the point, “ok how can we make this better?”
If someone had an idea for how to do something better, there may have been friction, but that was rare if it actually was better, and as soon as you could show how adoption was fast even at the less standardized operations I worked at.
Not saying it’s all unicorns and rainbows, but I feel like quality, and decision making, and “doing the right thing” were an integral part of the culture of aviation. “The weather is too bad and I cannot do this safely” grounds the flight, you don’t blast off into the shit (at reputable operators) to get the job done anymore (it’s not 1995), and it feels like this new industry is the opposite.
The entire concept of a “minimum viable product” is somewhat illustrative of the problem. It shouldn’t be the “minimum viable” it should be the “minimum quality product we’re willing to accept as a starting point.” But that doesn’t roll off the tongue in the same way.
We shouldn’t be striving for anything that’s the “minimum.” The minimum is only the beginning.
> But the goal is “better, always better, what can I do better?”
Is that not the case in software? The incentive to improve may not be quite as strong as in aviation (crashing software isn't quite the same as crashing airplanes), but it is still pretty strong. Life is very miserable in software when quality isn't present.
What happens when you work under a group of people who are satisfied at stage one of project X? You know you can iterate to get two stages further, but they want you to work on projects Y and Z. This is a very common situation where you, or even the whole development team has very little control.
Of course, management should be supportive of quality improvements, but their reality is either one where they are under genuine pressure to deliver projects X and Y to stage of quality through to not understanding or caring about quality.
My own experience is that individual programmers have vastly different ideas of quality is based on their experience and education. You can be struggling to get a team to improve and then you hire a somewhat normal individual with a very different background who makes a sizeable impact on quality and the culture of this in the team. I'm thinking specifically of someone who joined from aerospace, but I've seen it with finance backgrounds. I think the background matters less than the perspective and ability to hold people accountable (including yourself.)
> What happens when you work under a group of people who are satisfied at stage one of project X?
No doubt the same as when the members of your garage band are happy to stay in the garage while you have your sights set on the main stage. You either suck it up and live in the misery, or you get better on your own time and leverage those improvements in the quality of your performance to move into a better position where quality is valued.
Practice only matters if you try to produce quality. If you just practice producing crap you'll only get good at producing crap. But I suppose if someone doesn't care about quality (and these people do exist) all bets are off really.
or maybe, them who know how to do this are just unable to spread this knowledge... something about how they think their private secret codes are the source of their wealth
when in fact, it's merely the scheme by which they mantain an advantageous capacity to extract energy from them seeking to learn how to build quality software
There's an obvious comprehensibility complexity to code to anyone who has spent almost any time what so ever trying to make something happen in software. However, we've got zero academics or theory around it.
Just 'best practices' (ie a thing other people are known to do so if things go wrong we can deflect blame).
And code smells (ie the code makes your tummy feel bad. yay objective measures).
And dogma (ie "only ONE return point per function" or "TDD or criminal neglect charges").
Sure, please do something for QA because it'll be better than nothing. But we're probably a few decades of waiting for actual theoretical underpinnings that will actually allow us to make objective tradeoffs and measurements.
You may still not buy into it, but note that single exit was established for languages like C where an early exit can make it difficult to ensure that all resources are freed. It isn't meant for every language – and, indeed, languages that are not bound by such constraints usually promote multiple exit because of the reasons you bring up.
And even that is wrong, single entrance/exit was originally because you had subroutines designed to be goto'd into at different points for
different behavior and would goto different points outside the subroutine as the exit.
There are pretty much no languages left today where it's even possible to violate this principle without really trying, it's not about having single a return it's about all the functions starting at the top and return statements always taking you back to the same place in the code.
There is plenty of academics on it, as real engineers, those that studied Software Engineering or Informatics Engineering, instead of fake engineering titles from bootcamps, should be aware.
Usually available as optional lectures during the degree, or later as Msc and PhD subjects.
Although, so far I've only bumped into cyclomatic complexity (with some studies showing that it has worse predicting power than lines of code) and lines of code.
I don't know. I was hoping for something like: "We know inheritance is bad because when we convert the typical example over to this special graph it forms a non-compact metric space" Or something like that.
Even though I find cyclomatic complexity uncompelling, it at the very least can slurp up code and return a value. Nicely objective, just not particularly useful or insightful to whether or not things are easy to understand.
The provided link looks suspiciously like they're going to talk about the difference between system, integration, and unit tests. The importance of bug trackers. And linters / theorem provers maybe.
I don't think these are bad things, but it's kind of a statistical approach to software quality. The software is bad because the bug chart looks bad. Okay, maybe, but maybe you just have really inexperienced people working on the project. Technically, the business doesn't need to know the difference, but I would like to.
I don't suppose you know where I can get their list of references without hitting a paywall? Specifically [16] and [24].
EDIT: [For anyone following along]
The linked paper is Measuring Complexity of Object Oriented Programs. Although, the paper isn't free. They reference several other papers which they assert talk about OO complexity metrics as well as procedural cognitive complexity, but unfortunately the references aren't included in the preview.
Apparently, there's also a list of Weyuker's 9 Properties which look easier to find information on. But these look like meta properties about what properties a complexity measurement system would need to have [interesting, but they don't really seem to comment on whether or not such measurement is even possible].
It looks like a lot of this research is coming out of Turkey, and has been maybe floating around since the early 2000s.
EDIT EDIT: References are included at the bottom of the preview.
EDIT EDIT EDIT: Kind of interesting, but I'm not sure this is going to yield anything different than cyclomatic complexity. Like, is this still an area of active research or did it all go by the wayside back in the early 2000s when it showed up? The fact that all the papers are showing up from Turkey makes me concerned it was a momentary fad and the reason it didn't spread to other countries was because it doesn't accomplish anything. Although, I suppose it could be a best kept secret of Turkey.
Renamed programs are defined to have identical complexity, which is pretty intuitively untrue, so I've got my concerns.
EDIT ^ 4: Doesn't seem to be able to take data complexity into account. So if you're dividing by input, some inputs are going to cause division by zero, etc. You might be able to jury rig it to handle the complexity of exceptions, but it looks like it can mostly handle static code. I'm not sure if it's really going to handle dynamically calling code that throws very well. I also don't think it handles complexity from mutable shared references.
Nice try, but unless there's a bunch of compelling research that no actually this is useful, I'm not sure this is going to cut it. And at the moment the only research I'm finding is more or less just defining functions that qualify as a cognitive measure under the Weyuker principles. I'm not seeing anyone even pointing it at existing code to see if it matches intuition or experience. Happy to be found wrong here, though.
I wish more ppl felt this way. What a compliment it is to oneself when I hear ppl saying "write clean code" as if they know its address and had dinner with clean code just last night.
I was thinking there should be some metric around d(code)/dt . That is, as the software is used, 'bad' code will tend to change a lot but add no functionality. 'Good' code will change little even when it's used mode.
d(code)/dt isn't a very good metric though. Think of the Linux kernel. Drivers get some of the least maintenance work and are broadly the lowest quality part of the kernel. arch/ is busier than drivers/, but anything you find in the parts being touched are also significantly higher quality.
The scientific groundwork for excellent testing, anyway, has already been done-- but not in the realm of computer science. This is because computer scientists are singularly ill equipped to study what computer scientists do. In other words, if you want to understand testing, you have to watch testers at work, and that is social science research. CS does not take social science seriously.
An example of such research done well can be found in Exploring Science, by Klahr. The author and his colleagues look very closely at how people interact with a system and experiment with it, leading to wonderful insights about testing processes. I've incorporated those lessons into my classes on software testing, for instance.
> you cannot be taught what nobody knows how to do
It's worse than that. No one can agree what "quality" means.
Mostly, the word is used as a weapon.
The pointy end of the weapon is what management pokes you with whenever anything unexpected happens. Typically they do this instead of making sure that problems do not happen (a.k.a. "management").
The weapon's handle is sometimes flourished by process gatekeepers who insist on slowing everything down and asserting their self-worth. This is not good for throughput, anyone else's mood, or eventually even for the gatekeepers.
People usually refuse to talk about quality in terms of actual business metrics because if anything unexpected happens that's not covered by the metrics, there will be fault-finding. And the fingers pointed for wrong metrics are typically pointed at middle management.
I was actually thought how to build quality software (which is not limited to "having no bugs") in college, but I do not have the time or resources to apply this knowledge consistently in a corporate setting because of the pressure to deliver.
Because frankly, too much quality is not necessary, in many many cases. To know when you should or should not emphasize quality over quantity and speed, to meet a certain financial objective, is actually harder than writing quality software in the first place, I think.
I agree in principle, but in my experience quality is not nearly prioritized highly enough. There is not enough understanding of quality attributes beyond the most visible ones like compute performance and stability (i.e. lack of bugs). And even for those I work on projects where people complain about lack of proper test coverage constantly, but it is impossible to dedicate time to improve that.
I'm pretty sure even those basic two, of performance and stability are extreemely undervalued, when you objectively look at how fast modern hardware is and yet how easy it is to find slowness simply in using devices in a day to day environments
This is false.
Its just that the costs of low quality code are much less obvious and harder to measure then the dev time.
But the ammount of bad code just piles on itself over and over and over and we end up in a world where hardwares becomes incrementally faster while software becomes slower and slower, and more bugier.
I mean, in the strict sense of the world an individual company will not pay those costs, but on a societal scale, how much time (and thus money) is wasted daily by all the people who are waiting 30secconds for windows explorer to load? If your app have millions of users, literally every additional second your app wastes multiplies to tangential numbers.
It's akin to pollution, really:
Individual company making 'dirty' things won't see the consequences.
But scale this mindset out and suddenly we wake up in a world when trillions of dollars are spend to counteract those effects.
I wonder where you get the confidence to make such a strong statement, which is clearly not warranted. I want to challenge you to broaden your view a bit: Not a lot of software is like Windows explorer. Not a lot of software is performance critical. A lot of software can do with bugs, with many many bugs, to be honest. A lot of code is run fewer than 100 times, before it's retired. Also, not a lot of software written has many users. Or enough users to support maintaining it. "Pollution" often affects just the author of the software themself. Software is just a means to an end, and the end decides, how much effort was warranted in the first place.
> Obviously we aren't taking about some simple automation scripts here.
This is moving the goalpost, and also ignores the fact that software exists on a spectrum from "simple automation script" to "messaging app used by millions". It seems you have a very narrow view of what software is, or what it is used for, and the constrains that apply when building it.
This is not moving a goalpost. Running a program less then 100 times total, across all its user, is just very little for anything that could be considered commercial.
That really isn't a controversial statement.
So I am simply excluding this category as an extremum.
What software are you running that gets less then 100 usages before it gets retired?
>Great aproach
Unironically better, then trying to make prescriptions as broad and general as possible, because those usually are too generic to carry any actual value
Yearly reports. They can be buggy, can be off by millions, due to rounding errors. They can crash. They can run for days. Nobody cares enough to rewrite them, because regulation will change before that effort amortizes.
Also note that I wrote "code" originally, because there can be programs which are run very often, but certain code paths are not, so my statement applies even for some parts of popular software.
The image I think would be valuable for you to consider is a curve, where 20% of code has 80% of all executions, and 80% of code get's the rest. It makes sense to put in a lot of effort into writing the top 20%, but on any given day it is very likely you'll be working on the lower 80%.
> 'If we don't do it now, development efforts (and therefore also costs) will be up 15% in 4 months.'
Yeah you won't get to a point where you'll have a valid-enough metric to make this point.
I was at a startup once. The two founders said "don't write unit tests". I wasn't going to argue with them. I understood what they really meant. We've been too slow, we need to ship as fast as possible. I shipped fast and I shipped quality (ie low defects and outages). I wrote unit tests. They didn't need to know. They just needed the outcome.
The elephant in the room in all of these conversations is that you walk into any software development shop and they just don't how to ship both fast and at quality. No matter how much an organization or team tries to revisit/refactor their dev process year-to-year, they're still shipping too slow and mediocre quality.
The truth is there isn't a magic formula. It's an individual craft that gets you there. It's a team sport. And the context is different enough everywhere you go, yeah, sure, some lightweight processes might abstract across the universe but very little bang for those bucks. Far beyond any other facet of things, you really just have to have a good team with experience and the right value-system/value-delivery-focused wisdom.
There are Computer Engineering programs and a few universities that really emphasize internships and hands on practice. But at many universities, the CS department came out of the Math department and is focused on theory. Chemistry isn't Chemical Engineering either. I think that's okay. University isn't just a trade school--the idea behind almost any degree is to train the mind and demonstrate an ability to master complex material.
Yeah but at those internships you aren’t taught how to build quality software, just how to ship a SPA that connects to an API in 15 weeks (or you’re not hired).
It is a good peek into the professional software world though!
Before you can write quality software you need to be able to write large software. Most interns I see are learning how to work on non-trivial programs as this is their first chance to see something non-trivial. Then they get a "real job" and are shoved into extremely large programs.
Writing a thousand lines of bug free code isn't that hard, so the need for QA practices won't be apparent. Then you get the ten thousand line intern project and discover things are not always that easy. Then we throw you into a multi-million line project and good luck: you need a lot of that QA stuff to make it work.
What society needs is a mix of trade school a traditional university. If a university is not providing both they are failing everyone. (except the straw-man rich kid who will inherit a lot of money but not be expected to either also inherit/run a company or pass the money onto their kids - this is something that happens in story books but doesn't seem to be real world where the rich give their kids lots of advantages but eventually expect them to take over and run the family business)
A pure university education without considering is this degree useful in the real world is a disservice to education. However a pure trade school education that teaches how to do something without understanding is not useful (I don't think any trade school is that pure: they tell you to ignore hard stuff but generally give you deep understanding of some important things)
> If a university is not providing both they are failing everyone.
Why?
> A pure university education without considering is this degree useful in the real world is a disservice to education.
I think this line of thinking is a much bigger disservice to higher education. It was very tiresome as an undergraduate to be surrounded by people that thought this way - and detrimental to everyone's education.
"I'll never use this knowledge" is the single worst thing you can say as a student, and it needs to be beaten out of undergrads' heads. Not encouraged.
I agree with you in principle, but it's very easy to have this attitude when the education isn't obscenely expensive.
Which is why the "I'm never going to use this, what a waste of time" feeling among American undergrad students is so common.
If you fix the affordability problem and bring it back to where is was in the mid 70s (inflation adjusted) I think things would be a lot better.
My point is that higher education isn't job training and doesn't pretend to be, and people who think it is or should are the ones that need education the most because they don't seem to get it.
>My point is that higher education isn't job training and doesn't pretend to be, and people who think it is or should are the ones that need education the most because they don't seem to get it.
That was true 50 years ago, but employers turned it into job training. My father in law retired a well off businessman with a History degree from Yale he got in the 50s. You know what a History degree from Yale qualifies you for today? Teaching History and maybe writing some books. The degree didn't change and Yale didn't change.
I'm not sure about this part... A very common pattern in my conversations with working class friends and family from my parents' generation is: "we were told that if we sent our kids to college, they'd have better lives than we did, but instead we all just ended up with more debt than we could handle".
It's tricky! If you tell teenagers and their parents the truth - this purely academic program will not train you for any job besides pure academia, which, while it can be a fantastic career, is a super risky hits business in which only a few will truly succeed - then that's only going to sound like a reasonable risk to take for wealthy families. But then you've badly limited your pool of academic researchers to this extremely small and honestly often not as promising set of rich kids.
Maybe one solution (which is not workable in the real world) would be: any academic program that does not have a viable "job training" component should only accept students on academic scholarship, regardless of their own means. If some neutral party thinks they are promising enough in that field to pay their way, they get to go for free, otherwise they don't get to go at all. The programs that do graduate people with directly marketable job skills could keep working the current mercenary way.
The reason this wouldn't work in reality is that the wealthy would still just game the scholarships in some way. Alas.
There is a big difference in value between different degrees in the real world. Yet the costs are similar. What someone studies is very important and universities do not do a good job of telling people that.
There is nothing wrong with art/music/history. If you are interested by all means take a lot of courses in them. You can learn a lot of valuable skills which is why good universities required a diverse background of "generals" that these (and many more) fit into. However they give far more degrees in these things than are needed. (even physics gets more degrees than the world needs - but most getting a physics degree can better pivot to something else well paying).
If you want to know what a university will teach your kids, ask them. They'll even tell you without asking them - it was pretty obvious to me as a dumb high school kid on campus visits what the emphasis of one program or another was going to be.
What I'm saying is: universities are incentivized to mislead people (including themselves!) about this.
If you are a working class family with a kid who is very talented at math, and you go sit down with the counselors and ask them: If my child studies pure theoretical math, will that open them up to a life full of possibilities? they will say "yes, it absolutely will". But that's not true. It might be true, but it's a big risk. It's a risk a wealthy family can very easily absorb. But if this child from this working class family takes on this risk using student debt, it might go poorly. They might very well be good at pure math but not be good enough to go into academia. Then they might be unsure what else they can do with that degree, unable to get their foot in the door at the kinds of employers where just a general proof-of-being-smart degree is enough. And now they have debt and uncertainty about what to do.
It also might work out great! But it's a risk. And I know a number of people who feel they ended up on the wrong side of that risk.
No, I don't think that's it. I think it is simply that you have to put an awful lot of people through the explore part of the learning loop, to get a handful who will reach the exploit part of the loop, for any given subject.
99% of what we all learn in college is a waste of time for us. But we all have a unique 1% that is vital to who we become. Over time I expect that 1% to become 0.1%, then 0.01%, and for that vitality to become ever more concentrated in that sliver.
Because like it or not most people are going to university to get a better jobs. Companies like university educated people because they learn deep thinking. However they often come out lacking important skills that are needed.
Sure there are a few going to university just for the fun of it. However most are expecting a job. Thus universities should train and emphasize thinking in more specific areas.
> "I'll never use this knowledge" is the single worst thing you can say as a student, and it needs to be beaten out of undergrads' heads. Not encouraged.
This is tricky. I agree undergrads say this all the time when they are wrong but they don't know it. They have no clue what they will use and what they won't. This is something universities should figure out so they push people to avoid things they won't use. OTOH, a lot of what they are really teaching isn't the specific skill, but how to research and analyze data to find complex answers - it doesn't matter if you look at data from art or from science, what you are really learning is how to think and the specific knowledge gained is isn't important or the point (I think this is the point you were trying to make?).
> However they often come out lacking important skills that are needed.
Companies that offer the jobs are the ones that need to offer the job training.
> (I think this is the point you were trying to make?)
Not really, it's that university education is kind of meta/self serving (the goal is not to train X number of students to do Y jobs, it's to give every student at the institution what that institution defines to be an education).
But like you said, the fact this produces better workers is a second-order effect. It's not the goal of most institutions. But not all institutions; some define "well educated" to have lots of industry practicum, and if you want that, go study at those institutions.
My main point is that it's not a "disservice" to eschew practicum or industry training as an educational institution.
What society needs is the second order effect though. I don't care about education for the sake of education, I can for what education can do for me/society. Now some of what most institutions define as a good education is good for society (the ability to think is very useful), but I don't value/support education because of arbitrary definitions that an institution might come up with. I value/support education because people who have education tend to show specific abilities in society that I want more people to have. The more universities are in line with that and try to produce that the more I value/support them. (note that I didn't not formally define what those things are - this is a tricky topic that I'm sure to get wrong if I tried!)
When institutions allow student to take degrees that society finds less valuable (art,music...) they are doing society a disservice by not producing what society needs. Now if the student is wealthy (not rich) enough to afford that the price then I don't care: I don't need to impose my values on anyone else. However most people in a university are not that wealthy (most are young) and so if the degree granted isn't valuable to society the university robbed that student.
>When institutions allow student to take degrees that society finds less valuable (art,music...) they are doing society a disservice by not producing what society needs.
1. what's wrong with a student pursuing their own personal goals? A person doesn't need to produce for society's sake.
2. despite that sentiment you hold, it's clear many people do value art and music. Maybe not in its pure form, but those artists do in fact fuel industries worth billions. Clearly "society" values something that requires such skills and thinking.
> Companies like university educated people because they learn deep thinking.
No. Companies love hiring higher-ed graduates because it removes a lot of cost and risk for them:
- hiring only people with degrees weeds out everyone unable to cope with a high-stress environment, for whatever reason - crucially, also including people who would normally be protected by ADA or its equivalent provisions in Europe.
- it weeds out people in relationships or with (young) children, which makes them easier to exploit and reduces the amount of unexpected time-off due to whatever bug is currently sweeping through kindergarten/school/whatever. Sure, eventually they will get into relationships and have children as they age, but looking at the age people start to have kids these days [0], that's a solid 5-10 years you can squeeze them for overtime.
- it saves companies a ridiculous amount of training. The old "tradespeople apprenticeship" way is very cost-intensive as you have to train them on virtually anything, not just stuff relevant to the job, e.g. using computers and common office software. Instead, the cost is picked up either by the taxpayer (in Europe) or by the students themselves in the form of college debt. The latter used to be reserved for high-paying jobs such as pilots who have to "work off" their training cost but got compensated really well, nowadays it's standard practice.
- it keeps the employee diversity relatively homogenous. There is a clear bias towards white and asian ethnicity in the US for higher ed [1], and among top-earning job, males still utterly dominate [2].
- related to the above, it also weeds out people from lower economic classes, although at least that trend has been seriously diminishing over the last decades [3].
Strongly disagree with this. If a class (at any level) is strictly teaching "the subject" then that is a very good issue to raise by a student or anyone else. Great teachers don't just teach the subject though, they teach the skills necessary to engage with the subject and then apply them to said subject.
Unfortunately many programs are not designed this way and learning the appropriate skills is left as an exercise to the student usually in a sink or swim approach. So some students come out with the meta skills that a university education is touted for and others do not.
I do agree that "I'll never use this knowledge" can be a miserable attitude to have or engage with - especially when it's just a proxy for "I'm not interested in learning, just in getting good grades" but the idea itself is valid.
>"I'll never use this knowledge" is the single worst thing you can say as a student, and it needs to be beaten out of undergrads' heads.
Everyone will think differently. I've never truly be research-minded and there's very much a bunch of odd classes that felt like a waste of my money (something to consider as education gets more expensive). But I do agree that there should be a space to foster researchers and especially one to overall round out a student, even if that space is more niche. I just don't think that everyone needs to go far into debt for that experience if they just want job training.
So I too desire a more explicit divide than "research university vs. industry university" and wish there were some better trade schools focused on software (not 6 month boot-camps. Think of a condensed university program without requirements of electives and maybe less supporting classes). But no one seems to be protesting this much.
> If you believe you can ship bug free code, it's time to switch careers.
Unfortunately, you are correct. Shipping in time and bug free are inversely proportional, and in a world were usually it's hard to argue with PMs for more time to have better testing, or paying tech debt... it's just a reality
An infinite amount of time would not necessarily yield zero bugs.
But more importantly, once you've fixed the "show-stopping bugs," putting the software in front of customers is probably the best next step, as even if it's bug-free, that doesn't mean it solves the problem well.
there is no such thing as zero bugs. There is only a marker in time for a suite of tests that show no bugs. Doesn't mean larva aren't living under the wood. You can't control all the bits (unless you built your own hardware/software stack).
I think we're saying the same thing? That was my point. You're never going to achieve zero bugs no matter how much time you give yourself. Focus on getting the critical path right and creating a good experience, and then get it to customers for feedback on where to go next.
[The above does not necessarily apply in highly regulated industries or where lives are on the line]
I would say that also applies on highly regulated industries or where lives are on the line.
On those you're of course expected to do safety and testing up to the limit of the "value of a statistical life"s within the expected project impacts, but it still has time and budget limits.
The part I was suggesting does not apply is the statement "Focus on getting the critical path right and creating a good experience, and then get it to customers for feedback on where to go next."
Most software engineering is about making sure the happy path works well. When lives are on the line, you need to also plan to minimize the possible damage that can happen when things go wrong.
I like to think of "zero bugs" as the asymptote. As you spend more time, you discover increasingly fewer (and less significant) bugs per unit of time. POSSIBLY at the limit of infinite time you hit 0 bugs, but even if you could, would it be worth it? Doubtful.
I can think of far better ways to spend infinite time.
0 bugs is actually impossible. A cosmic ray can flip a bit and change the behavior of your software. We live in a fundamentally unreliable universe.
We aren't taught how to write reliable software because very few people know how to write reliable software. It doesn't help that academia has a hard crush on OOP, which is a bag of accidental complexity - and complexity is a breeding ground for unreliability.
I think if a cosmic ray flips the bit and changes the behavior of your software, you can still reasonably brag that you wrote 0-bug code. It's not your fault that happened, you didn't do that. The code you wrote had 0 bugs.
Yup, I also like how you call out "get it in-front of customers" as a step in the whole chain. Often sorely missed. Sometimes a bug to you, is a feature to them (gasp!)... so either make it a first class thing or train them on the correct path (while you fix the "bug").
Ok, I think we’ve gone too far. There absolutely is such thing as 0 bugs and sometimes code changes don’t have bugs. That is not to say it can be garunteed.
We need to define bug but if bug is anything a customer (internal or external) is not happy with that passes triage and you can’t throw it back in their face. Then zero bugs would be impossible with even infinite time.
That's only true up to a point. There are some quality assurance and control activities that are essentially "free" in that they actually allow for shipping faster by preventing rework. But that requires a high level of team maturity and process discipline, so some teams are simply incapable of doing it. And even in ideal circumstances it's impossible to ship defect free software (plus the endless discussions over whether particular issues are bugs or enhancement requests).
yeah, it's a spectrum. Clearly no one is expecting an app to be truly bug free if the underlying compiler itself has bugs. But how often do users truly run into compiler level bugs?
I think when the author says "bug free", it's from the user perspective. where bugs either need to go out of your way to trigger or are so esoteric it's impossible to think about hitting without that user themself knowing the code inside out. Games is definitely an industry where the quality of code has always dipped to a point where users can easily hit issues in normal use, and only gets worse as games get more complex. That's where it gets truly intolerable
There are tools that help, but you still need time to integrate those tools, learn how to use them, etc. If you are doing unit and integration tests, you need time to not only write those, but also actually plan your tests, and learn how to write tests. Which... needs time
This kind of wisdom only comes from experience I think. Either that or higher order think. Like the article says, most of the time testing/tdd/qa is bolt on after-the-fact. Or a big push at the end with "QA Sprints" (are you sprinting or are you examining? what exactly is a QA sprint? I know what it is).
Once you get beyond "I wrote a function" and "I tested a function" and even still "I tested a function that was called by a function over the wire", you will come to a realization that no matter how edgy your edge cases, no matter how thorough your QA, there will always - ALWAYS be 0-day "undefined behavior" in certain configurations. On certain hardware. On certain kernels. It's an assurance that I guarantee that I'm almost positive it's bug free, since it passed tests, it passed human eyes, and it passed review - fingers crossed.
You might be correct today but that’s a pretty sad state of affairs, don’t you think we can do better? Most other engineering domains can deliver projects without bugs, with various definitions of “bug” of course
I'm not sure about that. Which engineering domain do you have in mind?
Maybe show-stopping bugs are somewhat unique to software engineering, but all somewhat-complex products are flawed to some extent imho.
It might be an unergonomic handle on a frying pan, furniture that visibly warps under the slightest load (looking at ikea shelfing) or the lettering coming off the frequently used buttons on a coffee machine.
But there do exist shelves that don’t warp, when used within some reasonable bounds.
I’d also quibble about the buttons on the coffee machine. They might be properly designed, just subject to the normal wear-and-tear that is inevitable in the real world. This is not a defect, physical devices have finite lifespans.
As far as computers go… if we got to the point where the main thing that killed our programs was the hard drives falling apart and capacitors drying out, that would be quite impressive and I think everyone would be a little bit less critical of the field.
Formally verified, bug free software exists. It just costs a LOT to produce, and typically isn't worth it, except for things like cryptographic libraries and life or death systems.
As the discipline has evolved, the high integrity tools are slowly being incorporated into typical languages and IDEs to generally improve quality cheaper. Compare C++ to rust for example, whole classes of bugs are impossible (or much harder to make) in rust.
A shelve is a dumb primitive static object though. Even a simple hello world goes over a huge amount of lines of code before it is displayed on a screen, ANY one of which being faulty could result in a bug visible to the enduser. And most of that is not even controlled by the programmer — they might call into libc, which calls into the OS, which calls into drawing/font rendering libraries, that calls into video card drivers that “calls” into the screen’s firmware.
I think “hello world” is not really the simplest program in this context, in the sense that printing, as you note, involves touching all that complicated OS stuff. In terms of, like, actual logic complexity implemented by the programmer compared to mess carried along by the stack, it is really bad.
But I mean, I basically agree that the ecosystem is too complicated.
To be an engineer is to know the expected system requirements and build a product that is extremely optimized for the system requirements.
There's a saying that I think fits very well here: "Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands."
You don't want a bridge to cost 50 years and quadrillions of dollars to build, you want a cheap bridge safe for the next 50 years done in 2 years.
I would not call the resulting bridge "bug free", of course.
We can certainly do better, but it takes a _lot_ of time, effort, care and discipline; something most teams don't have, and most projects can't afford.
Bugs arise from the inherent complexity introduced by writing code, and our inability to foresee all the logical paths a machine can take. If we're disciplined, we write more code to test the scenarios we can think of, which is an extremely arduous process, that even with the most thorough testing practices (e.g. SQLite) still can't produce failproof software. This is partly because, while we can control our own software to a certain degree, we have no control over the inputs it receives and all of its combinations, nor over the environment it runs in, which is also built by other humans, and has its own set of bugs. The fact modern computing works at all is nothing short of remarkable.
But I'm optimistic about AI doing much better. Not the general pattern matching models we use today, though these are still helpful with chore tasks, as a reference tool, and will continue to improve in ways that help us write less bugs, with less effort. But eventually, AI will be able to evaluate all possible branches of execution, and arrive at the solution with the least probability of failing. Once it also controls the environment the software runs in and its inputs, it will be able to modify all of these variables to produce the desired outcome. There won't be a large demand for human-written software once this happens. We might even ban software by humans from being used in critical environments, just like we'll ban humans from driving cars on public roads. We'll probably find the lower quality and bugs amusing and charming, so there will be some demand for this type of software, but it will be written by hobbyists and enjoyed by a niche audience.
It is sad that people on here would believe this and that for whole platforms it is actually true, however, it absolutely is not universally true and the proof is all around us.
Yes, but that software is not bug-free. The claim was not "it's impossible to make software that does not exhibit bugs too a casually noticeable degree".
People who know how the sausage is made will always know of a bunch of bugs that haven't been fixed exactly because they aren't impactful enough to be worth the effort required to fix them.
If it works within specs it is bug free. It doesn’t matter how it is made if it works within specs, which is one of the real unfortunate truths of software.
The other is working out the correct specification is far harder than coding is.
For example it is trivial to write a bug free program that multiplies an integer between 3 and 45 by two.
To have no bugs, which is extremely unlikely for a program of any real complexity. Having bugs, and being functional, are fairly self-explanatory and independent of each other. No need to try to conflate them.
Not sure what your quote is supposed to mean. That's a textbook example of someone who doesn't understand software at all making laughable requests of their engineers.
To be bug free we must be able to define what a bug is. So, what is a bug?
The reason for that quote is from what you have said a bug would be anything you didn't expect, even if it is consistent or not with the specification as that merely affects if we classify it as functional or not (a classification I profoundly disagree with, obviously). It is simply a negative rephrasing of what the marketing guy said and laughable in the same way.
> One plausible definition is “system deviates from its specification”
And that's quite reasonable. So I actually retract my argument.
For my own definition, I was considering a bug to be any behavior that the software engineers weren't expecting. Because those can exist invisibly for a long time until they become so bad they become visible. They can also exist for decades without causing any problems to functionality at all.
Fair enough. I considered a bug to be any behavior the engineers didn't plan in the code. They have their own specification, in their heads, that is more technical/exact than the business specification. Your definition is also reasonable but it's not what people mean when they say "there's no such thing as bug-free code", because bugs of my definition are almost unavoidable.
Most devices work within the spec 99.9% of the time, but that last .1% it is outside the spec. The exact % is different for different projects of course, but the idea is still there: no software operates according to spec 100% of the time.
Some people obviously aren't true Scotsman... I'm from the US and have no attachment to Scotland; if I claimed to be a Scotsman and you pointed out that I'm not, and I said "well that's just the no true Scotsman fallacy!", then I would be totally full of it.
In the same way I am not a real Scotsman, your toy example of an easily specified snippet of a function that doesn't do anything useful is not real software.
Sure, but adding two ints is trivial. Hello world probably operates to spec all the time too. Almost all software is vastly more compelx and isn't perfect.
As you alluded, in practice no specs fully specify a truly bug free implementation. If you want to consider bugs that are within the specification as being bugs in the spec rather than bugs in the implementation, fine, but in my view that is a distinction without a difference.
(Personally, I think code is itself more analogous to the specification artifacts of other professions - eg. blueprints - and the process of creating the machine code of what is analogous to construction / manufacturing something to those specs.)
And even having said that, even the vast majority "bug free" software that nearly always appears to be operating "within spec" will have corner cases that are expressed in very rare situations.
But none of this is an argument for nihilism about quality! It is just not the right expectation going into a career in software that you'll be able to make things that are truly perfect. I have seen many people struggle with that expectation mismatch and get lost down rabbit holes of analysis paralysis and overengineering because of it.
> in practice no specs fully specify a truly bug free implementation.
Except for ones that do, obviously.
The key reason to make the distinction is because the fuzzy business of translating intention into specification needs to be fully accepted as an ongoing negotiation process of defining exactly what the specification is, and integrated into repeated deterministic verification that that is what has been delivered. Failing to do that is mainly a great way for certain software management structures to manipulate people by ensuring everything is negotiable all the time, and has the side effect that no one can even say if something is a bug or not. (And this pattern is very clear in the discussion in this thread - there is a definite unwillingness to define what a bug is).
IME the process of automated fuzzing radically improves all round quality simply because it shakes out so many of the implicit assumptions and forces you specify the exact expected results. The simple truth is most people are too lazy and/or lack the discipline needed to do it.
Those don't exist. There are too many free variables. Some get much closer than others (for instance via formal verification), but all specs are by necessity a model of reality, not the full reality itself.
Nobody actually has any trouble knowing what a bug is. Like, this is just a non-issue, I've never in my career spent a non-negligible amount of time debating with anybody whether something is or isn't a bug. We discuss whether fixing bugs have worthwhile return on investment, and we discuss the relative priority between fixing bugs and doing other things, but this meta-debate about "well technically it complies with this spec so is it even a bug, really?" just never comes up. We all know what bugs are.
It's perfectly acceptable to let bugs escape into production if those "cost" of fixing that bug higher than the "cost" to the user experience / job to be done. A bug that takes a week to fix that will only be encountered by a small amount of users in a small number of obscure scenarios may not need to be fixed.
This presupposes that you know what/where bugs will be found and how they'll impact future users. In my experience knowing either of these is very rare at the point where you're "building quality".
I think a common error is taking this view in isolation on each bug.
Fact is, if you ship enough 'low probability' bugs in your product, your probabilities still add up to a point where many customers are going to hit several of them.
I've used plenty of products that suffer from 'death by a thousands cuts'. Are the bugs I hit "ship blockers"? No. Do I hit enough of them that the product sucks and I don't want to use it? Absolutely.
Software is commonly built on non-fungible components and monopolies.
Right, you don't want to use Microsoft Word, or SalesForce, or Apple vs Android, or X Whatever. It's highly unlikely you'll have a choice if you use it though.
Very much this, and low risk bugs compound at scale.
If you're in a very large FANNG type company, and say you have 1000 components that each ship 1 bug each day that has a 0.1% chance of breaking something critical, that translates to a less than 50% chance you ship a working OS on any given day. And that may mean the entire company's productivity is impacted for the day depending on how broken it is.
A saying that I once heard and appreciate goes like this:
"A programmer who releases buggy software and fixes them is better than a programmer who always releases perfect software in one shot, because the latter doesn't know how to fix bugs."
Perhaps similar to the saying that a good driver will miss a turn, but a bad driver never misses one.
I think you misunderstand, I'm talking about a programmer who makes perfect, bug-free code in one shot. There are no bugs to catch and fix, because this "perfect" programmer never writes buggy code.
The moral of the sayings is, that "perfect" programmer is actually a bad programmer because he wouldn't know how to fix bugs by virtue of never needing to deal with them.
To reuse the driver analogy, the driver who never misses a turn is a bad driver because he doesn't know what to do when he does miss a turn.
If a software developer consistently delivers high-quality software on time and on budget, that means they're good at their job, pretty much by definition. It would make no sense to infer they're bad at fixing bugs.
It would make sense to infer instead that they're good at catching and fixing bugs prior to release, which is what we want from a software development process.
> the driver who never misses a turn is a bad driver because he doesn't know what to do when he does miss a turn
Missing a turn during a driving test will never improve your odds of passing.
The driver who never misses a turn presumably has excellent awareness and will be well equipped to deal with a mistake should they make one. They also probably got that way by missing plenty of turns when they were less experienced.
What we are discussing isn't a real programmer we might actually find. No, we are talking about a hypothetical "perfect" programmer. This "perfect" programmer never wrote a bug in his entire life right from the moment he was born, he never had a "when they were less experienced" phase.
Obviously, that means this "perfect" programmer also never debugged anything. For all the perfect code he writes, that makes him worse than a programmer who writes buggy code but also knows how to go about debugging them.
Yep. If you have written production grade software at real companies, you know that the moment you make that new commit (even if 1 liner change), you are now ready to accept that it could break something. yes you can do your unit tests, integration test, User Acceptance Tests and what not. But every code change = new possible bug that you may not be able to catch until it occurs to a customer.
Whenever I hear a developer say "I never ship buggy code", I am always cautious to dig in more and understand what they mean by that.
There are well-formed statements that can be proved but which assert that its godelized value represents a non-provable theorem.
Therefore, you must accept that it and its contradiction are both provable (leading to an inconsistent system), or not accept it and now there are provable theorems that cannot be expressed in the system.
Furthermore, that this can be constructed from anything with base arithmetic and induction over first-order logic (Gödel's original paper included how broadly it could be applied to basically every logical system).
The important thing to note is that it doesn't have anything to do with truth or truth-values of propositions. It breaks the fundamental operation of the provability of a statement.
And, since many proofs are done by assuming a statement's inverse and trying to prove a contradiction, having a known contradiction in the set of provable statements can effectively allow any statement to be proven. Keeping the contradiction is not actually an option.
No. It merely prevents you from confirming every arbitrarily complex proof. Incompleteness is more like: I give you a convoluted mess of spaghetti code and claim it computes prime numbers and I demand you try to prove me wrong.
Or perhaps you'll prove it from first principles. Although if turns out to be difficult, that's okay. Somebody mentioned something about systems being either complete or consistent but never both. Some things can be true but not proveably so. Can't quite remember who it was though.
Gödel's really was a rather unique mind, and the story of his death is kind of sad.. but I wonder if it takes such a severe kind of paranoia to look for how math can break itself, especially during that time when all the greatest mathematicians were in pursuit of formalizing a complete and consistent mathematics.
I never really got how proofs are supposed to solve this issue. I think that would just move the bugs from the code into the proof definition. Your code may do what the proof says, but how do you know what the proof says is what you actually want to happen?
A formal spec isn't just ordinary source-code by another name, it's at a quite different level of abstraction, and (hopefully) it will be proven that its invariants always hold. (This is a separate step from proving that the model corresponds to the ultimate deliverable of the formal development process, be that source-code or binary.)
Bugs in the formal spec aren't impossible, but use of formal methods doesn't prevent you from doing acceptance testing as well. In practice, there's a whole methodology at work, not just blind trust in the formal spec.
Software developed using formal methods is generally assured to be free of runtime errors at the level of the target language (divide-by-zero, dereferencing NULL, out-of-bounds array access, etc). This is a pretty significant advantage, and applies even if there's a bug in the spec.
I think the reason that formal proofs haven't really caught on is because it's just adding more complexity and stuff to maintain. The list of things that need to be maintained just keeps growing: code, tests, deployment tooling, configs, environments, etc. And now add a formal proof onto that. If the user changes their requirements then the proof needs to change. A lot of code changes will probably necessitate a proof change as well. And it doesn't even eliminate bugs because the formal proof could include a bug too. I suppose it could help in trivial cases like sanity checking that a value isn't null or that a lock is only held by a single thread but it seems like a lot of those checks are already integrated in build tooling in one way or another.
Yes, with the current state of the art, adopting formal methods means adopting a radically different approach to software development. For 'rapid application development' work, it isn't going to be a good choice. It's only a real consideration if you're serious about developing ultra-low-defect software (to use a term from the AdaCore folks).
> it doesn't even eliminate bugs because the formal proof could include a bug too
This is rather dismissive. Formal methods have been successfully used in various life-critical software systems, such as medical equipment and avionics.
As I said above, formal methods can eliminate all 'runtime errors' (like out-of-bounds array access), and there's a lot of power in formally guaranteeing that the model's invariants are never broken.
> I suppose it could help in trivial cases like sanity checking that a value isn't null or that a lock is only held by a single thread
No, this doesn't accurately reflect how formal methods work. I suggest taking a look at the PDFs I linked above. For one thing, formal modelling is not done using a programming language.
You mix up development problem with computational problem.
If you can't use formal proof just because the user can't be arsed to wait where it is supposed to be necessary, then the software project conception is simply not well designed.
> A formal spec isn't just ordinary source-code by another name, it's at a quite different level of abstraction
This is the fallacy people have when thinking they can "prove" anything useful with formal systems. Code is _already_ a kind formal specification of program behavior. For example `printf("Hello world");` is a specification of a program that prints hello world. And we already have an abundance of tooling for applying all kind of abstractions imaginable to code. Any success at "proving" correctness using formal methods can probably be transformed into a way to write programs that ensure correctness. For example, Rust has pretty much done so for a large class of bugs prevalent in C/C++.
The mathematician's wet dream of applying "mathematical proof" on computer code will not work. That said, the approach of inventing better abstractions and making it hard if not impossible for the programmer to write the wrong thing (as in Rust) is likely the way forward. I'd argue the Rust approach is in a very real way equivalent to a formal specification of program behavior that ensures the program does not have the various bugs that plagues C/C++.
Of course, as long as the programming language is Turing Complete you can't make it impossible for the programmer to mistakenly write something they didn't intend. No amount of formalism can prevent a programmer from writing `printf("hello word")` when they intended "hello world". Computers _already_ "do what I say", and "do what I mean" is impossible unless people invent a way for minds to telepathically transmit their intentions (by this point you'd have to wonder whether the intention is the conscious one or the subconscious ones).
> thinking they can "prove" anything useful with formal systems
As I already said in my reply to xmprt, formal methods have been used successfully in developing life-critical code, although it remains a tiny niche. (It's a lot of work, so it's only worth it for that kind of code.) Google should turn up some examples.
> Code is _already_ a kind formal specification of program behavior.
Not really. Few languages even have an unambiguous language-definition spec. The behaviour of C code may vary between different standards-compliant compilers/platforms, for example.
The SPARK Ada language, on the other hand, is unambiguous and is amenable to formal reasoning. That's by careful design, and it's pretty unique. It's also an extremely minimal language.
> `printf("Hello world");` is a specification of a program that prints hello world
There's more to the story even here. Reasoning precisely about printf isn't as trivial as it appears. It will attempt to print Hello world in a character-encoding determined by the compiler/platform, not by the C standard. It will fail if the stdout pipe is closed or if it runs into other trouble. Even a printf call has plenty of complexity we tend to just ignore in day to day programming, see https://www.gnu.org/ghm/2011/paris/slides/jim-meyering-goodb...
> Any success at "proving" correctness using formal methods can probably be transformed into a way to write programs that ensure correctness
You've roughly described SPARK Ada's higher 'assurance levels', where each function and procedure has not only an ordinary body, written in SPARK Ada, but also a formal specification.
SPARK is pretty challenging to use, and there can be practical limitations on what properties can be proved with today's provers, but still, it is already a reality.
> Rust has pretty much done so for a large class of bugs prevalent in C/C++
Most modern languages improve upon the appalling lack of safety in C and C++. You're right that Rust (in particular the Safe Rust subset) does a much better job than most, and is showing a lot of success in its safety features. Programs written in Safe Rust don't have memory safety bugs, which is a tremendous improvement on C and C++, and it manages this without a garbage collector. Rust doesn't really lend itself to formal reasoning though, it doesn't even have a proper language spec.
> The mathematician's wet dream of applying "mathematical proof" on computer code will not work
Again, formal methods aren't hypothetical.
> I'd argue the Rust approach is in a very real way equivalent to a formal specification of program behavior that ensures the program does not have the various bugs that plagues C/C++.
It is not. Safe languages offer rock-solid guarantees that certain kinds of bugs can't occur, yes, and that's very powerful, but is not equivalent to full formal verification.
It's great to eliminate whole classes of bugs relating to initialization, concurrency, types, and object lifetime. That doesn't verify the specific behaviour of the program, though.
> No amount of formalism can prevent a programmer from writing `printf("hello word")` when they intended "hello world"
That comes down to the question of how do you get the model right? See the first PDF I linked above. The software development process won't blindly trust the model. Bugs in the model are possible but it seems li...
As always, the branding of formal methods sucks. As other commentators point out, it isn't technically possible to provide a formal proof that software is correct. And that is fine, because formal software methods don't do that.
But right from the outset the approach is doomed to fail because its proponents write like they don't know what they are talking about and think they can write bug-free software.
It really should be "write software with a formal spec". Once people start talking about "proof" in practice it sounds dishonest. It isn't possible to prove software and the focus really needs to be on the spec.
> It really should be "write software with a formal spec".
The code is already a formal spec.
Unless there are bugs in the language/compiler/interpreter, what the code is essentially formally well defined.
As programming languages get better at enabling programmers to communicate intention as opposed to being a way to generate computer instructions, there's really no need for a separate "spec". Any so called "spec" that is not a programming language is likely not "formal" in the sense that the behavior is unambiguously well defined.
Of course, you might be able to write the "spec" using a formal language that cannot be transformed into machine code, but assuming that the "spec" is actually well defined, then it's just that "compiling" the spec into machine code is too expensive in some way (eg. nobody has written a compiler, it's too computationally hard to deduce the actual intention even though it's well defined, etc.). But in essence it is still a "programming language", just one without a compiler/interpreter.
You can formally prove that it doesn't have certain kinds of bugs. And that's good! But it also is an enormous amount of work. And so, even for life-critical software, the vast majority is not formally proven, because we want more software than we can afford to formally prove.
This is an interesting point that I think a lot of programming can miss.
Proving that the program has no bugs is akin to proving that the program won't make you feel sad. Like ... I'm not sure we have the math.
One of the more important jobs of the software engineer is to look deep into your customer's dreams and determine how those dreams will ultimately make your customer sad unless there's some sort of intervention before you finish the implementation.
Exactly, it's fundamentally impossible. Formal proofs can help with parts of the process, but it can guarantee no bugs in the product. These are the steps of software, and their transitions. It's fundamentally a game of telephone with errors at each step along the way.
What actually would solve the customer's problem -> What the customer thinks they want -> What they communicate that they want -> What the requirements collector hears -> What the requirements collector documents -> How the implementor interprets the requirements -> What the implementor designs/plans -> What the implementor implements.
Formal proofs can help with the last 3 steps. But again that's assuming the implementor can formalize every requirement they interpreted. And that's impossible as well, there will always be implicit assumptions about the running environment, performance, scale, the behavior of dependent processes/APIs.
It helps with a small set of possible problems. If those problems are mission-critical then absolutely tackle them, but there will never be a situation where it can help with the first 5 steps of the problem, or with the implicit items in the 6th step above.
Even formally proved code can have bugs. If your requirement is wrong is the obvious thing. I don't work with formal proofs (I want to, I just don't know how), but I'm given to understand they have other real world limits that make them sometimes have other bugs.
It's always amazing when I get a bug report from a product that's been running bug free in production for years with minimal changes but some user did some combination of things that had never been done and it blows up.
Usually it's something extremely simple to fix too.
This happens a lot more than one may think especially with products that have lot of features. Some features are used sparingly and the moment a customer uses that feature a bit more in depth, boom. Something is broken.
> especially with products that have lot of features
No kidding. I'm 2 or 3 years into working on a SaaS app started in ~2013 and I still get bug reports from users that make me say "what!? we have that feature!?"
Trust me when I say this:
even "other" engineering domains have to do patches.
The difference is that software can be used before it is fully ready, and it makes sense to do so. No one can really use a 90% finished power plant, but software at 95% capacity is still usually "good enough"
I install high voltage switchgear on site.
A common problem is all the changes that has been added during the design stage, circuits that have been removed or altered, work that has kind of mostly been done to the schemes by the overworked secondary engineer. Sometimes, the schemes have been changed after all the wiring is completed and shipped to site, making it my pain in the ass when it's time to do the commissioning.
The end result is never 100% perfect, but somewhere in between "not too bad" and "good enough".
I think you're 90% there. There is also the cost to apply a patch.
If you want to patch a bridge, it's gonna cost you. Even if you only need to close down a single lane of traffic for a few hours you are looking at massive expenses for traffic control, coordination with transportation agencies, etc.
For most software it's pretty inexpensive to ship updates. If you're a SaaS company regular updates are just part of your business model. So the software is never actually done. We just keep patching and patching.
In some contexts, it is much more expensive to push out updates. For example, in the 00s, I worked on a project that had weather sensors installed in remote locations in various countries and the only way to get new software to them was via dial-up. And we were luck that that was even an option. Making international long distance calls to upload software patches over a 9600 baud connection is expensive. So we tested our code religiously before even considering an update, and we only pushed out the most direly needed patches.
Working on SaaS these days and the approach is "roll forward through bugs". It just makes more economic sense with the cost structures in this business.
Thanks for this insight! It has pretty strong explanatory power. It also explains why rushed development can stall. It explains 'move fast and break things'.
There's even an added factor of learning more about what is really needed by putting a 95% done product into use.
Heck, it explains (stretching it here) space-x's success with an iterative approach to rocket design.
My wife works as an acoustical consultant at a global construction firm. The things you hear about factories, offices, and even hospitals is wild. Don’t get me wrong the construction world works very hard to avoid issues but I think we in software tend to hold other engineering disciplines up on a pedestal that doesn’t quite match the messiness of reality.
Thanks for saying this. I think we in software engineering tend to think too binary: either the product is perfect (100% bug-free) or it's shit. There's always room for improvement, but compared to other engineering, overall, I think we're doing pretty good. As an example similar to your wife's, my friend used to work for one of the major car manufacturers doing almost the exact same job as Edward Norton's character in Fight Club. The cars had "bugs", they knew about it, but they didn't publicly acknowledge it until they were forced to.
>when do not accept the same in other engineering domains?
No, you just complain that your taxes are being used to build expensive roads and bridges. Or you think airplanes are far too expensive. Or that new cars are insanely expensive.
There are cost trade offs. In general, better quality more expense.
Also in software there is not an excessive amount of software engineers in relation to demand for software. So SWEs can get paid a lot to go build crappy software.
There are a few aspects. One is that we don't understand the fundamentals of software as well as the underpinnings of other engineering disciplines.
More importantly though, for the most part we choose not to do engineering. By which I mean this - we know how to do this better, and we apply those techniques in areas where the consequences of failure are high. Aerospace, medical devices, etc.
It differs a bit industry to industry, but overall the lessons are the same. On the whole it a) looks a lot more like "typical" engineering than most software development and b) it is more expensive and slower.
Overall, we seem to have collectively decided we are fine with flakier software that delivers new and more complex things faster, except where errors tend to kill people or expensive machines without intending to.
The other contributing thing is it's typically vastly cheaper to fix software errors after the fact than, say, bridges.
I think your salary observation is more of a firmware vs. hardware, rather then "soft" vs "hard" engineering.
Further to that, it's often informative to figure out what makes a company money. The highest paid software development roles tend to be doing things that are closer to revenue, on average. If you are a software developer at a hardware company (or an insurance company, or whatever), you aren't that close. Even worse if you are viewed as a cost center.
>Further to that, it's often informative to figure out what makes a company money. The highest paid software development roles tend to be doing things that are closer to revenue, on average.
yeah. Who are those trillion dollar businesses and what do they rely on?
- Apple: Probably the better example here since they focus a lot on user-facing value. But I'm sure they have their own deals, B2B market in certain industries, R&D, and ads to take into account
- Microsoft: a dominant software house in nearly every aspect of the industry. But I wager most of their money comes not from users but other businesses. Virtually every other companies uses Windows, Word, and those that don't may still use Azure for servers.
- Alphabet: ads. Need I say more? Users aren't the audience, they are the selling point to other companies.
- Amazon: a big user facing market, but again similar to Microsoft. The real money is b2b servers.
- Nvidia: Again, user facing products but the real selling point is to companies that need their hardware. In this case, a good 80% of general computing manufacturers.
- Meta: Ads ans selling user data once again
- Tesla: CEO politics aside, it's probably the 2nd best example. Split bewteen a user facing product that disrupted an industry and becoming a standard for fuel in the industry they disrupted. There's also some tangential products that shouldn't be underestimated, but overall a lot of value seems to come from serving the user.
General lesson here is that b2b and ads are the real money makers. if you're one level removed that financial value drops immensely (but not necessarily to infeasible levels, far from it).
> One is that we don't understand the fundamentals of software as well as the underpinnings of other engineering disciplines.
That sounds like an awfully bold claim. I have the feeling we understand software a lot better than we understand mechanical engineering (and by extension material sciences) or fluid dynamics. By a big margin.
I worked with finite element software and with CFD solvers, you wouldn't believe how hard it is to simulate a proper airflow over a simple airfoil and get the same results as in the wind tunnel.
To the contrary, it's nearly canonical. Most of the problems pointed out in the 70s (mythical man month) have still not been resolved, 50 years later.
>you wouldn't believe how hard it
Oh, I'd believe it (I've designed and built similar things, and had colleagues in CFD).
But you are definitely cherry picking here. The problem with CFD is we don't understand the fluid dynamics part very well; turbulence is a big unsolved problem still, though we have been generating better techniques. This is so true that in an undergraduate physics degree, there is usually a point where they say something like: "now that you think you know how lots of things work, let's introduce turbulence"
But a lot of mechanical engineering and the underlying physics and materials science is actually pretty well understood, to the degree that we can be much more predictive about the trade offs than typically is possible in software. Same goes for electrical, civil, and chem. Each of them have areas of fuzziness, but also a pretty solid core.
The article is about delivering a complete, working project "on time". I have a neighbor whose home is being renovated and it is already 2x the time the contractor originally quoted.
Of course it is easier for a developer to walk away from something incomplete than an architect and the contractors involved in a physical project, but still, I hardly think that there is really much difference in terms of timelines.
FWIW in my experience delays in e.g. home renos (or for that matter larger scale projects) are mostly for reasons unrelated to the engineering. In software projects, it's probably the #1 reason (i.e. we didn't know how to do it when we started).
Software is still absolutely king for number of large scale projects that just never ship, or ship but never work.
> To the contrary, it's nearly canonical. Most of the problems pointed out in the 70s (mythical man month) have still not been resolved, 50 years later.
Even with all of those applied, we wouldn’t be magically better. Complexity is simply unbounded. It’s almost impossible to reason about parallel code with shared mutable state.
We accept this in all fields of engineering. Everything is "good enough" and the seems to work reasonably well. You should remember this next time you hear about car recalls, maintenance work on bridges, or when some component in your laptop flakes out.
In addition to the other answers, there is the perennial and depressing one: Software bugs haven't killed enough people in a suitably visible/dramatic way to be regulated that heavily.
Team are flying the airplane, the se time rebuild it to the zeppelin, testing new engines inflight.
Or construction. Let's build apartment block, but for few apartments we will test new materials, new layout, etc. Once there are walls of the first apartments we will let people live there. We will build how we can, according to the napkin plan. In the end we will put all tenants in and stress test strength of the structures. Or one day people return home and their apartments have totally different design and layout because someone from the HOA decided so to get a promotion.
I mean, bridges collapse. That hasn't meant we gave up on engineering bridges.
Point being, we have some risk tolerance, even for civil engineering.
Now we don't accept an engineer saying, "this bridge will probably collapse without warning", which we do accept with software. So there is a difference.
it will be necessary to deliver software without bugs that could have reasonably been avoided in time
ive had this sentiment thrown at me too often by peak move fast and break things types. it's too often a cudgel to dispense with all QA in favor of more new feature development. shipping shit that has the same pattern of flaws youve encountered in the past when youve been shown ways to catch them early but couldnt be bothered isnt accepting that you cant catch everything, it's creating a negative externality.
you usually can make it someone else's problem and abscond with the profits despite, but that doesn't mean you should
Not buildings, not bridges, not cars, not airplanes, not software. There are mistakes in every field of engineering and the best we can hope for is to minimize them as much as possible.
Engineering is knowing (among other things) how to categorize the potential mistakes, develop procedures to reduce the chance of them being made and in the case that some slip through (and they will), estimate their impact and decide when you're "good enough."
I think with formal analysis, whole bug classes can be eliminated. Add to that a rigorous programming style, and 'bug-free' code is within reach.
There will remain bugss that make it through, but they will be rare, and will need a chain of mistakes.
Currently ways of coding to this kind of standard exist. But they are stupid. It involves things like no dynamic memory allocation, only fixed length for-loops, and other very strict rules. These are used in aerospace, where bugs are rather costly and rare.
Reminds me of a story about an engineer who was participating in a meeting with managers and partners. Manager was speaking of his team and how they will deliver software. Then, he asked the engineer to assure that the software will be bug-free. To this the engineer responded by saying he cannot guarantee there will be no bugs. The manager went nuts and started screaming.
Engineers cannot be responsible for all the vertical stack and the components which were built by others. If somebody claims it is bug free then they have not enough experience. Pretty much anything can fail, we just need to test as many possible cases as possible with a variety of available tools to reduce the chances of bugs.
When I first started in the 1990s I was told testing was 60% of the product budget and traditional writing code 20%. The remaining 20% was architecture and other design work. We didn't have unit test frameworks, but we did spend a lot of time writing throw away test fixtures (which was already generalizing into test frameworks by great developers and kicking off the unit test revolution)
Making good (!perfect) software is a function of three constraints: knowledge, economic resources, and time.
You can mix those three together and produce a desired output, but don't expect perfection, perfect software only appears when the three variables tend to infinity
I dunno about this authors CS program but at GT we had significant coursework related to all aspects of SDLC (including unit and acceptance testing), business case value, etc.
The process of building quality applications starts long before any code is written.
You need to understand the domain, you have to design something that solves an actual problem or delivers tangible value to someone, you need a holistic approach to user experience.
> In addition, at least in my studies, there was a semester about project management approaches and scrum. All of which is great, but QA is missing completely.
Just an anecdote, but, we had a "Software Development" class like this in CS (I took it in the '90s) and even though it followed a waterfall development model[0] and we used Gantt charts, QA (testing) was a big part of it and 1 of our 4 team members (or maybe 2 of 4 worked on it together) was primarily responsible for it. (I wrote the parser and the made diagrams/documentation for the parser.)
The description (in an old catalog[1]) is:
Software specification, design, testing, maintenance, documentation; informal proof methods; team implementation of a large project.
Turns out I didn't need to look up the old catalog because the description is exactly the same still! Except it's CS 371 now, and the longer "Learning Outcomes" for the course has some newer stuff (agile and version control) but otherwise is all the same things I learned at the time.
While not a panacea, visual/snapshot testing tools like Cypress + Percy that perform clicks and take screenshots can be tremendously helpful to ensure that building in a programmatic QA plan is a predictable part of the scope of what is coded.
And the good thing about snapshots is that they provide a visual way to communicate a preview to stakeholders of what may change, both for the currently-being-developed workflow as well as to prevent visual regressions elsewhere - so they're inherently more easily justifiable than "we're spending time on testing that you'll never see."
The article is correct that treating QA as the last phase of the project, and a disposable phase at that, is a recipe for disaster. But if you make it an ongoing part of a project as individual stories are burned down, and rebrand it as part of a modern development workflow, then it's an entirely different conversation.
As sibling comment `@wellpast` commented, but to extend it with my point of view.
We can roughly say we can have three out of: (high) quality, (low) time, (low) communication complexity, and (low) money. (time is a dependent here.)
People are trying to apply factory processes and structures to a team sport, an engineering discipline. You do not teach or build a basketball team by breaking down each attack phase into steps and checkmarks.
You try to minimize communication and make the team work as one. It is a team and individual building, not a process building exercise. You make a plan, and follow the Moltke's the Elder conclusion:
"no plan of operations extends with any certainty beyond the first contact with the main hostile force."
(Or paraphrased as you have heard: No plan survives contact with the enemy.)
All (types of) Engineers know this. But software engineering is "special."
And it is not a "move fast and break things issue."
That is part of all engineering or team playing too.
It is the type of business mentality, that because a plan did not go exactly as expected we need to add more process. Whatever that process may be. Because if "I as a manager add a process, then the next failed plan, I am covered, and I will blame the individuals."
Process has a place to ensure things happen in a legal and moral framework. And minimize adverse circumstances -- e.g. we bet all the hedge fund money accidentally when running tests.
Process is used differently in most startups and corporations with not the team in mind.
The construction metaphor is a bad analogy. The compiler does the construction, dev teams do iterative design, ideally with frequent feedback and adjustment.
Do you ever yell at a traditional architect and ask them when it's going to be done? It's always when the client is happy or makes their mind up about it. A lot of dev is like this.
Honestly I think the root problem is that universities have a degree in computer science, whereas what most people want is to learn to build computer software.
The two overlap most of the time in subtle ways where the science gives an important foundation, such as learning Big O notation and low level memory concepts where exposure helps. I've personally seen this with a smart coworker who didn't go through university and is great at programming but I'll catch him on certain topics such as when he didn't know what sets and maps were and when he tries to sleep a second instead of properly wait on an event.
However, the differences between computer science and building software are problematic. Watching my wife go through university, she's had to struggle with insanely hard tasks that will not help her at all with software, such as learning Assembly and building circuits. The latest example is the class where she's learning functional programming is not actually teaching it to her. Instead, they combined it with how to build a programming language, and so instead of giving her toy problems to teach the language she is having to take complex code she doesn't understand well that generates an entirely different programming language and do things like change the associativity of the generated language. In the end, she feels like she's learned nothing in that class, despite it being her first experience with functional programming.
On the flip side are the things that are necessary for software that aren't taught in university, like QA. For me personally, back when I was in university a decade ago I never learned about version control and thought it was just for back up. Similarly, I never learned databases or web, as the required classes were instead focused on low level concepts as Assembly and hardware. My wife is at least learning these things, but even then they often seem taught badly. For example, when they tried to teach her QA, instead of hardcoded unit tests, they made her give random inputs and check to make sure the output was correct. Of course, checking the output can only be done by rewriting all of your code in the testing files, and if there's a bug in your code it'll just get copied, so that kind of defeats the purpose. Even when the assignments are relevant there is often no teaching on them. For example, her first ever web code was a project where they told her to hook up 6 different technologies that they had not gone over in class, with only the line "I hope you've learned some of these technologies already".
I wonder how much of the lack of QA is rational. That is to say, for most projects does shipping with lots of somewhat hard-to-find bugs actually hurt the bottom line?
For some classes of bugs it can (e.g. if the software is so bad as to open you to a class-action lawsuit; in b2b software bugs that put you in breech of contract), but for many classes of consumer software, it's not clear to me that shipping software that works better is rewarded. Picking not-too-buggy software ahead of time is hard, people are slow to switch (even when the people encountering the bugs are the people selecting the software, which is often not the case), and people are good at subconsciously avoiding triggering bugs.
It starts mattering more for consumer software when you reach mass scale. Somewhat hard-to-find bugs at the scale of hundreds of millions of users (like a social media company), turn into bugs faced by hundreds of thousands of users.
But at that scale (in my experience), QA is up front and center and is typically a core pillar of engineering orgs.
> It starts mattering more for consumer software when you reach mass scale. Somewhat hard-to-find bugs at the scale of hundreds of millions of users (like a social media company), turn into bugs faced by hundreds of thousands of users.
From a cynical point of view, if those hundreds of thousands of users will use your product despite the bugs, does it matter?
People are only as loyal as their opportunities; if the competition is mostly the same as your product but has either fewer bugs or bugs in a less painful flow, buh-bye
I'm super cognizant that the cited example of "social media company" is its own special ball of wax, due to the network effect, and I wish to holy hell I knew how to fix that bug :-)
1. The premise that college teaches you how to build software in industry is a pretty wild claim.
2. Is this article from the 90s where we ship software on CDs or floppy disks? In today's world where the concept of a "release" is often blurred by continuous delivery pipelines (and this is considered a good practice), having a quality insurance department manually assuring that no bugs are in that release seems downright archaic.
Not everyone is writing a webapp where you can roll out upgrades anytime CI passes, or a phone app that you can upgrade every week. some of us work on code that will be shipped in a device that is not easy to upgrade.
Absolutely, and industries like medical devices and aviation have extremely strict regulation and procedures regarding the testing of software. The article not mentioning any of those made me conclude that author is referring to regular software.
I'll take a stab. Quality software is software that is testable, able to adapt to new features and is architected to match the current organizational structure of the software team so that communication and dependencies don't have an impedance mismatch.
- why is testable software higher quality? Does it add value to the software? I'd venture that untestable software has the same value (if not more) than testable software (due to time-to-market). You can write software that is 'obviously correct' and "high quality" at the same time, without any tests.
- Why does software that can adapt to new features increase the quality? If that is the case, we must argue that WordPress is extremely high-quality software. Or SAP.
- How does architecture influence quality? If that is the case, then there isn't any need for different architectural styles since there should be "one true style" that has the best quality software.
Testable software usually has a better quality because you can automate some parts of the quality assurance.
Sacrificing quality assurance to favour other aspects is common, but the quality usually suffers.
A company favouring time to market over testability is likely to release buggy software. They can get away with it.
Adaptability is a common quality, but you can find counterexamples. WordPress and SAP are successful software that may not check all the quality boxes.
Some architectures are for sure worse than others, and there isn’t one good architecture for all kinds of problems.
> why is testable software higher quality? Does it add value to the software? I'd venture that untestable software has the same value (if not more) than testable software (due to time-to-market). You can write software that is 'obviously correct' and "high quality" at the same time, without any tests.
Note I said testable software, not software with tests (there is a difference!)...I'd agree that software with tests (which is by definition testable software) has a huge developer cost to it that may not always be in the best interest of the company (like you said, time to market might be important). But in my experience, writing code in a way that can be tested later is only marginally more costly (time-wise) than writing code that isn't. A good example of this is writing modules that communicate with message passing and have state machines over direct function calls. The former has a slightly higher cost for dev time, but you can always retro-fit tests to it once you've achieved market penetration. You can't always do that with direct function calls.
> Why does software that can adapt to new features increase the quality? If that is the case, we must argue that WordPress is extremely high-quality software. Or SAP.
This is a good point that you bring up. I think what we are getting at ultimately is that quality and value are distinct entities. Software can have high value without being high quality. In my mind, being able to provide the business with new value-producing functionality without causing a spike in bug reports is my (admittedly vague) standard.
> How does architecture influence quality? If that is the case, then there isn't any need for different architectural styles since there should be "one true style" that has the best quality software.
Architecture has to match how the software teams communicate with each other. Like actually communicate, not how the org chart is made (see Conway's Law). So my point is then that if there are two separate teams, your code should communicate between two "modules" that have an interface between them. Just like real life. It would be silly to implement a micro service architecture here. That's why Amazon's SOA design works for them: it matches how teams are organized.
Good start, but too broad and open for interpretation.
- Who gets to define testability?
- I want to add a coffee maker to my crash test dummy; is the lack of room for the filter and water tank a sign of a bad design? Or not flexible enough for my feature?
- (cue meme) "You guys have organizational structure?"
- Who gets to claim the impedence mismatch? What are those consequences? Wait, where are the dependencies defined again outside of the software?
I do (just kidding!)...Testability is the ability to add testing at a later point. There is no hard definition of this, but if you can't test at least 75% of your public facing functions then I'd say you don't have testability. Remember testability means you can have a tigher feedback loop which means that you don't have to test in production or in the physical world. This means you get where you want to go faster.
> - I want to add a coffee maker to my crash test dummy; is the lack of room for the filter and water tank a sign of a bad design? Or not flexible enough for my feature?
I know you are joking, but imagine for a second that your business did in fact invent a brand new way to test crashes and that coffee makers were the key to breaking into that market. If the dummy can't accommodate that then...yes! It is a bad design, even if it was previously a good design.
> - (cue meme) "You guys have organizational structure?"
Remember: there always is an organizational structure, with or without a formal hierarchy. You want to match your software to the real one.
> - Who gets to claim the impedence mismatch? What are those consequences? Wait, where are the dependencies defined again outside of the software?
There are no "the company blew up" consequences with this type of failure mode. Instead you get a lot of "knock on" effects: high turnover, developer frustration, long time to complete basic features and high bug re-introduction rates. This is because software is inherently a human endeavor: you need to match how it is written to how requirements and features are communicated.
My entry: one that is easy to refactor regardless of code size.
If this is given, every other metric, like features, bugs, performance is just a linear dependence on development resources (maybe except documentation, but that is kinda an externality).
Yeah the dimensions discussed in this article is somewhat advanced stuff and comes from experience, DRY is relatively basic concept and easy to grasp and unfortunately mid level engineer do really dangerous stuff in the name of DRY and horrible abstractions get created that break down and create a horrible mess when the requirements change.
In my experience it is generally wise to avoid abstractions and copy/paste things a couple of times, once the code base matures good abstractions will be more obvious. Even then it's good to think about future changes, will these 2 things want to evolve separately in the future? If the answer is YES, then maybe coupling them is not a great idea. I think there was a really good Kent Beck talk about coupling vs cohesion somewhere.
Another thing to think about is breaking things, if changes are local to one single endpoint then any changes there can only break that endpoint, edge cases and scenarios to consider are only relevant to that endpoint. When changes to a core abstraction are required then hundreds of use cases/edge cases need to be considered - why are we creating so many core abstractions in our systems in the name of DRY?
I've also found that the more moving parts you add the harder a system becomes to learn, the S in SOLID is probably to blame for that. The only single responsibility principle is useful for is unit tests (easier to mock), but many times harder to understand. If the actual functionality is not local to the file things become ungreppable via code search, understanding the entire system requires an IDE and jumping around to each and every beautiful lpad() implementation and trying to piece what is happening one 3 line function at a time.
Then there is also layering to consider, if 2 pieces of code look somewhat similar but belong to different layers (example controller and DAO layer, then also care must be taken to not make an abstraction that couples these together, or to couple 2 unrelated modules together that could otherwise have their own life cycle).
These are just some aspects I could think of that I think about when creating abstractions, but somehow I see engineers focus too much on DRY. Maybe they got burned so bad some time in the career by forgetting to change something in 2 places?
The whole article has a straw-man feel to it. It is not the senior developers responsibility to create a proper QA policy for the project. It is good that the lead should know how to implement good QA processes.
HOWEVER, the real culprit are the MANAGERS.
After 50+ years of skanky software development policy aimed at low balling cost its time to blame the right people. No amount of cajoling, "business/budget" speak manipulation is going to fix a fundamental flaw in how managers at that level are trained and behave.
We have stop being apologists for mistakes not of our making.
If using QA techniques would increase the managers bonus then we will see it being used. If all it does is make better software then this post will be rewritten in 50 years and would still be relevant.
Berkeley has a class that teaches TDD and other XP practices. (Or at least used to.) Pivotal used to recruit a lot of new grads from Berkeley for that reason.
IME, “QA” doesn’t really correlate with quality software, nor is there really a time vs. quality trade off. Bad software is often also delivered poorly, and high quality software can be delivered quickly.
493 comments
[ 5.1 ms ] story [ 315 ms ] threadEvery industrial practice.
On the other hand, the title just means that programming is not an industrial practice. What should be obvious to anybody that looked, but some people insist on not seeing it.
For engineers, the situation is very similar to software. There are many tools for enforcing quality, but there's no structure for teaching the engineers, and no, there isn't a widely accepted theory for how to teach design quality either.
The one place where people are consistently taught how to build quality is on manufacturing.
Imagine if every software company was obliged by law to:
- Every single release has to have been signed off by someone who got their "software release engineer" certification at the software equivalent of the FAA.
- This engineer is required by law to not sign off unless every box on a 534 item checklist has been manually verified.
- Any time an unplanned downtime happens at any company, a government team comes in to investigate the root cause and add points nr 535 through 567 to the checklist to make sure it never happens again.
If such a system was mandated for software companies, most of the common bugs would very rapidly become a thing of the past. Development velocity would also fall through the floor though, and most startups would probably die overnight. Only companies that could support the overhead of such a heavyweight process would be viable, and the barrier to entry would massively increase.
It's specific to the complex and unsafe C++ language though, rather than addressing broader software development methodology.
[0] [PDF] https://www.stroustrup.com/JSF-AV-rules.pdf
The rebuttal to your implied Goodhart's Law <https://en.wikipedia.org/wiki/Goodhart%27s_law> that was offered by my manager was "tension metrics" <https://en.wikiversity.org/wiki/IT_Service_Management/Contin...>
If I understand his theory correctly, in your case there would be a competing metric to the "test coverage" one that said for any changeset, a test cannot itself change by more than 20% in the same changeset as non-test code. So you can change the code such that it still passes the existing tests, or you can change the test to adapt to new requirements, but you cannot rewrite the tests to match your newly changed code
I'm acutely aware this is a terrible example, the devil's in the details, and (in my experience) each company's metrics are designed to drive down their own organizational risk <https://en.wikipedia.org/wiki/Conway%27s_law>, combined with "you're always fighting the last war" :-D
The entire thing is terrible from principle. You won't find a good example, because that's not how you use a process checklist.
That's the reason why aviation can only shine when it becomes a private means of transportation, and I don't mean 70mm private jets but, 150k light helicopters.
When a critical mass is hit then accidents will become no more traumatic to the collective psyche than car accidents, the lighter the aircraft the better because it would seem exactly like a car crash as opposed to leaving a huge burning hole into the ground
Machine handling requires a lot of training and a lot of well thought out procedures.
I cannot fly professionally anymore due to health, but this is something we are taught in aviation and something I too find lacking from tech so far.
Like, you’re taught the standards as part of learning to fly, but as time goes on, you’re told to narrow your tolerance of what is acceptable. So if you are learning how to do steep turns, for instance, the altitude standard is +- 100’. You’re taught, “that’s the minimum, you should be trying for 50’” and then 20’, then the absolute best performance would be where you do your turn, the needle doesn’t move, and as you roll out on a calm day you fly through your wake. But the goal is “better, always better, what can I do better?” And flying is not graded on the overall performance, if you don’t do satisfactory everywhere you fail. Culturally satisfactory, isn’t, it’s the starting point.
That encourages a much more collaborative model I feel like. I’ve only worked one or two flying jobs that were not collaborative. In the outside world it sometimes feels the opposite. In flying, you truly want everyone to succeed and do well, and the company does. Even the guys I hated that I flew with, I didn’t want them to fail. If they failed, I was partially responsible for that.
It wasn’t always perfect, and I worked with some legendary assholes while I was flying, but truly, they supported me and I supported them, and if I screwed up (or they screwed up) the culture required that we found a way to minimize the future potential screwups.
You’re actually trained on what quality means in a wide variety of contexts too, and flight operations quality assurance (FOQA) is a big part of many airlines. In smaller bush operations where I worked, it is significantly more informal, but we truly had a “no fault” culture in nearly all the places I worked. It’s not perfect, but that’s the point, “ok how can we make this better?”
If someone had an idea for how to do something better, there may have been friction, but that was rare if it actually was better, and as soon as you could show how adoption was fast even at the less standardized operations I worked at.
Not saying it’s all unicorns and rainbows, but I feel like quality, and decision making, and “doing the right thing” were an integral part of the culture of aviation. “The weather is too bad and I cannot do this safely” grounds the flight, you don’t blast off into the shit (at reputable operators) to get the job done anymore (it’s not 1995), and it feels like this new industry is the opposite.
The entire concept of a “minimum viable product” is somewhat illustrative of the problem. It shouldn’t be the “minimum viable” it should be the “minimum quality product we’re willing to accept as a starting point.” But that doesn’t roll off the tongue in the same way.
We shouldn’t be striving for anything that’s the “minimum.” The minimum is only the beginning.
Is that not the case in software? The incentive to improve may not be quite as strong as in aviation (crashing software isn't quite the same as crashing airplanes), but it is still pretty strong. Life is very miserable in software when quality isn't present.
Of course, management should be supportive of quality improvements, but their reality is either one where they are under genuine pressure to deliver projects X and Y to stage of quality through to not understanding or caring about quality.
My own experience is that individual programmers have vastly different ideas of quality is based on their experience and education. You can be struggling to get a team to improve and then you hire a somewhat normal individual with a very different background who makes a sizeable impact on quality and the culture of this in the team. I'm thinking specifically of someone who joined from aerospace, but I've seen it with finance backgrounds. I think the background matters less than the perspective and ability to hold people accountable (including yourself.)
No doubt the same as when the members of your garage band are happy to stay in the garage while you have your sights set on the main stage. You either suck it up and live in the misery, or you get better on your own time and leverage those improvements in the quality of your performance to move into a better position where quality is valued.
or maybe, them who know how to do this are just unable to spread this knowledge... something about how they think their private secret codes are the source of their wealth
when in fact, it's merely the scheme by which they mantain an advantageous capacity to extract energy from them seeking to learn how to build quality software
There's an obvious comprehensibility complexity to code to anyone who has spent almost any time what so ever trying to make something happen in software. However, we've got zero academics or theory around it.
Just 'best practices' (ie a thing other people are known to do so if things go wrong we can deflect blame).
And code smells (ie the code makes your tummy feel bad. yay objective measures).
And dogma (ie "only ONE return point per function" or "TDD or criminal neglect charges").
Sure, please do something for QA because it'll be better than nothing. But we're probably a few decades of waiting for actual theoretical underpinnings that will actually allow us to make objective tradeoffs and measurements.
(One exit point is a pet peeve of mine since it often makes the code a lot harder to read and think about vs exit asap)
There are pretty much no languages left today where it's even possible to violate this principle without really trying, it's not about having single a return it's about all the functions starting at the top and return statements always taking you back to the same place in the code.
Usually available as optional lectures during the degree, or later as Msc and PhD subjects.
Although, so far I've only bumped into cyclomatic complexity (with some studies showing that it has worse predicting power than lines of code) and lines of code.
Even though I find cyclomatic complexity uncompelling, it at the very least can slurp up code and return a value. Nicely objective, just not particularly useful or insightful to whether or not things are easy to understand.
The provided link looks suspiciously like they're going to talk about the difference between system, integration, and unit tests. The importance of bug trackers. And linters / theorem provers maybe.
I don't think these are bad things, but it's kind of a statistical approach to software quality. The software is bad because the bug chart looks bad. Okay, maybe, but maybe you just have really inexperienced people working on the project. Technically, the business doesn't need to know the difference, but I would like to.
"Measuring Complexity of Object Oriented Programs"
https://link.springer.com/chapter/10.1007/978-3-540-69848-7_...
I don't suppose you know where I can get their list of references without hitting a paywall? Specifically [16] and [24].
EDIT: [For anyone following along]
The linked paper is Measuring Complexity of Object Oriented Programs. Although, the paper isn't free. They reference several other papers which they assert talk about OO complexity metrics as well as procedural cognitive complexity, but unfortunately the references aren't included in the preview.
Apparently, there's also a list of Weyuker's 9 Properties which look easier to find information on. But these look like meta properties about what properties a complexity measurement system would need to have [interesting, but they don't really seem to comment on whether or not such measurement is even possible].
It looks like a lot of this research is coming out of Turkey, and has been maybe floating around since the early 2000s.
EDIT EDIT: References are included at the bottom of the preview.
EDIT EDIT EDIT: Kind of interesting, but I'm not sure this is going to yield anything different than cyclomatic complexity. Like, is this still an area of active research or did it all go by the wayside back in the early 2000s when it showed up? The fact that all the papers are showing up from Turkey makes me concerned it was a momentary fad and the reason it didn't spread to other countries was because it doesn't accomplish anything. Although, I suppose it could be a best kept secret of Turkey.
Renamed programs are defined to have identical complexity, which is pretty intuitively untrue, so I've got my concerns.
EDIT ^ 4: Doesn't seem to be able to take data complexity into account. So if you're dividing by input, some inputs are going to cause division by zero, etc. You might be able to jury rig it to handle the complexity of exceptions, but it looks like it can mostly handle static code. I'm not sure if it's really going to handle dynamically calling code that throws very well. I also don't think it handles complexity from mutable shared references.
Nice try, but unless there's a bunch of compelling research that no actually this is useful, I'm not sure this is going to cut it. And at the moment the only research I'm finding is more or less just defining functions that qualify as a cognitive measure under the Weyuker principles. I'm not seeing anyone even pointing it at existing code to see if it matches intuition or experience. Happy to be found wrong here, though.
The main point still stays, "You are never taught how to build quality software" is wrong, and plenty of Engineering degrees do teach about it.
I was thinking there should be some metric around d(code)/dt . That is, as the software is used, 'bad' code will tend to change a lot but add no functionality. 'Good' code will change little even when it's used mode.
An example of such research done well can be found in Exploring Science, by Klahr. The author and his colleagues look very closely at how people interact with a system and experiment with it, leading to wonderful insights about testing processes. I've incorporated those lessons into my classes on software testing, for instance.
The Power of 10: Rules for Developing Safety-Critical Code[1]
Assessing the Relationship between Software Assertions and Code Quality: An Empirical Investigation[2]
Cyclomatic Complexity and Basis Path Testing Study[3]
[1]: https://web.eecs.umich.edu/~imarkov/10rules.pdf
[2]: https://www.microsoft.com/en-us/research/wp-content/uploads/...
[3]: https://ntrs.nasa.gov/api/citations/20205011566/downloads/20...
It's worse than that. No one can agree what "quality" means.
Mostly, the word is used as a weapon.
The pointy end of the weapon is what management pokes you with whenever anything unexpected happens. Typically they do this instead of making sure that problems do not happen (a.k.a. "management").
The weapon's handle is sometimes flourished by process gatekeepers who insist on slowing everything down and asserting their self-worth. This is not good for throughput, anyone else's mood, or eventually even for the gatekeepers.
People usually refuse to talk about quality in terms of actual business metrics because if anything unexpected happens that's not covered by the metrics, there will be fault-finding. And the fingers pointed for wrong metrics are typically pointed at middle management.
I agree with you on that one. IMHO it's because of that difficulty to know when to optimize for quality, but also because of sheer incompetence.
It's akin to pollution, really: Individual company making 'dirty' things won't see the consequences. But scale this mindset out and suddenly we wake up in a world when trillions of dollars are spend to counteract those effects.
I wonder where you get the confidence to make such a strong statement, which is clearly not warranted. I want to challenge you to broaden your view a bit: Not a lot of software is like Windows explorer. Not a lot of software is performance critical. A lot of software can do with bugs, with many many bugs, to be honest. A lot of code is run fewer than 100 times, before it's retired. Also, not a lot of software written has many users. Or enough users to support maintaining it. "Pollution" often affects just the author of the software themself. Software is just a means to an end, and the end decides, how much effort was warranted in the first place.
Not being performance critical doesn't mean it is justified to diss-respect users by wasting their time.
>A lot of code is run fewer than 100 times, before it's retired. Also, not a lot of software written has many users.
Obviously we aren't talking about some simple automation scripts here.
>"Pollution" often affects just the author of the software themself.
You are misunderstanding the pollution analogy. I'm not talking about polluting the codebase with code smell.
I am talking about costs of low quality being non obvious and only revealing themselves at global scale
This is moving the goalpost, and also ignores the fact that software exists on a spectrum from "simple automation script" to "messaging app used by millions". It seems you have a very narrow view of what software is, or what it is used for, and the constrains that apply when building it.
Running that kind of software for the central bank here. So kind of disputing your statement.
> So I am simply excluding this category as an extremum.
Which ignores the long tail. Great approach.
>Great aproach
Unironically better, then trying to make prescriptions as broad and general as possible, because those usually are too generic to carry any actual value
Also note that I wrote "code" originally, because there can be programs which are run very often, but certain code paths are not, so my statement applies even for some parts of popular software.
The image I think would be valuable for you to consider is a curve, where 20% of code has 80% of all executions, and 80% of code get's the rest. It makes sense to put in a lot of effort into writing the top 20%, but on any given day it is very likely you'll be working on the lower 80%.
I have written non-trivial systems deployed in 20 different sites that have never had a bug ever.
My best are usually second systems, focus on simplicity, standardisation and resist scope creep.
100% coverage with unit tests. 100% coverage with integration tests.
I've written a many things with zero bugs after delivery.
(and other that were never ending quality nightmares)
If I am under pressure to deliver I get strict with TDD, since no time for bug fixing.
Yeah you won't get to a point where you'll have a valid-enough metric to make this point.
I was at a startup once. The two founders said "don't write unit tests". I wasn't going to argue with them. I understood what they really meant. We've been too slow, we need to ship as fast as possible. I shipped fast and I shipped quality (ie low defects and outages). I wrote unit tests. They didn't need to know. They just needed the outcome.
The elephant in the room in all of these conversations is that you walk into any software development shop and they just don't how to ship both fast and at quality. No matter how much an organization or team tries to revisit/refactor their dev process year-to-year, they're still shipping too slow and mediocre quality.
The truth is there isn't a magic formula. It's an individual craft that gets you there. It's a team sport. And the context is different enough everywhere you go, yeah, sure, some lightweight processes might abstract across the universe but very little bang for those bucks. Far beyond any other facet of things, you really just have to have a good team with experience and the right value-system/value-delivery-focused wisdom.
"I want to start programming, what language should I learn?"
"One of these two most popular languages, where you will not even know about basic errors until you run the code!"
It is a good peek into the professional software world though!
Writing a thousand lines of bug free code isn't that hard, so the need for QA practices won't be apparent. Then you get the ten thousand line intern project and discover things are not always that easy. Then we throw you into a multi-million line project and good luck: you need a lot of that QA stuff to make it work.
A pure university education without considering is this degree useful in the real world is a disservice to education. However a pure trade school education that teaches how to do something without understanding is not useful (I don't think any trade school is that pure: they tell you to ignore hard stuff but generally give you deep understanding of some important things)
Why?
> A pure university education without considering is this degree useful in the real world is a disservice to education.
I think this line of thinking is a much bigger disservice to higher education. It was very tiresome as an undergraduate to be surrounded by people that thought this way - and detrimental to everyone's education.
"I'll never use this knowledge" is the single worst thing you can say as a student, and it needs to be beaten out of undergrads' heads. Not encouraged.
Which is why the "I'm never going to use this, what a waste of time" feeling among American undergrad students is so common. If you fix the affordability problem and bring it back to where is was in the mid 70s (inflation adjusted) I think things would be a lot better.
That was true 50 years ago, but employers turned it into job training. My father in law retired a well off businessman with a History degree from Yale he got in the 50s. You know what a History degree from Yale qualifies you for today? Teaching History and maybe writing some books. The degree didn't change and Yale didn't change.
I'm not sure about this part... A very common pattern in my conversations with working class friends and family from my parents' generation is: "we were told that if we sent our kids to college, they'd have better lives than we did, but instead we all just ended up with more debt than we could handle".
It's tricky! If you tell teenagers and their parents the truth - this purely academic program will not train you for any job besides pure academia, which, while it can be a fantastic career, is a super risky hits business in which only a few will truly succeed - then that's only going to sound like a reasonable risk to take for wealthy families. But then you've badly limited your pool of academic researchers to this extremely small and honestly often not as promising set of rich kids.
Maybe one solution (which is not workable in the real world) would be: any academic program that does not have a viable "job training" component should only accept students on academic scholarship, regardless of their own means. If some neutral party thinks they are promising enough in that field to pay their way, they get to go for free, otherwise they don't get to go at all. The programs that do graduate people with directly marketable job skills could keep working the current mercenary way.
The reason this wouldn't work in reality is that the wealthy would still just game the scholarships in some way. Alas.
There is nothing wrong with art/music/history. If you are interested by all means take a lot of courses in them. You can learn a lot of valuable skills which is why good universities required a diverse background of "generals" that these (and many more) fit into. However they give far more degrees in these things than are needed. (even physics gets more degrees than the world needs - but most getting a physics degree can better pivot to something else well paying).
If you want to know what a university will teach your kids, ask them. They'll even tell you without asking them - it was pretty obvious to me as a dumb high school kid on campus visits what the emphasis of one program or another was going to be.
If you are a working class family with a kid who is very talented at math, and you go sit down with the counselors and ask them: If my child studies pure theoretical math, will that open them up to a life full of possibilities? they will say "yes, it absolutely will". But that's not true. It might be true, but it's a big risk. It's a risk a wealthy family can very easily absorb. But if this child from this working class family takes on this risk using student debt, it might go poorly. They might very well be good at pure math but not be good enough to go into academia. Then they might be unsure what else they can do with that degree, unable to get their foot in the door at the kinds of employers where just a general proof-of-being-smart degree is enough. And now they have debt and uncertainty about what to do.
It also might work out great! But it's a risk. And I know a number of people who feel they ended up on the wrong side of that risk.
99% of what we all learn in college is a waste of time for us. But we all have a unique 1% that is vital to who we become. Over time I expect that 1% to become 0.1%, then 0.01%, and for that vitality to become ever more concentrated in that sliver.
Sure there are a few going to university just for the fun of it. However most are expecting a job. Thus universities should train and emphasize thinking in more specific areas.
> "I'll never use this knowledge" is the single worst thing you can say as a student, and it needs to be beaten out of undergrads' heads. Not encouraged.
This is tricky. I agree undergrads say this all the time when they are wrong but they don't know it. They have no clue what they will use and what they won't. This is something universities should figure out so they push people to avoid things they won't use. OTOH, a lot of what they are really teaching isn't the specific skill, but how to research and analyze data to find complex answers - it doesn't matter if you look at data from art or from science, what you are really learning is how to think and the specific knowledge gained is isn't important or the point (I think this is the point you were trying to make?).
Companies that offer the jobs are the ones that need to offer the job training.
> (I think this is the point you were trying to make?)
Not really, it's that university education is kind of meta/self serving (the goal is not to train X number of students to do Y jobs, it's to give every student at the institution what that institution defines to be an education).
But like you said, the fact this produces better workers is a second-order effect. It's not the goal of most institutions. But not all institutions; some define "well educated" to have lots of industry practicum, and if you want that, go study at those institutions.
My main point is that it's not a "disservice" to eschew practicum or industry training as an educational institution.
When institutions allow student to take degrees that society finds less valuable (art,music...) they are doing society a disservice by not producing what society needs. Now if the student is wealthy (not rich) enough to afford that the price then I don't care: I don't need to impose my values on anyone else. However most people in a university are not that wealthy (most are young) and so if the degree granted isn't valuable to society the university robbed that student.
1. what's wrong with a student pursuing their own personal goals? A person doesn't need to produce for society's sake.
2. despite that sentiment you hold, it's clear many people do value art and music. Maybe not in its pure form, but those artists do in fact fuel industries worth billions. Clearly "society" values something that requires such skills and thinking.
No. Companies love hiring higher-ed graduates because it removes a lot of cost and risk for them:
- hiring only people with degrees weeds out everyone unable to cope with a high-stress environment, for whatever reason - crucially, also including people who would normally be protected by ADA or its equivalent provisions in Europe.
- it weeds out people in relationships or with (young) children, which makes them easier to exploit and reduces the amount of unexpected time-off due to whatever bug is currently sweeping through kindergarten/school/whatever. Sure, eventually they will get into relationships and have children as they age, but looking at the age people start to have kids these days [0], that's a solid 5-10 years you can squeeze them for overtime.
- it saves companies a ridiculous amount of training. The old "tradespeople apprenticeship" way is very cost-intensive as you have to train them on virtually anything, not just stuff relevant to the job, e.g. using computers and common office software. Instead, the cost is picked up either by the taxpayer (in Europe) or by the students themselves in the form of college debt. The latter used to be reserved for high-paying jobs such as pilots who have to "work off" their training cost but got compensated really well, nowadays it's standard practice.
- it keeps the employee diversity relatively homogenous. There is a clear bias towards white and asian ethnicity in the US for higher ed [1], and among top-earning job, males still utterly dominate [2].
- related to the above, it also weeds out people from lower economic classes, although at least that trend has been seriously diminishing over the last decades [3].
[0] https://www.nytimes.com/interactive/2018/08/04/upshot/up-bir...
[1] https://hechingerreport.org/proof-points-new-higher-ed-data-...
[2] https://www.bankrate.com/loans/student-loans/top-paying-coll...
[3] https://www.pewresearch.org/social-trends/2019/05/22/a-risin...
And thus you destroyed you whole point: females dominate universities these days.
[1] https://www.stemwomen.com/women-in-stem-percentages-of-women...
Unfortunately many programs are not designed this way and learning the appropriate skills is left as an exercise to the student usually in a sink or swim approach. So some students come out with the meta skills that a university education is touted for and others do not.
I do agree that "I'll never use this knowledge" can be a miserable attitude to have or engage with - especially when it's just a proxy for "I'm not interested in learning, just in getting good grades" but the idea itself is valid.
Everyone will think differently. I've never truly be research-minded and there's very much a bunch of odd classes that felt like a waste of my money (something to consider as education gets more expensive). But I do agree that there should be a space to foster researchers and especially one to overall round out a student, even if that space is more niche. I just don't think that everyone needs to go far into debt for that experience if they just want job training.
So I too desire a more explicit divide than "research university vs. industry university" and wish there were some better trade schools focused on software (not 6 month boot-camps. Think of a condensed university program without requirements of electives and maybe less supporting classes). But no one seems to be protesting this much.
Seems like a pretty bad premise to start an article on quality software. If you believe you can ship bug free code, it's time to switch careers.
Unfortunately, you are correct. Shipping in time and bug free are inversely proportional, and in a world were usually it's hard to argue with PMs for more time to have better testing, or paying tech debt... it's just a reality
But more importantly, once you've fixed the "show-stopping bugs," putting the software in front of customers is probably the best next step, as even if it's bug-free, that doesn't mean it solves the problem well.
[The above does not necessarily apply in highly regulated industries or where lives are on the line]
On those you're of course expected to do safety and testing up to the limit of the "value of a statistical life"s within the expected project impacts, but it still has time and budget limits.
Most software engineering is about making sure the happy path works well. When lives are on the line, you need to also plan to minimize the possible damage that can happen when things go wrong.
I can think of far better ways to spend infinite time.
We aren't taught how to write reliable software because very few people know how to write reliable software. It doesn't help that academia has a hard crush on OOP, which is a bag of accidental complexity - and complexity is a breeding ground for unreliability.
Ok, I think we’ve gone too far. There absolutely is such thing as 0 bugs and sometimes code changes don’t have bugs. That is not to say it can be garunteed.
Never said that, just that quick turnaround for deliveries will usually mean more bugs, and having some extra time usually means less bugs
I think when the author says "bug free", it's from the user perspective. where bugs either need to go out of your way to trigger or are so esoteric it's impossible to think about hitting without that user themself knowing the code inside out. Games is definitely an industry where the quality of code has always dipped to a point where users can easily hit issues in normal use, and only gets worse as games get more complex. That's where it gets truly intolerable
"Cheap. Fast. Good. Pick two."
The pessimistic viewpoint is that you get to pick up to one.
I assume I’m not infallible.
Writing some unit tests, C++ and mocking in my case, give both the team and myself some confidence I’ve not made things worse.
I’m the most experienced dev in the department.
Once you get beyond "I wrote a function" and "I tested a function" and even still "I tested a function that was called by a function over the wire", you will come to a realization that no matter how edgy your edge cases, no matter how thorough your QA, there will always - ALWAYS be 0-day "undefined behavior" in certain configurations. On certain hardware. On certain kernels. It's an assurance that I guarantee that I'm almost positive it's bug free, since it passed tests, it passed human eyes, and it passed review - fingers crossed.
Maybe show-stopping bugs are somewhat unique to software engineering, but all somewhat-complex products are flawed to some extent imho.
It might be an unergonomic handle on a frying pan, furniture that visibly warps under the slightest load (looking at ikea shelfing) or the lettering coming off the frequently used buttons on a coffee machine.
I’d also quibble about the buttons on the coffee machine. They might be properly designed, just subject to the normal wear-and-tear that is inevitable in the real world. This is not a defect, physical devices have finite lifespans.
As far as computers go… if we got to the point where the main thing that killed our programs was the hard drives falling apart and capacitors drying out, that would be quite impressive and I think everyone would be a little bit less critical of the field.
As the discipline has evolved, the high integrity tools are slowly being incorporated into typical languages and IDEs to generally improve quality cheaper. Compare C++ to rust for example, whole classes of bugs are impossible (or much harder to make) in rust.
And this is almost the simplest possible program.
But I mean, I basically agree that the ecosystem is too complicated.
New bridges collapses, dams overflow s, planes crashes, vaccines kills, food kills, leaning towers and skyscrapers, capsized ships - catastrophic flaws are everywhere.
There's a saying that I think fits very well here: "Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands."
You don't want a bridge to cost 50 years and quadrillions of dollars to build, you want a cheap bridge safe for the next 50 years done in 2 years.
I would not call the resulting bridge "bug free", of course.
Bugs arise from the inherent complexity introduced by writing code, and our inability to foresee all the logical paths a machine can take. If we're disciplined, we write more code to test the scenarios we can think of, which is an extremely arduous process, that even with the most thorough testing practices (e.g. SQLite) still can't produce failproof software. This is partly because, while we can control our own software to a certain degree, we have no control over the inputs it receives and all of its combinations, nor over the environment it runs in, which is also built by other humans, and has its own set of bugs. The fact modern computing works at all is nothing short of remarkable.
But I'm optimistic about AI doing much better. Not the general pattern matching models we use today, though these are still helpful with chore tasks, as a reference tool, and will continue to improve in ways that help us write less bugs, with less effort. But eventually, AI will be able to evaluate all possible branches of execution, and arrive at the solution with the least probability of failing. Once it also controls the environment the software runs in and its inputs, it will be able to modify all of these variables to produce the desired outcome. There won't be a large demand for human-written software once this happens. We might even ban software by humans from being used in critical environments, just like we'll ban humans from driving cars on public roads. We'll probably find the lower quality and bugs amusing and charming, so there will be some demand for this type of software, but it will be written by hobbyists and enjoyed by a niche audience.
People who know how the sausage is made will always know of a bunch of bugs that haven't been fixed exactly because they aren't impactful enough to be worth the effort required to fix them.
The other is working out the correct specification is far harder than coding is.
For example it is trivial to write a bug free program that multiplies an integer between 3 and 45 by two.
No, it's functional. If it has bugs, it's not bug-free. By definition.
To quote a former marketing guy “it should work out what the user intends to do and do it”?
Not sure what your quote is supposed to mean. That's a textbook example of someone who doesn't understand software at all making laughable requests of their engineers.
The reason for that quote is from what you have said a bug would be anything you didn't expect, even if it is consistent or not with the specification as that merely affects if we classify it as functional or not (a classification I profoundly disagree with, obviously). It is simply a negative rephrasing of what the marketing guy said and laughable in the same way.
> One plausible definition is “system deviates from its specification”
And that's quite reasonable. So I actually retract my argument.
For my own definition, I was considering a bug to be any behavior that the software engineers weren't expecting. Because those can exist invisibly for a long time until they become so bad they become visible. They can also exist for decades without causing any problems to functionality at all.
Really bug free commercial software does not exist. And can't exist. There are always bugs which are known but would not be fixed.
You would have to introduce things like tolerance to hardware failure, but that is outside the spec of the software as stated.
In the same way I am not a real Scotsman, your toy example of an easily specified snippet of a function that doesn't do anything useful is not real software.
No-one in the real world gives a damn about your 'spec of the software as stated'
When testing should start? BEFORE the first line of code is written.
(Personally, I think code is itself more analogous to the specification artifacts of other professions - eg. blueprints - and the process of creating the machine code of what is analogous to construction / manufacturing something to those specs.)
And even having said that, even the vast majority "bug free" software that nearly always appears to be operating "within spec" will have corner cases that are expressed in very rare situations.
But none of this is an argument for nihilism about quality! It is just not the right expectation going into a career in software that you'll be able to make things that are truly perfect. I have seen many people struggle with that expectation mismatch and get lost down rabbit holes of analysis paralysis and overengineering because of it.
Except for ones that do, obviously.
The key reason to make the distinction is because the fuzzy business of translating intention into specification needs to be fully accepted as an ongoing negotiation process of defining exactly what the specification is, and integrated into repeated deterministic verification that that is what has been delivered. Failing to do that is mainly a great way for certain software management structures to manipulate people by ensuring everything is negotiable all the time, and has the side effect that no one can even say if something is a bug or not. (And this pattern is very clear in the discussion in this thread - there is a definite unwillingness to define what a bug is).
IME the process of automated fuzzing radically improves all round quality simply because it shakes out so many of the implicit assumptions and forces you specify the exact expected results. The simple truth is most people are too lazy and/or lack the discipline needed to do it.
Nobody actually has any trouble knowing what a bug is. Like, this is just a non-issue, I've never in my career spent a non-negligible amount of time debating with anybody whether something is or isn't a bug. We discuss whether fixing bugs have worthwhile return on investment, and we discuss the relative priority between fixing bugs and doing other things, but this meta-debate about "well technically it complies with this spec so is it even a bug, really?" just never comes up. We all know what bugs are.
I have been writing code for almost a decade now, and still make errors. I don't believe anyone is capable of producing bug free software.
I have also seen plenty of bugs in apps and games. I don't think I have ever witnessed a major game patch that was bug free.
Microwave has random errors from time to time.
Robo vacuum freezes.
Parking meter malfunction.
Public transport ticket machine don't want to give me a ticket.
Online banking failing to make a transfer because I use UI with other language.
Mobile banking failing to make a transfer because I use not native currency.
Car has issues as well, incorrect fuel amount is injected by computer.
Online pages have tons of bugs, many are barely usable.
Most people in this thread understand that users' interests only matter insofar as they impact business profit.
I just think you're having a different conversation.
Fact is, if you ship enough 'low probability' bugs in your product, your probabilities still add up to a point where many customers are going to hit several of them.
I've used plenty of products that suffer from 'death by a thousands cuts'. Are the bugs I hit "ship blockers"? No. Do I hit enough of them that the product sucks and I don't want to use it? Absolutely.
Right, you don't want to use Microsoft Word, or SalesForce, or Apple vs Android, or X Whatever. It's highly unlikely you'll have a choice if you use it though.
If you're in a very large FANNG type company, and say you have 1000 components that each ship 1 bug each day that has a 0.1% chance of breaking something critical, that translates to a less than 50% chance you ship a working OS on any given day. And that may mean the entire company's productivity is impacted for the day depending on how broken it is.
"A programmer who releases buggy software and fixes them is better than a programmer who always releases perfect software in one shot, because the latter doesn't know how to fix bugs."
Perhaps similar to the saying that a good driver will miss a turn, but a bad driver never misses one.
The doesn't know how to fix bugs idea seems pretty silly.
The moral of the sayings is, that "perfect" programmer is actually a bad programmer because he wouldn't know how to fix bugs by virtue of never needing to deal with them.
To reuse the driver analogy, the driver who never misses a turn is a bad driver because he doesn't know what to do when he does miss a turn.
If a software developer consistently delivers high-quality software on time and on budget, that means they're good at their job, pretty much by definition. It would make no sense to infer they're bad at fixing bugs.
It would make sense to infer instead that they're good at catching and fixing bugs prior to release, which is what we want from a software development process.
> the driver who never misses a turn is a bad driver because he doesn't know what to do when he does miss a turn
Missing a turn during a driving test will never improve your odds of passing.
The driver who never misses a turn presumably has excellent awareness and will be well equipped to deal with a mistake should they make one. They also probably got that way by missing plenty of turns when they were less experienced.
What we are discussing isn't a real programmer we might actually find. No, we are talking about a hypothetical "perfect" programmer. This "perfect" programmer never wrote a bug in his entire life right from the moment he was born, he never had a "when they were less experienced" phase.
Obviously, that means this "perfect" programmer also never debugged anything. For all the perfect code he writes, that makes him worse than a programmer who writes buggy code but also knows how to go about debugging them.
Whenever I hear a developer say "I never ship buggy code", I am always cautious to dig in more and understand what they mean by that.
I jest, but that should be the gold standard for anything life-critical and good to have for mission-critical software. Alas, we're not there yet.
https://www.fastcompany.com/28121/they-write-right-stuff
There are well-formed statements that can be proved but which assert that its godelized value represents a non-provable theorem.
Therefore, you must accept that it and its contradiction are both provable (leading to an inconsistent system), or not accept it and now there are provable theorems that cannot be expressed in the system.
Furthermore, that this can be constructed from anything with base arithmetic and induction over first-order logic (Gödel's original paper included how broadly it could be applied to basically every logical system).
And, since many proofs are done by assuming a statement's inverse and trying to prove a contradiction, having a known contradiction in the set of provable statements can effectively allow any statement to be proven. Keeping the contradiction is not actually an option.
Godel means that we can't have an algorithmic box that we put a program into and out comes a true/false statement of halting.
Nothing is stopping you from writing the proof manually for each program that you want to prove properties for.
ALSO, you can write sub-turing complete programs. Those are allowed to have automated halting proofs (see idris et al).
While, yes, computability and provability are very closely related, it's important to get attribution correct.
More details on what Gödel's Incompleteness Theorem really said are in a sibling comment so I won't repeat them here.
Really? Says who?
Or perhaps you'll prove it from first principles. Although if turns out to be difficult, that's okay. Somebody mentioned something about systems being either complete or consistent but never both. Some things can be true but not proveably so. Can't quite remember who it was though.
[I believe that] it's important to get attribution correct.
- Futurama (kind of)
Gödel's really was a rather unique mind, and the story of his death is kind of sad.. but I wonder if it takes such a severe kind of paranoia to look for how math can break itself, especially during that time when all the greatest mathematicians were in pursuit of formalizing a complete and consistent mathematics.
It might be wrong, but it's pretty easy to inspect and has a much higher chance of being right than your code does.
You then use proof refinement to eventually link this very high level statement down to the code implementing it.
That's the vision, at least, and it's sometimes possible to achieve it. See, for example, Ironfleet: https://www.microsoft.com/en-us/research/publication/ironfle...
Bugs in the formal spec aren't impossible, but use of formal methods doesn't prevent you from doing acceptance testing as well. In practice, there's a whole methodology at work, not just blind trust in the formal spec.
Software developed using formal methods is generally assured to be free of runtime errors at the level of the target language (divide-by-zero, dereferencing NULL, out-of-bounds array access, etc). This is a pretty significant advantage, and applies even if there's a bug in the spec.
Disclaimer: I'm very much not an expert.
Interesting reading:
* An interesting case-study, albeit from a non-impartial source [PDF] https://www.adacore.com/uploads/downloads/Tokeneer_Report.pd...
* An introduction to the Event-B formal modelling method [PDF] https://www.southampton.ac.uk/~tsh2n14/publications/chapters...
Yes, with the current state of the art, adopting formal methods means adopting a radically different approach to software development. For 'rapid application development' work, it isn't going to be a good choice. It's only a real consideration if you're serious about developing ultra-low-defect software (to use a term from the AdaCore folks).
> it doesn't even eliminate bugs because the formal proof could include a bug too
This is rather dismissive. Formal methods have been successfully used in various life-critical software systems, such as medical equipment and avionics.
As I said above, formal methods can eliminate all 'runtime errors' (like out-of-bounds array access), and there's a lot of power in formally guaranteeing that the model's invariants are never broken.
> I suppose it could help in trivial cases like sanity checking that a value isn't null or that a lock is only held by a single thread
No, this doesn't accurately reflect how formal methods work. I suggest taking a look at the PDFs I linked above. For one thing, formal modelling is not done using a programming language.
If you can't use formal proof just because the user can't be arsed to wait where it is supposed to be necessary, then the software project conception is simply not well designed.
This is the fallacy people have when thinking they can "prove" anything useful with formal systems. Code is _already_ a kind formal specification of program behavior. For example `printf("Hello world");` is a specification of a program that prints hello world. And we already have an abundance of tooling for applying all kind of abstractions imaginable to code. Any success at "proving" correctness using formal methods can probably be transformed into a way to write programs that ensure correctness. For example, Rust has pretty much done so for a large class of bugs prevalent in C/C++.
The mathematician's wet dream of applying "mathematical proof" on computer code will not work. That said, the approach of inventing better abstractions and making it hard if not impossible for the programmer to write the wrong thing (as in Rust) is likely the way forward. I'd argue the Rust approach is in a very real way equivalent to a formal specification of program behavior that ensures the program does not have the various bugs that plagues C/C++.
Of course, as long as the programming language is Turing Complete you can't make it impossible for the programmer to mistakenly write something they didn't intend. No amount of formalism can prevent a programmer from writing `printf("hello word")` when they intended "hello world". Computers _already_ "do what I say", and "do what I mean" is impossible unless people invent a way for minds to telepathically transmit their intentions (by this point you'd have to wonder whether the intention is the conscious one or the subconscious ones).
As I already said in my reply to xmprt, formal methods have been used successfully in developing life-critical code, although it remains a tiny niche. (It's a lot of work, so it's only worth it for that kind of code.) Google should turn up some examples.
> Code is _already_ a kind formal specification of program behavior.
Not really. Few languages even have an unambiguous language-definition spec. The behaviour of C code may vary between different standards-compliant compilers/platforms, for example.
It's possible to reason formally about C, but it's not an ideal match. https://www.eschertech.com/products/ecv.php
The SPARK Ada language, on the other hand, is unambiguous and is amenable to formal reasoning. That's by careful design, and it's pretty unique. It's also an extremely minimal language.
> `printf("Hello world");` is a specification of a program that prints hello world
There's more to the story even here. Reasoning precisely about printf isn't as trivial as it appears. It will attempt to print Hello world in a character-encoding determined by the compiler/platform, not by the C standard. It will fail if the stdout pipe is closed or if it runs into other trouble. Even a printf call has plenty of complexity we tend to just ignore in day to day programming, see https://www.gnu.org/ghm/2011/paris/slides/jim-meyering-goodb...
> Any success at "proving" correctness using formal methods can probably be transformed into a way to write programs that ensure correctness
You've roughly described SPARK Ada's higher 'assurance levels', where each function and procedure has not only an ordinary body, written in SPARK Ada, but also a formal specification.
SPARK is pretty challenging to use, and there can be practical limitations on what properties can be proved with today's provers, but still, it is already a reality.
> Rust has pretty much done so for a large class of bugs prevalent in C/C++
Most modern languages improve upon the appalling lack of safety in C and C++. You're right that Rust (in particular the Safe Rust subset) does a much better job than most, and is showing a lot of success in its safety features. Programs written in Safe Rust don't have memory safety bugs, which is a tremendous improvement on C and C++, and it manages this without a garbage collector. Rust doesn't really lend itself to formal reasoning though, it doesn't even have a proper language spec.
> The mathematician's wet dream of applying "mathematical proof" on computer code will not work
Again, formal methods aren't hypothetical.
> I'd argue the Rust approach is in a very real way equivalent to a formal specification of program behavior that ensures the program does not have the various bugs that plagues C/C++.
It is not. Safe languages offer rock-solid guarantees that certain kinds of bugs can't occur, yes, and that's very powerful, but is not equivalent to full formal verification.
It's great to eliminate whole classes of bugs relating to initialization, concurrency, types, and object lifetime. That doesn't verify the specific behaviour of the program, though.
> No amount of formalism can prevent a programmer from writing `printf("hello word")` when they intended "hello world"
That comes down to the question of how do you get the model right? See the first PDF I linked above. The software development process won't blindly trust the model. Bugs in the model are possible but it seems li...
But right from the outset the approach is doomed to fail because its proponents write like they don't know what they are talking about and think they can write bug-free software.
It really should be "write software with a formal spec". Once people start talking about "proof" in practice it sounds dishonest. It isn't possible to prove software and the focus really needs to be on the spec.
The code is already a formal spec.
Unless there are bugs in the language/compiler/interpreter, what the code is essentially formally well defined.
As programming languages get better at enabling programmers to communicate intention as opposed to being a way to generate computer instructions, there's really no need for a separate "spec". Any so called "spec" that is not a programming language is likely not "formal" in the sense that the behavior is unambiguously well defined.
Of course, you might be able to write the "spec" using a formal language that cannot be transformed into machine code, but assuming that the "spec" is actually well defined, then it's just that "compiling" the spec into machine code is too expensive in some way (eg. nobody has written a compiler, it's too computationally hard to deduce the actual intention even though it's well defined, etc.). But in essence it is still a "programming language", just one without a compiler/interpreter.
You can formally prove that it doesn't have certain kinds of bugs. And that's good! But it also is an enormous amount of work. And so, even for life-critical software, the vast majority is not formally proven, because we want more software than we can afford to formally prove.
Proving that the program has no bugs is akin to proving that the program won't make you feel sad. Like ... I'm not sure we have the math.
One of the more important jobs of the software engineer is to look deep into your customer's dreams and determine how those dreams will ultimately make your customer sad unless there's some sort of intervention before you finish the implementation.
What actually would solve the customer's problem -> What the customer thinks they want -> What they communicate that they want -> What the requirements collector hears -> What the requirements collector documents -> How the implementor interprets the requirements -> What the implementor designs/plans -> What the implementor implements.
Formal proofs can help with the last 3 steps. But again that's assuming the implementor can formalize every requirement they interpreted. And that's impossible as well, there will always be implicit assumptions about the running environment, performance, scale, the behavior of dependent processes/APIs.
It helps with a small set of possible problems. If those problems are mission-critical then absolutely tackle them, but there will never be a situation where it can help with the first 5 steps of the problem, or with the implicit items in the 6th step above.
Formal proving doesn’t really scale much further, definitely not to “enterprise” product scale.
Usually it's something extremely simple to fix too.
No kidding. I'm 2 or 3 years into working on a SaaS app started in ~2013 and I still get bug reports from users that make me say "what!? we have that feature!?"
The difference is that software can be used before it is fully ready, and it makes sense to do so. No one can really use a 90% finished power plant, but software at 95% capacity is still usually "good enough"
The end result is never 100% perfect, but somewhere in between "not too bad" and "good enough".
If you want to patch a bridge, it's gonna cost you. Even if you only need to close down a single lane of traffic for a few hours you are looking at massive expenses for traffic control, coordination with transportation agencies, etc.
For most software it's pretty inexpensive to ship updates. If you're a SaaS company regular updates are just part of your business model. So the software is never actually done. We just keep patching and patching.
In some contexts, it is much more expensive to push out updates. For example, in the 00s, I worked on a project that had weather sensors installed in remote locations in various countries and the only way to get new software to them was via dial-up. And we were luck that that was even an option. Making international long distance calls to upload software patches over a 9600 baud connection is expensive. So we tested our code religiously before even considering an update, and we only pushed out the most direly needed patches.
Working on SaaS these days and the approach is "roll forward through bugs". It just makes more economic sense with the cost structures in this business.
There's even an added factor of learning more about what is really needed by putting a 95% done product into use.
Heck, it explains (stretching it here) space-x's success with an iterative approach to rocket design.
No, you just complain that your taxes are being used to build expensive roads and bridges. Or you think airplanes are far too expensive. Or that new cars are insanely expensive.
There are cost trade offs. In general, better quality more expense.
Also in software there is not an excessive amount of software engineers in relation to demand for software. So SWEs can get paid a lot to go build crappy software.
More importantly though, for the most part we choose not to do engineering. By which I mean this - we know how to do this better, and we apply those techniques in areas where the consequences of failure are high. Aerospace, medical devices, etc.
It differs a bit industry to industry, but overall the lessons are the same. On the whole it a) looks a lot more like "typical" engineering than most software development and b) it is more expensive and slower.
Overall, we seem to have collectively decided we are fine with flakier software that delivers new and more complex things faster, except where errors tend to kill people or expensive machines without intending to.
The other contributing thing is it's typically vastly cheaper to fix software errors after the fact than, say, bridges.
1. The ECU ("hard" engineering)
2. The infotainment system ("soft" engineering)
Now, an interesting thing I have noticed is that "soft" software engineering pays more. Often substantially more.
Further to that, it's often informative to figure out what makes a company money. The highest paid software development roles tend to be doing things that are closer to revenue, on average. If you are a software developer at a hardware company (or an insurance company, or whatever), you aren't that close. Even worse if you are viewed as a cost center.
yeah. Who are those trillion dollar businesses and what do they rely on?
- Apple: Probably the better example here since they focus a lot on user-facing value. But I'm sure they have their own deals, B2B market in certain industries, R&D, and ads to take into account
- Microsoft: a dominant software house in nearly every aspect of the industry. But I wager most of their money comes not from users but other businesses. Virtually every other companies uses Windows, Word, and those that don't may still use Azure for servers.
- Alphabet: ads. Need I say more? Users aren't the audience, they are the selling point to other companies.
- Amazon: a big user facing market, but again similar to Microsoft. The real money is b2b servers.
- Nvidia: Again, user facing products but the real selling point is to companies that need their hardware. In this case, a good 80% of general computing manufacturers.
- Meta: Ads ans selling user data once again
- Tesla: CEO politics aside, it's probably the 2nd best example. Split bewteen a user facing product that disrupted an industry and becoming a standard for fuel in the industry they disrupted. There's also some tangential products that shouldn't be underestimated, but overall a lot of value seems to come from serving the user.
General lesson here is that b2b and ads are the real money makers. if you're one level removed that financial value drops immensely (but not necessarily to infeasible levels, far from it).
That sounds like an awfully bold claim. I have the feeling we understand software a lot better than we understand mechanical engineering (and by extension material sciences) or fluid dynamics. By a big margin.
I worked with finite element software and with CFD solvers, you wouldn't believe how hard it is to simulate a proper airflow over a simple airfoil and get the same results as in the wind tunnel.
To the contrary, it's nearly canonical. Most of the problems pointed out in the 70s (mythical man month) have still not been resolved, 50 years later.
>you wouldn't believe how hard it
Oh, I'd believe it (I've designed and built similar things, and had colleagues in CFD).
But you are definitely cherry picking here. The problem with CFD is we don't understand the fluid dynamics part very well; turbulence is a big unsolved problem still, though we have been generating better techniques. This is so true that in an undergraduate physics degree, there is usually a point where they say something like: "now that you think you know how lots of things work, let's introduce turbulence"
But a lot of mechanical engineering and the underlying physics and materials science is actually pretty well understood, to the degree that we can be much more predictive about the trade offs than typically is possible in software. Same goes for electrical, civil, and chem. Each of them have areas of fuzziness, but also a pretty solid core.
Of course it is easier for a developer to walk away from something incomplete than an architect and the contractors involved in a physical project, but still, I hardly think that there is really much difference in terms of timelines.
Software is still absolutely king for number of large scale projects that just never ship, or ship but never work.
Even with all of those applied, we wouldn’t be magically better. Complexity is simply unbounded. It’s almost impossible to reason about parallel code with shared mutable state.
Team are flying the airplane, the se time rebuild it to the zeppelin, testing new engines inflight.
Or construction. Let's build apartment block, but for few apartments we will test new materials, new layout, etc. Once there are walls of the first apartments we will let people live there. We will build how we can, according to the napkin plan. In the end we will put all tenants in and stress test strength of the structures. Or one day people return home and their apartments have totally different design and layout because someone from the HOA decided so to get a promotion.
Now we don't accept an engineer saying, "this bridge will probably collapse without warning", which we do accept with software. So there is a difference.
ive had this sentiment thrown at me too often by peak move fast and break things types. it's too often a cudgel to dispense with all QA in favor of more new feature development. shipping shit that has the same pattern of flaws youve encountered in the past when youve been shown ways to catch them early but couldnt be bothered isnt accepting that you cant catch everything, it's creating a negative externality.
you usually can make it someone else's problem and abscond with the profits despite, but that doesn't mean you should
Not buildings, not bridges, not cars, not airplanes, not software. There are mistakes in every field of engineering and the best we can hope for is to minimize them as much as possible.
Engineering is knowing (among other things) how to categorize the potential mistakes, develop procedures to reduce the chance of them being made and in the case that some slip through (and they will), estimate their impact and decide when you're "good enough."
Software engineering is no different.
Currently ways of coding to this kind of standard exist. But they are stupid. It involves things like no dynamic memory allocation, only fixed length for-loops, and other very strict rules. These are used in aerospace, where bugs are rather costly and rare.
Engineers cannot be responsible for all the vertical stack and the components which were built by others. If somebody claims it is bug free then they have not enough experience. Pretty much anything can fail, we just need to test as many possible cases as possible with a variety of available tools to reduce the chances of bugs.
Where I work is 50% qa effort at least, 50% by budgets, and devs do much >50% automated testing as part of what we call development.
Anything less and projects take longer, because bugs found later cost so much more to fix.
You can mix those three together and produce a desired output, but don't expect perfection, perfect software only appears when the three variables tend to infinity
What you read on the internet is mostly ... bullshits.
You need to understand the domain, you have to design something that solves an actual problem or delivers tangible value to someone, you need a holistic approach to user experience.
I've seen this too many times to count.
Just an anecdote, but, we had a "Software Development" class like this in CS (I took it in the '90s) and even though it followed a waterfall development model[0] and we used Gantt charts, QA (testing) was a big part of it and 1 of our 4 team members (or maybe 2 of 4 worked on it together) was primarily responsible for it. (I wrote the parser and the made diagrams/documentation for the parser.)
The description (in an old catalog[1]) is:
Software specification, design, testing, maintenance, documentation; informal proof methods; team implementation of a large project.
Turns out I didn't need to look up the old catalog because the description is exactly the same still! Except it's CS 371 now, and the longer "Learning Outcomes" for the course has some newer stuff (agile and version control) but otherwise is all the same things I learned at the time.
[0] https://en.wikipedia.org/wiki/Waterfall_model
[1] https://nmsu.contentdm.oclc.org/digital/collection/catalogs/... (C S 372 in the lower right corner)
And the good thing about snapshots is that they provide a visual way to communicate a preview to stakeholders of what may change, both for the currently-being-developed workflow as well as to prevent visual regressions elsewhere - so they're inherently more easily justifiable than "we're spending time on testing that you'll never see."
The article is correct that treating QA as the last phase of the project, and a disposable phase at that, is a recipe for disaster. But if you make it an ongoing part of a project as individual stories are burned down, and rebrand it as part of a modern development workflow, then it's an entirely different conversation.
We can roughly say we can have three out of: (high) quality, (low) time, (low) communication complexity, and (low) money. (time is a dependent here.)
People are trying to apply factory processes and structures to a team sport, an engineering discipline. You do not teach or build a basketball team by breaking down each attack phase into steps and checkmarks.
You try to minimize communication and make the team work as one. It is a team and individual building, not a process building exercise. You make a plan, and follow the Moltke's the Elder conclusion:
"no plan of operations extends with any certainty beyond the first contact with the main hostile force."
(Or paraphrased as you have heard: No plan survives contact with the enemy.)
All (types of) Engineers know this. But software engineering is "special." And it is not a "move fast and break things issue." That is part of all engineering or team playing too.
It is the type of business mentality, that because a plan did not go exactly as expected we need to add more process. Whatever that process may be. Because if "I as a manager add a process, then the next failed plan, I am covered, and I will blame the individuals."
Process has a place to ensure things happen in a legal and moral framework. And minimize adverse circumstances -- e.g. we bet all the hedge fund money accidentally when running tests.
Process is used differently in most startups and corporations with not the team in mind.
Do you ever yell at a traditional architect and ask them when it's going to be done? It's always when the client is happy or makes their mind up about it. A lot of dev is like this.
The two overlap most of the time in subtle ways where the science gives an important foundation, such as learning Big O notation and low level memory concepts where exposure helps. I've personally seen this with a smart coworker who didn't go through university and is great at programming but I'll catch him on certain topics such as when he didn't know what sets and maps were and when he tries to sleep a second instead of properly wait on an event.
However, the differences between computer science and building software are problematic. Watching my wife go through university, she's had to struggle with insanely hard tasks that will not help her at all with software, such as learning Assembly and building circuits. The latest example is the class where she's learning functional programming is not actually teaching it to her. Instead, they combined it with how to build a programming language, and so instead of giving her toy problems to teach the language she is having to take complex code she doesn't understand well that generates an entirely different programming language and do things like change the associativity of the generated language. In the end, she feels like she's learned nothing in that class, despite it being her first experience with functional programming.
On the flip side are the things that are necessary for software that aren't taught in university, like QA. For me personally, back when I was in university a decade ago I never learned about version control and thought it was just for back up. Similarly, I never learned databases or web, as the required classes were instead focused on low level concepts as Assembly and hardware. My wife is at least learning these things, but even then they often seem taught badly. For example, when they tried to teach her QA, instead of hardcoded unit tests, they made her give random inputs and check to make sure the output was correct. Of course, checking the output can only be done by rewriting all of your code in the testing files, and if there's a bug in your code it'll just get copied, so that kind of defeats the purpose. Even when the assignments are relevant there is often no teaching on them. For example, her first ever web code was a project where they told her to hook up 6 different technologies that they had not gone over in class, with only the line "I hope you've learned some of these technologies already".
For some classes of bugs it can (e.g. if the software is so bad as to open you to a class-action lawsuit; in b2b software bugs that put you in breech of contract), but for many classes of consumer software, it's not clear to me that shipping software that works better is rewarded. Picking not-too-buggy software ahead of time is hard, people are slow to switch (even when the people encountering the bugs are the people selecting the software, which is often not the case), and people are good at subconsciously avoiding triggering bugs.
But at that scale (in my experience), QA is up front and center and is typically a core pillar of engineering orgs.
From a cynical point of view, if those hundreds of thousands of users will use your product despite the bugs, does it matter?
I'm super cognizant that the cited example of "social media company" is its own special ball of wax, due to the network effect, and I wish to holy hell I knew how to fix that bug :-)
2. Is this article from the 90s where we ship software on CDs or floppy disks? In today's world where the concept of a "release" is often blurred by continuous delivery pipelines (and this is considered a good practice), having a quality insurance department manually assuring that no bugs are in that release seems downright archaic.
Anything connected to a network could be released frequently if people wanted to. Not everything is connected to a network though.
Some things should not roll out updates without extensive testing, including manual testing to verify nothing was missed.
There are tons of software without continuous delivery.
- why is testable software higher quality? Does it add value to the software? I'd venture that untestable software has the same value (if not more) than testable software (due to time-to-market). You can write software that is 'obviously correct' and "high quality" at the same time, without any tests.
- Why does software that can adapt to new features increase the quality? If that is the case, we must argue that WordPress is extremely high-quality software. Or SAP.
- How does architecture influence quality? If that is the case, then there isn't any need for different architectural styles since there should be "one true style" that has the best quality software.
Sacrificing quality assurance to favour other aspects is common, but the quality usually suffers.
A company favouring time to market over testability is likely to release buggy software. They can get away with it.
Adaptability is a common quality, but you can find counterexamples. WordPress and SAP are successful software that may not check all the quality boxes.
Some architectures are for sure worse than others, and there isn’t one good architecture for all kinds of problems.
> why is testable software higher quality? Does it add value to the software? I'd venture that untestable software has the same value (if not more) than testable software (due to time-to-market). You can write software that is 'obviously correct' and "high quality" at the same time, without any tests.
Note I said testable software, not software with tests (there is a difference!)...I'd agree that software with tests (which is by definition testable software) has a huge developer cost to it that may not always be in the best interest of the company (like you said, time to market might be important). But in my experience, writing code in a way that can be tested later is only marginally more costly (time-wise) than writing code that isn't. A good example of this is writing modules that communicate with message passing and have state machines over direct function calls. The former has a slightly higher cost for dev time, but you can always retro-fit tests to it once you've achieved market penetration. You can't always do that with direct function calls.
> Why does software that can adapt to new features increase the quality? If that is the case, we must argue that WordPress is extremely high-quality software. Or SAP.
This is a good point that you bring up. I think what we are getting at ultimately is that quality and value are distinct entities. Software can have high value without being high quality. In my mind, being able to provide the business with new value-producing functionality without causing a spike in bug reports is my (admittedly vague) standard.
> How does architecture influence quality? If that is the case, then there isn't any need for different architectural styles since there should be "one true style" that has the best quality software.
Architecture has to match how the software teams communicate with each other. Like actually communicate, not how the org chart is made (see Conway's Law). So my point is then that if there are two separate teams, your code should communicate between two "modules" that have an interface between them. Just like real life. It would be silly to implement a micro service architecture here. That's why Amazon's SOA design works for them: it matches how teams are organized.
- Who gets to define testability?
- I want to add a coffee maker to my crash test dummy; is the lack of room for the filter and water tank a sign of a bad design? Or not flexible enough for my feature?
- (cue meme) "You guys have organizational structure?"
- Who gets to claim the impedence mismatch? What are those consequences? Wait, where are the dependencies defined again outside of the software?
I do (just kidding!)...Testability is the ability to add testing at a later point. There is no hard definition of this, but if you can't test at least 75% of your public facing functions then I'd say you don't have testability. Remember testability means you can have a tigher feedback loop which means that you don't have to test in production or in the physical world. This means you get where you want to go faster.
> - I want to add a coffee maker to my crash test dummy; is the lack of room for the filter and water tank a sign of a bad design? Or not flexible enough for my feature?
I know you are joking, but imagine for a second that your business did in fact invent a brand new way to test crashes and that coffee makers were the key to breaking into that market. If the dummy can't accommodate that then...yes! It is a bad design, even if it was previously a good design.
> - (cue meme) "You guys have organizational structure?"
Remember: there always is an organizational structure, with or without a formal hierarchy. You want to match your software to the real one.
> - Who gets to claim the impedence mismatch? What are those consequences? Wait, where are the dependencies defined again outside of the software?
There are no "the company blew up" consequences with this type of failure mode. Instead you get a lot of "knock on" effects: high turnover, developer frustration, long time to complete basic features and high bug re-introduction rates. This is because software is inherently a human endeavor: you need to match how it is written to how requirements and features are communicated.
It’s not that crazy.
If this is given, every other metric, like features, bugs, performance is just a linear dependence on development resources (maybe except documentation, but that is kinda an externality).
But how do you define the requirements is the real question (and problem)…
In my experience it is generally wise to avoid abstractions and copy/paste things a couple of times, once the code base matures good abstractions will be more obvious. Even then it's good to think about future changes, will these 2 things want to evolve separately in the future? If the answer is YES, then maybe coupling them is not a great idea. I think there was a really good Kent Beck talk about coupling vs cohesion somewhere.
Another thing to think about is breaking things, if changes are local to one single endpoint then any changes there can only break that endpoint, edge cases and scenarios to consider are only relevant to that endpoint. When changes to a core abstraction are required then hundreds of use cases/edge cases need to be considered - why are we creating so many core abstractions in our systems in the name of DRY?
I've also found that the more moving parts you add the harder a system becomes to learn, the S in SOLID is probably to blame for that. The only single responsibility principle is useful for is unit tests (easier to mock), but many times harder to understand. If the actual functionality is not local to the file things become ungreppable via code search, understanding the entire system requires an IDE and jumping around to each and every beautiful lpad() implementation and trying to piece what is happening one 3 line function at a time.
Then there is also layering to consider, if 2 pieces of code look somewhat similar but belong to different layers (example controller and DAO layer, then also care must be taken to not make an abstraction that couples these together, or to couple 2 unrelated modules together that could otherwise have their own life cycle).
These are just some aspects I could think of that I think about when creating abstractions, but somehow I see engineers focus too much on DRY. Maybe they got burned so bad some time in the career by forgetting to change something in 2 places?
HOWEVER, the real culprit are the MANAGERS.
After 50+ years of skanky software development policy aimed at low balling cost its time to blame the right people. No amount of cajoling, "business/budget" speak manipulation is going to fix a fundamental flaw in how managers at that level are trained and behave.
We have stop being apologists for mistakes not of our making. If using QA techniques would increase the managers bonus then we will see it being used. If all it does is make better software then this post will be rewritten in 50 years and would still be relevant.
IME, “QA” doesn’t really correlate with quality software, nor is there really a time vs. quality trade off. Bad software is often also delivered poorly, and high quality software can be delivered quickly.