Curious if the author has read anything by Watts Humphrey? He spent his entire career at SEI arguing this point and attempting to develop the tools and processes to bear it out. This article reads like a repetition of his arguments from A Disicipline for Software Engineering (1995).
I learned in a discussion with "kragen" that "software engineering" sort of took on a life of its own that turned into a nightmare of management-focused stuff harmful to programming. Most programmers, especially mainstream crowd, have an aversion to the term from the permanent association. I started saying CompSci and focusing on prior techniques rather than "engineering" due to that permanent scarring. As in my other comment, I just focus on specific techniques and processes that help programmers do their job with evidence they work.
It's my suggestion to you. I'm going to try to dig up and read that paper out of curiosity as I know many useful things came out of software engineering research. Most won't, though, the second they see "SEI" or "software engineering." So, might be best for us to just drop that and focus on improvements to "programming" or "development" with specific techniques.
I'm mixed about the post, esp the science part. The science of developing robust software is great and pretty consistent going back decades varying mostly in specific tools and tactics. Mainstream programming just doesn't apply it although more adoption in past decade of key techniques. Here's some computer science from the 1960's-1980's used in robust and secure system development (esp Orange Book B3 or CC EAL6) people might want to copy. I'm taking an empirical route where I reference techniques that were applied to many real-world projects with lessons learned in papers or studies that were consistent. All one can do with limited data & these aren't in order of importance.
1. Formal, non-English (eg math/logical) specifications of requirements or abstract design. English is ambiguous and misreadings of it caused countless errors, even back then. CompSci researchers tried formal specs with both English as a start and precise notations (eg Z, VDM, ASM's, statecharts) for clarity in specifics. Result was many inconsistencies caught in highly assured systems and protocol specs before coding even began.
2. High assurance stuff often used mathematical (formal) verification. Whether that worked or made sense was hit and miss. More on it later. Yet, virtually all of them said there was benefit in restrictions on the specs, design, and coding style to fit the provers' limitations. Essentially, they used boring constructs that were easy to analyse and this prevented/caught problems. Don't be too clever with design or code. Wirth and Hansen applied this to language design to bake safety & comprehension in with minimal to low loss in performance.
Note: Led to Nick P's Law of Trustworthy Systems: "Tried and true beats novel or new." Always the default.
3. Dijkstra's THE project showed that modular, layered design with careful attention to interfaces (and interface checks) makes for most robust and maintainable software. Later results confirmed this where each module must fit in your head and control graph that's pretty predictable with minimal cycles prevented all kinds of local-becomes-global issues. Many systems flawless (or nearly so) in production were built this way. Dijkstra correctly noted that it was very hard to do this even for smart people and average developer might screw structuring up a lot. Solid prediction... but still worth striving for improvement here.
4. Fagan ran empirical studies at IBM that showed a regular, systematic, code review process caught many problems, even what tests missed. Turned that into formal inspections with the periodicity and prioritizing tuned per organization for right cost-benefit. Was generalized to whole SDLC by others in high robustness areas. Improved every project that used it from then on. Exactly what parameters to use is still open-ended but periodically looking for well-known flaws with reference sheet always works.
5. Testing for every feature, code-path, prior issues outside of code base, and common use-case. All of these have shown repeated benefits. There's a cut-off point for each that's still an open, research problem. However, at a minimum, usage-based testing and regression testing helped many projects achieve either zero or near-zero, user-facing defects in production. That's a very important differentiator as 100 bugs user never experiences is better than 5 that they do regularly. Mills' Cleanroom process combined simple implementation, code review, and usage-testing for insanely-high, statistically-certifiable quality even for amateur teams.
6. By around 60's-70's, it became clear that the language you choose has a significant effect on productivity, defects, maintenance, and integration. Numerous studies were run in industry and military comparing various ones. Certain languages (eg Ada) showed vastly lower defects, equal/better productivity, and great maintenance/integration in every study. Haven&...
> The science of developing robust software is great and pretty consistent going back decades varying mostly in specific tools and tactics
Nope (since uh-huh/nuh-uh is the starting point you've taken). Most science is BAD SCIENCE and I have doubts about the claims made by either the author or the topics you reference. Almost all formal studies are about specific tools and tactics in small sample sizes. This isn't proven science as much as they are test results. - https://vimeo.com/9270320 + http://quorumlanguage.com/evidence.php should orient you toward what proof is and how we can reference it in a reasonable manner. There will be plenty of bias to argue about how the tests are conducted or what the data demonstrates.
You reply with a combination of trolling and endorsing a specific link that you believe represents doing it better. The first thing I looked at, syntax study, was already weaker than prior work I've seen on the subject. The language choice was highly biased while not focusing on enough metrics for comparison of strengths and weaknesses. Between field use and its first "evidence," your Quorum is already weaker than most of what I presented in terms of empirical weight. Due to the trolling, I'm hesistant to even watch the video in case his presentation is something likes yours in terms of content.
@ non-trolling, HN readers also concerned about science in software
Back to science, though, given other readers might be interested in that point. Software development is a combination of human- and machine-driven processes to turn some requirements into working, maintainable code. A subset of this, significant it turns out, can be analyzed for effectiveness with enough different case studies, sample sizes, objective measures, and repeats to make a claim about it with believable evidence. For process stuff, it's not going to be as logically formulated or minutely analyzed as a math equation or something. Not usually, anyway. Instead, it works more like this:
1. A recurring problem exists in the development process or artifact.
2. Measurements are taken of how often that problem occurs in general and for specific scenarios.
3. A hypothesis is formed as to why it exists and a solution to it.
4. The hypothesis is tested by applying the technique to several projects likely to experience the problem with objective records on outcomes and subjective accounts from users for a take on less tangible aspects.
5. Significant reductions in the problem in the objective data provide supporting evidence that the theory is correct.
6. The theory (solution) is tested by others against the problem to find more supporting evidence or counter-examples to the claim.
7. If it's correct, the results continue to speak for themselves in it being a solution with likely modifications as the problem and solution space are further explored.
Note: If it's performed by humans, it's wise to either look out for or control for issues like Hawthorne effect. Must be sure that it's the overall solution that's working rather than people's heightened attention to an issue being studied.
The above 1-7 have applied to my list of techniques to varying degrees ranging from strong analysis + repeated, field results (eg formal specs, code reviews) to clear theory, models, assessment, and massive field results (eg type-systems, usage-driven testing). The evidence is quite clear and favors them as effective with repeated successes that matched the theory. Many other aspects of programming I left off the list because the evidence or studies were not clear. Or were non-existent. Hence me not including the parent's strawmen ("most science") or red herring (Quorum) on a list of techniques with decades of measured results behind them across many software projects.
Science, what little exists in our field, is on my side if the reader is focused on evidence aspects such as the Why, How, and Got Job Done. They collectively make up The Right Thing in robust, system development. YMMV for individual techniques or tools on a given project. Pick best tools for the job, prioritize your development budget, and so on. Common sense. People looking at where to start dealing with recurring issues or overall robustness have an evidence-backed list in my main comment, though. There are also papers and books on each showing where to apply them and most effectively.
"Saying you have presented empirical weight, is not compelling when you haven't and, frankly, can't."
The proof is in the literature I'm referencing rather than me dumping all of it in the comment box. Anyone studying PL, QA, and so on literature should have seen research on effectiveness of basic techniques. Here's a few examples of methods of evaluation I claimed were scientific vs people guessing that their stuff seems better in vague ways.
1. Experiments conducted between similarly skilled groups doing same things with only difference being language or methodology. Those with new technique do significantly better in ways that match the expected properties of technique. Hypothesis based on real-world data, experimental confirmation, fix/reject if necessary, rinse, repeat.
2. Study of problems in software from a specific language identifies specific features as root cause, proposes a fix, the fix is implemented in significant test cases, and the problem goes away entirely or is significantly reduces. This is replicated across groups and codebases. Easiest example here is safe (Ada) or automatic (GC) techniques to manage memory to reduce or eliminate its many, common issues. Supported by theory, sometimes proofs, & countless applications in industry.
3. A certain verification approach espouses specific modeling, analysis, refinement, etc approach to achieve certain goals. The work on formal side translates into finding defects, enhancing performance, etc on the code side. Implies the model or method were correct, esp with diverse replication of results. This happened a lot in Orange Book days for catching security violations but most applicable to hardware with formal, equivalence checking. Theory, some proofs, and confirmation in production.
4. In some cases, properties are mathematically formulated, proven in theorem prover, checked by humans + machines, and tested on real-world examples to further validate them. Two words: type systems. Again, theories, proofs, and real-world validation. Tons of testing, too.
Most of the best evidence in the literature works something like that. You're saying none of the above constitute scientific evidence for a claim about a software tool or method? That's a big claim itself that contradicts what most of research community thinks. The people successfully finding bugs with Astree, avoiding out-of-bounds with languages that don't allow it, catching safety violations with protocol checkers, and so on will all be surprised about your claim. They and I thought the tools would work based on evidence presented and field results that confirmed the theories. Turns out we were all drinking spiked Kool-Aid, leading to hallucinations that followed. Or wrong on a technicality.
Got a link that briefly explains why all of that is unscientific and what methods count as scientific evidence in software tools? (without 1hr videos)
Note: Looking up more on Quorum is on my agenda. Already saved What Works Clearinghouse document. I'm not rejecting it outright in our discussion. Yet, you've indicated proven models, worked case studies, analysis techniques effective on 1+ million plus samples, mathematical proofs... that none of this is science. So, I'm starting with wanting an easy resource on what you think counts as scientific evaluation, argument, and method. Then, I can go from there in future discussions about specific topics and what science says about them.
I think we agree more than you think ;) In my original article, I quoted some papers and studies. And on twitter, somebody occused me that "All science is crap". So I tried to write another article with only my opinion on the subject - The article linked here on HN.
Reading it instead of skimming it shows there's indeed plenty of agreement even without the science part. The minimizing f_n reflects well what the science established.
Far as defects, problems people report are definitely a start. Fagan Software Inspection Process had the nice idea of standardizing in a list the kind of issues in code that would be treated as defects and fixed by priority. They could be lack of a bounds check, code style issue, whatever team thought was important. So, my definition for defect in such a scheme was "unacceptable code."
"Low defect potential." Interesting metric. Often a function of complexity, coupling, amount of state being thrown around, and handling of interface assumptions. Scientists should come up with ways of assessing stuff like that then see what effect it had on defects during changes among many codebases. Of course, the minimizing f_n stuff should help here.
"When you find a defect in production a week after deploying the code, it is probably still rather cheap to fix: All the original developers are still there, they still remember what they did, the documentation is still accurate."
That's actually a really good point. I don't remember seeing anyone else say it. I'll have to remember that.
"Well crafted code is self documenting. When I read the code together with it's tests, I want to be able to understand what is going on. Without reading some external documentation (wikis, ...)"
Well-crafted code section is overall good but I'm not sure about this. I've seen code that can work like this. Yet, I've also seen high assurance work where there were abstract specs of what it does w/ pre and post conditions that were easier to understand for the use rather than modification of a module. Could also be fed into analysis tools. Some modern languages include that stuff in the code itself and can export interface documentation.
That's not even getting into issues like consistently using a filesystem where all kinds of extra commands are needed for non-obvious reasons that need a whole article to describe. A 1 line operation suddenly takes more like 10. So, I'd say make the code self-documenting where possible but there's potentially good reasons to have the other things. If they're needed or useful.
"Good software design is, in my opinion, less subjective than some other terms I described above."
I agree. You did alright explaining that.
"It's Not Much Slower Anyway The Microsoft study on TDD"
You're conflating quality and testing here. They're not the same. There's plenty of low-quality software with lots of testing. Likewise, Cleanroom (extreme example) normally resulted in high quality from app/code structure and verification via code revies before testing. Same with Fagan inspections and OpenBSD's auditing process. All of those had vastly higher quality than TDD projects far as I've seen. And that's without arguing against testing, as you know I'm for it.
Maybe renaming anything like that in your articles to specifically focus on benefit or cost of testing. So, this would be "TDD/Testing is not much slower anyway." The arguments you made for it are good and actually apply to all quality techniques. I think you intended that but set it up wrt to testing. So, maybe it's not the title and just something about that section in particular. Sounds nitpicking but my reaction is more that we need to always convey quality as a combo of prevention, code review, and testing. Code review & testing at the least. Gotta keep putting it into their heads until they can't forget. ;)
"Let's look back to "Minimizing F_n (For Large n)":"
"Executable specifications (a part of the test suite) make sure everybody understands the current functionality that is implemented. And that "documentation" cannot be outdated, because otherwise t...
Thank you so much for your detailed reply! That's a lot to think about :)
Thanks for the hints about defects. I like "unacceptable code", but it's probably still a definition not everyone can live with :)
"It's Not Much Slower Anyway The Microsoft study on TDD"
"You're conflating quality and testing here. They're not the same."
You are right. My arguments here are only for TDD, and I actually did not want to write a TDD article :) I'll think about how to improve that section. Also, CleanRoom is really interesting, but I don't know very much about it and never experienced it.
"At some point, your users will recognize that it takes longer and longer..."
"Should be one of the top points."
I was thinking about the order of the points a lot, and no particular order seemed really right. It now comes at a point where I think I have established all the preconditions: Speed means lead time, cost depends on speed, high external quality means faster, high internal quality means faster. But you're right, the main points I want to bring across come very late in the article. As for the illustrations: Maybe I'll write another article about how users care about internal quality - Because I've worked in so many legacy code projects that I think I have a good view on that :)
I like your points about the iOS stuff and the short-term jobs. I'll think about them some more. I'm still not sure if I believe the "unlimited downside" idea 100% (because only very few things are truly unlimited), but it's an interesting concept.
"
Thank you so much for your detailed reply! That's a lot to think about :)"
You've very welcome. :)
"Thanks for the hints about defects. I like "unacceptable code", but it's probably still a definition not everyone can live with :)"
I kind of came up with it off the top of my head. Point is that the code will be rejected on relatively arbitrary criteria. It might be known defects, coding style, what libraries were used... anything. Hard to find a word or phrase covering all that which people will agree with.
"Also, CleanRoom is really interesting, but I don't know very much about it and never experienced it."
...summarizes it for you in about 5 min of reading. The first attempts to make software in a scientific way, engineered software I call it, were done by Margaret Hamilton (founder of our field) at NASA plus academics like Dijkstra and Hoare. They all produced nearly defect-free systems in a systematic, often mathematical way. Harlan Mills took the lessons to industry when developing Cleanroom: a combo of lightweight formal methods, design constraints, spec/code verification, incremental development, and usage-driven testing. He wanted SW products to be as low-defect as HW clean rooms. Result was code at extremely low defect rate that was consistent enough for statistical certification. While popular, Cleanroom teams actually warrantied their software.
Altran/Praxis is another company I know that (a) truly engineers software and (b) warranties it at specific defect rates. Defect rates are usually better than the Linux kernel and close to the Space Shuttle's control code. Here's a case study of their Correct by Construction process applied to high-assurance, certificate authority:
So, there's you two methods that already maxed out quality (and/or security) on real-world projects. Altran charges an acceptable premium for quality but Cleanroom often cost nothing extra or saved money for same reasons as you article says. Appears the modern programmers still have lessons to learn in software quality from the 1980's. Like I tell people, the old wisdom is often pretty solid even after you fix outdated parts. Our field is just terrible at passing it down. I try to fix that with posts like this so the smart, young people get a boost toward whatever great things they create. Also reduce endless rediscovering of fire and reinventing the wheel. :)
"I was thinking about the order of the points a lot, and no particular order seemed really right. It now comes at a point where I think I have established all the preconditions: Speed means lead time, cost depends on speed, high external quality means faster, high internal quality means faster. "
I'm going to let you keep working on that formula. You've got the right ingredients. Far as order, there might not be any. Many things in engineering or business are holistic where the parts all feed into each other. The goal is an emergent property of that. So, rather than an order, you'd visualize it like a bubble chart or something that with connections between the components and labels for the effect they have. Just something that's clear that they're all connected, all feed into each other, and ignoring one breaks the process for achieving the goal. Just my dos centimos.
"Maybe I'll write another article about how users care about internal quality - Because I've worked in so many legacy code projects that I think I have a good view on that :)"
You should. Matter of fact, you should do a lot mor...
13 comments
[ 3.2 ms ] story [ 46.0 ms ] threadIt's my suggestion to you. I'm going to try to dig up and read that paper out of curiosity as I know many useful things came out of software engineering research. Most won't, though, the second they see "SEI" or "software engineering." So, might be best for us to just drop that and focus on improvements to "programming" or "development" with specific techniques.
I did not read that paper (book?), but will add it to my "To Read" pile. Thank you ;)
1. Formal, non-English (eg math/logical) specifications of requirements or abstract design. English is ambiguous and misreadings of it caused countless errors, even back then. CompSci researchers tried formal specs with both English as a start and precise notations (eg Z, VDM, ASM's, statecharts) for clarity in specifics. Result was many inconsistencies caught in highly assured systems and protocol specs before coding even began.
2. High assurance stuff often used mathematical (formal) verification. Whether that worked or made sense was hit and miss. More on it later. Yet, virtually all of them said there was benefit in restrictions on the specs, design, and coding style to fit the provers' limitations. Essentially, they used boring constructs that were easy to analyse and this prevented/caught problems. Don't be too clever with design or code. Wirth and Hansen applied this to language design to bake safety & comprehension in with minimal to low loss in performance.
Note: Led to Nick P's Law of Trustworthy Systems: "Tried and true beats novel or new." Always the default.
3. Dijkstra's THE project showed that modular, layered design with careful attention to interfaces (and interface checks) makes for most robust and maintainable software. Later results confirmed this where each module must fit in your head and control graph that's pretty predictable with minimal cycles prevented all kinds of local-becomes-global issues. Many systems flawless (or nearly so) in production were built this way. Dijkstra correctly noted that it was very hard to do this even for smart people and average developer might screw structuring up a lot. Solid prediction... but still worth striving for improvement here.
4. Fagan ran empirical studies at IBM that showed a regular, systematic, code review process caught many problems, even what tests missed. Turned that into formal inspections with the periodicity and prioritizing tuned per organization for right cost-benefit. Was generalized to whole SDLC by others in high robustness areas. Improved every project that used it from then on. Exactly what parameters to use is still open-ended but periodically looking for well-known flaws with reference sheet always works.
5. Testing for every feature, code-path, prior issues outside of code base, and common use-case. All of these have shown repeated benefits. There's a cut-off point for each that's still an open, research problem. However, at a minimum, usage-based testing and regression testing helped many projects achieve either zero or near-zero, user-facing defects in production. That's a very important differentiator as 100 bugs user never experiences is better than 5 that they do regularly. Mills' Cleanroom process combined simple implementation, code review, and usage-testing for insanely-high, statistically-certifiable quality even for amateur teams.
6. By around 60's-70's, it became clear that the language you choose has a significant effect on productivity, defects, maintenance, and integration. Numerous studies were run in industry and military comparing various ones. Certain languages (eg Ada) showed vastly lower defects, equal/better productivity, and great maintenance/integration in every study. Haven&...
Nope (since uh-huh/nuh-uh is the starting point you've taken). Most science is BAD SCIENCE and I have doubts about the claims made by either the author or the topics you reference. Almost all formal studies are about specific tools and tactics in small sample sizes. This isn't proven science as much as they are test results. - https://vimeo.com/9270320 + http://quorumlanguage.com/evidence.php should orient you toward what proof is and how we can reference it in a reasonable manner. There will be plenty of bias to argue about how the tests are conducted or what the data demonstrates.
@ non-trolling, HN readers also concerned about science in software
Back to science, though, given other readers might be interested in that point. Software development is a combination of human- and machine-driven processes to turn some requirements into working, maintainable code. A subset of this, significant it turns out, can be analyzed for effectiveness with enough different case studies, sample sizes, objective measures, and repeats to make a claim about it with believable evidence. For process stuff, it's not going to be as logically formulated or minutely analyzed as a math equation or something. Not usually, anyway. Instead, it works more like this:
1. A recurring problem exists in the development process or artifact.
2. Measurements are taken of how often that problem occurs in general and for specific scenarios.
3. A hypothesis is formed as to why it exists and a solution to it.
4. The hypothesis is tested by applying the technique to several projects likely to experience the problem with objective records on outcomes and subjective accounts from users for a take on less tangible aspects.
5. Significant reductions in the problem in the objective data provide supporting evidence that the theory is correct.
6. The theory (solution) is tested by others against the problem to find more supporting evidence or counter-examples to the claim.
7. If it's correct, the results continue to speak for themselves in it being a solution with likely modifications as the problem and solution space are further explored.
Note: If it's performed by humans, it's wise to either look out for or control for issues like Hawthorne effect. Must be sure that it's the overall solution that's working rather than people's heightened attention to an issue being studied.
The above 1-7 have applied to my list of techniques to varying degrees ranging from strong analysis + repeated, field results (eg formal specs, code reviews) to clear theory, models, assessment, and massive field results (eg type-systems, usage-driven testing). The evidence is quite clear and favors them as effective with repeated successes that matched the theory. Many other aspects of programming I left off the list because the evidence or studies were not clear. Or were non-existent. Hence me not including the parent's strawmen ("most science") or red herring (Quorum) on a list of techniques with decades of measured results behind them across many software projects.
Science, what little exists in our field, is on my side if the reader is focused on evidence aspects such as the Why, How, and Got Job Done. They collectively make up The Right Thing in robust, system development. YMMV for individual techniques or tools on a given project. Pick best tools for the job, prioritize your development budget, and so on. Common sense. People looking at where to start dealing with recurring issues or overall robustness have an evidence-backed list in my main comment, though. There are also papers and books on each showing where to apply them and most effectively.
Saying you have presented empirical weight, is not compelling when you haven't and, frankly, can't.
It's telling that the rest of your pointless counter-troll? is devoid of proof when leading with "back to science", which is sad.
The proof is in the literature I'm referencing rather than me dumping all of it in the comment box. Anyone studying PL, QA, and so on literature should have seen research on effectiveness of basic techniques. Here's a few examples of methods of evaluation I claimed were scientific vs people guessing that their stuff seems better in vague ways.
1. Experiments conducted between similarly skilled groups doing same things with only difference being language or methodology. Those with new technique do significantly better in ways that match the expected properties of technique. Hypothesis based on real-world data, experimental confirmation, fix/reject if necessary, rinse, repeat.
2. Study of problems in software from a specific language identifies specific features as root cause, proposes a fix, the fix is implemented in significant test cases, and the problem goes away entirely or is significantly reduces. This is replicated across groups and codebases. Easiest example here is safe (Ada) or automatic (GC) techniques to manage memory to reduce or eliminate its many, common issues. Supported by theory, sometimes proofs, & countless applications in industry.
3. A certain verification approach espouses specific modeling, analysis, refinement, etc approach to achieve certain goals. The work on formal side translates into finding defects, enhancing performance, etc on the code side. Implies the model or method were correct, esp with diverse replication of results. This happened a lot in Orange Book days for catching security violations but most applicable to hardware with formal, equivalence checking. Theory, some proofs, and confirmation in production.
4. In some cases, properties are mathematically formulated, proven in theorem prover, checked by humans + machines, and tested on real-world examples to further validate them. Two words: type systems. Again, theories, proofs, and real-world validation. Tons of testing, too.
Most of the best evidence in the literature works something like that. You're saying none of the above constitute scientific evidence for a claim about a software tool or method? That's a big claim itself that contradicts what most of research community thinks. The people successfully finding bugs with Astree, avoiding out-of-bounds with languages that don't allow it, catching safety violations with protocol checkers, and so on will all be surprised about your claim. They and I thought the tools would work based on evidence presented and field results that confirmed the theories. Turns out we were all drinking spiked Kool-Aid, leading to hallucinations that followed. Or wrong on a technicality.
Got a link that briefly explains why all of that is unscientific and what methods count as scientific evidence in software tools? (without 1hr videos)
Note: Looking up more on Quorum is on my agenda. Already saved What Works Clearinghouse document. I'm not rejecting it outright in our discussion. Yet, you've indicated proven models, worked case studies, analysis techniques effective on 1+ million plus samples, mathematical proofs... that none of this is science. So, I'm starting with wanting an easy resource on what you think counts as scientific evaluation, argument, and method. Then, I can go from there in future discussions about specific topics and what science says about them.
Far as defects, problems people report are definitely a start. Fagan Software Inspection Process had the nice idea of standardizing in a list the kind of issues in code that would be treated as defects and fixed by priority. They could be lack of a bounds check, code style issue, whatever team thought was important. So, my definition for defect in such a scheme was "unacceptable code."
"Low defect potential." Interesting metric. Often a function of complexity, coupling, amount of state being thrown around, and handling of interface assumptions. Scientists should come up with ways of assessing stuff like that then see what effect it had on defects during changes among many codebases. Of course, the minimizing f_n stuff should help here.
"When you find a defect in production a week after deploying the code, it is probably still rather cheap to fix: All the original developers are still there, they still remember what they did, the documentation is still accurate."
That's actually a really good point. I don't remember seeing anyone else say it. I'll have to remember that.
"Well crafted code is self documenting. When I read the code together with it's tests, I want to be able to understand what is going on. Without reading some external documentation (wikis, ...)"
Well-crafted code section is overall good but I'm not sure about this. I've seen code that can work like this. Yet, I've also seen high assurance work where there were abstract specs of what it does w/ pre and post conditions that were easier to understand for the use rather than modification of a module. Could also be fed into analysis tools. Some modern languages include that stuff in the code itself and can export interface documentation.
That's not even getting into issues like consistently using a filesystem where all kinds of extra commands are needed for non-obvious reasons that need a whole article to describe. A 1 line operation suddenly takes more like 10. So, I'd say make the code self-documenting where possible but there's potentially good reasons to have the other things. If they're needed or useful.
"Good software design is, in my opinion, less subjective than some other terms I described above."
I agree. You did alright explaining that.
"It's Not Much Slower Anyway The Microsoft study on TDD"
You're conflating quality and testing here. They're not the same. There's plenty of low-quality software with lots of testing. Likewise, Cleanroom (extreme example) normally resulted in high quality from app/code structure and verification via code revies before testing. Same with Fagan inspections and OpenBSD's auditing process. All of those had vastly higher quality than TDD projects far as I've seen. And that's without arguing against testing, as you know I'm for it.
Maybe renaming anything like that in your articles to specifically focus on benefit or cost of testing. So, this would be "TDD/Testing is not much slower anyway." The arguments you made for it are good and actually apply to all quality techniques. I think you intended that but set it up wrt to testing. So, maybe it's not the title and just something about that section in particular. Sounds nitpicking but my reaction is more that we need to always convey quality as a combo of prevention, code review, and testing. Code review & testing at the least. Gotta keep putting it into their heads until they can't forget. ;)
"Let's look back to "Minimizing F_n (For Large n)":"
"Executable specifications (a part of the test suite) make sure everybody understands the current functionality that is implemented. And that "documentation" cannot be outdated, because otherwise t...
Thanks for the hints about defects. I like "unacceptable code", but it's probably still a definition not everyone can live with :)
"It's Not Much Slower Anyway The Microsoft study on TDD"
"You're conflating quality and testing here. They're not the same."
You are right. My arguments here are only for TDD, and I actually did not want to write a TDD article :) I'll think about how to improve that section. Also, CleanRoom is really interesting, but I don't know very much about it and never experienced it.
"At some point, your users will recognize that it takes longer and longer..." "Should be one of the top points."
I was thinking about the order of the points a lot, and no particular order seemed really right. It now comes at a point where I think I have established all the preconditions: Speed means lead time, cost depends on speed, high external quality means faster, high internal quality means faster. But you're right, the main points I want to bring across come very late in the article. As for the illustrations: Maybe I'll write another article about how users care about internal quality - Because I've worked in so many legacy code projects that I think I have a good view on that :)
I like your points about the iOS stuff and the short-term jobs. I'll think about them some more. I'm still not sure if I believe the "unlimited downside" idea 100% (because only very few things are truly unlimited), but it's an interesting concept.
You've very welcome. :)
"Thanks for the hints about defects. I like "unacceptable code", but it's probably still a definition not everyone can live with :)"
I kind of came up with it off the top of my head. Point is that the code will be rejected on relatively arbitrary criteria. It might be known defects, coding style, what libraries were used... anything. Hard to find a word or phrase covering all that which people will agree with.
"Also, CleanRoom is really interesting, but I don't know very much about it and never experienced it."
This presentation...
http://groups.engin.umd.umich.edu/CIS/course.des/cis376/ppt/...
...summarizes it for you in about 5 min of reading. The first attempts to make software in a scientific way, engineered software I call it, were done by Margaret Hamilton (founder of our field) at NASA plus academics like Dijkstra and Hoare. They all produced nearly defect-free systems in a systematic, often mathematical way. Harlan Mills took the lessons to industry when developing Cleanroom: a combo of lightweight formal methods, design constraints, spec/code verification, incremental development, and usage-driven testing. He wanted SW products to be as low-defect as HW clean rooms. Result was code at extremely low defect rate that was consistent enough for statistical certification. While popular, Cleanroom teams actually warrantied their software.
Altran/Praxis is another company I know that (a) truly engineers software and (b) warranties it at specific defect rates. Defect rates are usually better than the Linux kernel and close to the Space Shuttle's control code. Here's a case study of their Correct by Construction process applied to high-assurance, certificate authority:
http://www.anthonyhall.org/c_by_c_secure_system.pdf
So, there's you two methods that already maxed out quality (and/or security) on real-world projects. Altran charges an acceptable premium for quality but Cleanroom often cost nothing extra or saved money for same reasons as you article says. Appears the modern programmers still have lessons to learn in software quality from the 1980's. Like I tell people, the old wisdom is often pretty solid even after you fix outdated parts. Our field is just terrible at passing it down. I try to fix that with posts like this so the smart, young people get a boost toward whatever great things they create. Also reduce endless rediscovering of fire and reinventing the wheel. :)
"I was thinking about the order of the points a lot, and no particular order seemed really right. It now comes at a point where I think I have established all the preconditions: Speed means lead time, cost depends on speed, high external quality means faster, high internal quality means faster. "
I'm going to let you keep working on that formula. You've got the right ingredients. Far as order, there might not be any. Many things in engineering or business are holistic where the parts all feed into each other. The goal is an emergent property of that. So, rather than an order, you'd visualize it like a bubble chart or something that with connections between the components and labels for the effect they have. Just something that's clear that they're all connected, all feed into each other, and ignoring one breaks the process for achieving the goal. Just my dos centimos.
"Maybe I'll write another article about how users care about internal quality - Because I've worked in so many legacy code projects that I think I have a good view on that :)"
You should. Matter of fact, you should do a lot mor...