Code coverage and static analysis are such valuable tools, I can't believe they are not used more and more frequently .. it really seems that a lot of open source projects would benefit from a coverage/analysis phase on checkin.
It seems that Coverity[0] is offering free static analysis to open source projects. Even if it is a marketing move, it will benefit the open source ecosystem in the long run. You'll waste some time analyzing false positive defects, the handful of critical/security error found make it worth it in my opinion. Notably the duplicate "goto fail" in one of the recent apple vulnerability should have triggered the dead code detection.
Using a language with a more powerful type system (e.g. Haskell) you can derive the same benefits as you would from static analysis tools, but integrated with the rest of your development tools, extensible, and supported by other users of the same language (e.g. library writers). So I think most people who appreciate the benefits tend to move on from C (particularly given how large the library ecosystem in safer languages is these days).
Depends where they come from. The average Python or Perl library seems better written and more tested that something I would write myself. JavaScript seems to have a lot less quality in my experience.
I was thinking about that in response as well; then I began to wonder. Is it that the quality is actual higher, or that I'm more likely to gravitate towards established, well-tested libraries (of which there are fewer in javascript)?
The more I think about it, the more I think it's the latter. There are tons of crap libraries out there for any language - but the more mature languages/ecosystems naturally tend to have a wider selection of more mature libraries.
Think about node modules. How often have you done an n install only to have 10+ other libraries brought in. Some are going to be decent, some not.
I've found oddities like improper implementations of http digest auth and what not. When libraries are small you can sometimes catch these things easily.
As someone who prefers Python to JavaScript (as far as dynamic languages go), but who has been recently using nodejs a lot, I disagree. I've found that there is a very strong testing culture among node hackers.
In my opinion, the biggest security liabilities in Node are with npm:
1. The culture that emphasizes tons of tiny packages, each bringing in dozens of other dependencies managed by multiple maintainers. The increases the attack surface dramatically. All it will take will be one single maintainer of a small package used by dozens of other packages to expose his npm password, and exploit could easily spread to hundreds of node apps in a very short period of time. I'm surprised this hasn't happened already.
2. On a related note, npm's lack of package signing. Most of the other popular dynamic languages (Python, Ruby, Perl) offer or even require package signing. npm does require SSL, but package signing is even more important, since only package signing can prevent the scenario I describe above.
That said, I do see lots of unit testing in node packages, so I can't criticize them there.
Test the hell out of anything you use, both hardware, software, and infrastructure -- and write the tests before you write any code. Double-down on that for stuff like medical tech.
Now, is anybody doing that? Anybody? (Sound of crickets chirping)
The article is about static analysis. Static analysis is the most efficient way to catch a whole host of bugs. Testing would be a waste of everyone's time for these types of bugs. Now, the tools in the article are commercial and perform static analysis on C and C++. Other programming languages have static analysis and code path coverage analysis built into their compilers. Many of those languages are free. Writing tests to cover those types of bugs is time wasted on a problem that is already solved.
I won't change my initial comment, but please be aware that by "testing" I also meant static code analysis. (Thought had it covered with software, hardware, and infrastructure, but forgot one)
Funny enough in a sad way, static analysis tooling was part of the original C tooling, but when developers started creating C compilers for systems other than UNIX, lint seldom came along.
It also did not help the way C compiler phases were split.
If anything, we have to thank LLVM guys for making static analysis part of clang.
> If there's a bug in a library you use it would show up when testing your code that uses the library.
That's not true at all. You may catch some lib bugs, but not all, even through testing of your own code that leverages the lib. Unless you're implying that you'll pass every possible value to your tests which will then cover 100% of any and all code in the lib.
You can still have bugs in parts of code that you use (or would use) given the right inputs. The problem is making your tests so diverse that they test every possible input.
Of course not. You can't even trust the firmware on the storage your OS runs on. We're in a very weird period of time computer security wise. There was the first golden age, when computers weren't easily connected and harmful software was generally not very impactful. Then there was a brief dark age when computers became connected and the utter lack of concern for security was revealed to be hugely exploitable on an unprecedented scale with an enormous impact. A lot of the worst mistakes were corrected and a lot of the most important software was tightened up, leading to a long "security cold war" lasting from roughly the late '90s or early 2000s to today. Right now we're sort of in a cuban missile crisis period in the cold war, or something similar, the impact of computer security holes could result in events (of several possible kinds) which might have as much global economic and geopolitical impact as a major war. Whether that happens or not is as yet unknown, but the potential is there, and will remain there for likely quite sometime (several key vulnerabilities: core libraries such as openssl, essentially every major operating system, firmware on usb devices and storage devices in general, BGP, and SSL/TLS under the CA system).
I don't think most tech companies and governments are taking the risks and threats seriously enough as of yet so I suspect some sort of calamity will have to happen first before things are truly tightened up well enough for anyone to be able to trust anything and not be deluding themselves in doing so.
My opinion is that technology development is going to have to bifurcate -- or perhaps even split into three paths. No longer can you simply sit down and code something. Instead, you'll need to code some set of "rules" (this system can't use the internet, it should never access the GPS, and so on) at the same time. Technology will "run" when all the separate systems agree that everything is behaving correctly.
We've tried to do this with various XML/manifest declarative constructs, but the problem is that these constructs just rely on subsystems which are themselves insecure.
These systems will have to be baked into the metal, and they're going to need to come from different vendors.
Unlikely to happen, but that looks like the way forward.
Well, yes. A big part of the problem is the lack of terminology and models. "Software development" is a stupidly large scope, it encompasses activities that are comparatively as varied as backyard astronomy and heart surgery. There are many different kinds of developers and many different kinds of software, but we lack the words to express the differences clearly. Which makes it that much more difficult to improve processes and standards. It's common for bromides and recommendations to be made as blanket statements applying to all software development, when it's clear that no one set of practices could possibly apply universally.
Worse, many devs aren't even aware this is a problem, and few people are working on fixing it. We're deep in a morass that will take a long, long time to get out of.
You have to, because there's no alternative. And even if it were possible to assure 100% safety from 100% of the code that runs on your computer, how do you know you can trust your hardware?
I guess one approach would be to dramatically simplify required functionality so that it doesn't require a general purpose computer to implement. You could even take things back to the point of having mechanical systems rather than electronic...
[NB I can remember in 1990s being on a project where someone was telling us about the safety shutdown equipment on a UK AGR nuclear power station - it was mechanical logic as when the plants were designed in the 1960s they wanted to use really well tried and tested technology.]
Let's down-vote these types of comments that answer, vaguely, the question proposed in the headline and reveal that they did not even skim the topics in the OP. Totally worthless.
Maybe the title shouldn't propose a question that is controversial (and obviously click-bait), and then have the bulk of the article be about something completely different.
You're right, I should have read more of the article to realize that it's basically about nothing (someone looking for, and fixing bugs in a library he uses, which everyone who cares does).
Who cares. It's finding and reporting genuine bugs in widely used open source libraries, and most of the times it's even suggesting a fix or a workaround. Think about it: each article is worth half a dozen to a dozen high quality bug reports.
Moreover, the guys behind PVS-Studio seem like great guys. The CTO, Andrey Karpov, was interviewed in early 2014: http://www.viva64.com/en/b/0231/.
Selling software development tools isn't easy. PVS-Studio "adverts" are interesting and useful to the community.
I actually really enjoy these articles. Seeing real pitfalls and thinking about how to correct them is useful, as well as a reminder about how bugs lurk everywhere. "The Last Line Effect" in particular will stick with me forever.
(The title in this case should have been less link-baity, though. I almost didn't even click on it, til I saw the URL.)
Might be, but it also shows how many C and C++ developers still don't care about static analysis tools, given the amount of issues that always get found and used as advertising material.
If your definition of trust is that it passes static analysis without throwing any warnings, then you probably can't trust it. But this is hardly a new or useful insight, almost all code has known bugs in it at the time it ships and the additional value of discovering another bug is generally quite low. The value of discovering bugs lies in understanding how they relate to an undesired behaviour of a running system.
You can't guarantee trusted software even your code and the libraries are 100% trusted / bug less, as the combined software creates emergent properties which will also have to be properly tested. As such you can only trust a software system if it has been properly tested.
If this is the case, then you can provide 100% trusted software even if it has been built with insecure code bases / libraries, as long as all the external interfaces has been tested for all possible inputs.
Off course this is extremely difficult / impossible. This is why we will never arrive at a point where software is 100 trustworthy.
Your comment provides no value. The OP has specific examples of tools being used to find bugs in software all without the need for testing. Testing is the least efficient way to fix those bugs. Tooling, as demonstrated in the OP, and other applications of programming language theory are far more efficient ways to have correct software. First, let the computer find the bugs we don't know about, then we can focus on higher goals.
Testing is the least efficient way to fix those bugs.
Testing isn't a way of fixing bugs. It's a way of making sure you don't introduce bugs when things change. The important thing to remember that 'code the first implementation' - where you go from having no code to having some code - is a change just like any other.
Writing code without bugs is a lot more efficient than writing code with bugs and then fixing them.
The title doesn't really fit the article. The author just picked up an open source C/C++ library and found some bugs in its dependencies using their commercial static analysis tool.
This is hardly a surprising result, and does not tell us anything new about how one might evaluate or build highly trusted software.
That seems unlikely to be the case; the variable being assigned is through a pointer to a struct, which is already being accessed in the context. (Plus, of course, __attribute__((unused)) or similar in a macro would be preferable for silencing warnings) It's pretty likely that, had this occured at the top of a function, and been a direct assignment, the author would not have flagged it (indeed, perhaps PVS-Studio ignores that case; seems easy enough to implement).
E.g. in older GCC you would get
int x;
switch (!!foo) {
case 0:x=0;break;
case 1:x=1;break;
/no default/
}
print(..x)
Or many other cases and the compiler will give an erroneous uninitialized warning. you can assign x to itself at the stop to eliminate the warning without changing the generated code.
Java has a very good sand-boxing mechanism (AcessController), which helps limit the effect of bugs in libraries. Firstly, the memory model is highly abstract; no pointer arithmetic and type-less references. To mess with the memory, the thread needs to drop down to native code, and this where sandboxing can trap this behavior.
The same mechanism can also help increase trust in a third-party library from a security point of view. Every access to a file or network resource, for example, is checked before being granted.
Compare this with the worst library include language: JavaScript. There is no sand-boxing at all - code is simply inlined. (PHP's require does this too.) But for some reason web developers pile on the shady libraries when they should be extremely discerning.
54 comments
[ 3.8 ms ] story [ 121 ms ] thread[0] https://scan.coverity.com/
Sadly many managers see this as extra cost, rather something that will reduce costs in project lifetime.
The more I think about it, the more I think it's the latter. There are tons of crap libraries out there for any language - but the more mature languages/ecosystems naturally tend to have a wider selection of more mature libraries.
I've found oddities like improper implementations of http digest auth and what not. When libraries are small you can sometimes catch these things easily.
In my opinion, the biggest security liabilities in Node are with npm:
1. The culture that emphasizes tons of tiny packages, each bringing in dozens of other dependencies managed by multiple maintainers. The increases the attack surface dramatically. All it will take will be one single maintainer of a small package used by dozens of other packages to expose his npm password, and exploit could easily spread to hundreds of node apps in a very short period of time. I'm surprised this hasn't happened already.
2. On a related note, npm's lack of package signing. Most of the other popular dynamic languages (Python, Ruby, Perl) offer or even require package signing. npm does require SSL, but package signing is even more important, since only package signing can prevent the scenario I describe above.
That said, I do see lots of unit testing in node packages, so I can't criticize them there.
Now, is anybody doing that? Anybody? (Sound of crickets chirping)
I won't change my initial comment, but please be aware that by "testing" I also meant static code analysis. (Thought had it covered with software, hardware, and infrastructure, but forgot one)
It also did not help the way C compiler phases were split.
If anything, we have to thank LLVM guys for making static analysis part of clang.
That's not true at all. You may catch some lib bugs, but not all, even through testing of your own code that leverages the lib. Unless you're implying that you'll pass every possible value to your tests which will then cover 100% of any and all code in the lib.
Your tests will be testing the parts of code in the library that your application uses, which is the part that matters to you.
I don't think most tech companies and governments are taking the risks and threats seriously enough as of yet so I suspect some sort of calamity will have to happen first before things are truly tightened up well enough for anyone to be able to trust anything and not be deluding themselves in doing so.
We've tried to do this with various XML/manifest declarative constructs, but the problem is that these constructs just rely on subsystems which are themselves insecure.
These systems will have to be baked into the metal, and they're going to need to come from different vendors.
Unlikely to happen, but that looks like the way forward.
Worse, many devs aren't even aware this is a problem, and few people are working on fixing it. We're deep in a morass that will take a long, long time to get out of.
[NB I can remember in 1990s being on a project where someone was telling us about the safety shutdown equipment on a UK AGR nuclear power station - it was mechanical logic as when the plants were designed in the 1960s they wanted to use really well tried and tested technology.]
You're right, I should have read more of the article to realize that it's basically about nothing (someone looking for, and fixing bugs in a library he uses, which everyone who cares does).
Moreover, the guys behind PVS-Studio seem like great guys. The CTO, Andrey Karpov, was interviewed in early 2014: http://www.viva64.com/en/b/0231/.
Selling software development tools isn't easy. PVS-Studio "adverts" are interesting and useful to the community.
(The title in this case should have been less link-baity, though. I almost didn't even click on it, til I saw the URL.)
If this is the case, then you can provide 100% trusted software even if it has been built with insecure code bases / libraries, as long as all the external interfaces has been tested for all possible inputs.
Off course this is extremely difficult / impossible. This is why we will never arrive at a point where software is 100 trustworthy.
edit - spelling
Testing isn't a way of fixing bugs. It's a way of making sure you don't introduce bugs when things change. The important thing to remember that 'code the first implementation' - where you go from having no code to having some code - is a change just like any other.
Writing code without bugs is a lot more efficient than writing code with bugs and then fixing them.
Testing is good for doing one thing: Validating that the code meets a specific quality target.
This is hardly a surprising result, and does not tell us anything new about how one might evaluate or build highly trusted software.
Self assignment is used to silence overly aggressive and incorrect uninitialized warnings in GCC (esp prior to 4.5).
I'm not sure if thats what the code there is doing— not enough context, but I'd guess it's likely. Odd that the author hasn't seen that before.
Another one that I've seen quite often is casting to void.
E.g. in older GCC you would get int x; switch (!!foo) { case 0:x=0;break; case 1:x=1;break; /no default/ } print(..x)
Or many other cases and the compiler will give an erroneous uninitialized warning. you can assign x to itself at the stop to eliminate the warning without changing the generated code.
The same mechanism can also help increase trust in a third-party library from a security point of view. Every access to a file or network resource, for example, is checked before being granted.