I haven't worked with Haskell myself (and we don't currently support Haskell), but it looks like Cabal is the package manager, which resolves the full graph of dependencies for your project.
Most dependencies are specified like "network-info >= 0.2" (taken from a random example: https://github.com/bitemyapp/blacktip/blob/master/blacktip.c...), which means you don't know which version will actually be used until you build the project. Which is our point in this blog post. If you just scanned that text file, you'd just be guessing at which versions of open source libraries are actually being used.
I don’t have a very in-depth understanding of dynamic analysis myself, so this may be somewhat of a simplistic question: Why not just resolve to the latest version of each library/module present on the system during the first pass through the list of dependencies?
The package manager decides which packages are used based on how dependencies are specified. For example, you might ask for library-A >= 1.1, but a transitive dependency specification (a dependency of a dependency) may specify library-A >= 1.0 && <= 1.6. If versions up to 1.7 are available, you'd probably get 1.6. Probably. Because with some package managers, if you have version 1.5 sitting around in a local cache, it may use that instead.
Basically, never try to guess what the package manager is going to resolve. Just let it do its job, just as it does for your production builds, and use that information to look up any associated vulnerabilities.
You may be talking about two different situations. "r-w" might be thinking about a build style where compilation just uses the versions of libraries available on the system, rather than going to the package manager on a per-build basis and re-fetching dependencies (so, something more like a C build than a Rust, Haskell, or Node build).
As an aside, and echoed by a few other comments in the thread, what you're calling "dynamic analysis" is what I know as "static analysis", like what I want Coverity to do (watch the build process, and monitor the source that actually goes into each of my build artifacts). "Dynamic analysis" brings to mind something more like Valgrind, or some other tool that monitors and profiles a program during execution.
Good point. The wikipedia definition is my understanding as well. Analyzing during build time is what scan-build does. They call it the Clang Static Analyzer.
Coverity has been doing "dynamic analysis" in the way this article uses the term for a long time; they ended up creating a system call tracer that lets them monitor an arbitrary build process to find code to analyze:
Agreed. Their definition of static analysis is laughably bizarre as well; there are many security and safety properties that can be proved by static analysis but not by dynamic analysis. Additionally, many of the problems they attribute to static analysis (non-specific versioning and recursive dependencies) are caused by the package managers themselves, and can be solved using static analysis if the build tool is well defined and well behaved.
Their tool has a few major weaknesses:
1. Builds are not reproducible - Reproducible builds require pinned package versions, which they specifically avoid. This could result in security holes if a dependent package version was bumped after the parent project was tested.
2. Subject of analysis - Their main target of analysis is the build file. This severely limits the extent of their analysis and requires them to build new tools for each build tool used.
3. Underestimation - Since a code path must be exercised in order to analyze it, you are guaranteed that the set of vulnerabilities detected is a subset of (or equal to) the true set of vulnerabilities. This is the opposite of what I believe should be the default: Always prefer false positives over false negatives. Both static analysis and binary analysis allow the programmer to over-estimate their analysis, guaranteeing that all vulnerabilities will be found.
I was going to say same about build systems being able to do static analysis. Some can even be correct-by-construction. Further, if compiler like CompCert or CakeML is used, then static analysis on source will apply to binary too. Finally, static tools can analyze a lot more than just one file or whatever.
I'm not sure what you mean by #1, the builds are as reproducible as the build system being used.
For #2, SourceClear doesn't build the software under test, it's hooked into the build via various methods. For Maven and Gradle, those are plugins. For NPM and Bundler, the existing build files contain the complete dependency graphs as determined by the build system. The analysis is quite accurate, I daresay more so than any other tool. Yes, it requires implementation work for each build stack, but that's the price you pay for accuracy.
In response to your #3, SourceClear doesn't report only vulnerabilities verified by call paths, it reports on all components will known vulnerabilities and denotes if a call path was found.
Disclaimer, I'm a Co-Founder and have spent a great deal of time writing scanning code.
None of the terms they define share the definitions that are understood by everyone in the field.
Static analysis refers to analysis techniques that understand code without recourse to execution, usually during compilation. (Static binary analysis does exist, but the difficulties of disentangling code from data in native code makes it usually impracticable). The Java FindBugs tools or Clang's static analyzer are good examples of static analysis techniques.
Dynamic analysis refers to analysis techniques that rely on gathering information using actual executions of a program. Even pedestrian techniques like profiling and code coverage are considered dynamic analysis techniques.
Binary analysis is a completely orthogonal concept. It refers to the analysis of programs without recourse to the source code, and is usually understood to refer almost exclusively to native machine code as opposed to bytecode languages like JVM or .NET IL. Most binary analysis tools are dynamic (such as Pin or Valgrind) but there do exist static binary analysis tools such as GrammaTech's CodeSurfer.
This doesn't really capture the whole picture of techniques, though. Symbolic execution and its step-child concolic execution aren't best thought of as static or dynamic analysis.
If you consider the build system to be a language in itself, I think this is appropriately called dynamic analysis.
For example, a profiler for a Makefile should record the time taken for each build step as the Makefile is running. It's not about profiling the program being built.
I don't understand what exactly they're saying. Seems to me they're comparing apples, oranges, and cherries. I don't know if they're willingly doing this or if they're trying to make the problem sound harder than it is.
You only need to perform whatever security analysis is necessary after all your dependencies are resolved. This does not require anything "dynamic". You just call the packager manager, wait for all the dependencies to be resolved, then verify there are no vulnerable versions, which is just a matter of looking up the relevant pieces in some database somewhere.
Which is fine if that's what they're doing but this article seems to be just smoke and mirrors mostly.
We published this because there are lots of legacy tools that approach this problem with two specific weaknesses:
* Many tools just scan a text file where you declare your dependencies, which misses transitive dependencies and won't tell you exactly which versions are in use
* They use incomplete datasets for vulnerabilities, like CVE. Most OSS projects don't create CVEs for vulnerabilities, so it is a mostly useless datasource.
Besides having a complete view of the dependencies, we have a research team that is finding and disclosing new vulnerabilities all the time, which you can see here:
https://www.sourceclear.com/registry/explore
That's a lot more clear and a much better description. I didn't get any of that from the post. Instead you explained several disparate processes each of which has their place in security analysis and declared a specific version the winner.
Interesting. So if I understood correctly, you dynamically analyze the build process (and that's why you use that term) instead of just parsing the build file, because you don't know exactly how the dependencies are going to be resolved from the package manager during the build process. How do you verify that a specific version of a library is used during the build and not some other version (you just do a hash lookup or have a way to generate signatures with small false positives) ? Also, what happens if the package manager is compromised (for example it informs you that she used version 2.0 but instead she used a vulnerable version) ? For the call graphs, do you find the relationships between each procedure for the whole project and if so, isn't that literally static analysis ? Sorry if my questions don't make sense or are trivial, I am just looking it from a research perspective, because I am working on somehow similar things.
I agree w/all the confusion regarding terminology. It's still static if you're not executing the code.
But, back to the premise: it would be really helpful if you [the author] could illustrate security defects which can be detected using "dynamic analysis" which cannot be detected with "static analysis." Legitimate, actually exploited/able vulnerabilities would be ideal.
22 comments
[ 2.3 ms ] story [ 53.4 ms ] threadMost dependencies are specified like "network-info >= 0.2" (taken from a random example: https://github.com/bitemyapp/blacktip/blob/master/blacktip.c...), which means you don't know which version will actually be used until you build the project. Which is our point in this blog post. If you just scanned that text file, you'd just be guessing at which versions of open source libraries are actually being used.
Basically, never try to guess what the package manager is going to resolve. Just let it do its job, just as it does for your production builds, and use that information to look up any associated vulnerabilities.
As an aside, and echoed by a few other comments in the thread, what you're calling "dynamic analysis" is what I know as "static analysis", like what I want Coverity to do (watch the build process, and monitor the source that actually goes into each of my build artifacts). "Dynamic analysis" brings to mind something more like Valgrind, or some other tool that monitors and profiles a program during execution.
(you could think of stack as analogous to ruby's Gemfile.lock plus rbenv, or python's virtualenv - it makes the whole build reproducible)
http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lin...
(starting from "Laws of Bug Finding")
Their tool has a few major weaknesses:
This article is BS.
For #2, SourceClear doesn't build the software under test, it's hooked into the build via various methods. For Maven and Gradle, those are plugins. For NPM and Bundler, the existing build files contain the complete dependency graphs as determined by the build system. The analysis is quite accurate, I daresay more so than any other tool. Yes, it requires implementation work for each build stack, but that's the price you pay for accuracy.
In response to your #3, SourceClear doesn't report only vulnerabilities verified by call paths, it reports on all components will known vulnerabilities and denotes if a call path was found.
Disclaimer, I'm a Co-Founder and have spent a great deal of time writing scanning code.
Static analysis refers to analysis techniques that understand code without recourse to execution, usually during compilation. (Static binary analysis does exist, but the difficulties of disentangling code from data in native code makes it usually impracticable). The Java FindBugs tools or Clang's static analyzer are good examples of static analysis techniques.
Dynamic analysis refers to analysis techniques that rely on gathering information using actual executions of a program. Even pedestrian techniques like profiling and code coverage are considered dynamic analysis techniques.
Binary analysis is a completely orthogonal concept. It refers to the analysis of programs without recourse to the source code, and is usually understood to refer almost exclusively to native machine code as opposed to bytecode languages like JVM or .NET IL. Most binary analysis tools are dynamic (such as Pin or Valgrind) but there do exist static binary analysis tools such as GrammaTech's CodeSurfer.
This doesn't really capture the whole picture of techniques, though. Symbolic execution and its step-child concolic execution aren't best thought of as static or dynamic analysis.
For example, a profiler for a Makefile should record the time taken for each build step as the Makefile is running. It's not about profiling the program being built.
You only need to perform whatever security analysis is necessary after all your dependencies are resolved. This does not require anything "dynamic". You just call the packager manager, wait for all the dependencies to be resolved, then verify there are no vulnerable versions, which is just a matter of looking up the relevant pieces in some database somewhere.
Which is fine if that's what they're doing but this article seems to be just smoke and mirrors mostly.
* Many tools just scan a text file where you declare your dependencies, which misses transitive dependencies and won't tell you exactly which versions are in use
* They use incomplete datasets for vulnerabilities, like CVE. Most OSS projects don't create CVEs for vulnerabilities, so it is a mostly useless datasource.
Besides having a complete view of the dependencies, we have a research team that is finding and disclosing new vulnerabilities all the time, which you can see here: https://www.sourceclear.com/registry/explore
But, back to the premise: it would be really helpful if you [the author] could illustrate security defects which can be detected using "dynamic analysis" which cannot be detected with "static analysis." Legitimate, actually exploited/able vulnerabilities would be ideal.