I've started it and a few pages in I would largely agree with this review, it's very America-centric. You have to simultaneously believe that the NSA are the best in the world, and that foreign hackers despite being nowhere near as smart managed to steal all their hacking tools, and it's lucky for them that they did otherwise they would have no chance against the NSA because they are the BEST! USA! USA!
If it doesn't get better than that pretty quickly I'm unlikely to finish it.
I listened to a discussion with Perlroth on a podcast where she said things that implied strongly that exploits, in general, writ large, were leaked NSA tradecraft. I don't have even 1/5th the skin in this game Halvar does, and I found it offensive enough to yell out loud in my car, just at the implication.
In reality I think it's kind of an embarrassment how much CNE technology goes the other way, from industry and research to the IC. Not for moral reasons (though: I would have ethical problems selling bugs to attackers of any sort and am thankful I don't produce the kinds of bugs that have this market) so much as "it's not that hard to do this work and we should be getting more for our tax dollars".
I don't know if the NSA are the best in the world, but they're probably among the best-funded in the world.
Stuxnet is close to a work of art; the degree of effort involved in hiding self-reinstalling malware inside hard-drive firmware is staggering.[0] Anyone who's ever tried to write Linux drivers for undocumented hardware can appreciate how insane it is that they managed to do it for a dozen different hard drive manufacturers.
But yes, making it work for a bunch of manufacturers would be a lot of additional work (though easier if you can obtain the data sheets for the embedded microcontroller, which is probably doable if you're the NSA).
Side note: don't blame me for the URL, it was just the first place I found that had a PDF of the paper.
Americans are quite insular so not really knowledgeable about the "outside" world. Maybe because the country is so big and important, maybe because arrogance or intellectual laziness. I would never forget this fragment from a class on El Quijote at Yale.
"So what is spoken in the Bronx, Mexico City and Madrid? Castilian, which is Spanish, is what’s spoken, it’s spoken in those places. So you can erase that American prejudice from your mind if you ever had it. It is true — I mean, I had a colleague at Cornell, a very distinguished American medievalist who would ask me, “In what language do you write your scholarship, Roberto?” I said, “What do you mean? In English and Spanish.” He was worried that if I didn’t write in English being in Latin America, I didn’t have a language of culture in which I could write, because he didn’t think that in Latin America we had a language of culture. It is an American prejudice."
My organization was part of the author's book tour, we had a webinar with her earlier this year as she provided the book to the entire company. A couple of takeaways:
- Author believes (or is at least attempting to make others believe that she believes) that the NSA is benevolent and not committed to any specific domestic program
- Author frequently stated that 0-day purchasing platforms should not exist as it is a matter of opaque moral policy.
- Author frequently states that contractors like Immunity are problematic because they sell tradecraft to "bad nations" for profit.
- Worth noting that the author is also a NYT contributor and clings to a lot of those same notions.
The overall sentiment of the author after her webinar was that she was egotistically confined to believing in US supremacy and very much in the midst of an ideological war against competing beliefs. Probably not going to get far in her book either in light of that webinar.
This might be a cliché answer, but the (ISC)2 CISSP CBK Reference. Among cybersecurity certifications, the CISSP is notable for it's focus on cybersecurity policy laws & implementation frameworks.
I like Network Attacks and Exploitation: A Framework by Matthew Monte. It makes a clear and comprehensible case about the various asymmetries in cybersecurity and gives a detailed overview of failed past strategies, and strategies currently being tried.
> policy (Government, Politics & Diplomacy) a plan of action adopted or pursued by an individual, government, party, business, etc [1]
Are you genuinely interested in a book on policy? (If not, I might have some recommendations, depending on what you're interested in).
If so, you might need to adjust your expectation bar toward the lower spectrum. I honestly can't imagine a book about policy that isn't trash, but maybe that's just me.
Not exactly what you asked for but I always recommend The Cuckoo's Egg by Cliff Stoll whenever hacking comes up. And he deals with the early days of computer networks when there basically was no cybersecurity, policy or otherwise.
"This Is How They Tell Me the World Ends" tackles an important question: What causes the vulnerability of our modern world to "cyberattacks"?
What causes that vulnerability? Lack of liability for software vendors. If Microsoft had to pay the costs of vulnerabilities, we'd have far more secure systems.
There's one area of the industry where companies are held financially responsible for their mistakes - gambling systems. A few percent of revenue goes to paying for mistakes. For GTech, before they were acquired by a non-US company, you could see the numbers in their annual report. It's not a killer.
Since nobody knows how to reliably ship secure commercial software, liability will mostly have the effect of making it difficult to start new software businesses. Liability would make more sense to me if we could converge on a common understanding of a secure development process, but working on that has taken up most of my career and I don't think it's even on the horizon. The industry is still debating memory safety.
We have things like cybersecurity insurance that's required to be carried for vendors when working with the government, but much greater investment in both private and public sectors into secure coding and operational practices (namely by making commonly insecure things much more secure) would be helpful to at least some trends away from the current dominant mentality of "ship it first, ship it fast" dominating the software business.
I don't think it makes much sense for a random one-off script written by some lone developer starting a company to be subjected to all the alphabet soup of regulation, but I don't think letting everyone get away with security breaches forever is a good idea nor is just throwing up our hands and going "oh well, we're going to keep getting break-ins" for another 50 years.
It could quite possibly cost over a billion dollars to make a provably secure SAP/Salesforce, and it's not a one-time cost since SAP at least is usually customized.
"Liability would make more sense to me if we could converge on a common understanding of a secure development process, but working on that has taken up most of my career and I don't think it's even on the horizon."
That seems unlikely to happen as long as it is in commercial software businesses' best financial interests that it not happen.
That is only a problem if you create mandatory fixed-cost liability requirements. You can solve the problem by allowing companies to opt-in to liability requirements for some benefit. As an example, we could require that any company that wants to advertise security must do so in the form of a number specifying how much they will pay their customers in the event of a breach. So, they would not be able to advertise something like: "We have a secure cloud database.". They would instead be required to say: "We have a cloud database with $10M security." and in the event of a breach be forced to pay out $10M to its customers.
So, if a company is small and unable to afford any security, they can just set the number at $0 which means it is the customer's problem. However, if the company is big and people have security expectations, they would need to specify a reasonable number that would alleviate those concerns. As long as the number is clearly communicated and you are not allowed to fraudulently or misleadingly advertise a different number, then customer's would be able to make an informed decision with respect to the security level and liability they are accepting from their vendors.
Obviously there are some complexities with respect to properly communicating this information. For instance, for a consumer-facing company you would probably want a per-consumer number instead of an aggregate number. As an example, if Apple were to claim $300M for the iPhone, that might seem like a large number to an average person, but that would only amount to ~$1.50 per iPhone sold in a year. You also need to prevent misleading advertising that might attempt to divert from the quantitative liability they are actually accepting. However, the base scheme of mandatory labelling requirements along with opt-in liability should allow for a solution that is hard to bypass while being flexible enough to support both small and large businesses.
So, if a company is small and unable to afford any security, they can just set the number at $0 which means it is the customer's problem.
And if a company is large and low margin, they can do this too so nothing changes. A few companies will be out there promising things, like today, and maybe they can deliver and maybe you can sue them if they don't, like today.
Well, right now everyone and everything claims to be secure; this puts a number on that claim, and gives you companies explicitly claiming that they’re not secure (at least, they wouldn’t bet on it), which is quite a bit more information than you have today.
Being able to actually get your money back... is another problem. But obviously a toothless regulation is toothless, and changes nothing, so its not a useful assumption to make
If a company is large and low margin and does not care about security and they write in big bold print that they provide exactly zero guarantees about security and it is your problem if something happens and they never insinuate that there is any appreciable security, then I do not see the problem with that. The customer has been informed in no uncertain terms what they are getting and can make the cost-benefit analysis themselves. Obviously, there are complexities with regard to what exactly constitutes proper notice such that the customer can do a proper cost-benefit analysis, but the spirit of such a rule is clear.
This is would be different than the current state of affairs because lying or overstating your security story would have direct, pre-committed, quantitative downside. This is contrast to the outcomes of a lawsuit which are expensive, highly variable, generally require proof of intent, and depend on demonstrating actual quantitative damages.
The decision to take on risk for the purpose of marketing is not at all palatable to decision makers. Why would you want to accept an increase in risk for an unknown-unknown?
Liability may also be to third parties. Actually, I'm surprised that someone facing a DDOS attack from compromised Windows systems hasn't sued Microsoft for negligence. The victim is not a party to Microsoft's EULA in that case.
Attribution in security is really hard. But attribution in health supplements is far harder. Nobody's ever gotten a PoC for a health improvement to work every time in living organisms, even on the same organism version and platform. So there isn't a real option for supplement producers to make quantitative claims about their efficacy.
Meanwhile, quantitative, money-linked claims about software security actually do happen in rare edge cases, already.
What's the issue with requiring a formal spec and being liable for deviations from it? Though obviously this would have to be restricted to certain areas.
Because chances are the the formal spec itself won't be perfect. But even if it is you still have to worry about every layer below you in the software stack on top of the hardware. You might do everything perfectly at great cost and it all gets rendered moot by spectre/meltdown/rowhammer etc.
>Because chances are the the formal spec itself won't be perfect
It's not about creating 'perfect software', but about holding developers accountable. If a customer accepts a contract with a faulty spec, that's on them, not the developer.
>But even if it is you still have to worry about every layer below you in the software stack on top of the hardware. You might do everything perfectly at great cost and it all gets rendered moot by spectre/meltdown/rowhammer etc.
So what? It's not your fault. There's only one party that should be held liable for spectre, Intel.
Well, it also affects certain AMD and ARM processors as far as I know, I was just talking about a specific example. If software can be exploited because of a bug in hardware, I think it only makes sense to assign liability to the hardware company.
But how would you know this? All you know is that the 'software' was compromised and the data leaked. Do you think the initial blame will go to the right place or just the company after all? You won't always figure out how something was definitely compromised. You'll look around and definitely find bugs in the software though.
Well, if you have formally verified your system, you know that if the underlying systems work correctly, your work will correctly adhere to the spec. So if a breach occurs, there's two options - either the spec was bad or there's a hardware/OS/library bug.
Otherwise, you can't always know it, that doesn't strike me as a realistic goal in the first place. It's not about always being able to perfectly assign liability - obviously if you can't figure out how something was compromised, you simply can't assign any liability. But one could investigate and if you find deviations from the spec, you know where to correctly place blame.
>You'll look around and definitely find bugs in the software though.
If these are deviations from the spec, I think it would only be right to hold the developer liable. If a developer doesn't want this to occur, they could instead formally verify their software.
If the bugs are in the spec instead, the blame should lie with the party that accepted the spec.
> Since nobody knows how to reliably ship secure commercial software, liability will mostly have the effect of making it difficult to start new software businesses.
Do you literally mean "nobody" here? As in if I wanted to hire you to put a team together to reliably ship secure commercial software, you couldn't do it either?
I think we have, as an industry, tacitly agreed that it's better to ship cheap, insecure software than it is to ship expensive, secure software. A lot of the innovation in software has occurred specifically because it is so inexpensive.
It seems clear to me that, so far, this tradeoff has been a net benefit, but at some point we may want to trade the inexpensiveness and innovation for some security, particularly has we start to put software into more things that can kill and maim us.
> Do you literally mean "nobody" here? As in if I wanted to hire you to put a team together to reliably ship secure commercial software, you couldn't do it either?
It's fundamentally impossible given that you won't control everything about the operating environment of the software, and can't see into the future.
I mean, even if you could create a perfect piece of software with no bugs present at the time, that's still not enough, because vulnerabilities in dependent components will arise which were unknown at the time.
For example, if you'd written the perfect web browser a few years back, how would you prevent vulnerability to CPU-level attacks such as Spectre and Meltdown discovered later on?
Or consider if you write the perfect wireless networking firmware, but then end up being subject to vulnerabilities in the protocol standards (WEP was a good example of this)?
This is on top of software development being such a complex process that even the best of its practitioners write code strewn with bugs.
> It's fundamentally impossible given that you won't control everything about the operating environment of the software...
Why not? Given the cost of writing secure software, for all but the highest volumes, you can ship it as a dedicated app on its own hardware.
> For example, if you'd written the perfect web browser a few years back, how would you prevent vulnerability to CPU-level attacks such as Spectre and Meltdown discovered later on?
Given that we are talking about liability here. Presumably Spectre and Meltdown would be primarily liabilities for hardware vendors. Software shipped after the vendors recommend software mitigations would shift that, but I don't think
> Or consider if you write the perfect wireless networking firmware, but then end up being subject to vulnerabilities in the protocol standards (WEP was a good example of this)?
First we must put aside the fact that WEP raised red-flags for cryptographers far before an exploit was found, and the fact that those concerns were dismissed with a hand wavy "People can tap into Ethernet lines too, this is just 'Wired Equivalent Privacy'"
Now, I think that if you write software that faithfully implements a protocol, advertise it as implementing that protocol, and it is exploited not in-spite-of but because it did so, what exactly are you liable for?
Even knowing that WEP, WPA, WPA2 and WPA3 are all flawed protocols, I'm sure there will be a WPA4 some day that is also flawed. If the only thing we had to worry about in WiFi stacks were protocol flaws, attackers would have a much harder time of things. No flaw in the WEP protocol allowed gaining execution access on the WiFi adapter, much less the host computer (though given the bus designs of the time, the former inevitably led to the latter).
First, I'm a full-time, normal-mode software developer at Fly.io right now, so you can't hire me to do security stuff at the moment (Fly didn't bring me on board to do security stuff either! I spend my days writing BPF code and a bad Rust DNS server).
But rewind to last year, when I was still at Latacora. Ask me the question: can I hire you to make my development process more secure? Sure, I think I can add value there. Ask me instead: can I hire you to make my software liability-proof? Not a chance.
Take 9 strong software security people and divide them into 3 teams. Give each team a stretch of time to find vulnerabilities. There will be a ton of overlap in the findings, but each team will find different things. One team will happen to have a specialty in the low-level details of the platform it's running on; another will spend 80% of their cycles thinking about SSRF and HTTP desync vulnerabilities; still another will happen to have a reliable process for exhaustively testing authorization. Don't get me started on K8s or Electron or cryptography or iOS or browser extensions or Node.js.
And that's all for pretty standard app stack stuff. None of those problems are the subtext for Perlroth's book; the vulnerabilities she's trying to write about are in low-level C/C++ code. The biggest security teams in the world have build enormous, carefully-instrumented fuzzing farms for this stuff and they don't find everything.
I literally mean nobody.
I think you could stand a chance if you designed your product to fit the limitations of what we can deliver reliably. You'd lose most of your features. You'd ship boring CRUD stuff exclusively on your own dedicated hardware, and you'd build it with minimal library support. It would be excruciating, but people do it. Just not for any of the kinds of software we normally interact with.
> I think you could stand a chance if you designed your product to fit the limitations of what we can deliver reliably. You'd lose most of your features. You'd ship boring CRUD stuff exclusively on your own dedicated hardware, and you'd build it with minimal library support. It would be excruciating, but people do it. Just not for any of the kinds of software we normally interact with.
I think we are in agreement then. If you are targeting K8S or Electron, you've already lost the battle against many threat-models that matter for liability. I think what you are pushing back against is people imagine bug-free software and they think "oh maybe 4x more expensive and it will look like it's from 2005" while the reality is more like table-stakes of 10x more expensive and it will look like it's from 1980.
I personally think that the tradeoffs are worth it for more industries than currently make it, but I don't think the law can be nuanced enough to both be effective and not cause massive collateral damage to many other industries. Look at the chilling effect regulation has had on software in the medical industry, and yet those regulations seem to have done little to increase the resources required for an attacker who wishes to exfiltrate patient data.
I think the end of your second paragraph is a good way to put it. I think you could get to 1998 securely, for what it's worth, but your expense number seems dead-on.
Nate Lawson used to tell me (I assume he'd still tell me) that companies trying to ship secure software need to budget 10x for verification what they do for implementation. The best companies in the world don't even budget 1x. I'm not sure it's reasonable to ask them to. I think we'll know stuff in 10 years that will change how we look at things.
What do you mean by secure? Actually impossible? An attacker needs more than $100B to breach it? $1B? $10M?
In my opinion, if Jeff Bezos, Satya Nadella, Sundar Pichai, and Tim Cook all got together and wanted to put together a team that ship secure commercial software, they would probably be unable to put together a team that knew how to get to the $10M level, and there is zero chance at the $1B level. If they all worked together and put their full focus and resources on the problem, they might be able to put together a team that could invent how to get to the $10M level, but they would not be able to put together a team that already knew how and had experience doing so. The key difference between these two cases being that in the former the team would need to invent good practices and experiment to see if they worked, where as the latter only needs to apply already existing known good methodology.
If you want some 3rd party evidence for my claims, not that I am claiming this is exhaustive proof, bug bounties that get claimed are, generally speaking, less than the cost of discovery for a bug otherwise security researchers would not put in the effort to discover than claim them. Not a single one of their premier products that they ship and sell [1][2][3] offers a bug bounty over $1M for total and complete remote zero-interaction compromise. It is very unlikely that they can do an order of magnitude better than the premier products just because they want to.
No one knows how to make a perfectly safe car, yet there are some car safety problems for which car manufacturers are liable and others for which they are not. And those things can change over time, particularly as we learn about new manufacturing techniques and features which can improve safety!
I see no reason why the same can't be applied to software. It's probably true that we can't feasibly create software and computer systems that are completely free from security vulnerabilities, but we absolutely have discovered some vulnerabilities which can be protected against using certain software development techniques, and I see no reason why software vendors ought not be liable for damages caused by their failure to follow these established techniques.
Just like with cars, there is always going to be some tradeoffs between cost and security/safety, and exactly where to draw the line is something reasonable people can debate.
This sort of analogy, just like its popular variants (e.g. 'how come we can build bridges that don't collapse but windows crashes', etc) falls apart under very superficial scrutiny. For one thing, the failure semantics and modes of physical things are vastly simpler. It's not really a fruitful approach to this discussion.
The point of an analogy is to point out that certain aspects of two things are comparable in a certain context. It's true that there are significant differences between cars and software, like, say, the presence of wheels, but those aren't relevant to the specific point I was making. What is it exactly that makes the development of regulation of car safety so different from the development and (potential) regulation of software security?
It's not the presence of wheels, it's the difficulty of even defining what software operating 'correctly' means. This is a fairly well-understood aspect of software in general, not just the security part - hence everything from the important (e.g. halting problem) to the silly ('software will eat the world'). Software represents models we make up and on top of that, ones we interpret. Cars are not like that. You don't say your car made a mistake when it refuses to start one cold morning. There's essentially universal agreement on the 'semantics' of cars and bridges.
The notion of software operating correctly didn't really come up. We're not talking about regulating the reliability or effectiveness of software, but rather the security vulnerabilities, and I suspect the regulations would treat the seriousness of different vulnerabilities by the actual harm they can do, just like flaws in the manufacturing of cars. And sure, there are still cases where one could argue about whether a certain attribute of a certain piece of software constitutes a security vulnerability or not, but that's okay. Laws aren't mathematical proofs. There is undoubtedly some ambiguity and room for reasonable debate in the specifics of car safety regulations as well.
> You don't say your car made a mistake when it refuses to start one cold morning. There's essentially universal agreement on the 'semantics' of cars and bridges.
Since you mentioned it...I don't know whether this is the case, but I would guess that there might be regulations about the temperatures at which car batteries and other components must be capable of operating at. And, as car automation software becomes more advanced and prominent, I think we'll be discovering that there is not in fact universal agreement on the "semantics" of cars. There is probably already considerable overlap in the subjects of my analogy, for recent self-driving capabilities but also for much older standard things like airbags and anti-lock braking systems.
I'm not sure how the security is different than the correctness. Security is just an aspect of it - the software doesn't do what it's supposed to, in some specific way.
But my broader point was that this analogy is basically what 'can god make a rock so big they can't pick it up' is to a discussion of theology.
"No one knows how to make a perfectly safe car, yet there are some car safety problems for which car manufacturers are liable and others for which they are not."
The situation you are describing is that automobile safety is implemented primarily by establishing standards and only secondarily by the existence of liability (though it's important).
Now, if you had a large company that bought cars and modified them, that company would itself have an obligation to do it safely too. And having regulations about that would be an absolutely necessary part of liability and safety.
...we absolutely have discovered some vulnerabilities which can be protected against using certain software development techniques, and I see no reason why software vendors ought not be liable for damages caused by their failure to follow these established techniques.
I would note that the existence of zero-days is absolutely not the most pressing problem of software security. Having a reasonable system of connecting X to Y is far more pressing. You can get in as much trouble with a misconfigured system as with a vulnerable system. But broadly, your point that there should be standards, is good.
> The situation you are describing is that automobile safety is implemented primarily by establishing standards and only secondarily by the existence of liability (though it's important).
Indeed, regulatory standards is what I was hinting at, not just civil liability (although I believe both exist for car manufacturers, and should probably exist for software vendors).
Yes, we desperately need broad but appropriate regulatory standards. Especially, security standards and research should focus on securing infrastructure and infrastructure type resources. The hacked water supply article from just a few days ago was shocking even to cynical me.
Things like institutes that study this situation outside the government itself may be necessary to establish trust at this point.
And you can't make the standards primarily about software vendors.
> Since nobody knows how to reliably ship secure commercial software
Secure software lies on a spectrum. We know memory unsafe languages are dangerous yet critical software is still being written in them. Liability would drastically change the domain in terms of discarding practices we've known for decades are deadends, and raise the bar significantly.
>Since nobody knows how to reliably ship secure commercial software, liability will mostly have the effect of making it difficult to start new software businesses.
Doesn't sound so bad of a tradeoff, enforced for certain classes of software. Like one expects liability for medical, banking, flight control, etc. software, it could be extended to OSes, server programs, and anything handling user accounts...
Software with less impact if it's not secure, like e.g. a media player or a text editor, could continue to ship free from liability.
> Since nobody knows how to reliably ship secure commercial software, liability will mostly have the effect of making it difficult to start new software businesses.
I think you would agree that having critical string parsing logic written in C and shipped in an opaque binary is both negligent and more dangerous than a reasonable person would expect. Traditional products liability claims focus on those kinds of questions: was there negligent conduct? is the product more dangerous than a reasonable person would expect?
Attaching some kind of liability is a good way to encourage the industry to settle on standards of non-negligent conduct. If you provide vendors an opt-out mechanism (e.g. disclose your source and there's no liability), it will not dramatically interfere with the cadence of development.
Common law is very well suited to establishing standards in a dynamic environment. It got us through the first industrial revolution.
If MSFT had to pay the cost of vulnerabilities then Windows would cost a lot more. Also, who would pay for Linux vulnerabilities? Would anyone contribute to open source if they were liable for any damage caused by their contribution?
That’s a good point about Linux. The GPL boilerplate already includes:
> This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
The author including that line is saying “despite what you think this software may do (creating implied warranties and the like), it doesn’t matter. I can’t be held liable for your usage of this code/software.” Would forced liability mean open source licenses like the GPL wouldn’t be able to have those sentences mean anything?
You need a way to generally enforce an appropriate effort to security. This seems extremely important. I don't see liability, after-the-fact-punishment, only with software vendors, being the way to achieve this.
Numerous threads on HN lately have discussed the way that "connect everything to everything" approaches wind-up inherently insecure and a wide range of organizations have no incentive to stop them. Water systems have no need to connect to the Internet even indirectly and neither automobile remote start system but organization managing these things have no incentive to take this stuff seriously.
This is what's called an "externality" and a difficult one. Externalities are generally best dealt with be direct regulation but security is a bit different externalities because defining best practicing isn't. I could imagine a "security institute", only focused on defense and independent enough to be trustworthy to most institutions. But I'm not all that optimistic such a thing could be created.
They already do this to a certain extent. Sometimes it's after the vulnerability has already been exploited by them or a sister agency. But other times it's soon after discovery.
At one time the NSA did that. The way it worked at one point in the 1980s is that a vendor could submit an OS for evaluation. NSA employees would test it, and if they could break in, they'd tell the vendor how they got in. The vendor got a chance to resubmit the system. One chance. If it failed on round 2, it was rejected.
Vendors hated it. I've seen a vendor group document asking NSA not to publicly certify systems at the highest levels because it make them look bad. They got DoD to go for a system where the testing was outsourced to private labs, and you could keep going back for a retest over and over again until you "passed".
Amusingly, computer security in those days wasn't a high priority at NSA for a bureaucratic reason. Computer security was at FANX, not NSA HQ at Fort Meade. FANX is the "Friendship Annex", a few miles from NSA HQ, near what used to be called Friendship Airport and is now BWI. FANX had all the minor functions of NSA - personnel, procurement, training, etc., and was considered a dumping ground for losers. Putting computer security out there doomed it.
What really killed security is that from 1940 to 1980, DoD was the major consumer of advanced computer and electronics technology. By 1990, the commercial sector was dominant.
>"If Microsoft had to pay the costs of vulnerabilities"
Yeah, ok but lets trade out MSFT for FOSS, Firefox or small iOS app developers.
At some point, we should ask to see if we are inadvertently advocating for scaring people into doing the right thing. Worse yet, for something that they couldn't see or anticipate.
What about the standards makers? Are they going to be held accountable too? When is the last time you've read OWASP articles? Have you seen the state of that sort of literature? Its a mess. Have you seen how arcane and arbitrary the NIST-800 series is? What is the limit to this sort of "accountability"?
At the risk of being even more trite, ideologically motivated statements like this look cool only because its on the internet.
72 comments
[ 3.0 ms ] story [ 84.9 ms ] threadIf it doesn't get better than that pretty quickly I'm unlikely to finish it.
No one knows who the Shadow Brokers are, unless that’s covered later in the book.
https://threadreaderapp.com/thread/1224880979258441729.html
In reality I think it's kind of an embarrassment how much CNE technology goes the other way, from industry and research to the IC. Not for moral reasons (though: I would have ethical problems selling bugs to attackers of any sort and am thankful I don't produce the kinds of bugs that have this market) so much as "it's not that hard to do this work and we should be getting more for our tax dollars".
Stuxnet is close to a work of art; the degree of effort involved in hiding self-reinstalling malware inside hard-drive firmware is staggering.[0] Anyone who's ever tried to write Linux drivers for undocumented hardware can appreciate how insane it is that they managed to do it for a dozen different hard drive manufacturers.
[0] https://www.theverge.com/2015/2/16/8048243/nsa-hard-drive-fi...
But yes, making it work for a bunch of manufacturers would be a lot of additional work (though easier if you can obtain the data sheets for the embedded microcontroller, which is probably doable if you're the NSA).
Side note: don't blame me for the URL, it was just the first place I found that had a PDF of the paper.
"So what is spoken in the Bronx, Mexico City and Madrid? Castilian, which is Spanish, is what’s spoken, it’s spoken in those places. So you can erase that American prejudice from your mind if you ever had it. It is true — I mean, I had a colleague at Cornell, a very distinguished American medievalist who would ask me, “In what language do you write your scholarship, Roberto?” I said, “What do you mean? In English and Spanish.” He was worried that if I didn’t write in English being in Latin America, I didn’t have a language of culture in which I could write, because he didn’t think that in Latin America we had a language of culture. It is an American prejudice."
https://oyc.yale.edu/spanish-and-portuguese/span-300/lecture...
- Author believes (or is at least attempting to make others believe that she believes) that the NSA is benevolent and not committed to any specific domestic program
- Author frequently stated that 0-day purchasing platforms should not exist as it is a matter of opaque moral policy.
- Author frequently states that contractors like Immunity are problematic because they sell tradecraft to "bad nations" for profit.
- Worth noting that the author is also a NYT contributor and clings to a lot of those same notions.
The overall sentiment of the author after her webinar was that she was egotistically confined to believing in US supremacy and very much in the midst of an ideological war against competing beliefs. Probably not going to get far in her book either in light of that webinar.
Are you genuinely interested in a book on policy? (If not, I might have some recommendations, depending on what you're interested in).
If so, you might need to adjust your expectation bar toward the lower spectrum. I honestly can't imagine a book about policy that isn't trash, but maybe that's just me.
[1]: freedictionary.com
What causes that vulnerability? Lack of liability for software vendors. If Microsoft had to pay the costs of vulnerabilities, we'd have far more secure systems.
There's one area of the industry where companies are held financially responsible for their mistakes - gambling systems. A few percent of revenue goes to paying for mistakes. For GTech, before they were acquired by a non-US company, you could see the numbers in their annual report. It's not a killer.
I don't think it makes much sense for a random one-off script written by some lone developer starting a company to be subjected to all the alphabet soup of regulation, but I don't think letting everyone get away with security breaches forever is a good idea nor is just throwing up our hands and going "oh well, we're going to keep getting break-ins" for another 50 years.
That seems unlikely to happen as long as it is in commercial software businesses' best financial interests that it not happen.
So, if a company is small and unable to afford any security, they can just set the number at $0 which means it is the customer's problem. However, if the company is big and people have security expectations, they would need to specify a reasonable number that would alleviate those concerns. As long as the number is clearly communicated and you are not allowed to fraudulently or misleadingly advertise a different number, then customer's would be able to make an informed decision with respect to the security level and liability they are accepting from their vendors.
Obviously there are some complexities with respect to properly communicating this information. For instance, for a consumer-facing company you would probably want a per-consumer number instead of an aggregate number. As an example, if Apple were to claim $300M for the iPhone, that might seem like a large number to an average person, but that would only amount to ~$1.50 per iPhone sold in a year. You also need to prevent misleading advertising that might attempt to divert from the quantitative liability they are actually accepting. However, the base scheme of mandatory labelling requirements along with opt-in liability should allow for a solution that is hard to bypass while being flexible enough to support both small and large businesses.
And if a company is large and low margin, they can do this too so nothing changes. A few companies will be out there promising things, like today, and maybe they can deliver and maybe you can sue them if they don't, like today.
Being able to actually get your money back... is another problem. But obviously a toothless regulation is toothless, and changes nothing, so its not a useful assumption to make
This is would be different than the current state of affairs because lying or overstating your security story would have direct, pre-committed, quantitative downside. This is contrast to the outcomes of a lawsuit which are expensive, highly variable, generally require proof of intent, and depend on demonstrating actual quantitative damages.
The decision to take on risk for the purpose of marketing is not at all palatable to decision makers. Why would you want to accept an increase in risk for an unknown-unknown?
Meanwhile, quantitative, money-linked claims about software security actually do happen in rare edge cases, already.
It's not about creating 'perfect software', but about holding developers accountable. If a customer accepts a contract with a faulty spec, that's on them, not the developer.
>But even if it is you still have to worry about every layer below you in the software stack on top of the hardware. You might do everything perfectly at great cost and it all gets rendered moot by spectre/meltdown/rowhammer etc.
So what? It's not your fault. There's only one party that should be held liable for spectre, Intel.
Well, if you have formally verified your system, you know that if the underlying systems work correctly, your work will correctly adhere to the spec. So if a breach occurs, there's two options - either the spec was bad or there's a hardware/OS/library bug.
Otherwise, you can't always know it, that doesn't strike me as a realistic goal in the first place. It's not about always being able to perfectly assign liability - obviously if you can't figure out how something was compromised, you simply can't assign any liability. But one could investigate and if you find deviations from the spec, you know where to correctly place blame.
>You'll look around and definitely find bugs in the software though.
If these are deviations from the spec, I think it would only be right to hold the developer liable. If a developer doesn't want this to occur, they could instead formally verify their software.
If the bugs are in the spec instead, the blame should lie with the party that accepted the spec.
Do you literally mean "nobody" here? As in if I wanted to hire you to put a team together to reliably ship secure commercial software, you couldn't do it either?
I think we have, as an industry, tacitly agreed that it's better to ship cheap, insecure software than it is to ship expensive, secure software. A lot of the innovation in software has occurred specifically because it is so inexpensive.
It seems clear to me that, so far, this tradeoff has been a net benefit, but at some point we may want to trade the inexpensiveness and innovation for some security, particularly has we start to put software into more things that can kill and maim us.
It's fundamentally impossible given that you won't control everything about the operating environment of the software, and can't see into the future.
I mean, even if you could create a perfect piece of software with no bugs present at the time, that's still not enough, because vulnerabilities in dependent components will arise which were unknown at the time.
For example, if you'd written the perfect web browser a few years back, how would you prevent vulnerability to CPU-level attacks such as Spectre and Meltdown discovered later on?
Or consider if you write the perfect wireless networking firmware, but then end up being subject to vulnerabilities in the protocol standards (WEP was a good example of this)?
This is on top of software development being such a complex process that even the best of its practitioners write code strewn with bugs.
Why not? Given the cost of writing secure software, for all but the highest volumes, you can ship it as a dedicated app on its own hardware.
> For example, if you'd written the perfect web browser a few years back, how would you prevent vulnerability to CPU-level attacks such as Spectre and Meltdown discovered later on?
Given that we are talking about liability here. Presumably Spectre and Meltdown would be primarily liabilities for hardware vendors. Software shipped after the vendors recommend software mitigations would shift that, but I don't think
> Or consider if you write the perfect wireless networking firmware, but then end up being subject to vulnerabilities in the protocol standards (WEP was a good example of this)?
First we must put aside the fact that WEP raised red-flags for cryptographers far before an exploit was found, and the fact that those concerns were dismissed with a hand wavy "People can tap into Ethernet lines too, this is just 'Wired Equivalent Privacy'"
Now, I think that if you write software that faithfully implements a protocol, advertise it as implementing that protocol, and it is exploited not in-spite-of but because it did so, what exactly are you liable for?
Even knowing that WEP, WPA, WPA2 and WPA3 are all flawed protocols, I'm sure there will be a WPA4 some day that is also flawed. If the only thing we had to worry about in WiFi stacks were protocol flaws, attackers would have a much harder time of things. No flaw in the WEP protocol allowed gaining execution access on the WiFi adapter, much less the host computer (though given the bus designs of the time, the former inevitably led to the latter).
But rewind to last year, when I was still at Latacora. Ask me the question: can I hire you to make my development process more secure? Sure, I think I can add value there. Ask me instead: can I hire you to make my software liability-proof? Not a chance.
Take 9 strong software security people and divide them into 3 teams. Give each team a stretch of time to find vulnerabilities. There will be a ton of overlap in the findings, but each team will find different things. One team will happen to have a specialty in the low-level details of the platform it's running on; another will spend 80% of their cycles thinking about SSRF and HTTP desync vulnerabilities; still another will happen to have a reliable process for exhaustively testing authorization. Don't get me started on K8s or Electron or cryptography or iOS or browser extensions or Node.js.
And that's all for pretty standard app stack stuff. None of those problems are the subtext for Perlroth's book; the vulnerabilities she's trying to write about are in low-level C/C++ code. The biggest security teams in the world have build enormous, carefully-instrumented fuzzing farms for this stuff and they don't find everything.
I literally mean nobody.
I think you could stand a chance if you designed your product to fit the limitations of what we can deliver reliably. You'd lose most of your features. You'd ship boring CRUD stuff exclusively on your own dedicated hardware, and you'd build it with minimal library support. It would be excruciating, but people do it. Just not for any of the kinds of software we normally interact with.
> I think you could stand a chance if you designed your product to fit the limitations of what we can deliver reliably. You'd lose most of your features. You'd ship boring CRUD stuff exclusively on your own dedicated hardware, and you'd build it with minimal library support. It would be excruciating, but people do it. Just not for any of the kinds of software we normally interact with.
I think we are in agreement then. If you are targeting K8S or Electron, you've already lost the battle against many threat-models that matter for liability. I think what you are pushing back against is people imagine bug-free software and they think "oh maybe 4x more expensive and it will look like it's from 2005" while the reality is more like table-stakes of 10x more expensive and it will look like it's from 1980.
I personally think that the tradeoffs are worth it for more industries than currently make it, but I don't think the law can be nuanced enough to both be effective and not cause massive collateral damage to many other industries. Look at the chilling effect regulation has had on software in the medical industry, and yet those regulations seem to have done little to increase the resources required for an attacker who wishes to exfiltrate patient data.
Nate Lawson used to tell me (I assume he'd still tell me) that companies trying to ship secure software need to budget 10x for verification what they do for implementation. The best companies in the world don't even budget 1x. I'm not sure it's reasonable to ask them to. I think we'll know stuff in 10 years that will change how we look at things.
In my opinion, if Jeff Bezos, Satya Nadella, Sundar Pichai, and Tim Cook all got together and wanted to put together a team that ship secure commercial software, they would probably be unable to put together a team that knew how to get to the $10M level, and there is zero chance at the $1B level. If they all worked together and put their full focus and resources on the problem, they might be able to put together a team that could invent how to get to the $10M level, but they would not be able to put together a team that already knew how and had experience doing so. The key difference between these two cases being that in the former the team would need to invent good practices and experiment to see if they worked, where as the latter only needs to apply already existing known good methodology.
If you want some 3rd party evidence for my claims, not that I am claiming this is exhaustive proof, bug bounties that get claimed are, generally speaking, less than the cost of discovery for a bug otherwise security researchers would not put in the effort to discover than claim them. Not a single one of their premier products that they ship and sell [1][2][3] offers a bug bounty over $1M for total and complete remote zero-interaction compromise. It is very unlikely that they can do an order of magnitude better than the premier products just because they want to.
[1] https://www.microsoft.com/en-us/msrc/bounty
[2] https://www.google.com/about/appsecurity/android-rewards/
[3] https://developer.apple.com/security-bounty/
I see no reason why the same can't be applied to software. It's probably true that we can't feasibly create software and computer systems that are completely free from security vulnerabilities, but we absolutely have discovered some vulnerabilities which can be protected against using certain software development techniques, and I see no reason why software vendors ought not be liable for damages caused by their failure to follow these established techniques.
Just like with cars, there is always going to be some tradeoffs between cost and security/safety, and exactly where to draw the line is something reasonable people can debate.
> You don't say your car made a mistake when it refuses to start one cold morning. There's essentially universal agreement on the 'semantics' of cars and bridges.
Since you mentioned it...I don't know whether this is the case, but I would guess that there might be regulations about the temperatures at which car batteries and other components must be capable of operating at. And, as car automation software becomes more advanced and prominent, I think we'll be discovering that there is not in fact universal agreement on the "semantics" of cars. There is probably already considerable overlap in the subjects of my analogy, for recent self-driving capabilities but also for much older standard things like airbags and anti-lock braking systems.
But my broader point was that this analogy is basically what 'can god make a rock so big they can't pick it up' is to a discussion of theology.
The situation you are describing is that automobile safety is implemented primarily by establishing standards and only secondarily by the existence of liability (though it's important).
Now, if you had a large company that bought cars and modified them, that company would itself have an obligation to do it safely too. And having regulations about that would be an absolutely necessary part of liability and safety.
...we absolutely have discovered some vulnerabilities which can be protected against using certain software development techniques, and I see no reason why software vendors ought not be liable for damages caused by their failure to follow these established techniques.
I would note that the existence of zero-days is absolutely not the most pressing problem of software security. Having a reasonable system of connecting X to Y is far more pressing. You can get in as much trouble with a misconfigured system as with a vulnerable system. But broadly, your point that there should be standards, is good.
Indeed, regulatory standards is what I was hinting at, not just civil liability (although I believe both exist for car manufacturers, and should probably exist for software vendors).
Things like institutes that study this situation outside the government itself may be necessary to establish trust at this point.
And you can't make the standards primarily about software vendors.
Secure software lies on a spectrum. We know memory unsafe languages are dangerous yet critical software is still being written in them. Liability would drastically change the domain in terms of discarding practices we've known for decades are deadends, and raise the bar significantly.
Doesn't sound so bad of a tradeoff, enforced for certain classes of software. Like one expects liability for medical, banking, flight control, etc. software, it could be extended to OSes, server programs, and anything handling user accounts...
Software with less impact if it's not secure, like e.g. a media player or a text editor, could continue to ship free from liability.
I think you would agree that having critical string parsing logic written in C and shipped in an opaque binary is both negligent and more dangerous than a reasonable person would expect. Traditional products liability claims focus on those kinds of questions: was there negligent conduct? is the product more dangerous than a reasonable person would expect?
Attaching some kind of liability is a good way to encourage the industry to settle on standards of non-negligent conduct. If you provide vendors an opt-out mechanism (e.g. disclose your source and there's no liability), it will not dramatically interfere with the cadence of development.
Common law is very well suited to establishing standards in a dynamic environment. It got us through the first industrial revolution.
> This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
The author including that line is saying “despite what you think this software may do (creating implied warranties and the like), it doesn’t matter. I can’t be held liable for your usage of this code/software.” Would forced liability mean open source licenses like the GPL wouldn’t be able to have those sentences mean anything?
Numerous threads on HN lately have discussed the way that "connect everything to everything" approaches wind-up inherently insecure and a wide range of organizations have no incentive to stop them. Water systems have no need to connect to the Internet even indirectly and neither automobile remote start system but organization managing these things have no incentive to take this stuff seriously.
This is what's called an "externality" and a difficult one. Externalities are generally best dealt with be direct regulation but security is a bit different externalities because defining best practicing isn't. I could imagine a "security institute", only focused on defense and independent enough to be trustworthy to most institutions. But I'm not all that optimistic such a thing could be created.
Vendors hated it. I've seen a vendor group document asking NSA not to publicly certify systems at the highest levels because it make them look bad. They got DoD to go for a system where the testing was outsourced to private labs, and you could keep going back for a retest over and over again until you "passed".
Amusingly, computer security in those days wasn't a high priority at NSA for a bureaucratic reason. Computer security was at FANX, not NSA HQ at Fort Meade. FANX is the "Friendship Annex", a few miles from NSA HQ, near what used to be called Friendship Airport and is now BWI. FANX had all the minor functions of NSA - personnel, procurement, training, etc., and was considered a dumping ground for losers. Putting computer security out there doomed it.
What really killed security is that from 1940 to 1980, DoD was the major consumer of advanced computer and electronics technology. By 1990, the commercial sector was dominant.
Here are the early security standards books.[1]
[1] https://i.ibb.co/CJTTzDL/ivorytowerspeedtest.png
Yeah, ok but lets trade out MSFT for FOSS, Firefox or small iOS app developers.
At some point, we should ask to see if we are inadvertently advocating for scaring people into doing the right thing. Worse yet, for something that they couldn't see or anticipate.
What about the standards makers? Are they going to be held accountable too? When is the last time you've read OWASP articles? Have you seen the state of that sort of literature? Its a mess. Have you seen how arcane and arbitrary the NIST-800 series is? What is the limit to this sort of "accountability"?
At the risk of being even more trite, ideologically motivated statements like this look cool only because its on the internet.