I really hope Apple will disclose its findings on this matter, in name of the transparency they're advocating.
Imagine if Apple itself states something along the lines of "we found out that the line of code responsible for the bug was planted on purpose".
Should they make such a statement? Better: COULD they make it?
I'd love to see it, but I seriously doubt it. Apple is secretive to an incredibly annoying degree.
Has Apple even publicly discussed the basic details of this bug? As far as I've seen, every single bit of analysis has come from outside the company, based on simply viewing the source code. Apple acknowledged that there is a vulnerability, but they haven't even said what it is in any detail.
I'd like to believe Apple simply as no code review but it's odd they removed a specific check -- was it not suppose to be there? Does anyone know if the patch added the line back in or if it removed the duplicate goto?
The one thing that points to it not being a backdoor is I doubt Apple would open source the code. Surely they'd maintain a separate branch or something?
They could have made it open source because they knew they were under pressure to backdoor it, so wanted people to see what happened without explicitly telling the world... Or not. It's a fun theory though.
You think a billion dollar, cash-rich company known for software excellence would understand testing and hardening? A company known for draconic quality assurance, would skimp? Silly me. HN must be right.
As noted in the article, plausible deniability is a key criterion for thinking of this as a possible NSA insertion.
In other words, it's only possible that it's an NSA job if it's also possible that it isn't. Something tells me therefore that we're unlikely ever to know for sure.
Don't scare the geeks of HN. They have well-tuned, waffle-brains with syrup-tight barriers. Suggesting there is a world with blurry boundaries scares them.
Doesn't it strain credulity that the NSA would (bother to) insert a flaw that would be detected by many commercially available analysis tools (or the right compiler flags)?
(I certainly shouldn't pretend to know much about C, but the above matches my understanding of the situation)
Often when things like this happen there's a large conspiracy at play in some peoples minds, "Apple deliberately left a security vulnerability". But it falls apart pretty quickly, it's in an open sourced package, so the assumption is someone is eventually going to see it, so it's not going to remain a secret and thus is useless as a stealth backdoor.
The likelihood is pretty simple, someone fucked up. On a potentially huge level, but a fuck up none the less. These things do unfortunately happen, and no doubt it'll prompt an internal review of their change management process, and their build chain, and what they can do to isolate issues like this in the future.
While I believe that it was just a screw-up that wasn't caught because of gaps of process (the code essentially repeats blocks with minor changes, so I can envision someone doing some refactoring -- removing the passing of the context -- and trying to save time by copying the first changed block to the second, failing to overwrite that single additional line), considering the possibility that it was intentional in no universe suggests that it was Apple the organization that chose and instituted the vulnerability.
A single employee receiving a second paycheck from a three-letter organization could have been responsible. A vulnerability could have been used to plant it. And so on. Remember when everyone was railing about Google giving NSA a backdoor (cue a thousand "don't be evil" cries) when in reality the NSA and friends were simply exploiting a weakness of Google's network. The net effect was the same.
It could be intentional and still entirely unintentional as far as Apple the company is concerned.
And to the open source thing -- it sat there for 18 months. Indeed, it was caught [EDIT: Actually I don't know how it was caught. I faintly recall someone posting an issue someone submitted to Apple detailing some weird behavior with invalid private keys, but can't find it now]
"And to the open source thing -- it sat there for 18 months." !!! Just .. wow. If it was the first time we heard such stories. But it happends many time with the linux kernel source code and all. I definitively conclude that source code begin open to everyone is not a mark of higher quality.
"It could be intentional and still entirely unintentional as far as Apple the company is concerned."
Agreed, as far as Apple know it was a bug, but one of their staff could have been turned. But they'll be able to audit the change logs, and I imagine a company famously known for their OTT internal security would be preparing the car batteries and nipple clamps if they thought they'd found a bad actor.
> it's in an open sourced package, so the assumption is someone is eventually going to see it, so it's not going to remain a secret and thus is useless as a stealth backdoor.
The fact that people can see the nature of the bug in the source code for themselves just makes the case that it was a deliberate backdoor even more deniable which, as Schneier is pointing out, is a very desirable property.
You're also assuming that a backdoor you can use indefinitely is desirable.
As an added side-benefit, any other proprietary or FOSS products that adopted this code are now vulnerable.
Sure, it makes it even more deniable, but on the balance between 'well thought out conspiracy with I know-you know-I know you know I know' logic, and a fuck up, I'd err towards it being a fuck up.
You're just casually brushing this off with Hanlon's Razor. You shouldn't discount the fact that as technical folk and programmers here on HN we're probably biased[0]. Our ability to recognise just how easily this flaw could have been both introduced and detected can both be used both against, and in defence, of the conspiracy theory.
I've been a proponent of Hanlon's Razor a number of times, so guilty as charged.
As technically minded people we intrinsically understand that it could be the work of someone malicious, but we should also understand from personal experiences that fuck ups happen significantly more often than brown envelopes full of cash.
The thing that bugs me is so many people are saying there is no way that could be a merge/editing error so it had to have been deliberate bad actor, but all they are comparing is a diff between the sources that match two released versions and using that as evidence that it was not very plausible that it was such en error. What I think happened is very plausible though:
if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
goto fail;
Some other check was added:
if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;
+ if ((err = SSLHashSHA1.foobar(&hashCtx, &signedParams)) != 0)
+ goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
goto fail;
Then that check was merged into another check, otherwise refactored, or decided it was not needed there anymore - but mistakenly only the added if ... line was removed:
if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;
- if ((err = SSLHashSHA1.foobar(&hashCtx, &signedParams)) != 0)
goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
goto fail;
And that's how that ended-up the way that it did.
if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;
goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
goto fail;
Then of course it was not the only change submitted, it must have been a pretty sizable diff of many files, and it slipped-by in all that.
The scenario you've described would require multiple changes to the same function, but the file diffs making their around the blogosphere show only cosmetic changes to that file and none of those affect the "goto fail" function.
Do you see how there was some simple refactoring going on there in the structure and how the entire in-kernel functionality was added? I could totally see debugging function being created and sprinkled in all the places after all the update()s were called while verifying kernel stuff was working then removed and in this one place one line not enough deleted.
edit: Ha, I had my own copy paste error after getting bad gateway error, I fixed the duplicated code, sorry.
That's a maxim I apply in general, but given that we know there are, in fact, known malicious actors purposely introducing flaws to security software, I think it might be a mistake to reflexively apply it here.
Hanlon's razor is just the worst.
Systematic incompetence is malice.
Here's my alternative but equally reasonable platitude:
"Never attribute to incompetence what could be attributed to malice with plausible deniability."
Another sad indicator of the level Schneier is playing at today, in the same vein as "avoid elliptic curves, we don't trust the math".
Once again: the only reason this bug got so much attention and press is that it's easy for laypeople to get their heads around. All you have to understand is how "goto" works. The bug is vivid, and so (paradoxically) seems scarier.
Significantly worse bugs are found every week. Within a few days of the announcement of this TLS bug, a Flash bug was announced, after being detected in exploits in the wild, that enabled reliable drive-by hijackings of browsers --- multiple browsers. It was off the HN front page within an hour.
TLS bugs aren't even unusual. We get a new one every few years ago. Firefox managed a PKCS1v15 parsing bug that allowed anyone with a Python script and 30 milliseconds to generate a certificate for any domain. Other browsers have screwed up certificate chaining, so that any domain could sign any other domain. But nobody understands PKCS1v15 padding, nobody understands certificate chaining, and so nobody writes stories about these bugs. But their impact is identical to this one.
Whatever may or may not be wrong with Schneier's article, it's certainly a hell of a lot better than (say) conflating a vulnerability that was in released, stable versions of iOS and OS X for over a year with one that never made it into an actual release of Firefox in order to make it look like the media were making a big deal about nothing[1], as you did previously on this topic. I'd suggest you take a good, hard look at your own tactics before accusing others of misinformation, and that anyone else doesn't take your word on these matters. (Sadly that comment got upvoted all the way to the top of the discussion, just like this one no doubt will.)
(Edit: also, while it was 4 years ago and my memory's a bit fuzzy, I seem to recall the earlier, actually exploitable in the wild, PKCSv15 vulnerability was discussed a lot in the tech media and community. It just didn't get the mainstream media attention, most likely because it wasn't as badly mishandled and "obscure security issue fixed already" is much less of a story than "you're at risk now and there's nothing you can do, because someone fucked up".)
You are confused. The NSS bug you're referring to was in False Start. Mozilla found and fixed the False Start bug before any release of Firefox (which is not the only user of NSS) could be impacted. I am not talking about the False Start bug. The PKCS1v15 parsing bug had nothing to do with False Start. Which makes your comment a little silly.
You're not talking about it anymore, most likely because someone from Mozilla called you out on it last time. Instead you're talking about a vulnerability that's 4 years old and have quietly stopped mentioning exactly how old it was, because it'd make it obvious just how weak your argument actually is.
So your problem with an argument you don't like is that I didn't make it? You didn't understand the point I was making on that thread and now, embarrassingly, are attempting to port your misconception into a thread that has nothing to do with either my point or your fictitious version of it. I don't think you know what you're talking about.
Okay, what exactly is your point then? You seem to be arguing that the only reason this got coverage was because it was easy for laypeople to understand, but none of the other examples you're comparing to involved the spectacular vendor incompetence in rolling out the patch that made Apple a laughing-stock online and gave the media such a sensational story to cover, and the only one [edit: well, only TLS one] that even made it into a release was so long ago most people on here have probably forgotten the details. (I wasn't on Hacker News at the time, but it definitely made it to Slashdot and as I recall resulted in a lot of people complaining about SSL/TLS being a disaster by design.)
I agree that TLS bugs aren't unusual. This is one of the reasons I don't trust TLS for anything serious. ;-)
But as I said a few days ago (https://news.ycombinator.com/item?id=7284957) I think this far more likely to be an NSA-exploited bug than an arbitrary-code-execution bug in Flash, because it's something which can be exploited widely and quietly without worrying about accidentally causing new versions of the program to core dump and reveal the presence of the bug.
I hate to say it, but I think you're being insufficiently paranoid.
We already know that NSA exploits Flash bugs, and bugs like it, in the large, with impunity. They have an elaborate MITM apparatus to direct people to clientside exploiters. These things we all disclosed in the leaks. Why make new stuff up? I know security people at Apple. I do not believe for a heartbeat that they would cooperate with NSA on a backdoor.
I'm coming around to your take on TLS, for what it's worth, although the only thing I can think of that's worse than TLS are the crypto transports generalist developers come up with.
> I know security people at Apple. I do not believe for a heartbeat that they would cooperate with NSA on a backdoor.
In other words, you're not being objective. You already attribute good character to the people you know at Apple, and are therefore are looking for ways to defend them against an accusation that hasn't even been made. The fact that you unnecessarily called Schneiers current occupation as 'sad', imo, tends to support this rationalisation.
None of the issues with Flash or certificate verification bugs you bring up are relevant to the discussion Schneier is having about this being the perfect kind of backdoor, and that, if it were a backdoor, it would have been hugely successful over the last year or more.
Are we perhaps operating from different definitions of the word "backdoor"? If so, I think mine's correct: a backdoor is a security vulnerability deliberately introduced into a codebase.
I have no doubt at all that if NSA knew about this bug, they were exploiting it. That's what they do. It is not my argument that this bug is irrelevant to NSA.
If you assume that the only way that the NSA could get a bug into apple products is by making a call to Apple execs and asking them to do it. What do you think of the possibility of them approaching someone who already codes at Apple, possibly someone who they've had previous contact with, or have leverage against that would guarantee that they won't go public about an (unsuccessful) approach from the NSA, and employing them to find places to insert deniable vulnerabilities into Apple products?
It's unlikely that Apple would agree to sabotage their own products in a way that will eventually be detected and affect their own credibility. That's restricting the argument that people are making to a straw man, though. Have spies ever worked that way? What people are mostly speculating about is whether this was an intentionally introduced vulnerability, not the mechanism by which that would come about.
"The fact that you unnecessarily called Schneiers current occupation as 'sad', imo, tends to support this rationalisation."
Let me give you the perspective of someone who is outside the security business but knows a good deal about PR and the media and how it operates. And selling books (one that I was doing was mentioned on the front page of the WSJ - all from PR that I organized on my own).
Schneier has certainly transformed himself and it's all about his image and selling books with his new found fame. I know it when I see it. Because as he just said in a way "it's what I would do if I wanted to..."
So in other words he has an axe to grind in getting mentioned in the media. Period. That doesn't mean what he is saying isn't correct in some fashion of course. But it does give credence to the statement "Another sad indicator of the level Schneier is playing at today". That level is publicity for Schneier. So to repeat myself, exactly what I'd be doing if I was him.
> I know security people at Apple. I do not believe for a heartbeat that they would cooperate with NSA on a backdoor
So you're absolutely sure that there is no developer (in the position) at Apple that would take a discrete paper bag of $$$ to make just a tiny error by hitting ctrl-v twice?
For one thing, it's hard to imagine an amount of money that would be simultaneously large enough to entice an Apple engineer to risk his job and reputation, but small enough to not trigger questions about where it came from. You'd have to concoct elaborate schemes that convince the IRS to look the other way, or target someone who could be blackmailed without fear that they'll run immediately to the authorities or their boss. Not anywhere near as easy or safe as handing over a bag of cash.
If I were the NSA, a strategy with much lower variance would be to hire a lot of smart folks to do the intense security reviews and testing that Apple (and many other companies, presumably) fail to do, and try to find bugs before they're patched. As tptacek has pointed out, there are plenty of these, and an organization with the resources of the NSA can probably find them frequently enough for most purposes without sticking its neck out.
Surely this hypothetical bribed engineer will be identified during Apple's post-mortem. Hopefully he's a good liar, otherwise the NSA risks exposure that way. Even if he's a terrific liar, of course his reputation will be hurt: His 'mistake' caused Apple a major PR hit and engineering scramble that likely cost millions of dollars.
And for what? The NSA can hide money, but you or I would quickly find ourselves talking with the IRS if our spending or bank accounts rose dramatically without a commensurate rise in reported income. We could probably hide a few thousand dollars a month under the table, but that doesn't sound like it would turn the head of even a particularly amoral Apple engineer already making six figures.
I'm not saying this scenario is impossible. But to be plausible, it requires both the perpetrator and the NSA to take on a degree of risk that doesn't seem to match the reward for either.
If this is standard procedure for the NSA and happens often, given the risk involved, why hasn't anyone been caught at it and publicly exposed?
If the NSA made an exception and executed some unusually elaborate cloak and dagger to seduce or blackmail an Apple employee for this change, why do we think it was for this vulnerability and not the dozens of others that are discovered annually?
Since we're at a nosebleed altitude of hypothetical here, I think now might be the time to point out that this thread is demanding that people prove a negative.
> If I were the NSA, a strategy with much lower variance would be to hire a lot of smart folks to do the intense security reviews and testing that Apple (and many other companies, presumably) fail to do, and try to find bugs before they're patched.
Very very good point.
Until you made it, I might have thought "well of course NSA looks for bugs to exploit", but I didn't think through the implications.
Here we have Apple open sourcing their security code. They made the NSA's job 10x easier than w/o the source. So of course the NSA will lint and review and test and exploit Apple's source code. Intensively. It's low-hanging fruit, of the Apple variety. :)
I know security people at Apple. I do not believe for a heartbeat that they would cooperate with NSA on a backdoor.
I agree with you based on the security people I know at Apple. However, I would be surprised if Apple's security people have enough time to carefully audit every line of security-relevant code which other people at Apple touch.
>Once again: the only reason this bug got so much attention and press is that it's easy for laypeople to get their heads around. All you have to understand is how "goto" works.
I don't agree. Most people don't even have a basic understanding of coding. I think that it has more to do with the fact that there are about 500 million vulnerable devices out there, and that we are talking about Apple. Remember Antennagate, it wasn't even that big of a deal and look at what the press made of it.
It's hard to design a good TLS test suite, for at least a couple reasons:
* The TLS stacks all have different APIs, and you'll need some fine-grained configuration over client options to exercise the code.
* There are only a few important TLS stacks, and the effort required to build a test suite is probably greater than the effort required to build a TLS stack.
It would still be a cool project. If it exists, I'd love to hear about it.
I think you're using a different definition of "worse" when it comes to backdoors, compared to Schneier (in this post). His yardsticks seems to be "low chance of discovery, high deniability if discovered, and minimal conspiracy to implement."
When you say "significantly worse bugs are found every week", are you considering them worse using Schneier's yardsticks above, or your own (which to me as a lay person appear to be their potential for damage)?
I cared a lot more than usual about the bug because iOS is generally my go-to advice for "secure web browsing platform", and because I personally use iOS and OSX, and the slow response on OSX patching. I expect people care about Apple products in the media for this reason -- Microsoft bugs would have similar reach (as did the Android RNG bug), but Apple bugs are relatively rare and thus more a dog bites man story.
Flash I know to be insane. I don't use Firefox. So, less reason to care.
But yes, Schneier has become the USA Today of security advice. A lot of people read USA Today, so it's important, but it's probably not where you go for deep or substantial analysis. (and, generally a week after everyone else has discussed an issue to death)
Bruce Schneier does have vast expertise when it comes to security/encryption and the fact that he is formulating his conclusions in a way that they can be consumed by the general public, is something that is commendable and makes him way more dangerous for the NSA. Which is one of the reaseons why he gets attacked for it, i guess.
Does he? What makes you think that? Can you be specific? I understand this to be an article of faith, but cryptography has changed significantly since the 1990s --- and software security has (a) changed even more and (b) been a pretty consistent weak spot in Schneier's analyses even in the '90s.
Bruce Schneier is an exceptionally good writer and a very good public speaker. That alone is sufficient to make him incredibly important in the security community.
>Once again: the only reason this bug got so much attention and press is that it's easy for laypeople to get their heads around.
I think that's the reason why this bug should get a lot of attention and press. Most programmers can see this bug instantly, without even knowing the purpose of the code. How long does something like that in a (maybe the most) security-critical piece of code in a product go unseen and uncorrected without it being a result of maliciousness or incompetence?
How many dopey errors like this would have to be inserted into how many products to guarantee that the vast majority of communications could be monitored for (lets say) 5-10 years? 50 errors in 15 products? This single error compromised all iPhones, all iPads, and all Macs.
This sort of strategy would be the most efficient way for the NSA to spend money that I've ever heard of it doing. If money isn't exchanged (the companies aren't being paid to compromise themselves) we don't even have the evidence trail that outed RSA, who sabotaged their products in an equally deniable way.
I'm with the people asking who did the commit that introduced the bug. I'd like to know if they had any intelligence connections.
You think that because you learned about the bug from blog posts that just showed the diff. If that was what the developers of the stack saw, I'd agree. But of course, that isn't all they saw: you're looking at a needle in a haystack that someone has taken the time to find for you, and then wondering why it was so hard for anyone to distinguish between a needle and a blade of grass.
I'm not thinking that from the diff, I'm thinking that from the two gotos in a row.
Can you think of a reason why that would be in any computer program, and if you saw it in while reading any code, wouldn't it either indicate some weird trick that you maybe don't understand (and should look up) or an obvious error?
It doesn't look like anything that any static analysis tool wouldn't instantly scream about, either. I can't think of a less subtle error to accomplish the same thing.
Have you been reading the same technical coverage of this story that I have? Because what I'm reading is people saying that their static analysis regimes wouldn't have caught this, starting with Adam Langley. Veracode, which exists to sell a tool designed specifically to catch stuff like this, wrote the same thing.
I haven't. I'd like links, though. If I can see with my eyes that code will never be executed, I'd like to know what's wrong with the state of current static analysis.
Veracode does static analysis of the binary, and says that it would be thwarted because it wouldn't know that some functions were meant to be called. Static code analysis would tell you that it was impossible for some of the code to be run.
edit 2:(the second comment on the vericode blog)
caf | February 24, 2014 11:05 pm
In the Evil Unit Tests part you can use code coverage tools to at least verify that your unit tests exercise all code paths. It won’t catch every bug, but it would have caught this one.
The ability to understand it when it's pointed out is unrelated to the ease of spotting it among a thousand lines of other code when you don't know what you're looking for.
Also there is oclint not listed there, which is pretty amazing, but it has a strange notation for ignoring wqrnings. splint is pretty good too though and still supports the legacy lint stuff like /* NOTREACHED /
BSD lint seems to work still (though it looks like the lintlib is no longer built on FreeBSD and stdlib is not lint clean!) but even it would have caught this gotofail:
01: #include <stdlib.h>
02:
03: int
04: main(int argc, char *argv[])
05: {
06: if (argc == 0)
07: goto fail;
08: goto fail;
09:
10: exit(0);
11:
12:
13: fail:
14: exit(1);
15: }
$ lint a.c
a.c:
stdlib.h(286): warning: ANSI C does not support 'long long' [265]
stdlib.h(287): warning: ANSI C does not support 'long long' [265]
stdlib.h(287): warning: ANSI C does not support 'long long' [265]
a.c(10): warning: statement not reached [193]
a.c(15): warning: function main falls off bottom without returning value [217]
a.c(4): warning: argument argv unused in function main [231]
_types.h(61): warning: struct __timer never defined [233]
_types.h(62): warning: struct __mq never defined [233]
lint: cannot find llib-lc.ln
Lint pass2:
exit used( a.c(10) ), but not defined
$ cc a.c
$ ./a.out
$ echo $?
1
edit: grrr, sorry small copy/paste mistakes before.
The question I have is how many Open Source projects use static analysis tools as part of their development process?
Given that the code is publicly available it seems likely that bad actors would be using such tools (both proprietary and open) to find exploits for each release.
I remember Eric Raymond blogging about using Coverity to analyze gpsd but beyond that I don't recall any wider discussion about the issue.
Where am I mistaken that having this bug deliberate is meaningful only if you control most of the networks? In which case there are a lot scarier things to worry about.
Yes, and this is actually a sane coding rule. Without it, badly written macros (using several ;-separated lines for example) will cause subtle bugs like this one.
I would argue that in this case the problem lies in the macro, not the lack of braces. You can't program defensively against broken macros.
I've seen quite a lot of discussion surrounding the coding style around this "goto fail" vulnerability, but IMO that's missing the forest for the tree. If this coding error was truly a mistake (which I tend to believe) it was probably the result of a bogus copy/paste. Shit happens, no matter the coding style.
In my opinion the real problem is: why wasn't this code properly tested and audited? For crypto code it borders on gross negligence.
In particular, I disagree with TFA that "The flaw is subtle, and hard to spot while scanning the code". When you're used to read properly indented C code the 2nd "goto fail" with its weird indentation jumps to the eye IMO. Also the fact that the same statement is repeated twice in exactly the same way. I'm used to reading a lot of kernel code using the same kind of error handling and it really stands out IMO.
I don't think the NSA is nearly as technologically capable as people think they are. If they want info from someone all they have to do is detain them and bust open their kneecaps with a hammer. No one would ever find out. So there's really no reason to go through all the shadow games.
If this is an intentional bug, I think it's likely a hacker or just a disgruntled employee. But I'd be willing to wager that it was a tiredness error, rushed to implementation by an overworked and likely underpaid engineer deep within Apple.
Have you seen the more technical slides with some of the hacks they use? They are very technilogically capable. Actually, they are much more capable than most people used to think.
I'm sure they can do these things. I'm just not sure why they would. If the NSA wants my encrypted email keys they can ask me, they can hack my server, or they can break my jaw. The last option is considerably cheaper. So the entire organization seems like a gigantic waste of money to me.
I understand your point but you might be missing NSA view on this. Their goal is (apparently) to source as much data as possible and, although breaking a jaw doesn't sound expensive as a one-off, it gets extremely expensive when doing it for a given percentage of the world population. In our own language, it's not scalable. And neither are the other options. Global automation of the gathering of personal data is much cheaper and much more useful (again, apparently).
Breaking your jaw is most definitely not cheaper. Not by a long shot.
The NSA could, for example, compromise Comcast routers, use this bug to MITM every single SSL connection to e-mail servers, and collect literally millions of people's credentials and messages.
The cost of doing the equivalent with "break my jaw" techniques is astronomical. You'd need massive manpower, huge facilities, and of course the public backlash would be almost incomprehensibly vast.
Programmers of all people should be able to understand that an automated, computerized solution can be applied to a large scale extremely cheaply, and that this is pretty much the only way to scale up to millions of users. In the NSA's case, it's targets instead of users, but the same principle applies.
The NSA wants all of the data. Technological measures is how to obtain it, and that's what they've been doing.
You're absolutely wrong on this. The NSA are far, far more technologically competent than we've been giving them credit - and that is precisely why the NSA releases have been so imporant, and chilling, to the computer industry and hacker community. The NSA are way ahead of the pack. To think of them as a stupid bungling agency, the way most people associate government TLA's, is doing the world a disservice. Their competence is terrifying - as in it literally induces terror.
I think they are paranoid and have a blank check budget. But I don't think they are competent. If they were competent then they would know that no serious threats will be discussed in world of Warcraft or over yahoo messenger. The supposed enemy in the shadow is at least smart enough to know better. If they aren't, then they aren't smart enough to warrant the invasion if basically everyone's privacy. It just doesn't add up to me. If we are fighting super technical enemies, we must assume they are smart enough to not use basic, consumer grade chat. If they are not technically capable enough to know better than to use something like yahoo, then just send them an email with Sub7 in it. Either way, spending billions of dollars of my money to watch some millionaire banker sext with his secretary is a waste of money and of time.
> If they were competent then they would know that no serious threats will be discussed in world of Warcraft or over yahoo messenger.
Or they're pretending to be incompetent so they have an excuse to swallow up World of Warcraft and Yahoo Messenger chat logs. Because they don't care about terrorists, they care about logging /everything/. It's about power.
>If they were competent then they would know that no serious threats will be discussed in world of Warcraft or over yahoo messenger.
I don't know, game chat systems seem like a pretty good place to conspire. Imagine something like Call of Duty, you can pretty obviously talk about attacks and targets without setting off any red flags since it would be indistinguishable from real game chat.
> If they were competent then they would know that no serious threats will be discussed in world of Warcraft or over yahoo messenger.
No, that would be a sign that they were incompetent, because secretly communicating groups do use unusual chats, such as graphical children's chat applets, in-game communications, shoutboxes and such. Why wouldn't they?
It's not just an "iOS" flaw. It's a "latest, greatest Apple OS" flaw.
And anyone who given a choice between SSL and TLS relies on TLS is not putting security as a top priority.
And why would anyone who cares even the slightest about security use Flash? Are you serious?
I guess this is why security consulting could be easy money... clients want to use Flash and "stay secure". Yeah, sure, we can handle that for you.
Well, now you cannot even use a Mac without the potential for HTTPS authentication not working. Better make sure the OS is updated. Sounds a lot like Microsoft. Maybe you could start a business updating Mac OS's.
"Does anyone know what's going on inside Apple?"
If they did they couldn't say. All employees are sworn to secrecy.
I blackhole all traffic from Apple devices to *.apple.com
You would not believe (or maybe you would, if you are a "security consultant" or some such)... you would not believe the amount of "phoning home" that these devices do.
I agree you can't trust "security consultants" who do their marketing via blogs and forums.
But you surely cannot trust Apple either.
The flaw was one line of code.
I'm curious. What is the size on the update?
Imagine if you could make the change yourself, recompile and dd an image to your device.
I know that function calls are fundamentally just jmp calls underneath, and that's exactly what a goto is too, but we built high level language compilers (including C compilers) so that we didn't have to deal with goto-centric spaghetti any more.
As with everything, gotos work just fine if you restrict yourself to a few specific uses. At that point, however, you've basically just invented function calls, switch statements, and maybe exceptions if your rules aren't strict enough, which can all be managed by a compiler which always generates the correct number of jmp's to do the job.
EDIT: you could more accurately replace each instance of "function call" in the above with "subroutine" as FORTRAN defines them, which is closer to what you get for free with gotos than a function call with an actual stack pointer.
A function call transparently returns to the point at which the call originated and resumes execution. A GOTO departs from the original location, never to return. That is a crucial difference.
> EDIT: you could more accurately replace each instance of "function call" in the above with "subroutine" as FORTRAN defines them, which is closer to what you get for free with gotos than a function call with an actual stack pointer.
Still wrong -- you're confusing two very different things. A function call, in both the assembly and high-level sense, is completely different from a GOTO instruction.
I feel that your answer proves the point my original comment - that there is literally Silicon level support for completely avoiding gotos. I stand by my disgust at finding that they exist in any non-legacy code, especially in something I generally think of as high quality (iOS[1]/OS X).
(combined with a global pointer to a stack, a few instructions and a jmp should be enough to reimplement functions, including storing the return jmp address somewhere, obviously. As I think I made clear, I believe we invented compilers so that we don't have to know how this stuff is actually implemented. I do realise that I'm never going to know more about assembly than the set of HN readers who will pounce on this kind of comment - I was just trying to show that I have heard of jmp, and that I know it is spat out by compilers to implement high level language features, and that any nontrivial compiled program will contain many jmp instructions. A bug from using goto where a switch or if/else statement would have worked is inexcusable, and also nothing to do with assembly instructions.)
> I feel that your answer proves the point my original comment - that there is literally Silicon level support for completely avoiding gotos.
Yes, and there's a reason -- chip design should reflect how computers are programmed at a higher level, and function calls produce code that's easier to understand, create and maintain. The reason is that function calls obey a logical hierarchy that often reflects the meaning of the algorithm being executed.
To see the point, try creating a recursive-descent parser that uses GOTO and avoids function calls.
This understanding isn't limited to an HN gaggle of nattering bystanders:
> I was just trying to show that I have heard of jmp, and that I know it is spat out by compilers to implement high level language features, and that any nontrivial compiled program will contain many jmp instructions.
Yes, but what a compiler does, and what a high-level language does, are often completely different. As one example, many recursive algorithms are compiled into functionally equivalent loops, to make them more efficient and to keep from blowing up the stack.
My point is that a compiler's output doesn't have to be read by a human, so it can break every rule. Not so for a high-level code listing that might have to be understood and maintained over decades.
It might be easier, but I wouldn't say that it's safer. Fake certificates leave a paper trail -- that's how hacked CAs have been discovered in the past.
If I were at the NSA and wanted to introduce a bug like this, I'd get access to Apple's build servers (either through an exploit or by just talking to an employee who has access) and arrange for a binary patch to be applied to the generated object files at build time. It would be basically undetectable, as no amount of source code auditing would reveal it. Could probably make it look like a compiler bug without too much difficulty.
Of course, this presumes that the NSA needs to introduce bugs like this. I imagine they do just fine for now merely taking advantage of naturally occurring bugs.
Again I think this was just a mistake, but there are other groups that could want to add something like this other than the NSA where it would be more plausible for them to have such a need. It shows that Apple needs to reconsider its processes though, and I guess they are now, which should make the likelihood of something like gotofail for nefarious means working very small from now on.
I'm hoping that their security folks are now going through and writing extensive automated tests for all of this code, and anything similar, and turning on lots of automated static analysis to go along with it.
I fear that it may end up as, "we already found the bug, we don't need tests now". But that's probably pessimistic of me.
I honestly think it was deliberate. What class of Operating System developer ships their OS releases without 100% CODE COVERAGE? Apple do code coverage testing, surely? I mean, more than the "-warn-dead-code" args that get flung around.
I can't understand how this would have gotten released into the wild if they were doing industry-standard code coverage tests. And .. if they're not doing industrial-strength code-coverage testing on their iOS/OSX release builds, thats the real news here ..
123 comments
[ 22.7 ms ] story [ 3477 ms ] threadHas Apple even publicly discussed the basic details of this bug? As far as I've seen, every single bit of analysis has come from outside the company, based on simply viewing the source code. Apple acknowledged that there is a vulnerability, but they haven't even said what it is in any detail.
The one thing that points to it not being a backdoor is I doubt Apple would open source the code. Surely they'd maintain a separate branch or something?
In other words, it's only possible that it's an NSA job if it's also possible that it isn't. Something tells me therefore that we're unlikely ever to know for sure.
(I certainly shouldn't pretend to know much about C, but the above matches my understanding of the situation)
The likelihood is pretty simple, someone fucked up. On a potentially huge level, but a fuck up none the less. These things do unfortunately happen, and no doubt it'll prompt an internal review of their change management process, and their build chain, and what they can do to isolate issues like this in the future.
A single employee receiving a second paycheck from a three-letter organization could have been responsible. A vulnerability could have been used to plant it. And so on. Remember when everyone was railing about Google giving NSA a backdoor (cue a thousand "don't be evil" cries) when in reality the NSA and friends were simply exploiting a weakness of Google's network. The net effect was the same.
It could be intentional and still entirely unintentional as far as Apple the company is concerned.
And to the open source thing -- it sat there for 18 months. Indeed, it was caught [EDIT: Actually I don't know how it was caught. I faintly recall someone posting an issue someone submitted to Apple detailing some weird behavior with invalid private keys, but can't find it now]
Agreed, as far as Apple know it was a bug, but one of their staff could have been turned. But they'll be able to audit the change logs, and I imagine a company famously known for their OTT internal security would be preparing the car batteries and nipple clamps if they thought they'd found a bad actor.
The fact that people can see the nature of the bug in the source code for themselves just makes the case that it was a deliberate backdoor even more deniable which, as Schneier is pointing out, is a very desirable property.
You're also assuming that a backdoor you can use indefinitely is desirable.
As an added side-benefit, any other proprietary or FOSS products that adopted this code are now vulnerable.
[0] https://en.wikipedia.org/wiki/Attribution_%28psychology%29#D...
As technically minded people we intrinsically understand that it could be the work of someone malicious, but we should also understand from personal experiences that fuck ups happen significantly more often than brown envelopes full of cash.
edit: Ha, I had my own copy paste error after getting bad gateway error, I fixed the duplicated code, sorry.
[1]: http://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines
Here's my alternative but equally reasonable platitude: "Never attribute to incompetence what could be attributed to malice with plausible deniability."
Once again: the only reason this bug got so much attention and press is that it's easy for laypeople to get their heads around. All you have to understand is how "goto" works. The bug is vivid, and so (paradoxically) seems scarier.
Significantly worse bugs are found every week. Within a few days of the announcement of this TLS bug, a Flash bug was announced, after being detected in exploits in the wild, that enabled reliable drive-by hijackings of browsers --- multiple browsers. It was off the HN front page within an hour.
TLS bugs aren't even unusual. We get a new one every few years ago. Firefox managed a PKCS1v15 parsing bug that allowed anyone with a Python script and 30 milliseconds to generate a certificate for any domain. Other browsers have screwed up certificate chaining, so that any domain could sign any other domain. But nobody understands PKCS1v15 padding, nobody understands certificate chaining, and so nobody writes stories about these bugs. But their impact is identical to this one.
[1] https://news.ycombinator.com/item?id=7284290
(Edit: also, while it was 4 years ago and my memory's a bit fuzzy, I seem to recall the earlier, actually exploitable in the wild, PKCSv15 vulnerability was discussed a lot in the tech media and community. It just didn't get the mainstream media attention, most likely because it wasn't as badly mishandled and "obscure security issue fixed already" is much less of a story than "you're at risk now and there's nothing you can do, because someone fucked up".)
But as I said a few days ago (https://news.ycombinator.com/item?id=7284957) I think this far more likely to be an NSA-exploited bug than an arbitrary-code-execution bug in Flash, because it's something which can be exploited widely and quietly without worrying about accidentally causing new versions of the program to core dump and reveal the presence of the bug.
I hate to say it, but I think you're being insufficiently paranoid.
I'm coming around to your take on TLS, for what it's worth, although the only thing I can think of that's worse than TLS are the crypto transports generalist developers come up with.
Do you have any information, whether they're using any kind of of code -security- auditing tools?
Could it be that the programmer simply got a backdoor request, which s/he in protest implemented this way to ensure to be caught.
In other words, you're not being objective. You already attribute good character to the people you know at Apple, and are therefore are looking for ways to defend them against an accusation that hasn't even been made. The fact that you unnecessarily called Schneiers current occupation as 'sad', imo, tends to support this rationalisation.
None of the issues with Flash or certificate verification bugs you bring up are relevant to the discussion Schneier is having about this being the perfect kind of backdoor, and that, if it were a backdoor, it would have been hugely successful over the last year or more.
I have no doubt at all that if NSA knew about this bug, they were exploiting it. That's what they do. It is not my argument that this bug is irrelevant to NSA.
It's unlikely that Apple would agree to sabotage their own products in a way that will eventually be detected and affect their own credibility. That's restricting the argument that people are making to a straw man, though. Have spies ever worked that way? What people are mostly speculating about is whether this was an intentionally introduced vulnerability, not the mechanism by which that would come about.
Let me give you the perspective of someone who is outside the security business but knows a good deal about PR and the media and how it operates. And selling books (one that I was doing was mentioned on the front page of the WSJ - all from PR that I organized on my own).
Schneier has certainly transformed himself and it's all about his image and selling books with his new found fame. I know it when I see it. Because as he just said in a way "it's what I would do if I wanted to..."
So in other words he has an axe to grind in getting mentioned in the media. Period. That doesn't mean what he is saying isn't correct in some fashion of course. But it does give credence to the statement "Another sad indicator of the level Schneier is playing at today". That level is publicity for Schneier. So to repeat myself, exactly what I'd be doing if I was him.
So you're absolutely sure that there is no developer (in the position) at Apple that would take a discrete paper bag of $$$ to make just a tiny error by hitting ctrl-v twice?
If I were the NSA, a strategy with much lower variance would be to hire a lot of smart folks to do the intense security reviews and testing that Apple (and many other companies, presumably) fail to do, and try to find bugs before they're patched. As tptacek has pointed out, there are plenty of these, and an organization with the resources of the NSA can probably find them frequently enough for most purposes without sticking its neck out.
The odds are that this engineer will never be revealed publicly, and it will not effect his/her reputation in any way.
>but small enough to not trigger questions about where it came from.
You don't think that the NSA has the ability to hide money?
And for what? The NSA can hide money, but you or I would quickly find ourselves talking with the IRS if our spending or bank accounts rose dramatically without a commensurate rise in reported income. We could probably hide a few thousand dollars a month under the table, but that doesn't sound like it would turn the head of even a particularly amoral Apple engineer already making six figures.
I'm not saying this scenario is impossible. But to be plausible, it requires both the perpetrator and the NSA to take on a degree of risk that doesn't seem to match the reward for either.
If this is standard procedure for the NSA and happens often, given the risk involved, why hasn't anyone been caught at it and publicly exposed?
If the NSA made an exception and executed some unusually elaborate cloak and dagger to seduce or blackmail an Apple employee for this change, why do we think it was for this vulnerability and not the dozens of others that are discovered annually?
Very very good point.
Until you made it, I might have thought "well of course NSA looks for bugs to exploit", but I didn't think through the implications.
Here we have Apple open sourcing their security code. They made the NSA's job 10x easier than w/o the source. So of course the NSA will lint and review and test and exploit Apple's source code. Intensively. It's low-hanging fruit, of the Apple variety. :)
I agree with you based on the security people I know at Apple. However, I would be surprised if Apple's security people have enough time to carefully audit every line of security-relevant code which other people at Apple touch.
I don't agree. Most people don't even have a basic understanding of coding. I think that it has more to do with the fact that there are about 500 million vulnerable devices out there, and that we are talking about Apple. Remember Antennagate, it wasn't even that big of a deal and look at what the press made of it.
[edit]
Also I bet most readers of Schneier's bug know what goto is as well.
Is there a public test-suite that implementers can check against?
Plus, if there were tests, they just become another target.
* The TLS stacks all have different APIs, and you'll need some fine-grained configuration over client options to exercise the code.
* There are only a few important TLS stacks, and the effort required to build a test suite is probably greater than the effort required to build a TLS stack.
It would still be a cool project. If it exists, I'd love to hear about it.
When you say "significantly worse bugs are found every week", are you considering them worse using Schneier's yardsticks above, or your own (which to me as a lay person appear to be their potential for damage)?
Flash I know to be insane. I don't use Firefox. So, less reason to care.
But yes, Schneier has become the USA Today of security advice. A lot of people read USA Today, so it's important, but it's probably not where you go for deep or substantial analysis. (and, generally a week after everyone else has discussed an issue to death)
http://www.theregister.co.uk/2014/02/20/flash_adobe_posts_em...
That was the day before the iOS update. You scared me for a moment that there was a flash update I had missed cause of all the hoopla around gotofail.
I think that's the reason why this bug should get a lot of attention and press. Most programmers can see this bug instantly, without even knowing the purpose of the code. How long does something like that in a (maybe the most) security-critical piece of code in a product go unseen and uncorrected without it being a result of maliciousness or incompetence?
How many dopey errors like this would have to be inserted into how many products to guarantee that the vast majority of communications could be monitored for (lets say) 5-10 years? 50 errors in 15 products? This single error compromised all iPhones, all iPads, and all Macs.
This sort of strategy would be the most efficient way for the NSA to spend money that I've ever heard of it doing. If money isn't exchanged (the companies aren't being paid to compromise themselves) we don't even have the evidence trail that outed RSA, who sabotaged their products in an equally deniable way.
I'm with the people asking who did the commit that introduced the bug. I'd like to know if they had any intelligence connections.
Can you think of a reason why that would be in any computer program, and if you saw it in while reading any code, wouldn't it either indicate some weird trick that you maybe don't understand (and should look up) or an obvious error?
It doesn't look like anything that any static analysis tool wouldn't instantly scream about, either. I can't think of a less subtle error to accomplish the same thing.
edit: http://blog.veracode.com/2014/02/do-not-pass-qa-do-not-goto-...
Veracode does static analysis of the binary, and says that it would be thwarted because it wouldn't know that some functions were meant to be called. Static code analysis would tell you that it was impossible for some of the code to be run.
edit 2:(the second comment on the vericode blog)
caf | February 24, 2014 11:05 pm
In the Evil Unit Tests part you can use code coverage tools to at least verify that your unit tests exercise all code paths. It won’t catch every bug, but it would have caught this one.
I'm talking about tools like http://oclint.org/ and http://clang-analyzer.llvm.org/
If they work anything like similar tools in other languages, they would have found this bug trivially.
To the more important questions 'Did the NSA almost immediately discover this bug' and 'Did they exploit it', I answer with a resounding yes.
The author can't be serious. This particular bug has made the rounds and is understood even by non-programmers.
The clang static analyzer: http://clang-analyzer.llvm.org
Valgrind: http://www.valgrind.org
Address Sanitizer: http://clang.llvm.org/docs/AddressSanitizer.html
The various -fsanitize options in clang (undefined behaviours, integer overflows etc): http://clang.llvm.org/docs/UsersManual.html#id28
http://en.wikipedia.org/wiki/List_of_tools_for_static_code_a...
Also there is oclint not listed there, which is pretty amazing, but it has a strange notation for ignoring wqrnings. splint is pretty good too though and still supports the legacy lint stuff like /* NOTREACHED /
BSD lint seems to work still (though it looks like the lintlib is no longer built on FreeBSD and stdlib is not lint clean!) but even it would have caught this gotofail:
edit: grrr, sorry small copy/paste mistakes before.Given that the code is publicly available it seems likely that bad actors would be using such tools (both proprietary and open) to find exploits for each release.
I remember Eric Raymond blogging about using Coverity to analyze gpsd but beyond that I don't recall any wider discussion about the issue.
But on the other hand, why didn't the compiler generate an "Unreachable Code" warning during build?
We have explicitly set this warning to "Treat as Error" during our builds.
http://gcc.gnu.org/ml/gcc-help/2011-05/msg00360.html
if (cond) { goto fail; }
instead of:
if (cond) goto fail;
I've seen quite a lot of discussion surrounding the coding style around this "goto fail" vulnerability, but IMO that's missing the forest for the tree. If this coding error was truly a mistake (which I tend to believe) it was probably the result of a bogus copy/paste. Shit happens, no matter the coding style.
In my opinion the real problem is: why wasn't this code properly tested and audited? For crypto code it borders on gross negligence.
In particular, I disagree with TFA that "The flaw is subtle, and hard to spot while scanning the code". When you're used to read properly indented C code the 2nd "goto fail" with its weird indentation jumps to the eye IMO. Also the fact that the same statement is repeated twice in exactly the same way. I'm used to reading a lot of kernel code using the same kind of error handling and it really stands out IMO.
If this is an intentional bug, I think it's likely a hacker or just a disgruntled employee. But I'd be willing to wager that it was a tiredness error, rushed to implementation by an overworked and likely underpaid engineer deep within Apple.
And completely legal (from their perspective), whereas attacking foreign citizens gets into grey areas.
The NSA could, for example, compromise Comcast routers, use this bug to MITM every single SSL connection to e-mail servers, and collect literally millions of people's credentials and messages.
The cost of doing the equivalent with "break my jaw" techniques is astronomical. You'd need massive manpower, huge facilities, and of course the public backlash would be almost incomprehensibly vast.
Programmers of all people should be able to understand that an automated, computerized solution can be applied to a large scale extremely cheaply, and that this is pretty much the only way to scale up to millions of users. In the NSA's case, it's targets instead of users, but the same principle applies.
The NSA wants all of the data. Technological measures is how to obtain it, and that's what they've been doing.
Or they're pretending to be incompetent so they have an excuse to swallow up World of Warcraft and Yahoo Messenger chat logs. Because they don't care about terrorists, they care about logging /everything/. It's about power.
I don't know, game chat systems seem like a pretty good place to conspire. Imagine something like Call of Duty, you can pretty obviously talk about attacks and targets without setting off any red flags since it would be indistinguishable from real game chat.
Its not.
It is creating one.
That is why its imperative that we do something about the NSA.
No, that would be a sign that they were incompetent, because secretly communicating groups do use unusual chats, such as graphical children's chat applets, in-game communications, shoutboxes and such. Why wouldn't they?
And anyone who given a choice between SSL and TLS relies on TLS is not putting security as a top priority.
And why would anyone who cares even the slightest about security use Flash? Are you serious?
I guess this is why security consulting could be easy money... clients want to use Flash and "stay secure". Yeah, sure, we can handle that for you.
Well, now you cannot even use a Mac without the potential for HTTPS authentication not working. Better make sure the OS is updated. Sounds a lot like Microsoft. Maybe you could start a business updating Mac OS's.
"Does anyone know what's going on inside Apple?"
If they did they couldn't say. All employees are sworn to secrecy.
I blackhole all traffic from Apple devices to *.apple.com
You would not believe (or maybe you would, if you are a "security consultant" or some such)... you would not believe the amount of "phoning home" that these devices do.
I agree you can't trust "security consultants" who do their marketing via blogs and forums.
But you surely cannot trust Apple either.
The flaw was one line of code.
I'm curious. What is the size on the update?
Imagine if you could make the change yourself, recompile and dd an image to your device.
As with everything, gotos work just fine if you restrict yourself to a few specific uses. At that point, however, you've basically just invented function calls, switch statements, and maybe exceptions if your rules aren't strict enough, which can all be managed by a compiler which always generates the correct number of jmp's to do the job.
EDIT: you could more accurately replace each instance of "function call" in the above with "subroutine" as FORTRAN defines them, which is closer to what you get for free with gotos than a function call with an actual stack pointer.
No, this is false, both at the high-level-language and assembly-language level:
http://en.wikipedia.org/wiki/X86_calling_conventions
A function call transparently returns to the point at which the call originated and resumes execution. A GOTO departs from the original location, never to return. That is a crucial difference.
> EDIT: you could more accurately replace each instance of "function call" in the above with "subroutine" as FORTRAN defines them, which is closer to what you get for free with gotos than a function call with an actual stack pointer.
Still wrong -- you're confusing two very different things. A function call, in both the assembly and high-level sense, is completely different from a GOTO instruction.
(combined with a global pointer to a stack, a few instructions and a jmp should be enough to reimplement functions, including storing the return jmp address somewhere, obviously. As I think I made clear, I believe we invented compilers so that we don't have to know how this stuff is actually implemented. I do realise that I'm never going to know more about assembly than the set of HN readers who will pounce on this kind of comment - I was just trying to show that I have heard of jmp, and that I know it is spat out by compilers to implement high level language features, and that any nontrivial compiled program will contain many jmp instructions. A bug from using goto where a switch or if/else statement would have worked is inexcusable, and also nothing to do with assembly instructions.)
[1] actually, iOS 7? I take that back.
Yes, and there's a reason -- chip design should reflect how computers are programmed at a higher level, and function calls produce code that's easier to understand, create and maintain. The reason is that function calls obey a logical hierarchy that often reflects the meaning of the algorithm being executed.
To see the point, try creating a recursive-descent parser that uses GOTO and avoids function calls.
This understanding isn't limited to an HN gaggle of nattering bystanders:
http://www.u.arizona.edu/~rubinson/copyright_violations/Go_T...
> I was just trying to show that I have heard of jmp, and that I know it is spat out by compilers to implement high level language features, and that any nontrivial compiled program will contain many jmp instructions.
Yes, but what a compiler does, and what a high-level language does, are often completely different. As one example, many recursive algorithms are compiled into functionally equivalent loops, to make them more efficient and to keep from blowing up the stack.
My point is that a compiler's output doesn't have to be read by a human, so it can break every rule. Not so for a high-level code listing that might have to be understood and maintained over decades.
Of course, this presumes that the NSA needs to introduce bugs like this. I imagine they do just fine for now merely taking advantage of naturally occurring bugs.
I fear that it may end up as, "we already found the bug, we don't need tests now". But that's probably pessimistic of me.