What a great writeup. Comprehensive without being overly verbose, answers to "what does this mean?" and "does this affect me?", and clear calls to action.
While I'm not happy at having to spend my Monday patching a kajillion machines, I welcome more vulnerability writeups in this vein.
Well, I don't think it's anything in memory, but whatever was up to 64k from wherever the downloaded packet was put in userspace (Edit: Er, 64k at a time, but the attacker can try again over and over). Since the kernel should be handing only zeroed pages to userspace to use as a buffer then it should only be memory used by the process using openssl at risk.
The big problem is that this is still a gigantic range of processes (and possible memory buffer contents). But SSH at least would appear to be fine, unless you've ever transferred an SSH key over TLS using OpenSSL.
What you probably want is to re-key your cert, do not revoke it. Revoking with some CA's (such as GoDaddy) means to essentially cancel the remainder of the valid date forever and requires purchasing a new cert to secure the same domain. You are forfeiting the rest of its value.
When you re-key, it will automatically deactivate the previous cert and is free. It also gives you the opportunity to update to SHA-2 or increase the key to 2048 bit, which you should do unless you have unusual and extreme legacy support needs (and must keep SHA-1 a while longer).
I disagree. Revoking the certificate is a requirement. If you re-key without revoking, that means someone who has stolen your key could impersonate you until the validity period expires. So revoking is a needed if you want to inoculate yourself against a potential active man-in-the-middle attack.
If you want to be secure, make sure the certificate based on your old key is showing up in the certificate revocation list (CRL), and/or any online certificate status protocol (OCSP) servers it specifies.
(What I want now is an exploit.c, PoC.py, pwnSSL.rb, etc... but I guess it would be irresponsible to provide that to the script-kiddies of the interwebz right now)
The part that's caused me to read this page several times over without a clear answer is that they mention that private keys may be leaked, but their calls to action do not recommend generating new private keys. How does that make any sense?
>this leak requires patching the vulnerability, revocation of the compromised keys and reissuing and redistributing new keys. Even doing all this will still leave any traffic intercepted by the attacker in the past still vulnerable to decryption. All this has to be done by the owners of the services.
Yes, down in the Q&A of details of what's leaked, not in the "here's what you need to do" section. It makes you think, "wait, the details say reissue keys...why does the 'what you need to do' section not say that? Did I misunderstand?" It's not very clearly written. Not to mention "revocation of the compromised keys" is itself vague: which keys are compromised? "The crown jewels" of course. We must infer that we're talking about the SSL private keys. And again, because revoking those keys is not mentioned in the call to action, you're forced to question whether your inference is correct.
As an actionable bulletin, this page leaves a lot to be desired. Nice logo and domain name, though.
One (selfish) question I have is whether this can affect primary key material stored in an HSM. I'm assuming not, but that the session key generated by the HSM would still be susceptible.
Yeah, I haven't seen any new RPMs for RHEL/CentOS/Fedora yet. Kinda concerning, since I'd expect vendors to be given advance notice and the chance to prep updates to coincide with the announcement.
All my RHEL5 boxes are running 0.9.8, though, at least.
Homebrew has updated to 1.0.1g since 6:00PM GMT. It's important to note that this isn't an issue unless you have an outward facing service that uses TLS and the brew/macports library
It does take time for these things to be tested and deployed. Regardless of severity of bug, distributions must test packages before sending them out to all their users.
It would be unfortunate if a new package were to be released immediately only to be soon masked/recalled due to unforeseen consequences.
Of note, the Gentoo package was bumped approximately 2 hours after the advisory was published.
That is my concern as well. We are still running CentOS 6.4 which does not have the affected version of OpenSSL, but we terminate SSL at the ELB so if they are affected then are keys are not safe.
The forum thread has just been updated with this reply:
"We can confirm that load balancers using Elastic Load Balancing SSL termination are vulnerable to the Heartbleed Bug (CVE-2014-0160) reported earlier today. We are currently working to mitigate the impact of this issue and will provide further updates."
Rackspace guy here. We have been digging in and it appears that we did have the impacted version of openssl installed but the heartbeat extension was disabled. Regardless, we have updated everything on the Cloud Load Balancer side to 1.0.1g. I will update here if we find anything different.
I'm very curious to see the change that introduced the bug in the first place. According to the announcement it was introduced in 1.0.1. That's the version that added Heartbeat support, so maybe it was a bug from the beginning.
Probably to make it more clear what you're referring to, and double-check yourself. There are probably components that are 1 byte, 2 bytes, and 16 bytes long. Writing it out makes it clear and eliminates a chance for human error in the sum, more than a magic 19 does. (I guess 16 is pretty magical too, though. At least it's a "round" number, and in context may be a well-known field size of something in the protocol.)
Yeah, this. It is fairly common to see things like this in C-like languages when it comes to times, like when including the milliseconds in a day you might see:
int timeout = 1000 * 60 * 60 * 24;
(milliseconds in a second * 60 for minute, * 60 for hour, * 24 for hours in a day).
Much more obvious than just putting in 86400000, and the compiler will optimize away the math and putting the math in there explicitly is arguably better than a comment that could easily become unanchored from the real value (if someone changes the value and forgets to update the comment).
When it comes to byte-sizes of things, though, most code will use sizeof() to both make it more clear where these numbers are coming from and to make them automatically adjust if the structure sizes change (granted this is unlikely to happen on a mature protocol).
At the very least having them be preprocessor defines would certainly make things a lot more clear here, so even for C I'd consider this a bit of a "code smell" (though the people who work on this code regularly are probably versed enough in the ssl3 record structure enough that they immediately grok this when they see it).
Great writeup but I guess I'm still a bit confused. As someone responsible for rails servers I can see that I need to update nginx and openssl as soon as packages become available or compile myself. What about keys though? Do I need to get our SSL certs re-issued? regenerate SSH keys? Anything else that I should be doing?
If you're running a vulnerable version of OpenSSL and want to be truly careful, assume your private keys (not just certs) are already compromised. Once new packages are available, you need to update and then re-roll your crypto.
Also, if you're using those keys to protect other secrets like passwords - say, DB credentials or AWS keys stored in an HTTP-hosted Git repo behind - you can't really assume those are safe either.
Does SSH (specifically sshd) on major OSes use affected versions of OpenSSL? [answer pulled up from replies below: since sshd doesn't use TLS protocol, it isn't affected by this bug, even if it does use affected OpenSSL versions]
What's the quickest check to see if sshd, or any other listening process, is vulnerable?
(For example, if "lsof | grep ssl" only shows 0.9.8-ish version numbers, is that a good sign?)
The bug is in the handling of the TLS protocol itself (actually, in a little-used extension of TLS, the TLS Record Layer Heartbeat Protocol), and isn't exposed in applications that just use TLS for crypto primitives.
when openssh uses certificates it still uses its own protocol (even with the x509 patch - without that, the certificates used by openssh are not even the same kind of certificate as those used by openssl).
the problem with the openssl library is in the implementation of the TLS protocol. this is not used by openssh. so openssh is not affected by this problem. even when certificates are used. and even when x509 certificates are used (which requires a separate patch).
Not only HTTPS. Many other protocols are TLS-based: modern email, some VPNs, etc. Really almost everything secret on the Internet is protected by TLS; SSH is a rare exception.
First off, TLS is crypto bread-and-butter that's used for a lot more than HTTPS. You're not out of the woods because you're not running a webserver.
Second, SSH itself doesn't use TLS; it has its own protocol, so sshd isn't vulnerable.
But third, read overflows like this can be escalated in countless ways to total compromise if some credential, key, canary, or such gets leaked. So just because sshd isn't vulnerable doesn't mean you're not screwed.
I am having a lot of trouble figuring out what you were attempting to convey in the first four sentences in your comment.
The one thing that I can discern is that you printed a list of every package that depends on libssl1.0.0 for your configured repositories. But you have no idea if those programs make use of heartbeat. ssh (and everything related like libpam-ssh) is on that list and does not use TLS. The same can be said for many others such as tpm-tools, ntp/ntpdate/openntpd, xca and so on.
Ideally, no, but with SSL, and a very-strong password, it might not be worse than other options for automating payouts... until a major bug like this comes along.
For comparison, the Bitcoin RPC password timing bug - https://github.com/bitcoin/bitcoin/issues/2838 - would have been a more slower and blatant/detectable way to compromise the same sorts of bitcoin RPC daemons.
I'm fairly confident that you're wrong about "nobody"... but you'll have to find the examples yourself.
Yes, it requires extra settings in bitcoin.conf: to enable RPC, accept connections from non-local addresses, and use SSL. But it's only 5 lines if the host is not otherwise firewalled from the net:
But couldn't it be the case that with this bug you could sweep private keys from server's memory if they happen to be in there, because bitcoind is using them at the moment?
Yes, I would put that under the "implications for Bitcoin web services that use HTTPS" category. For context, the deleted question asked about the "Bitcoin blockchain" so I was responding to that specifically, with the added caveat for Bitcoin services that use HTTPS.
Basically yes. However, from my experience, package update urgencies are no good indicator of the updates's actual priority.
It's in the +*-security" channels and you're supposed to apply all updates from there.
I just wanted to point out that you really do not need the `apt-get clean.` Obviously your work flow is your business but I wanted to speak up in case you thought it was needed before upgrading packages.
It seems like you are somewhat new to the Debian utopia. Here is another great package that a lot of people are not aware of `apt-listbugs.` After you say "yes" to apt-get upgrade, apt-listbugs queries bts for bugs in the packages:version you are about to install. If any bugs are found you have the chance to review the report to see if it applies to you and if it does you can have apt-listbugs pin the package so that the new buggy version is not installed. Every night at midnight (i think) apt-listbugs queries bts to see if the bugs are still relevant and unpins the package if the bug is no longer relevant. It is especially handy for testing/unstable/experimental.
By default it only prompts you for grave-serious bugs. I have been bitten a couple of times by "important" bugs so set listbugs up so that it also checks for "important" bugs. This makes it a tiny bit noisier but not enough to make me switch to the defaults. Changing the severities is easy:
Just received an upgrade on Ubuntu 12.04 LTS as well, apt-get clean issued before updating.
EDIT: If you are using DigitalOcean, the update is not yet on their mirrors. Issue 'sudo sed -i "s/mirrors\.digitalocean/archive.ubuntu/g" /etc/apt/sources.list;sudo apt-get clean;sudo apt-get update;sudo apt-get upgrade' to get the patch. Check the comment by 0x0 above ( https://news.ycombinator.com/item?id=7549842 ) to find any services which need restarting.
I don't quite understand how this bug works. I would appreciate any input from someone knowledgeable.
It sounds like the heartbeat code is sending some data in the handshake. That data should be harmless (padding? zeroes?) but the bug results in reading off the end of an array and from whatever other data happens to be there. Someone sniffing the connection can then see those bytes fly by. If they happened to contain private info, game over.
Is that a correct read on the situation? If so, my followup questions are: 1) Why is there any extra data being sent at all beyond a simple command to "heartbeat"? 2) How much data is being leaked here and at what rate? Is it a byte every couple of hours, is it kilobytes per minute, or what?
I am particularly interested in #1, since that's the part I really don't get at the moment. I suspect the answer to #2 will be implied by the answer to #1.
Can attacker access only 64k of the memory?There is no total of 64 kilobytes limitation to the attack, that limit applies only to a single heartbeat. Attacker can either keep reconnecting or during an active TLS connection keep requesting arbitrary number of 64 kilobyte chunks of memory content until enough secrets are revealed.
...so I guess the answer to 2 is only limited by how frequently you can change the heartbeat settings, and how frequently OpenSSL will send a heartbeat packet.
One obvious - if slightly paranoid - answer is that this was a deliberate backdoor. There appears to be a length field specific to the heartbeat packet that's used to determine how much data from the original packet is included in the response, isn't checked against the actual packet length, and allows lengths up to 64k which is unnecessarily generous for the intended purpose but very useful for this attack.
>>> TLS heartbeat consists of a request packet including a payload; the other side reads and sends a response containing the same payload (plus some other padding).
So, what happens is that the payload comes in as a pointer and a size (up to 64kb). The server then prepares a response and copies the memory block [pointer, pointer+payloadSize] into the request.
The attack happens when the payload is smaller than the payload size passed in the request. This results in the response preparation dumping the memory block [pointer+realPayloadSize, pointer+payloadSize] into the response.
Any data in this block is now exposed to the callee; and could contain any data from the process.
Thanks. That lines up with what I've seen elsewhere too. I think the main thing I was missing was that this is not a sniffing attack, but rather an active attack where you talk to a peer over SSL and basically trick it into sending you some content from its memory.
> Recovery from this bug could benefit if the new version of the OpenSSL would both fix the bug and disable heartbeat temporarily until some future version... If only vulnerable versions of OpenSSL would continue to respond to the heartbeat for next few months then large scale coordinated response to reach owners of vulnerable services would become more feasible.
This sounds risky to me. I'm afraid attackers would benefit more from this decision than coordinated do-gooders.
I believe the reason they got access was one of their customers found it and reported it to them, and they reported it to OpenSSL, and then it somehow leaked (either with the OSSL release, or someone else) and then they posted their now-public writeups of it.
That's not correct. One of the individuals who discovered the bug contacted us as a large provider of SSL termination services. We were asked not to further disclose the details until it was officially patched and announced by OpenSSL. The official announcement occurred today after which we put up a post to let our customers know that they were protected.
I wonder who else was notified early? I noticed Apple's ocspd was downloading an unusual amount of data back on March 31. Could be unrelated, but Apple and other big software vendors would make sense for early notification.
Any chance this bug originated with the NSA? It seems like it would fall under their goal of subverting the infrastructure that keeps secrets on the internet. Of course this is exactly why such a goal is a bad idea - an unprotected internet causes widespread damage.
they did not try to "weaken RSA", as in the RSA algorithm. They paid off and/or infiltrated RSA the corporation. You were not attacked, your posts simply contained wrong information and useless speculation.
Screaming about the NSA every time a security bug comes up is not interesting, productive, insightful, or useful, please stop.
Asking "did they do this?" is not an accusation, seriously.
And even then, the NSA has had their fingers in enough places and lied about it enough times (infiltrating FOSS projects was explicitly one of their goals, IIRC) that the sane default position would be to assume shenanigans on their part unless proven otherwise.
The vetting process does absolutely nothing to prevent something like this from happening, especially since some very sneaky and pernicious bugs can be introduced in the guise of simple mistakes. It would be foolish to assume this isn't part of the standard playbook, and just as foolish to discount the possibility of maliciously introduced bugs just because the evidence doesn't immediately point to malicious intent - that is the nature of the attacker.
The alternative is remaining ignorant and vulnerable to the single most well funded and experienced adversary a crypto user will ever likely face.
I don't know -- why don't you try reasoning it out since you're the one lobbing the accusation. Upon a very simple review of the code change/patch, one can see this is a relatively new feature, agreed upon and passed by the publicly available IETF, implemented naively.
"Never attribute to malice that which can be adequately explained by incompetence" -- slightly-butchered quote, from someone smarter than me.
It's not an accusation, it's a speculation. I don't have the ability to judge it for myself, i.e. "a simple review of the code change/patch". That's why I put it out there. I don't mind being refuted, but I wish it would be refuted rather than just downvoted blindly.
P.S. I think your quote doesn't capture the situation properly when someone is known to have malicious intent.
I don't think so - while the NSA would dearly like to have the access that this vulnerability would allow, they would dislike even more if anyone could have it. If they're going to insert a backdoor they're going to be damn sure only they have the key.
This thing has been in the wild for two years. What are the odds it hasn't been systematically abused? And what does this imply?
To me it sounds kind of like finding out the fence in your backyard was cut open two years ago. Except in this case the backyard is two thirds of the internet.
Worse, it's retroactively unfixable: Even doing all this [revoking certs, new secret keys, new certificates] will still leave any traffic intercepted by the attacker in the past still vulnerable to decryption.
So it would be a good idea to change all your passwords to critical services like email and banks, once they have issued new certs and updated their openssl.
I was thinking of the scenario of old traffic being recorded by someone. Unless they also extracted the session key at that time, that traffic should be secure if PFC was enabled even if someone where to extract the server key now.
Yes, but there are other ways to compromise TLS sessions. For example, if you're using session tickets, the ticket key could be in RAM. Or, the session master keys themselves could be leaked. Still, you're _much_ better off with Forward Secrecy -- in most cases keys ticket keys are rotated with server restarts; so are session master keys.
That's slightly misleading. Every private key disclosure leads to decryption of past traffic unless forward secrecy is used.
However, if you switch to a fixed version of OpenSSL now, then an attacker cannot retroactively exploit this bug even if they have recorded all your past traffic, because exploiting the bug requires a live connection.
(Of course, this only applies to attackers who did not know about the bug before it was publicly released, so some worry is still justified. I only wanted to point out that the "retroactively unfixable" is a misleading exaggeration.)
I think what was meant is that since exploiting this bug leaves no trace, you should automatically consider every master key ever loaded to a vulnerable OpenSSL application to be already compromised. As nothing says this is the first discovery of the bug, one should consider that the black hats have already been exploiting this for long before the first public disclosure.
What popular SSL client software uses the vulnerable OpenSSL? (Any web browsers, for example on popular linuxes? How about 'curl' when connecting to HTTPS sites?)
How would a client be compromised? I mean I guess a malicious server could send these bad heartbeat packets and sniff the keys, but if the server is pwned then your secrets are already revealed, right?
Imagine you've got a script that, among other things, does a 'wget' against some innocent plain HTTP URL. But an attacker intercepts your request, and redirects you to an HTTPS URL of their choosing.
Yes, wget uses OpenSSL, and follows redirects silently by default.
Now that server uses heartbleed to x-ray your client process memory, collecting all sorts of confidential information, including perhaps credentials to other services.
This bug has a lot of nasty, unintuitive permutations and repercussions that will take time to fully grasp.
Oh it's even worse, basically every secret you had in your server processes' RAM was potentially read in real-time by an attacker for the last 2 years.
It can only access memory of the process running openssl. So if you got nginx in front of your webprocesses they are protected. However anything in the nginx process is accessible (e.g certificates).
Yes, to be clear (esp. for others reading this thread) this is really bad, but shouldn't be able to compromise your ssh server keys.
However -- ssl certs and session keys are a likely target, and combined with passively logging traffic that is enough to compromise all data going over ssl, such as login/passwords and data.
Problem servers include not only web servers, but also imap/pop and smtp servers supporting tls (via openssl -- afaik gnutls isn't vulnerable to this bug).
I got a "security warning" update when I logged in to the server (good), ran apt-get and installed, did openssl version, got the string as noted above (which seemed just a tad out of date).
So... I built and installed from source, and got... the same string.
and got the same results as you. How can it be built on January 8th if the patch was just made today?
[EDIT] running
sudo aptitude upgrade
upgraded properly and now I'm getting a version that was compiled earlier today. I'm guessing I needed to update another package as well. Probably `libssl`?
upgrade will work because it updates libssl1.0.0 which is the package you want upgraded :)
Openssl is the command line package and libssl1.0.0 is the library. i was able to upgrade openssl without upgrading libssl1.0.0.
ben@ip-10-0-0-76:~$ dpkg -s libssl1.0.0 |grep Version
Version: 1.0.1e-3ubuntu1
ben@ip-10-0-0-76:~$ dpkg -s openssl |grep Version
Version: 1.0.1e-3ubuntu1
ben@ip-10-0-0-76:~$ sudo apt-get install openssl
...
ben@ip-10-0-0-76:~$ dpkg -s libssl1.0.0 |grep Version
Version: 1.0.1e-3ubuntu1
ben@ip-10-0-0-76:~$ dpkg -s openssl |grep Version
Version: 1.0.1e-3ubuntu1.2
ben@ip-10-0-0-76:~$ openssl version -a
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Jul 15 12:44:45 UTC 2013
platform: debian-amd64
options: bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_NO_TLS1_2_CLIENT -DOPENSSL_MAX_TLS1_2_CIPHER_LENGTH=50 -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/usr/lib/ssl"
ben@ip-10-0-0-76:~$ sudo apt-get install libssl1.0.0
ben@ip-10-0-0-76:~$ dpkg -s libssl1.0.0 |grep Version
Version: 1.0.1e-3ubuntu1.2
ben@ip-10-0-0-76:~$ openssl version -a
OpenSSL 1.0.1e 11 Feb 2013
built on: Mon Apr 7 20:33:19 UTC 2014
platform: debian-amd64
options: bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_NO_TLS1_2_CLIENT -DOPENSSL_MAX_TLS1_2_CIPHER_LENGTH=50 -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/usr/lib/ssl"
i wonder how many people will do apt-get update openssl and assume they have fixed it
Agree. This needs a big fat the world is coming to an end stlye of warning.
I've just shut down the webservers running SSL that I can control.
If you are vuln and don't want to build openssl from source and can afford the outage. I'd reccomend to do the same.
OTHERWISE BUILD FROM SOURCE IMMEDIATELY, PATCH, AND GET NEW KEYS!
Let's hope CA's don't get swamped by all the CSR's. Or rather let's hope they do so we see people are doing something...
For me right now these are just my hobby projects. So I don't care if they're down. But I imagine it will be fun tomorrow.
Ok, anyone could assist me on how to update openssl without breaking anything? I've fetched newest sources from openssl.org and compiled them, but "make install" doesn't actually install it, it only got compiled, but issuing "openssl version" still gives me the old version.
What I want to do is to patch it so our webserver uses new version.
I would tread lightly here if you aren't comfortable with compiling. Rather than break your website, it might be better to take it down until your distro's packages are available.
You should probably spend your time investigating a good method of reissuing keys for when you get to a stable OpenSSL version.
Some apps have OpenSSL statically compiled into the binaries. Beware that what you think is fixed may not be.
Depending on the distro on which you're based, you may find that making a new package from a source package (e.g. srpm) would be the safest route even if you're in a hurry.
If you're on Ubuntu, it would appear at least the updated base (OpenSSL itself) packages are now in the repos.
Not to sound like a commercial for Cloudflare or anything. But putting your infrastructure behind their services can protect users while they perform their patching. According to their latest blog post http://blog.cloudflare.com/staying-ahead-of-openssl-vulnerab...
[this command generates a private key and server cert and outputs to pem's]
[Note also the key sizes are 4096, you may want 2048. AND I use -sha256, as sha1 is considered too weak nowadays. These certs are valid for 3650 days...10 years]
Since the command overwrites certs/keys in the current directory of the same name as the outfiles...that's it...you're done. Just restart nginx.
If you change a self-signed cert, like above, expect a new warning from the client on the next connection...this is just your new cert being encountered. Click permantly accept..blah blah.
This doesn't sound like "responsible disclosure" to me - how can Codenomicon dump this news when all the major Linux vendors don't have patches ready to go ?
Well someone was able to give Cloudflare a heads up last week [1].
It would have been nice if the package maintainers could have had time to build ready-to-roll solutions with Heartbeat compiled out prior to the official OpenSSL fix.
546 comments
[ 4.2 ms ] story [ 216 ms ] threadWhile I'm not happy at having to spend my Monday patching a kajillion machines, I welcome more vulnerability writeups in this vein.
The big problem is that this is still a gigantic range of processes (and possible memory buffer contents). But SSH at least would appear to be fine, unless you've ever transferred an SSH key over TLS using OpenSSL.
When you re-key, it will automatically deactivate the previous cert and is free. It also gives you the opportunity to update to SHA-2 or increase the key to 2048 bit, which you should do unless you have unusual and extreme legacy support needs (and must keep SHA-1 a while longer).
If you want to be secure, make sure the certificate based on your old key is showing up in the certificate revocation list (CRL), and/or any online certificate status protocol (OCSP) servers it specifies.
http://i.imgur.com/ZwTclan.png
(What I want now is an exploit.c, PoC.py, pwnSSL.rb, etc... but I guess it would be irresponsible to provide that to the script-kiddies of the interwebz right now)
As an actionable bulletin, this page leaves a lot to be desired. Nice logo and domain name, though.
One (selfish) question I have is whether this can affect primary key material stored in an HSM. I'm assuming not, but that the session key generated by the HSM would still be susceptible.
However, everything else in that process (like, all the traffic you were hoping to protect) is basically toast.
I'm running Fedora 19 and Arch on my main dev machines/VMs and as of this posting are considered up-to-date. Both are vulnerable:
And the current beta of 14.04: OpenSSL 1.0.1f 6 Jan 2014
All my RHEL5 boxes are running 0.9.8, though, at least.
https://rhn.redhat.com/errata/RHSA-2014-0376.html
It would be unfortunate if a new package were to be released immediately only to be soon masked/recalled due to unforeseen consequences.
Of note, the Gentoo package was bumped approximately 2 hours after the advisory was published.
https://admin.fedoraproject.org/updates/openssl-1.0.1e-37.fc...
https://admin.fedoraproject.org/updates/openssl-1.0.1e-37.fc...
Edit: I've posted on the support forum, hopefully they get back to us https://forums.aws.amazon.com/thread.jspa?threadID=149690
"We can confirm that load balancers using Elastic Load Balancing SSL termination are vulnerable to the Heartbleed Bug (CVE-2014-0160) reported earlier today. We are currently working to mitigate the impact of this issue and will provide further updates."
http://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=...
http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=bd69...
I don't know C well - why write 19 like this?
int timeout = 1000 * 60 * 60 * 24;
(milliseconds in a second * 60 for minute, * 60 for hour, * 24 for hours in a day).
Much more obvious than just putting in 86400000, and the compiler will optimize away the math and putting the math in there explicitly is arguably better than a comment that could easily become unanchored from the real value (if someone changes the value and forgets to update the comment).
When it comes to byte-sizes of things, though, most code will use sizeof() to both make it more clear where these numbers are coming from and to make them automatically adjust if the structure sizes change (granted this is unlikely to happen on a mature protocol).
At the very least having them be preprocessor defines would certainly make things a lot more clear here, so even for C I'd consider this a bit of a "code smell" (though the people who work on this code regularly are probably versed enough in the ssl3 record structure enough that they immediately grok this when they see it).
root# USE='-tls-heartbeat' emerge openssl
Also, if you're using those keys to protect other secrets like passwords - say, DB credentials or AWS keys stored in an HTTP-hosted Git repo behind - you can't really assume those are safe either.
Fun times!
What's the quickest check to see if sshd, or any other listening process, is vulnerable?
(For example, if "lsof | grep ssl" only shows 0.9.8-ish version numbers, is that a good sign?)
the problem with the openssl library is in the implementation of the TLS protocol. this is not used by openssh. so openssh is not affected by this problem. even when certificates are used. and even when x509 certificates are used (which requires a separate patch).
First off, TLS is crypto bread-and-butter that's used for a lot more than HTTPS. You're not out of the woods because you're not running a webserver.
Second, SSH itself doesn't use TLS; it has its own protocol, so sshd isn't vulnerable.
But third, read overflows like this can be escalated in countless ways to total compromise if some credential, key, canary, or such gets leaked. So just because sshd isn't vulnerable doesn't mean you're not screwed.
$ apt-cache showpkg libssl1.0.0 => http://perl514.cpanel.net/libssl1.0.0.depends (186 deps)
The one thing that I can discern is that you printed a list of every package that depends on libssl1.0.0 for your configured repositories. But you have no idea if those programs make use of heartbeat. ssh (and everything related like libpam-ssh) is on that list and does not use TLS. The same can be said for many others such as tpm-tools, ntp/ntpdate/openntpd, xca and so on.
The big one that comes to mind aside from https is smtp/tls, e.g. port 587
Edit: Apparently a PoC on STARTTLS has already been written, so smtp/tls is definitely vulnerable
"Yes, we can sell you our software!"
It could have implications for Bitcoin web services that use HTTPS, of course.
But, with the "rpcssl=1" option, the reference Bitcoin client's RPC interface would be using SSL, and specifically OpenSSL.
I'd guess a ton of online Bitcoin services reliant on hot wallets do this. So cue the thefts, or 'thefts', any minute now.
For comparison, the Bitcoin RPC password timing bug - https://github.com/bitcoin/bitcoin/issues/2838 - would have been a more slower and blatant/detectable way to compromise the same sorts of bitcoin RPC daemons.
Yes, it requires extra settings in bitcoin.conf: to enable RPC, accept connections from non-local addresses, and use SSL. But it's only 5 lines if the host is not otherwise firewalled from the net:
(Actually, I'm not sure if rpcuser and rpcpassword absolutely must be set...)http://www.networksolutions.com/whois/results.jsp?domain=hea...
Got a long night ahead :/
libssl1.0.0 openssh-client openssh-server openssl ssh
Something like "lsof -n | grep ssl | grep DEL" can identify processes using the DELeted old version of libssl after apt-get upgrading.
By default it only prompts you for grave-serious bugs. I have been bitten a couple of times by "important" bugs so set listbugs up so that it also checks for "important" bugs. This makes it a tiny bit noisier but not enough to make me switch to the defaults. Changing the severities is easy:
EDIT: If you are using DigitalOcean, the update is not yet on their mirrors. Issue 'sudo sed -i "s/mirrors\.digitalocean/archive.ubuntu/g" /etc/apt/sources.list;sudo apt-get clean;sudo apt-get update;sudo apt-get upgrade' to get the patch. Check the comment by 0x0 above ( https://news.ycombinator.com/item?id=7549842 ) to find any services which need restarting.
Setting up libssl-doc (1.0.1-4ubuntu5.12) ... Setting up libssl-dev (1.0.1-4ubuntu5.12) ... Setting up openssl (1.0.1-4ubuntu5.12) ...
Here's the changelog: http://changelogs.ubuntu.com/changelogs/pool/main/o/openssl/...
It sounds like the heartbeat code is sending some data in the handshake. That data should be harmless (padding? zeroes?) but the bug results in reading off the end of an array and from whatever other data happens to be there. Someone sniffing the connection can then see those bytes fly by. If they happened to contain private info, game over.
Is that a correct read on the situation? If so, my followup questions are: 1) Why is there any extra data being sent at all beyond a simple command to "heartbeat"? 2) How much data is being leaked here and at what rate? Is it a byte every couple of hours, is it kilobytes per minute, or what?
I am particularly interested in #1, since that's the part I really don't get at the moment. I suspect the answer to #2 will be implied by the answer to #1.
Can attacker access only 64k of the memory? There is no total of 64 kilobytes limitation to the attack, that limit applies only to a single heartbeat. Attacker can either keep reconnecting or during an active TLS connection keep requesting arbitrary number of 64 kilobyte chunks of memory content until enough secrets are revealed.
...so I guess the answer to 2 is only limited by how frequently you can change the heartbeat settings, and how frequently OpenSSL will send a heartbeat packet.
>>> TLS heartbeat consists of a request packet including a payload; the other side reads and sends a response containing the same payload (plus some other padding).
So, what happens is that the payload comes in as a pointer and a size (up to 64kb). The server then prepares a response and copies the memory block [pointer, pointer+payloadSize] into the request.
The attack happens when the payload is smaller than the payload size passed in the request. This results in the response preparation dumping the memory block [pointer+realPayloadSize, pointer+payloadSize] into the response.
Any data in this block is now exposed to the callee; and could contain any data from the process.
This sounds risky to me. I'm afraid attackers would benefit more from this decision than coordinated do-gooders.
How's that for responsible disclosure?
Thank you for the information, though.
Screaming about the NSA every time a security bug comes up is not interesting, productive, insightful, or useful, please stop.
And even then, the NSA has had their fingers in enough places and lied about it enough times (infiltrating FOSS projects was explicitly one of their goals, IIRC) that the sane default position would be to assume shenanigans on their part unless proven otherwise.
The vetting process does absolutely nothing to prevent something like this from happening, especially since some very sneaky and pernicious bugs can be introduced in the guise of simple mistakes. It would be foolish to assume this isn't part of the standard playbook, and just as foolish to discount the possibility of maliciously introduced bugs just because the evidence doesn't immediately point to malicious intent - that is the nature of the attacker.
The alternative is remaining ignorant and vulnerable to the single most well funded and experienced adversary a crypto user will ever likely face.
Nonsense. Are you a child-molester? I'm just asking, not accusing you of anything.
"Never attribute to malice that which can be adequately explained by incompetence" -- slightly-butchered quote, from someone smarter than me.
P.S. I think your quote doesn't capture the situation properly when someone is known to have malicious intent.
To me it sounds kind of like finding out the fence in your backyard was cut open two years ago. Except in this case the backyard is two thirds of the internet.
So it would be a good idea to change all your passwords to critical services like email and banks, once they have issued new certs and updated their openssl.
That's slightly misleading. Every private key disclosure leads to decryption of past traffic unless forward secrecy is used.
However, if you switch to a fixed version of OpenSSL now, then an attacker cannot retroactively exploit this bug even if they have recorded all your past traffic, because exploiting the bug requires a live connection.
(Of course, this only applies to attackers who did not know about the bug before it was publicly released, so some worry is still justified. I only wanted to point out that the "retroactively unfixable" is a misleading exaggeration.)
To me, this implies that it's not too easy to exploit, or we would've seen it fixed much sooner.
Email clients may be more vulnerable — Thunderbird doesn't, Mail.app doesn't, but I'm unaware what most use.
$ openssl version -a OpenSSL 0.9.8y 5 Feb 2013
Yes, wget uses OpenSSL, and follows redirects silently by default.
Now that server uses heartbleed to x-ray your client process memory, collecting all sorts of confidential information, including perhaps credentials to other services.
This bug has a lot of nasty, unintuitive permutations and repercussions that will take time to fully grasp.
If i got that right ALL openssl private keys are now potentially compromised.
I hope vendors push fixes soon, and then I guess I'm busy for a few days regenerating private keys.
So it's bad, but it's not that bad unless something exposing this bug (webserver with ssl, vpn, or other service) runs as root?
However -- ssl certs and session keys are a likely target, and combined with passively logging traffic that is enough to compromise all data going over ssl, such as login/passwords and data.
Problem servers include not only web servers, but also imap/pop and smtp servers supporting tls (via openssl -- afaik gnutls isn't vulnerable to this bug).
Suggestion: big, bold TLDR ("The sky is falling. Check your OpenSSL version right now") with a link on what to do sorted by OS vendor.
Step 1: Here's a command to spit out your OpenSSL version. If it is the following string, go to step 2.
Step 2: Here's how to update your OpenSSL. Here are links to guides on reissuing keys.
Probably OK the whole remediation bit links to a wiki that gets updated as the various vendors push their patches.
I got a "security warning" update when I logged in to the server (good), ran apt-get and installed, did openssl version, got the string as noted above (which seemed just a tad out of date).
So... I built and installed from source, and got... the same string.
Annoying.
[EDIT] running
upgraded properly and now I'm getting a version that was compiled earlier today. I'm guessing I needed to update another package as well. Probably `libssl`?I'm guessing that tons of people will run into this. I bet a blog post would get you some traffic... :)
I've just shut down the webservers running SSL that I can control. If you are vuln and don't want to build openssl from source and can afford the outage. I'd reccomend to do the same.
OTHERWISE BUILD FROM SOURCE IMMEDIATELY, PATCH, AND GET NEW KEYS!
Let's hope CA's don't get swamped by all the CSR's. Or rather let's hope they do so we see people are doing something...
For me right now these are just my hobby projects. So I don't care if they're down. But I imagine it will be fun tomorrow.
And when it's fixed, get new keys.
Btw: I'm a dev. Not a sysadmin though :P
Edit: Debian is patched. I'm online again \o/
What I want to do is to patch it so our webserver uses new version.
You should probably spend your time investigating a good method of reissuing keys for when you get to a stable OpenSSL version.
Some apps have OpenSSL statically compiled into the binaries. Beware that what you think is fixed may not be.
If you're on Ubuntu, it would appear at least the updated base (OpenSSL itself) packages are now in the repos.
http://people.canonical.com/~ubuntu-security/cve/2014/CVE-20...
1. Open a terminal[cd into] /etc/path_to_ssl_certs_folder[per site].
Ex. /etc/ssl/nginx
2. Regen the certs [example nginx mail server]
openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:4096 -keyout mailkey.pem -out mailcert.pem
[this command generates a private key and server cert and outputs to pem's] [Note also the key sizes are 4096, you may want 2048. AND I use -sha256, as sha1 is considered too weak nowadays. These certs are valid for 3650 days...10 years]
Since the command overwrites certs/keys in the current directory of the same name as the outfiles...that's it...you're done. Just restart nginx.
If you change a self-signed cert, like above, expect a new warning from the client on the next connection...this is just your new cert being encountered. Click permantly accept..blah blah.
------------------------------------------------------------------------
On a Windows box:
1. open an admin cmd window and run 'mmc'.
2. Add a new snap-in for Certificates as local machine.
3. Find and 'Disable all purposes for this cert'.
4. Import your new certs from your 3rd party or that you rolled yourself from your enterprise CA.
5. Test new cert.
6. Delete old cert.
[If you run your own CA, you should already know what to do...]
It would have been nice if the package maintainers could have had time to build ready-to-roll solutions with Heartbeat compiled out prior to the official OpenSSL fix.
[1] http://blog.cloudflare.com/staying-ahead-of-openssl-vulnerab...