There are different kinds of virtualization. OS-level virtualization like Linux-VServer and FreeBSD jails have little to no overhead while being very secure.
It was a serious question, I didn't mean to sound snarky.
I'm asking if you think it'd be a better idea (from a security perspective) to keep customers on separate VMs. My understanding of this vulnerability is that it relies on several customers sharing a filesystem; if Heroku simply spawned a new instance (they're already running on EC2) for each customer, wouldn't this be mitigated?
Also, I was looking up info on Pivotal Tracker yesterday (after the announcement that they were going paid) and saw your comment.
It really depends. Like most pentesters, I'm looking at two factors when I consider a stack like this: first, how big/complex is the exposed attack surface of the stack, and second, how well tested is that attack surface.
For many applications it seems entirely possible to do better with a single multi-tenant app stack than you could with OS virtualization with the same service model. In particular, if, to get the same service model, you had to customize your OS virtualization stack, I'd tack sharply towards preferring a multi-tenant app stack.
I can easily get my head around auditing the Ruby interpreter and its deployment on a Unix host. I've done projects assessing custom virtualization solutions and they are gigantic and complex to test. I'm pretty sure interrupt timing isn't going to screw up Heroku, but that's a flavor of the kind of stuff you have to test with full virtualization.
Finally, I have to add: everything is going to have bugs. You can fully virtualize every customer and every app and you will still eventually be exposed to something bad. The measure of the team is how they handle the exposure, how they fix it, and what they learn from the experience to make the next one harder to find.
"if Heroku simply spawned a new instance (they're already running on EC2) for each customer, wouldn't this be mitigated?"
Yes, but that'd cost a ton more than what they're doing right now. Consider the number of apps that they are hosting at the free level right now. Offering private instances as an upsell might make some folks happy though.
I don't agree with your "very slow" sentiment -- can you provide a source that shows that linux-vservers are very slow compared to running on bare metal? To my knowledge, vservers have almost no overhead vs running directly on the machine.
Heroku needs to be able to spin up and spin down server processes ("dynos") very quickly. Their process right now, so far as anyone can tell, is basically to untar a "slug" and start a ruby process. Booting a VM would be a lot more cumbersome (and there might be technical issues doing it on top of EC2 instances which are already virtualized).
If you're using a fairly recent kernel, 2.6.26 supports linux containers natively. You can "boot" a VM in less than 200ms, then start whatever process you need to start. I think when people think of virtualized instances, they think that the VM boots all kinds of other unnecessary processes.
Given that one could compile custom kernels on EC2, I think it's quite possible to run LXC on top of amazon instances. When I run "top" on my vserver running on top of vmware fusion, running on my laptop, I really only see my sandboxed process and nothing more.
Also, I see that heroku keeps these "disks" in a squashfs filesystem file. It seems like you might gain better performance if you just had all the files on your filesystem, used vservers, then use something like vserver hashify to prevent file duplication (to reduce space usage).
Summary of the vuln, from the original post: Heroku was relying on unguessable filenames to keep processes associated from one app from reading files containing the config files and source code of another app. However, the random portion of those filenames was showing up in the output of 'ps' as run from (I think) 'heroku console'.
The upshot was that the filenames were easily guessable, and their contents (config files and source code of the app) easily accessible.
So, if you've had a heroku app running between December 28 and January 17, your config files are possibly compromised.
(The scary part, to me, is what a malicious party could have done with access to somebody else's database.yml, but I haven't seen any writeup specifically address that point.)
The filenames weren't so much easily guessable as a change Heroku made last month accidentally exposed files. Relying on unpredictable strings isn't a fundamentally bad architecture; it's not an exaggeration to say that the whole web depends on that design.
My recommendation to Heroku might start with auditing every Heroku tool to make sure it calls "setproctitle".
I meant to say that the filenames were easily guessable after the random strings were exposed by 'ps'.
Relying on unpredictable strings works fine, so long as you keep them away from prying eyes. But that requires attention to detail, and can get harder over time (witness, say, Firesheep, which made it a whole lot easier to hide "secret" login cookies than it used to be).
Can I just say that "secrets" sent over bare HTTP aren't secret, and that even before Firesheep, nobody really thought they were hard to find? I love what Firesheep did to popularize that problem, but it was an old, very widely-exploited problem.
The point is the type of security scenario being talked about here is wildly different than the one made simpler to exploit by Firesheep. This was not security through obscurity. If your servers are configured correctly, unguessible filenames are just fine from a security perspective, at least as secure as passwords.
26 comments
[ 3.4 ms ] story [ 63.7 ms ] threadBlog post: http://daverecycles.com/post/2858880862/heroku-hacked-dissec...
HN Discussion: http://news.ycombinator.com/item?id=2128175
Disclaimer: I know David E. Chen personally from college.
http://news.ycombinator.com/item?id=362082
Do you still think that's a good idea?
I'm asking if you think it'd be a better idea (from a security perspective) to keep customers on separate VMs. My understanding of this vulnerability is that it relies on several customers sharing a filesystem; if Heroku simply spawned a new instance (they're already running on EC2) for each customer, wouldn't this be mitigated?
Also, I was looking up info on Pivotal Tracker yesterday (after the announcement that they were going paid) and saw your comment.
For many applications it seems entirely possible to do better with a single multi-tenant app stack than you could with OS virtualization with the same service model. In particular, if, to get the same service model, you had to customize your OS virtualization stack, I'd tack sharply towards preferring a multi-tenant app stack.
I can easily get my head around auditing the Ruby interpreter and its deployment on a Unix host. I've done projects assessing custom virtualization solutions and they are gigantic and complex to test. I'm pretty sure interrupt timing isn't going to screw up Heroku, but that's a flavor of the kind of stuff you have to test with full virtualization.
Finally, I have to add: everything is going to have bugs. You can fully virtualize every customer and every app and you will still eventually be exposed to something bad. The measure of the team is how they handle the exposure, how they fix it, and what they learn from the experience to make the next one harder to find.
Yes, but that'd cost a ton more than what they're doing right now. Consider the number of apps that they are hosting at the free level right now. Offering private instances as an upsell might make some folks happy though.
Given that one could compile custom kernels on EC2, I think it's quite possible to run LXC on top of amazon instances. When I run "top" on my vserver running on top of vmware fusion, running on my laptop, I really only see my sandboxed process and nothing more.
Also, I see that heroku keeps these "disks" in a squashfs filesystem file. It seems like you might gain better performance if you just had all the files on your filesystem, used vservers, then use something like vserver hashify to prevent file duplication (to reduce space usage).
And I wonder how many of those databases remain vulnerable because of previously exposed credentials.
The upshot was that the filenames were easily guessable, and their contents (config files and source code of the app) easily accessible.
So, if you've had a heroku app running between December 28 and January 17, your config files are possibly compromised.
(The scary part, to me, is what a malicious party could have done with access to somebody else's database.yml, but I haven't seen any writeup specifically address that point.)
My recommendation to Heroku might start with auditing every Heroku tool to make sure it calls "setproctitle".
Relying on unpredictable strings works fine, so long as you keep them away from prying eyes. But that requires attention to detail, and can get harder over time (witness, say, Firesheep, which made it a whole lot easier to hide "secret" login cookies than it used to be).