How likely is it that this sort of a thing stopped being a technical item of discussion and turned into a political one by the security contacts at Facebook?
I'm always curious about what sort of internal pressures would lead people to take a well-reported bug that the author did not take malicious action on and blow it up to the point that the CSO is getting involved.
Only way I can see this happening would be finger pointing and finding others to blame. Eventually, the problem starts with a few people then becomes inter-team issue. Then higher ups start to get involved.
Precisely - at the very least, taking care before bandying around legal action or calling someone's boss. Even without context, its a pretty weird sequence of events.
Facebook's calling his employer could be slanderous, possibly even criminal harassment.
Between stories like this demonstrating companies' apparent lack of understanding of whitehat infosec, and Weev's incarceration demonstrating the American legal system's apparent lack of understanding of whitehat infosec, it's hard to believe people still participate in such endeavors.
I don't see anything in the description of that call that qualifies as either slander (which requires a false statement of fact) or harassment (which requires a pattern of repeated contact intended to cause emotional distress).
If bringing unrelated parties to dialogue in the background is not harrasment, then what is?
Imagine I will contact your significant other over your comment on HN. Would not you be deeply disturbed even if I do it just once? Bonus points for frivolous legal threats on my side.
"If bringing unrelated parties to dialogue in the background is not harrasment (sic), then what is?"
Exactly what parent said, " . . . a pattern of repeated contact intended to cause emotional distress". Yes, doing as you said would be deeply disturbing, but it wouldn't be harassment.
Nobody is arguing that this person shouldn't be disturbed. But that doesn't make it "criminal harassment".
Unfortunately --- and this is not a normative argument, so please don't wig out --- the criminal action here is the researcher's. Nobody's been prosecuted, but could they have been? YES.
What possible motivation did Facebook have for contacting the company with whom this person had a contract employee relationship with, other than to implicitly threaten problems for both? There was no implication that he was doing this other than on his own, and he had cleared it with his employer. Presumably he didn't email Facebook with a corporate email account, and presumably his employer wasn't in a position where Facebook's first assumption would be "corporate espionage! (which was voluntarily reported to a bug bounty program)" - that's disingenuous.
No, it was "We're bigger than you and we have the power to fuck with your life and livelihood", and nothing more.
To ensure that this person deleted the credentials they had taken from the server they popped with the RCE, obviously.
Again: read the timeline. He submitted a finding with AWS creds taken from the server he popped on October 22 --- on December 1, more than a month after Facebook shut the server down. He took AWS creds from a Facebook server and saved them on his laptop for more than a month.
Then threaten him with legal action (not that I necessarily condone this - I will say that I did like your breakdown down thread to providing another perspective and balance).
It's neither harassment or slander, but it could be tortious business interference, where one party induces a second party to break a contract with a third party.
His contract employers are neither his parents or legal guardians - who have no more power to "ensure" this as anyone else.
As a corollary to this - this is exactly why it's illegal for debt collectors to call your friends and family to "encourage" them, or you by humiliation, to pay up.
Do you see what a ridiculous no-win situation this is?
Option 1: Call the pentest firm he works for.
Option 2: Threaten legal action.
Option 3: This dude might still be walking around with god knows what shit he's pulled out of S3 buckets or lord knows what else was accessible with those AWS keys or what other keys were in other S3 buckets or
wow i'm having a small panic attack just trying to complete that sentence
I don't know whether they tried, or if they didn't, why they didn't. Like I wrote below[1], I'm wondering what the rest of the story is. This isn't the first person to have submitted an RCE to Facebook, but it's the first person to get Facebook to go nuclear over a submission. Why?
This isn't a single key. It's, like, maybe all the keys? The bad stuff that happened here all happened in a single day, the day that the researcher disclosed the AWS creds for the first time, more than a month after the server he dumped them from was shut down.
Surely a competent technology company would realize the using creds that were stored on a known-compromised server is bad and change them immediately, right?
Without having a position in this debate myself: I think that's not quite fair.
My understanding is this: They got a report that a server can be compromised and fixed that vulnerability. Unbeknownst the reporter grabbed a huge amount of (remote! not on that server, btw) data to play with.
Later the reporter returns to Facebook and says 'Btw, I got all these valuable pieces of information and have those for quite a while'.
Only at that point can you panic and rotate keys, but now you notice that a third party had access to all these keys for a month already. What else did they get? Maybe the researcher sat in a posh coffee place and grabbed interesting Instagram credentials (using the certificate) or escalated this further, gaining even more access based on the exposed information so far.
In my world, Facebook/Instagram are basically completely owned and have to assume that this guy grabbed ~everything~. They probably need to hire (vs. doing a bug bounty) people to grab the same data from the same buckets to look for potential follow-up targets that were _not_ disclosed, but might've fallen to the same bug hunter.
Who's to say that the guy doesn't come around on New Year's Eve with Yet Another Disclosure based on the same 'attack'?
I hate the 'contact the employer' part, but I'd hate to be in FB's shoes far more. I can hate the company and feel for its CSO/IT staff in this aftermath at the same time.
That's fair, but it implies that at no point did FB ask themselves "what would someone who exploited this vulnerability have access to?" If they had, they would have realized they were completely owned before the researcher pointed it out to them and taken steps to fix it (changing keys, etc.) At that point the researcher would be the least of their worries, and they would have tried to figure out if anyone else had completely owned them.
However, since they were unable to figure out that the friendly researcher owned them until he told them we now know that FB itself doesn't know who has their data.
Why does it matter if he deleted the credentials? As soon as anybody was able to get the credentials them, you'll have to make the assumption that others have been able to do the same.
And that means rotating all your credentials the very same day you learn about that happening. Why does it make a difference then if he still has stale credentials?
Pointing out that this doesn't meet the legal definition of slander or criminal harrassment doesn't mean sticking up for Facebook or defending its motivation.
I'm not sure whether I'm sticking up for Facebook. I'm not just lawyering this thread. I think, if I was in Alex's shoes, I might have done something similar. I'm very glad I didn't have to make that call myself, because, what a nightmare this is.
> Presumably he didn't email Facebook with a corporate email account
"At this point, it was reasonable to believe that Wes was operating on behalf of Synack. His account on our portal mentions Synack as his affiliation, he has interacted with us using a synack.com email address, and he has written blog posts that are used by Synack for marketing purposes."
> I never contacted Facebook or Alex using my work email account. It was only after Alex contacted my employer via email that I sent a reply from my work account. Alex indirectly contacted me at work, not the other way around.
Also remember that the story we have here is a one sided narration from a bug bounty researcher.
The story tells us his side of things but what specifically Facebook perceived as threat is still unknown ? Why would a CSO get involved unless they specifically think that the data has been accessed violating the goodwill of the bug bounty research in the first place.
That's true, there could be large portions of the story that are omitted or inaccurate. We may never even get the full story.
Assuming the story as stated is truthful or even plausible, what options do whitehat hackers have to defend themselves in such a scenario? I mean the whole point seems to be to try to penetrate a secure system, and the consequences of that action seems to be fairly obvious from the start. If a whitehat hacker is successful, that carries with it the inherent potential that they will have some sort of access to some sort of sensitive data, right?
Surely telling Facebook "I was able to access these exact things" means he expected Facebook to update passwords and change keys accordingly, making the possibility that he retained those keys moot.
He claims he had access to so many credentials. That's a P1 security protocol. You can't just let a manager to handle it. Your executive boss, CSO, has to step in.
> Why would a CSO get involved unless they specifically think that the data has been accessed...
The most likely reason I can think of is that he was getting some heat from some other C*O.
Edit: Now that Alex's side of the story has been released, his actions don't seem out of line (assuming it's reasonably accurate, and I have no reason to suspect it isn't).
While my explanation is still a valid answer, I agree with the parent. Sounds like he was just doing his job. Though it would be interesting to listen to the audio of the conference call....
So if I'm reading this correctly, this massively compromising attack was made possible by doing a little research? e.g. Knowing about one of the admin services used by Instagram, looking in that admin's public repo, and musing whether Instagram had bothered to change the secret key from the default entry in the repo?
We'll probably never see a post mortem on this but it'd be interesting to hear how this got moved to production...: was the Sensu admin panel a nice scaffold for internal use and by the time they decided to make it remote, everyone just assumed the secret key had been changed at some point?
I can tell you from experience working at another similar company that this is not surprising at all. Especially as startups transition into larger companies (with formal security controls and policies), a lot of things can get missed or forgotten. Your primary production servers may be completely up-to-date and secure, but somewhere along the way, there's a high chance that an engineer deployed an internal admin tool or a test build somewhere that ends up being public, but ultimately lost and forgotten. The problem is, that kind of "lost" infrastructure often contains keys, credentials, or network access to other more critical parts of the infrastructure, and no one realizes the severity of the mistake until it's too late.
Even if that is the case, why is it exposed to the public? They firewalled it off almost immediately, so I assume it didn't need to be...
Never expose anything that doesn't need to be, SSH tunnels, openvpn... heck, use HTTP authentication wrapped SSL tunnels if you have to. Web servers tend to be more secure than webapps.
Mistakes like this are far too easy to make. So anything that isn't part of your business application needs to be tunneled with authentication or isolated completely.
> With the RCE it was simple to read the configuration file to gain the credentials necessary for this database. I connected and dumped the contents of the users table.
This was his mistake. This is a huge no-no. You never dump data unless you have permission. It's against the terms of most bounty programs.
But like he said in the article, he was unable to find a clear policy that gave him the "Stop, no further" point. It may have been a bad assumption to think Facebook was going with the Tumblr stance of "give us a thorough POC," but where should he have drawn the line in his hack and why here instead of where he did?
Whereof one cannot speak, one should be silent. Dumping the user table is the literal next step in a standard vulnerability assessment (in order to acquire reused credentials), wasn't prohibited by the terms of FB's bug bounty program, and was crucial to the development of the bug.
No, that's the next step in an external penetration test, which is not the same thing as a vulnerability assessment.
In an external pentest, you get a set of netblocks and rules of engagement, and you get as far as you can. That's why it's called a "penetration test".
In a vulnerability assessment, you get a target (usually an application), and you find as many flaws in that target as you can.
Big annual pentests often have wide-open rules of engagements, where you (as a consultant) win big by, for instance, dumping the CEO's mail spool. But those projects also start with several meetings worth of negotiating rules of engagement.
Vulnerability assessments virtually never have those rules of engagement!
Nobody that I know of runs a bug bounty program on pentest norms. To do so would be grossly irresponsible, because on every network with more than 1000 hosts I've ever tested, ever, RCE behind the firewall is gameover for the whole test: you can get everything.
You're HN's anointed expert, so I suppose all I can say is that's not my experience.
Among the many reasons bug bounties are bad ideas is that they generally fail to write clear rules -- as Facebook did. As written, what he did is not against the rules and while it may fall into some best-practices bucket you assert to be universal, that's hardly sufficient for a field in which participants can come from any background. But please, continue to defend your friend whose multi-billion company had a month to cycle their popped keys and failed to do so, then responded by threatening a researcher's employment after multiple conciliatory e-mails.
The "then" isn't temporally proximal. The quoted e-mails (unless you feel like asserting that they're fake, which I think is the next step in your arguments in this thread) demonstrate that he's trying to work within the unwritten rules of the program and asking for clarification in good faith. Then after that, rather than attempting any communication with his, Stamos threatens his employment.
I agree with you that something seems off, but you're happily giving all the charity to FB and none to this guy, which is your prerogative but hardly makes for good conversation.
Not really, no. Your should not have had is still presupposing a set of bug-bounty-hunter-professional-guidelines that don't actually exist unless they're specified in the program guidelines, and from a philosophical perspective the actual security vulnerability under discussion now is that their sec team is so lackluster that they can't or won't change out a credential set known to have been externally accessible (and, the critical point, to anyone who could have found this not-particularly-obscure vuln, not just this researcher).
I agree. If there's no clear rule "all data stays in our network", dumping data is not an unreasonable move. I don't care whether some experts in their offices mull about what's alright to do in a pentest or when finding vulnerabilities for a bounty program - most people aren't experts in that sector, so better make it clear. The researcher is in the right here.
Not only is dumping data an unreasonable move, but it's one that will get you referred to prosecutors. That didn't happen here, but it just did happen somewhere else last week. Don't ever do that.
I wouldn't do that (I'd be scared to death about what would happen, even without reading this article). But I also don't find it an unreasonable move. Just make it clear - you dump data, we're going to sue you. Right now, the researcher is in the clear, even though what he did was incredibly stupid.
I don't understand why a company would ever say "you can snoop around in our stuff" without very clearly stating what they can do. You're leaving open a legal loophole where a blackhat can claim to be a whitehat.
> As described above, I used the web interface to gain code execution, but at this point I still hadn't actually gained access to the web interface as a normal user.
He had code execution, there was no need for him to go any further.
Yes. As much or more so than an end user table. You can't dump data and use dumped data acquired from a legitimate vulnerability to continue to gain access to additional resources.
Either way, it's awesome for the world. This kind of attack is great to tell people one more reason why they should not trust Facebook, WhatsApp, Instagram, etc. It'd only have been better if someone malicious had done it and made some data public (perhaps slightly redacted).
In particular, it might help with Signal vs WhatsApp.
1. Never trust Facebook. If he had signed some messages using their keys and sent it to Wired/NYTimes/etc he'd be getting fame and job offers instead of $1250 and call home to his parents.
Note to self: Don't report any chained attacks to any large companies bug bounty programs. Alex Stamos contacting the employer of the bug reporter is completely out of line.
This is the fastest and easiest way for Facebook to stop good submissions to their bug bounty program.
nope I was too. Interesting illustration (assuming it wasn't just a typo) that exploitation of vulnerabilities doesn't necessarily require deep understanding of the tech. stack in question.
And on the flip side, deep understanding of the technology stack in question doesn't necessarily lead to implementing it securely. This is division of labor at work.
In stories like this, try first to remember that Facebook isn't a single entity with a single set of opinions, but rather a huge collection of people who came to the company at different times and different points in their career.
Alex Stamos is a good person† who has been doing vulnerability research since the 1990s. He's built a reputation for understanding and defending vulnerability researchers. He hasn't been at Facebook long.
To that, add the fact that there's just no way that this is the first person to have reported an RCE to Facebook's bug bounty. Ask anyone who does this work professionally: every network has old crufty bug-ridden stuff laying around (that's why we freak out so much about stuff like the Rails XML/YAML bug, Heartbleed, and Shellshock!), and every large codebase has horrible flaws in it. When you run a bug bounty, people spot stuff like this.
So I'm left wondering what the other side of this story is.
Some of the facts that this person wrote up are suggestive of why Facebook's team may have been alarmed.
It seems like what could have happened here is:
1. This person finds RCE in a stale admin console (that is a legit and serious finding!). Being a professional pentester, their instinct is that having owned up a machine behind a firewall, there's probably a bonanza of stuff they now have access to. But the machine itself sure looks like an old deployment artifact, not a valuable asset Fb wants to protect.
2. Anticipating that Fb will pay hundreds and not thousands of dollars for a bug they will fix by simply nuking a machine they didn't know was exposed to begin with, the tester pivots from RCE to dumping files from the machine to see where they can go. Sure enough: it's a bonanza.
3. They report the RCE. Fb confirms receipt but doesn't respond right away.
4. A day later, they report a second "finding" that is the product of using the RCE they already reported to explore the system.
5. Fb nukes the server, confirms the RCE, pays out $2500 for it, declines to pay for the second finding, and asks the tester not to use RCEs to explore their systems.
6. More than a month after Facebook has nuked the server they found the RCE in, they report another finding based on AWS keys they took from the server.
So Facebook has a bug bounty participant who has gained access to AWS keys by pivoting from a Rails RCE on a server, and who apparently has retained those keys and is using them to explore Instagram's AWS environment.
So, some thoughts:
A. It sucks that Facebook had a machine deployed that had AWS credentials on it that led to the keys to the Instagram kingdom. Nobody is going to argue that, though again: every network sucks in similar ways. Sorry.
B. If I was in Alex's shoes I would flip the fuck out about some bug bounty participant walking around with a laptop that had access to lord knows how many different AWS resources inside of Instagram. Alex is a smart guy with an absurdly smart team and I assume the AWS resources have been rekeyed by now, but still, how sure were they of that on December 1?
C. Don't ever do anything like what this person did when you test machines you don't own. You could get fired for doing that working at a pentest firm even when you're being paid by a client to look for vulnerabilities! If you have to ask whether you're allowed to pivot, don't do it until the target says it's OK. Pivoting like this is a bright line between security testing and hacking.
This seems like a genuinely shitty situation for everyone involved. It's a reason why I would be extremely hesitant to ever stand up a bug bounty program at a company I worked for, and a reason why I'm impressed by big companies that have the guts to run bounty programs at all.
† (and, to be clear, a friend, though a pretty distant one; I am biased here.)
a private key. It's not uncommon to have multiple simultaneously-valid certificates for the same domain. I'd argue that it's actually sort of irresponsible and therefore surprising for a site at the scale of Instagram not to, for backup purposes.
but using that private key can still grant him access to someone's traffic to their machines. isn't revocation necessary to imply security in that domain ever again?
> † (and, to be clear, a friend, though a pretty distant one; I am biased here.)
Alex is good friend of mine and I've known him since college. He's definitely a good guy and understands the ins and outs of security vulnerability research, having done it himself for many years. I'm sure he didn't take the action of calling the researcher's employer lightly, and probably had a really good reason to do so.
There has to be a side of this story we aren't hearing, and probably never will.
> I'm sure he didn't take the action of calling the researcher's employer lightly
He's the CSO, and this occurred under his watch. The exploit was 2 years old, and well known. It highlights an internal security problem at Facebook et al, of-which Alex sits at the top.
In this situation, his years of "doing it himself" is unlikely to have factored in - rather, he felt like he dropped the ball and could be facing some consequences, or at the very least felt embarrassment.
This would have led to a rash thought process, and perhaps Alex jumped to the conclusion of some sort of sabotage by another company.
> Wes counters that, "[Alex] never for a second believed I was operating on behalf of Synack"
I'm not sure how Wes knows what is going through the mind of Alex
As blazespin[1] mentioned in this thread, Facebook's own terms states that they only pay individuals. That's how Wes knows - because Facebook's bounty program never deals with companies. The only other explanation would be Alex is ill-informed about the terms of Facebook's bounty program.
Coming from a pentesting background (and now working as a CISO), I can see both sides to this. tptacek is almost certainly correct in his characterization of the events, and I agree wholeheartedly with what he's said. It's important to note that this researcher didn't just chain several exploits together, but sat on sensitive data unbeknownst to Facebook in order to exploit other vulnerabilities later. Those vulnerabilities could not have been exploited without the initial (fixed) compromise.
Think about it a different way. If this researcher had found SQL injection in a webapp, dumped the usernames and passwords, and reported the vulnerability for a bug bounty, he should get paid. If he kept each of those credentials, and then logged into other systems using higher-privilege accounts that he'd compromised even after the SQLi is fixed, he is basically continuing the exploitation of an already-fixed bug. Those don't deserve payouts. Similarly, if he'd established some sort of persistence (such as a reverse shell, etc) on compromised assets, he can't keep coming in and getting more and more bounty payoffs. Fruit of the forbidden tree, in this case.
Where I disagree with tptacek is with regard to the benefit of bug bounty programs. Although I'm not currently running one, I find the idea fascinating and helpful for two primary reasons: first, you're almost definitely going to see generally better results in a well-managed bug bounty program (not necessarily something like Facebook's White Hat program) than traditional pentests or application security assessments. More eyes are almost always better when searching for tricky problems. Secondly, if you're a large enterprise, there are already people "testing" your security. I'd much rather be able to pay out a researcher than drive them to more nefarious buyers. You will probably encourage many people to test your security (which screws up metrics) but if finding security problems is the ultimate goal, it's worth it.
Even in this case in point, Facebook did discover an RCE that could have been (and kind of was) extensively exploited due to the fact that they held the bounty. If an actual malicious hacker had found that problem first, they would have been in significantly worse shape.
Whilst I'd agree that bug bounty programmes can be a good idea for Internet facing assets, I thought that this story actually neatly illustrated their limitations.
With a bug bounty programme you don't generally authorise the kind of post-exploitation activities which we see here as leading to the really serious exposures, and that's not surprising as you can't easily authorise a set of unknown people to be processing your customer data.
This differs from an engaged penetration testing firm, with whom you have a contract which covers things like handling of data gained during a test.
So I don't really see bug bounties ever replacing penetration testing companies for internal work or anything that requires accessing customer data as part of the exploit...
> If he kept each of those credentials, and then logged into other systems using higher-privilege accounts that he'd compromised even after the SQLi is fixed, he is basically continuing the exploitation of an already-fixed bug.
Why did those credentials still work post-report?
What if those credentials were accessed from a public dump?
The outcome of this entire clusterfuck of a bounty is one of the reasons there are still very well paid blackhats. There are no rules or terms to follow.
If their terms aren't clear (the terms he's citing certainly weren't intended for keys, rather Facebook user accounts/information), pay out and fix them.
Thanks for the writeup. Based on what you've written, it sounds like you would have been surprised if Facebook had paid $1 million for the original report (and no further nefarious behavior by OP) since it was probably due to a simple oversight, even though it was a RCE that obviously could have been turned into total ownage of instagram. Is that accurate? If so, what class of vulnerability would make you say "Yep that's totally worth $1 million".
Or do you think he should have just stopped and Facebook should have realized how bad it was and paid him a lot more than $2500?
There isn't a parallel universe in which this finding is worth $1,000,000. It it was, every pentester in the country is getting way underpaid, because this is not an uncommon pentest finding.
Makes sense, I'm just trying to get a sense of what sort of thing would be worth that much. Obviously only Facebook can answer that for sure. Heartbleed?
It's really dependent on the company. Ruby RCE would have the same affect heartbleed would to an entirely Ruby stack company.
I don't believe any company would pay $1M for a bounty on their own systems. Only people who intend to use the vuln, or to fix it as they are the vendor.
Fr a vuln to go for $1M requires "discovering SQL injection"-levels of vuln. MS paid $100K for an entire vuln class for ASLR/DEP bypass discovery, and promptly patched the shit out of it. For a remote vuln class, I could see them paying $1M quite happily to not have all of their products re-owned.
What about the parallel universe in which bug bounty hunters are blackhats who directly profit from the exploit? It seems like someone with that level of access could run up, among other things, a decent AWS bill.
> It it was, every pentester in the country is getting way underpaid, because this is not an uncommon pentest finding.
No wonder there's a flourishing (and well paying) blackmarket for vulnerabilities. I wonder how much this keys-to-the-kingdom vuln would be worth (Mitm Instagram, bootstrap a botnet, steal celebrity pics, ... the possibilities are endless)
I think you're right on most points, but after reading the write up and response I do think Alex reached out to the employer first instead of the researcher as an intended act of intimidation. That was a mistake.
If it was not done for the purpose of intimidation, then Alex simply would have asked the CEO if the researcher was acting on the company's behalf and after hearing "no" would have ended the call and contacted the researcher directly.
Seems simple doesn't it? Perhaps you are not seeing it due to your friendship, but it seems like a dirty move and only serves to call into question how Alex handled other aspects of the situation.
> If it was not done for the purpose of intimidation, then Alex simply would have asked the CEO if the researcher was acting on the company's behalf and after hearing "no" would have ended the call and contacted the researcher directly.
Then the CEO is going to contact the researcher and he's screwed either way. God knows what the CEO would have say to the researcher privately. Having a middle man to translate is a bad idea in an emergency.
Let's face it, when you used your work email and made another company paranoid, you are putting people on the spot. Employer needs to know (they have legal responsibility), and given the prior research they did and the researcher's claim, I think the reach out is absolutely correct.
Instgram's infrastructure has flaw. That's bad but everyone's infrastructure has flaw. Shit has to be fixed. Doing more than what was needed is bad. If I am told to stop dumping data, I would stop.
If there's a grey area in your ToS, and a security researcher/hacker type is in the middle of it - the smart route is to appease them and fix the grey area. FB has a lot of resources, and it wouldn't have to deal with the blowback from this.
Why make such a bad situation worse, if you don't have to?
FB messed up. The researcher partly messed up too. Fix it and move on.
every network has old crufty bug-ridden stuff laying around
"stuff" was the keys to the kingdom, do you think this is acceptable for a company like facebook? So instead of them making an apology, the CSO is trashing the guy who gave them the wake up call?
If you're biased, you should do the ethical thing and stay out of it, honestly. There is a ton of asymmetry here, and you and your Facebook CSO friend are being bullies. This is pretty grey, you don't have first hand knowledge, and obviously Alex can do no wrong in your eyes.
Your characterization of the AWS keys being sat on for over a month does make sense, now that you frame it in that light.
That said, Alex Stamos and the rest of the security team should have tried to figure out what vulnerabilities existed from this server instead of just nuking it and thinking that the problem was solved. That was lazy and stupid.
> 5. Fb nukes the server, confirms the RCE, pays out $2500 for it, declines to pay for the second finding, and asks the tester not to use RCEs to explore their systems.
The issue here is that, in hindsight, FB failed at this step.
They nuked the server, but they didn't determine what sensitive information was available on that server, and take steps to mitigate those risks.
I think that's an understandable mistake - cleaning up after a server intrusion is hard. Knowing how much to do after a possible intrusion is even harder. But it is still a mistake and it happened on Alex's watch.
If the purpose of the bounty program is to find out about your security mistakes, then the program did its job here, and Alex should be pleased that the problem was reported so that they could fix it.
That the researcher found the mistake by overstepping what is considered ethical (and I have no doubt that they did overstep) creates a very difficult situation - you don't want to reward that behaviour, but you do want to know about security problems and this one was only discovered/reported because of that bad behaviour.
In that difficult situation it is all the more important to tread carefully. The easy cases where you're paying out a $10k bounty typically don't require much finesse. It's the tricky cases where you need to make sure your actions are well considered and above-board at every step.
From Alex's own summary it's evident that he didn't handle it as well as he could have.
Two of the longest paragraphs in Alex's write up cover what he said to the CEO of Synack, even though Synack had nothing to do with this. Even if we accept that Alex thought it likely the Wes was acting on behalf of Synack (personally, I don't think that was a reasonable conclusion to draw, thought I assume Alex is sincere in his view that it was), he should have determined that up front, and then, once he knew it was not work related, he should have avoided:
- making accusations about Wes's ethics to his boss ("Wes ... had acted unethically")
- suggesting that his external behaviour has implications for his employment ("Wes's behavior reflected poorly ... on Synack")
- bringing in the threat of lawyers ("keep this out of the hands of the lawyers")
When faced with the difficult situation of legitimate security research that has (well) overstepped the ethical boundaries, all the evidence is that Alex jumped to the position of protect yourself, protect the company, intimidate and control the researcher and though that is a common and understandable reaction, it's not the way you turn a bad situation like this into a good one.
There is no reason for the researcher not to retain those keys, IMO - Once those keys were found to be compromised by the company, they should have been revoked immediately, and considered 'in the wild'. The fact that they didn't revoke these keys is basically a security violation itself.
Dumping the users table on an 'internal' (heh) dashboard -- any company that is doing these bounty programs needs to clarify what a 'user' is. Is it someone using their application, or all employee information as well. It's an important distinction.
If the researcher didn't try to find what he could do with those AWS keys, they would likely be still valid. It's conceivable that some other people have found them too and did the same the researcher did, only kept everything to themselves. Thus, if the researcher didn't do the thing you consider bad, users of instagram would currently be more vulnerable. Why is then the thing the researcher did bad?
Not only did this person make several large and irresponsible mistakes in the process of uncovering and reporting the bug (dumping tons of private user information without permission, going far beyond simply discovering and reporting the bug, etc.), but they also keep referring to Ruby ("running Ruby 3.x, which is susceptible to code execution via the Ruby session cookie") as the vulnerable piece, when in reality, it's the version of Rails that had the vulnerability.
Sort of an interesting conflict these bug bounties create. You have someone who wants to hack as deeply as possible to have a bigger bug bounty based on stated rules, but at the same time they will invalidate your bounty if they arbitrarily determine it as too much?
I imagine the initial report by his friend that the server was accessibly would not be a very high paying bounty compared to one accessing the server. But how deep is too deep?
Right? If he left it at the RCE he would have gotten the $2,500 split between him and his friend... but he continued and was able to get access to all the S3 buckets which you would assume would warrant a much higher payout. Instead he got a huge amount of backlash.
Right, this feels like a way for Facebook to simply not payout a bigger bounty after they realized how big an appropriate bounty would be.
If the author submitted the RCE, and nothing else: is someone at Facebook actually going in and trying to simulate what he actually did? Who knows, because the process is pretty opaque. If you argue with Facebook's assessment, and go and further exploit the system to say "no, this is actually how bad the RCE is, in the grand scheme", you've now actually gone and proved what can be done, against their guidelines, which potentially disqualifies your initial discovery altogether.
Exactly how I see it. People want a higher bounty, and are also curious of any more bugs deeper. But companies want them to stop at the first layer.
It seems too difficult to define how deep is too deep, especially since at least he reported him doing it. He didn't decide to go that deep and then just report the RCE and collect $10 million from people far more interested in this.
Not only that, but dangling the $1 million bounty means they are encouraging the bounty hunters to try to make it larger. And ultimately it also leaves them in a position to find out how big it is (for whatever negotiations) and prove it to the company (in order to make an argument to its magnitude).
Once again we see how people act hard-ass in sight of gaping vulnerability in their system. Be it law system, computer system or moral system, you will see denial and intimidation.
We should have "pastebin hat" list and Facebook should definitely be on it.
The problem with humans is that they will rather go extinct over such things than behave properly. You could try to teach us by painful example but death will probably come first.
The thing that gets to me is the lack of gratitude on Facebook's end. Instead, they turn him into the villain for breaking imaginary rules. What would have been the harm in slapping him on the wrist and giving him some sort of reward for exposing a huge vulnerability? Instead, they eat the reward and shit on the guy who produced it. Real classy FB.
Yea I did and I realize he got paid out a little, but it was short of the $1 million.
I realize a million is a bit unrealistic, but if you're going to make a public statement, at least back it up or prove to the guy why his findings don't constitute a "million-dollar bug". It's not right to just cold-shoulder the guy and hide behind vague rules that were never clearly outlined. In fact, you might even conclude Facebook brought his behavior on themselves by making such a statement as "if a million-dollar bug is found, we'll pay it out." $2500 is nothing when you're thinking $1,000,000
Nobody is going to pay you a million dollars in 2015 for the 2013 Rails YAML bug in a stale server. Nobody is going to pay you a million dollars for a reliable Firefox RCE, and those take months to prove out and develop, and there's a liquid market for them.
But that's not going to stop Facebook from publicizing that they will. You're glossing over the details and attributing an aire of "old news" to the bug. Well, yes / no. If he didn't find such an ancient bug but instead someone devious did, they could have dumped all the private user photos. If that happened, what do you think the financial implications might have been?
How does that matter in any way? This was a series of fuck-ups. Facebook wouldn't pay $1M to anyone, ever, since it would encourage this kind of behaviour. It was the "zero-dollar bug that lead to the million-dollar fuck-up" though.
I think the point though, is that it's more than just a single old Rails YAML bug. The privilege escalation shouldn't have been there. Their infrastructure would still be vulnerable even without the initial exploit.
How much do you suppose blackhats would pay for instagram's ssl keys, mobile app signing keys, push notification keys, etc?
Yeah, the researcher went deep into the grey area, but I find Alex Stamos's reaction barely short of unbelievable - it's almost as though he's so new to the internet he's never heard of the Streisand Effect... (Either that, or he's just so accustomed to bullying and intimidating people who might embarrass him that he's now got that corrupt politician "Waddaya mean I'm 'abusing my power'? We grant multimillion dollar contracts to old school buddies all the time? What's the problem?" look on his face.)
I was thinking more of the Zerodium/Gleg/BoozAllenHamilton class of buyers - who'd on-sell it to, say, the Egyptian or Thai Government, rather than run-of-the-mill carders or identity thieves.
(But yeah, I'm perfectly happy with my life where I have no real understanding of how the black market for this kind of thing works...)
If companies are going to keep trying to get out of paying bounties for insane vulnerabilities like this, white hat researchers will just move onto something else, leaving the bounties to be paid out by the black market. Bounties aside, contacting his employer is a disgusting move.
He claims to have downloaded the content listed below. And he is surprised that Facebook responds coldly? Note the string "private keys" in this list... Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?
On the other hand, it does sort of re-enforce the idea that he should be paid handsomely, doesn't it? :)
* Static content for Instagram.com websites. Write access was not tested, but seemed likely.
* Source code for fairly recent versions of the Instagram server backend, covering all API endpoints, some image processing libraries, etc.
* SSL certificates and private keys, including both instagram.com and *.instagram.com
* Secret keys used to sign authentication cookies for Instagram
* OAuth and other Instagram API keys
* Email server credentials
* iOS and Android app signing keys
* iOS Push Notifications keys
* Twitter API keys
* Facebook API keys
* Flickr API keys
* Tumblr API keys
* Foursquare API keys
* Recaptcha key-pair
That's the Catch-22 of whitehat, isn't it? The whole idea is to find breaches and report them, but any worthwhile breach is going to expose sensitive data. How could you possibly know you've found a hole unless you've peered through it?
Is this not the point of a Whitehat bounty program? To entice someone to discover and disclose a bug in a trustworthy manner?
If they react this way, and can't trust people to attempt to find exploitable security holes on their system (even those that yield private keys), then what is the point at all? The only people that find them then, are not going to be as cooperative about it.
> Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?
This is not the author's fault. He did nothing but disclose bugs that Facebook themselves set in place, and seemed to be very open with them about it, at that.
What is the protocol for assuming that a bug might have previously been exploited and keys already compromised? Is that just not worried about unless they see evidence in logs?
I don't know much about this which is why I asked.
It seems that severity-based payouts have created incentives that do not match the program rules? Maybe all rce bugs should be paid out on an assumption that if used they'll lead to access to a shell or to user data.
Yeah - but it's 100% clear from this that FB wanted to brush the RCE under the carpet with a "not at all severe $2500" classification - without ever admitting to losing their private ssl keys or auth token seeds.
He clearly _did_ have a "security vulnerability" that gave him the keys to the kingdom. He knew it, and Facebook know it - and they wanted to pretend it was no big deal.
Any bets on how many months till there's a large-scale breach of Facebook user data? The reality of the balance between responsible disclosure and selling an exploit is much easier to evaluate now.
Especially considering Alex Stamos apparently requested reassurance that he _hadn't_ accessed particular classes of data - instead of looking in their own presumably non-existent audit logging of people who've had access to the private keys ssl of instgram.com and *.instagram.com!!!
(Seriously??? That's some world-class enterprise-grade "moving fast and breaking things"...)
Why wouldn't it be considered a bug that accessing one low-permission S3 bucket allowed him to access all the other buckets, including user data and keys?
Both tptacek here and Facebook claim that he found one bug. He found at least two, depending on how you classify things: even if Facebook would not like to admit that their security architecture around token amanagement was/is deficient, and the fuzziness of internal security boundaries makes "bug" somewhat hard to define, it was deficient by industry standards (especially for such a large and tech-focused company), and he got way more access than that RCE should have given him. Whether or not he was supposed to go looking for such additional bug(s), it's discourteous not to at least acknowledge that he found them, and thereby provided Facebook additional value over just finding the RCE.
If he had told Facebook that at the same time as he reported the credentials he harvested from the database --- which his timeline suggests he could have --- I'd agree with you.
But he didn't. He put the credentials in his back pocket so he could pull them out when they suggested he hadn't found his "million dollar bug". And so for a month after they fixed the bug, some fucking rando is walking around with credentials to all of Instagram's AWS assets, totally unbeknownst to anyone at Facebook. They turn down his bid for his "million dollars", and he busts the credentials out on them. You think they're going to thank him?
He's lucky it was Stamos and not Mary Ann Davidson.
I think the point is that, after the first bug report those credentials SHOULD NOT WORK because their job should have included revoking ANYTHING that system have access to. How did they know Wes was the first person to find that bug and the linked credentials?
So, the fact that those credentials still worked a month later is a HUGE FUCKING DEAL! Alex, the consummate professional, didn't do his job and instead had a knee jerk reaction to someone slapping that fact in his face.
It has been incredibly interesting reading through those threads. People are arguing two completely different arguments. tptacek is saying that the dude keeping AWS keys without disclosing this was bad and guy is lucky to not get a early morning wake-up call from men with guns. slewis, comex et al are saying that Facebook not locking down and later disabling AWS keys was bad and Facebook was lucky they didn't get sold on black market. Both sides are correct but it's informative who makes which arguments.
Bug bounty appears to be a misnomer in this instance. Facebook is specifically asking for reports of security vulnerabilities in their policy:
> If you believe you have found a security vulnerability on Facebook, we encourage you to let us know right away.[1]
Which then begs the question to me: how do you differentiate an acceptable and unacceptable probing of security vulnerabilities when you can't capture the full impact of an issue without attempting to exploit it to its fullest? Because it is certainly not outlined in their policy.
And when you're asking for any whitehat to attempt to discover and disclose security vulnerabilities in your system with only the limpest of guidelines around how to do so, I don't feel that it is warranted to react such as Facebook has here.
I don't know. I feel bad for Alex but if we want to suggest that Facebook's vulnerability disclosure policy was poorly written, I will ruefully agree.
When you stand up a bug bounty program, you are giving strangers permission to do something that they would otherwise be prosecuted for doing. You should be extraordinarily careful when you do that, and your rules of engagement should be crystal clear. These weren't.
EDIT: Having read the CSO's explanation that the guy was using his company work email, it makes more sense why the CSO would contact the company (and explains away the pettiness my comment was referring to)
One thing I notice: if the CSO felt like this person did something grossly illegal and irresponsible, why not go straight to the police? Why instead go to the man's employer and speak passively aggressively?
Paradoxically, contacting the authorities could have helped facebook's argument. It would communicated to the community at large: "Hey Facebook believes it has clear standing to pursue this guy. Maybe, he really did do something wrong."
Instead, what I'm reading is: "Facebook doesn't actually believe what the guy did was illegal per se... but they wanted to spite the guy anyway."
Zero is the number of people on HN who would feel better about this situation if Alex Stamos had referred this person to the police to be prosecuted under CFAA.
The researcher has already updated his post regarding the use of his company email. Apparently your original point still stands:
> I never contacted Facebook or Alex using my work email account. It was only after Alex contacted my employer via email that I sent a reply from my work account. Alex indirectly contacted me at work, not the other way around.
Also, why would he be doing this work at the behest of his employer when (IIRC) Facebook's bounty program only pays out to individuals? It would automatically make him ineligible to claim the bounty.
To me it seems like Alex Stamos tried to use some good old threaten-your-livelihood intimidation tactics and failed miserably.
I commented earlier to sort of the same effect, and was thinking a little more about this.
I don't think the goal, or desire, is to be told the full extent or impact of a problem. The goal is to be alerted to spots that may lead to a large problem, or re in and of themselves a large problem.
This seems like it has a few facets to it. You end up reducing the space of things to mostly "ways to get in the front door." Thinking about it, I would probably be frustrated, in general, if I knew someone had important keys to the kingdom I was in charge of. It doesn't change the fact that others may or may not have also gotten the same access, now it's 1-* instead of 0-* people who have it and shouldn't.
I'm still slightly skeptical on the bounty reward itself. This was a simple exploit that got pivoted into some major shit, so do you reward the exploit of the logical conclusion of the exploit? I lean towards the latter, but again, as you said... how do you figure out the impact without... actually trying to figure out the impact?
Bug bounties are an interesting concept, to be sure.
-
I'll rephrase the question. Is the broader vulnerability apparent based on the first discovery OR does it only become clear the further down the rabbit hole you get?
I don't know. If we're going to speculate, I'll say: the Facebook security team didn't know this system existed (it's a 3rd party admin console on a public IP address!), and their immediate reaction to it was "nuke it from orbit, pay out the bounty for finding it, and forget about it".
My guess is that they discovered the AWS credential thing on December 1.
If they discovered the AWS credential thing on December 1 after the security researcher reported it, and wouldn't have discovered it otherwise, and it could be the case that someone else found the exact same attack path first, shouldn't they reward him for making them aware of a problem they would not have otherwise noticed? That they wouldn't have fixed? That others that discovered the same attack path might otherwise still openly exploit to MITM all the traffic, to do arbitrary things with arbitrary user accounts?
You don't know that's what happened, even the researcher didn't say that. You're extrapolating.
A much more reasonable and likely explanation of the same set of things we've been told:
Alex Stamos called Synack and said that the AWS credentials, which, by the researchers own admission, he'd chosen to retain long after the vulnerability he reported was fixed, had to be deleted, and that if they weren't and the researcher continued to use them, the situation would be out of Stamos' hands and into Facebook legal's, at which point he couldn't keep him from being prosecuted.
In that interpretation, Alex isn't threatening the researcher; he's (very reasonably) saying "you cannot use these credentials you've taken from the server, and if you keep doing that, I can't take responsibility for how Facebook will handle this, so you should stop right away before you harm yourself."
it's utterly trivial to revoke and reissue aws access keys. trying to paint this as a necessary security measure is incredibly dishonest. the only plausible reasons to loop in his employer and mention legal remedies are intimidation and incompetence and as you've assured us incompetence is off the table...
blazespin > > But threatening to call the cops was really bad.
tptacek > You don't know that's what happened, even the researcher didn't say that. You're extrapolating.
From Wes' blog (presumably based on his boss' oral description of the call): "Alex then stated that he did not want to have to get Facebook's legal team involved, but that he wasn't sure if this was something he needed to go to law enforcement over."
Your bias is apparently badly incapacitating your reading comprehension, because "stated that [...] he wasn't sure if this was something he needed to go to law enforcement over" is exactly threatening to call the cops. Not even your friend mr Stamos, who has presumably read Wes' blog post, is claiming that he didn't. So whom are you saying is lying; Wes, or his boss?
Oh, and "(very reasonably) saying 'you cannot use these credentials you've taken from the server, and if you keep doing that, I can't take responsibility for how Facebook will handle this, so you should stop right away before you harm yourself.'" really, really, really sounds like Vito The Baseball Bat "very reasonably" saying "You cannot use this testimony you got off Loanshark Louie, and if you keep doing that, I can't take responsibility for how the boys will handle this, so you should stop right away before you harm yourself."
Seeing that as SERIOUSLY (as opposed to sarcastically) "very reasonable"... Well, hello, friendship-bias Bizarro World.
Holy christ that is SO wrong. The system should not be so easy to pivot in that way. That was definitely the real bug. If getting the keys to the kingdom is easy as exploiting a trivial bug than Instagram is really really screwed.
As I'm sure it's not the only trivial bug!
Instagram should be thanking Wes for the wakeup call instead of making him the enemy.
Facebook's point is that he found a vulnerability, and exploited it instead of stopping there. I kind of understand their point of view though. "See you have a vulnerability there, and then I can get access to this, and then this, and see now I have the password of your user, and then I'm just one click away from accessing all the instagram pictures I want."
Although Facebook's handling of the problem is poor (why didn't the CSO call the author directly to get things squared out? He does not talk to people who are not C*O?), they do have a point.
I think the author acted in good faith, but got carried away by his findings unfortunately.
"See you have a vulnerability there, and then I can get access to this, and then this, and see now I have the password of your user"
However, these weak passwords could have been exploited separately as part of an attack. It is fair to call it a new vulnerability, even though it was discovered by exploiting the first vulnerability.
Exploiting the bug would have been downloading the actual contents of the S3 bucket (the instagram source and other things). He specifically says he did not do that.
He clearly made a big effort not to violate privacy. The problem is that he made their security look like a joke by getting the keys to the kingdom without anyone noticing. Did that big expensive IDS catch him? Nope. Did any of the log watchers babysitting the AWS logs? Nope. One researcher made the CSO look incompetent in the matter of minutes.
If he had found a bug with something a developer wrote that would be a different story. What he found was layer after layer of Operations (particularly Security Operations) failures. This is something you hire a CSO to think about (or at least hire/manage others to think about).
"Since the Faceboook Whitehat rules state that researchers need to "make a good faith effort to avoid privacy violations", I avoided downloading any content from those buckets"
Listing the contents of the bucket is very different from fetching them. Without listing the contents, he wouldn't know the severity of the vulnerability. There's nothing wrong with that.
While I'm on his side, his wording seems to indicate he did download data from SOME of the buckets, just not those specifically containing user sensitive data.
I'm of the opinion that not downloading user data, but grabbing source code, backups, and secret keys - is a perfectly reasonable interpretation of "making a good faith effort to avoid privacy violations".
> Doesn't the author know how long it will take them to recover from this breech?
I assume Facebook would need to regenerate API keys anyways. Simply showing that author could have accessed the API keys is reason enough to think that he may have even if he claims to not have, or that someone else may have access the API keys.
The initial bug in Ruby/Rails is striking in its stupidity.[1] You can send something to Ruby/Rails in a session cookie which, when unmarshalled, stores into any named global variable in the namespace of the responding program. It's not a buffer overflow or a bug like that. It's deliberately designed to work that way. It's like doing "eval" on untrusted input. This was on YC years ago.[2] Why was anything so idiotic ever put in Ruby at all?
Something like this makes you suspect a deliberate backdoor. Can the person who put this into Ruby/Rails be identified?
I think you're overextrapolating here, though I admit my knowledge on this isn't totally up to date.
As I understand it, Ruby's Marshal function, which takes text data and deserializes it, is not safe by default. So, is that a flaw of Ruby? I guess...except that this kind of serialization seems to be a standard feature in languages (well, Ruby and Python, the two things I currently use):
> Warning The pickle module is not secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.
So the true bug seems to be that in Rails ActiveSupport (in a deprecated class, which uses some of Ruby's fun meta magic to deal with missing methods -- so basically, the classic obfuscation of functionality as a tradeoff for some sugary magic, all in a deprecated function that likely no one revisits), you can trigger a set of functions and routines in which the final decoding step, for whatever reason, ends up invoking Ruby's Marshal (via Rack: http://www.rubydoc.info/github/rack/rack/Rack/Session/Cookie...)
Also, only the server is allowed to put things into the session cookie, which is enforced by checking the cookie's signature which is generated from a key that only the server is supposed to know. Using a "native object" serializer (like Marshal or pickle) for session data and storing the secret token in a file that is easy to accidentally check into source control are both stupid things to do, but they're also common mistakes and you have to do both at the same time for this attack to work, so it seems quite overboard to suggest it was done deliberately.
Completely right. If the secret server token is compromised, it is presumed that you can fake any data. Should that allow for RCE? That's where Ruby steps in and provides the double whammy.
This might be a useful attack vector against ad servers and trackers. Those use complex cookies. The next step in the ad blocker war may be taking over ad servers.
As a security researcher and engineer, I'd like to point out the following, without taking sides:
1. Facebook is not going ballistic because this is a RCE report. They have received high and critical severity reports many times before and acted peaceably, up to and including a prior RCE reported in 2013 by Reginaldo Silva (who now works there!).
2. The researcher used the vulnerability to dump data. This is well known to be a huge no-no in the security industry. I see a lot of rage here from software engineers - look at the responses from actual security folks in this thread, and ask your infosec friends. Most, perhaps even all, will tell you that you never pivot or continue an exploit past proof of its existence. You absolutely do not dump data.
3. When you dump data, you become a flight risk. It means that you have sensitive information in your possession and they have no idea what you'll do with it. The Facebook Whitehat TOS explicitly forbid getting sensitive data that is not your own using an exploit. There is a precedent in the security industry for employers becoming involved for egregious "malpractice" with regards to an individual reporting a bug. A personal friend and business partner of mine left his job after publicly reporting a huge breach back in 2012 (I agree with his decision there), and Charlie Miller was fired by Accuvant after the App Store fiasco. Consider that Facebook is not the first company to do this, and that while it is a painful decision, it is not an insane decision. You might not agree with it, but there is a precedent of this happening.
I'm not taking sides here. I don't know that I would have done the same as Alex Stamos here, but it's a tough call. I do believe the researcher here is being disingenuous about the story considering that a data dump is not an innocuous thing to do.
I'm balancing out the details here because I know it will be easy to see "Facebook calls researcher's employer and screws him for reporting a huge security bug" and get pitchforks. Facebook might be in the wrong here, but consider that the story is much more nuanced than that and that Facebook has an otherwise excellent bug bounty history.
Edited for visibility: 'tptacek mentioned downthread that Alex Stamos issued a response, highlighting this particular quote:
At this point, it was reasonable to believe that Wes was operating on behalf of Synack. His account on our portal mentions Synack as his affiliation, he has interacted with us using a synack.com email address, and he has written blog posts that are used by Synack for marketing purposes.
Viewed in this light (and I don't believe Stamos would willfully fabricate a story like this), it is very reasonable to escalate to an employer if they seem to be affiliated with a security researcher's report.
I think instagram should be asking themselves: Would they rather have an honest researcher report this or North Korean hackers not saying anything and just slurping data? Security Researchers are always going to see things they shouldn't. That's just a fundamental rule. You have to know who your real enemies are and not come down on someone just because they got a little enthusiastic.
Wes [edit] is one of the good guys - he went overboard, sure, but he should be rewarded, he should be asked not to go crazy next time, and the rules should be updated.
Personally, I think by saying the exploit was trivial shows that the CSO should be fired. If he has to make a phone call, it's not trivial.
>If you give us reasonable time to respond to your report before making any information public, AND MAKE A GOOD FAITH EFFORT TO AVOID PRIVACY VIOLATIONS, destruction of data, and interruption or degradation of our service during your research, we will not bring any lawsuit against you or ask law enforcement to investigate you.
I certainly wouldn't consider dumping credentials to test for reuse/continued use a privacy violation. If FB wants people not to dump data, they need to make that explicit and specific.
Really? The article states: "To say that I had gained access to basically all of Instagram's secret key material would probably be a fair statement". How on earth would holding on to that data not be a privacy violation?
Holding credentials is not violating privacy. It would be possible to use those credentials to violate privacy, but merely having them is not that act.
Holding sensitive credentials is absolutely a violation of privacy. This is like saying that having a user's password is not a privacy violation unless you use it to gain access to their account.
So would you agree that holding the keys to someone's house is also a privacy violation? What if instead of keys, you were holding a set of lockpicks? Would everyone's privacy of home be immediately violated?
Holding a manually chosen password can be a privacy violation because it's a small peek into the user's psyche. (I wouldn't say the employee "changeme", "instagram" etc. passwords count, although the act of running a password cracking tool meant that he could have seen a more personal password.)
Holding some randomly generated numbers that could be used to access a server is not.
Surely they would have to revoke all the keys anyway as they would have no idea if a blackhat got their first and took the keys before the vulnerability was reported?
Privacy in this case is in an infosec context. Not a personal information context. Finding the open/unsecured/unpatched server is a bug. Downloading and testing a password keyring found as a result of that bug is not finding a bug. That is exploiting a bug for additional gain.
Finding a sql injection in a query string is finding a bug. Is using the injection to dump a table exploiting the bug for additional gain?
It sounds like you're only allowed to penetrate one layer of a defence in depth system. If you gain access to some edge system that isn't sensitive, I'd assume that would pay little. If you gain access to some core system, I'd assume that would pay lots. Why then are you not allowed to pivot from some nothing system to some larger system?
The purpose of bug bounties is to secure your systems. If you only ever secure the first layer, if some malicious actor finds another vector into the same system and there is a really easy pivot in sight (like full access to an S3 account!) then you've lost. If the bug bounty hunter found the escalation though and responsibly reported that, then a potential second vector loses its potency.
I'm not a security person at all so I'd like to hear some perspective on my thoughts above. It just seems fairly short sighted to specifically forbid pivoting.
FWIW dumping S3 buckets as a white hat does seem wrong to me. Listing them probably ok.
> The Facebook Whitehat TOS explicitly forbid getting sensitive data that is not your own using an exploit.
This seems to be the crux of this whole thing. The article suggests that is not true, including some quotes from what I assume is "The Facebook Whitehat TOS" at [0] along with his interpretation of those quotes. As an unsophisticated person reading through that document, I don't see anything I would describe as "explicitly forbidding getting sensitive data that is not your own using an exploit". The closest seems to be: "make a good faith effort to avoid privacy violations". I'm inclined to believe you and others in this thread that this was not the most responsibly done, but the seeming repeated claim that there is an explicit policy against this, which doesn't seem to be findable makes me scratch my head. Is there some other document, that is more explicit, or is this just supposed to be implicit knowledge, or what?
The "privacy violations" statement is what I was talking about. I suppose you could make an argument that this is not sufficiently explicit for this scenario, but I believe it covers this ground. It is a privacy violation to retrieve sensitive data via an exploit.
It is worth pointing out that Wesley specifically avoiding dumping data from the S3 buckets which were directly related to User Data / Information. "There were quite a few S3 buckets dedicated to storing users' Instagram images, both pre and post processing. Since the Faceboook Whitehat rules state that researchers need to "make a good faith effort to avoid privacy violations", I avoided downloading any content from those buckets" In fact, the only 'sensitive data' he retrieved in regards to user account information were the weak employee logins.
Is gathering up the credentials of employees not also a privacy violation? At this point you're going way beyond proving that you have access to something - you're actively trying to probe and see how deep the rabbit hole goes. I don't (personally) believe that this is acceptable behaviour under a white hat program.
I see your point but I'm not sure if having passwords like 'changeme' qualifies as being a privacy violation... You should almost expect it to happen at that point.
But I do recognize that cracking passwords goes a step too far.
Fair enough, I can only say that it seems like they could be more explicit on that point, but I don't see anybody arguing against the idea that that their rules could use clarification.
I feel he meant the original RCE Ruby bug which then allowed all this extra access. It was not some huge, architecture-changing security problem, just a simple upgrade to fix.
What he revealed however, was that Facebook doesn't pay attention to least privilege with key access, what those keys access[1] and more importantly where those keys access data from[2]. I have a feeling there's some scrambling to cover these blind spots over at Facebook.
The problem that Alex is skimming over here is that if Wes got access to this data, you have to ask yourself - WHO ELSE GOT THE DATA?
If Alex knows anything about his job he should know that he has to refresh all those keys even if Wes didn't report it or say anything.
The diff between Wes and everyone else is Wes just explained to Facebook how completely screwed they are. Alex is just pissed because Wes made it bluntly clear how much he screwed up.
Alex has been a vulnerability research since the 1990s, and co-ran iSEC Partners, one of the best-known software security firms in the world, through the 2000s. I'm pretty sure they're on top of the key situation.
judging by this exploit and the fact that they didn't rotate keys and other folks probably got this data, I would say this wasn't one of their finest moments, wouldn't you agree?
Select the most senior security person at those companies.
Roll 1d10 and substitute that person for Alex in this exact situation.
Now bet your life that you won't have your life wrecked by a prosecutor based on the outcome of that die roll.
I don't love Stamos calling the guy's boss, but if it's between "call his boss" and "tell legal that a bounty participant has FUCKING GONE ROGUE WITH ALL OF INSTAGRAM'S CREDS", I think he made the right goddamn call.
That just sounds like ass covering to me. The fact of the matter is that Alex had no idea and no one at Facebook had any idea if this researcher indeed went rogue with their credentials, because of the lack of security that the hack exposed. No logs on S3 buckets? No separation of access between user data and operations buckets? Give me a break.
Calling the guy's boss or the guy himself wouldn't give any authoritative answers as to what's on the researcher's laptop, so I really don't see how calling the researcher's boss was a way out of "telling legal that a bounty participant had THE KEY TO THE KINGDOM BECAUSE CAPS ARE REALLLLLY AWESOME!"
If you think that simply calling the guy's boss was the right call and not acknowledging the massive security holes that this guy exposed, then I hope you work for a company that has a more clear bounty program and deal with equally ethical researchers who will tell you about a full systems exploit without violating any user privacy and hope he's happy with your $2500. That will happen..
Now bet your life that you won't have your life wrecked by a prosecutor based on the outcome of that die roll
What I get from your comment is that it's never a smart move to take one's chances dealing with company security people. The only smart move is to sell anonymously to the highest bidder.
> but if it's between "call his boss" and "tell legal that a bounty participant has FUCKING GONE ROGUE WITH ALL OF INSTAGRAM'S CREDS"
False dichotomy - those weren't his only options, had he bothered to think more on it. There was an even better option, which strangely he chose not to take (assume an actual rogue actor got there before Wes and react accordingly: rotate the AWS keys, password reset for affected users, update SSL signing keys).
It bears asking - what exactly was he trying to achieve by calling Wes' boss, and has he achieved it? This is not his brightest moment.
Or... one could actually read the response article:
"This bug has been fixed, the affected keys have been rotated, and we have no evidence that Wes or anybody else accessed any user data. "
I understand how dumping SENSITIVE data can make you a flight risk, but he specifically outlined that he avoided dumping anything sensitive (that is anything directly related to Users and their data). He did dump S3 buckets that had a treasure trove of other files (such as the API keys for the other services and static content), so I guess my question here is at what point does dumping of any kind become bad?
In infosec keychains are about as sensitive as private as it gets. They should probably change it to "do not pull or retain any data from any server except that which is explicitly needed to identify the vulnerability" for those who might not understand.
But I feel like it would have been the same if he got to the point he did and recognized that he had access to keychains. Whether or not he actually accessed them,especially since they weren't auditing (from what I understand), is sort of irrelevant at that point, they would have to be cycled either way.
I understand that they're top secret, but that sort of proves the extent of the vulnerability.
It would have been the same. Bug bounties are for quality of the bug/vulnerability - for instance they find a configuration error that directly affects every server Facebook has open. Or they find a zero day exploit with root capabilities. Those would be million dollar bugs. Facebook definitely needs to clarify that the bounty is for the severity and widespread nature of the bug itself and not an invitaion to penetration testing. They also need to be more explicit about what is not allowed. Maybe they should give bonuses for the value of the target, but the current policy is for the bug itself. He certainly did expose an embarrassing lack of procedure and awareness of key security and that's certainly worth a lot more to Facebook than the bug. However they definitely do not want to encourage penetration testing. And it's infosec code of ethics (probably should be written down somewhere) that when you find a bug you don't use the bug to download anything from the target. It means a lot of people won't be interested because they want to hack and penetration test. To be whitehat about that requires a lot closer communication and contractual obligation.
Facebook needs to get its shit together in key security and clarity of its bounty program. On the other hand this guy writing a blog about downloading a keychain and probing how deep it leads is definitely not responsible infosec.
Running a bug bounty is not a suicide pact. A team had to convince a finance group that it was valuable to give money away to people who might be assholes. Bounty hunters are not a community- but if you are a bounty hunter, you should understand that many of your peers are total assholes. The company that wants to pay you a reward has to figure out if you are going to make them regret offering you a reward.
There are 4 categories of reporters: great, good, shit and crazy. Again- if you are a reporter, you should be trying to make it easy for the team to distinguish you in one of the first two categories by being simply being polite & respectful.
I will take a side- it's Facebook. Dumping data is the end of the Proof of Concept. Trying to determine if there is more data you can access through a single vulnerability chain is over the line.
Boats sink. The engineers know it. If you sink a boat in order to prove the boat had a hole, you will not get your payout.
And one final thought-
In my experience, bounty hunters almost never realize the full consequences of a vulnerability that receives a reward. Most of the time, the "Bad thing" that they identify is just the tip of the iceberg.
The choices of the researcher reflect inexperience and immaturity. The researcher has a significant misunderstanding about what is happening in the bug bounty marketplace. I think they need to apologize if they want a future in the infosec world.
Publishing this blog post was a huge error. Going to the journalist was another huge error. I don't see how this person could ever be considered employable by a reputable company.
Are you saying that if Wes hadn't pointed it out, than Alex wouldn't have to refresh all those keys? That if Wes hadn't dumped the keys than they were 100% secure?
Database is just a tool to store data, just like such tool is a filesystem. Can you explain the difference between dumping user logins from a table and just reading them from a file? How first one is a no-no and the second one is fine?
Regardless of whether or not he followed etiquette or the rules he did report it and obviously had no intention of utilizing it to be a bad guy. And calling his employer? This was ass covering by the CSO.
> This is well known to be a huge no-no in the security industry. I see a lot of rage here from software engineers - look at the responses from actual security folks in this thread, and ask your infosec friends...
The problem is that on the one side you have security professionals who do this full time. They build up a background of implicit knowledge through extensive interaction with other security professionals, via training, mentoring, team activities, etc.
On the other side you have folks like the guy who found this vulnerability -- don't specialize in security, basically moonlighting / hobby, not necessarily connected to other security professionals or even other hobbyists. They won't have the same kind of implicit knowledge.
When someone from the first category communicates with someone from the second category, the communication can break down. That's what happened here.
Offering a million dollar bounty makes this kind of communication problem more likely -- a potential million-dollar payout catches the interest of people who have spare time and encourage them to pick this as the thing they do on the side. And further, encourages them to try anything and everything you don't explicitly forbid in by giving them hope that if they just try hard enough, they'll be able to turn what initially looks like a ho-hum two-year-old Ruby exploit into a million-dollar payday.
On some level isn't the security testing a farce if you can't use local data to escalate your breach? It seems kind of like a bank that wants to know if their front door is unlocked but doesn't want you to tell them the vault's open.
561 comments
[ 2.5 ms ] story [ 354 ms ] threadI'm always curious about what sort of internal pressures would lead people to take a well-reported bug that the author did not take malicious action on and blow it up to the point that the CSO is getting involved.
Between stories like this demonstrating companies' apparent lack of understanding of whitehat infosec, and Weev's incarceration demonstrating the American legal system's apparent lack of understanding of whitehat infosec, it's hard to believe people still participate in such endeavors.
Imagine I will contact your significant other over your comment on HN. Would not you be deeply disturbed even if I do it just once? Bonus points for frivolous legal threats on my side.
Exactly what parent said, " . . . a pattern of repeated contact intended to cause emotional distress". Yes, doing as you said would be deeply disturbing, but it wouldn't be harassment.
Unfortunately --- and this is not a normative argument, so please don't wig out --- the criminal action here is the researcher's. Nobody's been prosecuted, but could they have been? YES.
No, it was "We're bigger than you and we have the power to fuck with your life and livelihood", and nothing more.
Again: read the timeline. He submitted a finding with AWS creds taken from the server he popped on October 22 --- on December 1, more than a month after Facebook shut the server down. He took AWS creds from a Facebook server and saved them on his laptop for more than a month.
WHY?
It's neither harassment or slander, but it could be tortious business interference, where one party induces a second party to break a contract with a third party.
His contract employers are neither his parents or legal guardians - who have no more power to "ensure" this as anyone else.
As a corollary to this - this is exactly why it's illegal for debt collectors to call your friends and family to "encourage" them, or you by humiliation, to pay up.
Option 1: Call the pentest firm he works for.
Option 2: Threaten legal action.
Option 3: This dude might still be walking around with god knows what shit he's pulled out of S3 buckets or lord knows what else was accessible with those AWS keys or what other keys were in other S3 buckets or
wow i'm having a small panic attack just trying to complete that sentence
[1]: https://news.ycombinator.com/item?id=10754627
My understanding is this: They got a report that a server can be compromised and fixed that vulnerability. Unbeknownst the reporter grabbed a huge amount of (remote! not on that server, btw) data to play with.
Later the reporter returns to Facebook and says 'Btw, I got all these valuable pieces of information and have those for quite a while'.
Only at that point can you panic and rotate keys, but now you notice that a third party had access to all these keys for a month already. What else did they get? Maybe the researcher sat in a posh coffee place and grabbed interesting Instagram credentials (using the certificate) or escalated this further, gaining even more access based on the exposed information so far.
In my world, Facebook/Instagram are basically completely owned and have to assume that this guy grabbed ~everything~. They probably need to hire (vs. doing a bug bounty) people to grab the same data from the same buckets to look for potential follow-up targets that were _not_ disclosed, but might've fallen to the same bug hunter.
Who's to say that the guy doesn't come around on New Year's Eve with Yet Another Disclosure based on the same 'attack'?
I hate the 'contact the employer' part, but I'd hate to be in FB's shoes far more. I can hate the company and feel for its CSO/IT staff in this aftermath at the same time.
However, since they were unable to figure out that the friendly researcher owned them until he told them we now know that FB itself doesn't know who has their data.
And that means rotating all your credentials the very same day you learn about that happening. Why does it make a difference then if he still has stale credentials?
"At this point, it was reasonable to believe that Wes was operating on behalf of Synack. His account on our portal mentions Synack as his affiliation, he has interacted with us using a synack.com email address, and he has written blog posts that are used by Synack for marketing purposes."
From Alex Stamos's writeup: https://www.facebook.com/notes/alex-stamos/bug-bounty-ethics...
> I never contacted Facebook or Alex using my work email account. It was only after Alex contacted my employer via email that I sent a reply from my work account. Alex indirectly contacted me at work, not the other way around.
The story tells us his side of things but what specifically Facebook perceived as threat is still unknown ? Why would a CSO get involved unless they specifically think that the data has been accessed violating the goodwill of the bug bounty research in the first place.
Assuming the story as stated is truthful or even plausible, what options do whitehat hackers have to defend themselves in such a scenario? I mean the whole point seems to be to try to penetrate a secure system, and the consequences of that action seems to be fairly obvious from the start. If a whitehat hacker is successful, that carries with it the inherent potential that they will have some sort of access to some sort of sensitive data, right?
Surely telling Facebook "I was able to access these exact things" means he expected Facebook to update passwords and change keys accordingly, making the possibility that he retained those keys moot.
The most likely reason I can think of is that he was getting some heat from some other C*O.
Edit: Now that Alex's side of the story has been released, his actions don't seem out of line (assuming it's reasonably accurate, and I have no reason to suspect it isn't).
While my explanation is still a valid answer, I agree with the parent. Sounds like he was just doing his job. Though it would be interesting to listen to the audio of the conference call....
We'll probably never see a post mortem on this but it'd be interesting to hear how this got moved to production...: was the Sensu admin panel a nice scaffold for internal use and by the time they decided to make it remote, everyone just assumed the secret key had been changed at some point?
Never expose anything that doesn't need to be, SSH tunnels, openvpn... heck, use HTTP authentication wrapped SSL tunnels if you have to. Web servers tend to be more secure than webapps.
Mistakes like this are far too easy to make. So anything that isn't part of your business application needs to be tunneled with authentication or isolated completely.
This was his mistake. This is a huge no-no. You never dump data unless you have permission. It's against the terms of most bounty programs.
In an external pentest, you get a set of netblocks and rules of engagement, and you get as far as you can. That's why it's called a "penetration test".
In a vulnerability assessment, you get a target (usually an application), and you find as many flaws in that target as you can.
Big annual pentests often have wide-open rules of engagements, where you (as a consultant) win big by, for instance, dumping the CEO's mail spool. But those projects also start with several meetings worth of negotiating rules of engagement.
Vulnerability assessments virtually never have those rules of engagement!
Nobody that I know of runs a bug bounty program on pentest norms. To do so would be grossly irresponsible, because on every network with more than 1000 hosts I've ever tested, ever, RCE behind the firewall is gameover for the whole test: you can get everything.
Among the many reasons bug bounties are bad ideas is that they generally fail to write clear rules -- as Facebook did. As written, what he did is not against the rules and while it may fall into some best-practices bucket you assert to be universal, that's hardly sufficient for a field in which participants can come from any background. But please, continue to defend your friend whose multi-billion company had a month to cycle their popped keys and failed to do so, then responded by threatening a researcher's employment after multiple conciliatory e-mails.
That is NOT what happened. Look at the timeline again.
* He popped the server.
* He submitted the RCE.
* He submitted dumped file from the compromise as a finding.
* They fixed the RCE.
* They told him not to dump files.
* They paid out the RCE finding.
* A month later, they declined to pay out on the dumped file.
* In response, he submits a new finding, with AWS creds that he stored for more than a month after they shut down the server
* (Whatever else happens that day)
* Stamos calls Synack.
I agree with you that something seems off, but you're happily giving all the charity to FB and none to this guy, which is your prerogative but hardly makes for good conversation.
1. Second finding is declined.
2. New third finding, which includes AWS credentials that this person should not have had, is written and submitted.
3. Stamos calls Synack.
I believe the relative timing of these events is, in fact, established.
Now: stipulate that I'm right, even if you're not sure. Does your opinion of the story change?
I don't understand why a company would ever say "you can snoop around in our stuff" without very clearly stating what they can do. You're leaving open a legal loophole where a blackhat can claim to be a whitehat.
60 accounts on the admin console are not users, and he did not touch the buckets with actual user data.
> As described above, I used the web interface to gain code execution, but at this point I still hadn't actually gained access to the web interface as a normal user.
He had code execution, there was no need for him to go any further.
An apparent refusal to admit that in the write up is making it hard to put 100% support behind him.
There is no excuse: dumping the user table was too far.
Facebook went rather far too, of course.
* Day 1: Report RCE
* Day 2: Report finding from dumped file
* Day 4: RCE's gone
* Day 8: Asked not to dump files using RCEs in the future
* Day 26: Paid out for the RCE.
* Day 40: Bug based on dump is rejected
* Day 41: Report new bug based on dump, which shouldn't have been accessible for over a month!
* Day 41+: All hell breaks loose.
1. Never trust Facebook
In particular, it might help with Signal vs WhatsApp.
1. Never trust Facebook. If he had signed some messages using their keys and sent it to Wired/NYTimes/etc he'd be getting fame and job offers instead of $1250 and call home to his parents.
2. Ruby programmers are garbage.
This is the fastest and easiest way for Facebook to stop good submissions to their bug bounty program.
Alex Stamos is a good person† who has been doing vulnerability research since the 1990s. He's built a reputation for understanding and defending vulnerability researchers. He hasn't been at Facebook long.
To that, add the fact that there's just no way that this is the first person to have reported an RCE to Facebook's bug bounty. Ask anyone who does this work professionally: every network has old crufty bug-ridden stuff laying around (that's why we freak out so much about stuff like the Rails XML/YAML bug, Heartbleed, and Shellshock!), and every large codebase has horrible flaws in it. When you run a bug bounty, people spot stuff like this.
So I'm left wondering what the other side of this story is.
Some of the facts that this person wrote up are suggestive of why Facebook's team may have been alarmed.
It seems like what could have happened here is:
1. This person finds RCE in a stale admin console (that is a legit and serious finding!). Being a professional pentester, their instinct is that having owned up a machine behind a firewall, there's probably a bonanza of stuff they now have access to. But the machine itself sure looks like an old deployment artifact, not a valuable asset Fb wants to protect.
2. Anticipating that Fb will pay hundreds and not thousands of dollars for a bug they will fix by simply nuking a machine they didn't know was exposed to begin with, the tester pivots from RCE to dumping files from the machine to see where they can go. Sure enough: it's a bonanza.
3. They report the RCE. Fb confirms receipt but doesn't respond right away.
4. A day later, they report a second "finding" that is the product of using the RCE they already reported to explore the system.
5. Fb nukes the server, confirms the RCE, pays out $2500 for it, declines to pay for the second finding, and asks the tester not to use RCEs to explore their systems.
6. More than a month after Facebook has nuked the server they found the RCE in, they report another finding based on AWS keys they took from the server.
So Facebook has a bug bounty participant who has gained access to AWS keys by pivoting from a Rails RCE on a server, and who apparently has retained those keys and is using them to explore Instagram's AWS environment.
So, some thoughts:
A. It sucks that Facebook had a machine deployed that had AWS credentials on it that led to the keys to the Instagram kingdom. Nobody is going to argue that, though again: every network sucks in similar ways. Sorry.
B. If I was in Alex's shoes I would flip the fuck out about some bug bounty participant walking around with a laptop that had access to lord knows how many different AWS resources inside of Instagram. Alex is a smart guy with an absurdly smart team and I assume the AWS resources have been rekeyed by now, but still, how sure were they of that on December 1?
C. Don't ever do anything like what this person did when you test machines you don't own. You could get fired for doing that working at a pentest firm even when you're being paid by a client to look for vulnerabilities! If you have to ask whether you're allowed to pivot, don't do it until the target says it's OK. Pivoting like this is a bright line between security testing and hacking.
This seems like a genuinely shitty situation for everyone involved. It's a reason why I would be extremely hesitant to ever stand up a bug bounty program at a company I worked for, and a reason why I'm impressed by big companies that have the guts to run bounty programs at all.
† (and, to be clear, a friend, though a pretty distant one; I am biased here.)
It doesn't look like the SSL cert on instagram.com has changed recently, and the pentester specifically claims to have obtained its private key.
Alex is good friend of mine and I've known him since college. He's definitely a good guy and understands the ins and outs of security vulnerability research, having done it himself for many years. I'm sure he didn't take the action of calling the researcher's employer lightly, and probably had a really good reason to do so.
There has to be a side of this story we aren't hearing, and probably never will.
He's the CSO, and this occurred under his watch. The exploit was 2 years old, and well known. It highlights an internal security problem at Facebook et al, of-which Alex sits at the top.
In this situation, his years of "doing it himself" is unlikely to have factored in - rather, he felt like he dropped the ball and could be facing some consequences, or at the very least felt embarrassment.
This would have led to a rash thought process, and perhaps Alex jumped to the conclusion of some sort of sabotage by another company.
Sounds like a jerk to me.
Escalating issues with a company to the CEO of that company doesn't seem like jerk behavior.
Wes counters that, "[Alex] never for a second believed I was operating on behalf of Synack"
I'm not sure how Wes knows what is going through the mind of Alex, so I'm inclined to take Alex's word on this.
As blazespin[1] mentioned in this thread, Facebook's own terms states that they only pay individuals. That's how Wes knows - because Facebook's bounty program never deals with companies. The only other explanation would be Alex is ill-informed about the terms of Facebook's bounty program.
1. https://news.ycombinator.com/item?id=10755746
Think about it a different way. If this researcher had found SQL injection in a webapp, dumped the usernames and passwords, and reported the vulnerability for a bug bounty, he should get paid. If he kept each of those credentials, and then logged into other systems using higher-privilege accounts that he'd compromised even after the SQLi is fixed, he is basically continuing the exploitation of an already-fixed bug. Those don't deserve payouts. Similarly, if he'd established some sort of persistence (such as a reverse shell, etc) on compromised assets, he can't keep coming in and getting more and more bounty payoffs. Fruit of the forbidden tree, in this case.
Where I disagree with tptacek is with regard to the benefit of bug bounty programs. Although I'm not currently running one, I find the idea fascinating and helpful for two primary reasons: first, you're almost definitely going to see generally better results in a well-managed bug bounty program (not necessarily something like Facebook's White Hat program) than traditional pentests or application security assessments. More eyes are almost always better when searching for tricky problems. Secondly, if you're a large enterprise, there are already people "testing" your security. I'd much rather be able to pay out a researcher than drive them to more nefarious buyers. You will probably encourage many people to test your security (which screws up metrics) but if finding security problems is the ultimate goal, it's worth it.
Even in this case in point, Facebook did discover an RCE that could have been (and kind of was) extensively exploited due to the fact that they held the bounty. If an actual malicious hacker had found that problem first, they would have been in significantly worse shape.
With a bug bounty programme you don't generally authorise the kind of post-exploitation activities which we see here as leading to the really serious exposures, and that's not surprising as you can't easily authorise a set of unknown people to be processing your customer data.
This differs from an engaged penetration testing firm, with whom you have a contract which covers things like handling of data gained during a test.
So I don't really see bug bounties ever replacing penetration testing companies for internal work or anything that requires accessing customer data as part of the exploit...
Also, you can't just expect that "oh, just delete your data pls" will work, can you? You can't trust anyone that literally hacks your system.
Why did those credentials still work post-report?
What if those credentials were accessed from a public dump?
The outcome of this entire clusterfuck of a bounty is one of the reasons there are still very well paid blackhats. There are no rules or terms to follow.
If their terms aren't clear (the terms he's citing certainly weren't intended for keys, rather Facebook user accounts/information), pay out and fix them.
Or do you think he should have just stopped and Facebook should have realized how bad it was and paid him a lot more than $2500?
I don't believe any company would pay $1M for a bounty on their own systems. Only people who intend to use the vuln, or to fix it as they are the vendor.
Fr a vuln to go for $1M requires "discovering SQL injection"-levels of vuln. MS paid $100K for an entire vuln class for ASLR/DEP bypass discovery, and promptly patched the shit out of it. For a remote vuln class, I could see them paying $1M quite happily to not have all of their products re-owned.
No wonder there's a flourishing (and well paying) blackmarket for vulnerabilities. I wonder how much this keys-to-the-kingdom vuln would be worth (Mitm Instagram, bootstrap a botnet, steal celebrity pics, ... the possibilities are endless)
If it was not done for the purpose of intimidation, then Alex simply would have asked the CEO if the researcher was acting on the company's behalf and after hearing "no" would have ended the call and contacted the researcher directly.
Seems simple doesn't it? Perhaps you are not seeing it due to your friendship, but it seems like a dirty move and only serves to call into question how Alex handled other aspects of the situation.
Then the CEO is going to contact the researcher and he's screwed either way. God knows what the CEO would have say to the researcher privately. Having a middle man to translate is a bad idea in an emergency.
Let's face it, when you used your work email and made another company paranoid, you are putting people on the spot. Employer needs to know (they have legal responsibility), and given the prior research they did and the researcher's claim, I think the reach out is absolutely correct.
Instgram's infrastructure has flaw. That's bad but everyone's infrastructure has flaw. Shit has to be fixed. Doing more than what was needed is bad. If I am told to stop dumping data, I would stop.
Alex just went the drama route.
Why make such a bad situation worse, if you don't have to?
FB messed up. The researcher partly messed up too. Fix it and move on.
"stuff" was the keys to the kingdom, do you think this is acceptable for a company like facebook? So instead of them making an apology, the CSO is trashing the guy who gave them the wake up call?
I do think you are heavily biased ;)
This reminds me of the illusion of objectivity in journalism. If you pretend to be perfectly objective and unbiased, you're lying.
That said, Alex Stamos and the rest of the security team should have tried to figure out what vulnerabilities existed from this server instead of just nuking it and thinking that the problem was solved. That was lazy and stupid.
The issue here is that, in hindsight, FB failed at this step.
They nuked the server, but they didn't determine what sensitive information was available on that server, and take steps to mitigate those risks.
I think that's an understandable mistake - cleaning up after a server intrusion is hard. Knowing how much to do after a possible intrusion is even harder. But it is still a mistake and it happened on Alex's watch.
If the purpose of the bounty program is to find out about your security mistakes, then the program did its job here, and Alex should be pleased that the problem was reported so that they could fix it.
That the researcher found the mistake by overstepping what is considered ethical (and I have no doubt that they did overstep) creates a very difficult situation - you don't want to reward that behaviour, but you do want to know about security problems and this one was only discovered/reported because of that bad behaviour.
In that difficult situation it is all the more important to tread carefully. The easy cases where you're paying out a $10k bounty typically don't require much finesse. It's the tricky cases where you need to make sure your actions are well considered and above-board at every step.
From Alex's own summary it's evident that he didn't handle it as well as he could have.
Two of the longest paragraphs in Alex's write up cover what he said to the CEO of Synack, even though Synack had nothing to do with this. Even if we accept that Alex thought it likely the Wes was acting on behalf of Synack (personally, I don't think that was a reasonable conclusion to draw, thought I assume Alex is sincere in his view that it was), he should have determined that up front, and then, once he knew it was not work related, he should have avoided:
- making accusations about Wes's ethics to his boss ("Wes ... had acted unethically")
- suggesting that his external behaviour has implications for his employment ("Wes's behavior reflected poorly ... on Synack")
- bringing in the threat of lawyers ("keep this out of the hands of the lawyers")
When faced with the difficult situation of legitimate security research that has (well) overstepped the ethical boundaries, all the evidence is that Alex jumped to the position of protect yourself, protect the company, intimidate and control the researcher and though that is a common and understandable reaction, it's not the way you turn a bad situation like this into a good one.
Dumping the users table on an 'internal' (heh) dashboard -- any company that is doing these bounty programs needs to clarify what a 'user' is. Is it someone using their application, or all employee information as well. It's an important distinction.
And the bug has been existing for 2 years.
Wonder where the person who tipped him off had the info from – could very well have been a common target in the black hat scene.
I imagine the initial report by his friend that the server was accessibly would not be a very high paying bounty compared to one accessing the server. But how deep is too deep?
If the author submitted the RCE, and nothing else: is someone at Facebook actually going in and trying to simulate what he actually did? Who knows, because the process is pretty opaque. If you argue with Facebook's assessment, and go and further exploit the system to say "no, this is actually how bad the RCE is, in the grand scheme", you've now actually gone and proved what can be done, against their guidelines, which potentially disqualifies your initial discovery altogether.
It seems too difficult to define how deep is too deep, especially since at least he reported him doing it. He didn't decide to go that deep and then just report the RCE and collect $10 million from people far more interested in this.
We should have "pastebin hat" list and Facebook should definitely be on it.
The problem with humans is that they will rather go extinct over such things than behave properly. You could try to teach us by painful example but death will probably come first.
I realize a million is a bit unrealistic, but if you're going to make a public statement, at least back it up or prove to the guy why his findings don't constitute a "million-dollar bug". It's not right to just cold-shoulder the guy and hide behind vague rules that were never clearly outlined. In fact, you might even conclude Facebook brought his behavior on themselves by making such a statement as "if a million-dollar bug is found, we'll pay it out." $2500 is nothing when you're thinking $1,000,000
Yeah, the researcher went deep into the grey area, but I find Alex Stamos's reaction barely short of unbelievable - it's almost as though he's so new to the internet he's never heard of the Streisand Effect... (Either that, or he's just so accustomed to bullying and intimidating people who might embarrass him that he's now got that corrupt politician "Waddaya mean I'm 'abusing my power'? We grant multimillion dollar contracts to old school buddies all the time? What's the problem?" look on his face.)
A script to create new bogus accounts on Facebook is probably worth more than mass Facebook account compromise.
People really don't seem to understand how the "black market" works.
(But yeah, I'm perfectly happy with my life where I have no real understanding of how the black market for this kind of thing works...)
He claims to have downloaded the content listed below. And he is surprised that Facebook responds coldly? Note the string "private keys" in this list... Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?
On the other hand, it does sort of re-enforce the idea that he should be paid handsomely, doesn't it? :)
Facebook might've never known
If they react this way, and can't trust people to attempt to find exploitable security holes on their system (even those that yield private keys), then what is the point at all? The only people that find them then, are not going to be as cooperative about it.
> Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?
This is not the author's fault. He did nothing but disclose bugs that Facebook themselves set in place, and seemed to be very open with them about it, at that.
This person took a bug bounty and ran it as a penetration test.
Facebook fixed the one bug he found and paid him for it.
It seems that severity-based payouts have created incentives that do not match the program rules? Maybe all rce bugs should be paid out on an assumption that if used they'll lead to access to a shell or to user data.
He clearly _did_ have a "security vulnerability" that gave him the keys to the kingdom. He knew it, and Facebook know it - and they wanted to pretend it was no big deal.
Any bets on how many months till there's a large-scale breach of Facebook user data? The reality of the balance between responsible disclosure and selling an exploit is much easier to evaluate now.
(Seriously??? That's some world-class enterprise-grade "moving fast and breaking things"...)
But he didn't. He put the credentials in his back pocket so he could pull them out when they suggested he hadn't found his "million dollar bug". And so for a month after they fixed the bug, some fucking rando is walking around with credentials to all of Instagram's AWS assets, totally unbeknownst to anyone at Facebook. They turn down his bid for his "million dollars", and he busts the credentials out on them. You think they're going to thank him?
He's lucky it was Stamos and not Mary Ann Davidson.
So, the fact that those credentials still worked a month later is a HUGE FUCKING DEAL! Alex, the consummate professional, didn't do his job and instead had a knee jerk reaction to someone slapping that fact in his face.
Notwithstanding the fact that AWS credentials should be very narrow in scope.
> If you believe you have found a security vulnerability on Facebook, we encourage you to let us know right away.[1]
Which then begs the question to me: how do you differentiate an acceptable and unacceptable probing of security vulnerabilities when you can't capture the full impact of an issue without attempting to exploit it to its fullest? Because it is certainly not outlined in their policy.
And when you're asking for any whitehat to attempt to discover and disclose security vulnerabilities in your system with only the limpest of guidelines around how to do so, I don't feel that it is warranted to react such as Facebook has here.
[1]: https://www.facebook.com/whitehat
When you stand up a bug bounty program, you are giving strangers permission to do something that they would otherwise be prosecuted for doing. You should be extraordinarily careful when you do that, and your rules of engagement should be crystal clear. These weren't.
One thing I notice: if the CSO felt like this person did something grossly illegal and irresponsible, why not go straight to the police? Why instead go to the man's employer and speak passively aggressively?
Paradoxically, contacting the authorities could have helped facebook's argument. It would communicated to the community at large: "Hey Facebook believes it has clear standing to pursue this guy. Maybe, he really did do something wrong."
Instead, what I'm reading is: "Facebook doesn't actually believe what the guy did was illegal per se... but they wanted to spite the guy anyway."
For me, it seems petty.
> I never contacted Facebook or Alex using my work email account. It was only after Alex contacted my employer via email that I sent a reply from my work account. Alex indirectly contacted me at work, not the other way around.
Also, why would he be doing this work at the behest of his employer when (IIRC) Facebook's bounty program only pays out to individuals? It would automatically make him ineligible to claim the bounty.
To me it seems like Alex Stamos tried to use some good old threaten-your-livelihood intimidation tactics and failed miserably.
I don't think the goal, or desire, is to be told the full extent or impact of a problem. The goal is to be alerted to spots that may lead to a large problem, or re in and of themselves a large problem.
This seems like it has a few facets to it. You end up reducing the space of things to mostly "ways to get in the front door." Thinking about it, I would probably be frustrated, in general, if I knew someone had important keys to the kingdom I was in charge of. It doesn't change the fact that others may or may not have also gotten the same access, now it's 1-* instead of 0-* people who have it and shouldn't.
I'm still slightly skeptical on the bounty reward itself. This was a simple exploit that got pivoted into some major shit, so do you reward the exploit of the logical conclusion of the exploit? I lean towards the latter, but again, as you said... how do you figure out the impact without... actually trying to figure out the impact?
Bug bounties are an interesting concept, to be sure. -
My guess is that they discovered the AWS credential thing on December 1.
A much more reasonable and likely explanation of the same set of things we've been told:
Alex Stamos called Synack and said that the AWS credentials, which, by the researchers own admission, he'd chosen to retain long after the vulnerability he reported was fixed, had to be deleted, and that if they weren't and the researcher continued to use them, the situation would be out of Stamos' hands and into Facebook legal's, at which point he couldn't keep him from being prosecuted.
In that interpretation, Alex isn't threatening the researcher; he's (very reasonably) saying "you cannot use these credentials you've taken from the server, and if you keep doing that, I can't take responsibility for how Facebook will handle this, so you should stop right away before you harm yourself."
tptacek > You don't know that's what happened, even the researcher didn't say that. You're extrapolating.
From Wes' blog (presumably based on his boss' oral description of the call): "Alex then stated that he did not want to have to get Facebook's legal team involved, but that he wasn't sure if this was something he needed to go to law enforcement over."
Your bias is apparently badly incapacitating your reading comprehension, because "stated that [...] he wasn't sure if this was something he needed to go to law enforcement over" is exactly threatening to call the cops. Not even your friend mr Stamos, who has presumably read Wes' blog post, is claiming that he didn't. So whom are you saying is lying; Wes, or his boss?
Oh, and "(very reasonably) saying 'you cannot use these credentials you've taken from the server, and if you keep doing that, I can't take responsibility for how Facebook will handle this, so you should stop right away before you harm yourself.'" really, really, really sounds like Vito The Baseball Bat "very reasonably" saying "You cannot use this testimony you got off Loanshark Louie, and if you keep doing that, I can't take responsibility for how the boys will handle this, so you should stop right away before you harm yourself."
Seeing that as SERIOUSLY (as opposed to sarcastically) "very reasonable"... Well, hello, friendship-bias Bizarro World.
As I'm sure it's not the only trivial bug!
Instagram should be thanking Wes for the wakeup call instead of making him the enemy.
Facebook's point is that he found a vulnerability, and exploited it instead of stopping there. I kind of understand their point of view though. "See you have a vulnerability there, and then I can get access to this, and then this, and see now I have the password of your user, and then I'm just one click away from accessing all the instagram pictures I want."
Although Facebook's handling of the problem is poor (why didn't the CSO call the author directly to get things squared out? He does not talk to people who are not C*O?), they do have a point.
I think the author acted in good faith, but got carried away by his findings unfortunately.
However, these weak passwords could have been exploited separately as part of an attack. It is fair to call it a new vulnerability, even though it was discovered by exploiting the first vulnerability.
If he had found a bug with something a developer wrote that would be a different story. What he found was layer after layer of Operations (particularly Security Operations) failures. This is something you hire a CSO to think about (or at least hire/manage others to think about).
> [...] I queued up several buckets to download, and went to bed for the night.
> The next day, I began to go through some of what I'd downloaded, [...]
"Since the Faceboook Whitehat rules state that researchers need to "make a good faith effort to avoid privacy violations", I avoided downloading any content from those buckets"
Listing the contents of the bucket is very different from fetching them. Without listing the contents, he wouldn't know the severity of the vulnerability. There's nothing wrong with that.
I assume Facebook would need to regenerate API keys anyways. Simply showing that author could have accessed the API keys is reason enough to think that he may have even if he claims to not have, or that someone else may have access the API keys.
Doctor diagnoses, patient has cancer.
Doesn't the doctor know how long the patient will take to recover from this disclosure? How much it will cost the patient?
Something like this makes you suspect a deliberate backdoor. Can the person who put this into Ruby/Rails be identified?
[1] http://robertheaton.com/2013/07/22/how-to-hack-a-rails-app-u... [2] https://news.ycombinator.com/item?id=6110386
As I understand it, Ruby's Marshal function, which takes text data and deserializes it, is not safe by default. So, is that a flaw of Ruby? I guess...except that this kind of serialization seems to be a standard feature in languages (well, Ruby and Python, the two things I currently use):
https://docs.python.org/3/library/pickle.html
> Warning The pickle module is not secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.
So the true bug seems to be that in Rails ActiveSupport (in a deprecated class, which uses some of Ruby's fun meta magic to deal with missing methods -- so basically, the classic obfuscation of functionality as a tradeoff for some sugary magic, all in a deprecated function that likely no one revisits), you can trigger a set of functions and routines in which the final decoding step, for whatever reason, ends up invoking Ruby's Marshal (via Rack: http://www.rubydoc.info/github/rack/rack/Rack/Session/Cookie...)
- Java: WebSphere, WebLogic, JBoss, Jenkins : http://foxglovesecurity.com/2015/11/06/what-do-weblogic-webs... . Admittedly most of these are through sidechannels and nothing as obvious as sessions, but it's the same mistake.
- Python: https://blog.nelhage.com/2011/03/exploiting-pickle/ . Unpickling got at least Cisco Web Security Appliances: http://tools.cisco.com/security/center/content/CiscoSecurity...
- PHP : Of course, it's PHP and of course, it's WordPress . https://vagosec.org/2013/09/wordpress-php-object-injection/
It's hard to attribute malice to an obvious mistake that everyone makes.
edit: if it's indeed true, but I have my doubts that's the case. Hard to say either way.
1. Facebook is not going ballistic because this is a RCE report. They have received high and critical severity reports many times before and acted peaceably, up to and including a prior RCE reported in 2013 by Reginaldo Silva (who now works there!).
2. The researcher used the vulnerability to dump data. This is well known to be a huge no-no in the security industry. I see a lot of rage here from software engineers - look at the responses from actual security folks in this thread, and ask your infosec friends. Most, perhaps even all, will tell you that you never pivot or continue an exploit past proof of its existence. You absolutely do not dump data.
3. When you dump data, you become a flight risk. It means that you have sensitive information in your possession and they have no idea what you'll do with it. The Facebook Whitehat TOS explicitly forbid getting sensitive data that is not your own using an exploit. There is a precedent in the security industry for employers becoming involved for egregious "malpractice" with regards to an individual reporting a bug. A personal friend and business partner of mine left his job after publicly reporting a huge breach back in 2012 (I agree with his decision there), and Charlie Miller was fired by Accuvant after the App Store fiasco. Consider that Facebook is not the first company to do this, and that while it is a painful decision, it is not an insane decision. You might not agree with it, but there is a precedent of this happening.
I'm not taking sides here. I don't know that I would have done the same as Alex Stamos here, but it's a tough call. I do believe the researcher here is being disingenuous about the story considering that a data dump is not an innocuous thing to do.
I'm balancing out the details here because I know it will be easy to see "Facebook calls researcher's employer and screws him for reporting a huge security bug" and get pitchforks. Facebook might be in the wrong here, but consider that the story is much more nuanced than that and that Facebook has an otherwise excellent bug bounty history.
Edited for visibility: 'tptacek mentioned downthread that Alex Stamos issued a response, highlighting this particular quote:
At this point, it was reasonable to believe that Wes was operating on behalf of Synack. His account on our portal mentions Synack as his affiliation, he has interacted with us using a synack.com email address, and he has written blog posts that are used by Synack for marketing purposes.
Viewed in this light (and I don't believe Stamos would willfully fabricate a story like this), it is very reasonable to escalate to an employer if they seem to be affiliated with a security researcher's report.
LAUGH.. Where does it say this?
https://www.facebook.com/whitehat/
I think instagram should be asking themselves: Would they rather have an honest researcher report this or North Korean hackers not saying anything and just slurping data? Security Researchers are always going to see things they shouldn't. That's just a fundamental rule. You have to know who your real enemies are and not come down on someone just because they got a little enthusiastic.
Wes [edit] is one of the good guys - he went overboard, sure, but he should be rewarded, he should be asked not to go crazy next time, and the rules should be updated.
Personally, I think by saying the exploit was trivial shows that the CSO should be fired. If he has to make a phone call, it's not trivial.
Holding some randomly generated numbers that could be used to access a server is not.
My browser is currently showing an ssl cert for instagram.com that was issued in April and expires on Dec 31.
Doesn't look like they're in any hurry to revoke that one. (I guess like Alex Stamos told his employer - it's "trivial and of little value"...)
Seems … contradictory.
It sounds like you're only allowed to penetrate one layer of a defence in depth system. If you gain access to some edge system that isn't sensitive, I'd assume that would pay little. If you gain access to some core system, I'd assume that would pay lots. Why then are you not allowed to pivot from some nothing system to some larger system?
The purpose of bug bounties is to secure your systems. If you only ever secure the first layer, if some malicious actor finds another vector into the same system and there is a really easy pivot in sight (like full access to an S3 account!) then you've lost. If the bug bounty hunter found the escalation though and responsibly reported that, then a potential second vector loses its potency.
I'm not a security person at all so I'd like to hear some perspective on my thoughts above. It just seems fairly short sighted to specifically forbid pivoting.
FWIW dumping S3 buckets as a white hat does seem wrong to me. Listing them probably ok.
This seems to be the crux of this whole thing. The article suggests that is not true, including some quotes from what I assume is "The Facebook Whitehat TOS" at [0] along with his interpretation of those quotes. As an unsophisticated person reading through that document, I don't see anything I would describe as "explicitly forbidding getting sensitive data that is not your own using an exploit". The closest seems to be: "make a good faith effort to avoid privacy violations". I'm inclined to believe you and others in this thread that this was not the most responsibly done, but the seeming repeated claim that there is an explicit policy against this, which doesn't seem to be findable makes me scratch my head. Is there some other document, that is more explicit, or is this just supposed to be implicit knowledge, or what?
[0]: https://www.facebook.com/whitehat
But I do recognize that cracking passwords goes a step too far.
"[Alex] then explained that the vulnerability I found was trivial and of little value"
coupled with the fact that he seemed to be very worried about the problems that could be caused by the author in exploiting it. Something seems amiss.
[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.ht...
[2] http://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.htm...
Facebook considers the keys to their kingdom to be worth $2,500. OR Facebook doesn't know what the keys to it's kingdom look like.
Facebook will not update keys/credentials even if they are known to be compromised.
If you have the keys to the kingdom, you can use them and Facebook won't find out about it unless you tell them.
https://www.facebook.com/notes/alex-stamos/bug-bounty-ethics...
If Alex knows anything about his job he should know that he has to refresh all those keys even if Wes didn't report it or say anything.
The diff between Wes and everyone else is Wes just explained to Facebook how completely screwed they are. Alex is just pissed because Wes made it bluntly clear how much he screwed up.
Select the most senior security person at those companies.
Roll 1d10 and substitute that person for Alex in this exact situation.
Now bet your life that you won't have your life wrecked by a prosecutor based on the outcome of that die roll.
I don't love Stamos calling the guy's boss, but if it's between "call his boss" and "tell legal that a bounty participant has FUCKING GONE ROGUE WITH ALL OF INSTAGRAM'S CREDS", I think he made the right goddamn call.
Jesus.
What I get from your comment is that it's never a smart move to take one's chances dealing with company security people. The only smart move is to sell anonymously to the highest bidder.
False dichotomy - those weren't his only options, had he bothered to think more on it. There was an even better option, which strangely he chose not to take (assume an actual rogue actor got there before Wes and react accordingly: rotate the AWS keys, password reset for affected users, update SSL signing keys).
It bears asking - what exactly was he trying to achieve by calling Wes' boss, and has he achieved it? This is not his brightest moment.
The competent responses would be:
"We DO have evidence that X DID NOT happen", or
"We DO have evidence that X DID happen".
A bag of rocks also has "no evidence that Wes or anybody else accessed any user data". Would you trust a bag of rocks with your computer security?
I understand that they're top secret, but that sort of proves the extent of the vulnerability.
Facebook needs to get its shit together in key security and clarity of its bounty program. On the other hand this guy writing a blog about downloading a keychain and probing how deep it leads is definitely not responsible infosec.
There are 4 categories of reporters: great, good, shit and crazy. Again- if you are a reporter, you should be trying to make it easy for the team to distinguish you in one of the first two categories by being simply being polite & respectful.
I will take a side- it's Facebook. Dumping data is the end of the Proof of Concept. Trying to determine if there is more data you can access through a single vulnerability chain is over the line.
Boats sink. The engineers know it. If you sink a boat in order to prove the boat had a hole, you will not get your payout.
And one final thought-
In my experience, bounty hunters almost never realize the full consequences of a vulnerability that receives a reward. Most of the time, the "Bad thing" that they identify is just the tip of the iceberg.
The choices of the researcher reflect inexperience and immaturity. The researcher has a significant misunderstanding about what is happening in the bug bounty marketplace. I think they need to apologize if they want a future in the infosec world.
Publishing this blog post was a huge error. Going to the journalist was another huge error. I don't see how this person could ever be considered employable by a reputable company.
I am saying explicitly- Wes went past the point at which he should have stopped.
He also should have known better, and the fact that he didn't is a problem in itself.
The problem is that on the one side you have security professionals who do this full time. They build up a background of implicit knowledge through extensive interaction with other security professionals, via training, mentoring, team activities, etc.
On the other side you have folks like the guy who found this vulnerability -- don't specialize in security, basically moonlighting / hobby, not necessarily connected to other security professionals or even other hobbyists. They won't have the same kind of implicit knowledge.
When someone from the first category communicates with someone from the second category, the communication can break down. That's what happened here.
Offering a million dollar bounty makes this kind of communication problem more likely -- a potential million-dollar payout catches the interest of people who have spare time and encourage them to pick this as the thing they do on the side. And further, encourages them to try anything and everything you don't explicitly forbid in by giving them hope that if they just try hard enough, they'll be able to turn what initially looks like a ho-hum two-year-old Ruby exploit into a million-dollar payday.