Impressive display of persistence, stringing together those vulnerabilities. I also see your English has gotten noticeably better :) Keep up the good work!
8 hours at 400$/hour will still only be 3200$ and he can presumably spend the remaining 4-3 hours doing more security analysis with less overhead, so it might still be cheaper to hire him as a consultant.
True, there are always bugs and security issues, but security issues tend to ramp even quicker than general bugs up from trivial to find to very, very difficult to find, so finding bug n+1 may be substantially harder than finding bug n.
Given a reasonably competent development team, you can usually make a first pass and find quite a number of low-hanging fruit security issues. Everyone makes mistakes, especially when under pressure to get a product out. Once those are gone you can use fuzzing and/or static analysis type techniques to find another set, but after that you get to the point where the bugs start getting quite obscure and require a fairly deep knowledge of how the system works so you can start stringing multiple problems together to get to a real security issue.
Of course this can be offset somewhat by the fact that software is usually a moving target, so if you're security testing a live, active codebase the developers are likely introducing new issues all the time, though hopefully at a reduced rate as they learn from their previous errors.
They launched bounty, this was the motivation to check things i always wanted to check. It took me about 4-5 hours, most of that time I was watching TV shows.
But they'd have to pay those $3200 without knowing if there were results. They might have to pay dozens of such consultants before one of them found bugs like this. Bug bounties, paid only on successful discoveries, are much cheaper.
But also much riskier. What if it transpires that the $4000 isn't enough? We know roughly what they're paying now, so when people find an issue like this they know they could sell it for much more.
I'm pretty sure Egor's first language isn't English, so OK might mean 'meh it's alright' through to 'hey this is great'. I know a few non-native speakers who do similar things.
As soon as I saw the new bounty program the first thought through my head was "Any Github Hacking leaderboard without homakov at tthe top is an inaccurate one". Congrats on your newest discovery!
If we're shaming any code with security flaws, no one is free of shame. I'm excited by the bounty program, it's a great way to get things like this identified and responsibly disclosed
Remember that you only see the interesting stories and successful investigations. Before making such a decision you should try to arrange a chat with someone already doing comp-sec, and figure out how much time they spend on all the other stuff.
I'd put this in the same category as mobile app dev. There are a few people making money by the truckload, plenty of people making a decent living, and lots of folks who strike out.
If it's something you're interested in, go for it. I just worry that people see this like the promise of gold in a faraway land and go rushing in, not thinking about the real distribution of success.
I totally believe that he's worth that amount of money. I'm sorry if you thought I was questioning that. I'm questioning the juxtaposition of his hourly rate with a request for donations.
I think the contract makes sense for clients, and the donation makes sense for other security researchers who want an incentive for him to keep publishing ideas.
Understood. But I imagine that his work isn't quite as "steady" as one might expect. He invests time by trying to find security exploits in hopes that the affected company compensates him. He doesn't set his price or even determine if he gets paid for his time.
I think that might be the rationale...or it might just be that he's found himself in a position where he can collect bounties AND donations :).
At least in my experience, I donate to groups that do good work but aren't getting paid for it. I wouldn't donate to people who are being paid (quite handsomely, in this case) for their labor. Especially when he's already clarified that GitHub paid him more than he thought his time was worth.
Perhaps you could clarify that part in your future posts, to appease the Internet haters on both sides. "I do paid contract work. However I also spend lots of time fixing open source stuff for free. If you want to encourage me to keep doing the latter, here's how to donate."
Donate or don't donate, that's your call. But why are you complaining about him asking for a donation? Why try to "shame" him? What is he doing to harm you?
Not sure why you're viewing my comment with such hostility. I was mistakenly under the impression that most of his work is contracted / bounty. He's already clarified his reason for accepting donations below, and I understand. I just think the placement/wording was less than ideal.
I doubt Egor is being paid for posting these summaries to his own blog for all of us to see. Even if he weren't contributing code to various libraries and applications, these write-ups are a great benefit to everyone else who has yet to be a target.
Send him an e-mail saying, "Hey, I sent you $100. I would deeply appreciate it if you spent it on your beverage of choice, or a nice dinner with a friend, rather than on necessities."
Charging $400/hour does not mean he does not need extra money. His nature of business is a short term projects, it's not like a regular web developer who has to work 40 hours a week for many month to finish a project, he only does audits which don't last long because of that you see this "high" (I personally don't think it's high) hourly rate.
It's actually a good strategy to price high hourly but over-deliver (doing lots of free work behind the scenes, or speculative unpaid work, etc.) -- rather than the market-clearing rate of ~100-150/hr, at least when you're trying to build a brand. At $400, he's clearly a specialist, and will get more interesting work; at $100/hr, you could hire him and just treat him like another developer, have him do cookie-cutter assessments, etc.
Personally, I think he'd make more money at $400-600/hr if he could also get some kind of manager to handle the interactions with clients; it doesn't seem to be what he enjoys, or is particularly good at.
(I've had drinks with him before, so probably the most effective way to accomplish my goal is to buy him drinks when I'm in town.)
Personally, I think he'd make more money at $400-600/hr if he could also get some kind of manager to handle the interactions with clients; it doesn't seem to be what he enjoys, or is particularly good at.
Completely agree. I'm not doing security, but my hourly is similar, and it was a game changer for me to have someone in a manager-like role working with me. Client relations are a huge time suck, but are also absolutely necessary. If he can find someone (or maybe someone on HN should volunteer), it'd be more than worth it.
BTW My manager takes a flat 15%. I'm much happier, clients are way happier, and my total income has increased as a result—not to mention another person is gainfully employed at something they're good at and enjoy. A win-win all the way around.
I meant manager in the sense of a band's manager. It's more than just introducing you to deals; it's handling the communications back and forth with the client on an ongoing basis; not merely the negotiation to set up the deal but the actual work-product communications as well.
Sure, I had a similar first reaction, but thought about it. If you have skills but haven't yet developed a deep-enough client base, you're in a quandary. You can't bill for $10/hour, or no one will take you seriously. You need perceived value, so you have to quote some reasonably high rate, even if you case-by-case discount it or work gratis.
(At least that's how I imagine it must work. I've never consulted.)
Although you probably should factor in the possibility of several years of compulsory $0.30/hr labour, plus forfeiture of all your ill-gotten gains (and probably some healthily-gotten ones too, they're not so fussy)
And that's before legal costs and possible restitution.
If @homakov is finding security holes without access to Github repositories, imagine what he'd find if you had him code audit for a few days... He's clearly been going about this the proper white-hat way and ensuring holes are patched before open disclosure... what's there to lose?
On the flip side, you could go about doing what you're doing under the presumption nobody is maliciously targeting your user base. In this scenario, it's possible you have a couple bad actors that see a net benefit greater than your bug bounties and are silently stealing and selling supposedly secure code from your users. You could be supporting a hacker black market where they sell and trade codebases to popular online sites. Imagine how easy it would be for them to find vulnerabilities in these sites if given access to the source code.
Completely agree, GitHub private repos are a huge target. Even if you use 2FA, after login it's just a cookie that separates the good from the bad. How could GH improve that? Client-side SSL Certs?
I'm pretty sure many more codebases have been lost through failures to secure internal networks by corporate IT departments than through vulnerabilities in cloud hosting providers.
I think he is referring to many people failing to secure their networks and having code stolen. It can be just as insecure, if not worse than a cloud provider if done wrong.
Important storage can be done 'in the cloud', but you need to audit and verify the cloud vendor is providing the proper controls. Just like you need to do 'privately'.
For code projects that are between me and a couple of other devs, none of whom are infrastructure security experts, I trust a company like Github a lot more than one of us trying to hack something together on a server.
With the exception that if you have three guys hacking something together a dedicated server or a box off your cable modem, with git tunneled over ssh using keys and a proper firewall, you'd probably be miles ahead. That might take you an afternoon to set up with almost no experience.
Not to say that it couldn't be compromised, but your not a target like github might be. If you're working with an enterprise level project with more complex auth and access methods, more users, performance and scaling needs, you'd need a real security implementation.
Yeah, the first Github and Rails exploit is the one that still sticks out in my mind. That kind of thing can be hard to shake, but it helps that you were quite young at the time. I'm happy to see you've matured a lot since then.
I don't think that is a fair assessment of him, even then.
At any case, I hired him fairly recently for a security audit and he worked quickly, and was very effective (he found several important vulnerabilities and reported them in a crystal clear manner). He was also a pleasure to deal with (no bullshit stance, something I find enjoyable).
The 4000 USD for ~20 hours of work were definitely well spent!
The parent was asking why Github haven't hired him, not why nobody has hired him. If you remember, Github actually banned him for hacking the Rails account in his pentesting.
I'm incredibly interested in angling my career towards security and have no real experience.
Wouldn't it also be wise to keep people like him 'out of the loop', I imagine it's much harder to audit when they have access to internal code/architecture that would be difficult for an outsider to stumble-upon?
I think his behaviour was commendable - he tried many many times to warn, going to multiple people and projects, but they all ignored him - they were too busy being Gem installing Ruby hipster Brogrammers to consider security, and it bit them hard in the backside.
he gets paid $400/hr doing consulting for YC Companies and other startups and companies, he is from Russia, and now lives in Bangkok, when he becomes rich he wants to live in Hong Kong, pretty nice for a 20 year old, I don't see any glaring reasons to work for Github http://egorhomakov.com/
400 USD/hour is a great rate anywhere in the world, even the most expensive cities.
But abcd_f's comment is right about one-off 4-hour projects vs. long-term contracting. Non-billable time overhead spent on finding clients, negotiating contracts, mentally switching projects, or just sitting idle can negate the benefits of a high hourly rate.
But hiring him offers no guarantee that he will be able to find any other bugs.
That's the beauty of bounties, it allows people to decide whether they want to do the right thing or not, if there was no bug bounty more people are just tempted to exploit the bug.
How can I start learning about how to identify exploits like this? I know some basics about web application security and work as a software engineer on a day-to-day basis but security has always been a passion of mine and I have always wanted to be able to support myself through working on security alone (by collecting rewards through bounty programs, self-employed security consulting, working at a security consulting firm like Matasano, or some combination thereof) but I don't know where to start. I want to learn the ins and outs of web application security instead of just understanding the OWASP top 10 and having a strong interest in certain topics (like HTTPS/SSL vulnerabilities). When I read disclosures from people like Egor I grasp the steps they are taking to craft an exploit like this as they are explained but I don't know how to identify these exploits on my own.
Can anyone recommend some reading material or some first steps I can take to work towards moving to a more security-focus career?
Like a lot of other things, practice matters. OWASP has some deliberately insecure webapps which are meant to give people practice spotting and exploiting vulnerabilities (WebGoat, RailsGoat, PyGoat, probably others). There are also "capture the flag" competitions of the sort run every so often by Stripe; Matasano currently has one going as well, focused on embedded systems:
See my comment below for Web App hacking, my personal favorite, but if you enjoy working in the lower level and reading assembly then the matasano ctf is very well made.
Github uses ruby on rails, which is a pretty mature framework, perhaps covering most of the common security pitfalls. Additionally, I assume github has excellent programmers because of the nature of their job.
Could someone explain in simple english, how did they overlook known & well documented bugs that got them hacked (e.g. Bug 3 about cross domain injection). I'm wondering if someone of Github's caliber can be hacked so easily, what about the rest of the masses developing web apps. Especially all those new crypto-currency exchanges popping up left & right.
I've been toying with Django. Reading through the docs makes me feel that as long as I follow the safety guidelines, my app should be safe. It feels as if they've got you covered. But this post rattles my confidence.
As briefly as possible? Infosec is hard. Most companies have virtually no security policies. Nobody listens. Black hats are ahead in the arms race and anyone who has decent knowledge (doesn't even have to be anywhere near on a level like Homakov or Zalewski) can pull off all sorts of exploits. Even if they don't strike the application itself, they'll get you through infrastructure that your application relies on. Look at how script kiddies like the SEA can pull off high-profile hacks through social engineering, domain and DNS hijacking.
It's assured that a ton of Rails apps are vulnerable, it's just that no one has found them, or more likely, is not publicly releasing or actively exploiting them.
Also, Rails doesn't address for all security pitfalls. Some of its mechanisms are actually underdeveloped and require rolling lots of checks by yourself, such as for proper session termination, IIRC.
This highlights to me that our infrastructure is horrendously overcomplicated. We have all these great abstractions, but you have to worry about bugs and exploits in every possible layer of every system. Even the simplest modern web-application has an enormous surface-area to secure, and that makes getting it "right every single time" damned near impossible.
No one gets it right every single time. No one. That's a completely unrealistic expectation. What you do is establish a bar, which you share with everyone who will use your software. Then you evaluate your efforts against that bar.
One of the keys to developing good software is hiring third-parties to conduct audits. A bug bounty program is one way to incentivize people who are already probing your software to take the next step and tell you about the bugs they find.
This is a little myopic but understandable in the context of a discussion on HN. Infosec is hard,
but it is just one example of a bigger truth:
Defense is hard.
This comes up time and time again in any defensive discipline:
Over two decades the CIA had learned again and again that it could not hope to
defend against terrorists by relying solely on its ability to detect specific
attacks in advance. No matter how many warnings they picked up, no matter how
many terrorist cells they disrupted, at least some attackers were going to
get through. Officers in the CTC privately compared themselves to soccer
goalies: They wanted to be the best in their league, they wanted to record as
many shutouts as possible, but they knew they were going to give up scores to
their opponents. Ultimately, many of them believed, the only way to defeat
terrorists was to get out of the net and try to take the enemy off the field.[1]
The final sentence above highlights the one pecularity of InfoSec; you do not have any
offensive capabilities.
This is why I think some more work into client (or active) honeypots may be beneficial. If we can get an easy to install, auto updating honeypot that fights back, we may have a better offensive capability.
This may just end, like nuclear warfare, in MAD... But it would be great fun to watch!
It's worth mentioning that Github has forked Rails and is working off their own private branch of Rails 2.3. Not saying that was relevant to this exploit, mind you.
It's a dream job for developers, in some ways a lot more so than the big boys like Google and Facebook. They have a hiring pipeline any tech company would kill for. They probably don't have the deep security talent that say Google or Microsoft have, but they should have enough.
Really? I'm not so sure, AFAICT Github doesn't have any new or interesting problems to deal with. It's just a Rails app that's constantly developed on. You can do that, well, anywhere.
Having met several of them and spent a day in their offices (they gave Kiva engineering a tour day a couple years back), I can say they have an awesome company culture and space and great leadership and great brand recognition. Plus, people generally like them. I'd put it near the top of my list if I had one.
I'm wondering if someone of Github's caliber can be hacked so easily, what about the rest of the masses developing web apps.
They're all pretty bad. SQL injection was a boondoggle for years until people wised up, or more likely moved to the then-newly-popular ORMs, but it still got Bell Canada recently. Target is #36 on the Fortune 500. That wasn't a webapp based attack, but even companies of their considerable resources still get security that wrong. Sure, you can tell yourself a startup is more tech focused and better positioned to get security right. But do devops building for server stacks and platforms they don't fully understand while pushing code multiple times a day really have both the skills and time to focus on security?
@homakov finds 5 different bugs with github and manages to align them so that a bigger vulnerability is exposed in under 5 hours? That's amazing! I used to think I'm a fast delivery-focused developer but I'm probably just a fraction of how fast some people are.
Something I don't really understand is why people use github in the first place when it's so easy to set up your own git repos with gitolite.
I understand using github for large open source projects that you want people to freely branch, but it seems the vast majority of accounts are small personal projects and even some small companies.
$400 is such chump change compared to the PR disaster that can come from exploited, or even just leaked, vulnerabilities. I honestly think any SaaS needs to have this somewhere in their budget once a year.
It would be great for educational purposes if a sample app was setup so this vulnerability could be tried on it. Most of the white hack vulnerabilities are fixed by the time white hat blog posts come out so there is no way to actually try them out.
201 comments
[ 3.4 ms ] story [ 125 ms ] thread$4000 !? Wow, I'd love to be able to make $4000 on the side just doing what I love.
> Interestingly, it would be even cheaper for them to buy like 4-5 hours of my consulting services at $400/hr = $1600.
This sounds like a pretty clever strategy for marketing yourself as an effective security consultant.
EDIT: $4000!? wow. so money. such big.
Given a reasonably competent development team, you can usually make a first pass and find quite a number of low-hanging fruit security issues. Everyone makes mistakes, especially when under pressure to get a product out. Once those are gone you can use fuzzing and/or static analysis type techniques to find another set, but after that you get to the point where the bugs start getting quite obscure and require a fairly deep knowledge of how the system works so you can start stringing multiple problems together to get to a real security issue.
Of course this can be offset somewhat by the fact that software is usually a moving target, so if you're security testing a live, active codebase the developers are likely introducing new issues all the time, though hopefully at a reduced rate as they learn from their previous errors.
I would make this your tagline in some way -
"I will find vulnerabilities. If I don't, I will become a vulnerability to my own body and attack myself until I do!"
http://en.wikipedia.org/wiki/Yakov_Smirnoff - referred to as a Russian Reversal
Ok, learned something completely wasn't aware of before.
But, no, no intent to make that kind of joke.
http://krebsonsecurity.com/tag/0day/
http://www.sakurity.com/hustlers
Is there an easy way to see what vulnerabilities other websites have had and fixed, and to check if your site has them as well?
I see that your a sysadmin so if network hacking is more you speed I would download Metasploit[2] and start hacking old linux or windows distros.
[0]http://www.amazon.com/The-Web-Application-Hackers-Handbook/d... [1]http://itsecgames.blogspot.com/2013/07/bee-box-hack-and-defa... [2] http://www.metasploit.com/
If it's something you're interested in, go for it. I just worry that people see this like the promise of gold in a faraway land and go rushing in, not thinking about the real distribution of success.
Maybe it's just me, but asking for donations after saying you bill clients at $400/hr seems weird to me. I wish I could bill at that rate.
I think that might be the rationale...or it might just be that he's found himself in a position where he can collect bounties AND donations :).
There were always people complaining "Add a donate address"
Now "why you added a donate address". Oh, Internet.
You are giving people that you have helped an opportunity to pay you without having any kind of contract with them.
Nothing wrong with that at all.
He also commented on his site that he "is poor", so it could be that he simply hasn't landed enough gigs @ $400/hr to be in good financial shape yet.
There's a clearly huge market.
http://www.youtube.com/watch?v=kQFKtI6gn9Y
Ad supported. Abusive ads berating potential users are encouraged.
Get on that before someone filches it!
Point, set and match.
It's no guarantee, of course. :)
Personally, I think he'd make more money at $400-600/hr if he could also get some kind of manager to handle the interactions with clients; it doesn't seem to be what he enjoys, or is particularly good at.
(I've had drinks with him before, so probably the most effective way to accomplish my goal is to buy him drinks when I'm in town.)
Completely agree. I'm not doing security, but my hourly is similar, and it was a game changer for me to have someone in a manager-like role working with me. Client relations are a huge time suck, but are also absolutely necessary. If he can find someone (or maybe someone on HN should volunteer), it'd be more than worth it.
BTW My manager takes a flat 15%. I'm much happier, clients are way happier, and my total income has increased as a result—not to mention another person is gainfully employed at something they're good at and enjoy. A win-win all the way around.
(At least that's how I imagine it must work. I've never consulted.)
And that's before legal costs and possible restitution.
</jokeruiner>
On the flip side, you could go about doing what you're doing under the presumption nobody is maliciously targeting your user base. In this scenario, it's possible you have a couple bad actors that see a net benefit greater than your bug bounties and are silently stealing and selling supposedly secure code from your users. You could be supporting a hacker black market where they sell and trade codebases to popular online sites. Imagine how easy it would be for them to find vulnerabilities in these sites if given access to the source code.
That, my friends, would be a catastrophe.
Verify.
Important storage can be done 'in the cloud', but you need to audit and verify the cloud vendor is providing the proper controls. Just like you need to do 'privately'.
Not to say that it couldn't be compromised, but your not a target like github might be. If you're working with an enterprise level project with more complex auth and access methods, more users, performance and scaling needs, you'd need a real security implementation.
At any case, I hired him fairly recently for a security audit and he worked quickly, and was very effective (he found several important vulnerabilities and reported them in a crystal clear manner). He was also a pleasure to deal with (no bullshit stance, something I find enjoyable).
The 4000 USD for ~20 hours of work were definitely well spent!
Wouldn't it also be wise to keep people like him 'out of the loop', I imagine it's much harder to audit when they have access to internal code/architecture that would be difficult for an outsider to stumble-upon?
https://news.ycombinator.com/item?id=7136027
But abcd_f's comment is right about one-off 4-hour projects vs. long-term contracting. Non-billable time overhead spent on finding clients, negotiating contracts, mentally switching projects, or just sitting idle can negate the benefits of a high hourly rate.
That's the beauty of bounties, it allows people to decide whether they want to do the right thing or not, if there was no bug bounty more people are just tempted to exploit the bug.
Can anyone recommend some reading material or some first steps I can take to work towards moving to a more security-focus career?
Thanks.
http://www.matasano.com/matasano-square-microcontroller-ctf/
Could someone explain in simple english, how did they overlook known & well documented bugs that got them hacked (e.g. Bug 3 about cross domain injection). I'm wondering if someone of Github's caliber can be hacked so easily, what about the rest of the masses developing web apps. Especially all those new crypto-currency exchanges popping up left & right.
I've been toying with Django. Reading through the docs makes me feel that as long as I follow the safety guidelines, my app should be safe. It feels as if they've got you covered. But this post rattles my confidence.
Cheers !
It's assured that a ton of Rails apps are vulnerable, it's just that no one has found them, or more likely, is not publicly releasing or actively exploiting them.
Also, Rails doesn't address for all security pitfalls. Some of its mechanisms are actually underdeveloped and require rolling lots of checks by yourself, such as for proper session termination, IIRC.
Here is some OWASP material specific to Django.
If you like reading http://blog.mikeleone.com/2011/10/security-django-and-owasp-...
If you like watching http://www.youtube.com/watch?feature=player_embedded&v=sra9x...
In computer security, you have to get it right every single time. The bad guys only need to get it right once.
One of the keys to developing good software is hiring third-parties to conduct audits. A bug bounty program is one way to incentivize people who are already probing your software to take the next step and tell you about the bugs they find.
until that's different it's harder to answer your actual question. my guess, it'll be better but inevitably still have some holes.
It seems to me that it would drastically reduce the surface areas of attack.
Defense is hard.
This comes up time and time again in any defensive discipline:
The final sentence above highlights the one pecularity of InfoSec; you do not have any offensive capabilities.[1]: "Ghost Wars" (Steve Coll) pg 505
This may just end, like nuclear warfare, in MAD... But it would be great fun to watch!
http://en.wikipedia.org/wiki/Client_honeypot
http://books.google.com.au/books?id=YQmWtsqlvfMC&dq=active+h...
http://en.wikipedia.org/wiki/Mutual_assured_destruction
https://github.com/github/rails
http://www.kalzumeus.com/2013/06/17/if-your-business-uses-ra...
> I . . . decoded _gist_session cookie (which is regular Rails Base64 encoded cookie)
In Rails 4 the session cookie is encrypted with a server-side secret, so the end user can't decipher it.
What do you think makes Github that much better than all the rest?
Really? I'm not so sure, AFAICT Github doesn't have any new or interesting problems to deal with. It's just a Rails app that's constantly developed on. You can do that, well, anywhere.
They're all pretty bad. SQL injection was a boondoggle for years until people wised up, or more likely moved to the then-newly-popular ORMs, but it still got Bell Canada recently. Target is #36 on the Fortune 500. That wasn't a webapp based attack, but even companies of their considerable resources still get security that wrong. Sure, you can tell yourself a startup is more tech focused and better positioned to get security right. But do devops building for server stacks and platforms they don't fully understand while pushing code multiple times a day really have both the skills and time to focus on security?
Just a quote while we're at it..
>> We make expand and then defense it.
But github, seriously? Why do you guys fail so hard at security?
Too much Brogrammer rather than programmer methinks.
Friends don't let friends code in Fails frameworks.
I understand using github for large open source projects that you want people to freely branch, but it seems the vast majority of accounts are small personal projects and even some small companies.