The so what is that the majority of people on HN are developers and the majority of them use Github either personally or with their teams. So a major outage is costing potentially millions of dollars in potentially lost developer time. If you calculate an average hourly of just $50 and a developer loses 20 minutes. Then that outage costs $17 x the number of affected developers. That's a pretty big loss.
I'm reading HN right now because I can't manage pull requests and do code reviews of my distributed team's code. So I'm just spinning in my chair.
You can't pull code from another node, i.e another developer? Git is distributed, if you want centralised source control use Perforce or SVN or something.
sure, and I can build a commenting and pull request system while I'm at it. It's highly inefficient to change workflow for an outage that might not last an hour.
Github.com is a profitable entity that makes a fairly large amount of money. Additionally, Github.com hosts a lot of content. Content that other people may not like.
Possibilities are: 1) Extortion of some sort i.e. We want your money, or we'll DDoS you every week 2) Content i.e. Remove repositories a, b, and c with content we don't like, or we'll DDoS you every week 3) Weapons demonstration i.e. We want to show off what l33t hax0rs we are, so we'll DDoS Github.com every week
So maybe they can build a distributed system themselves instead of making a point the f*c%s everyone while they have their childish tantrum. This is like nuking a city because you don't like the mayor.
blackmailing ? "if you dont pay we'll DDOS you" or whatever. Given all these infected Windows computers all over the world ,it's not going to stop soon. That's the Microsoft legacy.
Am I crazy in thinking that there's zero chance Github would pay a "ransom"? Seems like they'd just work with their vendors/providers and mitigate it to avoid setting that precedent (making them even more vulnerable to future attempts).
The attacker's bet is that they can apply pressure beyond the victim's ability to mitigate. If they win that bet, the victim may reach a point where paying out is preferable to the lost business they'd suffer because of the attack.
It could be to cause a failure in an ancillary github service that results in opening up a new attack vector which could then allow the attackers to steal code from private repos.
There is at least one Vagrant+puppet installer I know of but, well, uhhh... it's on Github and won't help you much at this point :P https://github.com/sbadia/puppet-gitlab
I've been very happy with Gitlab. The CI server seems a bit simplistic at this point and I haven't tried it, but I'm slowly moving all of the stuff I need to have more consistent access to, from Github.
GitLab co-founder here. Thanks for commenting druiid. GitLab CI 3.1 will have very nice integration with GitLab 6.0. It will still be simple, but should cover most cases.
Does anyone else find it ironic that Git is a distributed version control system, yet we rushed to centralize it and base our entire workflow around GitHub?
Github is just a node. IF Github dies , people still have the full repos on their computers.That's not always the case with SVN... And most people are too lazy/dont have resources to set up a git server anyway.
Most teams that work on GitHub have none of their own workflow or infrastructure for sharing code when GH goes down.
You can probably blame those teams for that more than GitHub but people do seem to have bought into the idea that we can just jam everything up to GitHub and get it back later.
Of course from a risk-management perspective, the likelihood of a long term outage is statistically low enough to not justify the expenditure for setting up and maintaining alternative systems (other than simply having repos backed up.) It's the anti-TSA approach. The TSA spends billions and hassles everyone despite the relative rareness of air-travel attacks. But, the severity of a potential attack weighed against the expenditure has led politicians to believe that the cost is worth it.
The Heroku/AWS outages a few years back had far more of an impact, yet Heroku still (to my knowledge) relies exclusively on AWS-East, because presumably the risk-profile doesn't exceed the threshold to justify the expenditures required to mitigate the risk.
I just wish the attackers would be considerate enough to share the schedule for these attacks ahead of time, so I can plan a longer lunch. The rudeness of these attackers is unparalleled. I'm inclined to write them a strongly worded letter suggesting same.
The problem isn't developer access to the code, but that GitHub gets tightly linked to other tools, such as continuous integration and test tools. If Jenkins is expecting to pull the latest code from GitHub, but GitHub is down it's a PITA.
There's also stuff like GitHub Issues, which are separate from the actual Git hosting. So even if the source code is available and the 3rd party tools aren't broken, work can still be interrupted.
Nobody denies your blobs are safe and available elsewhere when Github goes down.
It's the infrastructure around those blobs that makes Github so popular. It's your issues and pull requests, your wiki, your connection to Travis-CI that are gone when Github goes down.
I've tinkered with the idea of setting up a git repo on a hosting provider like Digital Ocean for just an occasion. Maybe back it up hourly from github, and/or also push to it regularly by adding it as a remote in our git repos.
Anything business related we use beanstalkapp, for its better price-per-private repo and deployments in any case (no issues with them ever!)
Have you tried Jenkins as a CI server? While the core Gitlab product is quite nice, the feedback I've heard on the CI component is that it becomes problematic once builds become non-trivial (though to be fair, I've heard this mostly from Jenkins folks).
We just built our own build scripts (in make(1) like god intended) and wrap jenkins around that. Pretty hard to go wrong with tech that's been tested for the last 35 years
I always wonder if people prefer to use a centralised model that github seems to promote, or if people are not tooled up to take advantage of the distributed nature of git?
How technically hard is it to have an internal roundrobin git system that's synced up with the github repos needed, and then redirect to the internal git servers as needed when github goes down? I'm just wondering, no snark.
Github will go down by the nature of it being a web service. Web services aren't 100% by nature.
We are using Gitlab for about half a year with Jenkins, and we could not be happier. We are about 50 devs working with it every day. It has some quirks (for example AD integration) but it has served us well, and we would not want anything else.
GitLab co-founder here, thanks for the comment y0ghur7_xxx. Please let me know if you have any idea's to improve Active Directory Lightweight Directory Service support.
Can someone with experience mitigating an attack like this describe how it's done? A known set of hosts/address spaces is fine, but it's the "distributed" part I don't understand how to deal with.
Usually there is some pattern to the traffic that can be identified and filtered out. There are manual ways to do this (iptables, basic firewalls) or more sophisticated "anti-DDOS" boxes you can buy to automate or simplify the process (e.g. Cisco Anomaly Guard or Juniper Junos DDoS Secure).
We don't know anything about the attack, but in general:
1. All of this requires that you have more bandwidth than the attacker.
2. ACL drop everything but the ports/protocols you use on your frontend IP. For github.com that would be TCP 80 and 443. (ACLs are cheap, can be done at wire-rate on any capable edge router, so they're a good "first step." In this case it would have the bonus of dropping DNS amplification attacks which are increasingly common.)
3. If it's a TCP SYN flood (probably from spoofed IPs) enable SYN cookies. (Of course now the bottleneck is their load balancer.)
4. If it's an L7 attack, they'll need to identify patterns in the requests to identify and drop the traffic. For example, the attacker may be requesting a single URL only. If it's an L7 attack on a TCP service, they can also automatically add a drop rule for that source address, because at this point the client IP will have been validated (via the SYN challenge).
They may also be able to identify certain patterns in the traffic that can be ACL'd at the edge. For example, IP TTL being identical.
There are services that specialize in handling DDOS attacks with distributed data centers and large pipes. It is very hard to do it on a shared infrastructure (e.g. AWS) w/o affecting other customers thus hosting providers will be more interested in protecting other clients.
Can someone with experience mitigating an attack like this describe how it's done?
I believe Step 1 is to submit the fact that GitHub is being DDoS'd to Hacker News, reddit, Slashdot and other social networking sites with a direct link to GitHub's web site. This will cause a flood of real people to load the web site, thereby crowding out the DDoS.
The site will still be down, but at least it will be due to real people and not the DDoS.
(Actually, there are long established yet less high profile alternatives with decent features and better pricing for commercial teams, like beanstalk or unfuddle or codebasehq. No hip cred, but for example supporting archived projects that you can read w/o using up your repo count license.)
Unfuddle, while good for its day, is not even close to GitHub. It's woefully out of date, slow, and temperamental in ways that are frustrating. The closest thing going is Bitbucket which is a pretty good competitor for GitHub...in 2010.
This argument also works the other way. We payed less for beanstalk and got half the features. Sure it was a remote git repo, but our use case for GitHub is more than just external storage, and we were able to remove other tools that we were paying for.
We've been developing custom software for over a decade. GitHub's pricing model counts every private repo, active or not, while others let you put repos into R/W or archive mode, or don't count repos at all and only count against your storage quota.
The difference in annual cost is significant when you have 20 active independent projects, and hundreds of archived projects, not to mention all the customers needing accounts and access.
I do not understand why GitHub get attacked so often - honestly it's not like they are doing anything wrong, they are a fabulous service and the go-to place for most (if not all) developers for all sorts of projects, be it open source or in an enterprise/corporate context.
The goal may be to potentially open up a new attack vector to steal code, some of which may contain secrets. This is pure speculation, but a definite possibility.
My guess would be attention and publicity. Targeting a technical, developer centered site is likely to require more "skill" as one would expect such a site to have their defenses ready for this. All the greater challenge. Secondly, every time this happens it causes quite a discussion on developer forums and sites such as HN. If you're seeking attention and "recognition" amongst technical people, what better site to attack?
Extortion, research, or for evil. Attacking Github may be an attempt to effect someone that uses it (Github may not be the direct target). What if it's possible to effect HFT trading.
---
We do know that they've been consistently attacked. Github has been unsuccessful in mitigating attacks.
Github needs to explain why these attacks are happening. What is being done to stop them. What has been done to stop them. And if they are different from previous attacks.
What a coincidence. We actually just finished migrating all our projects over to GitLab about an hour before this outage.
I still very much love GitHub, it just ended up not scaling for us (we have a lot of repositories that seldom needs to be touched, which results in a $20 / month Linode + Backup being a much better solution)
GitLab also allows us to group repositories and gives a little bit more flexibility in regards of git server-hooks. Also, server-side branch locking! (Does anybody know how to lock branches server-side with GitHub?)
You can have multiple remotes for your Git repositories, one of them (Github) being down should not affect your work since you can coordinate a new one with your team in the manner described above.
Yep! I usually setup a mirror at bitbucket and set my local to push to both github and bitbucket and only pull from github. It worked well when github was down and I needed a branch from a remote developer.
Snarky comments about distributed version control aside, the bigger problem is the ecosystems which revolve around GitHub. Two which are immediately screwing my day up are Composer and Homebrew. Currently, both are totally dependent on GitHub.
104 comments
[ 4.1 ms ] story [ 99.5 ms ] threadI'm reading HN right now because I can't manage pull requests and do code reviews of my distributed team's code. So I'm just spinning in my chair.
That's the so-what.
Sheesh.
Surely by now, GitHub must know who's responsible and putting more in place to mitigate as much as possible before this happens. Right?
https://status.github.com/messages
Anybody has experience with Gitlab and Gitlab CI? How's the flow compared to Github, especially for pull requests, commenting and collaboration?
Not a good situation.
Does anyone else find it ironic that Git is a distributed version control system, yet we rushed to centralize it and base our entire workflow around GitHub?
You can probably blame those teams for that more than GitHub but people do seem to have bought into the idea that we can just jam everything up to GitHub and get it back later.
The Heroku/AWS outages a few years back had far more of an impact, yet Heroku still (to my knowledge) relies exclusively on AWS-East, because presumably the risk-profile doesn't exceed the threshold to justify the expenditures required to mitigate the risk.
I just wish the attackers would be considerate enough to share the schedule for these attacks ahead of time, so I can plan a longer lunch. The rudeness of these attackers is unparalleled. I'm inclined to write them a strongly worded letter suggesting same.
There's also stuff like GitHub Issues, which are separate from the actual Git hosting. So even if the source code is available and the 3rd party tools aren't broken, work can still be interrupted.
Nobody denies your blobs are safe and available elsewhere when Github goes down.
It's the infrastructure around those blobs that makes Github so popular. It's your issues and pull requests, your wiki, your connection to Travis-CI that are gone when Github goes down.
That is why people get angry when it goes down.
Anything business related we use beanstalkapp, for its better price-per-private repo and deployments in any case (no issues with them ever!)
How technically hard is it to have an internal roundrobin git system that's synced up with the github repos needed, and then redirect to the internal git servers as needed when github goes down? I'm just wondering, no snark.
Github will go down by the nature of it being a web service. Web services aren't 100% by nature.
Then look at something like Gitweb [1], cgit [2], or Gerrit [3] to front the repo. I personally use git, gitolite, and gitweb.
[1] https://git.wiki.kernel.org/index.php/Gitweb
[2] http://git.zx2c4.com/cgit/
[3] http://code.google.com/p/gerrit/
1. All of this requires that you have more bandwidth than the attacker.
2. ACL drop everything but the ports/protocols you use on your frontend IP. For github.com that would be TCP 80 and 443. (ACLs are cheap, can be done at wire-rate on any capable edge router, so they're a good "first step." In this case it would have the bonus of dropping DNS amplification attacks which are increasingly common.)
3. If it's a TCP SYN flood (probably from spoofed IPs) enable SYN cookies. (Of course now the bottleneck is their load balancer.)
4. If it's an L7 attack, they'll need to identify patterns in the requests to identify and drop the traffic. For example, the attacker may be requesting a single URL only. If it's an L7 attack on a TCP service, they can also automatically add a drop rule for that source address, because at this point the client IP will have been validated (via the SYN challenge).
They may also be able to identify certain patterns in the traffic that can be ACL'd at the edge. For example, IP TTL being identical.
I believe Step 1 is to submit the fact that GitHub is being DDoS'd to Hacker News, reddit, Slashdot and other social networking sites with a direct link to GitHub's web site. This will cause a flood of real people to load the web site, thereby crowding out the DDoS.
The site will still be down, but at least it will be due to real people and not the DDoS.
(Actually, there are long established yet less high profile alternatives with decent features and better pricing for commercial teams, like beanstalk or unfuddle or codebasehq. No hip cred, but for example supporting archived projects that you can read w/o using up your repo count license.)
The difference in annual cost is significant when you have 20 active independent projects, and hundreds of archived projects, not to mention all the customers needing accounts and access.
- Getting people to use BB instead
- Fake DDOS to get more companies to pay for github:enterprise
- Preventing a system upgrade in a company which has deployment relying on github
There are lots of potential ways to use it...
---
We do know that they've been consistently attacked. Github has been unsuccessful in mitigating attacks.
Github needs to explain why these attacks are happening. What is being done to stop them. What has been done to stop them. And if they are different from previous attacks.
I still very much love GitHub, it just ended up not scaling for us (we have a lot of repositories that seldom needs to be touched, which results in a $20 / month Linode + Backup being a much better solution)
GitLab also allows us to group repositories and gives a little bit more flexibility in regards of git server-hooks. Also, server-side branch locking! (Does anybody know how to lock branches server-side with GitHub?)
git daemon --verbose --export-all --base-path=.git --reuseaddr --strict-paths .git/
Can you please elaborate?
You can have multiple remotes for your Git repositories, one of them (Github) being down should not affect your work since you can coordinate a new one with your team in the manner described above.
Here's how to set multiple remotes. (See answer 2). http://stackoverflow.com/questions/849308/pull-push-from-mul...