I happily created a new Github repo, imported some code, made some changes and pushed stuff, all OK. And just now when I want to post announcements of my new library, bam! Github is down.
Mostly I don't care if they are periodically unvailable, but the timing on this was amusing.
2:01 UTC We're still experiencing a large DDoS attack. The site is experiencing major packet loss and is mostly unavailable. We're working to further mitigate the attack.
Genuine question – who DDoSes Github? From what I've seen, DDoS attacks often seem to be a response to a site or company entering the political sphere in a controversial way.
In previous threads about Github being DDoS'd, I asked the same question.
Apparently it's some scheme to ask ransom from companies/startups that have enough money to pay and/or not enough infrastructure to sustain the attack without giving up.
Yep. There's a well-written article about an early DDoS extortion attempt where the victim fought instead of paying, developing some novel protection technology in the proces: http://www.csoonline.com/article/print/220336
Yea, all of us successful web companies face these all the time. We get them multiple times a day. Most of us end up giving up and pay for Arbor gear and ridiculously over provisioned pipes as well as good relations with our hosting facilities.
Unfortunately this occurred at the exact moment I was deploying some changes. Makes me think that being so dependent on GitHub for a production code repo is a little big dangerous.
Redundancy would certainly solve this issue, though.
There's already built-in redundancy. Wherever you pushed to Github from has a complete copy of the current state of your repository. Github is effectively just a publicly accessible mirror, but you can always push to a different remote elsewhere, push directly to your production instance, or even host your repo in an s3 bucket, all while Github is down.
I've been in some development environments where the GitHub master branch is what the build system pulls from in order to deploy to dev, staging and production. With that setup, it's more challenging since the build process probably involves more steps and potentially ssh keys that are only allow the build server to deploy.
Depending on how many repos you need, I would say set up an alternative remote on bitbucket.org.
Right, I get that...I meant a separate remote when I said redundancy. GitHub has just been so reliable I haven't even needed other remotes for the past 8 or so months we've been using it like joeblau mentioned.
Guys, please don't forget the advantage of using DVCS to begin with. I like Github too, but if Github goes down it shouldn't put the brakes on your work. Push to multiple remotes. Solved.
I've been telling people for a fair while that I wish Github issues were stored as part of the git repos themselves, not so much for availability reasons as that it'd be awesome if the state of bugs got branched and merged along with everything else, so that a bug could be fixed in a branch, but not yet in master, and would get fixed in master when the branch got merged. But I guess availability would be a nice additional perk.
If you use the "fixes #7" syntax in a branch commit, the issues will be closed only when that branch is merged to your default branch (usually master). I realize that this is only part of what you desire, but it's a useful recent addition: https://github.com/blog/1386-closing-issues-via-commit-messa...
CDNs are only useful for things that can actually be cached, and which have been accessed recently. Static content is easy, and I'm sure Github does it, but dynamic content doesn't benefit as much for a CDN.
As another note, anyone doing a DDoS is going to target parts of the system that are more intensive. They'd just skip the cached things and go after the ones that have to be regenerated.
That makes sense, but can't a CDN also help identify the DDoS and block communications from certain servers. I remember seeing a captcha on Adria Richards website after she got those two devs kicked/fired. Maybe GitHub just needs better tools to identify and block the DDoSes quicker.
29 comments
[ 0.21 ms ] story [ 74.2 ms ] threadMostly I don't care if they are periodically unvailable, but the timing on this was amusing.
https://status.github.com/messages
Today
1:45 UTC Major service outage.
Apparently it's some scheme to ask ransom from companies/startups that have enough money to pay and/or not enough infrastructure to sustain the attack without giving up.
Redundancy would certainly solve this issue, though.
Depending on how many repos you need, I would say set up an alternative remote on bitbucket.org.
The protocol could even be email based.
e.g. a pull request is an email with the git formatted patch attached. Issues are emails with discussions as threaded mail/news, etc.
As another note, anyone doing a DDoS is going to target parts of the system that are more intensive. They'd just skip the cached things and go after the ones that have to be regenerated.