Random somewhat on-topic question. How does one measure ones technical capability to keep up services that are more available and dependent than third party solutions? I mean obviously starting out, almost everybody can benefit from doing a lot of stuff not in-house, but at some point you'd be able to do better yourselves. But then maybe for certain services from certain providers, raw technical capability never catches up?
How do you know when that point is? Is there even data out there on how frequently big providers go down? (Not just Github, but stuff like aws, etc)
By the time you're talking about 99.9% uptime there is only 9 hours downtime a year so there isn't much wiggle room for failure. There is never a reason any one person should assume they can do better than a third party because you're likely sleeping 2920 hours a year. Basically, you're never good enough on your own. If you care about uptime you're going to need to pay good money for it.
Redundancy. Lots of it. A simple example would be hitting your website with a script every so often and when it stops responding then update your DNS to point to another data center that's up and ready to go.
In short, no, unless you think people replacing broken hardware or patching bugs in your software can be automated.
Your question is too vague. Github is up enough that I don't care. However, it's down enough I wouldn't want to not be able to deploy because it's down. Therefore, I may mirror my repo somewhere else. That's easy because git is decentralized. It's a lot cheaper than running some alternative that I guarantee is always running. You can do this by simply pushing to a mirrored remote branch.
If you're just interested in the subject, research high availability.
In short, no, unless you think people replacing broken hardware or patching bugs in your software can be automated.
Your question is too vague. Github is up enough that I don't care. However, it's down enough I wouldn't want to not be able to deploy because it's down. Therefore, I may mirror my repo somewhere else. That's easy because git is decentralized. It's a lot cheaper than running some alternative that I guarantee is always running. You can do this by simply pushing to a mirrored remote branch.
If you're just interested in the subject, research high availability.
I think the general consensus for running infrastructure in house isn't superior uptime but superior control.
Generally, you don't care if GitHub [or your inhouse equivalent] is down when no one is working. Also, if it goes down in the middle of the night and the first guy in fixes it...the difference between 1 man hour down in the morning vs. 20 man hours during the day across an entire team is significant.
Global service should have at least 3 teams (Asia, Europe, America). 4 teams would be better. They can take shifts: Asia team joins the standup meeting of Europe, then goes home. And so on.
It might not be affordable for daily works, but perfect for critical issues.
How is this a useful comment? Everybody knows that Git is distributed. Everytime Github's down, somebody feels the urge to remind everybody else about that.
Github's not only Git. Lots of services integrate with Github. Stuff like continuous integration, continuous deployment. If your system is built on those things, Github being down will prevent you from deploying.
What's the alternative, replicate everything in-house/self hosted? Should startup stop using third party service providers?
If your workflow involves services that integrate with GitHub, you should have a plan for when GitHub is not available. You've got your source code elsewhere, you should have docs and procedures for running the tests in your own dev environments and for utilizing whatever staging/deploy process you prefer without GitHub as an intermediary.
I continue to post comments like that because while I love GitHub, watching the community put all their eggs in GitHub's basket, especially when GitHub lives on top of a tool designed to avoid SPoFs, is concerning.
100%. GitHub is amazing. GitHub is a wonderful tool. GitHub is also just a service cooked up by a handful of jokers who got enough funding to make it happen, and they do a reasonable job of keeping it running most of the time. I wish them all the best, and yet making my own company's infrastructure reliant on it is not something I'm comfortable with. Sometimes they're going to blow it, and that's ok. If them blowing it means my business is affected, I'm the dummy.
While I agree with the general gist of your comment, GitHub was entirely self-funded up until about a year ago or so. they didn't "get" funding, they made real money and reinvested it and grew.
Fine, bootstrapping jokers are my favorite kind of jokers. No offense intended. My point is it's just a small company with no SLA, express or implied. It could have been anybody with the same great idea, and just because they had a great idea doesn't make them automatically reliable enough to pin your infrastructure on.
No? The fact that GitHub is based on a distributed tool is a large part of what makes it so awesome. I can take advantage of the sweet social/collaborative elements of GitHub without locking myself to a central server. If the lower-level they used weren't so open/distributed, I doubt GitHub would have taken off like it has.
What's the alternative, replicate everything in-house/self hosted?
Erm... Yes? You write that as if hosting your own repository is somehow a foolish or implausible thing to do.
Should startup stop using third party service providers?
Stop? Maybe, maybe not, it obviously depends on the circumstances.
Stop relying on them without a back-up plan? Absolutely. If you can't run your essential systems without GitHub, or any other third party system that isn't under your direct control, then you need better contingency planning.
If you're using a cloud running test service like Travis CI pushing to GitHub can trigger your whole deployment process starting with running the unit and integration tests.
Now you deploying production hinges on a third party. What if you push a broken build that your tests didn't cover, and GitHub goes down when you have the patch prepared? What if Travis goes down?
Third-party services being in the critical path for applying code to systems is a recipe for outages and other trauma. If it impacts you enough to come to HN and leave the comment I replied to, you're doing it wrong -- there had absolutely better be a failsafe that does not involve GitHub and Travis in your architecture.
Github's uptime is good enough that I don't see anything inherently awful about a workflow that makes development difficult when Github is down, but any production deployment process that strictly requires anything beyond a working internet connection and a working server to deploy to is insane.
Works pretty well for me. The argument you just made could have - and has - been about any other piece of infrastructure that we now work with third parties for. DNS, EMail, managed hosting, VPS hosting, off-site backups, etc, etc etc. I rely on GitHub to store my code and handle code sharing from separated networks behind NAT - why can't I rely on it for code deployment?
The days of isolating yourself from third parties are over be cause it really doesn't make sense to spend a bunch of man hours setting up and maintaining an infrastructure that GitHub - with the rare exception of tonight - specializes in and has an entire staff dedicated to keeping up and running. Third parties have become mission critical because they save money and time and are generally reliable.
If Travis CI or GitHub stop working I can do a number of things if I really need to deploy - for instance, push to production anyway, which, btw, is also hosted on a third party hosting provider. If several large services start falling like dominoes at once we have bigger problems than just whether my service is running.
Of course there is - pushing directly to the server. Which is hosted on a third party, the third party itself is hosted on another third party, and that third party probably relies on a couple of other third parties. And if my patch brings down the service I get a bunch of emails - which are sent to my email address, which is also hosted on a third party. And that email found its way using an MX record from my DNS records, which are hosted on another third party's DNS servers.
Do you really run an operation where you have soldered together all of your servers, created a data center inside your headquarters, within which you run all of your mission critical CI/testing/deployment services? Because unless you're one of a handful of tech companies, you didn't need to do that. If you still did I'd like to know why you did that, because in an era where even the CIA relies on third parties to accomplish mission critical tasks it doesn't seem to make a whole lot of sense.
And that trauma you are talking about - it doesn't happen. Not rarely, not ever. For years. Tonight I only happened to see that GitHub was down because I was looking at HN anyway.
> Do you really run an operation where you have soldered together all of your servers, created a data center inside your headquarters, within which you run all of your mission critical CI/testing/deployment services? Because unless you're one of a handful of tech companies,
The contrary. You represent the handful that didn't, and the "handful" of tech companies you suggest is much larger than you think.
Based on this comment and the recent reply in this thread I can tell you've mostly worked with small-scale architectures. Third party tooling and workflows do make a lot of sense at a small scale, but the point at which you outgrow those solutions comes a lot sooner than you think. When I arrived at Foursquare the entire operation was on Amazon; when I left a year and a half later, much of it was on physical equipment. And Foursquare is not a Google-scale operation -- virtualization and customer cotenancy just have a serious impact on SLA that is less pronounced at smaller scales.
It's easy to think your experiences represent the industry, as your comments suggest. It's also easy to think HN represents the industry, where startups reign supreme and everybody loves working Lean Devops. The fact is, neither of those statements are accurate, and beyond a six (or maybe seven) figure architecture you start having a harder time justifying third parties financially and operationally.
I do use Amazon currently, just as an off-site backup for on-site monitoring that I've built. That's common.
I recognize that the costs become less straightforward the larger you get, and that a number of larger companies have to develop their own bare metal infrastructure. But I was including companies exactly like Foursquare in that handful of tech companies.
I'd still like to see an actual cost breakdown, because to me hiring a full-time staff of 5 or 10 people to run such an infrastructure, at $80,000/year or more a pop, seems like it could quickly get just as expensive or more expensive than outsourcing those costs to another organization.
There has to be some reason Netflix is able to justify placing their entire high bandwidth streaming operation on AWS, a third party that relies on full virtualization.
Haha, I wonder how many of us came here because we couldn't push or pull commits, so just taking a HN reading break... and boom the top post is about Github!
Yep. All metrics down and exception rate is up gentlemen: http://imgur.com/Ncm5Rrx . I just noticed when SQLMap install was hung and every tarball/zip download on their site also leads to Github , oy vey!
Probably there are a handful of gravitons or something embedded right in the floor, or there is massive inertia to the system. We tend to forget that not all solutions require active power.
This does makes one wonder that if you need to make an app that has as little downtime as possible perhaps the best solution is to port as much as possible functionality client-side with client-side caching of data.
(theoretically, if I could cache every repo I visit using squid-proxy, even if github has an hour downtime I can still access the repo's I visited in the past, not perfect, but it's something!)
It is now up to 175% a couple minutes later (and 205% a couple minutes after that). The issue seems to be that it is actually currently more like 10000%, and is being averaged down throughout the month to a mere 100-200%. What is this even measuring?!
If I were implementing something tracking the rate of exceptions a reasonable thing to measure it against is the total number of requests. In such a system, if each request generated multiple exceptions you would have >100% exception rate.
Could Github store mirrors of all repositories on some third-party service, like S3?
Even if they were only updated every 24 hours and had a limited history for each repo, it still seems like it would be a really useful fallback that they could put up when things like this happen.
Sure, it's easy to do with your own code. But what about apps that don't vendor their dependencies? If you need to pull in third-party dependencies when you deploy or test, it's still possible to maintain your own mirror, but it would be nice if you could depend on the code being "up".
Alright...well, it looks like this whole programming thing was a bust. Was good while it lasted, but yeah, uh, I guess I'm gonna go home, er, something - so...see you guys later?
It's amazing how often github actually goes down - 90% of the time they don't log it on their uptime / status page - it got so bad at one point I added it to our Nagios monitoring to prove a point to our devs that they shouldn't rely on external services for builds when our internal git(lab) server hasn't gone down once this year.
BTW the official standing on dependencies in Go is to actually fork all deps you need, possibly using submodules or even vendorizing by adding src/* to your project VCS.
88 comments
[ 7.0 ms ] story [ 99.0 ms ] threadLet's see who the Saturday team is!
How do you know when that point is? Is there even data out there on how frequently big providers go down? (Not just Github, but stuff like aws, etc)
Your question is too vague. Github is up enough that I don't care. However, it's down enough I wouldn't want to not be able to deploy because it's down. Therefore, I may mirror my repo somewhere else. That's easy because git is decentralized. It's a lot cheaper than running some alternative that I guarantee is always running. You can do this by simply pushing to a mirrored remote branch.
If you're just interested in the subject, research high availability.
Your question is too vague. Github is up enough that I don't care. However, it's down enough I wouldn't want to not be able to deploy because it's down. Therefore, I may mirror my repo somewhere else. That's easy because git is decentralized. It's a lot cheaper than running some alternative that I guarantee is always running. You can do this by simply pushing to a mirrored remote branch.
If you're just interested in the subject, research high availability.
Generally, you don't care if GitHub [or your inhouse equivalent] is down when no one is working. Also, if it goes down in the middle of the night and the first guy in fixes it...the difference between 1 man hour down in the morning vs. 20 man hours during the day across an entire team is significant.
There are very valid reasons for both choices.
It is fortunate, then, that all GitHub customers are in the same timezone!
It might not be affordable for daily works, but perfect for critical issues.
Github's not only Git. Lots of services integrate with Github. Stuff like continuous integration, continuous deployment. If your system is built on those things, Github being down will prevent you from deploying.
What's the alternative, replicate everything in-house/self hosted? Should startup stop using third party service providers?
I continue to post comments like that because while I love GitHub, watching the community put all their eggs in GitHub's basket, especially when GitHub lives on top of a tool designed to avoid SPoFs, is concerning.
While I agree with the general gist of your comment, GitHub was entirely self-funded up until about a year ago or so. they didn't "get" funding, they made real money and reinvested it and grew.
You're saying it as if it were a bad thing.
Erm... Yes? You write that as if hosting your own repository is somehow a foolish or implausible thing to do.
Should startup stop using third party service providers?
Stop? Maybe, maybe not, it obviously depends on the circumstances.
Stop relying on them without a back-up plan? Absolutely. If you can't run your essential systems without GitHub, or any other third party system that isn't under your direct control, then you need better contingency planning.
Third-party services being in the critical path for applying code to systems is a recipe for outages and other trauma. If it impacts you enough to come to HN and leave the comment I replied to, you're doing it wrong -- there had absolutely better be a failsafe that does not involve GitHub and Travis in your architecture.
Github's uptime is good enough that I don't see anything inherently awful about a workflow that makes development difficult when Github is down, but any production deployment process that strictly requires anything beyond a working internet connection and a working server to deploy to is insane.
The days of isolating yourself from third parties are over be cause it really doesn't make sense to spend a bunch of man hours setting up and maintaining an infrastructure that GitHub - with the rare exception of tonight - specializes in and has an entire staff dedicated to keeping up and running. Third parties have become mission critical because they save money and time and are generally reliable.
If Travis CI or GitHub stop working I can do a number of things if I really need to deploy - for instance, push to production anyway, which, btw, is also hosted on a third party hosting provider. If several large services start falling like dominoes at once we have bigger problems than just whether my service is running.
Do you really run an operation where you have soldered together all of your servers, created a data center inside your headquarters, within which you run all of your mission critical CI/testing/deployment services? Because unless you're one of a handful of tech companies, you didn't need to do that. If you still did I'd like to know why you did that, because in an era where even the CIA relies on third parties to accomplish mission critical tasks it doesn't seem to make a whole lot of sense.
And that trauma you are talking about - it doesn't happen. Not rarely, not ever. For years. Tonight I only happened to see that GitHub was down because I was looking at HN anyway.
The contrary. You represent the handful that didn't, and the "handful" of tech companies you suggest is much larger than you think.
Based on this comment and the recent reply in this thread I can tell you've mostly worked with small-scale architectures. Third party tooling and workflows do make a lot of sense at a small scale, but the point at which you outgrow those solutions comes a lot sooner than you think. When I arrived at Foursquare the entire operation was on Amazon; when I left a year and a half later, much of it was on physical equipment. And Foursquare is not a Google-scale operation -- virtualization and customer cotenancy just have a serious impact on SLA that is less pronounced at smaller scales.
It's easy to think your experiences represent the industry, as your comments suggest. It's also easy to think HN represents the industry, where startups reign supreme and everybody loves working Lean Devops. The fact is, neither of those statements are accurate, and beyond a six (or maybe seven) figure architecture you start having a harder time justifying third parties financially and operationally.
I do use Amazon currently, just as an off-site backup for on-site monitoring that I've built. That's common.
I'd still like to see an actual cost breakdown, because to me hiring a full-time staff of 5 or 10 people to run such an infrastructure, at $80,000/year or more a pop, seems like it could quickly get just as expensive or more expensive than outsourcing those costs to another organization.
There has to be some reason Netflix is able to justify placing their entire high bandwidth streaming operation on AWS, a third party that relies on full virtualization.
per docs at https://github.com/bower/bower oh....
As for the docs, it is somewhat crazy to think how much documentation is hosted by GitHub these days...
(theoretically, if I could cache every repo I visit using squid-proxy, even if github has an hour downtime I can still access the repo's I visited in the past, not perfect, but it's something!)
I've personally wondered about their stack but I imagine that has little to do with that--mostly Ruby though right?
"github.com is currently unreachable. We are investigating mysql cluster issues."
Even if they were only updated every 24 hours and had a limited history for each repo, it still seems like it would be a really useful fallback that they could put up when things like this happen.
https://gist.github.com/kyledrake/e6046644115f185f7af0
More info:
http://www.theverge.com/policy/2014/5/9/5699510/web-hosting-...
BTW the official standing on dependencies in Go is to actually fork all deps you need, possibly using submodules or even vendorizing by adding src/* to your project VCS.