Just noticed an interesting design decision on that page. An 'everything OK' message once a day. One way to push previous downtime below the fold of the page.
To me, that looks like an angry unicorn ... does that mean it's another DDoS?
It would also be interesting to track loss of productivity in the software engineering professions when GitHub is down. Is this a single point of failure for your company?
Exactly ... we've got Ansible scripts that do that. One of the current Java rockstars (not my description), Adam Bien, recommends pulling the source code for all third-party dependencies to build them locally and put them in a local repository. I suppose you can push directly back upstream to avoid having an extra step.
For OS-level package management like Apt or Yum this is less of an issue, but for things like Composer, Gem, NPM and the like, is the concept of "closest available mirror" even a thing?
I guess for mission critical applications you could have a local mirror of all your vendor package dependencies (probably not a bad idea) but I would bet most people don't do that (I don't)..
If you're using Java most of the build tools use the Maven repositories, for which you can set up a local mirror too. The company i work for runs a Sonatype Nexus so we're not really affected by others' downtime unless we need stuff that we hadn't ever used before.
Yes, for our Java projects we have a Nexus repository which mirrors everything locally. It doesn't protect us from someone deleting a project ... then we're stuck on the last available version without a way to address issues, but it's worked pretty well so far.
Yes, but these decisions aren't under me plus setting up a github kind of thing is way too time consuming with all the features that we're used to now.
Understandable, but surely the potential disruption if github was to go down for a long time or indefinitely would be more than setting up a local git server that you control?
They're not GitHub. i.e. The chance of GitHub and your GitLab instance on DO going down simultaneously is very slim. Spend an extra $10/month and you can set up another DO server in another data-center, and reduce the odds of simultaneous failure even further.
Setting up a GitHub alternative is quite expensive but if you're just making your deployment system more robust it doesn't need to be more time consuming as having a server which you can SSH to.
how does that help for the long long lists of external dependencies, for the bugtrackers of projects where I'd like to look up if someone else had the same issue and maybe there's a workaround, to view and compare code for older revisions of projects I'm using?
With a ton of projects now hosted on github, github being down is a major dent in my overall productivity even though we _do_ have an internal git server.
For repositories your projects depend on, you can set up a local proxy so you at least have local copies for continuous integration. E.g. PHP's Composer project has _Toran_ and _Satis_ for this: http://tech.m6web.fr/composer-installation-without-github.ht...
doesn't give me easy access to the source, browseable and searchable, I can't link to it when discussing with a colleague via chat, can't browse the docs and the wiki and I can't update them. It's a minor advantage at the cost of running yet another piece in my infrastructure that may die or exhibit problems. We use nexus as a proxy for various repositories and so far I've had nexus down more often than github.
You can host your own Nexus servers, for whatever that's worth, if you were unaware. Again, adding more infrastructure, but it at least solves the "it's down on the internet!" problem.
we are hosting our own nexus servers - but they tend to have issues of their own. Bugs, repositories get broken, storage full, machines down for maintenance and patching, yadda yadda. Githubs track record at being up is better than our nexus track record.
OK I get what you're saying, but honestly, who is really 'losing productivity'? I suspect the ones saying they are in fact just getting on with other tasks they can do because they are, despite github, still using distributed version control and do have access to and the ability to commit to code repositories. You don't need to host your own git daemon or github clone to get that functionality.
I don't use github because it gives me a git daemon. I use it because the interface allows me interact with non-technical people when dealing with code and resources. I use github because, even though it is down right now, my stuff is backed up, I don't have to deal with the nitty gritty of ensuring backups succeed, I don't have to deal with the system when it goes down - someone else does that for me. Hence I would never want to run a local github clone.
GitHub isn't made for that though, if you need scripts to pull downm files for important work, then you should really be storing that stuff in S3. And probably some other place you can switch to if something goes wrong.
Someone pushed code to master or some other specified pre-master branch. Jenkins pulls down, builds, runs tests, uploads artifacts to S3. That doesn't sound unrealistic or risky on the surface, modulo relying on both GH and S3 being up.
Arguably you should be able to mirror on both an internal git repository somewhere, but GitHub is down infrequently enough that it's hard to justify that work + continuing costs if you don't have your own hardware + security patching + other admin costs to managers until you see multiple outages in a small enough window.
That sounds exactly like our deployment strategy, and I can push to production when GitHub is down. What's preventing you from copying & pasting the build script from Jenkins into a terminal in a local checkout? S3 being down would be a show stopper, but GH being down is just really annoying.
What about the code it has to build? Where does that come from? What about third party libraries it has to pull in? If the only thing the build server needed to build your code was the build script, this would be easy.
Checkout from where? I'm not being dense, but new code has to come from SOMEWHERE. If GH (or wherever your code is hosted) is down, the checkout won't be able to happen. Development isn't generally done on the build box.
Again, how does that get to the build box? I also have several local copies of my code on my computer, but that doesn't help the Jenkins machine get it unless I scp (or whatever) it over.
Developers don't always have access rights to the build systems or artifact repositories. Most of the devs I work with don't - not because I'm not willing to share, but because they're supremely uninterested in the deployment systems. If there's an outage, the people who can fix it aren't always available right then and there.
Yes, for production assets. But that's not the only thing deployment scripts serve. Tomorrow we're trying out a feature branch on our staging servers, and will be pulling from Bitbucket as we iterate. If Bitbucket goes down, we can work around it, but we still 'lose productivity' because we're spending our time working around it. Yes, the devs can 'work on other things', but tomorrow's feature branch is what's on their mind, and it's what the sprint target is.
It doesn't matter what kind of deployment you do. Even s3 has outages if the network between you and s3 bites the dust. Even if you deploy straight from the developer's workstation to the production server, you can have an outage - network splits, hardware failure, developer-with-commit-rights gets sick. Working around an outage means you're not working on the product, and that's a loss of productivity.
I diaagree on some points: github/git is not an appropriate backup strategy and github is not user friendly enough for non technical people (the default repository page, github pages is another matter).
Do you have any support for those assertions? In particular, the backup claim is rather dubious unless you're redefining “backup” to be a much higher bar than most common answers meet.
Git is a distributed revision control system and designed as such. The fact it can be used as a back-up tool doesn't mean it should be used as a back-up tool, especially when there are back-up solutions designed from the ground to do just that.
But what you actually said was “Github/git is not an appropriate backup strategy”. Git is tamper-evident and uses strong hashes to protect against bitrot, has full change tracking and numerous measures to avoid data-loss becoming permanent if you detect human error before weeks/months go by, has integrated remote tracking so you can tell how stale your off-site copies are and trivially update them, and most server implementations allow as much access control as you desire.
Assuming that we're talking about things which are already stored in Git and not e.g. an HD video collection, what would make your original statement true?
Funny, I switched our projects to Gitlab a week ago and told the team: Because we should not trust a centrealized platform to keep our code decentralized.
GitLab CEO here, I might be missing the joke here, GitLab is a decentralized solution you run on your own servers. Maybe you are confused with GitLab.com?
Is it? Having it hosted on my servers doesn't make it decentralized: it's just centralized in a place I own. If the server(s) go down, same problem as Github (except I may be able to take action). Right?
Compared to multi-tenancy it seems more decentralized to me since it dispersed functions from a central point. It is not redundant, if that is what you mean. In general having the server together with the rest of your infrastructure ensures it is available when the rest of your infrastructure is available and reduces network/ddos problems.
> Because we should not trust a centrealized platform to keep our code decentralized.
People should have all their code in place and just merge and rebase from time to time? At least that's what I (a SVN user) have read on the internets about Git.
GitHub is a much larger target, and has many, many more moving parts than your average Gitlab on, say, a Linode virtual server.
Github will probably have more downtime which you can do noting about, than your selfhosted gitlab, which will probably fail because you upgraded something, or tinkered with a configuration.
It's a numbers game. They have more people monitoring, they spend way more money than you do on uptime/services, but such a service doesn't get more stable as it grows.
I trust myself way more than I trust GitHub or other hosted/cloud services because I cannot affect/help them in any way when stuff happens.
Ultimately, I know this, I accept this, and I happily use GitHub, Bitbucket and such all the time.
>Github will probably have more downtime which you can do noting about, than your selfhosted gitlab, which will probably fail because you upgraded something, or tinkered with a configuration.
Personally I don't think you will see less downtime on your own server. The difference is, that when it goes down, there is something you can do about it, so you get busy fixing it.
When github is down we get to all get online and talk about our woes together.
The outages are usually pretty short. Go get a coffee and come back, or grab the team and go play a round of bowling. That has to be better than trying to manage and keep secure our own repos.
This - the last thing I want to do is be on the line when something happens to our git server and no one can work because our solution is my own custom one-off.
Each has it's pros/cons and everyone has their own opinions/needs.
For me - github doesn't provide free private repos (exceptions being for class use) and while bitbucket does support free private repos but I believe only a limited number of people can work on a private repo. While I don't collaborate with many people on my ~70 private projects - I also don't want to have to migrate my projects (again - after google code announced shutting down...).
Self hosting does allow me to pre-configure repos for use by other people or students.
I do self host my repos but I don't use gitlab.
The one thing to keep in mind with a service like github - I have seen services that you can run that leverage the github API that fires an event on a new commit to a public repo. Accidentally push your Amazon cloud service keys? Before you even realize it happened - someone will be spinning up VMs or using your S3 service[1].
Yes exactly, between trusting a third-party centralized platform and trusting my git hosted on my own serve .. I choose my own and everyone should .. this was the original point of creating Git in the first place
Our team did something similar after the problems GitHub had when it came under attack a few weeks ago. We have a team of 20 or so and when GitHub becomes unreliable it becomes a big problem for the team.
Last week we set up the community edition of GitLab on one of our local servers. We then mirror the repository to GitHub. So far so good.
We all like GitHub but after it's recent problems we realised we rely completely on it being up. We couldn't even deploy if it went down.. we still use it for open source things. For our main product it has simply become offsite back up.
So glad we don't rely on GitHub for our deployments anymore, we haven't had an outage to git in the past year yet it seems GitHub goes down more often than people realise. Obviously DDOS attacks are nearly impossible to mitigate for anyone but it should serve as a reminder of companies dependencies on external services.
Somewhat related I highly recommend both Gogs and Gitlab for internal git hosting.
153 comments
[ 3.1 ms ] story [ 214 ms ] threadCould be another DDoS?
---
Update 11:32 UTC "We're seeing high error rates on github.com and are investigating".
>Today
>11:32 UTC "We're seeing high error rates on github.com and are investigating".
>11:40 UTC "We're doing emergency maintenance to recover the site".
it's back
It would also be interesting to track loss of productivity in the software engineering professions when GitHub is down. Is this a single point of failure for your company?
Think of all the package managers that pull straight from Github repos as part of their installation processes..
I guess for mission critical applications you could have a local mirror of all your vendor package dependencies (probably not a bad idea) but I would bet most people don't do that (I don't)..
http://www.bash.org/?330261
It's all good though, I still have loco host to keep me busy until their resolve the issue.
With a ton of projects now hosted on github, github being down is a major dent in my overall productivity even though we _do_ have an internal git server.
I don't use github because it gives me a git daemon. I use it because the interface allows me interact with non-technical people when dealing with code and resources. I use github because, even though it is down right now, my stuff is backed up, I don't have to deal with the nitty gritty of ensuring backups succeed, I don't have to deal with the system when it goes down - someone else does that for me. Hence I would never want to run a local github clone.
Arguably you should be able to mirror on both an internal git repository somewhere, but GitHub is down infrequently enough that it's hard to justify that work + continuing costs if you don't have your own hardware + security patching + other admin costs to managers until you see multiple outages in a small enough window.
It doesn't matter what kind of deployment you do. Even s3 has outages if the network between you and s3 bites the dust. Even if you deploy straight from the developer's workstation to the production server, you can have an outage - network splits, hardware failure, developer-with-commit-rights gets sick. Working around an outage means you're not working on the product, and that's a loss of productivity.
Assuming that we're talking about things which are already stored in Git and not e.g. an HD video collection, what would make your original statement true?
WALTER MERCADO
Most people use GitLab to refer to the downloadable software you run locally https://about.gitlab.com/downloads/
People should have all their code in place and just merge and rebase from time to time? At least that's what I (a SVN user) have read on the internets about Git.
There's a lot of people spreading a holier than thou attitude in this thread.
Github will probably have more downtime which you can do noting about, than your selfhosted gitlab, which will probably fail because you upgraded something, or tinkered with a configuration.
It's a numbers game. They have more people monitoring, they spend way more money than you do on uptime/services, but such a service doesn't get more stable as it grows.
I trust myself way more than I trust GitHub or other hosted/cloud services because I cannot affect/help them in any way when stuff happens.
Ultimately, I know this, I accept this, and I happily use GitHub, Bitbucket and such all the time.
Personally I don't think you will see less downtime on your own server. The difference is, that when it goes down, there is something you can do about it, so you get busy fixing it.
When github is down we get to all get online and talk about our woes together.
The outages are usually pretty short. Go get a coffee and come back, or grab the team and go play a round of bowling. That has to be better than trying to manage and keep secure our own repos.
For me - github doesn't provide free private repos (exceptions being for class use) and while bitbucket does support free private repos but I believe only a limited number of people can work on a private repo. While I don't collaborate with many people on my ~70 private projects - I also don't want to have to migrate my projects (again - after google code announced shutting down...).
Self hosting does allow me to pre-configure repos for use by other people or students.
I do self host my repos but I don't use gitlab.
The one thing to keep in mind with a service like github - I have seen services that you can run that leverage the github API that fires an event on a new commit to a public repo. Accidentally push your Amazon cloud service keys? Before you even realize it happened - someone will be spinning up VMs or using your S3 service[1].
[1] - http://www.devfactor.net/2014/12/30/2375-amazon-mistake/
Last week we set up the community edition of GitLab on one of our local servers. We then mirror the repository to GitHub. So far so good.
We all like GitHub but after it's recent problems we realised we rely completely on it being up. We couldn't even deploy if it went down.. we still use it for open source things. For our main product it has simply become offsite back up.
wasn;t me
You would think you can still develop an iOS app when GitHub is down, but a simple `pod install` requires access to repos hosted on GitHub.
Somewhat related I highly recommend both Gogs and Gitlab for internal git hosting.