96 comments

[ 2.0 ms ] story [ 174 ms ] thread
Again.

Affects both the website and git.

Affects push to GitHub*

I really hope that your copy of git isn't broken because GitHub went down :p

Haha got me. :P

No I just tried both the website and a `git pull`.

How am I supposed to do any work if I can't copy and paste code.
My builds are broken because they have dependencies on github repos.
This is a teaching moment for you then. Minimize external dependencies when you've moved from "thank god it works" to "this is production ready."
One solution to that problem is to run your own local git server, with an automatic process for pulling github repositories of the dependencies you need. Then you can always know that you will have the access to our dependencies even with a third party is unavailable.
Has anyone heard what might be the cause?
It just went down about 10 minutes ago, you'll have to wait a bit for an explanation...
> 09:14 MST Major service outage.

It's been 4 minutes, so I highly doubt anybody at GitHub even knows yet. Maybe one guy who's thinking a lot of "oh crap, oh crap, oh crap" right now.

The last outage they were back up in 4 minutes.
Pretty sure devops at a service like Github have slack alerts and/or something like pagerduty. They'll be drowning in alerts.
Rumors are that they edited some JSON files. Not a good day to touch JSON.
Hacker News Time! :)
I count 4 threads on /new referencing GitHub down :D
but we are keeping this one :) I like it
Long live Gitlab...
Except. Pushes of new repos are still broken in the CE. No fix for over 3 weeks.

https://gitlab.com/gitlab-com/support-forum/issues/207

https://gitlab.com/gitlab-org/gitlab-ce/issues/26369

https://gitlab.com/gitlab-org/gitlab-ce/issues/21502

Only work around is to create a README on empty repos via the GUI and which then opens up the Protected Branches page. That said the rules around who is able to commit on the repos are broken.

Hey there, thanks for bringing this up. First, I want to apologize for the inconvenience you've experienced as a result of this issue. Second, I'd like to apologize for the delay in getting back to you on this as well.

You're right, 3+ weeks is a pretty long time for something like this to be affecting people. Unfortunately I don't have an ETA on when this issue will be resolved, but I'll be keeping tabs on it going forward. Please feel free to add additional comments to the issues if you haven't already, or you can reach out to me directly. Again, I'd like to apologize for any trouble this caused you and others affected by this issue.

This issue should be fixed as of today in 8.16.3
Gitlab can go down. Gitlab might have outages as well unless you are using the self-hosted version (although, in this case there is a self-hosted version of Github as well).

Just because Gitlab is not as famous as Github and hackers don't run their botnets on it, it doesn't mean it is safe or stable.

Github is down --> check out the unicorn --> refresh hacker news --> refresh again --> shitpost in comments
This submission is a shitpost.
(comment deleted)
The coding equivalent of accident rubbernecking.
I wonder how many companies are unable to deploy new updates now because they rely on Github?
mine.

Try to deploy -> bad key -> check github website -> go to hackernews

Yep, exactly the same process here.

:( Now my travis build might be red all weekend

Friends don't let friends deploy on Fridays... it's always a horrible idea.
Especially in France where it's 5:27.
excellent point sir.
Deploying on Friday is fine if you can rollback quickly :)
I think the point was that you don't want to be messing with production when people are going to be gone for days. Even if it's easy to roll back I'd just as soon not deal with it. If your application is of such importance that rolling a change out Friday v. Monday makes a different in the business's bottom line, that business can almost certainly afford to have weekend staff that could cover it.
Meanwhile, today is Friday the 13th.
(comment deleted)
(comment deleted)
Mine. Let this tree grow!
…and this is how we remember how much we rely on GitHub, and how much value it provides. Every day.
It sort of becomes your Internet and computer. No computer no work. No Internet no VPN no work.

But in this case I guess investing in local mirror is a great investment.

(comment deleted)
Friday the 13th...
The day following a full moon...
The day github had a major outage...
Obligatory Gogs plug: https://gogs.io

It also has an awesome automatic mirror mode that can remain sync'd to a remkte repository. I don't care what your company does or how you run it, but please don't just rely on Github to host your code without a mirror somewhere. There are drawbacks to this incessant centralization.

does it mirror issues, PRs, wiki pages etc?
I don't think so (we use other apps for those features) but Github's wiki is just a git repo, isn't it?
Just this week I setup gogs on my VPS to mirror all of my repos on github. It's part of my new year goal to avoid my dependence on 3rd party apps. Of course I still am depending on my VPS provider so...
Just set up a Banana Pi or whatever at home to mirror your mirror.
The specs for a beaglebone black are worse than a raspberry pi 3 in every way except the eMMC storage. Am I missing something? (I've actually been looking for something stronger to include in my raspberry pi cluster at home, but this doesnt seem like it)
Whoops you are right, I meant the Banana Pi m3(http://www.banana-pi.org/m3.html).
Banana pi is probably overkill for a low-key git repo. Its double the price of a raspberry pi.

That said, I'll actually be looking into this to possibly run arm stuff that needs more power.

The nice thing about git is that it's distributed. You can place as many mirrors as you like. If you make the project and repo names the exact same as the official ones, then switching between them can be as easy as replacing a hostname in your dependency manager.
Instead of Gogs it is now recommended to use Gitea, since it has more activity (~400 more commits last time I checked) and is community driven: https://gitea.io/

According to their Blog: Gitea is a community fork of the popular self-hosted Git service Gogs. We’re a growing group of former Gogs users and contributors who found the single-maintainer management model of Gogs frustrating and thus decided to make an effort to build a more open and faster development model. https://blog.gitea.io/2016/12/welcome-to-gitea/

I would love this if gitea had a docker image for arm prebuilt. gogs/gogs-rpi is several versions behind gogs/gogs, but it at least exists.

I'm capable of building my own, but it's nice when there's already an official x86 version to have an arm version too.

(comment deleted)
looks like it is back up now
(comment deleted)
It's back. [END OF THREAD ?]
Since I can't push to my repo, I'm gonna say no, it's not back.
Site came back online, but ssh/git still down.
Github has been down (or very slow) a lot the last few months. They used to have a good engineering reputation (at least with me), but it's been tough times over there.
(comment deleted)
(comment deleted)
So there was an outage yesterday which required failover to another data center. Wonder if both are related. Either way I want to read their postmortem. GitHub's postmortem is usually very educational.
Currently 98.9212% app server availability in the last month.

That means 7h 52m 49.7s of potential downtime/unavailability [1] in the last month.

And the trend of that percentage doesn't look that rosy.

[1] https://uptime.is/98.9212

(comment deleted)
I use JSPM on my projects and I have several dependencies on github repos that aren't in NPM. Anyone have any ideas on a fallback solution for when github goes out like this?
Is there a way to be able to point JSPM to a different, non-github repo? A remote clone of the repo might work then.
Have a local git server, with an automatic process for pulling in the repositories that are your dependencies. Then use the repos from that local git server instead of github directly.
I've configured one remote with two pushurls and set this remote to all my branches. This way I always push to BitBucket and GitHub at the same time. Just wondering, how this would work if I was not the single user contributing.
You'd have to either coordinate with the other users to have them do the same, or officially declare which one is "official" and which one is "backup" and set up something so the backup tracks the official (and then you only need to push to the official as well)