HN Explorer

Status.github.com: “We're failing over a data storage system”

337 points by donpdonp ↗ HN
Updates to gist are getting lost. It accepts them as normal but the next page load for the gist is the previous version. Twitter accounts describe the same sort of problem for git repos.

status.github.com reports "We're failing over a data storage system in order to restore access to GitHub.com."

172 comments

[ 2.8 ms ] story [ 163 ms ] thread
dfcowell
PR comments are also failing with HTTP 405 error code.

Great way to start the week.

keyle
Well for the poor sods fixing it in the US, it's still Sunday evening...
geerlingguy
It's when I typically get an hour or two to crank out some open source PRs and issue queue cleanup. Sadly, the outage means I don't get that time to devote this week :(
hayd
Surely they have SRE teams around the world?
karaokeyoga
Is it just me or is that status message wonky? Failing over something to restore access?
karaokeyoga
failing -> migrating
pinneycolton
Well, they've got the "failing" part right. It's the "over" that's taking a while ;)
rphillips
Sounds like an engineer knee deep in diagnosing the issue.
diegoperini
That person needs our support more than anyone else right now.
radicality
"Failing over" in this context most likely means either migrating to some replica (say doing a dead master mysql promotion) or spinning up some backup storage system to serve master reads/writes.
cesarb
In this context, "failing over" means switching the active system to a replica. Which means their primary data storage system had some issue, and they were switching to a secondary data storage system, which hopefully contains the same data (replicated in real time, or nearly real time). Clearly that switch didn't quite work as expected, otherwise it would have taken just a couple of minutes before everything went back to normal...
6t6t6t6
Next: "We are restoring tape backups from some of our storage systems"
6t6t6t6
"We are continuing to repair a data storage system for GitHub.com. You may see inconsistent results during this process."
[dead] newnewpdro
Enjoy hosting your source code with Microsoft.
hartator
I can access GitHub.com here (Austin,TX), but it's taking forever.
igni
Before all the trolling about Microsoft starts up, does anyone have current information on what these systems are?

In the enterprise space, a 'data storage system' could be an Array or a SAN or a lightpath etc, with usually quite long failover times. For an org like GitHub I'd think more like an object store (an Array-of-Hosts, if you will) or whatever storage mechanism holds their database files. Do they self host this sort of thing or is it an AWS/GCE/Azure service?

FWIW, all git commands are working fine for me (create a branch, push, colleagues can fetch my branch), but the UI doesn't show my branch & nor can I review/comment on PRs.

reindeerer
I think it's obvious that it's their SQL storage that holds the website up that is pretty much in read-only mode, and has been for a couple hours now, not the git repos themselves.
geerlingguy
Yeah, been trying to post comments and new issues and keep getting "405 Not Allowed" responses.
sciurus
For storing things other than git repos, GitHub is heavily invested in MySQL. AFAIK all of GitHub is hosted on their own hardware.

https://githubengineering.com/mysql-high-availability-at-git...

rurban
This blog post describes exactly the scenario we were experiencing here. A master (single writer) failure, with missing fail over. You can only guess what went wrong with this plan. Looks good on paper, but some unexpected network or HW or routing problem could have caused the problem to identify the single writer.
friedman23
I can push a branch but I cannot access the pull requests that I create off of that branch
[dead] dvfjsdhgfv
> Before all the trolling about Microsoft starts up, does anyone have current information on what these systems are?

To be fair, you can't call it trolling without knowing for sure that the problem hasn't been caused by Azure-related part of the infrastructure. Looking forward to reading the post-mortem.

jononor
It is incredibly unlikely that there is any Azure in the production infrastructure. The acquisition is too recent for big changes like that.
paavohtl
It's actually so recent GitHub hasn't even been acquired yet. That's going to happen some time next year.
[deleted]
(comment deleted)
dvfjsdhgfv
I admire your confidence but will withhold my judgement until the postmortem is published.
beamso
Seeing a lot of issues with Pull Requests.
RandomGuyDTB
Earlier GitHub pages was "down for maintenance". Probably not related but might be of note.

(source: my website is hosted on github pages)

gavreh
My issue comments are not being saved, and if I do a tag push I don't see that reflected on the "releases/tags" area of GitHub.
keyle
I don't know if they keep changing the text updates with a slightly different version to

- prove it's a human that typed it

- there is code the prevents repeating twice the same message

either way it's entertaining... But it's Monday morning in Australia and we need to release! (yep we do this via pr/tagging etc.)

Fuzzwah
It is to ensure that the updates get mirrored onto twitter, where exact duplicates can't be posted.
hiccuphippo
Isn't it enough to delete the older tweet? Or maybe just add a timestamp to the message.

Off-Topic: If it's not possible to write a twit with the exact text from a deleted twit, then a way to prove someone wrote a twit and then deleted it would be to have them try to write it again.

askmike
Deleting tweets is a terrible workaround. Once github start tweeting and people start linking to those tweets they can't go ahead and delete them 50 minutes later..
owl57
Re: off-topic: You have to do it fast. You certainly can tweet a lot of identical tweets if you aren't in a hurry: there is a "Rock in the forest"[1] that tweets "Nothing happened today" (in Russian) every day for several years, and most of the time the text is exactly the same. Interestingly enough, the time of tweet varies widely from day to day, so it looks like a human or a program specifically made to imitate one.

[1] https://twitter.com/kamen_v_lesu

mehrdadn
They can't alternate?
OJFord
For several hours it was alternating.
hnarn
If you include a unique hashtag, would the message be considered unique? In that case, you could include the epoch timestamp as a hashtag :-)
drb91
Or even just prefix the tweet with the update ordinal.
rococode
I like how the times are unevenly spaced too haha. I'm imagining a bunch of devs sitting at home in their pajamas (Sunday night) talking on Slack as they try to fix the site and every time there's a lull one person's like "hey we should probably refresh the status again".
ajb257
Given the latest update is "We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours", I can only assume someone at GH read your comment!
majewsky
Hacker News is, surprisingly, not the center of the universe.
jedberg
> But it's Monday morning in Australia and we need to release!

Ironic isn't it? The whole point of git was to be distributed, and yet we're at a place where a bunch of companies can't deploy software when a single git provider is down. I myself am in the same boat. Sure, I could reconfigure for a different repo, but is it even worth the effort?

beatgammit
I've seen some that are spaced at regular intervals with very subtle changes between messages, none of which actually say anything. I really like Github's messages since they seem like they're actually being updated by a human.
Memosyne
Anyone know if this could be related to the Youtube outage? It's been a while since I've seen these big websites go down.
bognition
In what way? Youtube runs on internal google hardware, GitHub runs on its own hardware too.

It could be a networking issue, but you'd expect more sites to be impacted.

If it were a software issue, you'd expect a big player like google to be aggressively patching and talking about a bad release.

More likely is its just a coincidence.

danielhlockard
I'm not sure how you could postulate that they're related
ObsoleteNerd
Targeted attacks on specific aspects/assets of a major websites infrastructure? Eg target a specific service that both sites have in common in their back end?

Not even sure if that's feasible but it's an idea.

Memosyne
I figured it was just a coincidence, but the HN network is so well informed I thought I'd just ask the question on the off-chance it wasn't.

I personally don't find a relation likely.

ObsoleteNerd
Oh I find the relation very unlikely. I was just trying to think up a potential connection for fun/curiosity, as the coincidence is pretty crazy that two huge sites known for very few/little outages had major outages in the same week.
jpatokal
Github is not hosted by Google and was acquired by Microsoft some time back, so that seems... unlikely.
geggam
Interesting how everyone uses a tool designed to eliminate SPOF in a way it has a SPOF.
geerlingguy
GitHub does not equal git; the reason I’m paused is because I use Github’s issue queues to organize my OSS work. Much easier than self hosting an issue repository/bug tracker. I am still able to do all my work, run new containers, etc., but GitHub is more tied into business processes than actual code (which is what causes the pain during these outages).
dfcowell
This is certainly making me rethink the workflows we are using in my team, particularly those around PRs and code review.
platinium
This has been super frustrating, as people have deadlines and are working to finish projects before Monday morning.

What are the (good) alternatives to Github? Gitlab supposedly is Google-backed, so I don't want to have my private code there. Is Bitbucket the only one left?

I don't mind paying monthly, which I already do for GitHub.

Tecuane
> Gitlab supposedly is Google-backed, so I don't want to have my private code there.

This strikes me as odd. May I ask why usage of GCP is a deal-breaker for you? While I can understand not wanting to use Google products directly as a consumer, I believe it would be - for lack of a better term - platform suicide for Google to intercept and perform its usual analytical shenanigans on the data content of transmissions to/from their platform.

Either way, Phacility's Phabricator[1] is $20/user/mo.

1. https://www.phacility.com/pricing/

Jedi72
Nobody trusts Google for any reason any more as they have proven unworthy of our trust.
hactually
That's a silly statement. By saying 'nobody' - a single point of data invalidates your assertion.

I use gmail, I'm quite happy to trust that contract.

jammygit
You're saying that you trust that contract today, or are you saying that you have always trusted that contract?

Its only recently that gmail's contract involved keeping out of your data. I think they also only say they abstain from using your data for targeted advertising, not that they don't use it for other purposes. I haven't read the terms in quite a while though and I could be mistaken.

Great products though. I really do wish I could pay for them in exchange for a real, trustworthy, comprehensive privacy promise.

deadbunny
There is a whole world of difference between paid for and not paid for Google services.
naner
This type of thing will always be a risk with cloud infrastrucure no matter what service you choose.
cmroanirgo
If you've a server of your own (and even if you don't, you can self host for a few $/mo), gitea is an easy choice: https://gitea.io
toomanybeersies
You can self-host Gitlab, as well as Gogs, Gitea, and a handful of other solutions.
patrickg_zill
I have been meaning to evaluate fossil-scm.org for a while...
ComputerGuru
I can't add comments to pull requests("you can't do that right now") and any commits pushed to branches are not updating updating the visible status in the web interface, no are newly created branches showing up. However, if you navigate to a new commit directly with its SHA (so you can share it with someone if you really want), it'll show up (so they're just not being indexed).

EDIT:

Obligatory "that's what happens when the whole world relies on a centralized git repo" and a reference to gitea, which has a very slick github-esque UI and is incredibly easy and light to deploy/run (on an existing server, your own PC, a raspberry pi, a docker vm, or whatever): https://gitea.io/en-us/

rodorgas
I’m receiving emails (a lot of duplicates) from comments in PR, but they won’t show up in the browser. I guess people are trying to submit multiple times, the email is sent but comment isn’t posted.
majidazimi
Welcome to eventual consistency.
hnarn
Mañana Consistency(tm)
samuel1604
yes, but does it scale tho ? is it DR ? I mean sure fine it's easy and pretty but there is some operation challenge to figure out,
perlgeek
If you host just your own Open Source projects, does it need to scale?
metildaa
Likely not, you can probably even host your friends too without much server load. Scaling is a concern when you have many users, just a handful of users won't ttend to create much load.
theSage
I'm really looking forward to the post mortem that comes out of this (if it does). I always learn a lot from reading those.
[dead] reindeerer
A post-mortem from Microsoft ? nfw
JoyrexJ9
GitHub are not part of Microsoft yet, the deal has only just been approved. They are still totally separate, and will be for some considerable time.

Also Microsoft post plenty of postmortems, like this detailed one from the VSTS outage in Sept. https://blogs.msdn.microsoft.com/vsoservice/?p=17485

[deleted]
(comment deleted)
olingern
Local to Tokyo, Github has been down for most of the day. I never realized how much of my day centers around it: pr review, creating / commenting on issues, etc.

Looking forward to the write-up. I'm also curious as to how this significant outage lines up with their SLA for enterprise users.

guywhocodes
What's strange to me is how many hours we've been given the same message. Today is an important day for my organization and not getting any information that hints how long this will last is a huge problem for our planning.
steventhedev
Are you a paying customer of GitHub Enterprise? If not, then you're getting your money's worth.

Snark aside, this is a great time to reassess your deployment strategies and look into things like local apt and pypi proxies. I'm confident you can find similar projects that will transparently cache your dependencies.

guywhocodes
I don't know how much my org is paying but it's no small amount, Enterprise no. Are we getting what we pay for? I don't think so, I think effective SLA of 99% isn't good enough for any SasS. But of course you understand that moving away from github is no small decision.

Absolutely is this a perfect time to assess deployment strategies and challenge all the advice of how big a company has to be before it's worth to do X.

steventhedev
It sounds like you're using GH for ticketing also, which means that your R&D org's productivity is tightly coupled with an external companies uptime. Self-hosting Gitlab is easy, as is running Jira, redmine, and a dozen other tools.

If you can't do your job without a tool, then you need to have a plan B on hand for when that tool fails. Both in the micro sense of the tools you use to code (editor, browser, laptop, mouse, coffee mug, etc) and in the macro sense of tools your organization uses (ticketing systems, chat, bathrooms). Show some initiative, figure out some mirrors for your dependencies, and try standing up a local caching proxy for your team. It'll probably take a lot less time than you think.

BadassFractal
Wonder if GitHub is "too useful to fail" at this point. As in, most people and companies won't switch git repo providers unless GitHub is down for many days at a time during the work week.
avip
Transition to BitBucket or gilab is one click away. Companies surely will move if incentives are there.
jononor
Not when you use third party integrations. Like when using Travis for CI, and Travis doing your deploy.
Skinney
Travis supports both Bitbucket and Gitlab
jwilk
No, Travis CI doesn't support any code hosting other than GitHub.
mellisdesigns
What particular services are down? I noticed I can hit the UI.
avip
For me: can't fork, can't login (tested incognito), can't clone new repos.
[dead] julienfr112
Is that a job of the Microsoft special project team ?
CodeM0nkey
GitHub has not yet been acquired by Microsoft, they're completely separate.
arthurcolle
I wonder if this will vaporize the deal.
manigandham
That's not how these deals work. A little infrastructure downtime is not going to stop a 7.5B platform acquisition.
[dead] gyvastis
Great takeover, Microsoft.
jypepin
It's been over 6hrs without update, except the hourly message which states the same. I'm really looking forward this post mortem, but having been in the situation where I've had to deal with large scale outages like this one, I guarantee some engineers are having a bad time right now, and I feel for them.

Github engineers, if you are reading me (probably not), KEEP IT UP, it happens to the best of us! <3

olingern
Posted at 15:51 Japan Standard Time

"...with the aim of serving fully consistent data within the next 2 hours."

That's somewhat significant

Cthulhu_
Yeah, sounds like a backup restore or a RAID synchronization is in progress.
shry4ns
I don't know if someone's mentioned it yet but TravisCI is also not running on the commits that do show up on git, in Houston