HN Explorer

Facebook Platform was down

74 points by haney ↗ HN
A check of https://developers.facebook.com/status/dashboard/ returns an error and I'm unable to login with facebook to some of my mobile apps.

EDIT: Commenters are reporting that this appears to only impact users that are hitting the US data center[1], and that it may only impact users through specific ISPs

1.https://news.ycombinator.com/item?id=18434457

EDIT: Appears to be back up

39 comments

[ 3.2 ms ] story [ 92.4 ms ] thread
CJKinni
So much for the pivot to "move fast with stable infrastructure."
seattle_spring
They've been down for, what, 20 minutes in the past year?
greglindahl
Most of their outages appear to be regional, and only part of their site. This one also appears to have been regional. So it's hard for us outsiders to know the statistics.

These kinds of outages are pretty typical for "move fast and break things" development -- instead of spending huge effort making sure that no outage ever happens, spend effort testing and limiting outages to a small part of production. The intended benefit is much faster development.

[deleted]
(comment deleted)
mkong1
broken image from static.facebook.com for the logo, too.
faitswulff
This should be an interesting postmortem, if they have one. I think Netflix never released a postmortem of their latest downtime.
dgritsko
Unrelated to OP, but did Google ever release a postmortem for the YouTube outage a few weeks back? Was interested to hear the cause of that one.
[deleted]
(comment deleted)
garysahota93
I love reading the postmortems. I've read some from a few different tech sites and it's fascinating to hear just how people are doing things and how they fixed it.
gadders
My semi-dormant website: https://www.outagereports.net/

I started compiling a bunch of outage reports from around the web. I have another bunch stored in Pocket that I need to add.

Simulacra
Yep but only through the Verizon connection. Phone works, Tor connection to FB works, so it's probably localized.
noarchy
I can confirm what you say about Tor. I get the login screen when hitting FB via Tor, but not otherwise.
cronix
Strange, I just logged into the website fine from Portland, OR via Comcast
yeldarb
Interestingly I VPN'd into a node in Dublin and it is still working there.

Iowa (direct) and SF (VPN) were both erroring out sitewide. Anecdotally it seemed to start in Analytics and spread to Facebook.com and finally to the platform over the course of ~15 minutes. But not sure if that's representative.

skizm
Using a completely oversimplified estimation method and 2017 revenue numbers, they're losing about $1300 every second their website is down.
nabla9
From EU:

  Facebook Platform is Healthy   
  Since October 23 at 4:10 PM
a13n
Different data center, looks like the outage is US-based.
Mc_Big_G
Ideally it would never come back up.
a13n
Back up for me.

Facebook makes ~$105k in revenue per minute [1]. Since the outage was seemingly only US-based, and it lasted ~30 minutes, this outage probably cost them just over a million bucks.

Pretty staggering to think about.

[1]: https://ycharts.com/companies/FB/revenues

abkumar
> this outage probably cost them just over a million bucks

I have always wondered whether it is accurate to think about it like this. It sure is interesting to know how much companies make in revenue per minute, but it doesn't necessarily 100% translate to lost revenue in an outage.

For example, if I wanted to purchase something on Amazon, but the retail site went down I would not visit another site to purchase it. I would just wait until the site came back up. In that sense, the revenue isn't lost just delayed.

derefr
That works with e-commerce, but not necessarily with advertising. If you were going to browse Facebook during lunch in the office (and be shown ads), but Facebook is down, then you won't necessarily just "browse Facebook later in the day instead." You weren't on Facebook to catch up; you were on Facebook because you had nothing better to do.
greglindahl
This is definitely true, I expect that users will spend a bit less time on Facebook today.

However, Facebook doesn't have a 100% fill rate of high value ads, so during the rest of the day they'll be showing somewhat more expensive ads to users, earning a higher average revenue. That will make up for some of the loss, despite lower usage.

a13n
For e-commerce that argument makes sense, but not for advertising.

They lost 30 minutes of primetime eyeballs from tens or hundreds of millions of people that would have seen ads but didn't.