112 comments

[ 3.3 ms ] story [ 184 ms ] thread
maybe its the 1 billion like buttons they serve a day that tipped them over the edge. ;)
perhaps the Stuxnet virus was built to target facebook? ;)
You don't get to 500 million friends without making a few enemies
Wait, doesn't Facebook already have 500 million enemies?
It's hard to overstate the damage this is causing for merchants that prominently display FB like on their pages - lots of big ugly blank spaces / errors out there.

Realize this is temporary but shows why if you install any external vendor tool as a part of your site make contingency plans that switch on errors.

Or people who rely on Facebook for shared login...that must be down as well.
Good point. We recently had a client set their "signup" to FB login by default and I cringed.

Great, simple services but the macro effect that this is going to have on commerce today is crazy.

You're telling me--of course this has to happen the day before client review of a major Facebook Connect site whose only authentication is via Facebook Accounts. We had discussed potentially having two types of accounts (Facebook- and email-based), but having two parallel authentication methods just seemed clunky. I guess this is the price we pay for trying to keep things simple...
This must really be hurting Quora.

Luckily, they added Twitter authentication as well. But still, now a large portion of their site cannot login because of Facebook.

Seeing as Facebook login lets you easily collect email addresses these days you could always have a page that says, facebook is down click here to get an email confirmation link to login or something similar.
Well frankly, If you rely on some other business for such critical functionality, you deserve to lose.
Unfortunately its difficult to avoid relying on other businesses for crucial systems.

Example: Payment Processing, try building a business, especially an online one, without depending on Paypal, or at least Mastercard/Visa.

Sure you can reduce it by say not having FB Connect as the only log-on option but the simple fact is that you will almost always have to depend on other businesses, the best you can do is diversify and hope for the best.

Exactly...people made this same argument a few years back when Amazon S3 went down for a few hours. Just have a few different options, and the chance that they will all be down at the same time is pretty slim.
If you have a few options, then you don't rely on a single company, do you?
Yes and No. If you have a few options than your entire business won't come grinding to a halt but if half of your customers choose to use FB Connect as their login option than they likely don't care that FB's down, or that people with email logins can still access your site. To 50% of your user-base your site has failed and so in that sense I would say that you still rely on FB for your business.
for like buttons you should iframe them (same for anything from a third-party service)

facebook uptime has been good overall, considering they doubled in size in 12 months to 550m users

Even iframed, you end up with an ugly "service unavailable" blot of text.
Do better systems exist? A few hours downtime in several years is still better than most brick-and-mortar stores, I think.
I thought I had a crappy router, I didn't realise it was that crappy!
They closed shop after DHH's scathing criticism.
DHH's fantasies realized.
More like zuckerberg making a statement to detractors, how important they are.
They knew the gig was up and decided to cash in before everyone realized they really weren't worth $33B.
that and the movie.

enough is enough!

But I can still harvest my cows ...
(comment deleted)
You're not a bowler, are you?
Farmville. Or it is not available in China?
(comment deleted)
Don't place 3rd party javascript on your pages if you don't really need it.
Facebook is our biggest traffic source after Google. I'm inclined to forgive them for occasional outages.
I did say:

> if you don't really need it.

In your case, you obviously do.

I don't think we're uncommon. Most sites that I see plastered with Facebook embeds are publications, and I'm sure they all see results similar to ours. Facebook is RSS for the masses.
PPC? CPM?
Not ads. We post links to people who like us on Facebook, and we also get traffic from people who share our articles with their friends.
The one day I choose to update my FBConnect code!
Rarely use it so I didn't notice...
And that, my friends, is why its risky to build your business on top of FB Graph API.

In terms of reliability, FB is not twitter, but it certainly isn't google either...

I came here to post this same thing, but I just started thinking about it. When was the last time FB went down like this and took a bunch of core functionality down with it? Has it ever happened? And how long are they down...30 mins? Is that 30 mins of downtime every few years really more damaging than that the (potential) value that integration with the graph API or FB Connect can bring? Yes, you look like an idiot to a few users, but maybe you have 100x as many users as you'd have otherwise. I think the real reason not to build your business on the FB API (or Twitter or whoever) is strategic, not because of potential downtime.
They had a BGP-related screw-up a couple of weeks ago as well.
I fully agree with you. If your app is heavily social it would be ridiculous not to leverage the worlds largest social network, even with very occasional down-time.
You're right, I should have phrased it "And that, my friends, is one of the many reasons its risky to build your entire business on top of FB Graph API."

Just went through a nightmare building a product entirely on top of fb graph. FB fails, a lot. And not just in terms of downtime, in terms of inconsistently implemented technology, poor documentation, lack of transparency, inconsistent policies etc.

Not saying don't use FB ever, just that it's risky.

Oh, don't get me wrong. I won't do FB apps for clients anymore, despite the high demand and great pay. It's just way too stressful to have the clients always angry at me because of the platform's shortcomings.
Amen! I tell ya, I have had so much stress over the last few months as the direct result of this. Well put.
If you have 100x as many users, you will look like an idiot to 100x as many users too.
As an aside I notice that 'Facebook' is not a trending topic on Twitter. Seems the term must be blacklisted in some way along with all the swears and other undesirable terms.

Which seems a bit underhand by Twitter.

Edit: I notice that 'Facebook isnt working' has appeared now. I've blogged about this issue here: http://blog.dansingerman.com/post/1174729229/twitter-mostly-... (if anyone cares)

during one of the twitter hacks the blacklist application was accessed and facebook was on the list (as was google and a bunch of other brand names)
I'm curious about this and Googling failed me, do you have a link?
I took a screenshot on my other machine - I will upload it when I am on there next

There were hundreds of items on the list

(Edit: here is this link for now: http://techcrunch.com/2009/07/15/another-security-tip-for-tw...

I am sure that I saved the entire page of blacklisted words)

That screenshot shows which of the current (more or less I guess) topics need to be blacklisted. Not which words are always blacklisted.
Wow. So twitter censors disparaging comments about companies? Wow.
More likely that terms that are continuously mentioned in huge numbers (and therefore don't count as "trending") are omitted.
There are more principled ways to do that than a blacklist, though, like looking at current frequency compared to typical frequency. That way even something commonly mentioned could still genuinely be trending if it gets mentioned much more than usual on a particular day.

(It's possible there are scale-related reasons that make this infeasible, though.)

Just not tracking the frequency of the most common words that will never be in the trending topics is probably a big performance win. I'm sure there is an additional blacklist that includes stuff like "and" and "the".
You know you've made it when your brand name is a stopword.
No, it just prevents comment from trending. Not just disparaging comments, all comments, it seems.
I see "Facebook down again" trending and earlier saw "Facebook down".
Trending topics: Mono Jojoy , Facebook Isnt Working , Bishop Eddie , Internal Server ,
Before, two of the trending topics were "Facebooks" and "isnt working".
I'm sure somebody out there told you so
Current Status: API Latency Issues We are currently experiencing latency issues with the API, and we are actively investigating. We will provide an update when either the issue is resolved or we have an ETA for resolution.
(comment deleted)
Well somebody has screwed it up on their end. Only the www record is not resolving, a lot of the others are, for eg:

http://developers.facebook.com/

  $ nslookup facebook.com ns1.facebook.com
  Server:	ns1.facebook.com
  Address:	204.74.66.132#53
  
  Name:	facebook.com
  Address: 69.63.189.11
  Name:	facebook.com
  Address: 69.63.181.12
  Name:	facebook.com
  Address: 69.63.189.16
  
  $ nslookup www.facebook.com ns1.facebook.com
  Server:	ns1.facebook.com
  Address:	204.74.66.132#53
  
  Non-authoritative answer:
  *** Can't find www.facebook.com: No answer
(YMMV based on local cache etc.)
(comment deleted)
Hilariously touch.facebook.com still seems to work.
I imagine that would be a relatively easy fix, more likely they took the record down to debug internally without getting flooded with requests.
It looks like you were right - they dropped that record to stop the requests coming in
IT Dept. around the world will now get washed with calls, on how the internets dont work.
the BOFH will be making a list of people wasting time on facebook for later blackmail.
joking aside, this actually had this effect, as so many pages actually resolve some components to facebook.com, the slower response was felt on a wide variety of sites.
I [Like] this.
Is this a coordinated blackhat attack with DHH's PR attack intended to benefit some investor somewhere? (joking)
First I just had an empty feed, then I got the feed back but posting links and comments timed out. Then it worked for a minute, and now I get the DNS error. Seems like this affects several of Facebook's applications.
Everyone should calm down and realize that Facebook being down for a few hours is not going to kill anyone's business. It may cause trouble and disruption, but not anything more intense than a daily commute in Los Angeles.

That being said, this makes me question Facebook's ability to deliver on the "Social Graph" promise. If they truly want to build a completely social internet, this kind of downtime can't happen.

> "If they truly want to build a completely social internet, this kind of downtime can't happen."

Why not?

Even commerce sites such as Amazon or Ebay have downtime. Studies demonstrate that unless you're selling a perfectly commoditized generic product, obtainable instantly from multiple sources, downtime doesn't materially impact revenue.

Users seem to manage to "queue up" their intended actions and do them when a site is back. And it's not like Facebook's users can gravitate en masse to anywhere else.

The founding principle of the internet is not "5 nines" uptime for all components, but resiliency. Switch off your mail server for a few hours; you'll still get all your mail when you turn it back on.

"Kids these days" build apps and APIs (and sites that rely on APIs) that are much more brittle, because today's developers are spoiled into thinking communication conduits are reliable. They're not.

The fabric fails, so your site, your apps, your protocols, should fail gracefully -- and recover gracefully later.

(comment deleted)
Nice to see the big boys experiencing mundane issues as well now and then...
I'm actually surprised they don't have weekly maintenance for a few hours...
In A.D. 2010

War was Beginning.

Internet: What happen?

MarkZ: Somebody set up us the bomb.

FB: We get signal

MarkZ: What?!

FB: Main screen turn on.

MarkZ: It's you!!

DDH: How are you gentlemen !!

DDH: All your Information are belong to us.

DDH: You are on the way to destruction.