This is the first time I experience this. Also note that current session on messenger.com still work, we can still send/receive message, but can't upload any image or send sticker. Looking forward for a post mortem analysis on this.
So yesterday Google had a major (and out of character) outage across its apps, and today Facebook has a major (and also out of character) outage across its apps.
I can't wait to see the RCA for both of these and if they're related.
Private post Morten:
The NSA middleware we are required to run (that took time to deploy to each of our social partners) is breaking something so let’s revert.
That's why it's called PRISM. It's exactly what you describe. Splitting an optical signal into 2 using, basically, a prism. One signal goes out to the net as normal, the other goes to their own datacenters, that they keep continually building and expanding. The newer ones are being build on military bases, for added security. Check em out. Look at the size and cost of them. Some are over a million sq. ft. That's a lot of data. They measure it in terms of zottabytes and zettabytes (in 2013, a lifetime ago in terms of storage space):
Nah, PRISM referred to the front door for lawful access to customer records under warrant. That's the sort of portal that China once hacked Gmail by gaining access to.. the companies explicitly built those access relays.
The beam splitter stuff (e.g. Room 641A) went by different codenames, TRAFFICTHIEF and TURMOIL iirc. That's the back door.
> The newer ones are being build on military bases, for added security.
IIRC, the NSA is organizationally part of the military, and it's currently headed by a military officer who gives congressional testimony in his uniform (https://www.youtube.com/watch?v=nMi241XLeQ8). It makes sense they'd build on military bases, it'd be kinda weird if they didn't.
Also, the majority of tech and data companies have closed this loophole by encrypting traffic between data centers. Nobody thought it was necessary to do it before over dark fiber before because, hey, who was listening? (answer: the NSA was)
One of my coworkers came from a large telecom. He mentioned they had to get technology from an Israeli firm that specializes in quantum cryptography on the fiber optic line to fend of NSA and GCHQ, who are apparently worse than NSA. (iirc) the tech apparently encrypts data streams on one side and check to see if the hash is the same on the other side, if somethings off (evidence of tamper) it instantly changes the cipher.
Optical tap for unauthorized access, but surely port mirrors for the national-security-letter stuff. Wouldn't make sense to go through the hassle of a tap install and the ongoing risk of it failing, versus using a capability available on almost all serious switching hardware to give you a guaranteed 1:1.
And they have been using the USS Jimmy Carter sub with the front huge cable splice bay for decades to compromise all undersea cables.
>>>The New York Times reported in 2005 that the USS Jimmy Carter, a highly advanced submarine that was the only one of its class built, had a capability to tap undersea cables. An Institute of Electrical and Electronics Engineers report speculated that a 45-foot extension added to the Jimmy Carter provided this capability by allowing engineers to bring the cable up into a floodable chamber to install a tap. But it is unlikely that the USS Jimmy Carter routinely taps cables since U.S. intelligence agencies can much more easily (and lawfully) obtain cable data through taps at above-ground cable landing stations.
I don't think it's the NSA this time, for once they don't have to do deep package analysis or install any MITM device since they get the whole info in bulk, maybe it's just a 400-pound hacker.
interesting that facebooks cavalrylogger is still being sucessfully injected despite their being nothing but a blank page
also interesting that cavalrylogger has a function that lets you bind key-presses to callbacks
even more interesting is that cavalrylogger seems to come prepackaged with any facebook like button! cheers for the keylogger facebook
Meanwhile in Russia they are talking about disconnecting their network from the rest of the world.
Some test gone south?... Maybe...
Someone has a traceroute handy?
^^^ I noticed this too. GCP is under all 50 shades of outages since past few days. Feel I might need to rush back to my house and start digging a bunker
The Bay Area Peninsula has been having strong winds and heavy rains for the past few months. The last 3 days, there have been major power outages across the area. Redwood City had power outages 2 days in a row and Pacifica lost power to a good chunk of the city for like 7 hours last night. It wouldn't surprise me if all these major tech outages that have been happening this week is all related to poor Bay Area infrastructure.
Can you imagine if Twitter and Google went down at the same time?
People would be reactivating their Facebook accounts and having to sift through conspiracy theory posts about Hillary Clinton still just to figure out what was going on.
Edit: The points on this post keep going up and down every time I check these comments. Yes, it was sarcasm, I was joking, but I was trying to point out that most people rely on a small set of services. "Cloud" has centralized things a lot.
I think the only reason your comment gets down voted is because it's biased only to one side. I see about what Jon Oliver states as if I remember correct 1/2 as many conspiracy posts from about Trump (Edit: I meant to write conservatives write 2x as many conspiracy posts as democrats. This was not represented in my previous sentence.). It's annoying.
I guess to accurately represent things on Facebook you'd have to say something like conspiracy posts about AOC, Bernie, and Dan Chrenshaw. 2x1 Ratio.
You could have also said, "political conspiracy theories". I get annoyed by conspiracies on either side personally, but I can see how some people would see more of one than the other based on their friends.
Edit: Re-read some of what I wrote and I want to clarify, this isn't me saying i down-voted you. Just an observation.
Jokes, typically hint at a truth or belief of the comedian. I'm not saying it's a bad joke, I actually liked it. I was merely making an observation which apparently a lot of people disliked, and it looks (based on the upvote/downvote feedback) I am probably wrong.
It might just be because a lot of people around here read thishttp://www.paulgraham.com/hackernews.html and think that any joke means it's a bad comment. IDK. As I said I thought your comment was funny.
it really goes to show how different all of our feeds are based on who we are friends with (and of those, who we interact with the most). Anecdotally, I have 3 people who will share any "conservative" attack meme (at this point, I don't even know if it counts as conservative so much as just outright attacking Democrats. Sometimes it reads more like an attack for the sake of attacking than a statement of belief in something different. Kind of weird. Part of me wonders if maybe some of these accounts do conservative attacks and some do liberal attacks in an attempt to get shares and what not with no political interest whatsoever: effectively acting as an arms dealer of the meme variety.) they can get their hands on and of my friends of a more liberal view, it is mostly policy things they share (pro-choice, anti-rape, etc.). There's one dude that is pretty anti-Trump, but his for the most part stand out as an exception. Most of the ones I see referencing Trump directly (if it isn't during a period where he did something that Democrats felt was highly suspect) are more in support of him than anything.
Which is why I have to refrain from taking an "over the top much?" slant when people post the pro-Trump/victim of the left type memes as I don't see a ton of attacks on him directly, but then again, their feed could be totally different from mine so who knows?
I've got a few people in my feed who just dump on the right no matter what. I've fact checked them a little bit. Again it's about a 2-1 for me fact checking my right leaning and left leaning friends (typically older). But I keep all of them on there just to have an ear to the ground.
Yeah, I appreciate the various view points and items shared in my feed from both conservative/liberal viewpoints have been wrong (almost consistently so).
I think the real bummer is when you present the actual video of what the person says, and the response is essentially “yeah? Well, they still suck” or something in that ballpark. I have zero issue with someone not liking another person’s views, but a lot of it is just outright libel.
I don't believe Gmail was ever fully down. For me, I was just having problems with attachments. I also noticed app icons in the play store failing to load.
I experienced issues with Drive last night but it was never fully down. I was trying to work on a 500GB file and the API would drop the link intermittently. I could parse the directory no problem, just couldn't reliably access the files.
Back in my tech support days I received an email from a customer "I am unable to send or receive emails" I replied "I am very sorry for your inconvenience, I have resolved the issue".
Customers in 1999 really couldn't believe no one had replied to their emails within a day or two.
When it all goes down at the same time you should be worried. Not because of the lack of Twitter or FB or Gmail bit for what it means if it's all down.
Whenever I hear when some service is down, I immediately go to that service to confirm. Then I repeatedly hit reload if it doesn't work to see if it can come up. I guess many people do the same and that may contribute to the problem...
Years ago I worked at a large online casual gaming company who's name ended in -ynga. Our web tier was split into two: one for serving static content required to load the HTML, Flash app, assets, etc. The other was for actual communication regarding actions taken in game.
Whenever we had any sort of issue we could generally get a good idea of what was happening by looking at changes in traffic in those two web tiers.
If people couldn't play for most reasons, game action traffic would drop to near zero, but the static asset tier traffic would usually at least triple.
So yeah, there are a lot of F5 buttons being hit out there when pages don't load.
Increased Error Rates Created by Gary Fitzpatrick · · Facebook Team — Today at 10:32 AM
Current State: Investigating
Description: We are currently experiencing issues that may cause some API requests to take longer or fail unexpectedly. We are investigating the issue and working on a resolution.
Start Time:
2 hours ago
Last Update:
about an hour ago
Updates:
There are currently no updates for this issue.
I've also seen issues uploading images to Whatsapp in the past half hour. I wonder if there's anything to do with the Google Cloud Storage outage that took down Gmail yesterday?
What manner of failure would cause such globally deployed and distributed systems to go down like this? I'm very interested to read up on this when they release details of the failure.
I work for a smaller but comparably large platform. "If everything is down check the DB" is at the top of one of our internal monitoring websites in red.
Screw ups related to data loss are rare (I've been here years and haven't seen one with the DBs that the stuff I work with uses) but failures at this scale tend to cascade a little ways and it takes time to dig out of the hole. They probably have the problem solved but they have to spend a bunch of time synchronizing things and verifying the fix before they press the big red "go live" button.
We have a different dedicated page that gives an overviews of what's going on with the DB. The page in question is supposed to be a single stop that lets you visually get an overview of the state of the application servers and whether things are "normal" and if not allow you to quickly identify what is not normal.
As someone who works at a large company in the networking space, you would be surprised that minor changes to configuration can cause catastrophic failures that are really challenging to come back from
Network failures are usually really bad when your system is globally deployed and distributed -- often times you can't even communicate with your machines to deliver fixes :p
I have no inside knowledge of this one, but broadly speaking, these sorts of failures can be caused by a change thought innocent at the time to the core software that is then widely deployed using automated systems. If the core's tests didn't catch a real issue in production (and for whatever reason, the rollout happens faster than the regular small-release verification process can catch the error), things can go sour in a way that's expensive to un-sour.
Amazon once pushed a seemingly-innocuous change to their internal DNS that caused all the routers between and within datacenters to drop their IP tables on the floor. They had to re-establish the entire network by hand---datacenter heads calling each other up and reading IP address ranges over the phone to be hand-entered into lookup tables. Cost a fortune in lost sales for the time the whole site was inaccessible.
My hunch is that it's the end of Q1 and people are trying to release code changes so they can pad their Q1 performance reviews "designed and delivered feature X on time in Q1".
I've seen all quarter ends being targets for releases. And things that have been delayed since start of the end of the year are usually pushed to the end of Q1.
We're up to around 7 hours of partial outage now. I have spoken to FB employees in the past, every hour they are even partially down they are loosing millions and millions in revenue.
350 comments
[ 3.0 ms ] story [ 286 ms ] threadI believe this is why Github's status page is now on its own domain; so a github.com DNS outage won't take it down.
I can't wait to see the RCA for both of these and if they're related.
I can't wait to see the RCA for both of these and if they're related.
Public post Mortem:
Entirely believable technical cause.
(Ignore Stuxnet, Ignore DUQU)
It's the rice grain implants, man
https://modernsurvivalblog.com/government-gone-wild/nsa-loca...
http://worldstopdatacenters.com/government-data-centers/
The beam splitter stuff (e.g. Room 641A) went by different codenames, TRAFFICTHIEF and TURMOIL iirc. That's the back door.
IIRC, the NSA is organizationally part of the military, and it's currently headed by a military officer who gives congressional testimony in his uniform (https://www.youtube.com/watch?v=nMi241XLeQ8). It makes sense they'd build on military bases, it'd be kinda weird if they didn't.
Also, the majority of tech and data companies have closed this loophole by encrypting traffic between data centers. Nobody thought it was necessary to do it before over dark fiber before because, hey, who was listening? (answer: the NSA was)
>>>The New York Times reported in 2005 that the USS Jimmy Carter, a highly advanced submarine that was the only one of its class built, had a capability to tap undersea cables. An Institute of Electrical and Electronics Engineers report speculated that a 45-foot extension added to the Jimmy Carter provided this capability by allowing engineers to bring the cable up into a floodable chamber to install a tap. But it is unlikely that the USS Jimmy Carter routinely taps cables since U.S. intelligence agencies can much more easily (and lawfully) obtain cable data through taps at above-ground cable landing stations.
https://www.lawfareblog.com/evaluating-russian-threat-unders...
https://www.washingtonpost.com/news/post-politics/wp/2014/03...
https://en.wikipedia.org/wiki/USS_Jimmy_Carter
I don't think he made any pre-Snowden comments on mass surveillance?
https://stackoverflow.com/questions/4188605/what-is-cavalryl...
Yet another alternative: Third World War has just started, and this was the first battle.
[1] https://www.abc15.com/news/national/facebook-down-social-med...
[1]https://www.manchestereveningnews.co.uk/news/uk-news/faceboo...
https://www.businessinsider.com/verizon-outage-on-east-coast...
People would be reactivating their Facebook accounts and having to sift through conspiracy theory posts about Hillary Clinton still just to figure out what was going on.
Edit: The points on this post keep going up and down every time I check these comments. Yes, it was sarcasm, I was joking, but I was trying to point out that most people rely on a small set of services. "Cloud" has centralized things a lot.
I guess to accurately represent things on Facebook you'd have to say something like conspiracy posts about AOC, Bernie, and Dan Chrenshaw. 2x1 Ratio.
You could have also said, "political conspiracy theories". I get annoyed by conspiracies on either side personally, but I can see how some people would see more of one than the other based on their friends.
Edit: Re-read some of what I wrote and I want to clarify, this isn't me saying i down-voted you. Just an observation.
It might just be because a lot of people around here read thishttp://www.paulgraham.com/hackernews.html and think that any joke means it's a bad comment. IDK. As I said I thought your comment was funny.
Which is why I have to refrain from taking an "over the top much?" slant when people post the pro-Trump/victim of the left type memes as I don't see a ton of attacks on him directly, but then again, their feed could be totally different from mine so who knows?
I think the real bummer is when you present the actual video of what the person says, and the response is essentially “yeah? Well, they still suck” or something in that ballpark. I have zero issue with someone not liking another person’s views, but a lot of it is just outright libel.
Google+ APIs were shutdown 9 March 2019
It was not fully down, there were issues with their blobstore.
Customers in 1999 really couldn't believe no one had replied to their emails within a day or two.
Google sure, but what people in the real world cares about twitter?
Twitter could be down for days and only the technocracy would notice.
It also seems popular with journalists and media companies (e.g. TV shows asking viewers to "tweet us your questions")
Wait, people are doing this already: https://twitter.com/SA_SES/status/1105969450698694656
Whenever we had any sort of issue we could generally get a good idea of what was happening by looking at changes in traffic in those two web tiers.
If people couldn't play for most reasons, game action traffic would drop to near zero, but the static asset tier traffic would usually at least triple.
So yeah, there are a lot of F5 buttons being hit out there when pages don't load.
https://developers.facebook.com/status/issues/55989644784543...
Current State: Investigating
Description: We are currently experiencing issues that may cause some API requests to take longer or fail unexpectedly. We are investigating the issue and working on a resolution.
Start Time: 2 hours ago
Last Update: about an hour ago
Updates: There are currently no updates for this issue.
Well someone tell him to stop, for pete's sake!
https://www.thesslstore.com/blog/expired-certificate-ericsso...
Source: Me. My career has been spent managing db's for internet scale sites.
Screw ups related to data loss are rare (I've been here years and haven't seen one with the DBs that the stuff I work with uses) but failures at this scale tend to cascade a little ways and it takes time to dig out of the hole. They probably have the problem solved but they have to spend a bunch of time synchronizing things and verifying the fix before they press the big red "go live" button.
Network failures are usually really bad when your system is globally deployed and distributed -- often times you can't even communicate with your machines to deliver fixes :p
Amazon once pushed a seemingly-innocuous change to their internal DNS that caused all the routers between and within datacenters to drop their IP tables on the floor. They had to re-establish the entire network by hand---datacenter heads calling each other up and reading IP address ranges over the phone to be hand-entered into lookup tables. Cost a fortune in lost sales for the time the whole site was inaccessible.
All joking aside, is this news? :/
Yes, that seems like appropriate news for "hacker news", a website where people discuss technology news.
i'll put the sarcasm machine back in the drawer.
My hunch is that it's the end of Q1 and people are trying to release code changes so they can pad their Q1 performance reviews "designed and delivered feature X on time in Q1".
You'd think being down for hours would be negative news and revenue impacting.