317 comments

[ 4.1 ms ] story [ 274 ms ] thread
Is just a temporary degradation, probably...

Edit: Err, probably more serious

And cue all other major services going down in the next few days.
I'm not at that level of paranoia yet. In fact, I'm a long way from it.

If it happens, though, my paranoia is going to climb dramatically...

Same here in socal, and look at those report numbers. Baseline 30, now almost 190,000 reports. I wonder what could be going on to cause this level of global service degradation.

Perhaps related, I uploaded a video earlier today that took nearly 4 hours to process, though it was only ten minutes long.

Microservices in action! Can visit the site, see comments, but can't play videos. Go to twitter, check the latest tweets containing "youtube down", thousands of results coming.

Now I know why I used to download all my favorite videos to USB/SSD!

Microservices out of action in this case
Select microservices in action! If AWS/GCP/Azure are taking score, then this isn't an outage because DNS resolves a-ok!
What on Earth are you on about
GCP doesn't consider it to be an "outage" if you are able to access a website (e.g "youtube.com") and get a 200 OK response. So, when you go to the status page for the cloud service, it will show green checkmarks, meaning everything is ok.

But, the video you are trying to watch won't load, and will throw a 500 error. But this isn't considered downtime, for them.

Ah I see.

Yeah I've had nothing but grief with those outage detectors. They seem to be really optimistic and not take into account the web of CDN's that contribute to the 'uptime' of a website

Same here in socal, and look at those DownDetector report numbers[0]. Baseline 30, now almost 200,000 reports. I wonder what could be going on to cause this level of global service degradation.

Perhaps related, I uploaded a video earlier today that took nearly 4 hours to process, though it was only ten minutes long.

[0]: https://downdetector.com/status/youtube/

So is speedtest.net. I wonder how they're related (if at all)
Maybe it's something Google Cloud related?
Maybe speedtest.net downloads a YouTube video to measure your speed /s
Seems like just the video player is down - homepage loads with thumbnails, and when you click on a video, everything but the video player renders. What I find interesting is that scrolling through a video still displays closed captions and even the thumbnail preview of the timestamp you're hovering over. Curious if anyone know what causes something like this?
Captions and thumbnails come from a different part of the CDN. Same with Netflix. The videos come from devices dedicated to large files, the captions and thumbnails come from a CDN designed for small files.
How fun - same goes with the previews on hover over timestamps then, I guess?
Most likely. Any small file comes from a different CDN than the video.
Interesting, thank you! :)
thumbnails both static and animated are served from i.ytimg.com/vi/ thumbnail preview is also server from i.ytimg.com/sb/

videos however come from redirector.googlevideo.com/videoplayback

Was just about to post this... pretty crazy
So is speedtest.net. I wonder how they're related (if at all)
190,000 people running a speed test at the same time.

Just a guess.

Sounds plausible, my first guess when youtube went down was that my connection was messed up
So mine both restored working at the same exact second. speedtest and youtube. Just saying...
Oh I’ve had my internet troubles fixed several times by going to that domain. Saw one guy write a script to ping it every 15 minutes.

Now to be fair, When in a meeting, I’ve al had to track down the kids and all devises that could possibly access YouTube.

After a really bad hour of Zoom; I leaned that you can watch YouTube on the switch. Clever kids.

I locked away a 10 year kindle just out of principe for that one.

Most routers support some form of qos
You can firewall their devices. I used to do that to my dad's cloud-based cameras back when I was in highschool. He never figured it out. Always assumed it was the ISP's fault.
You may want to invest in an IQrouter or an equivalent DIY, though if you DIY and your issue is wifi congestion you'll find that only applying fq_codel to the modem side won't help with the wifi side.
It looks like it's a CDN issue, which might affect many things using GCP for large files.
The Play Store also seems to be having problems downloading APKs. CDN issues sounds pretty likely.
Speedtest and youtube restored for me at the exact same second. I had the windows open side-by-side. Not my area of expertise but I believe you now
If I made a speed test I'd probably include streaming videos from the common sites.
Maybe speedtest.net downloads a YouTube video to measure your speed /s
I vpn'd around, this appears to be global. While reading something unrelated my video suddenly started playing.

I think it's load balancer related for whatever is serving their videos

EDIT: Once you have a connection you appear to keep it the entire way through (at least it did for me with youtubedl). Something related to connection queueing...

yep, had long video opened (in mplayer) when this started. I stopped it while reading your post, trying to reconnect results in 502.
Downdetector seems like a great place to advertise. You have a bunch of people who were forced to abandon the provider of whatever type of service that your company may offer. What a great opportunity to show customers your competing product.
Maybe for other types of service yes, I find difficult to plug a alternative to YouTube, other type of streaming services maybe? Netlfix etc
Vudu, Spotify, Pandora. I think there's a wide enough overlap that plenty of people could say "give us a try" when YouTube goes down.
Yeah... But you still don't get the content from the youtube creators that you follow, usually I go for youtube just to see a few selected channels
Vimeo is my first stop when I can’t get it on YouTube.
The problem is that most of my subscribed channels are not in Vimeo, so I lose a lot of meaningful content
I wonder why syndication is not more of a thing. Surely there must be a simple tool to upload your video to multiple platforms at once?
Would you risk losing ad revenue if you posted to content providers who pay less?

(I have no idea whether YT revenue is meaningful for smaller players, to be clear; I suspect not, but I also suspect it's worse elsewhere.)

LBRY and Bitchute can automatically mirror your YT channel, and IIUC the former can also automatically upload videos to YouTube, maybe even on a customizable delay. Some decent sized channels like minutephysics use LBRY, but even though anecdotal accounts report much higher revenue per view and no decrease in YouTube exposure, it’s still certainly a uphill battle to compete with a de facto monopoly that can afford to hemorrhage money and just get more from a massive gatekeeper of the web.
I'm pretty sure Vimeo is paid, so it's sadly a non-starter for many creators.
It's niche but I like that a lot of filmmakers seem to be on there. Often I can find the individual people behind well-produced Apple ads for example.
YouTube Music user here. Fired up Windows Media Player for the first time in ages, and told it to scan my old folder of ripped CDs.
If you're Windows user maybe try lightweight option like foobar2000?
or WinAmp?
Is WinAmp still alive? I last used it a couple of decades ago!
That llama's ass ain't gonna whip itself!

https://www.winamp.com/

There's a 2018 copyright date in the footer, so I guess that's as alive as it needs to be.

YouTube is the one-stop-shop of internet videos. Any specific niche has a competitor, but none of them seriously competes with the whole of YouTube.

I noticed the outage because I wanted to watch a stream, so I might go to Twitch now. Often I use Youtube for music, where Spotify is a solid alternative with many of the same creators. If I want to see that video from reddit of people throwing burning flares at a balcony I can probably find it on LiveLeak. Many of the educational channels I've subscribed are also on Nebula, but I could also watch a documentary on Netflix or Disney+ instead.

My daughter could not do her homework assignment, her Google classroom is linked to YouTube. Thank goodness I was able to find the video she was looking for on Dailymotion. I haven’t been to the site in over 10 years
'PeerTube: the federated video streaming service that's resilient to down time'
Wouldn't that create an incentive for third parties to block Downdetector, negating its value?
At least one vlogger I follow posts duplicate content on Bitchute (which is up).
Wow, the front page of bitchute is 100% conspiracy. Is that... umm... what it's known for?
> Wow, the front page of bitchute is 100% conspiracy. Is that... umm... what it's known for?

When your main differentiator is "free speech" and your competitors don't actually restrict the speech they carry that much, you're pretty much only going to get the dross that was cast off the other platforms and very little else.

I guess that there are a lot of fringe channels on Bitchute that are banned or restricted on Youtube or Facebook Video.

But Youtube doesn't only ban conspiracists and hate speech. They also will restrict content that violates their community standards such as explicit instructions on slaughtering chickens on a farm, or how to field-dress a deer when hunting.

Even the Rob the Ranger channel has had videos taken down for inappropriately explicit content, when showing lions mating for example.

Youtube throws a wide net, and thus rather unfairly censors innocent content. Bitchute, peertube, etc. provide alternative ways to get that content online while you're waiting patiently for your appeal to go through.

And yeah, there's some content that the monitors at Youtube consider objectionable and will never be allowed, and maybe should never be allowed -- snuff films, child porn, terrorism how-to training guides, stuff like that. But I think Bitchute also takes down such content.

The only time I've used it is for inrange.tv content that isn't allowed on YouTube - things like uncensored gun manufacturing or assembly instructions.
Was wondering so looked up how much YouTube lost per second.

> YouTube ads generated $15.15 billion in revenue in fiscal year 2019.

So plugged into wolfram alpha:

https://www.wolframalpha.com/input/? i=15.15+billion+dollars+per+number+of+seconds+in+a+year

They lose about $480 per second or $1.7MM per hour. Yikes.

And we know that it isn't flat throughout the day. I imagine this is probably peak earning time for them, assuming they bulk of the revenue is US based.
Why MM, is it the standard for million akin to "k" is for thousand?
MM looks like "million million" to me, which is the old standard for billion but nowadays a billion is just a thousand million and a million million is a trillion and I am constantly feeling frustrated about how wasteful we are with words for big numbers.

But I think in this context MM is a million.

That was my exact thoughts seeing MM too yeah, the Roman M symbol meaning thousand clears it up though. Just a historical oddity.
"M" is the roman numeral for thousand. It is therefore ambiguous. Especially when dealing with money, you really don't want a million to be confused with a thousand.
MM means mil mil or 1000 1000. Its the roman numeral for 1,000,000 and typically used today in finance.
MM is 2000 in roman numerals though. It's additive, not multiplicative.
In finance mm is generally accepted to mean million.
"Mille" is French for 1,000. So MM = 1,000 * 1,000. M is also used in advertising, like CPM: Cost Per Mille.
Also plus the cost of losing users due to long term adverse effects of this incident.
I don’t think anyone is going to stop using YouTube because of a small downtime. There is really no alternative to it out there.
What about the new users visiting the site for the 1st time in that small downtime window? If their first experience sucks there is likelihood they won't come back.

So:

(number of new users) x (probability of those users not coming back) x (lifetime value of a youtube user) = I highly doubt this is equal to 0

I truly believe that the number of people that navigated to youtube for the first time ever in their lives during that window might actually be 0.
So you believe YT's growth rate == 0?
So we've already broken 5 9's of uptime, will we also break 4 9's as well?
On client side looks like requests for the video stream return 502 'bad gateway'.

I actually get back a little webpage it says:

    502. That’s an error.

    The server encountered a temporary error and could not complete your request.

    Please try again in 30 seconds. That’s all we know.
> XXX. That’s an error. > > [...]. That’s all we know

How old is this? I think I've always seen this error message from Google. It's at least 15 years old.

Still dead in California.
502 bad gateway (21s response time) for redirector.googlevideo.com
5 bucks on BGP
Possibly, but the pages and ads load fine, just the videos are broken, so I'd take your bet (idiomatically; I'm not a betting person).
That makes sense to me, because dynamic pages are not served from a CDN, which do a lot more BGP and routing trickery that could go wrong.
That's fair, but Google's BGP errors tend to be global. To my knowledge they've never killed their CDN and not everything else. (I have no non-public knowledge, I just pay attention).

Will be looking forward to their report.

I get a 502 error trying to load the actual video stream
Networking would be a likely cause.

Usually this type of site-wide outage is always related to networking.

BGP, not sure. Google practices SDN. So their software error can rekt a large chunk of infrastructure...

My understanding is that there was a pretty major fiber cut somewhere on the backbone...
IT'S BACK
Perhaps context will help: Pacific Northwest (PNW) here.

Seattle.

EDITs: More specific in location.

Pacific Northwest? I've lived in the US all my life and had to guess, not sure PNW is known to most outside of the Western North America
I'm in Oregon, and it's still down for me.
Down in eastern Washington still..
Pages load, but videos don't.
Captions, thumbnails, webpage html, etc. come from a different part of the CDN. Same with Netflix. The videos come from devices dedicated to large files, the captions and thumbnails come from a CDN designed for small files.
my browser shown an CORS related error, its weird
CORS errors are false positives for a large amount of causes. For example, if loading a website causes an 502 or other error, that might not set the required CORS header, thus making it illegal for javascript to access the response in a number of ways.
Too bad, the popup for youtube premium still works.
"Do you want videos to continue playing in the background?" is an even more annoying ad when the video isn't playing at all
.. and I just managed to get the YouTube app on Android to play an ad. That's two for two of the annoying things that are still (sorta) working. ;)