174 comments

[ 2.7 ms ] story [ 223 ms ] thread
Telcos will have to invest in capacity where Google wants in order to get a good rating. Since Google is a de-facto monopoly, I think we'll see an Antitrust case soon.
How is youtube a monopoly?

There are many free-to-upload/host videosistes.

Monopoly is defined by market share or market power. In the strict sense, it means "single provider", but a sufficiently large company can exercise an effective monopoly.

Having a monopoly is not of itself illegal. Using it in certain ways is.

Google are effectively a monopoly in search space. They're a massive presence in online advertising and video, as well as email.

If you look at the actual requirements they're talking about, they aren't that crazy. Even for HD video, they're talking > 2.5Mbps. So for really good quality, it's probably 5-6Mbps.

I have 85Mbps FiOS at home, and I still sometimes have issues streaming Youtube videos.

If my ISP can't even guarantee 10% of what I'm paying for over a fiber optic line, then yes, they need to be called out and publicly shamed.

What about if youtube.com's CDNs are overloaded and slow? Will you blame your ISP?

This is the problem with such ratings systems -- they're inherently one-sided.

I actually have much more faith in Google's capacity and CDN than I do in Verizon's history of sleaze.

After all, Google doesn't have a conflict of interest, Verizon does. The better Netflix and YouTube get, the closer I am to canceling my Verizon TV service.

Why would this be the case? It is only in Google's best interests to provide the best streaming service possible. It's not like they have any shortage of servers, bandwidth, or funds with which to do so.
5mbps per user, times tens or hundreds of thousands of users in a metro area.

Video is the thing that will finally get us faster IP networks, as it slowly replaces TV as the modern opiate of the masses. Even BitTorrent wasn't enough.

$60-80 a month for each one of those users (our bill is about $120 counting TV as well).

Not counting the fact that local governments subsidize the last mile. Also that routers and networking equipment keeps getting cheaper and faster.

It'll take a lot before I start to feel sorry for my ISP.

The thing is, I get amazing speeds from other sites and very good speeds from my EC2 boxes. Downloading MS/Ubuntu ISOs are just beautiful.

But those just aren't mainstream uses so I'm sure Verizon doesn't throttle those servers.

For the record I hate Centurylink. But I have 12 Mbps DSL with them and have never had a problem with youtube.
Google isn't close to a monopoly in the legal sense - especially in the online video delivery market. This will shine a light on ISP data handling and throttling.
I don't know how this could be antitrust. On the contrary, it seems like it would be protected as free speech.
Good. I swear youtube is throttled by Time Warner Cable for me. I stream Netflix, Hulu, etc and never have problems with them. I can confirm that I'm getting my speeds of 20MBit/s down, and yet almost every video on YouTube buffers. If I use youtube-dl to download the video, I can confirm that my actual connection speed is somewhere around 60KB/s. If I then use youtube-dl on my VPS with the same video, I get somewhere between 2MB/s and 8MB/s.
^this. Same with Netflix though for me. Where are you based? I'm in Austin, Tx.

Any of the speed tests online will report 30mbps/5mbps, but I'm unable to stream HD on Youtube or Netflix during peak times.

I'm in Cleveland, OH. Like I said, never have problems with other services like Netflix or even streaming videos by vimeo or skydrive, just Youtube. It's so frustrating I've considered setting up a network-wide VPN for my home network to get around it
Same for me on Time Warner in Austin. I can't wait for Google Fiber to roll into town.
At peak times it isn't out of the question that the video service itself is a bit under the weather. I've had some troubles with Amazon at peak time, and it's always been with fresh content, such as yesterday's television show that just became available on Amazon four hours ago. It wasn't obviously my network, as switching over to Netflix worked just fine at full quality. (That's not a proof it's not still somehow network related, but it strongly suggests it wasn't my ISP.)
I'd be happy if my TWC in Austin would just stay up. I've had many techs out and the service goes down daily for sometimes hours at a time. ugh...
I think Comcast is doing the same thing here in Colorado. Lately, I've been seeing quality drop in videos where it shouldn't. Running speed tests or downloading from newsgroups I'm hitting 60mbps.
Same here for Comcast in Seattle. From 6-8pm I can't stream youtube on anything more than 360p or it buffers constantly.

Other video streaming services seem to stream fine (Netflix, Hulu, Amazon Instant Video) and speedtest.net reports 56Mbps down, it's just youtube that suffers.

Ditto here re Comcast, but in Mountain View. I am a stone throw away from YouTube HQ and still get bad performance. Tried the 8.8.8.8 DNS and not much different. Comcast is definitely squeezing the pipes!
You'll be happy to know that Comcast's Denver ibone switching/routing has been over capacity for many months at this point...Same goes for their Oregon switches (which service SV and the PNW seaboard)
Same thing happened to me with AT&T in Tulsa with their fastest offering (I think it was 24/3 at the time). All speed tests, downloads, uploads were almost always the advertised speeds, but I couldn't watch Netflix or HBO GO, or Hulu. I could, but it was unbearable. To the point of kicking over magazine stands and throwing Nintendo controllers at the TV.

When I called support about this they claimed it was my setup. A Time Capsule connected to their modem/router. Whatever.

I switched to Cox (100/25) and get better than advertised speeds in all downloads, uploads, and speed tests, and have not had a single issue with buffering on any service. I have even been choosing the 1080p option on YouTube without any buffering.

The only problem I have now is whenever my Time Capsule is directly connected to my Motorola modem it causes the modem to restart, so I had to use a slower, older Linksys in-between the two until I figure out what's the cause of that.

Similar prob with AT&T in SoCal, bad Netflix, Youtube, and actually, almost all large file transfers of any kind, including linux isos, apple dmgs & system updates, etc. Switched to an independent ISP carried over the same AT&T DSL lines and I could actually get the advertised bandwidth consistently.

I suspect the only reason DSL is accessible by third-party ISP is that it carries some regulatory access rules with it because only U-Verse can get to the higher tiers of bandwidth.

My broadband choices are lower speeds with third-party DSL 6Mbps, or higher speeds with AT&T U-Verse, or TWC both of which seem to selectively throttle competing video services like YouTube or Netflix.

Yup. This article is a not so subtle jab at this and trying to educate consumers that their poor video experience isn't Google's fault. Both Google and Netflix are raising the bar here, customer demand/complaints are providing the necessary energy and the ISPs are maneuvering to get more money out of someone.
I feel like Verizon Fios does the same. I have 50 down and YouTube is painful most times
Are you by any chance using non-TWC DNS servers? I had this same frustrating problem on Comcast, and it turned out that their CDN works by using DNS to direct you to local servers; by using the Google 8.8.8.8 etc I wasn't getting that.

Even still, though, I run into certain Youtube videos that absolutely refuse to ever fully load -- it's always non-popular ones, and it's like they cache miss and just never get surfaced for me to watch the whole way through.

Have had the same experience on Comcast.
I am using different nameservers, but I've tried switching to the default nameservers and nothing appeared to change at the time, maybe I'll have to try it again
> it turned out that their CDN works by using DNS to direct you to local servers

Are these servers' IPs static? Can they be hardcoded in your hosts file?

I use TomatoUSB on my home router, which includes dnsmasq (under Advanced / DHCP/DNS, there's a textarea for "Dnsmasq custom configuration"). I have a line in there:

    server=/netflix.com/75.75.75.75
which tells dnsmasq to use my ISP's dns (75.75.75.75) instead of my default (8.8.8.8 and 8.8.4.4), only when resolving domains that look like netflix.com.

I don't actually know whether this is an anchored regex or what, maybe I'm spuriously using my ISP's dns for foonetflix.combar.info, but I don't really care.

All devices that use DHCP use the router (and thus the custom dnsmasq config) for dns by default. It improved my ability to stream netflix at the time, though I didn't measure it. TomatoUSB has neat-looking graphs, maybe I should run a test.

Edit as to your question:

> Are these servers' IPs static? Can they be hardcoded in your hosts file?

I suspect they use AWS for at least some layer in between you and the bits you want (if not the actual CDN, the server pool that generates the token that you need to get the bits from the CDN), so I don't think there would be a set of IP addresses to hard-code.

Kinda off-topic, but why do you use Google's DNS (8.8.8.8 and 8.8.4.4)? Unless I'm missing something, by doing so you are basically giving them a record of every domain you access. I don't really trust my ISP either, but why share that information with anyone else (especially a company with great interest in data mining everyone)?
Assuming you don't believe they are lying:

We built Google Public DNS to make the web faster and to retain as little information about usage as we could, while still being able to detect and fix problems. Google Public DNS does not permanently store personally identifiable information.[1]

More details about exactly what is logged at [1]. As for why, speed and security are the main benefits touted [2].

[1] https://developers.google.com/speed/public-dns/privacy

[2] https://developers.google.com/speed/public-dns/docs/intro

Despite being a (bit of a) privacy enthusiast, I still use Google's DNS servers, for a few reasons:

1. I use chrome. If I didn't use Google DNS, they'd be able to track 90-99% of my internet usage just by watching what domains I visit in chrome. The remaining portion is e.g. irc, usenet, xmpp, ssh, etc, that doesn't go through chrome. I could use firefox or opera or w3m or telnet or whatever, but I like chrome.

2. I became disinterested in using comcast's dns because of previous internet-breaking shenanigans like http://corporate.comcast.com/comcast-voices/domain-helper-se... .

3. I work at YouTube (owned by Google), so I get to see the internal temperament and discussions regarding privacy, data safety, PII, and so on. I'm comfortable letting Google have this data.

  > I use TomatoUSB on my home router, which includes dnsmasq (under Advanced / DHCP/DNS, there's a textarea for "Dnsmasq custom configuration"). I have a line in there:

      server=/netflix.com/75.75.75.75

  > which tells dnsmasq to use my ISP's dns (75.75.75.75) instead of my default (8.8.8.8 and 8.8.4.4), only when resolving domains that look like netflix.com.
You are my hero, good sir! You've just inadvertently provided the solution I didn't know existed to fix my Netflix and Hulu issues.
To be fair, I'm just repeating something I saw on HN before. Here, I dug up the original post, give that person your upvotes: https://news.ycombinator.com/item?id=6393829

Also, please consider sending the author / maintainer of dnsmasq (Simon Kelley, simon@thekelleys.org.uk) a thank-you email. Same for the maintainer of your router's after-market firmware, if you use such a thing.

I found the same thing. I reports it to TW and (unsurprisingly) they refused to admi anything. Switched my DNS to OpenDNS and YouTube works perfectly.
I have TWC and have had a lot of success in speeding up Youtube by blocking access to the caching servers that TWC uses.

If I block 206.111.0.0/16, I can stream at full speed.

See: http://www.reddit.com/r/technology/comments/13kmvd/have_time...

I tried blocking some of the relevant IP ranges, but it seemed like it wouldn't fallback or something. Instead of getting perfect streaming quality, I would not be able to load the youtube video at all
AFAIK ISPs have special hardware from/for these big streaming services to efficiently handle the traffic. Every-so-often that peering/mirroring equipment cannot support the demand and they have to upgrade the capacity.
They probably do it, but Google is at fault, too. Their new DASH thing has made the experience so much poorer. They never should've done it to begin with, and I don't think whatever savings in bandwidth they might have from it are worth the compromise.

Youtube was perfect before, because it allowed you to cache videos, and go back on the videos whenever you wanted, with no need for reloading or whatever. DASH made it a much worse experience, at it only caches like a few seconds at most in front of the streaming, and it won't even allow you to go back in the video without reloading it again.

All of that is required by their advertising business model. The more YouTube emulates the pre-VCR TV world, the more they can charge for time and visibility.
> because it allowed you to cache videos,

Youtube has to pay for that bandwidth usage even if you dont watch the cached video later.

> go back in the video whenever you wanted without reloading

That's even more bandwidth that they have to pay for unnecessarily.

I wouldn't know. YouTube is actually not so bad now if you block Flash, save the videos via the HTML5 interface, and watch them at your leisure.
I wonder if proxying just those sites through a $5/month Digital Ocean server would work...
Worked for me when I had streaming problems with AT&T, but AWS micro was the cheapest at the time.
Same here! Youtube is faster for me from China through a VPN connection than in the US via Time Warner.
I'm really surprised nobody's mentioned peering/transit in this thread. Ars has a good set of write-ups on this topic, the latest being http://arstechnica.com/information-technology/2013/07/why-yo...

Whether it's the fault of (1) the residential ISPs for letting their ports run so hot, or (2) the transit providers for trying to stiff them on pricing for additional capacity, or (3) the content provider for choosing a cheap upstream who is known to engage in this activity is up for debate!

As someone else noted on another thread, it's remarkable how often the name 'Cogent' comes up in these kinds of issues...

Good step but in the absence of true competition between ISPs this does little for the end user. I can discover that my connection is bad, and then what?
If you have the largest online advertising and marketing company in the world telling everyone that an ISP sucks... well, then guess what the ISP will do?

Either nothing, or lose customers.

In the absense of meaningful competition, they'll likely just laugh. Still, it's good to have someone call them out.
Which suggests that it would be in Google's interest to start fostering some competition.

Hrm....

They've been playing with that in the fiber-to-home space. But running cable is expensive.

I'm familiar with some wireless Internet providers in a few markets. The cool thing is that they can poach high-value, high-concentration subscribers first. Since the model is based on transmitters, typically rooftop installations, and the like, if they can hop onto a building, run Cat-6 throughout, and plug it in, they've got the whole building, plus can run local connections throughout the neighborhood, and beam line-of-site to other tall points. It's a pretty cool gig.

Thinking maybe you set that up in a few major markets as a competition play.

At least you will know whos fault it is and direct your hate when the posibility arises, (new videosite or new isp).
Then you go out and vote. The absence of true competition is because the average Joe isn't quite clued into the underlying problem of why things are slow. I suspect this is Google's way of raising awareness.
Interestingly, this also puts ISPs at a severe disadvantage for ratings if they decide to do net neutrality violating throttling on youtube or netflix. I hope every bandwidth-heavy service starts publishing reports like this
Or, even better, a neutral 3rd party that reports throttling for many services/domains.
It's not that simple. Comcast/Time Warner are monopolies in their area because we tax payers subsidized the laying of the lines. Verizon tried to expand FIOS to compete and took huge losses. My wife works for a fiber company, and seeing the numbers, it's basically impossible to make the investment worth it unsubsidized at at a price competitive with Comcast.
Agreed that fixing the root problem isn't simple. But citizens being pissed off en masse about their internet connectivity is a necessary precondition to any real change, and Google certainly seems to be trying to fan those flames.
Well, if I were in the ISP business and looking to expand, areas with poorly rated ISPs would be interesting.
Frontier mysteriously got better after the FCC publicly shamed them, although it could have been a coincidence.
Regarding the graphics shown, so it's a series of youtubes...
Quite often videos are buffering on my 80 Mbit connection at home - however when I switch to tethering with my phones connection, there is no problem. I always had the impression that providers try to cache or throttle Youtube video downloads. So I really appreciate that Google is doing something about it.

On a sidenote, the biggest german provider was trying to get money from services like Youtube for delivering their content. This rating turns it upside down :-)

Well, I'm on a resold Telekom VDSL connection, and so far all I see is "Results from your location are not yet available."

I've had major problems with YouTube videos in the past (despite having a very reliable 50Mbit down), but recently it's been fine.

This part was particularly interesting "We often deploy servers within your ISP's network, vastly reducing the distance the video has to travel and minimizing the chance for congestion."
Google Cache Servers. It also saves the ISP money by keeping popular traffic within it's own network/reducing carrier interconnect bandwidth.
How will they technically do it?
It's likely they'll just start doing analytics on the streaming speeds as we go about streaming youtube videos. They'll extract the ISP from our IP.

That's how I'm guessing Netflix does it: http://ispspeedindex.netflix.com/usa

Google Fibre will soon top the list.

Well played Google .. very well played.

I'm unfamiliar with ISP scale networks - where is the actual bottleneck for them? One of the large ISPs in the UK is Virgin, and they have their own fibre network. They also throttle Youtube.

Is there only so much data that can go down a fibre line? Is it the routers (I'm not sure if calling them routers is correct?) that lie between the main backbone and the network connecting a street/apartment? Is it the hardware in a handful of large data-centers?

Usually it's the peering between google/youtube and the ISP that is the bottleneck, internal ISP network and google CDN are usually fine there's just an unwillingness to upgrade the connection between them (usually because parties can't agree on terms, some ISP would like to be paid when peering).
Traffic inside your network come free of cost of course except for your own bandwidth constraints, but once you need to communicate with other networks that's where you start to pay.

Google offers free interconnections, but sometimes you can't just connect directly to Google and have to go through backbones that are owned by private companies.

So ISPs need to create as many interconnections as possible with other networks to decrease their costs, invest in their own infrastructure and pay for everything their network can't reach directly.

In other words, Google thinks a PR campaign and angry customers is cheaper for the same bandwidth to edge networks than just paying the owners of those networks.
Under the net neutrality paradigm, the entities "paying" for the last-network bandwidth are the ISP customers, not the content providers. The content they choose to receive has traditionally been expected to be treated "neutrally" without requiring payment by the provider. This argument has been had, and in virtually all cases our community has agreed (contra the positions of many regulators it seems) that it's a good thing.

Have you flipped on net neutrality or are you just confused because Google is the party harmed in this case?

Yes, because the customers of the networks have already paid for the bandwidth!

I buy internet access and that includes access to youtube!

You might, for just a brief moment, consider a less cynical, equally plausible explanation.
Brighthouse (TWC subsidiary) throttles quite a few services these days, Youtube being the biggest one. The throttling is so intense 24 hours a day that Google's streaming algorithm (download a little, play a little, repeat) can't continue downloading after the first initial burst. I can easily break it by VPNing into Work or School.

They very recently started throttling Netflix. I have a 90mbps down connection, and I can hit those speeds even at peak times.

ISPs are trying to strangle companies like Netflix into paying them for use of their network...you know...because Netflix doesn't already pay for an internet connection...they get free internet...because they're Netflix.

This is like the Mob walking into your laundromat and saying:

Mobster: "We own this whole block, and we want you to pay protection".

Laundromat Owner: "But we already pay rent, and protection!"

Mobster: "No, Clothing Protection. You're making a lot of money off dirty clothes, dirty clothes that get cleaned in our building. We want a cut. It would be a shame if someone left a sharpie in every washing machine wouldn't it? Then no one would come to your laundromat."

Netflix: "What if we don't let anyone in who has a sharpie?"

Google: "What if I just tell everyone the Mob ruined their clothes?"

Google is smart to Name and Shame the bad ISPs. I just think they should build this into the Youtube player, whenever the buffering spinner lags for more than 5 seconds have a a little message fade in that explains what the hold up is.
Interesting. Couldn't they just run a tracert and, through broad data comparisons, prove quite elegantly what the holdups in the chain are?
I'm not totally buying this - why is it when I have this extension[1] enabled, YouTube FLIES and works just as I remember it to work back in 2007?

[1] - https://github.com/YePpHa/YouTubeCenter

I haven't used this extension but my guess is that it's defeating the host redirection that ISPs do to point you at their extremely shitty and underprovisioned edge caches. Most ISPs run them to cache popular youtube content and save on out of network bandwidth, however as with most things they are terrible at it and extremely cheap. The solution I've been most happy with is to run all traffic over VPN to an endpoint not under Comcast - keeps everything nice and speedy, albeit with slightly more latency.
Is it me, or is this website subtly paying homage to Sen. Ted Stevens' famous "Series of Tubes" anti-net-neutrality speech.
Maybe they used that imagery because a series of tubes is an apt metaphor for internet bandwidth.
People talked about Internet "pipes" for many years, but somehow Stevens was a moron for saying "tubes".
He wasn't a moron for saying "tubes". We was a moron for the extended, stammering, rambling and incomprehensible context he used that word in.
The rest all seems reasonable to me, but you think he's a moron for stammering? I have a noticeable problem with stammering myself, and so do a few of the cleverest people I know. Stammering is an involuntary speech disorder, not a sign of low intelligence. That's simply not accurate.
Stevens' whole quote:

"I just the other day got, an internet was sent by my staff at 10 o'clock in the morning on Friday and I just got it yesterday. Why? Because it got tangled up with all these things going on the internet commercially...

They want to deliver vast amounts of information over the internet. And again, the internet is not something you just dump something on. It's not a truck.

It's a series of tubes.

And if you don't understand those tubes can be filled and if they are filled, when you put your message in, it gets in line and its going to be delayed by anyone that puts into that tube enormous amounts of material, enormous amounts of material."

This doesn't strike me as someone who knows what they're talking about.

> I just the other day got, an internet was sent by my staff at 10 o'clock in the morning on Friday and I just got it yesterday.

See, this is the funny part, not the "tubes". His staff sent him an internet? Yeah, I can see why that might take a while to deliver.

I'm pretty sure he meant "email" and misspoke. The overall point that the people regulating things often don't know as much about them as a specialist would, but it seems rude to laugh at a small mistake like that.
The point is that he is trying to convince people to regulate the internet. His rhetoric includes nonsense analogies that he doesn't even get straight. It's ridiculous.

Far better to simply say "I've consulted with experts so-and-so, and their conclusion was [reads a quote from expert]"

The underappreciated brilliance of this quote is that it's not clear what regulated industry he doesn't understand from his quote. Is it the Internet? Transportation? Waste removal? Plumbing?
Also worth pointing out that if his staff (house.gov) sent him an email (house.gov) and they all presumably use the same mail server (they do, House IT is all in house), then the tubes he's referring to were literal wires going from the basement of the capitol building into his office. I suspect it'd be pretty hard for the "commercial" stuff to get caught up in that.

Sounds like House IT was bad at administering a mailserver (they are -- when I was at the Sunlight Foundation, we linked to them and they called us accusing us of a DDoS), and decided to blame it on "the commercial internet."

It's a not-terrible but very basic metaphor applied to an area of engineering which about 99% of his audience did not need explained to them with a very basic metaphor. People assumed that it was he who needed the metaphor, and... hilarity ensued.
Google, would you please stop making these pretty advertisements and actually fix the problems? Thanks, Internet.
Wow, angry googlers strike again ;)
or maybe you just have no clue about what "the problems" actually are, and who needs to fix them.
+1 that was useful! Took 20 seconds to see how my local ISP does relative to other providers.
Great, but I can't see my country (Brazil) in there, even though Netflix operates here.
I can't get it to work in Washington, DC. I smell a conspiracy...
I get great streaming from netflix, and I find youtube frequently has trouble. Maybe my ISP (Verizon FIOS) is throttling youtube and not netflix, but that would be strange.
I'm a netadm for my network (a /16) and "Results for [my] location are not currently available". Any idea how I can alleviate this? I'm curious how we're doing.
Who can one test throttling by an ISP because I think Comcast is throttling youtube for the last 3 months. Its been very slow for me but my other downloads are 7MBs.
Google, please don't break my Back button. I actually try to use it once in a while…
There is nothing to go back on...
Interestingly, one of the early Nest investors is the CEO of an ISP who is in a battle with youtube over network traffic.
I was really hoping this was going to address the seeking problem. It seems like whenever I want to seek, even backwards to already buffered content, there's a 50/50 chance the video player will stop working entirely and force me to do a refresh and lose my entire video buffer.
The past few times I've had this happen, refreshing the page just dropped me right where I'd left off before trying to seek backwards.
I didn't this happened elsewhere. Here in France it's a huge problem, providers even talk about youtube in their ads to get people's attention. Free is the most famous case in France because they constantly refuse to do their jobs thinking that it's google's job to do (I don't know who's right but I left them because of their problem with youtube). Few months ago (or was it a year?) they upgraded their modem to block Google advertisement by default. A pretty bold move.