I worked with Vodafone years ago to do an integration with ticketing systems. It seems like no one actually worked for Vodafone it was all contractors or contractors of contractors of contractors.
Outsourcing peering to a 3rd party seems like their playbook.
Is there anything that Vodafone customers can do legally to punish Vodafone or not delivering on their broadband contracts?
If you're paying for a 1Gbps connection and Netflix is only able to stream to you at 0.93 Mbps because Vodafone or Inter.link are choking off the supply, surely that's breach of contract on Vodafone's part?
I'm sure Cory Doctorow has a word for what's happening here.
The big difference here is that you're almost never forced to use Deutsche Telekom for wired internet (there are many DSL resellers, and many of them actually provide routing themselves), but in some buildings, there is literally only Vodafone, with no choice of any alternative service provider on top.
Fucking around with peering is the specialty of German ISPs. Telekom (our biggest provider) is sometimes unusable for YouTube/Netflix/Cloudflare/Steam in big cities because of similar shenanigans.
This isn't unique to Vodafone. Google has also been slowly withdrawing from IXes globally in favor of PNIs and "VPPs" (verified peering providers). This only makes it harder for smaller networks to establish presence on the internet and feels pretty anti-competitive.
On the flip side, IXes are becoming harder and less desirable to participate in: port fees are going up, useful networks are withdrawing, low quality network participants are joining and widening blast radius. I'm not sure what the answer to this is, but this has not been a great year for the "open" internet.
> IXes are becoming harder and less desirable to participate in
Could this be due to the rise of services like Equinix Fabric and Inter.link? Google doesn't need to peer directly with most anymore because there is always a middleman somewhere who can handle it, and for many businesses the convenience of a point and click web gui outways whatever it costs?
Notably, Chris Sacca (now a famous VC), when working at Google in the early 2000s, got in trouble for mentioning that telecoms were preventing Internet from being Net Neutral, and instead of being fired, he got promoted by Larry Page and given a huge budget to work on it. I can't find a reference but I'm almost certain this is correct.
Bell Canada also has had a long-standing policy of refusing to peer with internet exchanges. They'll only truly peer with other direct backbone providers and a handful of one-off peer with other large networks (google, cloud flare, etc), but their historical position as Canada's base backbone (not so much anymore, but it was definitely a thing pre-2005) has meant their policy is most people should pay them to peer. I'm not sure if it's still the case, but IIRC for awhile they also refused to peer with any other domestic backbone providers.
The result has been some funny routes sometimes. I live in Toronto and have seen trace routes bounce over to Chicago to connect to stuff colocated here in Toronto.
It's frustrating as their fibre is my only real high speed option; also their lack of IPv6 on anything but their mobile network is annoying.
> also their lack of IPv6 on anything but their mobile network is annoying.
This gives me even less confidence after BCE took over ZiplyFiber, US PNW provider. There's a long running joke about IPv6 just one more lab test away from deployment.
It's annoying how they're the only big ISP offering fiber everywhere when they're also the ones that don't support ipv6 and have the shitty peering policy. I've heard you can use another isp though (teksavvy maybe?) that uses Bell's fibre and supports ipv6
The solution - as always - is regulation. ISPs typically already have very generous business models with widespread monopolies on customers, overwhelming barriers to entry for new companies, and a lack of rate controls allowing then to price arbitrarily - all of which supports immensely profitable businesses without the need for additional extraction of capital from other parties. Regulators and consumers alike should be screaming in rage at the idea that their ISPs are now multi-dipping for revenue, but we’ve done a piss-poor job of explaining how this works to the common man and thus can’t count on them to support the Open Internet as we’d like to see it.
That being said, the threat to the open internet is also more than just ISPs being gigantic assholes: it’s centralization in general. A majority of web traffic passes into or through one of three main cloud compute providers; Cloudflare has such an outsized impact that regional IP blocks can disrupt global traffic; and ISPs have been permitted to consolidate through mergers and acquisitions into expansive monopolies. The internet is fiercely centralized and largely closed already, which is why these ploys by shitty ISPs are likely to work absent Government intervention.
You want to protect the open internet? Regulate the shit out of its major players again. Force them to keep it open, especially when it hinders expanding profit margins.
Is doing business with Inter.link really structurally different from getting connectivity to an exchange like DE-CIX and doing business there? I know that in theory, you get settlement-free peering at exchanges, but only for those networks that participate.
And who funds Inter.link? Their publicly available balance sheet shows significant, growing debts to a linked company, but it doesn't mention its name.
I'm surprised to read an obviously AI written article ("This isn't about efficiency—it's about extraction") from a tech/news site. Does anyone else find this weird? It make me question the editors note about how much background research was actually done.
The disclaimer at the top of the article is really mind blowing:
> We may have failed in some areas to grasp the issue entirely. The reader is advised that not everything might be correct and you should follow the sources and conduct your own research to get an adequate understanding of the subject at hand.
It's very simple. I host my stuff on a network with an open peering policy. If you as an ISP somehow have peering issues with that, then that's a you problem. I will not pay a ransom to some shady middleman that you decide to use because your network admins are too lazy. I will (rightfully) blame you and tell your customers to switch ISPs if they have issues.
Play stupid games, win stupid prices. Just wait until Vodafone Germany customers get slow speeds and an automated warning banner on every other website they visit. "Too big to fail" until it isn't.
I recently moved to a Dutch municipality that runs its own non-profit ISP. They installed a symmetric 1 Gbps fiber connection with a static IP at my house for 40 euros per month.
The service is solid, there’s no upselling or throttling, and hosting things from home just works. I bring this up because when we talk about “open”, “fair” and “monopolies” the model of a local, non-profit ISP backed by the municipality could offer a real alternative. It doesn’t directly solve the peering issues, but it shifts the balance of power (and cost) somewhat.
i've wondered for a long time why this isn't a more common solution to these services that are almost inevitably monopolous. power, water, and internet kind of things.
Somewhat similar over here in Chattanooga (aka "Gig City"): our city's Electric Power Board has offered synchronous gig fiber for $75/month, a 300mbps "slower option" for $58/month — it's plenty fast — or 25gbps for $300/month... to all electric customers [0].
Of course our lobbied state congress critters passed a law to restrict this, so EPB can only offer internet to a limited geographic area (under the auspices of network monitoring of power delivery) — wouldn't want their Comcast-bros to have any competition! Certain apartment complexes are exempted, which prevents you from using EPB.
Wish more jurisdictions were even allowed to do this; wish politicians weren't such whores.
Interesting take, but generally large, incumbent eyeball networks have refrained from open peering at IX's for decades at this point. They maintained presences, but usually just to grab a few specific peers they wanted, not to peer broadly with everyone across the exchange, and the bulk of traffic from large providers into eyeball networks comes across PNIs or on-net CDN nodes, not IX.
If anything, this move to centralized PNIaaS platforms makes interconnecting with the eyeball networks even easier for smaller providers. The portals allow for straightforward visibility on what they want to charge for paid peering, and instant automated EVCs and turnup, shortcutting the long and windy process of negotiating terms and establishing individual XCs in DCs that you agree to peer in.
The Telekom story mentioned in this article is 100% as bad as they make it out to be, most of the users we support with issues reaching our services are with Telekom Germany. Or in an authoritarian nation that blocks access to western services.
Awesome. The only internet connection we get IN THE MIDDLE OF BERLIN is Vodafone Cable. Deutsche Telekom wants to build fiber here, but our landlord refuses to open the door to the cellar because he wants to kick everybody out and raise the rent for new tenants.
> our landlord refuses to open the door to the cellar
The simple solution would be to make this illegal, i.e. require landlords to allow at least two competing wired ISPs to connect each household.
No need to make them pay for it; I suspect it would be more than enough to end their very lucrative arrangement of somehow rewarding exclusivity. (I don't have any evidence that landlords are getting paid for it by Vodafone directly, but I highly doubt that there's any above board reason for the status quo.)
When can we start doing meshnets again? I was excited about it in '05 long before I could afford networking equipment. 20 years later and we've only regressed.
Right now. There's Freifunk in Germany. But we're all terrified of laws now. There was a time when hackers would dare to sticky-tape cables out their window and across the street. Not any more, everything's all nice and regulated and a strict order is harshly enforced. If I place a fiber anywhere outside of my apartment I expect to be deported.
60 comments
[ 2.8 ms ] story [ 78.2 ms ] threadOutsourcing peering to a 3rd party seems like their playbook.
If you're paying for a 1Gbps connection and Netflix is only able to stream to you at 0.93 Mbps because Vodafone or Inter.link are choking off the supply, surely that's breach of contract on Vodafone's part?
I'm sure Cory Doctorow has a word for what's happening here.
Sure, you lose Vodafone germany. Then you explain clearly why to every major media.
This coukd be stopped fairly quickly.
I disagree with this move, but it is not without precedent.
On the flip side, IXes are becoming harder and less desirable to participate in: port fees are going up, useful networks are withdrawing, low quality network participants are joining and widening blast radius. I'm not sure what the answer to this is, but this has not been a great year for the "open" internet.
Could this be due to the rise of services like Equinix Fabric and Inter.link? Google doesn't need to peer directly with most anymore because there is always a middleman somewhere who can handle it, and for many businesses the convenience of a point and click web gui outways whatever it costs?
The result has been some funny routes sometimes. I live in Toronto and have seen trace routes bounce over to Chicago to connect to stuff colocated here in Toronto.
It's frustrating as their fibre is my only real high speed option; also their lack of IPv6 on anything but their mobile network is annoying.
This gives me even less confidence after BCE took over ZiplyFiber, US PNW provider. There's a long running joke about IPv6 just one more lab test away from deployment.
See Telstra(Australia), the Korean Telcos, NTT(in Asia), Globacom (Nigeria) etc
South Korea pioneered fair share govt regulations in 2016 (which caused Twitch to exit the market in 2024 due the exorbitant "fair share" fees).
That being said, the threat to the open internet is also more than just ISPs being gigantic assholes: it’s centralization in general. A majority of web traffic passes into or through one of three main cloud compute providers; Cloudflare has such an outsized impact that regional IP blocks can disrupt global traffic; and ISPs have been permitted to consolidate through mergers and acquisitions into expansive monopolies. The internet is fiercely centralized and largely closed already, which is why these ploys by shitty ISPs are likely to work absent Government intervention.
You want to protect the open internet? Regulate the shit out of its major players again. Force them to keep it open, especially when it hinders expanding profit margins.
And who funds Inter.link? Their publicly available balance sheet shows significant, growing debts to a linked company, but it doesn't mention its name.
> We may have failed in some areas to grasp the issue entirely. The reader is advised that not everything might be correct and you should follow the sources and conduct your own research to get an adequate understanding of the subject at hand.
Play stupid games, win stupid prices. Just wait until Vodafone Germany customers get slow speeds and an automated warning banner on every other website they visit. "Too big to fail" until it isn't.
The service is solid, there’s no upselling or throttling, and hosting things from home just works. I bring this up because when we talk about “open”, “fair” and “monopolies” the model of a local, non-profit ISP backed by the municipality could offer a real alternative. It doesn’t directly solve the peering issues, but it shifts the balance of power (and cost) somewhat.
Of course our lobbied state congress critters passed a law to restrict this, so EPB can only offer internet to a limited geographic area (under the auspices of network monitoring of power delivery) — wouldn't want their Comcast-bros to have any competition! Certain apartment complexes are exempted, which prevents you from using EPB.
Wish more jurisdictions were even allowed to do this; wish politicians weren't such whores.
[0] https://en.wikipedia.org/wiki/EPB
If anything, this move to centralized PNIaaS platforms makes interconnecting with the eyeball networks even easier for smaller providers. The portals allow for straightforward visibility on what they want to charge for paid peering, and instant automated EVCs and turnup, shortcutting the long and windy process of negotiating terms and establishing individual XCs in DCs that you agree to peer in.
What a time to be alive.
The simple solution would be to make this illegal, i.e. require landlords to allow at least two competing wired ISPs to connect each household.
No need to make them pay for it; I suspect it would be more than enough to end their very lucrative arrangement of somehow rewarding exclusivity. (I don't have any evidence that landlords are getting paid for it by Vodafone directly, but I highly doubt that there's any above board reason for the status quo.)
That's giving free transit/route leaks