Ask HN: How would you bootstrap a telco?
The telcos here are terrible. I'd love to hear how HN users would think of bootstrapping a telco?
We've had a few startups try and disrupt things in Canada (e.g., Wind Mobile). They were funded by large global investment groups and eventually simply got acquired by the incumbent telcos.
The other approach I've seen is piggybacking on existing telco infrastructure. This is required by law since the Canadian government subsidizes telco infrastructure... It's not uncommon for the telcos to renege on their contractual/legal obligations. Plus, this doesn't create any redundant networks.
So: how else can one launch a telco? What creative solutions have folks seen?
I'm thinking: VOIP-only service that has direct access to the Internet backbone in Toronto, Vancouver, etc. so you can at least have completely separate service in major cities... Or crowdfunding 5G towers (though I imagine spectrum licensing is an issue here).
Would love to hear what other HN readers think.
111 comments
[ 3.9 ms ] story [ 191 ms ] threadAlso, see about using StarLink.
Also, do you have ComCast in addition to the phone company? It's always good to have a backup route to the world.
While a reasonable comment in the US, in most cases the phone company and the cable company in Canada are usually one and the same or have a common parent (and if the Rogers-Shaw merger is closed would make this true to nearly 90% of Canadians).
As the owner and operator of a small non-dominant carrier, what would really help is to have access to fibre local loops and transport at reasonable costs that reflect their 30+ year operational life. Existing fibre owned by the incumbents doesn't need to be duplicated. Unfortunately that's not the path the CRTC has taken to encourage competition. I really need to write a new Part 1 application on this.
Starting to build a carrier is a really steep learning curve. You don't know what the rules are, and you don't know how and when to fight back against the incumbents. Hell, the CRTC is a huge problem in and of itself as it took 2 years to sort out a fairly simple filing by Videotron where they had permits improperly denied to access poles (see https://crtc.gc.ca/eng/archive/2022/2022-160.htm for the decision on that filing -- my comments were referenced several times in that decision).
If someone really wants to start a small telco here in Canada, my number 1 recommendation is: do your own engineering. Engineering consulting companies will dig you a very expensive hole for projects that go sideways -- avoid that. The second most important tip is to fight back and go to regulators the moment permits do not get issued in a timely manner. Learn from some of my pain to avoid your own expensive lessons learned.
I would say that redundant networks are precisely whats required.
Selling a government on the idea is the hard part. Actually maintaining communications links, wires or fiber etc, is apparently so trivial a challenge as to be a minority of most phone companies actual business. They spend much more effort on billing and marketing.
I keep thinking the only way to break this anywhere in the world would be something like WiFi as a Mobile Network. Although this idea has been tried but never succeeded, practically because WiFi wasn't designed for such use in the first place. But recent improvement in WiFi 6 and WiFI 7 ( 802.11ax or 802.11be ) could solve this.
Right now I see usage across my network at around 4-5 Mbps per subscriber at peak. It takes a few hundred users at those rates to overload a Starlink terminal with a gigabit port. Starlink is not a viable backhaul technology for anything but the smallest of networks.
1. WiFi is largely unmanaged so all the clients basically act like they're on an Ethernet hub. If they detect a signal they wait a random period before trying to send again.
2. Besides interference from nodes on the same network, you're getting interference from other networks on the same channel. Your WAP and WiFi card ignore those networks logically but they still need to yell at each other over the noise. This makes for dropped frames that need to be retransmitted.
3. WiFi is using unlicensed bands and has very low radiated power limits. The 2.4GHz and 5GHz bands in particular don't propagate well. This keeps a neighborhood of networks from making the spectrum totally unusable. Even still areas dense with WiFi networks make for really shitty interference.
Cellular on the other hand is aggressively managed. For one handsets and towers transmit on different channels. Your upstream won't affect my downstream. Besides every channel being multiplexed a tower can tell a handset to change channels or even entire bands in order to managed congestion.
Becoming the third player is an uphill battle in all regards. Fibre customers are very sticky, and getting people to change is nearly impossible. Bell is particularly awful as a competitor as they steeply discount the first 1-2 years: want gigabit fibre with all the TV channels and phone service for $60/month along with a $200 prepaid Visa gift card? Sure! Then month 13 comes along and the bill becomes $300 per month. As a competitor, there is absolutely no way I can offer a similar package or retain a customer when they take that package. Want that deal as an existing customer? No way!
On top of all that, Canada is a relatively small market that is very spread out geographically. If you want to provide coverage similar to existing carriers, you have to build a lot of infrastructure. Nobody can afford to do that. Plus there are buckets of cash being rained down on other market players under a number of programs, so your competitor may be running on assets that cost 50-100% below your cost to build a similar infrastructure.
The easiest thing for you to do is to get 2 internet connections from different carriers that do not share networks. In most markets the telco and cableco run different networks. My Telus and Bell services continued working fine yesterday. Heck, my 3 different Rogers wavelength services we not impacted at all (although L1 services have other failure modes that take longer to recover from since there is no automatic rerouting as with L2 networks).
I'm pretty sure you really don't want to start down this path...
This is not as easy as it may seem, as it's common for the carriers to lease from each other in the backbones. Telus and Bell for cellular services share a radio network, each cell site connects to both the Bell and Telus core networks. The big Bell backbone outage in the Toronto area, 6 or 7 years ago, had impacts on all of Bell, Telus, and Rogers networks. As for cellular services we had glitches nationwide that had to be fixed during that backbone outage.
I had an incident where two different trans-Atlantic links from two different providers, shared a 20 mile stretch of conduit.
So, of course, that's exactly where the fiber cut happened.
The outage I cited, only occurred in the Bell Wireline Core network, but caused service impacts in Bell, Telus, Rogers and other networks for Cellular, Internet, etc services.
Third party carriers using fibre local loops from another carrier would not have gone down like TPIA users did on Rogers' network yesterday. I strongly believe that access to fibre local loops is needed in Canada. Leased copper local loops enabled plenty of competition between small local ISPs in the 1990s and early 2000s while they could still deliver speeds that were sufficient for internet users of the time. Having a tariff for local fibre circuits similar to the old LDDS tariff would bring back that vibrant competition we once had. Bonus: it's a lot harder for incumbents to put together bogus cost studies as compared to the failed disaggregated regime the CRTC has wasted nearly a decade on now.
Sort of, our wave circuits will automatically reroute but we had to shell out the $$ for protected waves and then explicitly mention they had to be configured to be auto-revertive as that was not the default behavior for that telecom provider. Then L2 and L3 don't notice when it shifts between the working and protect paths, it happens fast enough the only visible change then is the latency difference between the two.
It's as good an example as any of the lack of real liberty that entrepreneurs have in certain industries.
See also: banking and payments.
For exchanges, Ontario has the Torix exchange in Toronto for peering, and there's QIX in Montreal. Ottawa has Ottix, but it does not have many peers. There are others out west (as I'm in Ontario and don't pay attention to them as much).
Most consumer traffic is going to be to CDNs (Limelight, Akamai, Cloudflare), AWS, Apple, Google/Youtube, Microsoft, Netflix and the like if you're not big enough to have a private interconnect to them. It turns out the big internet companies like IX's too and don't want backbone providers being gatekeeper.
That's not the whole truth.
At least here in Germany, there are publicly-ish peering points. You pay a small amount for colocating a single router in the data center, and each peering partner only accepts IP traffic for the IPs they feel responsible for (typically their own AS). Companies such as Google participate in those, so you can offload all of your Google and Youtube traffic at such a peering point. A small ISP can afford to participate in such a peering point, you only pay a fixed, monthly fee for colocation.
However, you likely want to also have a carrier to whom you can hand off all the other traffic that you don't have direct peers for.
(Source: I work for a company that hosts some of the POPs for https://www.peeringdb.com/ix/135)
At any rate a new ISP wouldn't need to do paid peering they would simply buy transit from a Tier 1 or Tier 2 provider.
[1] https://drpeering.net/FAQ/Who-are-the-Tier-1-ISPs.php
[2] https://www.thousandeyes.com/learning/techtorials/isp-tiers
[3] https://drpeering.net/FAQ/What-is-a-Tier-1-ISP.php
Maybe there’s a fintech type disruption on the horizon.
This would be a fun project to hack on with 5-10 skilled folks with complementary skills to cover the industry, legal, financial, operational and the hardware and software angles to see what might be possible today. There have to be folks with a lot of experience in Canada that might be worth seeking out before beginning that feel exactly like you do and I would say I do too.
How much cash investment would you start with?
I’d be curious to how easy it can be for a non technical consumer today to access an mvno service that is turnkey and sells 3-4gb a month tablet type data plan (currently $10-20/mo) that can handle calls and sms via a carrier grade voip for $20-30/mo might be an interesting. It’s not new, but it’s not always easy to setup unless you’re technical. Calls are secondary to data (WhatsApp calls) these days for many. Keeping it independent of one carrier doesn’t preclude the fact that existing carriers have likely dealt with this type of approaches before. But it’s ok to start and remain small and let growth be organic snarfed across multiple providers instead of one only.
There may be something at the convergence or combination of some of the existing approaches or new tech that might be ready for the mainstream..
- Building momentum by starting as a MVNO in a segment of the market where you can secure and build demand and grow services. There are some interesting data only services out there.
- Perhaps a LoRa play. Great range. There seems to be some interesting gear that will pair Lora via Bluetooth to any phone. Software might be needed. Maybe there could be Lora beacons to assist with coverage.
- What’s possible with other open spectrums? Are there other unused spectrums that could be used in interesting ways, for exa
- Begin with cheaper 3G and now 4g networks only like new providers.
- Wifi calling overlap / voip coverage.
- Could there be a way to setup a peer routing network between devices and to their internet connections?
- Crowdsource/coop/non-profit focused on revenue and sustainability but not paying profit to shareholders. Non profit corporations do not mean mean people can’t be paid extremely well for their work. Maybe there is a meaningful social entrepreneurship angle here. Might be some funding available too to provide access to those who are priced out of the market.
- Maybe there is a way to use the enhanced networking capabilities of Linux only phones, or alternative setups.
Not even worth trying at all. LoRa link speeds will barely satisfy even the most frugal internet users, especially in a world where modern webpages can run into the megabytes.
[1]: https://startyourownisp.com/
In my experience VERY few people in the optical networking space understand how to engineer a municipal fiber optic network - they invest thousands of dollars per customer when it can be done with best practices for well under $1k per customer (let me just say that 802.1w RSTP is your friend). You need to combine all the different ISO networking layers into a SINGLE business model (ie. physical plant and Ethernet/VLAN circuits all by the SAME engineer not by different departments, otherwise things get unnecessarily over-engineered).
Even more important than the technical engineering is the financial engineering. Once you understand you will never produce more capital from selling customers than it will cost to provision those customers, you need to consider more advanced financial engineering models - the one that worked great for the cellular industry and several optical networks is commercial paper. Find a lawyer that REALLY understands commercial paper - then every contract signed by a customer actually IS cash and does not need to be converted to cash. It's one of the only ways I know of releasing capital invested into physical plant.
Good luck!
(Though I am curious why RSTP is so deserving of a call out - I mean, yes, but there's a lot of basics here that someone starting out would need to know, including things like avoiding fate sharing and basic planning, equipment realities, etc. if laying fiber.)
I consider RSTP a core foundational technology for FTTH, not some condiment sprinkled on the network afterwards. Plus it is the Ethernet technology I find most typically missing from FTTH network design.
Let me explain by way of an example: imagine a typical small town, population less than 50,000, with 2.60 people on average per household => less than 19,231 households. Traditional FTTH networks were modeled much like a POTS network in a star pattern with a dedicated optical fiber to each household from a central office. That then evolved (or should I say got bastardized) to a HFC (hybrid fiber coax) model by the MSO's as they already had a coax local loop going to each house. Clearly any kind of centralized distributed FTTH is doomed to financial ruin.
As an alternative, image this: same community. You get the local school district as your first customer (or other geographically dispersed participant), where you deploy a ring of fiber circumnavigating the community, interconnecting every school. As above you over build their fiber, keeping some for yourself. Of course, you will have a peering point (or two) to interconnect 10 gigabit Ethernet with a tier one provider. You now have a ring of optical fibers less than a few miles from every home in town (typically less than a mile). Built on top of RSTP, the ring of fiber gives every node on the network route-diversified redundancy and protection. But not every residential customer is going to sign up with you, so the last thing you want to do is build out to every home. Instead you build out from your ring to those households that do sign up, where every home has a route-diversified optical connection extending the RSTP ring to the customer premise. This topology negates the need for a massive star network emanating from a central office. Literally with a handful of fibers you can provide for an entire community at a standard of engineering and quality of service the MSOs cannot even fathom. Massive reduction in physical plant requirement, massive reduction in capital per customer.
What does a fiber loop (bus?) let you do hardware-wise, that a star topology wouldn't? Either way, you're running a piece of fiber to the home from somewhere, and connecting it to your network somehow; and I would assume that it's that connection that's inherently expensive, because of the equipment involved. In star topologies, that's a switch, or a hierarchy of switches. For a shared medium, that's a hub—or, I guess, some kind of direct splice, if that's somehow cheaper in TCO.
Is the goal of using a loop, to avoid the cost of having neighbourhood-distribution-box optical network switches, instead directly splicing customers onto the ring, making the customer's fiber "part of" the shared medium of the loop, and then allowing customer equipment to directly push packets / transmission cells onto the loop?
If so, why does it sound like you're implying that in a traditional FTTH star topology, there wouldn't be per-neighbourhood optical network switches, and instead only one wide-branching layer where all customers' lines terminate directly at one central location? That seems kind of ridiculous. Are optical switches really that expensive, that those are the only two options? Especially when neighbourhood-distribution-box optical switches would be relatively-low fan-in + relatively-low bandwidth, and therefore could be built to use simple low-frequency TDM circuit-switching protocols like coax's ATM?
Also, is client-accessible fiber head-end equipment really trustworthy to do TDM (FDM?) onto a shared fiber medium, without ruining it for everyone else? We don't trust client equipment to do that on enterprise Ethernet networks (if we can at-all help it); we use switches instead of hubs, and we even put VLANs in place to further isolate flows in the merged path for QoS purposes. Is the difference just a matter of a fiber provider only certifying specific (presumably robust) equipment; and of that equipment not being coupled to power delivery issues of an arbitrary host device? Is that enough to get you peace-of-mind that your shared-medium network won't be being blasted with noise from new failing head-end equipment every few days?
It’s the building out of fiber to each premise that is the expensive part, not the topology or the equipment. Those are rounding errors. What is expensive is undergrounding the fiber or hanging it on poles.
That's it in one sentence. So you want a topology and engineering model that will allow you to reduce your total network cost.
Each site has two links, left and right, that have engineered different paths.
If your link is to home base, chances are your second link to home base is going to take the same path.
I suggest you look into routed networks and redundant GPON topologies.
IMO one of the core components to think about when doing a buildout is your total solution. I've worked with Infinera in the past for both long haul and PON solutions and the important thing is getting the most return on investment out of your deployed glass. DWDM / band splitting is where you really need to engineer up front.
Once the physical plant is design engineered, your Ethernet and Internet engineering is virtually unregulated. You will want specialists to set up your BGP constellation and other things that will only be done once, but if you have a good working knowledge of routing and Q-tagged VLANs and other core Ethernet technologies, you will be amazed how much you can do yourself.
They were busily installing fiber to hogging operations back around 2000.
As a byproduct, small town residents were paying something like $15 a month for cable and phone lines with every possible feature.
It totally blew my mind.
You sell B2B (you would need to solve for multi-tenancy), and help facilitate a lot of consumer choice and competition by decoupling the pipe from the services...
Canada doesn't have a lack of competition. Canada has a lack of people willing to consider competition.
https://www.youtube.com/watch?v=HlRPLBLkZK0
Everyone I know whines about high telecom and internet prices. I know very few people willing to consider anything other than a lower price at Bell, Rogers, or Telus.
Canadians are very risk averse people.
It would probably be worth talking to one of these to find out what the real experience is like. In my experience, people will gladly talk about what they've learned.
They have nascent 5G support, and it’s all “owned by the people”.
Not a big fan of crypto shenanigans, but Helium comes as close as possible to crypto that kinda works and solves a problem.
Besides, there are actually successful mesh nets already out there in the wild, like Freifunk in Berlin.
https://berlin.freifunk.net/index_en/
My point is that up until this last event I was cognizant of the fact that the prices are too high, but was ok to pay for the quality I got, would have laughed in the face to anyone saying LTE coverage or speed are bad.
As a side note about the outage - I think a lot of people underestimate incompetence in large companies. I often marvel at how for example large Canadian banks are making record profits with many disfuctioning teams in critical positions - a situation that got much worse with the wave of the great resignation. If I were to bet, I’d say some incompetent team broke the infra, so there’s no need to look for conspiracy theories, people are just not competent in their positions and make mistakes when something has to be changed and they have no more excuses to avoid that change.
Not surprised to see Rogers blow up. Looks good on them.
It’s 2022 and I can’t get a fibre to the home connection in Ottawa less than 2km from Parliament. I’d get on a ladder myself to string up the fibre line myself if they’d let me.
Every time I want to self flagellate I take a look at the Swiss ISP init7[1]. 25/25Gbps to your door for 777 CHF/year. Infuriating that we can’t have that for any price here anywhere.
1: https://www.init7.net/en/internet/fiber7/
Join with other providers like Teksavvy to help the CRTC to break up the monopoly on the infrastructure.
1. You didn't specify wireless or wired. It matters. With wired, your biggest problem is going to be getting permission to deploy towers and licensing part of the spectrum;
2. Wired avoids this but adds the problem (and the biggest cost) of the last mile. Where do you run those cables?
a) Poles: you need permission for this and there can be a bunch of hyperlocal laws that will get in your way. For example, you may have to wait for existing telcos to move their wires up or down to make room and they'll take their sweet time.
b) In trenches: you'll need a different kind of permission for this. Local conditions may make this more of less difficult (eg there can be alot of large rocks in the soil).
3. If you run a cable from a house and on a pole or in a trench where does it go? You need some kinf of substation for this depending on the max transmission length of the cables you employ. You need planning permission for that and teh big cost of the real estate required. If it's a hole in the ground, that's less of a problem. If it's a free standing structure containing switching equipment, expect residents to fight it on NIMBY grounds;
4. Depending on where you are you may face local or state laws that restrict your ability to do any of this. Some US states have passed laws against municipal broadband (at the behest of telco lobby). Telcos may have exclusive franchises with certain areas. This has been dismantled somewhat but you may find unofficial opposition anyway;
5. You're going to need installer for all of this and people to handle all the permitting;
6. Once it's built you need people to maintain it. People will come through and cut your wires or they'll be accidents like cheery pickers running through your cables or poles going down in a storm requiring cable repair.
7. Existing telcos are exceptionally good at playing this game and are typically in bed with local politicians who will make your life difficult.
8. Will you get enough customers? Just getting permission to build something will probably lead existing telcos to cut prices and lock in consumers with 12 or 24 month contracts to starve you.
9. Do you have a TV service? If not, you're going to lose a certain number of potential customers who still want this. If you do, the price you pay scales with your size. Verizon, Comcat or AT&T pay a lot less to content providers for their TV packages than you will. Telcos like to bundle TV and Internet so you may find your customers only pay $20/month extra to add Internet to theri TV package.
10. Who do you get upstream Internet connections from? It may well be one of your local competitors. Guess how that goes.
This is a capital intensive business that doesn't reward overbuilds. The only way this can really work is with heavy cooperation from your local municpality. As mentioned there may be legal barries from them doing so (as noted, telcos don't want a repeat of Chattanooga).
This'll work best in low population centers that are underserved or not served at all by existing players. Low density housing means getting trenching permission is going to be less of a problem. Potential customers will also be much more motivated if there other options are HughestNET or DSL.
Starlink probably means even these customers aren't going to be as motivated as they would've been 5-10 years ago.
In more rural areas you may even have the option of microwave relays that'll bypass a spectrum issues and it'll be easier to permission to be a directional wireless tower on someone's boundary.
https://www.nycmesh.net/
https://www.nycmesh.net/faq
If the objective is to compete on a national scale, we're talking capital investments of something like $10 billion CDN per year. As I recall when I worked in the industry Bell was spending 5-6 billion per year on capital investments, and if you're starting now you're playing catch up, so probably need to double that for a decade or two to catch up.
And that's just capital investments, equipment, software, and cables in the ground. Can this be run way more efficiently, and with less overhead in billing systems and pet projects and the like, sure. But use these numbers just for the realm we need to be in. Let alone the people to build, maintain, upgrade, monitor, respond to outages, etc. Just to try and put that into perspective, the entire federal government spent about $440 billion last year. So we're talking a good 3-4% of the federal budget, for what one of the big 3 is spending. And keeping in mind, they have a lot of power to compete on price and features wherever you get started.
So I'm personally believe that difficult to duplicate infrastructure should be government run. It's simply impractical to go and say we'll have 5 or 10 or 15 competitors, and startups, and bankruptcies when the cables in the ground are so capital intensive. We don't run 15 power lines to each house so everyone can change on a whim. Why are we pretending we can run 15 fiber cables and do the same thing. We don't really want one new big competitor, we want a vibrant market of new approaches and failures to occur.
So set that policy tirade aside, and let's scope down our expectations from nationwide.
I get internet from a small provider, who mainly serves multi-dwelling buildings in the Toronto area. What they've been able to carve out, is a niche where it's cost effective to compete and start from. They're basically leveraging some, I think it was Industry Canada rules changes, that basically say that the telcos aren't able to claim ownership of the wiring inside a building. The provider I use, then just hired some Bell technicians, bought the same DSL remotes Bell was using for the same services, purchased a leased MPLS or similar line back to their data center to the basement of the building, and started making deals with Condo Boards and building owners to offer services. So an internet connection, DSL remote, and some technicians to terminate wires was what they needed to get started. Of course it's more complicated than this, but that's what it sort of boiled down to.
The DSL remotes have a SIP client that can create a phone connection, so you can get home phone, and they built a cheapo IPTV service with some android boxes. Give away telephone, TV and internet for free in the common areas of the building for advertising.
911 services, they just contacted with whoever provides normal 911 services for voip providers.
When I switched to them, it was cheaper than I got with an employee discount in the industry. Last year, they put fiber in, so I've got unlimited gigabit fiber for half the price or less of the big companies. The Big companies of course, now offer unpublished promotions from time to time that are only available to my building. This of course doesn't cover cellular service, but we're back to that capital intensive replication of infrastructure.
I'm glad I'm out of this industry.