Ask HN: What are the next internet infra problems?
There are multiple companies that were born to solve specific internet's infrastructural problems (e.g. Equinix, Akamai). Looking at the way internet usage has evolved, what kind of infra challenges do we have to face (now or future)?
[books/papers suggestions are welcome!]
229 comments
[ 3.4 ms ] story [ 234 ms ] threadhttps://en.wikipedia.org/wiki/Play-by-mail_game
This is one of the no-go theorems of quantum information: <https://en.wikipedia.org/wiki/No-communication_theorem>
https://www.nature.com/articles/s41534-018-0090-2
This is like saying you can’t send information with electricity. You couldn’t at first (FYI), but then someone invented Morse code and the absence and non-absence of electricity made all the difference in the world.
But you can't send information through quantum entanglement itself without rewriting a significant chunk of physics as we know it. If you're going to do that, all bets are off and you should be consulting with wizards and not physicists.
Apparently people mail each other save games.
0: https://ssd.jpl.nasa.gov/horizons/
- The design seems to mandate source routing, where the entire path needs to be known by the sender. This is much less resilient than the current internet where each hop decides how to best get a packet to its route.
Akamai solved POPs (point of presence). Equinix solved DC. Both are matching towards table stakes in the context of internet infrastructural. (Not business models). We have lots of under-sea cables / international expansions on-going and planned. And it is now more of a cost efficiency problem, not an infrastructural problem.
We have a decent Ethernet roadmap [1], Terabit Ethernet, Petabit under-sea cable by 2030. If anything I see the only internet's infrastructural problems being closer to the consumer / client side of things where Fibre Cables are not being deployed. But I sense the pandemic has changed a lot of perspective on fast internet and Government are now willing to put more pressure into making FTTH as requirement.
If we look at Mobile, even carriers were a little too optimistic in Data usage projection. 5G proved to be sufficient enough in terms of Tower capacity with enough headroom for expansion without requiring Small / Nano Cells.
It might be different set of infrastructural problems, but more regulations on internet in a per country / jurisdictions basis, which would require Internet infrastructure to adapt to these scenario.
[1] https://ethernetalliance.org/technology/roadmap/
That blog post was released on April 1, but it isn't a joke (Cloudflare announces things that seem crazy but are real on April 1 instead of doing a joke)
Disclaimer: I'm an engineer at Cloudflare, but not on a team related to this.
https://developers.cloudflare.com/distributed-web/ipfs-gatew...
“It’s not possible to solve this problem, except by centralizing all the web through us. Aren’t we generous to not punish our customers when they get hit by this problem?”
This could be set to only be enabled if load is approaching a certain percent of capacity that the servers/CDN are able to handle.
Once reaching that threshold, P2P would kick in, and existing visitors could serve static content to newer visitors using something like the WebRTC + Service Worker + IndexedDB combo that www.arc.io uses for their P2P CDN.
Thoughts?
Cloudflare has absolutely no reason to invest in solving DDoS because the existence of them is one of your best sales leads. DDoSes are cancer and you run a cancer treatment center. Gotta make sure you can treat cancer well but you wouldn’t want people to avoid it in the first place.
Imagine if Cloudflare stubbornly stuck to providing DDoS services and never considered the idea that there might be a solution to DDoS at the protocol level. We'd die if someone else came up with the technical solution to DDoS. So, it would both be better for the Internet and better for us if we were involved in killing off DDoS at whatever level possible.
For example, on the network level we've pushed for BCP-38 over and over again to deal with spoofing. RFC 2267 is 24 years old (https://www.rfc-editor.org/info/rfc2267)! But, yeah, sure, Cloudflare that is half that age is keeping all those DDoS attacks happening because they love the smell of $$$. Give me a fucking break.
This reasoning does not prevent Google from slowly making itself irrelevant by changing the web to such an extent as to make its search algorithm impossible to get any useful results from.
> For example, on the network level we've pushed for BCP-38 over and over again to deal with spoofing.
That is a point in your favor, I will concede.
> Give me a fucking break.
Cloudflare is a huge company with more and more power over the entire internet, and it is constantly urging people to only use the internet through Cloudflare, in numerous ways. You do not get a “fucking break”.
We also solved the bcp38-like problem ipv6 had by using source specific routing throughout openwrt. A lot of other router makers are still not doing this right. Turris gets it right, I know.
It would be good to know what else cloudflare thinks would be a good set of DDOS protection features (route 666) home routers should have? Please add requests here: https://forum.openwrt.org/t/cerowrt-ii-would-anyone-care/110...
Even if you properly implement this system, network operators will expose themselves to firewall DDoS attacks by malicious actors that are trying to fill the firewall blacklists with garbage.
We've reached counter counter DDoS warfare. What do you do now?
Too many people are stuck with slow, expensive, and unreliable cable and/or DSL ISPs. Some have no choice whatsoever. Some others get to choose only from two equally awful options. We need legally available competition EVERYWHERE.
https://en.wikipedia.org/wiki/Rural_Utilities_Service
Contrast that to the reaction to Tesla's autopilot that will happily drive you into an embankment at fatal speeds.
In the scale of the US, that's a blatantly false statement.
It's < 50,000 people out of 330 million. Or less than 0.02% of the population. Most of that is because they're living in quite isolated locations far from any population.
The figure is so high the WorldBank lists the US at 100% access to electricity with all the other affluent nations. Brazil is at 99.8% for reference, Vietnam is 99.4%, India is 97.8%.
Areas, as in places you can live.
I wasn’t saying it was whole chunks of the country. Just that there are many places, not necessarily in the same place and not necessarily enough for anyone to care about. The latter kind of being the problem…
Here are the economics for a fiber optics deployment in 2021 for a rural town of ~800 premises, where about 60% got service. They charge customers about $100/month for symmetric 75/75mbps internet plus phone.
It costs about $40K per mile to run the fiber down the road on existing utility poles. It costs between $2K and $4K to run the drop from the utility pole to the home.
Those two numbers (miles of road, number of premises) get you in the ballpark of the cost of deployment in a town/county. More premises per mile obviously decreases the cost per premise - that's why utilities prefer dense areas (cities).
I can only imagine how fucked rural America is.
https://discuss.broadband.money/c/broadband-grant-events/ala...
Some background maybe first.
There is a massive Global Internet Distribution challenge which works around the cost/bit equation. They are:
1. Undersea cable networks - USD 0.5-1B to deploy over a multi-year project. 10s of millions to maintain with regular cable cuts. Typically now only deployed through consortiums of Internet and Telecom companies. Carry 99% of world’s international data.
2. Inter-City Distribution - National Fiber and Copper networks which connect tier 1-3 cities, towns and villages with a backbone to the nearest Internet Exchange OR telco data center (which in-turn would have a hard line back to Undersea landing stations).
3. Last Mile or within city/urban connectivity - last & middle mile within a city/town connecting homes, offices, towers and DCs.
IMHO the challenges still remain but get worse from top to bottom, costs and complexity often jumping in orders of magnitude from one to the other, with Last mile obv being the craziest.
Telcos nationally in most countries still own most of inter-city distribution and tier 2/3/4 POPs (point of presence), leasing out capacity from POPs to ISPs and Enterprises. The investment in laying these cables is EXTREMELY prohibitive and is the main cause for high Mbps rates, high latency and onerous terms when it comes to in-country network distribution (big e.g. is South Africa). Numbers range from orders of magnitude more expensive (e.g. $1.6B for Telstra Australia in Phase 1, $130-150B for US) than Undersea cables primarily due to Right of Way and operational costs of deployment.
People are now moving from Rural to Tier2-3 cities/towns and also there is reverse-migration from megacities like Manila to Tier2-3 cities/towns (as evidenced by rising cities like Cebu, Bali, Miami, Austin, Pune, etc where housing is more affordable and earning potential remotely is nearly the same). Bandwidth and latency demands are going up 100% Y-on-Y in Tier1-3 cities, especially in WFH COVID times. Starlink & others in LEO wil definitely help with most rural unconnected places (<1-2% of total bulk). Telcos will eventually build out Tier1 cities with fiber more robustly (since they have to deliver on 5G small cell and potentially 6G).
Mid-tier cities & towns where by far the larger total bulk is accumulation will need a LOT of attention and more latency optimized, cost/bit minimized backbones.
Finally, humanity's push to get into deep space in the next decade will require building out infra to support robotic and autonomous missions. Thinking of deep space objects as islands or continents is a helpful model and tightbeaming laser comms to them as "undersea cables but in space" could help address some bandwidth allocation problems in the early days (but local distribution will again have challenges)
There's a lot of room to optimize latency whether it's removing bufferbloat, L4S, or cISP.
It is all a steaming pile of garbage.
We're still typing single characters in text files as if we're on a terminal in 1960s.
We're in the middle of a SW quality crisis, because a lot of people have not the slightest idea what they are doing, but they are encouraged by their managers to ship immediately.
Comments, build configuration, platform specific implementations, alternate implementations (e.g. (re)implementations of an interface used by an application), tests, and etc. could all be part of the "program". In many ways we try to do this already by hacking together git repos with everything stored in there as text and then require very particular versions of programs to run everything in there anyways.
I think it's all sufficiently powerful already to build awesome things, but I do not doubt that in the next 10 years something will be built that realizes the ideas which have been kicked around since Smalltalk/Self and raises the bar of productivity.
Around 37 percent of the world's population (2.9 billion people) have never used the Internet (1 in 3 people), per the UN’s 2021 report on the topic.
https://www.itu.int/en/mediacentre/Pages/PR-2021-11-29-Facts...
Even homeless people in the west should be able to get money that way.
So an .eth domain for every person?
Digital notary. So a third person (digitally) signing a transaction or other document exchange.
It should be possible for you, the receiver of the email, to check if the email originated at company X.
Digital Notary: this came up in several data privacy discussions. You (A) are in contact with B, but you don't want to send B something like a scan of your passport (e.g. for age restricted services). So you disclose the passport scan to Notary and he sends B the message, the passport was disclosed to me and the person is >21.
You could check the DKIM signature of the email.
The point being that Company X does not have a copy of the sensitive information (and neither the liability of losing it) and the Digital Notary would (in theory) have better procedures for properly deleting or storing the data as needed.
[0]: https://www.personalausweisportal.de/Webs/PA/EN/home/home-no...
Ability to do SDR for wireless networks with smarphones. 5G is not a good solution.
Better security for routers, and generally better software security regulations, which are almost non existent right now. If cars have security regulations, software should, too.
Interesting viewpoint. Care to expand on your thoughts? To me, 5G seems like a stepping stone to UWB communication
My gut tells me it's fairly low per person served, and it'll only improve over time as more renewable electric sources come online.
https://www.nature.com/articles/d41586-018-06610-y
Compare that to https://www.epa.gov/ghgemissions/sources-greenhouse-gas-emis...
I don't know if it takes the lifecycle of hardware into account.
(See: that time that a bunch of Google traffic started getting routed through Russia. Or the time that YouTube became inaccessible to the entire world)
Yet BGP injection attacks (ASN or prefix theft) happen regularly. The reason is that not everybody follows the best practice here. It may well take a massively disruptive attack before this gets any better.
30 years ago, people would've said the same things about routers, so I think it's possible with the right ui/incentives
https://thehelm.com/
It's great that you don't have to start rolling your own cloud, but it would be a better offering for me if you could decide to tinker with it.
I want that, but for the masses of tech-illiterate Average Joes out there, it's tough to compete against the sheer convenience of "tap next to trust big tech with all your private data and sync it all in the cloud" that you get when you unbox your iPhone/Android. And for most people their phone is their primary computing device now so their lives are tied to those ecosystems and we've been conditioned for over a decade now to just give our private data to the phones' ecosystems without asking questions because everting is so convenient and ignorance is bliss.
Trying to get average consumers off the big-tech ecosystems at this point is like trying to unplug people from the matrix. It's nearly impossible, unless some new EU-style regulations break up these monopolies first so that third party alternatives can compete on feature parity.
Consumer NAS management should be way easier. Why can't I tap my phone to my NAS to pair and then go anywhere in the world with a network connection?
1) most consumers have no idea what a NAS is, and even when they do, consumers have been conditioned that all their data is automagically beamed to the Apple/Google cloud at any and all time there's an internet connection without any user involvement beyond entering their ID when they unbox their phone for the first time, so it's impossible for a third party device or service to compete with this level of out of the box integration and convenience
2) NAS devices aren't made by Apple and Google so NAS integration into these ecosystems is a second class experience at best, and Apple and Google will never make NAS devices as they're incentivized to get you to pay for their cloud storage subscriptions for your data. Plus, as a a cherry on top, this way they can silently data-mine you as well.
Basically the industry is moving, or we can argue that it has moved already, towards subscriptions, where you never really own your music/movies/data but have access to it as long as you pay your monthly/yearly fee, because this is so much more lucrative for big-tech than getting you to buy commodity HW like a NAS and physically owning your data.
I'm only half joking. I was just fantasizing about this the other day. I'd love this to become reality but I'm worried reality diverges from this idea further with each passing day. People produce more and more data that's useless outside of a closed platform. Nobody owns media to host.
A lot of non-techy people already have NAS, external hard drives, and things like that. I don't know how ISPs haven't done already this.
I fully appreciate owning your own data and hosting it somewhere, but no idea why we need to host anywhere but hyper-connected data centers.
I’d like to own my social graph, my profile, my permissions for who can contact me and read my data. But no reason for that to execute on my phone or home server. Have service providers do it, compete, scale and specialise. Let me host at home if I want to, but that doesn’t feel like the default we need.
But prove me wrong. I like learning!
Beyond that, if someone owns your data they can simply decide what you pay tomorrow and its burn it or pay.
The focus for the past 30 years has been on simplifying web content consumption. Everyone knows how to use a web browser. Why hasn't there been a similar push to make serving web content easier? There have been some attempts (Opera Unite, Sandstorm, Docker... web3?), but none have prevailed.
A possible answer could be because it has created a huge market for 3rd party services to step in and make things easier for web users. And now it's probably too late to stop the train. But there's no reason this couldn't work while empowering the user.
* Most personal devices are battery powered and intermittently operated.
* Many are mobile in terms of physical location and logical network (home WiFi, work/school WiFi, cellular).
Having a third, highly available, high upload capacity location to stage data between production and consumption wins because it is a genuinely effective solution to problems inherent in sharing data between end users.
There's no technical reason for this to be the case. ISPs adapted to the needs of the web, not the other way around.
> lack upload bandwidth
This is relative to each user. The vast majority wouldn't require much upload bandwidth, and there could be technical solutions to address this (caching nodes, data expiration, P2P, etc.).
The biggest technical challenges of large web services are because of the scale needed to support the large amount of users. If we had built services and tools around the inherent distributed nature of the web, centralization and all the problems caused by scale wouldn't be an issue.
> battery powered and intermittently operated
> mobile
Why should user data be highly available? If I'm physically unreachable or just want to be offline, shouldn't my data be unreachable as well? Besides, all of these can have technical solutions as well.
> Having a third, highly available, high upload capacity location to stage data between production and consumption wins
I'm not disagreeing, but a) this wouldn't be required by most users, and b) why couldn't this be under control of users as well? The fact no such (simple) solution exists today doesn't mean that it couldn't have gained traction back when browsers were getting adopted, and today we would've had a completely different web. Unfortunately incentives are turned on its head, users aren't educated about the harms of giving their data away because they've learned that it's the way the web works, we have to pass laws to protect user data, and a vocal minority of web developers have been swimming upstream and trying to undo the harms of centralization for decades with lackluster results.
One of the main reasons we “hire” the platform companies to convey our data to other people rather than opening TCP sockets to each other directly is precisely async delivery. No one wants to use a social network where we can only see each other’s content if we’re using our laptops at the same time. Just make a phone call at that point.
Now maybe you could ask for federated, open standard queueing mechanisms. But we have one of those! It’s called SMTP, and it’s not really up to the needs of modern social applications. And maybe we could do something about that. But it’s also in the nature of federated open standards that are widely deployed to ossify. The other service that Facebook and Twitter are doing for you besides pub/sub is having the coordination and agency to update their own deployments over time, something that e.g. the set of all relevant email operators does not have.
It's not just the data you obviously fill in yourself
and that's before considering cookies other than letting you auto login to a specific site you already logged in to before
Also, isn't Whatsapp owned by facebook now? Don't trust facebook or anything they own
Sadly publishing goes towards the lowest friction service. Was websites, then blogs, now Twitter. The backgrounds and themes weren’t as important as simple content. I’m not sure home hosting has any chance to beat that trend.
The mx records could point to this homeserver box and you would fetch your messages from there. It would be the norm that your graph, photos, whatever would be stored there and you would have to allow access to share them out with others. That would have lead to a very different internet than today.
But the main thing I'm thinking of is having a physical, secure enclave protected, device that allows all internet users to u2f to their ISP in order to have the option for an "ip permitted from" protocol for registration on websites. Something that can ensure that when grandmas pc gets hacked, it's not used to create 100 Twitter accounts to pump some crypto scam.
Today even if we could get ISPs to setup such a system, people wouldn't use such a system as they are already use to easy sign ups, and companies love their growth metrics
I wish for a world where battery tech wasn’t so limited. Imagine if everyone could just run a full-fledged server 24/7 on their phone, as a simple app, with a reasonable data plan.
This is the obvious and simplest solution. A built-in self-hosting platform right in the router, extensible with an external USB drive if the user needs it. But ISP's are notoriously terrible at everything and certainly can't be trusted with something like this.
https://en.wikipedia.org/wiki/Freebox
Edité: Liberté, égalité, cyberité!
(Did you notice the source in my link?(at all?))
I did. It has its own slew of problems.
> Or another ISP.
ISPs hold monopolies in many areas of the US.
> Or get a bizniz line
Not always available. Too expensive when it is.
> Or move?
Is this a joke?
> Or use StarLink?
Waiting list. Expensive. Latency.
> Did you notice the source in my link?(at all?)
Did you really expect me to read the source code from a bunch of patches for a french modem and gain some kind of magical insight?
Err, nope? I just posted that link to show you that it doesn't have to be like you said, especially not for the device in question.
> ISPs hold monopolies...
They did, and sometimes still do elsewhere too.
I don't give a shit, since I don't live in "free 3rd-worldistan with nukez, carz, and screaming lunatics on crack, or whatever else".
> Is this a joke?
Not at all. Totally serious. It's the thing to do, to escape learned helplessness.
I can't help but feel distributed computation is a really really fascinating problem and if the socioeconomic wave we're going through now sustains even a fraction of this current moment it'll be a longterm engineering focus.
It's impossible for me to not recognize that the diff blockchains mirror that of different database designs as the web scaled from nineties. First read capacity was needed to support e-commerce. Followed by social platforms where read/write needed to scale and adopt distributed models and eventual consistency.
Now we're scaling distributed computation and all sorts of interesting problems emerge. If things are gonna turn out to be even remotely what an idealist might lead you to believe we're at the cusp of rearchitecting every single layer of computation. Networking. Machine code compilation and execution. File storage.
PS I did a couple of cmd+f for keywords to find someone answering with this context and didn't find any. That seems crazy.