I find it interesting how computing is starting to mature. It used to be something that only the latest generation was privy of. Everything was state of the art and always changing. My parents didn't understand me playing video games, it was not something they could grasp. Now, my daughter will probably be there when the Internet is a 100 years old, some ancient invention from a generation long gone. Computing will feel very different from how it feels to me.
Ethernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch already.
TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it.
Of course people came up with workarounds, but that's mostly adding a bunch of different layers on top. So on the surface everything looks stable, but in reality there's a huge amount of protocols piggybacking on top of TCP and UDP.
Jumbo/9000 byte frames have been a thing for a long time now and a lot of switches enable it by default so you can definitely reap the benefits of 100G/400G. The problem is that every L3 hop also needs to support a higher MTU if you want to route this traffic without fragmentation.
And that’s reasonable, because loss of a large datagram has larger impact. Thus jumbos in data centers, but not in the wild Internet where losses are more common.
This would be good enough, and allow for progressive enhancement, except that path MTU detection is somewhere between fragile and broken on the internet.
So maybe you run 9000 packets on internal interfaces, and 1500 externally, or more often, that's too hard and 1500 is it.
There are 9000 internet exchanges, so maybe there is hope, but I haven't seen any progress reports.
> Ethernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch already.
How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is?
I can see some server applications, but we've had offloading for years now:
> TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it.
Tallk to the vendors of middleware boxes (e.g., (CPE) firewalls): we've had (say) SCTP and DCCP for twenty years that offer variations, but they're never allowed to pass through. No new Layer 4 can really be allowed through because of ossification, and so kludging everything to TCP/UDP port 80/443 is what we're left with.
NAT and a lack of end-to-end also doesn't help: one has to futz around with things like STUN and TURN.
> How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is?\
Well, here's my personal experience for instance.
If you take a Linux server and try to do put a 10G interface into a bridge, that disables a good amount of the offload capability of the network card. As a result you don't get 10G anymore even on fairly decent hardware.
So VMs in a bridge with a 10G interface is already a configuration that doesn't quite work.
VFIO does the trick fortunately.
Also, surely even for ASICs there's a cost that comes with having to handle 150M packets/s.
And yeah, you can do those speeds on switches and devices with limited functionality, but maybe I want some iptables rules on my server hooked up to 100G.
And all of those are really workarounds that shouldn't be necessary. Having more than a hundred million packets per second is something that only happens because things got fossilized on a tiny packet size, not because there's actually some point to splitting up a 2 GB download into more than a million tiny bits.
> How limiting is 1500 for the end-nodes for most people?
Not very. For most end users, connection speeds are much less than 1G and 1500 is a reasonable size.
> For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is?
Yes, for these systems, one of the limiting factors is routing lookups per second. You'll see some systems that can only process wire speed with large packets, not with small packets, maybe with a mix. Increasing the size of packets reduces the packet count, reducing the routing lookups or other per packet work.
Lots and lots of programs work by building some sort of message system on top, with some sort of message type/length/payload system. Thousands of slightly different versions of reinventing that particular wheel.
Not sure I understand, sorry. By “on top” do you mean L5-L7 protocols? Like, for simplicity, FTP or HTTP? Or you mean reimplementing the transport layer itself, like QUIC?
Regardless, I don’t see that it’s possible to get rid of TCP/IP in any meaningful perspective really. There’s millions of devices around that support TCP offload, thousands of thousands of network equipment with routing processors and ASICs onboard. See ipv6 transition as a good example, and we’re much, much more ready to ipv6 than to a new tcp.
There’s a ton of artifacts and limitations in the stack that don’t make sense today - tcp ports as connection ids being one of them - but that’s something that’s taken care of on L7. Transport layer is a different beast though.
The entire network stack and the whole OSI concept is awful... The way DNS and TLS and certs are done kind of sucks, ARP is pretty insane... not much in networking is all that nice, there's not really any piece of it that is as awesome as something like a nice Python library.
Encryption could have been IP layer, DNS could have always been secure, certificates don't need to exist if IP addresses and DNS were secure by themselves, separate IP and MAC addresses might not really be needed ...
But the fact that there is a global standard internet protocol is amazing!
Some things are the same because they've changed considerably.
Ethernet used to be a collision domain on coaxial cable, not a switched full-duplex medium on twisted pair. It changed piece by piece into something that's conceptually fairly different from what it was, especially when it ceased being a collision domain.
> Early Ethernet variants (10BASE5, 10BASE2) were based on a shared wire and inherently half-duplex, representing a single, potentially large collision domain. Collision domains are also found in an Ethernet hub or repeater environment where each host segment connects to a hub, and all segments represent only one collision domain within one broadcast domain. Collision domains are also found in other shared medium networks, e. g. wireless networks such as Wi-Fi.
> Modern wired networks use a network switch to reduce or eliminate collisions. By connecting each device directly to a port on the switch, either each port on a switch becomes its own collision domain (in the case of half-duplex links), or the possibility of collisions is eliminated entirely in the case of full-duplex links. For Gigabit Ethernet and faster, no hubs or repeaters exist and all devices require full-duplex links.
> Ethernet used to be a collision domain on coaxial cable, not a switched full-duplex medium on twisted pair.
And it's come back full-circle to (radio) collision domains (à la AlohaNet) with Wifi, which I think the vast majority of people use on a day-to-day basis rather than plugging in.
> the vast majority of people use on a day-to-day basis rather than plugging in.
For last 100 feet yes, but for the entire rest of the network it's either fiber or copper. Notably, Ethernet framing sometimes with additional encapsulation is used everywhere along the path.
That may have been true in the past, but recently the same principle (using a hybrid) for simultaneous tx/rx on GBE can be employed on a point-to-point radio channel.
It's not a standard, it's a technology. 802.3 defines full-duplex (wired) gigabit Ethernet with each end of the peer-to-peer link both transmitting and receiving at the same time over the same wires. Add a similar hybrid to your WiFi PHY and you're done. It does also require MIMO beam forming to cancel the interference from other users on the same WAP.
> a 10Mbps system running over copper wire will carry bits at 0.75 the speed of light in a vacuum, or at 224,844 kilometres per second. This means that 576 bits at 10Mbps will be contained in 12,951m of copper cable
There is something intriguing about this image of localizing information along the cable...
> pingfs is a filesystem where the data is stored only in the Internet itself, as ICMP Echo packets (pings) travelling from you to remote servers and back again.
Many may not be aware of this, but signals through fiber optic cables actually propagate more slowly than through copper. Fiber carries bits at about 0.67 the speed of light in a vacuum.
It doesn't go the speed of light in copper, either. In fact, that study has a nice table on page 5. Looks like other than thick coax, fiber is a bit faster than most copper options.
Computerphile did a video a few weeks ago, "Ethernet (50th Birthday)", that gives a nice overview of how it developed for those unfamiliar with it works:
45 comments
[ 3.5 ms ] story [ 97.4 ms ] threadEthernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch already.
TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it.
Of course people came up with workarounds, but that's mostly adding a bunch of different layers on top. So on the surface everything looks stable, but in reality there's a huge amount of protocols piggybacking on top of TCP and UDP.
So maybe you run 9000 packets on internal interfaces, and 1500 externally, or more often, that's too hard and 1500 is it.
There are 9000 internet exchanges, so maybe there is hope, but I haven't seen any progress reports.
How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is?
I can see some server applications, but we've had offloading for years now:
* https://en.wikipedia.org/wiki/TCP_offload_engine
Netflix is hitting (encrypted) 800 Gb/s:
* https://www.youtube.com/watch?v=36qZYL5RlgY
* https://news.ycombinator.com/item?id=32519881
> TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it.
Tallk to the vendors of middleware boxes (e.g., (CPE) firewalls): we've had (say) SCTP and DCCP for twenty years that offer variations, but they're never allowed to pass through. No new Layer 4 can really be allowed through because of ossification, and so kludging everything to TCP/UDP port 80/443 is what we're left with.
NAT and a lack of end-to-end also doesn't help: one has to futz around with things like STUN and TURN.
Well, here's my personal experience for instance.
If you take a Linux server and try to do put a 10G interface into a bridge, that disables a good amount of the offload capability of the network card. As a result you don't get 10G anymore even on fairly decent hardware.
So VMs in a bridge with a 10G interface is already a configuration that doesn't quite work.
VFIO does the trick fortunately.
Also, surely even for ASICs there's a cost that comes with having to handle 150M packets/s.
And yeah, you can do those speeds on switches and devices with limited functionality, but maybe I want some iptables rules on my server hooked up to 100G.
And all of those are really workarounds that shouldn't be necessary. Having more than a hundred million packets per second is something that only happens because things got fossilized on a tiny packet size, not because there's actually some point to splitting up a 2 GB download into more than a million tiny bits.
Not very. For most end users, connection speeds are much less than 1G and 1500 is a reasonable size.
> For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is?
Yes, for these systems, one of the limiting factors is routing lookups per second. You'll see some systems that can only process wire speed with large packets, not with small packets, maybe with a mix. Increasing the size of packets reduces the packet count, reducing the routing lookups or other per packet work.
The stream is an atom of data - it's a block of serialized memory. How else do we transfer arbitrary amounts of data?
What do you mean?
Lots and lots of programs work by building some sort of message system on top, with some sort of message type/length/payload system. Thousands of slightly different versions of reinventing that particular wheel.
Encryption could have been IP layer, DNS could have always been secure, certificates don't need to exist if IP addresses and DNS were secure by themselves, separate IP and MAC addresses might not really be needed ...
But the fact that there is a global standard internet protocol is amazing!
Ethernet used to be a collision domain on coaxial cable, not a switched full-duplex medium on twisted pair. It changed piece by piece into something that's conceptually fairly different from what it was, especially when it ceased being a collision domain.
https://en.wikipedia.org/wiki/Collision_domain
> Early Ethernet variants (10BASE5, 10BASE2) were based on a shared wire and inherently half-duplex, representing a single, potentially large collision domain. Collision domains are also found in an Ethernet hub or repeater environment where each host segment connects to a hub, and all segments represent only one collision domain within one broadcast domain. Collision domains are also found in other shared medium networks, e. g. wireless networks such as Wi-Fi.
> Modern wired networks use a network switch to reduce or eliminate collisions. By connecting each device directly to a port on the switch, either each port on a switch becomes its own collision domain (in the case of half-duplex links), or the possibility of collisions is eliminated entirely in the case of full-duplex links. For Gigabit Ethernet and faster, no hubs or repeaters exist and all devices require full-duplex links.
And it's come back full-circle to (radio) collision domains (à la AlohaNet) with Wifi, which I think the vast majority of people use on a day-to-day basis rather than plugging in.
[0] https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_... [1] https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_...
For last 100 feet yes, but for the entire rest of the network it's either fiber or copper. Notably, Ethernet framing sometimes with additional encapsulation is used everywhere along the path.
https://www.microwaves101.com/encyclopedias/simultaneous-tra...
There is something intriguing about this image of localizing information along the cable...
> pingfs is a filesystem where the data is stored only in the Internet itself, as ICMP Echo packets (pings) travelling from you to remote servers and back again.
https://www.youtube.com/watch?v=JcJSW7Rprio
if you draw a Poincare disk model [1] where one dimension is nerdiness and the other is humor this guy is floating somewhere outside the boundary.
[1] https://en.wikipedia.org/wiki/Poincar%C3%A9_disk_model
https://www.commscope.com/globalassets/digizuite/2799-latenc...
https://aviatnetworks.com/solutions/low-latency/
* https://www.youtube.com/watch?v=TkOVgkcrvbg
The Computer History Museum also had a discussion with Metcalfe (and others) on it:
* https://www.youtube.com/watch?v=T9On2L0-ObU
* https://computerhistory.org/blog/ethernet-turns-50/
* https://en.wikipedia.org/wiki/Robert_Metcalfe
Also perhaps see Computerphile's "Discussing PDF@30 Years Old", which celebrated its anniversary/birthday in January:
* https://www.youtube.com/watch?v=Bffm1Ie66gM
Ooh! Can't wait.