Oversimplified - it's also a multi-threaded driver with multiple queues. This should hopefully allow you to fully saturate the 20gbps available to you.
Does anyone know what type of hardware they are using to accomplish this? Is it just your standard Intel X520/540 10GbE adapter, or is there some secret hardware they have access to?
FWIW, saturating line rate 10GbE has been problematic at times in the past due to challenges like single-threaded drivers, which could only run on 1 core at a time, even if there are 128 logical cores in a system (like the X1 instances). There are also challenges offloading VXLAN encapsulation to hardware at 10GbE line rate, but it seems like AWS has figured that out.
Since it offers 20Gbps to the VM, i'd guess it's a 25GbE NIC, since those and accompanying switches have just come out.
VXLAN in the NIC (and the switch) at full line-rate offload has been a thing for at least 18-24 months.
Most Modern NICs also support all of the features listed here, like checksum offload, Receive steering, multi-queue, etc. Check out Mellanox CX4/5, Intel X710, QLogic, and Netronome if you want fancy stuff. They aren't even that expensive anymore.
Their Linux driver has support for every current and future standard: 1/2.5/5/10/25/40/50/100/200/400G, nice future proofing.
With current per port costs and economics, that doesn't make much sense. The $/Gbps/Watt improves with every spin of the chip, and every increase in serdes bandwidth.
10GbE (64x10G) - ~$4,500, $70/port
10GbE (32x40G) - ~$6,000, $50/port
25GbE (72x25G) - ~$7,000, $100/port (Seen as low as $5k)
25GbE (32x100G) - ~$9,000, $75/port (Seen as low as $7k)
2x 10GbE = $100
1x 25GbE = $75
Not to mention the extra cabling costs, and complexity.
Huh -- why is Amazon introducing ENA as a paravirtual virtual device? They already provide Xen netdevs to VMs.
The Xen PV device would be able to use the (more efficient) xen bus/event channels for interrupts vs. ENA (MSI-x, by looking at their driver on the Linux netdev mailing list).
The Xen PV device already also supports multiple queues for xmit, see netback_init() in netback.c.
2. The ENA LLQ interface (pushing short packets via CPU write-combiner writes) only makes sense when you are writing to a physical PCI-e/Hypertransport/QPI device.
I suspect that this ENA NIC is at least partly implemented in silicon/firmware rather than dom0 s/w.
Perhaps something like Azure's smart NICs? (SIGCOMM 2015 keynote)
10 comments
[ 4.9 ms ] story [ 26.2 ms ] threadFWIW, saturating line rate 10GbE has been problematic at times in the past due to challenges like single-threaded drivers, which could only run on 1 core at a time, even if there are 128 logical cores in a system (like the X1 instances). There are also challenges offloading VXLAN encapsulation to hardware at 10GbE line rate, but it seems like AWS has figured that out.
VXLAN in the NIC (and the switch) at full line-rate offload has been a thing for at least 18-24 months.
Most Modern NICs also support all of the features listed here, like checksum offload, Receive steering, multi-queue, etc. Check out Mellanox CX4/5, Intel X710, QLogic, and Netronome if you want fancy stuff. They aren't even that expensive anymore.
Their Linux driver has support for every current and future standard: 1/2.5/5/10/25/40/50/100/200/400G, nice future proofing.
They also have a DPDK driver: http://dpdk.org/browse/dpdk/tree/drivers/net/ena/base/ena_co...
10GbE (64x10G) - ~$4,500, $70/port
10GbE (32x40G) - ~$6,000, $50/port
25GbE (72x25G) - ~$7,000, $100/port (Seen as low as $5k)
25GbE (32x100G) - ~$9,000, $75/port (Seen as low as $7k)
2x 10GbE = $100
1x 25GbE = $75
Not to mention the extra cabling costs, and complexity.
The Xen PV device would be able to use the (more efficient) xen bus/event channels for interrupts vs. ENA (MSI-x, by looking at their driver on the Linux netdev mailing list).
The Xen PV device already also supports multiple queues for xmit, see netback_init() in netback.c.
2. The ENA LLQ interface (pushing short packets via CPU write-combiner writes) only makes sense when you are writing to a physical PCI-e/Hypertransport/QPI device.
I suspect that this ENA NIC is at least partly implemented in silicon/firmware rather than dom0 s/w.
Perhaps something like Azure's smart NICs? (SIGCOMM 2015 keynote)