Somehow I think the overhead of TCP will be a lot more significant than the overhead of a copy from cache to memory.
Wouldn't such a system make more sense for sending raw memory directly over the wire in some form of custom protocol? Even UDP would probably be able to approximate that.
Tcp is also extremely battletested and performs well not just theoretically but in the face of other traffic (congestion) and in many different environments.
Beating TCP reliability and performance is an extremely ambitious endeavor in practice. Add wide interop to that list and it becomes next to impossible.
It's also important to have an idea what are the theoretical gains to be had with this work. It would make no sense to mess with infrastructure at this level of the tech stack if the ballpark estimates of potential gains are in the single-digit percent points. In this scenario it would be much more efficient to either go with horizontal scaling or look into the possibility of shaving off some requests.
Sorry, no. At this scale, if google thought they could shave single digits off their cpu costs they would staff a team to do it. If they thought the digit was “9” they would put dozens of people on it.
Have a look at Netflix's 800Gbps CDN node presentation. If you want to hit high transfer speeds, your CPU may not have the bandwidth to read from your device and write it to the NIC. DMA between devices (assuming that's what is happening here), saves memory bandwidth for other stuff.
> Have a look at Netflix's 800Gbps CDN node presentation. If you want to hit high transfer speeds, your CPU may not have the bandwidth to read from your device and write it to the NIC.
Would it be feasible to horizontal scale those nodes instead?
Sometimes yes, but CDN nodes are competing for space in ISP edge hosting. Fitting an 800Gbps appliance into 2-4U means you can fit into a lot more ISPs. Additionally, if you use multiple hosts in a single deployment, you often also need a router, since the ISPs want to provide a single connection. That adds a significant cost to deployment in dollars and space.
Additionally, I don't know why you would choose not to use the capacity of the system you have, just because you can easily scale horizontally. If a bit of engineering time can significantly reduce the number of systems to manage and the hardware budget, seems like a good plan. Especially when these projects are fun to work on and large tech companies want to retain employees.
> Fitting an 800Gbps appliance into 2-4U means you can fit into a lot more ISPs.
The work and investment to turn that 4U deployment into two 2U deployments sounds like a far more realistic and effective way to go about the problem.
> Additionally, I don't know why you would choose not to use the capacity of the system you have, just because you can easily scale horizontally.
The point is that the capacity does not exist, hence these patches being shot down by kernel maintainers for being fundamentally broken.
Also, what are the real-world performance improvements of these memory hacks? Adding another COTS box next to the one already deployed should double capacity straight away.
from what I can see the kernel developers don't like the particular means used in the patch, there's no fundamental objection to the idea of enabling device-to-device DMA as a performance optimisation.
I assume the actual TCP packetization is done by the network hardware in the typical case. Then the overhead of TCP is hidden as long as the network hardware can operate at full line rate.
At 200+ Gbps, the copy from LLC where the packet landed to the userspace buffer dominates the performance profiles on most systems. The TCP processing isn't bad and the expensive parts can often be offloaded. I'd contend that this data copy on the TCP recv path is one of the most important performance issues to solve for the entire industry right now. DDIO let us kick the can way down the road, but it seems like the network speeds have outpaced the CPU speeds so much that it can't save us for much longer.
Devmem TCP sounds a lot like direct memory access. Am i mistaken if i think of it as a security nightmare? Do you have by chance any links to security considerations?
DMA is only a problem if transfers can be initiated by an untrusted party. This patch looks to basically just be making it so that transfers which would normally go device->RAM->device with no actual processing of the data transferred instead just go device->device instead (but still initiated by the OS, not the device or the network!). This doesn't affect access control at all, as the trip to RAM doesn't impact that (though it does mean it only works if the machine is not looking at or touching the data at all, which probably means it's only useful in situations where there's no encryption).
It's different. Rdma offloads the TCP transfer of host memory to the network device. This is about transferring device memory directly, avoiding going though the host.
Had to read through the lklm message, at it wasn't clear from the article.
From what I remember of Infiniband, the network cards are able to do direct PCIe transfers to/from compatible devices attached to the host (eg nvidia quadro gpus).
precisely this. The reason why its worth the expense is that data entering the network is guaranteed to come out the otherside. This kinda eliminates the need for TCP (apart from addressing)
This is my first thought, it sounds like inifiniband with with none of the guarantees.
I suppose it means that you can bust out of the local network, and take in "foreign" data. But if you're doing that, you've got bigger problems with data affinity.
Perhaps it's just avoiding the need to encapsulate an InfiniBand transport packet? I'm not sure how much overhead that introduces in RDMA, but DMTCP may just be an effort to divorce the idea from its original IB underpinnings.
RDMA is great and similar, but behaves very differently from TCP in the face of network congestion and longer distance traversals. This is essentially trying to get the best parts of TCP and the best parts of RDMA combined.
iWARP maybe, but I don't think you want to offload all of TCP to hardware. You want to leave congestion control and all that to software. I don't entirely know if that's why iWARP isn't very popular, but I suspect that's the reason. You want a software TCP stack that can land the data where you want it directly.
(I work for Google, not on anything related, opinions my own, etc)
I would suspect it's not going to be dead unless the maintainers think the fundamental premise, not the architecture, is broken - if it's been brought up on a real piece of hardware, it's almost certainly because it was solving a real problem being experienced, and even if it involves reworking it completely, getting it upstreamed is much better than maintaining an enormous local delta.
My experience watching google try to contribute to upstream kernel is their proposals are almost always based on things that have already been in production at google for years. So when I see an upstream gatekeeper accusing someone of not understanding what they are proposing, I am skeptical. The thing that is “dead on arrival” here is probably the ability of the general public to enjoy something that google already uses.
> So when I see an upstream gatekeeper accusing someone of not understanding what they are proposing, I am skeptical.
Your comment reads like a textbook example of an appeal to authority.
Google engineers don't walk over water, and what Google accepts in their internal networks considering on Google's tightly locked internal security infrastructure might not map to real world security environments.
It also boggles the mind how you somehow presume a Linux kernel maintainer who repeatedly pointed out the failures of the proposed patch does not know what he is talking about. Unbelievable.
> The thing that is “dead on arrival” here is probably the ability of the general public to enjoy something that google already uses.
I'd wager that what the general public enjoys is working code with an excellent security track record thanks to the outstanding work by Linux kernel maintainers, who repeatedly stop broken submissions from being accepted.
I don't doubt it. I also heard that when a Linux dev find a bug that might be a vulnerability, they often don't bother adding it to the CVE database (and say as much in discussions with the other kernel devs) which casts doubt on the usefulness of counting CVEs as a way to estimate how secure Linux is.
If you are taking MR's from the public you are somewhat DDoS'ed with patches that are of low quality and will likely cause problems in your codebase. I totally understand that not everyone has the patience to deal with that with tactful language.
Not implying that this mail must be an example of this.
46 comments
[ 2.6 ms ] story [ 88.5 ms ] threadWouldn't such a system make more sense for sending raw memory directly over the wire in some form of custom protocol? Even UDP would probably be able to approximate that.
- is supported everywhere
- allows massive reads and writes (whereas UDP often has quite small limits before packets are dropped)
- is reliable (you wont lose data, your data won't corrupt, etc)
So you could use this inside an existing datacenter, with existing hardware, and existing software to debug and monitor it.
If you send GB of memory, the TCP overhead is minimal.
Beating TCP reliability and performance is an extremely ambitious endeavor in practice. Add wide interop to that list and it becomes next to impossible.
How many companies do you know who operate nearly as many servers as Google?
Would it be feasible to horizontal scale those nodes instead?
Additionally, I don't know why you would choose not to use the capacity of the system you have, just because you can easily scale horizontally. If a bit of engineering time can significantly reduce the number of systems to manage and the hardware budget, seems like a good plan. Especially when these projects are fun to work on and large tech companies want to retain employees.
The work and investment to turn that 4U deployment into two 2U deployments sounds like a far more realistic and effective way to go about the problem.
> Additionally, I don't know why you would choose not to use the capacity of the system you have, just because you can easily scale horizontally.
The point is that the capacity does not exist, hence these patches being shot down by kernel maintainers for being fundamentally broken.
Also, what are the real-world performance improvements of these memory hacks? Adding another COTS box next to the one already deployed should double capacity straight away.
Security is/was pretty good once an LSM permits the connection creation.
At least with mellanox stuff the last time I touched it, there were I think four levels of reliability, one of which matched TCP.
Had to read through the lklm message, at it wasn't clear from the article.
Specifically to avoid bogging down the host.
I suppose it means that you can bust out of the local network, and take in "foreign" data. But if you're doing that, you've got bigger problems with data affinity.
[0] https://lore.kernel.org/dri-devel/8719bf65-6906-0426-1ce4-b0...
I would suspect it's not going to be dead unless the maintainers think the fundamental premise, not the architecture, is broken - if it's been brought up on a real piece of hardware, it's almost certainly because it was solving a real problem being experienced, and even if it involves reworking it completely, getting it upstreamed is much better than maintaining an enormous local delta.
> I strongly suggest that you read up on the basic of this before proposing any solution.
Ouch.
That’s just what code review is there to spot, not something that can’t be fixed.
Your comment reads like a textbook example of an appeal to authority.
Google engineers don't walk over water, and what Google accepts in their internal networks considering on Google's tightly locked internal security infrastructure might not map to real world security environments.
It also boggles the mind how you somehow presume a Linux kernel maintainer who repeatedly pointed out the failures of the proposed patch does not know what he is talking about. Unbelievable.
> The thing that is “dead on arrival” here is probably the ability of the general public to enjoy something that google already uses.
I'd wager that what the general public enjoys is working code with an excellent security track record thanks to the outstanding work by Linux kernel maintainers, who repeatedly stop broken submissions from being accepted.
It seems we are talking about different projects.
Please point out a operating system that you feel has a better security track record than Linux.
https://www.cvedetails.com/cvss-score-charts.php?product_id=...
What exactly is your definition of security track record? Do you have any, or are you parroting cliches?
The first Linux CVE I clicked at random has CVSS 0.0: https://www.cvedetails.com/cve/CVE-2022-4095/
The same CVE on the official NVD has CVSS 7.8: https://nvd.nist.gov/vuln/detail/CVE-2022-4095
I’m shocked someone would talk like this publicly from their company email. AMD doesn’t seem like a company I would want to work for.
Not implying that this mail must be an example of this.
Man, my former company could have chipped this device-memory mapped TCP stack for Linux kernel 2.14 to 2.24.
I created that. And it likely going to disappear forever although the product may still being in use today.
What a waste.
Was coded primarily to recapture the CPU burn on lower-end Pentium platform.
Good stuff, (sigh)
It had a project name but still unclear as to my contractual obligation with regard to reveal anything else about it (other than resume stuffing).