Wow, reported May 2014, and still no resolution for such a serious issue. GCE looks nice in theory, but I've heard no end of problems like this with shitty communication and support when something doesn't work. I'll stick with AWS for the time being. I really wish Google would get their act together and provide serious competition though. That's good for everybody who uses the cloud.
Whenever people talk about behavior/treatment of UDP traffic I consider DNS as a special case. I have no idea how AWS handles UDP but I will never use DNS as a generalizable example of UDP traffic.
At least in the past, it really was true. I've been burned by this before using UDP in AWS. In AWS, I've learned to be skeptical of using protocols other than TCP. That said, it consequently has been a long time since I've tested UDP over AWS.
DNS was precisely the area where we got bit by it.
We were using a Non-AWS DNS resolver (aka Google) and we would often get dns resolution errors despite our NAT not being remotely taxed by the traffic.
Yeah, I've run in to this. The point about NAT makes me wonder though if it is really de-prioritization or just the network straining to handle all that recalculation of checksums.
It's legal to send UDP packets with a zero checksum, indicating "no checksum." This can be set at a UDP socket level in Linux. I wonder if that would make any difference?
(Of course this assumes your protocol has some alternate method of verifying transferred data, which many do.)
The last update really hits the nail on the head for most Google products:
> Apparently the update from Google is to find another support channel to escalate, or only use TCP.
I've never had a Google product issue ever resolved using an official channel I was directed to. It's only by back channels, friends of friends, posting to HN, etc.
They've likely blocked your VPN because they think you're trying to bypass region restrictions. In this case they don't care whether you're human or not: they simply want to make sure you can't watch videos that would not normally be available in your country.
It's a ridiculous system, but it's unfortunately how media licensing still works.
You could try getting a cheap VPN endpoint device and putting it between your router and your residential internet connection. Then you could VPN to that device to access the (usually-blocked) YouTube while still using your residential IP (assuming the blocking is done on your router).
Well, the router is also the DSL modem – so there’s nowhere to put it inbetween, and my parents hate modifying that stuff.
So I’ll have to pay more for a VPS that’s in my country just to make sure I don’t get hit by this again? Well, then fuck it, I’ll just use a goddamn free VPN or Tor or whatever.
I think your best bet is to solve the youtube filter/access problem in a different way. If you have a cable modem and/or wireless router, it may let you filter web traffic by MAC address (mine does). If your sister uses a different computer than you, you could block access for her MAC, and allow access for yours. If there's only a single computer, software that does web filtering with a password-based override would probably work.
Well, I know that that won’t work – back when I was still young, my parents did that, and I – then also 14 – used aircrack-ng to sniff for my dad’s laptops MAC address and use that.
I’m sure my little sister doesn’t know how that works, but I’d bet she’d find a friend or friend of a friend who does.
Well, for now it seems to work again. I was just trying to test Tor, when I noticed that I hadn’t started Tor and I was watching YouTube actually over my VPN.
I've had a couple of issues with Google Apps and their support has been very helpful, getting a phone call from them and the issue solved in the same day. For example I got them to revert me from annual to the flexible plan by simply asking nicely and I even got them to operate settings that aren't normally available, like changing the primary domain. Nowadays I've been moving off Google Apps, but let me tell you, it's the support that I'll miss ;-)
Also back in the day when I was working on integrating with AdX, their reply was very slow, but they did reply and they did help us with our integration.
I do not have experience with GCE, but saying that you don't get support for any Google product is disingenuous.
My experience has been quite different. I signed up for the Cloud Platform free trial a couple of months ago and they have responded to me the same day at the latest when I have issues. Hell, they even got back to me on the weekend.
Broadly speaking, UDP applications which rely on IP packet fragmentation are broken as designed. If you wanted reliable transport you would have used TCP or a higher level abstraction. If you wanted simple transport of large data chunks, you would have chosen likewise. You picked UDP because you have latency requirements that cannot be met by TCP, and that means you need to know what your packets are actually doing, and that includes fragmentation.
If you want to play in that world you need to be prepared to handle MTU discovery on your own, or else design your app around deliberately small packet sizes.
That's not to say that this isn't a bug. But let's not start editorializing our titles: the apps for which GCE is "unusable" are buggy apps to start with.
While I (sort of) agree with the basic premise of what you're saying, in the real world, its not very helpful.
We didn't "pick" UDP. We operate a VoIP related service that interoperates with many different carriers via SIP. Almost all of those carriers ONLY use UDP. SIP UDP packets can often be fairly large. This is especially problematic because GCE uses a non standard MTU size (1460 bytes). This does not make our app, or every other SIP related app that is forced to use UDP, "Buggy".
Upon further consideration, I've deleted the bulk of this comment.
Rough summary of what I had here: I'm an engineer on GCE (in particular I built our current virtio-net device and a small fraction of the other fiddly bits that sit behind that) -- some details in the bug jumped out at me and I thought there might be a quick fix, but I hadn't processed all of the details and posted a bit prematurely. After further review my original post was essentially content free other than 'IP fragmentation works correctly between internal IPs', which is not germane to the actual customer-reported issue.
Thanks for your comment. The vast majority of networks outside of GCE don't use an MTU of 1460. Even if we set the instance MTU size to 1460, we still have problems. If a UDP packet sent to a GCE instance from outside GCE is larger than GCE's MTU, the first fragment is received and no subsequent fragments ever make it to the instance. This is the bug. This is very common with protocols like SIP when the SDP gets large due to a lot of media attributes.
I'm not sure if I'd go so far as to say that apps which rely on UDP fragment reassembly are "buggy", but I definitely agree that the article title is exaggerating by calling UDP on GCE "unusable". Many UDP services (including one operated by my company, on GCE) will work just fine.
Interestingly Linux will actually, by default, throw EMSGSIZE any time you try to send() a UDP datagram that is larger than the detected network MTU to the destination. As I understand it, you have to explicitly turn this behavior off to get fragmentation.
After a suitable time-out. Hm. That makes me wonder if you couldn't use a scheme such as fragment-fragment-dropped-fragment repeatedly to eat up router or IP stack resources.
> Isn't fragment reassembly essentially implying all the packets (fragments) will arrive, and will be rearranged in order? i.e. exactly what TCP does?
No. It is implying that it does what UDP does. TCP is different because it constructs a continuous stream. UDP is inherently packet/message based.
UDP ensures that all of the IP fragments for a given UDP packet get reassembled and rearranged and order. The UDP packets themselves are not arranged in order, and it is possible to see duplicates.
> For example imagine sending a single 1MB packet via UDP
Not possible. You can't send more than a 65,507 byte packet via UDP[1].
Sigh... unfortunately, there are a lot of misconceptions about the basic plumbing of TCP/IP.
[1] UPDATE: as was pointed out in comments, my statement was only true for IPv4. Since IP packets in IPv6 can have jumbograms larger than 65,535 packets, IPv6 UDP does support larger packets. Of course, in that case, UDP doesn't handle the fragmentation/defragmentation protocol itself, and of course GCE won't handle anything IPv6.
Sorry. I was presuming we were talking about about UDP fragmentation with IPv4, as this was the bug report and of course GCE doesn't support IPv6.
IPv6 jumbograms are a whole different kettle of fish, as they are really handled at the IP layer, and are supposed to only be for cases where you have an MTU > 65,575 bytes. As that link describes, the UDP protocol itself doesn't do any fragmentation or reassembly of those packets. RFC 2147 is really just an update to handle the fact that IPv6 can have a single packet that is larger than an IPv4 packet.
> Not possible. You can't send more than a 65,507 packet via UDP.
It's a thought experiment not a specific number. I'm trying to demarcate some of the differences between TCP and UDP, and UDP reassembling fragments looks a whole lot like TCP.
> UDP reassembling fragments looks a whole lot like TCP
There is only a tiny bit of similarity in that they are both dealing with reassembling a higher layer protocol payload from fragments split up over IP.
Let's count the ways they are different:
1. UDP does provide ordering guarantees within a UDP packet. It provides no ordering guarantees between UDP packets. Consequently, the protocol never has to worry about ordering more than 2^16 bytes of IP data (that makes the sorting algo really simple and efficient).
2. UDP doesn't have to deal with complex stalls or partial delivery of data. Either it has all the IP packets for a given UDP packet (in which case, it immediately delivers the UDP packet), or it doesn't (in which case it doesn't deliver it). If there is a UDP packet is missing a fragment, but a subsequent UDP packet is fully assembled, that one gets delivered.
3. There is no retransmission. UDP doesn't retransmit. It doesn't even guarantee deduplication. It is entirely possible that fragmented packets going different routes can end up creating an echo of multiple copies of a UDP packet being transmitted to the recipient.
The above makes UDP far simpler, and the contract application layers operate under very, very different. If you look at your typical TCP stack's logic for reassembling fragments, you'll find it far, far more complex than for UDP.
> UDP doesn't have to deal with complex stalls or partial delivery of data.
Are you sure? If some fragments show up and some not, doesn't it have to wait and see if the rest show up? Maybe it doesn't have to wait for a long time (like TCP), but it does have to wait.
Both point 1 and point 2 are a matter of quantity vs TCP, rather than quality. i.e. UDP+fragments is a sort of TCP-lite.
> If some fragments show up and some not, doesn't it have to wait and see if the rest show up? Maybe it doesn't have to wait for a long time (like TCP), but it does have to wait.
Much of the stall logic is tied in to whether to wait longer, whether to deliver partial data, whether to request a retransmit, whether you've already transmitted the data (once you reassemble a UDP packet, you can fire it and then promptly forget you ever saw it.. if defragmentation causes you to reconstruct and send the packet again, that's 100% okay!) and most importantly, what to do with an open ended amount of data that may need to be held up while waiting for that one lost fragment.
The logic is simple:
1) Allocate space for UDP packet.
2) Fill in UDP fragment data as it arrives.
3) If all the data for a given packet is there, transmit it and delete all memory of the fragment.
4) If you timeout, delete all memory of the fragment.
That is so, so, so much easier.
If you looked at the code, you'd not think it much similar.
UDP is 'spray and pray'. If a packet is fragmented (goes out on the wire as more than one IP frame), several things can happen. Frames can jitter (arrive with different timing); they can be reordered (e.g. packets sent 1,2,3 but arrive 2,1,3 or 3,1,2 etc) and a frame can be dropped (e.g. arrive 1,3 or 2,3).
UDP has no feedback from the receiver to the source. If a packet is dropped, the IP stack on the receiver will never complain and the IP stack on the sender will never resend. SO the receiver will silently drop the whole packet.
Notice it can be hard to tell if a packet is dropped, or is reordered. You receive 2,3 - maybe 1 is still coming but reordered? So detecting dropped packets is a heuristic, depending on timing. Various algorithms will use a timer to say when to give up on a packet, or drop a packet if a frame from a subsequent packet is received, or tolerate one frame from a subsequent packet (maybe the last frame of the previous packet was reordered with the first of the next) and so on. But all the work is on the receiver. The sender is not involved.
Unless there is some application-layer protocol that notices packets arriving with a gap, and sends something back (that might also not arrive) like "I got packet a,b,d but missed c". Etc. Notice the application cannot resend frames; only packets because it is unaware of fragmentation/reassembly which happens in the IP stack.
Unfortunately this relies on PMTU discovery working, which is frequently broken by misconfigured firewalls by eg blocking all ICMP traffic. Now that GCE has a more exotic broken firewall that also breaks traffic with PMTU disabled, you have a lose-lose situation.
No. It's broken. The title should be longer and include "which fragmentation".
Large UDP packets have a packets success rate of the single packet success rate raised to the number of fragments power, so don't get attached to them in situations with significant packet loss. But this behavior is just an unaddressed bug.
> If you wanted simple transport of large data chunks, you would have chosen likewise.
That's not entirely fair. UDP is actually a very convenient transport for packetized data of 65,000 bytes or less.
> You picked UDP because you have latency requirements that cannot be met by TCP.
That is not really a good reason to use UDP, as you won't necessarily get better latency (as has oft been demonstrated).
You should pick UDP because you have a connectionless protocol and/or one with data loss requirements that are different from those built in to TCP.
> that means you need to know what your packets are actually doing, and that includes fragmentation.
UDP is supposed to abstract IP fragmentation issues for you.
> If you want to play in that world you need to be prepared to handle MTU discovery on your own, or else design your app around deliberately small packet sizes.
I've used UDP based protocols all the time. I'd agree that I've had to be a bit more aware of path MTU when it comes to diagnosing or debugging problems, but in terms of the application code reading and writing UDP packets, I've never done MTU discovery.
> But let's not start editorializing our titles: the apps for which GCE is "unusable" are buggy apps to start with.
Considering there are bytes in the UDP header specifically designed to allow you send packets larger than MTU size, this is actually a huge problem. For many people, this requires reimplementing UDP's fragmentation & defragmentation logic up in layer-7.
Isn't it bad enough that people are reimplementing TCP on top of UDP? Do we really need to tell them to reimplement UDP on top of UDP?
> If you wanted reliable transport you would have used TCP or a higher level abstraction.
Nobody asked for that, just to send slightly larger packets that will probably get through.
> If you wanted simple transport of large data chunks, you would have chosen likewise.
Why? A 4KB data chunk doesn't need TCP more than a 1KB data chunk.
You know the 1500 byte MTU is completely arbitrary and many networks set it higher, right?
> You picked UDP because you have latency requirements that cannot be met by TCP, and that means you need to know what your packets are actually doing, and that includes fragmentation.
Fragmentation itself is a cost of microseconds. The main problem with TCP is head of line blocking, and moderately large UDP packets don't have to deal with that.
You don't need to trace every packet all the way through the network to get low latency and manually handle loss.
> the apps for which GCE is "unusable" are buggy apps to start with
I kind of wrote a lot but didn't summarize very well.
A key principle of UDP is quite the contrary of what you are saying. UDP applications can have guarantees that they don't have to engage in MTU discovery or fragmentation issues. It provides a way to have an abstract, static contract about packets that is agnostic to layer-2.
Because UDP is comparatively simple, it has been abused as a proxy to implement your own protocol on top of IP, and in that context you of course you really have to deal with al those concerns. However, it is a terrible mistake to think that is what UDP is about or how one should use UDP.
That this comment is on top of this thread is saddening. You just called about a dozen broadly used protocols that depend on nothing more than the basic guarantee provided by UDP (unreliable datagram delivery-non/delivery) "broken by design".
If I was Linus, now would be my cue to unleash a tirade about how you don't fing break userspace. Ever.
Protocol layering works because higher level protocols depend on long standing contracts with the lower layers. Google decided to willy nilly go and break the basic contract of UDP across their entire cloud and the top comment here faults protocols built on those guarantees. Disappointing.
It sounds like UDP packets that fit within the MTU work fine. If you need to transmit more than fits in one packet (1452 bytes), UDP is a bad choice.
SCTP is ideal for this use case but it is not well supported by OSs or networking APIs. TCP works but adds overhead. TFTP works, is UDP-only and has less overhead than TCP, but it does not respond well to packet loss. UDT is like TFTP done right, and is a good solution if you can setup a dependency on its large C++ library.
SCTP is a pretty good choice. It has been a bit since I used it, but it is a more complex protocol and I recall often run in to challenges getting it to perform as efficiently over various bits of network equipment.
Last I checked UDT was far more complex than UDP, and since it is layered on top of UDP, I'd think it'd be vulnerable to this problem (although it had all kinds of logic for correctly sizing packets and windows, so maybe it correctly avoids this problem). Either way though, from an application perspective UDT looks much more like TCP than UDP, so I wouldn't think it'd be an obvious choice to replace UDP.
> If you need to transmit more than fits in one packet (1452 bytes)
You must never make assumptions about what fits in one packet. The MTU could be 100, or less, or 8000, or more.
As soon as you start doing math based on MTU values that you don't permanently have end-to-end control of yourself, you're setting yourself up for trouble.
That's true, it's a bad idea to assume an MTU of 1500. Although there is no minimum MTU in IPv4, IPv6 specifies a minimum of 1280 bytes. So if you send your UDP packets over IPv6, you are guaranteed room for 1232 bytes of payload.
I've run into several misfires while using Google Cloud/Compute Engine: MySQL database access, email and encryption. These features didn't work without either a Google or third-party service. I set up postfix and use SendGrid for email and Google's Cloud SQL. You can get tech support at Silver or Gold level. I think everyone starts at Bronze.
This is certainly not the only case of "network subtly broken on cloud VMs". For example, every provider I have tested (including AWS, Rackspace, Digital Ocean, Linode) enables TCP segment reassembly offload, and provides no way to disable it (presumably because it is being done on the host not the VM). This will typically break TCP tunneling (e.g. using GRE) because PMTUD doesn't work under these conditions. fwiw a shout out to Soft Layer which is the only VM hosting provider I'm aware of that does not suffer from this blight (provided you pay for additional IP addresses routed to your box).
Any application might very reasonably choose to do this. Maybe version X never does, and you can satisfy yourself of this by peaking at the code, but there is no reason to believe that X+1 won't, or that another vendor's product/FOSS project you need to interoperate with won't, etc.
Please don't editorialize the titles of articles you submit to HN.
The submitted title was "Warning: Google Compute Engine Unusable for Applications Which Rely on UDP", which several commenters have objected to as exaggerated.
67 comments
[ 3.0 ms ] story [ 140 ms ] threadhttps://forums.aws.amazon.com/thread.jspa?messageID=536049
https://www.reddit.com/r/aws/comments/3ccn5o/real_ipv6_suppo...
The info I have came from a conversation with an AWS Solutions Architect.
It's a couple of years old, but based on my experience it's still true today.
We were using a Non-AWS DNS resolver (aka Google) and we would often get dns resolution errors despite our NAT not being remotely taxed by the traffic.
(Of course this assumes your protocol has some alternate method of verifying transferred data, which many do.)
https://github.com/zerotier/ZeroTierOne/blob/14264c2d6f7d20b...
> Apparently the update from Google is to find another support channel to escalate, or only use TCP.
I've never had a Google product issue ever resolved using an official channel I was directed to. It's only by back channels, friends of friends, posting to HN, etc.
It's a ridiculous system, but it's unfortunately how media licensing still works.
You could try getting a cheap VPN endpoint device and putting it between your router and your residential internet connection. Then you could VPN to that device to access the (usually-blocked) YouTube while still using your residential IP (assuming the blocking is done on your router).
So I’ll have to pay more for a VPS that’s in my country just to make sure I don’t get hit by this again? Well, then fuck it, I’ll just use a goddamn free VPN or Tor or whatever.
I’m sure my little sister doesn’t know how that works, but I’d bet she’d find a friend or friend of a friend who does.
Well, for now it seems to work again. I was just trying to test Tor, when I noticed that I hadn’t started Tor and I was watching YouTube actually over my VPN.
I do not have experience with GCE, but saying that you don't get support for any Google product is disingenuous.
If you want to play in that world you need to be prepared to handle MTU discovery on your own, or else design your app around deliberately small packet sizes.
That's not to say that this isn't a bug. But let's not start editorializing our titles: the apps for which GCE is "unusable" are buggy apps to start with.
We didn't "pick" UDP. We operate a VoIP related service that interoperates with many different carriers via SIP. Almost all of those carriers ONLY use UDP. SIP UDP packets can often be fairly large. This is especially problematic because GCE uses a non standard MTU size (1460 bytes). This does not make our app, or every other SIP related app that is forced to use UDP, "Buggy".
Rough summary of what I had here: I'm an engineer on GCE (in particular I built our current virtio-net device and a small fraction of the other fiddly bits that sit behind that) -- some details in the bug jumped out at me and I thought there might be a quick fix, but I hadn't processed all of the details and posted a bit prematurely. After further review my original post was essentially content free other than 'IP fragmentation works correctly between internal IPs', which is not germane to the actual customer-reported issue.
When I jumped on HN on my ride home I saw this and was a weeee bit over-eager to be helpful.
(Off topic: has anyone tried grafting a TLS handshake onto ESP?)
Interestingly Linux will actually, by default, throw EMSGSIZE any time you try to send() a UDP datagram that is larger than the detected network MTU to the destination. As I understand it, you have to explicitly turn this behavior off to get fragmentation.
http://man7.org/linux/man-pages/man7/udp.7.html
Isn't fragment reassembly essentially implying all the packets (fragments) will arrive, and will be rearranged in order? i.e. exactly what TCP does?
For example imagine sending a single 1MB packet via UDP, letting it fragment and be reassembled. What distinguishes that from TCP?
No. It is implying that it does what UDP does. TCP is different because it constructs a continuous stream. UDP is inherently packet/message based.
UDP ensures that all of the IP fragments for a given UDP packet get reassembled and rearranged and order. The UDP packets themselves are not arranged in order, and it is possible to see duplicates.
> For example imagine sending a single 1MB packet via UDP
Not possible. You can't send more than a 65,507 byte packet via UDP[1].
Sigh... unfortunately, there are a lot of misconceptions about the basic plumbing of TCP/IP.
[1] UPDATE: as was pointed out in comments, my statement was only true for IPv4. Since IP packets in IPv6 can have jumbograms larger than 65,535 packets, IPv6 UDP does support larger packets. Of course, in that case, UDP doesn't handle the fragmentation/defragmentation protocol itself, and of course GCE won't handle anything IPv6.
https://tools.ietf.org/html/rfc2147
(Good luck with that over the public internet, but it is possible to make udp packets larger than 64K)
IPv6 jumbograms are a whole different kettle of fish, as they are really handled at the IP layer, and are supposed to only be for cases where you have an MTU > 65,575 bytes. As that link describes, the UDP protocol itself doesn't do any fragmentation or reassembly of those packets. RFC 2147 is really just an update to handle the fact that IPv6 can have a single packet that is larger than an IPv4 packet.
"Note: Compute Engine networks only support point-to-point IPv4 traffic. Broadcast and Multicast are not supported."
https://cloud.google.com/compute/docs/networking?hl=en
And this:
https://code.google.com/p/google-compute-engine/issues/detai...
So: Sorry!
It's a thought experiment not a specific number. I'm trying to demarcate some of the differences between TCP and UDP, and UDP reassembling fragments looks a whole lot like TCP.
No handshaking, no blocking, no resending, no throttling.
It's the same as a small UDP packet in every way except for a higher chance of packet loss.
There is only a tiny bit of similarity in that they are both dealing with reassembling a higher layer protocol payload from fragments split up over IP.
Let's count the ways they are different:
1. UDP does provide ordering guarantees within a UDP packet. It provides no ordering guarantees between UDP packets. Consequently, the protocol never has to worry about ordering more than 2^16 bytes of IP data (that makes the sorting algo really simple and efficient).
2. UDP doesn't have to deal with complex stalls or partial delivery of data. Either it has all the IP packets for a given UDP packet (in which case, it immediately delivers the UDP packet), or it doesn't (in which case it doesn't deliver it). If there is a UDP packet is missing a fragment, but a subsequent UDP packet is fully assembled, that one gets delivered.
3. There is no retransmission. UDP doesn't retransmit. It doesn't even guarantee deduplication. It is entirely possible that fragmented packets going different routes can end up creating an echo of multiple copies of a UDP packet being transmitted to the recipient.
The above makes UDP far simpler, and the contract application layers operate under very, very different. If you look at your typical TCP stack's logic for reassembling fragments, you'll find it far, far more complex than for UDP.
Are you sure? If some fragments show up and some not, doesn't it have to wait and see if the rest show up? Maybe it doesn't have to wait for a long time (like TCP), but it does have to wait.
Both point 1 and point 2 are a matter of quantity vs TCP, rather than quality. i.e. UDP+fragments is a sort of TCP-lite.
Much of the stall logic is tied in to whether to wait longer, whether to deliver partial data, whether to request a retransmit, whether you've already transmitted the data (once you reassemble a UDP packet, you can fire it and then promptly forget you ever saw it.. if defragmentation causes you to reconstruct and send the packet again, that's 100% okay!) and most importantly, what to do with an open ended amount of data that may need to be held up while waiting for that one lost fragment.
The logic is simple:
1) Allocate space for UDP packet. 2) Fill in UDP fragment data as it arrives. 3) If all the data for a given packet is there, transmit it and delete all memory of the fragment. 4) If you timeout, delete all memory of the fragment.
That is so, so, so much easier.
If you looked at the code, you'd not think it much similar.
UDP has no feedback from the receiver to the source. If a packet is dropped, the IP stack on the receiver will never complain and the IP stack on the sender will never resend. SO the receiver will silently drop the whole packet.
Notice it can be hard to tell if a packet is dropped, or is reordered. You receive 2,3 - maybe 1 is still coming but reordered? So detecting dropped packets is a heuristic, depending on timing. Various algorithms will use a timer to say when to give up on a packet, or drop a packet if a frame from a subsequent packet is received, or tolerate one frame from a subsequent packet (maybe the last frame of the previous packet was reordered with the first of the next) and so on. But all the work is on the receiver. The sender is not involved.
Unless there is some application-layer protocol that notices packets arriving with a gap, and sends something back (that might also not arrive) like "I got packet a,b,d but missed c". Etc. Notice the application cannot resend frames; only packets because it is unaware of fragmentation/reassembly which happens in the IP stack.
Large UDP packets have a packets success rate of the single packet success rate raised to the number of fragments power, so don't get attached to them in situations with significant packet loss. But this behavior is just an unaddressed bug.
That's not entirely fair. UDP is actually a very convenient transport for packetized data of 65,000 bytes or less.
> You picked UDP because you have latency requirements that cannot be met by TCP.
That is not really a good reason to use UDP, as you won't necessarily get better latency (as has oft been demonstrated).
You should pick UDP because you have a connectionless protocol and/or one with data loss requirements that are different from those built in to TCP.
> that means you need to know what your packets are actually doing, and that includes fragmentation.
UDP is supposed to abstract IP fragmentation issues for you.
> If you want to play in that world you need to be prepared to handle MTU discovery on your own, or else design your app around deliberately small packet sizes.
I've used UDP based protocols all the time. I'd agree that I've had to be a bit more aware of path MTU when it comes to diagnosing or debugging problems, but in terms of the application code reading and writing UDP packets, I've never done MTU discovery.
> But let's not start editorializing our titles: the apps for which GCE is "unusable" are buggy apps to start with.
Considering there are bytes in the UDP header specifically designed to allow you send packets larger than MTU size, this is actually a huge problem. For many people, this requires reimplementing UDP's fragmentation & defragmentation logic up in layer-7.
Isn't it bad enough that people are reimplementing TCP on top of UDP? Do we really need to tell them to reimplement UDP on top of UDP?
Nobody asked for that, just to send slightly larger packets that will probably get through.
> If you wanted simple transport of large data chunks, you would have chosen likewise.
Why? A 4KB data chunk doesn't need TCP more than a 1KB data chunk.
You know the 1500 byte MTU is completely arbitrary and many networks set it higher, right?
> You picked UDP because you have latency requirements that cannot be met by TCP, and that means you need to know what your packets are actually doing, and that includes fragmentation.
Fragmentation itself is a cost of microseconds. The main problem with TCP is head of line blocking, and moderately large UDP packets don't have to deal with that.
You don't need to trace every packet all the way through the network to get low latency and manually handle loss.
> the apps for which GCE is "unusable" are buggy apps to start with
Nothing you listed is a bug.
A key principle of UDP is quite the contrary of what you are saying. UDP applications can have guarantees that they don't have to engage in MTU discovery or fragmentation issues. It provides a way to have an abstract, static contract about packets that is agnostic to layer-2.
Because UDP is comparatively simple, it has been abused as a proxy to implement your own protocol on top of IP, and in that context you of course you really have to deal with al those concerns. However, it is a terrible mistake to think that is what UDP is about or how one should use UDP.
If I was Linus, now would be my cue to unleash a tirade about how you don't fing break userspace. Ever. Protocol layering works because higher level protocols depend on long standing contracts with the lower layers. Google decided to willy nilly go and break the basic contract of UDP across their entire cloud and the top comment here faults protocols built on those guarantees. Disappointing.
SCTP is ideal for this use case but it is not well supported by OSs or networking APIs. TCP works but adds overhead. TFTP works, is UDP-only and has less overhead than TCP, but it does not respond well to packet loss. UDT is like TFTP done right, and is a good solution if you can setup a dependency on its large C++ library.
Last I checked UDT was far more complex than UDP, and since it is layered on top of UDP, I'd think it'd be vulnerable to this problem (although it had all kinds of logic for correctly sizing packets and windows, so maybe it correctly avoids this problem). Either way though, from an application perspective UDT looks much more like TCP than UDP, so I wouldn't think it'd be an obvious choice to replace UDP.
You must never make assumptions about what fits in one packet. The MTU could be 100, or less, or 8000, or more.
As soon as you start doing math based on MTU values that you don't permanently have end-to-end control of yourself, you're setting yourself up for trouble.
The submitted title was "Warning: Google Compute Engine Unusable for Applications Which Rely on UDP", which several commenters have objected to as exaggerated.