Part of the last few weeks involved me learning Rust and using it in anger (if hooking nfqueue up to tokio counts as "in anger") so if you'd like to irritate the hell out of 'pcwalton, feel free to ask me Rust questions.
> Obviously, to do stuff like this, you need to generate certificates. The reasonable way to do that in 2020 is with LetsEncrypt. We do that for our users automatically, but “it just works” makes for a pretty boring writeup, so let’s see how complicated and meandering I can make this.
I'm full-time at Fly. I'll let Jerome answer the fly-proxy question, since it's his code and I wouldn't want to inadvertently take credit.
I think I came across as grumbling about Rust when my real perspective was much more subtle. My take on Rust so far is that it has been, for me, a vindication of a lot of decisions the Go team made, because I've been directly exposed to some of the downsides of the opposite decisions. But, while that sounds like a critique of Rust, it's not! Rust is the way it is for real reasons: zero-cost abstractions and no runtime GC, which are, right now, requirements for some application domains.
For me, right now, writing in Rust feels almost identical to how writing in C++ felt 15 years ago. But I'll keep writing in it, and it'll get faster for me. We're a Rust-on-the-data-plane shop!
"I absolutely understand what y’all like so much about Rust, but I have to say that as an auditor, my blood pressure drops and my shoulders relax the moment I switch from reading a Rust project to reading a Go project."
As long as we're clear that I'm not saying "Rust is less secure than Go", which is not at all what I meant. I just meant that it's much easier for me to read Go code.
(I will however miss match expressions when I return to my home planet.)
I'd be very curious to hear if there are specific bits about the Rust language that you think make it harder to audit or that (so far) it's just the lack of experience.
Fly's proxy uses a mix of tokio, hyper and rustls. We don't need to use a crate that handles ACME because we're processing all the validation and certificate authorizations from a centralized, boring, Rails application.
We've had to submit a PR to the rustls project a few months ago to handle different ALPNs. Instead of resolving a certificate only from a SNI, the crate now provides the full ClientHello which contains negotiable ALPNs. With that information you can respond to the tls-alpn-01 challenge.
Firecracker is f'ing awesome. I have a lot of notes to write up about it. I know this isn't how products actually succeed in the real world, but I'll be honest and say that Kurt had me at Fly with "WireGuard and Firecracker".
(For the unfamiliar reader: Firecracker is a micro-vm system that sits sort of in between a fully virtualized host, like an EC2 instance, and a container like Docker; you get the security isolation of a hypervisor but the speed/simplicity of Docker. It's the engine that powers AWS Lambda and Fargate. The Usenix paper is a pretty great read, and the code [it's all in Rust] is simple and easy to follow.)
Is anyone else feeling quite sad reading this article? ALPN being used because only 80/443 are realistic these days, middleboxes causing the TLS handshake to have padding so it's not misinterpreted with an ancient protocol (SSLv2).
ALPN would make sense for something like HTTP2 even if you didn't have the problem of ports being blocked. If HTTP2 had its own port clients would have to make multiple TCP connection attempts for each host they connect to.
I have the opposite feeling, the clever "hacks" people use to build very useful stuff that bypasses most problems with legacy infrastructure are pretty exciting. It's very much like watching a complex organism evolve into something you never really could have imagined 8000 iterations ago.
Most of this could have been avoided by using DOH and SRV records for HTTP/HTTPS. I still don't understand why SRV records is not supported for HTTP/HTTPS in browsers.
I remember looking into why A/AAAA is still used over SRV, and it would seem performance is one of the big concerns; browsers do not want to make more DNS lookups than necessary.
I think they'd end up with 4 lookups; A, AAAA, SRV (_http2._tls), and SRV (_http._tls).
Though perhaps you are suggesting DoH could mean the resolver also returns SRV records if you request A or AAAA? i.e. proactively point out there's an HTTP server?
IIRC chrome is now racing QUIC and HTTP(2) connections instead of doing negotiation/upgrade to detect QUIC support. So this argument (if true) has fallen apart just a few years later.
Isn't that because QUIC doesn't overload 443/tcp, and sending a UDP packet is extremely cheap? It's not the packets that kill you, it's the round trip latency.
This debate comes up a lot, and it's hilarious how misguided it was.
I regularly work with load-balancers such as Citrix ADC (NetScaler) or F5 BIG IP. These do DNS-based load-balancing, dynamically returning "A" records to that the browsers so that they can get the "single working IP address" they're expecting. The browsers don't try very hard to fail over to secondary IPs because this is the established standard architecture, but they don't need to because of this common setup.
Sounds like an optimal solution, right? It does at first glance anyway, as long as you ignore the eye-watering price tag on those load balancer boxes.
The subtle but critical issue is that by returning "A" records, the load balancers have to use a short time to live (TTL)! This is because there's a trade-off: You can have fast failover, OR long-lived DNS caching. With A records you can't have both!
Typical response TTL times are 5-30 seconds, 5 minutes tops if you hate your users. This means that many browsers will be forced to repeatedly re-query the DNS servers on every page load for typical end-user workflows. It also means that for all but the biggest, most popular sites, the ISP DNS cache does practically nothing for these records.
Meanwhile with SRV records the TTL times can be much higher, hours even. This is how Active Directory works, for example, all of the Domain Controllers add themselves to various SRV records so that if you query "_ldap._tcp.dc._msdcs.test.com" you get back all the DCs. These records include priorities and weightings, so you can pull tricks like incrementally demote a DC or prioritise the shiny new one.
If you watch the AD connection traffic in WireShark, it's incredible. It very quickly steps through alternate services and then reorders the successful hits in front of the failures so that subsequent queries are lightning fast. It is astonishingly tolerant of partial networking failures, yet still fast to connect despite that!
The key mistake made by the original DNS design working groups was that SRV records should have returned a list of IP addresses instead of a list of host names.
Yes, there does seem to be something missing in the semantics of IP lookup. It ought to be a user agent can look up an IP with "keep using until it stops working" sematics. Or "choose one of these 3 IPs, can keep using the one you chose until there is a problem. Or something more similar to how we want client side site availablity logic to work.
We actually run into problems that are similar to what you'd have with SRV records.
Fly.io apps can define different service "handlers" (like TLS and HTTP). If you want to, you can accept TCP connections and bypass our logic. Which is great and flexible.
The problem is, when someone is deploying a new version of their app where they _change_ one of those things, we have to e really careful about how we (a) load balance and (b) decide to do things like TLS. If we're not careful we can end up sending the wrong type of connection to a new VM that's expecting something else.
SRV records sound like they'd have it worse. If you do a DNS lookup to detect something like http2, the IP you connect to _can't_ do anything else. It's much simpler / safer to negotiate stuff like that at connection time.
That all assumes that you use the same port for all those things, right? Which is one of the points of SRV, to not have to squeeze everything into one canonical port. With a SRV record you could route https to whatever port suited you and rotate that out when rolling out changes,
While folks may be thinking “number of queries” must mean it was performance concerns, the answer for Chromium was simpler: SRV records just didn’t resolve for a number of users. Neither do TXT records, for that matter ( https://www.imperialviolet.org/2015/01/17/notdane.html ). Early on in Chrome, it was discovered that attempting to resolve too many domains at once could cause 2Wire routers used by AT&T for DSL to lock up, requiring you to physically reboot, due to a bug in their port of dnsmasq.
Stories like this are stories like those mentioned in the post: everything on the Internet is awful, and you work with what you have and work around things by adding JPEGs of cats and silly session IDs (As TLS 1.3 does) and whatever it takes to “make it work”.
DoH/DoT, if it’s able to be adopted before ISPs and middleboxes ossify it with subpar implementations as part of their effort to avoid net neutrality ( https://www.theverge.com/2019/10/4/20898779/fcc-net-neutrali... ), is the first time in a long time that innovation and experimentation in DNS might happen again. This is why we see things like ESNI and SVCB ( https://tools.ietf.org/html/draft-ietf-dnsop-svcb-httpssvc ) having a chance at being deployed. At least, if clients can get there before the middleboxes ruin the party, like they’ve tried to do for QUIC.
It's probably better to compare Fly with Lambda or Fargate. It's not really meant to be cheaper than AWS, though, the real value is being able to run app servers all over the world without spending time maintaining servers or wrangling AWS.
... and use Caddy to do the heavy lifting. (I'm biased, yes. But the linked doc is multi-authored and applies to every sysadmin or developer who needs to manage certs, regardless of your software choice.)
I also love Caddy. In fact, you can run it on Fly.io (and even opt out of our TLS/cert stack). I would love it if it could just put certs in Vault, though.
It would be interesting to see stats from the CAs about which of the Blessed Methods is most popular. (This article is about Let's Encrypt using tls-alpn-01 which is an implementation of 3.2.2.4.10 "TLS Using a Random Number"). Doubtless Fly aren't the only people doing tls-alpn-01 in bulk but we don't have a good overview as far as I'm aware.
In principle they can all generate those statistics because they (are supposed to) log enough information to identify what went wrong when, inevitably, something is misissued. Logically that also includes at least which method was used to verify domain authorization or control.
One of the things wrong at Symantec is that it turns out some of the records were notionally kept at CrossCert, a separate Korean company. CrossCert simply did not keep any records (or if it did they were in such disarray that it seemed less likely to attract retribution by refusing to disclose them) and Symantec had seemingly never checked.
Knowing which methods are popular with Subscribers, and whether that varies considerably between CAs would be valuable in trying to figure out how more of the worst Blessed Methods can be deprecated or improved, and who we need to be talking to about that.
For example maybe Let's Encrypt is doing almost all the 3.2.2.4.19 ("Agreed Upon Change to Website - ACME") then there's no point ragging on other CAs for the shortcomings of relying on plaintext HTTP in this method. Or maybe DigiCert are doing a lot of 3.2.2.4.15 ("Phone Contact with Domain Contact") so they are the people to talk through any proposed improvements around stuff like leaving a Voice mail.
> We proxy traffic from edge servers to containers through a global WireGuard mesh.
I am more interested in the mesh. Do you have more details on that? Specifically why this architecture was chosen, what kind of latency does WireGuard add, etc.
Ooooh I love Wireguard, we'll have an article about this in the next couple of months.
We picked it because it's really simple to manage, and we wanted to ensure traffic between datacenters was always encrypted. We have a little tool called "flywire" that keeps wireguard peer configs updated from Consul. Once we accept a connection from a user, we pick a target VM, and then connect them over the wireguard mesh.
For our purposes, it basically doesn't add any noticeable latency. I think when we tested we say something on the order of 0.1ms of added latency over wireguard, but I don't quite remember. It's never been the source of latency problems when we do have them, at least!
Question about fly.io: do you support HTTP/2? I have wanted to put gRPC services directly on the edge but most managed services make it completely convoluted to set up (the lone exception being Google Cloud Run).
46 comments
[ 3.8 ms ] story [ 110 ms ] threadThis delighted me.
How is the Fly proxy implemented? Are you using rustls and/or any of the available ACME crates?
I've been wanting to implement tls-alpn-01 support for rustls (although it might be possible to do this just by mutating the ServerConfig over time).
Also interested to hear your general impressions of Rust so far (I think I read some Twitter grumbling...).
I think I came across as grumbling about Rust when my real perspective was much more subtle. My take on Rust so far is that it has been, for me, a vindication of a lot of decisions the Go team made, because I've been directly exposed to some of the downsides of the opposite decisions. But, while that sounds like a critique of Rust, it's not! Rust is the way it is for real reasons: zero-cost abstractions and no runtime GC, which are, right now, requirements for some application domains.
For me, right now, writing in Rust feels almost identical to how writing in C++ felt 15 years ago. But I'll keep writing in it, and it'll get faster for me. We're a Rust-on-the-data-plane shop!
"I absolutely understand what y’all like so much about Rust, but I have to say that as an auditor, my blood pressure drops and my shoulders relax the moment I switch from reading a Rust project to reading a Go project."
https://twitter.com/tqbf/status/1260678152084480008
(I will however miss match expressions when I return to my home planet.)
Fly's proxy uses a mix of tokio, hyper and rustls. We don't need to use a crate that handles ACME because we're processing all the validation and certificate authorizations from a centralized, boring, Rails application.
We've had to submit a PR to the rustls project a few months ago to handle different ALPNs. Instead of resolving a certificate only from a SNI, the crate now provides the full ClientHello which contains negotiable ALPNs. With that information you can respond to the tls-alpn-01 challenge.
(For the unfamiliar reader: Firecracker is a micro-vm system that sits sort of in between a fully virtualized host, like an EC2 instance, and a container like Docker; you get the security isolation of a hypervisor but the speed/simplicity of Docker. It's the engine that powers AWS Lambda and Fargate. The Usenix paper is a pretty great read, and the code [it's all in Rust] is simple and easy to follow.)
https://www.usenix.org/system/files/nsdi20-paper-agache.pdf
Disclaimer: interned with the team
It feels like the Internet is so fragile.
I think they'd end up with 4 lookups; A, AAAA, SRV (_http2._tls), and SRV (_http._tls).
Though perhaps you are suggesting DoH could mean the resolver also returns SRV records if you request A or AAAA? i.e. proactively point out there's an HTTP server?
I regularly work with load-balancers such as Citrix ADC (NetScaler) or F5 BIG IP. These do DNS-based load-balancing, dynamically returning "A" records to that the browsers so that they can get the "single working IP address" they're expecting. The browsers don't try very hard to fail over to secondary IPs because this is the established standard architecture, but they don't need to because of this common setup.
Sounds like an optimal solution, right? It does at first glance anyway, as long as you ignore the eye-watering price tag on those load balancer boxes.
The subtle but critical issue is that by returning "A" records, the load balancers have to use a short time to live (TTL)! This is because there's a trade-off: You can have fast failover, OR long-lived DNS caching. With A records you can't have both!
Typical response TTL times are 5-30 seconds, 5 minutes tops if you hate your users. This means that many browsers will be forced to repeatedly re-query the DNS servers on every page load for typical end-user workflows. It also means that for all but the biggest, most popular sites, the ISP DNS cache does practically nothing for these records.
Meanwhile with SRV records the TTL times can be much higher, hours even. This is how Active Directory works, for example, all of the Domain Controllers add themselves to various SRV records so that if you query "_ldap._tcp.dc._msdcs.test.com" you get back all the DCs. These records include priorities and weightings, so you can pull tricks like incrementally demote a DC or prioritise the shiny new one.
If you watch the AD connection traffic in WireShark, it's incredible. It very quickly steps through alternate services and then reorders the successful hits in front of the failures so that subsequent queries are lightning fast. It is astonishingly tolerant of partial networking failures, yet still fast to connect despite that!
The key mistake made by the original DNS design working groups was that SRV records should have returned a list of IP addresses instead of a list of host names.
Fly.io apps can define different service "handlers" (like TLS and HTTP). If you want to, you can accept TCP connections and bypass our logic. Which is great and flexible.
The problem is, when someone is deploying a new version of their app where they _change_ one of those things, we have to e really careful about how we (a) load balance and (b) decide to do things like TLS. If we're not careful we can end up sending the wrong type of connection to a new VM that's expecting something else.
SRV records sound like they'd have it worse. If you do a DNS lookup to detect something like http2, the IP you connect to _can't_ do anything else. It's much simpler / safer to negotiate stuff like that at connection time.
Stories like this are stories like those mentioned in the post: everything on the Internet is awful, and you work with what you have and work around things by adding JPEGs of cats and silly session IDs (As TLS 1.3 does) and whatever it takes to “make it work”.
DoH/DoT, if it’s able to be adopted before ISPs and middleboxes ossify it with subpar implementations as part of their effort to avoid net neutrality ( https://www.theverge.com/2019/10/4/20898779/fcc-net-neutrali... ), is the first time in a long time that innovation and experimentation in DNS might happen again. This is why we see things like ESNI and SVCB ( https://tools.ietf.org/html/draft-ietf-dnsop-svcb-httpssvc ) having a chance at being deployed. At least, if clients can get there before the middleboxes ruin the party, like they’ve tried to do for QUIC.
micro-2x shared 512MB $0.000003044 $8 VS t3a.nano 2 Variable 0.5 GiB EBS Only $0.0031 per Hour
I'm missing something? cause seeing the pricing I still feel AWS is cheaper.
... and use Caddy to do the heavy lifting. (I'm biased, yes. But the linked doc is multi-authored and applies to every sysadmin or developer who needs to manage certs, regardless of your software choice.)
I also love Caddy. In fact, you can run it on Fly.io (and even opt out of our TLS/cert stack). I would love it if it could just put certs in Vault, though.
In principle they can all generate those statistics because they (are supposed to) log enough information to identify what went wrong when, inevitably, something is misissued. Logically that also includes at least which method was used to verify domain authorization or control.
One of the things wrong at Symantec is that it turns out some of the records were notionally kept at CrossCert, a separate Korean company. CrossCert simply did not keep any records (or if it did they were in such disarray that it seemed less likely to attract retribution by refusing to disclose them) and Symantec had seemingly never checked.
Knowing which methods are popular with Subscribers, and whether that varies considerably between CAs would be valuable in trying to figure out how more of the worst Blessed Methods can be deprecated or improved, and who we need to be talking to about that.
For example maybe Let's Encrypt is doing almost all the 3.2.2.4.19 ("Agreed Upon Change to Website - ACME") then there's no point ragging on other CAs for the shortcomings of relying on plaintext HTTP in this method. Or maybe DigiCert are doing a lot of 3.2.2.4.15 ("Phone Contact with Domain Contact") so they are the people to talk through any proposed improvements around stuff like leaving a Voice mail.
I am more interested in the mesh. Do you have more details on that? Specifically why this architecture was chosen, what kind of latency does WireGuard add, etc.
We picked it because it's really simple to manage, and we wanted to ensure traffic between datacenters was always encrypted. We have a little tool called "flywire" that keeps wireguard peer configs updated from Consul. Once we accept a connection from a user, we pick a target VM, and then connect them over the wireguard mesh.
For our purposes, it basically doesn't add any noticeable latency. I think when we tested we say something on the order of 0.1ms of added latency over wireguard, but I don't quite remember. It's never been the source of latency problems when we do have them, at least!
P.S: I’ve built a service using Fly and can’t recommend it enough!
https://fly.io/docs/app-guides/run-a-private-dns-over-https-...
(Beyond that, for whatever it's worth: you can skip our HTTP/H2 termination entirely and speak TCP directly to your VMs).