No, there isn’t. The specification as implemented requires no invalidation mechanism, which means no such mechanism across all caches exists, nor will it ever. The long tail kills you in such a failure scenario, and remember, people who make kitchen appliances write DNS resolvers.
Yeah, this was my first thought...I am guessing the author has never accidentally pushed out a bad DNS entry and needed to revert/update.
Everyone probably starts with higher TTLs, then the first time they mess up an update they switch to a short one.
The author seems to not appreciate how big a problem a misconfigured, long TTL DNS entry could be for someone.
Other reasons for short TTLs... maybe an IP gets blocked/flagged by a large network and they need to change fast... or a network path is slow and they need to move to a new location.
It would have been interested to see actual delay rather than qualitative results of the nature "<x>% wasn't in cache so this is horrible!". Admins and users don't care if it's in cache, they care what the impact to operations and load time is. https://www.dnsperf.com/dns-speed-benchmark says lookup times for my personal domain results in 20ms-40ms. Ironically the same dns test for 00f.net is taking 100ms-150ms.
99% of apps will gladly trade a 30ms increase in session start (assuming the browser's prefetcher hasn't already beaten them to it) to not have to worry about things taking an hour to change. Not all efficiency is about how technically slick something is.
I just tested 00f.net and got as low numbers as 6ms. Latency is a question about network traffic between the client and the server, and unless you use anycast you will get different latency depending on what place in the world the client and server reside in, and if you use anycast it depend on how good the contracts and spread the anycast network has.
Very true, looks to be hosted out of Europe. The point about 00f.net optimizing for ease of operation vs optimizing milliseconds of performance only holds doubly true with this information though.
Should DNS providers have a setting that increase TTLs over time automatically? I.e the longer I leave my dns entry pointing to the same IP the longer my TTL is?
Obviously it would be possible to opt out for situations where you genuinely need a low TTL on a domain.
What DNs providers should do is to transition to traffic-based billing instead, if TTL is such an issue for them. There’s a variety of useful use-cases handled with low TTLs.
This is definitely a feature I've also thought would be useful to have in DNS providers.
I've worked on managing thousands of (sub)domains and the administrative overhead of changing the TTLs for everything manually would be considerable. I'd certainly like an automated way to say "These records should gradually increase TTL up to <X> time over <Y> time" (e.g., gradually raise TTL to 2 days over 2 weeks if there are no changes).
There are downsides to high TTLs though: (1) you need to remember to preemptively lower them ahead of any planned changes (if you want those changes to take effect quickly), and (2) you can't change the records quickly in an emergency. But, fortunately, lots of record types are ones that you probably don't need to change in an emergency -- and for ones that you do, you can use a low TTL.
Anyway, I'd personally like to see automated TTL management as a feature in DNS software.
I've seen some that have a somewhat reasonable minimum time. You can go below it but it will reset to their minimum after a day or two.
But it's a risky play for providers. It reduces their DNS load (which is pretty cheap to handle), but increases the risk that a customer will come yelling why they couldn't fix their outage quickly because the algorithm increased their TTL to something large.
Maybe up to a point, but really the TTL should be set for how long is acceptable for traffic to continue to flow to the old destination after a change. That's not necessarily correlated with time between changes: just because a service IP hasn't changed for two years doesn't mean I would want to wait a day for most traffic to move.
Of course, the reality is some traffic will continue to flow to the old destination for as long as you care to measure. There's plenty of absurdly broken DNS caching out there.
This is very common (Dynect, NS1, AWS, GCP, etc all depend on this for monitoring and failover). The author is an incorrect.
amazon.com, reddit.com, facebook.com, and others use low TTLs on their domains for this reason. Anyone who can't maintain an anycast infrastructure around the world and doesn't want to depend on Cloudflare will use this method.
For literally my entire career in SRE, well over a decade now, I’ve only interacted with systems that use DNS for this purpose, from small shops to parts of every HN reader’s life. That sentence in your quote, as well as the assertive nature of the post on such a weak foundation, are sufficient to disqualify a hiring candidate on account of lack of experience despite the authored software presented. It simply does not align with reality when presented with two logically separate networks and a required mechanism to transition between them.
The only other alternative for that scenario is using anycast addressing, and that has a colorful bag of limitations that are quite different from those of low-TTL DNS (including being out of reach for most).
You aren't idiots if you're using it where there are no better alternatives - it's preferable to use load balancers etc where available, but there are places where it's very much "DNS or nothing".
How would I use a load balancer to fail traffic between, say, London and Amsterdam with no fiber in place between them? Where would the load balancer physically exist in that scenario and how would it fail to the other when power is lost in one location? Would I make a third PoP to isolate it? What would then be my redundancy story for that PoP? How would I relocate traffic to my backup load balancer PoP number four?
Within a single network, sure, load balance all you want. That’s not the scenario low TTLs go after.
> How would I use a load balancer to fail traffic between, say, London and Amsterdam with no fiber in place between them?
What people use in those sitations is Anycast.
Of course, DNS itself, or e-mail, don’t need this kind of redundancy, since the NS (or MX) records themselves provides a list of failover servers. The corresponding alternative for HTTP, SRV records, has been consistently stonewalled by standard writers for HTTP/2, QUIC, etc.
There is an interesting draft RFC which I am keeping an eye on, but I don’t want to get my hopes up:
This requires that you blow a publicly-advertisable prefix for every unique combination of services you would want to fail-over.
E.g., if you wanted to be able to have independent fail-over between your customer-facing self-service portal and your webmail interface (each relying on specific state that you can't replicate synchronously, and can't guarantee replicate consistently with each other), you would need to /24s, one dedicated to anycast for the webmail interface, one for the self-service portal, and separate from any services which are active-active.
Whereas using DNS, you could use your other existing public /24s that you are already using for your active-active services.
In the last days of IPv4, an extra 2 /24s just for this is quite an expense.
Within a region, Anycast is how many big companies move things around mostly seamlessly. Why inside a region? If RIPE or ARIN catch you advertising their IP in the other's territory, they will send nasty emails and threaten to take back your CIDR blocks. I have no idea if they will follow through as we always stopped violating their rules when told to.
Outside of a region, they use DNS and sometimes a combination of WAN accelerators and VPN's, dark fiber.
DNS failover is used extensively, especially in the cloud world.
I see nothing wrong with using low DNS TTLs for failover - really don't understand the author's objections here, and them claiming that "DNS is not used for failover any more" significantly discredits them, IMO.
> I mean, my company does this for certain failure scenarios involving our CDNs. Can anyone tell me why we're idiots, or is this just hyperbole?
I came here to say exactly that. Our company uses DNS entries with low TTL for failover and load balancing purposes as well -- it's a very common approach. Services like AWS Route 53 and CloudFlare make it very easy to setup and low cost. I was surprised that the author didn't give much acknowledgement to this type of usage.
I operate authoritative name servers for almost 10.000 domains. Originally, I used a default TTL of 2 days, as recommended by RIPE-203¹ (which is also compatible with the recommendations of RFC 1912²), but this was not accepted by users, who didn’t want to wait two days. Therefore, for all records except SOA and NS records, I changed the default TTL to one hour, which I still use as the default value unless a change is scheduled and/or planned, in which case I lower it to 5 minutes. I do not want to lower it any more, as I’ve heard rumors of buggy resolvers interpreting “too low” TTLs as bad, and reverting to some very-high default TTL, and thereby wrecking my carefully planned DNS changeover. I have, however, not seen any real numbers or good references on what numbers are “too low”, and would like to hear from anyone who might have some information on this.
I am hesitant to say; we only target the local area, and our home page isn’t even available in English. Our main role is as a domain name registrar, also providing, in increasingly tangential order, domain name strategy planning, some trade mark strategy, DNS hosting, HTTP redirects, E-mail, and web hosting. Our main value proposition is support; call us and talk to us directly, or send an e-mail, and get an answer more or less immediately. We only very reluctantly provide self-service control panels, and we don’t mention its availability unless people directly ask for it, and we generally discourage its use, preferring that people simply tell us what they want done in their DNS. Some people, including some very large companies, prefer this arrangement, and if you are one of them, and you are part of our local market, I’m sure you’ll be able to find us.
Unless you have insight into the end users DNS deployments I would say this is the appropriate amount of caution to apply. Besides just TTL being low, a frequent issue I had when first migrating to AWS years ago was CNAME to CNAME records not resolving among some end users. Primary schools were the worst offenders, I assume some of them still have Novell deployed.
There was a dns record looked up primarily by large supercomputers that had a 0 ttl. It was used for stats via a UDP packet (because it was non blocking, nevermind that the dns query was blocking). This was set to 0 for "failover" but it hadn't changed in years. I worked out that our systems alone had caused billions of queries for this name.
After I complained I think they upped the ttl.. to 60.
Reminds me a of a server pair at the last healthcare place I worked. Between the two of them they'd generate something around 1200 DNS lookups per second (about 60% of the load on the DNS servers) of their own name. I think the logic was if the name stopped responding then server A was primary. If the name was responding the server that owned the IP it was responding for was primary. If the servers wanted to swap primary/secondary they would issue a DDNS request.
After about 8 years we were were restructuring our DNS infrastructure for performance and I rate limited those two to 10 or so queries per second each. In that time there must have been 300 billion or so requests from those two boxes alone.
In my experience that sort of thing is from the local hostname not being present in /etc/hosts and (of course) a caching resolver not in use.
Some process on the system wants to connect to itself, which then causes a dns lookup. Add a high transaction rate on top of that, and 1,200/second is easy.
The funniest one I remember seeing was thousands and thousands of lookups for HTACCESS. Turned out apache was running on top of a web root stored in AFS and not configured to stop looking for .htaccess files at the project root, so it would try to open
> Of course, a service can switch to a new cloud provider, a new server, a new network, requiring clients to use up-to-date DNS records. And having reasonably low TTLs helps make the transition friction-free. However, no one moving to a new infrastructure is going to expect clients to use the new DNS records within 1 minute, 5 minutes or 15 minutes. Setting a minimum TTL of 40 minutes instead of 5 minutes is not going to prevent users from accessing the service.
Note that you can still get the benefit of a low TTL during a planned switch to a new cloud provider, server, or network even if you run with a high TTL normally. You just have to lower it as you approach the switch.
For example, let's say you normally run with a TTL of 24 hours. 25 hours before you are going to throw the switch on the provider change, change the TTL to 1 hour. 61 minutes before the switch, change TTL to 1 minute.
Wouldn't you be canarying your switch over a period of longer than 24 hours anyway?
I can still imagine a benefit to short TTLs in the sense that you can maybe roll out your canary in a more controlled way. But that's a lot more complicated than the issue of quick switching.
If your cloud provider does an oopsie (e.g. https://news.ycombinator.com/item?id=20064169) and takes down your entire infrastructure, or you have to move quickly for some other reason, or you're recovering from a misconfiguration, the long TTL can add 24 hours to your mitigation time.
If you're just playing around with your personal project/web site, you just added a giant round of whack-a-cache to your "let's finally clean up my personal server mess" evening.
As most who's ever worked with web hosting can confirm, small business customers often have no idea of what they're doing, and I've talked to many people who switched providers after seeing an ad for cheap hosting, without realising that they have to a) wait for the DNS changes to propagate, b) that they have to actually move their web site from one provider to another.
Subsequently, my previous employer lowered the default TTL simply because it got rid of all the bad Trustpilot ratings about customers being "prevented from leaving", and started offering a "move my WordPress site for me" service to profit from all the panicking new-comers who had no idea about how to do trivial things like importing/exporting a database and transferring files.
I'm wondering what it would take to make a DNS caching service with updates based on reliable notifications rather than polling? After all, every cellphone does it.
This already exists (NOTIFY), but it's only used for master-slave setups (ie. a bunch of DNS servers serving some authoritative zone who want changes to be transmitted to all slaves ASAP)
It would be interesting to (ab)use this mechanism in the way you suggest. A recursive DNS server could ask to be NOTIFY'ed of changes in the zone they are querying...it would, of course, add load to the server, and it would need strict limits to avoid DoS, but it seems an interesting idea.
The big problem, to the extent there is one, is between the client and the recursive server. Not as much the recursive and the authoritative. Cost is highly amortized between recursive and authoritative for busy names.
"The urban legend that DNS-based load balancing depends on TTLs (it doesn’t)"
So whats the solution? We are using AWS ALB/ELB and it states in docs, that we should have low TTL, and it makes sense. Servers behind LB scale up and down. What is the option B?
In fact, if you use Route 53 with an alias to an ELB, the TTL is hard-coded at 60s -- it is not even configurable. If it were, we'd follow the practice of lowering it prior to changes, and raising again once things or stable, but as it is, that's not an option (moving DNS off AWS would be a hard sell, not cause it's terribly hard but afamic, there's not really any value to doing it).
The author says low TTLs are bad because of latency but never attempts to quantify how much latency we are actually talking about. Its hard to know how outraged I'm supposed to be without actually seeing the numbers.
It seems that a lot of sites are ok with slightly higher latencies if it means greater operational flexibility.
Latency is dependent on many things. DNS server location - accessing my website from australia will take 500ms for DNS lookup (or twice as much if I'm using cnames). If this is not cached somewhere, that's 500ms every few seconds with those <1minute TTLs. If I'm on GPRS, or similar, that will add a bunch more hundreds of ms to every useless DNS resolution, incl. unpredictable variability.
I’m honestly not sure what this author is complaining about. If the infrastructure can handle it and the zone owner is willing to pay for the excessive traffic, and DNS cache operators are fine with it, then this seems like a call for premature optimization.
I would maintain that if you are experiencing poor performance for a web site, there are MUCH more fruitful places to look than DNS latency. Third party objects, excessive page sizes, lack of overall optimization based on device are just the tip of the iceberg.
For many apps I've worked the DB connection setup was always the slow part (use PgBouncer). Then, the part was the queries. DNS, gziped CSS/JS - chasing a red-herring.
Just anecdotally, from running PiHole, looking at the logs, I have some sites being resolved 12K times over 11 days... That's over a thousands requests a day.
Probably ad or analytic sites? I know one app in particular where every other update seems to result in it sending a request per second to some blocked analytics site.
Uh? I always used 24 hours TTL for DNS. I reluctantly move it to 1 hour for some tests, then quickly set it back to 24 hours. What are these people thinking?
One use case where short TTL-s make sense is running a service on a residential network where a power outage or router reboot can trigger an IP address change. If the IP address changes then you won't be offline for too long.
Yes, it is not exactly great, but at least it works well enough for self-hosting services.
The author seem to be missing one of the big reasons ridiculously low TTLs are used: it lets passive eavesdroppers discover a good approximation of your browsing history. Passive logging of HTTP has (fortunately) been hindered as most traffic moved to HTTPS, but DNS is still plaintext.
Low TTLs mean a new DNS request happens (apx) every time someone clicks a link. Seeing which domain names someone is interacting with every 60s (or less!) is enough to build a very detailed pattern-of-life[1]. Remember, it's probably not just one domain name per click; the set of domain names that are requested to fetch the js/css/images/etc for each page can easily fingerprint specific activities within a domain.
Yes, TTLs need to have some kind of saner minimum. Even more important is moving to an encrypted protocol. Unfortunately DOH doesn't solve this problem[2]; it just moves the passive eavesdropping problem to a different upstream server (e.g. Cloudflare). The real solution is an encrypted protocol that allows everyone to do the recursive resolution locally[3].
HTTPS does not hinder that type of tracking.. in fact, using SNI (which is unencrypted) would be more accurate than trying to do it with DNS... since it's sent with every request.
> The author seem to be missing one of the big reasons ridiculously low TTLs are used: it lets passive eavesdroppers discover a good approximation of your browsing history.
I operate DNS for hundreds of thousands of domains. I've tried to reassemble browsing history from DNS logs, and I can tell you it is damn near impossible. You have DNS caches in the browser, the OS, broadband routers, and ISPs/public resolvers to account for - and half of them don't respect TTLs anyways.
The reason people set low TTLs is they don't want to wait around for things to expire when they want to make a change. DNS operators encourage low TTLs because it appears broken to the user when they make a change and "it doesn't work" for anywhere from a few hours to a few days.
The problem is that your ISP can log and mine your DNS requests, regardless of the servers you use. They definitely do this and one can only assume they then sell it after some sort of processing.
The comment you're replying specifies caching at the browser, OS, and router level. Not one of three would show up as DNS refreshes with the ISP because the DNS is not being refreshed.
I can't tell. I run Firefox at home, and set up my own DoH server (mainly because I saw the writing on the wall and and if Mozilla/Google are going to shove this down my throat, I want it shoved down on my terms, but I digress). If I visit my blog (which has a DNS TTL of 86,400) I get a query for my domain not only on every request, but even if I just hover over the link. It will also do a query when I click on a link to news.ycombinator.com (with a TTL of 300) but not when I hover over a link. It's bizarre.
Mostly, yes. In my experience (as a service provider) Chrome has a bad habit of caching records occasionally for much much longer than it should. Maybe bug maybe intentional, I dunno.
I seem to remember a paper a few years ago that (IIRC) tested this by setting a very low TTL (like 60), changing the value, and seeing how long they continued to receive requests at the old value... and most updated within the TTL, but there were some that took up to (I want to say) an hour. I'm probably getting bits of this wrong though..
The violations in that paper that are important are those that have increased the TTL. Reducing the TTL increases costs for the DNS provider, but isn't important here. The slowest update was about 2 hours (with the TTL set to 333).
Of those that violated the TTL, we don't know what portion of those would function correctly with a different TTL (increasing the TTL indicates they're already not following spec). So I wouldn't assume that increasing the TTL would get them to abide by your requested TTL. They're following their own rules, and those could by anything.
Considering how common low TTLs are... you're worrying about a DNS server that's already potentially causing errors for major well known websites.
It is important to note that this study used active probes asking selected recursive resolvers around the world.
From my own experience when changing records and seeing when the long tail of clients stops calling the old addresses (with the name), it is a really long tail. An extreme example that lasted almost six months was a web spider that just refused to update their DNS records and continued to request websites using the old addresses.
Is there a lot of custom written code that does their own DNS caching? Yes. One other example is internal DNS servers that shadow external DNS. There is a lot of very old DNS software running year after year. Occasionally at work we stumble onto servers which are very clearly handwritten by someone a few decades ago by people with only a vague idea of what the RFCs actually say. Those are not public resolvers of major ISPs, so the above study would not catch them.
Naturally if you have a public resolver where people are constantly accessing common sites with low TTLs then issues would crop up quickly and the support cost would get them to fix the resolver. If it's an internal resolver inside a company where non-work sites are blocked then you might not notice until the company moves to a new web hosting solution and suddenly all employees can't access the new site, an hour later they call the public DNS hosting provider, the provider diagnoses the issue to be internal of the customer's network, and then finally several hours later the faulty resolver gets fixed.
> An extreme example that lasted almost six months was a web spider that just refused to update their DNS records and continued to request websites using the old addresses.
Could it be that it wasn't using DNS at all? Just hardcoded the ip address?
> An extreme example that lasted almost six months was a web spider that just refused to update their DNS records and continued to request websites using the old addresses.
It may have been Java client that was not restarted. At least for older versions of Java the default was to cache result forever.
Yep, older java versions had some ridiculous caching of both positive and negative DNS responses. That was some weird problem to troubleshoot. We ended up writing our own caching then, back in Java7ish. And the first version of our DNS caching was broken and promptly triggered load alerts on 2 DNS servers of our operations team by issuing ... a lot of DNS queries very very quickly :)
I run authoritative DNS for a very busy domain - 30B queries per month. Originally we had 6 hour TTLs, but now I use 60s. We have had no problems. Uptime and fast failover comes before anything else.
> Here’s another example of a low-TTL-CNAME+low-TTL-records situation, featuring a very popular name:
> $ drill detectportal.firefox.com @1.1.1.1
Is captive portal detection not a valid use case for low TTL? The entire point is to detect DNS hijacking of a known domain, which takes longer when you cache the DNS results...
Captive portal detection involves more than just checking for DNS hijacking. The browser tries to load http://detectportal.firefox.com/success.txt and acts based on how that goes. Having a short TTL does not help.
If your captive portal is implemented by intercepting your DNS queries, then having a short TTL should ensure that the captive portal actually has a query to intercept.
But sure, there are other implementation approaches (e.g. injecting HTTP redirects), which I imagine is one reason why Firefox doesn't literally inspect the DNS reply.
If you're talking about keep-alive this times out far before most of the DNS TTLs mentioned in this article and don't persist after a connection is closed anyways.
Anycast addresses change all the time. Ask Google, Microsoft, Amazon, Akamai, Cloudlfare and so on if you don't believe me. About the only anycast IPs that don't change are public DNS resolvers but that's also true of unicast resolvers as well.
By that logic BGP is a layer 7 load balancer since it has an application layer. BGP only exchanges layer 3 reachability information to update route tables therefore you can only load balance layer 3 with it.
Personal attacks and other things in your comments are against the HN guidelines. The goal is to talk about DNS/TTLs and their impact on performance not insult each other. https://news.ycombinator.com/newsguidelines.html
This is true in that the range encapsulates most keep-alive timeouts not in that keepalives longer than a minute or two are actually the majority. nginx defaults to 100 seconds, Apache is less than that. Most don't mess with these let alone bump them to 900. Generally 60 to 120 is considered standard with some cap on the number of active keep-alive sessions as well. Some go ultra-low or disable it all together, very few go ultra-high.
> WTF is your timeout?
120.
Also please try to keep the conversations together.
You're argument about keep alive makes no sense. You're confusing the nginx documentation. 100 is the number of default connections it will hold open. 60 is the default timeout. Also it's sent as an HTTP response to the client in the headers...
Meant "TTLs" as in the plural of TTL but my phone capitalized the whole block on me.
> You're argument about keep alive makes no sense. You're confusing the nginx documentation. 100 is the number of default connections it will hold open. 60 is the default timeout. Also it's sent as an HTTP response to the client in the headers... Here you need to read up: http://nginx.org/en/docs/http/ngx_http_core_module.html#keep...
Are you're right about the 100 being the default active keepalives not the default timeout. According to your own nginx link 75 is the timeout not 60 though: "Default:
keepalive_timeout 75s;"
Either way, 75/60/100/120 are significantly far off from 15 minutes.
> The timeout only matters if you're not making requests.
Or if the server reaches max connections.
> Setting a low keep alive will actually result in more DNS requests...doh
Which is how the discussion on DNS TTLs comes about in the first place. It's trivial to set the DNS TTL astronomically higher than the HTTP keep-alive, in which case the browser & OS won't actually make a lookup request since it's cached.
151 comments
[ 371 ms ] story [ 3499 ms ] threadOr is there a better way to quickly invalidate DNS caches in case of emergency?
Everyone probably starts with higher TTLs, then the first time they mess up an update they switch to a short one.
The author seems to not appreciate how big a problem a misconfigured, long TTL DNS entry could be for someone.
Other reasons for short TTLs... maybe an IP gets blocked/flagged by a large network and they need to change fast... or a network path is slow and they need to move to a new location.
Not sure about any others.
[0]: https://developers.google.com/speed/public-dns/cache
Edit: https://outline.com/CrL5gf
https://web.archive.org/web/20191103211525/https://00f.net/2...
99% of apps will gladly trade a 30ms increase in session start (assuming the browser's prefetcher hasn't already beaten them to it) to not have to worry about things taking an hour to change. Not all efficiency is about how technically slick something is.
Obviously it would be possible to opt out for situations where you genuinely need a low TTL on a domain.
I bet this is one of the main reasons for low TTLs.
I've worked on managing thousands of (sub)domains and the administrative overhead of changing the TTLs for everything manually would be considerable. I'd certainly like an automated way to say "These records should gradually increase TTL up to <X> time over <Y> time" (e.g., gradually raise TTL to 2 days over 2 weeks if there are no changes).
There are downsides to high TTLs though: (1) you need to remember to preemptively lower them ahead of any planned changes (if you want those changes to take effect quickly), and (2) you can't change the records quickly in an emergency. But, fortunately, lots of record types are ones that you probably don't need to change in an emergency -- and for ones that you do, you can use a low TTL.
Anyway, I'd personally like to see automated TTL management as a feature in DNS software.
But it's a risky play for providers. It reduces their DNS load (which is pretty cheap to handle), but increases the risk that a customer will come yelling why they couldn't fix their outage quickly because the algorithm increased their TTL to something large.
Of course, the reality is some traffic will continue to flow to the old destination for as long as you care to measure. There's plenty of absurdly broken DNS caching out there.
https://tools.ietf.org/html/rfc1996
https://tools.ietf.org/html/rfc1995
I mean, my company does this for certain failure scenarios involving our CDNs. Can anyone tell me why we're idiots, or is this just hyperbole?
amazon.com, reddit.com, facebook.com, and others use low TTLs on their domains for this reason. Anyone who can't maintain an anycast infrastructure around the world and doesn't want to depend on Cloudflare will use this method.
The only other alternative for that scenario is using anycast addressing, and that has a colorful bag of limitations that are quite different from those of low-TTL DNS (including being out of reach for most).
Within a single network, sure, load balance all you want. That’s not the scenario low TTLs go after.
What people use in those sitations is Anycast.
Of course, DNS itself, or e-mail, don’t need this kind of redundancy, since the NS (or MX) records themselves provides a list of failover servers. The corresponding alternative for HTTP, SRV records, has been consistently stonewalled by standard writers for HTTP/2, QUIC, etc.
There is an interesting draft RFC which I am keeping an eye on, but I don’t want to get my hopes up:
https://tools.ietf.org/html/draft-nygren-dnsop-svcb-httpssvc...
This requires that you blow a publicly-advertisable prefix for every unique combination of services you would want to fail-over.
E.g., if you wanted to be able to have independent fail-over between your customer-facing self-service portal and your webmail interface (each relying on specific state that you can't replicate synchronously, and can't guarantee replicate consistently with each other), you would need to /24s, one dedicated to anycast for the webmail interface, one for the self-service portal, and separate from any services which are active-active.
Whereas using DNS, you could use your other existing public /24s that you are already using for your active-active services.
In the last days of IPv4, an extra 2 /24s just for this is quite an expense.
Outside of a region, they use DNS and sometimes a combination of WAN accelerators and VPN's, dark fiber.
I see nothing wrong with using low DNS TTLs for failover - really don't understand the author's objections here, and them claiming that "DNS is not used for failover any more" significantly discredits them, IMO.
I came here to say exactly that. Our company uses DNS entries with low TTL for failover and load balancing purposes as well -- it's a very common approach. Services like AWS Route 53 and CloudFlare make it very easy to setup and low cost. I was surprised that the author didn't give much acknowledgement to this type of usage.
1. https://www.ripe.net/publications/docs/ripe-203
2. https://tools.ietf.org/html/rfc1912#page-4
After I complained I think they upped the ttl.. to 60.
After about 8 years we were were restructuring our DNS infrastructure for performance and I rate limited those two to 10 or so queries per second each. In that time there must have been 300 billion or so requests from those two boxes alone.
Some process on the system wants to connect to itself, which then causes a dns lookup. Add a high transaction rate on top of that, and 1,200/second is easy.
The funniest one I remember seeing was thousands and thousands of lookups for HTACCESS. Turned out apache was running on top of a web root stored in AFS and not configured to stop looking for .htaccess files at the project root, so it would try to open
When it hit /afs/.htaccess then it would try to contact the AFS sell called .htaccess, which would do a bunch of dns lookups.This would be done something like twice for every incoming web request.
Note that you can still get the benefit of a low TTL during a planned switch to a new cloud provider, server, or network even if you run with a high TTL normally. You just have to lower it as you approach the switch.
For example, let's say you normally run with a TTL of 24 hours. 25 hours before you are going to throw the switch on the provider change, change the TTL to 1 hour. 61 minutes before the switch, change TTL to 1 minute.
I can still imagine a benefit to short TTLs in the sense that you can maybe roll out your canary in a more controlled way. But that's a lot more complicated than the issue of quick switching.
If your cloud provider does an oopsie (e.g. https://news.ycombinator.com/item?id=20064169) and takes down your entire infrastructure, or you have to move quickly for some other reason, or you're recovering from a misconfiguration, the long TTL can add 24 hours to your mitigation time.
If you're just playing around with your personal project/web site, you just added a giant round of whack-a-cache to your "let's finally clean up my personal server mess" evening.
Subsequently, my previous employer lowered the default TTL simply because it got rid of all the bad Trustpilot ratings about customers being "prevented from leaving", and started offering a "move my WordPress site for me" service to profit from all the panicking new-comers who had no idea about how to do trivial things like importing/exporting a database and transferring files.
It would be interesting to (ab)use this mechanism in the way you suggest. A recursive DNS server could ask to be NOTIFY'ed of changes in the zone they are querying...it would, of course, add load to the server, and it would need strict limits to avoid DoS, but it seems an interesting idea.
1: https://tools.ietf.org/html/draft-ietf-dnssd-push
So whats the solution? We are using AWS ALB/ELB and it states in docs, that we should have low TTL, and it makes sense. Servers behind LB scale up and down. What is the option B?
It seems that a lot of sites are ok with slightly higher latencies if it means greater operational flexibility.
So there's no single latency to report.
Don't forget to run
afterwards.Instead of a browser doing
1. Local DNS lookup (resulting in expired entry)
2. DNS query
3. DNS response
4. HTTP request
why not do
1. Local DNS lookup (resulting in expired entry)
2.1. DNS query
2.2. HTTP request
3. DNS response
4. If DNS response changed and HTTP request failed, HTTP request again
Maybe use two expiration lengths, one that results in flow 2 and a much longer one that results in flow 1.
Yes, it is not exactly great, but at least it works well enough for self-hosting services.
The author seem to be missing one of the big reasons ridiculously low TTLs are used: it lets passive eavesdroppers discover a good approximation of your browsing history. Passive logging of HTTP has (fortunately) been hindered as most traffic moved to HTTPS, but DNS is still plaintext.
Low TTLs mean a new DNS request happens (apx) every time someone clicks a link. Seeing which domain names someone is interacting with every 60s (or less!) is enough to build a very detailed pattern-of-life[1]. Remember, it's probably not just one domain name per click; the set of domain names that are requested to fetch the js/css/images/etc for each page can easily fingerprint specific activities within a domain.
Yes, TTLs need to have some kind of saner minimum. Even more important is moving to an encrypted protocol. Unfortunately DOH doesn't solve this problem[2]; it just moves the passive eavesdropping problem to a different upstream server (e.g. Cloudflare). The real solution is an encrypted protocol that allows everyone to do the recursive resolution locally[3].
[1] https://en.wikipedia.org/wiki/Pattern-of-life_analysis
[2] https://news.ycombinator.com/item?id=21110296
[3] https://news.ycombinator.com/item?id=21348328
I operate DNS for hundreds of thousands of domains. I've tried to reassemble browsing history from DNS logs, and I can tell you it is damn near impossible. You have DNS caches in the browser, the OS, broadband routers, and ISPs/public resolvers to account for - and half of them don't respect TTLs anyways.
The reason people set low TTLs is they don't want to wait around for things to expire when they want to make a change. DNS operators encourage low TTLs because it appears broken to the user when they make a change and "it doesn't work" for anywhere from a few hours to a few days.
So if some things are cached, you won't get a complete picture, but the picture you get might be enough.
It would be naive to think none do of course.
Why? :)
I did find this paper: https://labs.ripe.net/Members/giovane_moura/dns-ttl-violatio...
The violations in that paper that are important are those that have increased the TTL. Reducing the TTL increases costs for the DNS provider, but isn't important here. The slowest update was about 2 hours (with the TTL set to 333).
Of those that violated the TTL, we don't know what portion of those would function correctly with a different TTL (increasing the TTL indicates they're already not following spec). So I wouldn't assume that increasing the TTL would get them to abide by your requested TTL. They're following their own rules, and those could by anything.
Considering how common low TTLs are... you're worrying about a DNS server that's already potentially causing errors for major well known websites.
From my own experience when changing records and seeing when the long tail of clients stops calling the old addresses (with the name), it is a really long tail. An extreme example that lasted almost six months was a web spider that just refused to update their DNS records and continued to request websites using the old addresses.
Is there a lot of custom written code that does their own DNS caching? Yes. One other example is internal DNS servers that shadow external DNS. There is a lot of very old DNS software running year after year. Occasionally at work we stumble onto servers which are very clearly handwritten by someone a few decades ago by people with only a vague idea of what the RFCs actually say. Those are not public resolvers of major ISPs, so the above study would not catch them.
Naturally if you have a public resolver where people are constantly accessing common sites with low TTLs then issues would crop up quickly and the support cost would get them to fix the resolver. If it's an internal resolver inside a company where non-work sites are blocked then you might not notice until the company moves to a new web hosting solution and suddenly all employees can't access the new site, an hour later they call the public DNS hosting provider, the provider diagnoses the issue to be internal of the customer's network, and then finally several hours later the faulty resolver gets fixed.
Could it be that it wasn't using DNS at all? Just hardcoded the ip address?
It may have been Java client that was not restarted. At least for older versions of Java the default was to cache result forever.
> $ drill detectportal.firefox.com @1.1.1.1
Is captive portal detection not a valid use case for low TTL? The entire point is to detect DNS hijacking of a known domain, which takes longer when you cache the DNS results...
But sure, there are other implementation approaches (e.g. injecting HTTP redirects), which I imagine is one reason why Firefox doesn't literally inspect the DNS reply.
Connections are cached and reused.
Also what are you talking about? If the connection is closed, how would it be used?
Connections should only be closed due to? Inactivity. If a connection is closed, don't you think you would probably want to do another DNS request?
Also if your doing proper layer 4 load balancing using BGP, DNS is a moot point...
Magic...
Or if the user closes the browser of if the server/proxy restarts. But yes, mostly inactivity on the order of a couple of minutes.
> If a connection is closed, don't you think you would probably want to do another DNS request?
That's the whole point of the DNS TTL, to say how long to go before doing another lookup rather than doing it each time you reconnect.
> Also if your doing proper layer 4 load balancing using BGP, DNS is a moot point...
BGP load balancing operates on layer 3, and is irrelevant as you still need to DNS lookup an anycast address. EDNS client subnet is better anyways.
I hope you didn't pay for your education.
By that logic BGP is a layer 7 load balancer since it has an application layer. BGP only exchanges layer 3 reachability information to update route tables therefore you can only load balance layer 3 with it.
Personal attacks and other things in your comments are against the HN guidelines. The goal is to talk about DNS/TTLs and their impact on performance not insult each other. https://news.ycombinator.com/newsguidelines.html
Most sit between? 0-15 minutes. WTF is your timeout?
TTLS.
> Did you even read the article?
Sure did which is why I referenced it.
> Most sit between? 0-15 minutes.
This is true in that the range encapsulates most keep-alive timeouts not in that keepalives longer than a minute or two are actually the majority. nginx defaults to 100 seconds, Apache is less than that. Most don't mess with these let alone bump them to 900. Generally 60 to 120 is considered standard with some cap on the number of active keep-alive sessions as well. Some go ultra-low or disable it all together, very few go ultra-high.
> WTF is your timeout?
120.
Also please try to keep the conversations together.
You're argument about keep alive makes no sense. You're confusing the nginx documentation. 100 is the number of default connections it will hold open. 60 is the default timeout. Also it's sent as an HTTP response to the client in the headers...
Here you need to read up: https://nginx.org/en/docs/http/ngx_http_core_module.html#kee...
The timeout only matters if you're not making requests. Setting a low keep alive will actually result in more DNS requests...doh
Next you're going to tell me you only write blocking code and use a thread per connection...
I guess you haven't created many 10k concurrent app servers.
> You're argument about keep alive makes no sense. You're confusing the nginx documentation. 100 is the number of default connections it will hold open. 60 is the default timeout. Also it's sent as an HTTP response to the client in the headers... Here you need to read up: http://nginx.org/en/docs/http/ngx_http_core_module.html#keep...
Are you're right about the 100 being the default active keepalives not the default timeout. According to your own nginx link 75 is the timeout not 60 though: "Default: keepalive_timeout 75s;"
Either way, 75/60/100/120 are significantly far off from 15 minutes.
> The timeout only matters if you're not making requests.
Or if the server reaches max connections.
> Setting a low keep alive will actually result in more DNS requests...doh
Which is how the discussion on DNS TTLs comes about in the first place. It's trivial to set the DNS TTL astronomically higher than the HTTP keep-alive, in which case the browser & OS won't actually make a lookup request since it's cached.