Ask HN: Thoughts on /etc/hosts instead of DNS for production applications?
"It's always DNS" is a theme we're all familiar with when it comes to outages. I understand why DNS is critical for most users. But for applications that are managed/deployed using an "Infrastructure as Code" system, where changes can, and should always be pushed in a way that treats the changes the same way code changes are treated (Devops and all that), is there any harm with using /etc/hosts files everywhere?
That way name-to-ip association changes benefit from IaC, and DNS related instabilities are minimized. Of course, I am assuming the name-to-ip association is under the control of the system's engineers to begin with, for every other use case DNS can and should still be used.
Why aren't cloud providers and FAANGs doing this already, where saving costs by eliminating things like DNS request traffic and CPU cycles is encouraged?
7 comments
[ 4.5 ms ] story [ 25.3 ms ] threadYou do give up some good stuff, though. Load-balancing can be more tricky, for instance. And if any of the machines change their IP addresses, or you add new machines to the network, then you have to distribute a new hosts file to all of the machines that aren't using DNS.
Why aren't cloud providers and FAANGs doing this already
This probably requires that anyone touching the Infrastructure as Code are all critical thinkers and fully understand the implications of mapping applications to hosts including but not limited to applications having their own load balancing mechanisms, fail-over IP addresses, application state and ARP timeouts, broadcast and multicast discovery. It can be done but I would expect large companies to avoid this potential complexity trap. It might work fine in smaller companies that have only senior/principal engineers. Using /etc/hosts for boot-strapping critical infrastructure nodes required for dynamic DNS updates could still make sense in some cases. Point being, this gets really complex and whatever is managing the Infrastructure as Code would have to fully aware of every level of abstraction, NAT's, SNAT's, hair-pin routes, load balanced virtual servers and origin nodes. Some companies are so big and complex that one human can not know the whole thing so everyone's silo knowledge has to be merged into this Inf as Code beast. Recursive DNS on the other hand only has to know the correct up-stream resolvers to use or if they are supposed to talk directly to the root DNS servers. This simplifies the layers upon layers of abstraction that manage their own application mapping and DNS.
Another complexity trap people get lured into is split-views which should be avoided due to growing into a complexity trap over time and breaking sites when one dependency starts to interfere with another. Everyone has to learn the hard way for themselves on this topic.
My preference would be to instead make DNS more resilient. Running Unbound [1] on every node pointing to a group of edge DNS resolvers for external IP addresses with customized settings to retry and keep state up the fastest upstream resolving DNS nodes, also caching infrastructure addresses and their state, setting realistic min/max DNS TTL times is a small step in the right direction. Dev/QA environments should also enable query logging to a tmpfs mount to help debug application misconfigurations and spot less than optimal uses of DNS within the infrastructure and application settings before anything gets to staging or production. Grab statistical data from Unbound on every node and ingest it into some form of big-data/AI web interface so questions about resolution, timing, errors may potentially be analyzed.
This is just my two cents based on my experience. If it seems like I was just spewing words I was watching Shane Gillis and did not want to turn it off.
[1] - https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound...
DNS isn't just name resolution, I'd say it's kind of load balancing, service discovery, caching, and dynamic configuration "all in one".
The FAANGs do minimize external DNS calls, but they run massive internal DNS infrastructures because the alternatives (config management pushing files) are actually slower and more fragile at scale.
I wouldn't use /etc/hosts for something with those needs, but pushing a smallish file to 10,000 hosts in a minute is readily achievable if you need it. I don't have an estimate kf the number of ssh handshakes per second a typical client machine could do, but if you run 512 pushes in parallel, I bet you can get it pushed in way under a minute; assuming you have sufficient bandwidth I would avoid trying to do a differential update because the interactivity is too many round trips and you'll blow your time budget, but you could keep several known hashes and differential update from those and full push if no match). Up to you if you want to also run a command to reload the file, or if you'd rather kqueue/inotify/just a periodic poll for change.
If you have frequent changes, you probably want to do a phone tree approach where you push to a few servers per location and those push to the local servers. And probably a way to do single name announcements. Or maybe something else.
But, if your system communicates by server hostnames and your server hostnames don't change often and ips of active servers don't change often, using /etc/hosts for that can be pretty useful. Everytime you get new servers, you have to repush the hostfile, but it's really not a big deal; and it changes a dynamic dependency to a static one. You can even run monitors to confirm your servers aren't resolving each other and that they all have acceptable host files.
(Late edit, composed while I lost connectivity) Also, if your scale is hundreds of thousands of hosts, it might get pretty nasty, but chances are you wouldn't resolve every name on every host, so something smarter is needed; which could be DNS or could be selective pushing or could be some other dynamic system that's not entirely DNS.
e.g: consul-template pointing to consul or puppet looking at some gitops repo
In a perfect world, everything would have a unique IPV6 address, and we wouldn't need DNS. Instead of NAT, you would just have any computer/vm that wants to be connected to the internet tacked on to existing address space, and then instead of DNS record being synced, you just use the address directly, and routing would take care of everything.
At a big enough scale, though, with thousands of people making changes many times / day, the file becomes a single point of contention, and the logistics of updating it without losing or clobbering other people's changes are going to become a problem.