HN Explorer

Systemd makes life miserable, again, this time by breaking DNS

6 points by bcrl ↗ HN
So, I made the mistake of updating my laptop from Fedora 31 to Fedora 33 last night. Normally this is fairly painless, as my laptop is one of the last machines I perform distribution upgrades. Today while doing some pole survey work out in the field, I tethered my laptop to my phone as has been done hundreds of times before. To my surprise, DNS doesn't work anymore, but only in web browsers. Both Firefox and Chrome can't resolve names anymore. Command line tools like ping and host work normally. WTF?

Why are distributions continuing to allow systemd to extend its tentacles deeper and deeper into more parts of Linux userland with poorly tested subsystem replacements for parts of Linux that have been stable for decades? Does nobody else consider this repeating pattern of rewrite-replace-introduce-new-bugs a problem? Newer is not all that better if you break what is a pretty bog standard and common use-case.

7 comments

[ 2.6 ms ] story [ 15.0 ms ] thread
ratsmack
> ... again ...

I didn't know that it ever stopped.

rektide
it's probably dnssec & your time is probably off.

jorunalctl -xe should be showing errors if so. probably check 'joirnalctl -xeu systemd-resolved'

use 'date' to check the time. if it's off, I use ntpdate to update my clock. but since dns isn't resolving I use 'dig pool.ntp.org @9.9.9.9' to resolve a ntp pool server ip address. then ntpdate [that up].

one of those things that hits me a bunch that I haven't automated quite yet. supposedly newer systemd can detect dnssec being bad in some cases & disable it, after a time, after a bunch of failures, but it either hasn't tripped in a number of cases for me or something else was odd in my envs. manually syncing the clock via ntp usually gets my dns working again.

westurner
> manually syncing the clock via ntp usually gets my dns working again.

Why is this necessary?

rektide
normally systemd-timedated does a good job keeping things in sync!

but some of my systems seem to have not great real-time clock batteries, and the system will forget the time. i think there might be some other circumstance that sometimes causes my system clock to be way out of whack, but i'm not sure what.

so that's why my clock doesn't work and needs sync.

the dnssec internet protocols are designed to guarantee the user that they have up to date, accurate, trustable records. this depends on your system knowing what time it is now. if your system is way ahead or way behind the actual time, the dnssec records it gets dont appear as valid. And systemd-resolved will reject them, if it is set up to respect DNSSEC.

rayko
systemctl stop systemd-resolved; systemctl disable systemd-resolved; check the settings in /etc/resolv.conf

That brought things back to normal for me.

westurner
As well, Firefox now defaults to DoH (DNS over HTTPS), which may be bypassing systemd-resolved by doing DNS resolution in the app instead of calling `gethostbyname()` (`man gethostbyname`) and/or `getaddrinfo()`.

`man systemd-resolved` describes why there is new DNS functionality: security; "caching and validating DNS/DNSSEC stub resolver, as well as an LLMR and MulticastDNS resolver and responder".

From `man systemd-resolved` https://man7.org/linux/man-pages/man8/systemd-resolved.servi... :

> To improve compatibility, /etc/resolv.conf is read in order to discover configured system DNS servers, but only if it is not a symlink to /run/systemd/resolve/stub-resolv.conf, /usr/lib/systemd/resolv.conf or /run/systemd/resolve/resolv.conf

> [...] Note that the selected mode of operation for this file is detected fully automatically, depending on whether /etc/resolv.conf is a symlink to /run/systemd/resolve/resolv.conf or lists 127.0.0.53 as DNS server.

Is /etc/resolv.conf read on reload and or restart of the systemd-resolved service (`servicectl restart systemd-named`)?

Some examples of validating DNSSEC in `man delv` would be helpful.

NetworkManager (now with systemd-resolved) is one system for doing DNS configuration for zero or more transient interfaces:

  man nmcli

  nmcli connection help
  nmcli c help
  nmcli c h

  nmcli c show ssid_or_nm_profile | grep -i dns

  nmcli c modify help

  man systemd-resolved
  man delv
  man dnssec-trust-anchors.d