37 comments

[ 14.0 ms ] story [ 942 ms ] thread
This has been a "need" for a long time for home users with FDE. The go-to software solution for Linux has been:

https://github.com/gsauthof/dracut-sshd

Sure, if you're using dracut, which is not true for "Linux" in general.
I have a setup based on this, but I modified it to encrypt the SSH host key using the TPM. That way, I can detect a MiTM from an attacker who has stolen the drive or modified the boot policy because host key verification will fail.

/usr/lib/dracut/modules.d/46cryptssh:

    #!/bin/bash
    
    check() {
        require_binaries sshd || return 1
        return 0
    }
    
    depends() {
        return 0
    }
    
    install() {
        if [ ! -e /etc/ssh/dracut ]; then
            mkdir /etc/ssh/dracut &&
            tmp=$(mktemp -d) &&
            head -c128 /dev/random > $tmp/passphrase &&
            ssh-keygen -t ed25519 -f /etc/ssh/dracut/ssh_host_ed25519_key -N"$(base64 < $tmp/passphrase)" &&
            tpm2_createprimary -C o -c $tmp/primary.ctx &&
            tpm2_pcrread -o $tmp/pcr.bin sha256:0,7
            tpm2_createpolicy --policy-pcr -l sha256:0,7 -f $tmp/pcr.bin -L $tmp/pcr.policy
            tpm2_create -C $tmp/primary.ctx -L $tmp/pcr.policy -i $tmp/passphrase -c $tmp/seal.ctx &&
            tpm2_evictcontrol -C o -c $tmp/seal.ctx -o /etc/ssh/dracut/seal || {
                rm -r $tmp /etc/ssh/dracut
                exit 1
            }
            rm -r $tmp
        fi
        for file in /etc/ssh/dracut/*; do
            inst_simple "$file" "/etc/ssh/${file#/etc/ssh/dracut/}"
        done
    
        mkdir -p -m 0700 "$initdir"/root/.ssh
        /usr/bin/install -m 600 /etc/ssh/dracut_authorized_keys "$initdir"/root/.ssh/authorized_keys
    
        inst_binary /usr/sbin/sshd
        inst_binary /usr/sbin/ssh-keygen
        inst_binary /usr/bin/tpm2_unseal
        inst_binary /usr/bin/base64
        inst_simple /usr/lib/libtss2-tcti-device.so
        
        inst_simple "$moddir"/cryptsshd.service "$systemdsystemunitdir"/cryptsshd.service
        inst_simple "$moddir"/sshd_config /etc/ssh/sshd_config
    
        inst_binary /usr/lib/ssh/sshd-session
        inst_binary /usr/lib/ssh/sshd-auth
    
        mkdir -p -m 0755 "$initdir"/var/empty/sshd
        mkdir -p -m 0755 "$initdir"/usr/share/empty.sshd
        mkdir -p  -m 0755 "$initdir"/var/log
        touch "$initdir"/var/log/lastlog
    
        systemctl -q --root "$initdir" enable cryptsshd
    }
cryptsshd.service:

    [Unit]
    Description=OpenSSH Daemon for Disk Encryption Passphrase
    DefaultDependencies=no
    Before=cryptsetup.target
    After=network-online.target
    
    [Service]
    Type=notify-reload
    ExecStartPre=/bin/sh -c '/usr/bin/ssh-keygen -p -f /etc/ssh/ssh_host_ed25519_key \
        -N "" -P "$(/usr/bin/tpm2_unseal -c /etc/ssh/seal -p pcr:sha256:0,7 | base64)"'
    ExecStart=/usr/bin/sshd -D
    KillMode=process
    Restart=always
    
    [Install]
    WantedBy=sysinit.target
That encrypts the SSH host key using a password sealed with PCR7, which is invalidated if an attacker disables Secure Boot or tampers with the enrolled keys. Thus, an attacker can't extract the key from the drive or by modifying the kernel command line to boot to a shell (since that's not allowed without disabling secure boot).

It's still probably vulnerable to a cold boot attack, since the key is decrypted CPU-side. It would be interesting to perform the actual key operations on the TPM ...

I'm vaguely reminded of some of the third party disk encryption/preboot management utilities that exist in the Windows space that leverage similar technology. Authentication is done against an online source, and only then is the key sent back to the local machine to unlock the disk. The Bitlocker key is kept nowhere near the local TPM.

I've only seen it on some paranoid-level devices in industry (typically devices handling biometric identity verification services).

IIRC this one is a Linux image that boots up, unlocks the normal Bitlocker partition via whatever mechanism you need, then hands control back to the Windows bootloader to continue onwards.

https://winmagic.com/en/products/full-disk-encryption-for-wi...

FYI your decryption key can be MITMed during this process by anyone with physical access to the system, which defeats the purpose of encrypting the disk in the first place.

Just use dm-verity for remote servers.

A long time ago, I built my own crashcart adapter with a raspberry pi and a teensy to do something similar. I would sometimes get weird mdadm errors that would hang the boot process and other times, a reboot or power loss wouldn't actually cause the PC to boot back up. The teensy did USB HID emulation for keyboard inputs. I added the ability to push the power button with a fet and some resistors. I had a cheap VGA to composite adapter going into a USB composite capture device so I could at least get screenshots for any weird boot messages. I built a small webpage using flask to display the screenshot, allow for text input, control inputs, and to push the power button. It was a lot of fun building but a basement flood completely wrecked it. Server was sitting on a 6in platform but the crashcart had fallen off the top of the case and was laying on the ground. Oops.
(comment deleted)
There is an old but still reasonable solution with mkinitcpio hooks encrypt/sd-encrypt + ssh, which is very easy to set up with EFI or grub2 onward. Tailscale is probably overkill for this use case, given that you're already exposing pre-/early- boot to the network by setting up interfaces that early. This became much more hermetic with secureboot and TPMs, too.
I'd love to see this in the bootloader, along with a selection of binaries useful for recovery. Might sound silly but over the years I have had many a remote system get to the bootloader and then no further after an upgrade. Nowadays we've usually got a nicely sized EFI partition, why not stuff it all in there? Gimme a full Linux userspace from the bootloader, it would feel luxurious when I'm up at 3 am trying to recover a broken system halfway across the country.

Or is there already a solution to this that I've been missing? (Yeah, KVM/IPMI/etc, I know, but not all hosters make it easy to get to that.)

In new installs you do stuff everything in EFI partition and skip the old /boot partition as such.

The better solution is to use tpm, unified kernel image and secure boot skipping the network unlock.

The whole process is like this -

1. enable secure boot;

2. generate and install your own secure boot keys (using sbctl);

3. use clevis to enable automatic unlocking of the root fs only when secure boot check passes;

4. generate the unified kernel image (in EFI partition) that is signed by your secure boot key;

4. use efibootmgr to enable booting of said kernel image.

(5.) If your CPU supports it, enable memory encryption in BIOS (to mitigate cold boot attacks).

The unified kernel image doesn't accept additional kernel parameters, so only parameters that are set during generation of the initram are used. The secure boot makes sure no one else has tampered with the boot chain. And TPM stores the disk key securely.

You can still add some additional network level check to make sure that your computer is in your expected location before unlocking.

And you can also include some recovery tools + dropbear in your initram (within the unified kernel image), if you expect that you will have to do some recovery from the other side of the world.

> 3. use clevis to enable automatic unlocking of the root fs only when secure boot check passes;

Can also use systemd-cryptsetup/systemd-cryptenroll for this. I've not used clevis myself, but I'd imagine you have to do somewhat more rolling-your-own compared to the systemd tools.

> The unified kernel image doesn't accept additional kernel parameters, so only parameters that are set during generation of the initram are used. The secure boot makes sure no one else has tampered with the boot chain. And TPM stores the disk key securely.

FYI, multi-profile UKIs are a thing. You can have one UKI with multiple different command lines, e.g. one for regular boot, one for emergency mode, etc.

https://uapi-group.org/specifications/specs/unified_kernel_i...

Sounds like you want ZFSBootMenu.org which offers remote SSH access with FDE in addition to snapshots in case of update falures or other issues. As long as you don't format the disk itself or wipe the ZFSBootMenu efi file you can recover and revert from anything remotely.
If you want to be able to reboot remotely, and non-interactively (i.e. while you sleep), I (and one other person) created Mandos for this purpose: <https://www.recompile.se/mandos>. It gets the password over the network. If this seems insecure to you, read the FAQ: <https://www.recompile.se/mandos/man/intro.8mandos>

Mandos works with initramfs images created by both initramfs-tools and Dracut, and is present in Debian since 2011, so no need to use a third-party package.

Sounds great, I'm going to give this a try. Is there a plug-in for passkeys/webauthn for the mandos-server unlock?
The kind of tool that I'm waiting for would allow me to log into a machine remotely, then reboot it once into a provided image (which could then run Linux in a ramdisk). I.e., take over the entire machine until it is restarted again. Does something like this exist?

I want something like "kexec" but taking over the entire hardware at the lowest possible level.

Edit, found this:

https://github.com/marcan/takeover.sh

But it's not as low level as I hoped, though it keeps the network running which is nice :)

> Because initramfs is just a (mostly) normal Linux system, that means it has its own init PID 1. On Arch, that PID is in fact just systemd.

Debian has (or had; at least my Devuan still has) a simple shell script as first init. Was an interesting read and helped me understand were to add my remote rootfs decryption.

https://salsa.debian.org/kernel-team/initramfs-tools/-/blob/...

Glad to see another example of this! Remote unlocking of your personal server's encrypted hard drive is PITA.

Other options that I've investigated that involve having a second server:

* A second server with Tang, and Clevis in the initramfs OS

* Keylime

Putting tailscale in the initramfs, and then updating the certs on a frequent enough schedule, seems risky to me. I've already played around with limine enough that I know I don't want to install much in the initramfs...

Man, that's pretty excellent!

I once built a demo-ish encrypted network boot system using similar initrd techniques. It's a fun hack working in the preboot environment.

I currently have dropbear-ssh presenting the LUKS password prompt on my home server, but that has the very annoying quality that there's no way to do it from the console if you set that up too.

It's not a huge problem but it certainly means some recovery scenarios would be painful.

Seems like you should be able to launch the LUKS password prompt multiple times on different interfaces.
I have a very similar setup to the author, but instead of running Tailscale in my initramfs, I have a Raspberry Pi sitting next to the home server (which is on my Tailscale network) and I use it like a bastion host. Process is something like:

1. SSH into the Pi

2. Issue the Wake-on-LAN packet to boot the server

3. Tunnel that server's SSH port to my laptop

4. SSH into the initramfs SSH server (I use TinySSH), enter the decryption key

5. Wait for server to come up, then access via Tailscale

This is more complicated than the author's setup in that it requires another device (the Pi), but it's simpler in that you don't need to have the initramfs on your Tailnet.

It's not only more complicated, it also does not sound to me like it would scale. What do you do when you have N servers? Do you buy N raspis, or do you keep using one bastion host? How do you automate it when you sooner or later must (re-)deploy?

If you set this up once ("this" meaning adding networking, SSH and tailscale inside initramfs), you can just do the same thing for the next server you set up, and you don't have to worry about the failure of one node affecting the other(s).

I have something similar set up to unlock the drives on my home server. Just the SSH in initramfs though, tailscale is pretty cool.

I've done stuff with mkinitcpio / initramfs on arch before, can't remember exactly what for. I still run arch on my main laptop. I'm running nixos on my home server though, and adding something like this is so easy by comparison.

And me was thinking about having a monitor, camera pointed at monitor and robot arm that I could type password in controlling it remotely.
Aside from enrolling a token with the TPM to unlock the LUKS volume, this is actually a pretty novel idea. Perfect for older hardware without TPM. I guess it depends on your use-case.
is it just me, or feels scary that it is happening
It doesn't work if you use RAID 5 OR RAID 6 My friend have tried it
MacOS recently solved this "issue".

When rebooting a FileVault encrypted machine, where it normally "hangs" asking for a user to unlock it, you can now SSH into the machine, but instead of getting a prompt it interprets your SSH login as a user logging in, hangs up, and proceeds to booting up.

Can you also decrypt a PGP-encrypted hard disk now? I mean, nowadays...?