Guide to make remotely-accessible rescue system

[avibrazil]

On a headless Fedora 43 server (no keyboard, no screen, just network), I needed to login remotely to make some LVM maintenance before even root filesystem was mounted.

This is how I did it, all commands as root user:

Root user needs to have a password, even if won't be used, otherwise #99 happens:

passwd root

Make sure it has a password, so this command must show its encrypted password:

grep root /etc/shadow

Install dracut-sshd, configure it and generate a new mini system called /boot/initramfs*img that helps the kernel bring your system up:

dnf install dracut-sshd
ln -s ~MY_USER/.ssh/authorized_keys /etc/dracut-sshd/
ln -s /usr/share/doc/dracut-sshd/example/90-networkd.conf /etc/dracut.conf.d/
dracut -f -v

The dracut-ssh package is a dracut module that activates networking and an SSH server on this early system stage. In my case, my server gets constant IPv4/IPv6 from my DHCP server, so it is accessible via same hostname in that early system stage.

You may want to add more commands and functionality to this mini system. Refer to dracut documentation and components available. Dracut is very modular and extensible.

Next challenge is to force the system boot to stop at that early stage so I can login and do some maintenance.

Create a boot entry based on what you already have with slightly different options:

### cp /boot/loader/entries/YOUR_LATEST_KERNEL.x86_64.conf /boot/loader/entries/rescue.conf
cp /boot/loader/entries/`ls /boot/loader/entries/ | tail -1` /boot/loader/entries/rescue.conf

vi /boot/loader/entries/rescue.conf

The title must be “Rescue” and add rd.break to the end of options. Will look like:

cat /boot/loader/entries/rescue.conf


title Rescue
version 6.17.12-300.fc43.x86_64
linux /vmlinuz-6.17.12-300.fc43.x86_64
initrd /initramfs-6.17.12-300.fc43.x86_64.img
options root=/dev/mapper/fedora_vtest-root ro rd.lvm.lv=fedora_vtest/root rhgb quiet rd.break
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora

Perhaps you want to also remove all LVM references from options so your disks won't be even touched.

Set this entry for the next boot only:

grub2-reboot Rescue

And then reboot and login from your laptop like:

ssh -l root BOOTING_SERVER_HOSTNAME

The laptop user must have the private key that matches the public key that you installed above with ~MY_USER/.ssh/authorized_keys

You'll find your server in a boot state before most partitions, volume groups and services were activated. Do your stuff and reboot: the server will boot to a normal boot entry, not the Rescue one anymore.

Login again into a now fully functional server and delete the Rescue boot entry:

rm /boot/loader/entries/rescue.conf

Also, I noticed that this dracut-ssh-enabled initramfs messes up with networking even after rebooting to normal mode, so more cleanup is needed:

rm /etc/dracut.conf.d/90-networkd.conf
dracut -f -v
reboot

[gsauthof]

FWIW, slightly related, some time ago, I had a similar use case, i.e. to invoke a remotely accessible rescue system.

My ansatz then was to generate an initramfs that contains a full minimal fedora system, without even relying on dracut, and kexec-ing into it:

https://github.com/gsauthof/playbook/tree/master/fedora/initramfs