develop automation to build BSD images

[jbpratt]

*BSD images can't use the guestfs-tools, mainly due to UFS, therefore we need to write the customization of these images using an alternative tool. virt-lightning seems like an easy enough way forward thanks to all of the hard work from @goneri as well on the *BSD side and cloud-init support.

• FreeBSD
• NetBSD
• OpenBSD
• DragonFlyBSD

As per our general baseline requirements for an image, we should aim to customize the images in three main ways:

1. QEMU guest agent should start on boot
2. Ability to SSH using the service after qemu-guest-agent propagates keys
3. Console is available out of the box

Working with @goneri to support qemu-ga out of the box upstream. Thanks to this work, we will simply need to download the image, skip customization then package it.

From here, the standard automation for packaging can kick in as if the image has been downloaded and skip customization the same way we implemented for *CoreOS. After packaging, tests will be kicked off to validate qemu-guest-agent starts on boot and ssh is available via the k8s service exposed.

Resources:
https://www.freshports.org/emulators/qemu-guest-agent
https://undeadly.org/cgi?action=article;sid=20200514073852
https://github.com/virt-lightning/virt-lightning
https://bsd-cloud-image.org

Things to keep in mind:

• While testing cloud-init, we were seeing OpenBSD restarting every 1-2 minutes. This stopped after doubling the memory to 4G.
• cloud-init networking should be set up slightly different than usual (requiring matching on the mac_address)
• /bin/bash is not available, we should stick to the default shell

[goneri]

I've got a bunch of scripts to build bsd-cloud-image images. I can probably install qemu-agent in the images if it's available as a binary package. This being said, you can also rely on cloud-init to inject the SSH keys too.

[jbpratt]

Hey @goneri, that would awesome! Any way to minimize the amount of customize building we have to do would be great. We could start with FreeBSD since there is a maintained port (https://github.com/aborche/qemu-guest-agent) that I am aware of.

This being said, you can also rely on cloud-init to inject the SSH keys too.

Yes, we can take advantage of cloud-init for static ssh key inject and qemu-guest-agent providing dynamic SSH key injection.

https://kubevirt.io/user-guide/virtual_machines/accessing_virtual_machines/#dynamic-ssh-key-injection-via-qemu-user-agent

[usrbinkat]

Adding use case explanation to the value of qemu-guest-agent.

We leverage kubernetes secrets to supply user details and credentials such as username, password (if required), and ssh-keys. This provides a predictable user/administration model where the user creates an ssh key secret for any given deployment, and that secret is used to propagate the ssh key dynamically to the subscribed instances even as the instance count may scale up/down.

One feature that is convenient for this is the ability to dynamically rotate and or even add additional ssh pub keys to all subscribed instances by just updating the kubernetes credentials secret data.

This workflow is why we attempt to leverage qemu-guest-agent when ever possible as the first-class experience, and then fall back from there to direct user-data injected ssh keys only for distributions or bugs that prevent this experience.

TLDR, prefer to use qemu-guest-agent for dynamic long lived control over ssh keys in subscribed kubevirt instances, we are also able to use cloud-init as a fallback option when required and can maintain documentation to cover these edges as needed.

[jbpratt]

FreeBSD has been implemented with the ssh test disabled for now.