rebuilding node image fails because CentOS 8 is EOL

[boegel]

I was trying to rebuild the node image of the CitC cluster that has been running for a while, and that's failing because CentOS 8 is end-of-life, and the repositories are no longer available:

When /etc/citc/packer/prepare_ansible.sh is running yum install -y epel-release, it fails with:

==> amazon-ebs.aws: Provisioning with shell script: /etc/citc/packer/prepare_ansible.sh
    amazon-ebs.aws: CentOS Linux 8 - AppStream                       95  B/s |  38  B     00:00
==> amazon-ebs.aws: Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlist
==> amazon-ebs.aws: Provisioning step had errors: Running the cleanup provisioner, if present...

I'm looking into dancing around this by switching to Rocky 8 instead, will report back here how I did that (if I can sort it out)

[boegel]

These changes worked for me (for both x86_64 and aarch64), to migrate from CentOS 8 to Rocky Linux 8 early on (using the script from https://github.com/rocky-linux/rocky-tools/tree/main/migrate2rocky):

[citc@mgmt ~]$ sudo diff -u /etc/citc/packer/prepare_ansible.sh.orig /etc/citc/packer/prepare_ansible.sh
--- /etc/citc/packer/prepare_ansible.sh.orig    2022-05-05 17:32:39.783471058 +0000
+++ /etc/citc/packer/prepare_ansible.sh 2022-05-05 17:43:41.212415589 +0000
@@ -9,6 +9,12 @@
 cluster_id=fair-mastodon
 packer_run=yes
 EOF'
+pwd
+curl -OL https://raw.githubusercontent.com/rocky-linux/rocky-tools/main/migrate2rocky/migrate2rocky.sh
+mv migrate2rocky.sh /tmp/
+chmod a+x /tmp/migrate2rocky.sh
+sudo /tmp/migrate2rocky.sh -r
+sudo /tmp/migrate2rocky.sh -V
 sudo yum install -y epel-release
 sudo dnf config-manager --set-enabled powertools
 sudo dnf install -y ansible git

[boegel]

@milliams suggested to me to tweak the source_ami_filter section in /etc/citc/packer/all.pkr.hcl to start from a Rocky Linux 8 AMI rather than a CentOS 8 one, but that's less straightforward as it seems because the official Rocky Linux AMIs are provided through the AWS Marketplace, not as a "community image".

That implies that you can't just search for "Rocky Linux 8.*" and then filter based on owner, and be sure that you'll always get the official AMIs...

See also the discussion in https://forums.rockylinux.org/t/rocky-linux-official-aws-ami/3049/20 .