From 3ae91060ea0ceba8d911540f931268d52a7074e5 Mon Sep 17 00:00:00 2001 From: Sally Makin Date: Sat, 31 May 2025 08:56:47 +0100 Subject: [PATCH] Add mirroring pages --- docs/staging/mirror-scripts.md | 110 ++++++++++++++++++++ docs/staging/mirrors.md | 177 +++++++++++++++++++++++++++++++++ docs/staging/push-mirroring.md | 141 ++++++++++++++++++++++++++ 3 files changed, 428 insertions(+) create mode 100644 docs/staging/mirror-scripts.md create mode 100644 docs/staging/mirrors.md create mode 100644 docs/staging/push-mirroring.md diff --git a/docs/staging/mirror-scripts.md b/docs/staging/mirror-scripts.md new file mode 100644 index 00000000..05abc150 --- /dev/null +++ b/docs/staging/mirror-scripts.md @@ -0,0 +1,110 @@ +(mirror-scripts)= +# Mirror scripts + +```{note} +This content comes [from the wiki](https://wiki.ubuntu.com/Mirrors/Scripts) +It has not yet been reviewed for currency or accuracy. +(last edited, 2017) +``` + +To be able to create a mirror people can rely on, you need to have all the +files, and you need them at the right moment. +"File not found" errors cause a lot of issues and annoyances for users. +This page provides you with scripts to sync from other mirrors and prevent 404's. + +## ubumirror + +One can use the scripts from the +[ubumirror](https://code.launchpad.net/~ubumirror-devs/ubumirror/trunk) +project to keep their mirror in sync, or use the scripts provided below. + +This project has been [packaged since Lucid](https://launchpad.net/ubuntu/+source/ubumirror). +Please file any bug reports you find [against it](https://bugs.launchpad.net/ubuntu/+source/ubumirror). + +## Archive mirrors + +For archive mirrors, it is very important **not** to delete packages before the +`Packages.gz` files (which hold information about the packages available) are +updated. Therefore, you need a "two-stage sync". This means that you download +new packages first, and new `Packages.gz` after that. After you've downloaded +the `Packages.gz` files, it's safe to delete old packages. + +### The script + +``` +#/bin/dash + +fatal() { + echo "$1" + exit 1 +} + +warn() { + echo "$1" +} + +# Find a source mirror near you which supports rsync on +# https://launchpad.net/ubuntu/+archivemirrors +# rsync://.rsync.archive.ubuntu.com/ubuntu should always work +RSYNCSOURCE=rsync://archive.ubuntu.mirror.isp.com/ubuntu + +# Define where you want the mirror-data to be on your mirror +BASEDIR=/var/www/ubuntuarchive/ + +if [ ! -d ${BASEDIR} ]; then + warn "${BASEDIR} does not exist yet, trying to create it..." + mkdir -p ${BASEDIR} || fatal "Creation of ${BASEDIR} failed." +fi + +rsync --recursive --times --links --safe-links --hard-links \ + --stats \ + --exclude "Packages*" --exclude "Sources*" \ + --exclude "Release*" --exclude "InRelease" \ + ${RSYNCSOURCE} ${BASEDIR} || fatal "First stage of sync failed." + +rsync --recursive --times --links --safe-links --hard-links \ + --stats --delete --delete-after \ + ${RSYNCSOURCE} ${BASEDIR} || fatal "Second stage of sync failed." + +date -u > ${BASEDIR}/project/trace/$(hostname -f) +``` + +## Releases mirrors + +For Releases mirrors, things are a little less complicated. There are no +dependencies between files, so you can just `rsync` away. + +### The script + +``` +#/bin/dash + +fatal() { + echo "$1" + exit 1 +} + +warn() { + echo "$1" +} + +# Find a source mirror near you which supports rsync on +# https://launchpad.net/ubuntu/+cdmirrors +# rsync://.rsync.releases.ubuntu.com/releases should always work +RSYNCSOURCE=rsync://releases.ubuntu.mirror.isp.com/releases + +# Define where you want the mirror-data to be on your mirror +BASEDIR=/var/www/ubuntureleases/ + +if [ ! -d ${BASEDIR} ]; then + warn "${BASEDIR} does not exist yet, trying to create it..." + mkdir -p ${BASEDIR} || fatal "Creation of ${BASEDIR} failed." +fi + +rsync --verbose --recursive --times --links --safe-links --hard-links \ + --stats --delete-after \ + ${RSYNCSOURCE} ${BASEDIR} || fatal "Failed to rsync from ${RSYNCSOURCE}." + +date -u > ${BASEDIR}/.trace/$(hostname -f) +``` + diff --git a/docs/staging/mirrors.md b/docs/staging/mirrors.md new file mode 100644 index 00000000..0e96341b --- /dev/null +++ b/docs/staging/mirrors.md @@ -0,0 +1,177 @@ +(mirrors)= + +# Mirrors + +```{note} +This content comes [from the wiki](https://wiki.ubuntu.com/Mirrors). +It has not yet been reviewed for currency or accuracy. +Last updated: 2024 +``` + +Looking for a list of Ubuntu {ref}`archive or CD image mirrors `? + +The distribution of Ubuntu CD images and packages can always be improved. +[There's a team](https://launchpad.net/~ubuntu-mirror-admins) working on +improving user experiences every day, making Ubuntu available to you. You can +help too, by creating a mirror of your own and provide people near you with a +reliable mirror. + +These pages tell you what to expect, and what Ubuntu expects. You can also find +example scripts to sync, and guidelines on maintaining your mirror. + +There are two types of mirrors: + +* Country mirrors (e.g. `nl.archive.ubuntu.com`/`nl.releases.ubuntu.com`) +* Normal mirrors (reachable via their own hostname) + +If you wish to create a new mirror, please read this page and its sub-pages. +If you want to apply for being a country mirror, please read {ref}`these requirements `. + +## Registration + +If your mirror is meant to be used by others, please go and register it at +[Launchpad](https://launchpad.net/ubuntu/+newmirror). You may need to create a +Launchpad account first. When your registration is successful and approved, it +will show up on the mirror lists on Launchpad. It will also be checked by +Launchpad. + +If you registered a CD mirror, and registration is successful and approved, it +will also show up on the [Ubuntu download pages](http://www.ubuntu.com/download). + +## Mirror guidelines + +To keep your mirror up to date and working, please follow these guidelines: + +* **Be committed to being a mirror** + + Obviously, each offer to be an Ubuntu mirror is great and users appreciate it. + However, it's in nobody's interest to go and change the `sources.list` every + time a mirror disappears. So if you're not planning on creating a mirror for + the long run, don't register it. People might get disappointed in you and/or + Ubuntu. + +* **Know what you're starting with** + + A mirror may cause a lot of traffic. If your server has insufficient + bandwidth, users cannot download very well and your machine may become + unreachable. If you pay for traffic, please note that traffic might increase, + a lot. + +* **Make sure you have enough disk space** + + The Ubuntu archive, as of `2024-04-09`, uses about: + + * **2.6TB** of disk space for the Ubuntu package archive + * **32GB** for Ubuntu release CD images + + ... *and is slowly growing*. A full disk will get you out of sync and cause + problems for your users. + +* **Keep up to date** + + Please try to mirror about four times a day (so every six hours) for + archive mirrors. Since Ubuntu only releases every six months or so, a daily + check on a releases mirror is sufficient. + + [Push mirroring](https://wiki.ubuntu.com/Mirrors/PushMirroring) is available + as an alternative. + +* **Monitor the output of your sync scripts** + + Sometimes, syncing of the mirrors fails. That's OK, but it is very important + that you monitor that and correct faults. Missing packages are not very + user-friendly, so try to avoid that. + +* **Subscribe to the mirror mailing lists** + + There are two mailing lists for mirror admins. You have the + [`ubuntu-mirrors` mailing list](https://lists.ubuntu.com/mailman/listinfo/ubuntu-mirrors) + for discussion, tips and tricks about mirroring. There's also + [`ubuntu-mirrors-announce`](https://lists.ubuntu.com/mailman/listinfo/ubuntu-mirrors-announce) + which announces big updates and deletes on the mirrors. + +(country-mirror)= +## Country mirror requirements + +If you want to apply for being a country mirror, you **MUST** follow these +requirements: + +* In case of an archive mirror: + + * update every six hours (four times a day) + + * use a {ref}`two-stage sync ` + +* In case of a Releases mirror, update every 4 hours (six times a day) or have + {ref}`push mirroring ` set up. + +* Provide the following services: + + * HTTP + + * Optional, but beneficial: `rsync` (modules '`ubuntu`' for archive and + '`releases`' for releases). + +* Keep your Launchpad account up to date, so we can reach you if needed. + +* In case of an archive mirror, the archive must be available at the following + URL: + + * `http://.archive.ubuntu.com/ubuntu/` + +* In case of a releases mirror, the releases must be available at the following + URL: + + * `http://.releases.ubuntu.com/` + +* Subscribe to [`ubuntu-mirrors-announce`](https://lists.ubuntu.com/mailman/listinfo/ubuntu-mirrors-announce) + which announces big updates and deletes on the mirrors. + +```{note} +The way {term}`APT` works means that all of the hosts behind a given name +**must** be in sync. + +If there is already a country mirror for that country, then you need to work +out with that country mirror which one is the better one going forward, since +there can only be one. + +If there is more than one host, they should be on the same LAN and making sure +that they remain consistent in the view that they present to the users. +``` + +## Releases statistics + +[Releases](https://wiki.ubuntu.com/Releases) approximately take up the following +number of *bytes* (as of `2019-03-27`): + +| Release | Size in bytes | Size in GB | +| ------------- | ------------- | ---------- | +| Precise | 4625553121 | 4.3 | +| Trusty | 3775672451 | 3.5 | +| Xenial | 5324338074 | 5.0 | +| Bionic | 2876699930 | 2.7 | +| Cosmic | 2929092930 | 2.7 | +| ubuntu-core | 1783731688 | 1.7 | +| **Total** | 16589609199 | 20.0 | + +(existing-mirrors)= +## Existing mirrors + +You can find the list of currently registered mirrors on launchpad: + +* [Archive Mirrors](https://launchpad.net/ubuntu/+archivemirrors) + +* [CD image mirrors](https://launchpad.net/ubuntu/+cdmirrors) + +To identify which country mirror you are connecting to, you can do a `CNAME` +lookup on the URL and cross reference the result with the relevant list above. +You can use the `dig` command on the terminal to do this, e.g. `dig nl.archive.ubuntu.com`. + +(mirrors-communication)= +## Communication + +If you want to get in touch with other mirror admins, feel free to join us at +`#ubuntu-mirrors` on `https://libera.chat`. Also, if you have questions or an +issue with a mirror, email us at `mirrors@ubuntu.com` to [open a ticket](https://rt.ubuntu.com). + + diff --git a/docs/staging/push-mirroring.md b/docs/staging/push-mirroring.md new file mode 100644 index 00000000..15bab9ee --- /dev/null +++ b/docs/staging/push-mirroring.md @@ -0,0 +1,141 @@ +(push-mirroring)= +# Push mirroring + +```{note} +This content comes [from the wiki](https://wiki.ubuntu.com/Mirrors/PushMirroring) +It has not yet been reviewed for currency or accuracy. +Last update: 2022 +``` + +As explained on [Debian's documentation](http://www.debian.org/mirror/push_mirroring) +on this subject: + +> Push mirroring is a form of mirroring that minimizes the time it takes +> for changes to the main archive to reach mirrors. The server mirror uses +> a triggering mechanism to immediately inform the client mirror that it +> needs to be updated. +> +> Push mirroring takes a little more effort to set up since the maintainers +> of the upstream and downstream mirror must exchange information. The +> benefit is that the upstream mirror initiates the mirror process immediately +> after its archive has been updated. This allows changes to the archive to +> propagate extremely quickly. + +So instead of relying on cron jobs to keep things in sync, mirrors can be asked +to sync when it is necessary. SSH is usually used, however we can offer HTTP +triggers if preferred. + +## Setup + +Your mirror should have previously been set up to use cron jobs and mirroring +scripts. Such scripts {ref}`can be found here `. + +We recommend that you create an `ubuntu` user and grant it permissions to the +directory where your mirror is stored. Then place the keys below (or from +whichever mirror you are syncing from) into that user's `authorized_keys` file. + +When the upstream mirror connects as the `ubuntu` user to your mirror, it will +run the script and background the process. Meanwhile the `rsync` command called +by the script, will connect to the upstream mirror and sync any changes as needed. + +When this setup has been completed, please {ref}`let us know here ` +and we'll set up the necessary trigger commands on our end. + +If the command is not run in the background, you can try redirecting `stdin`, +`stdout` and `stderr` to `/dev/null` using the following command : + +``` +command="~/archive-sync /dev/null 2>/dev/null &" +``` + +## Keys + +These are the SSH keys used for triggering archive and releases mirrors +respectively. Please be sure to change the `command=` option to the location of +your mirroring script and do **not** remove the backgrounding character (`&`) +after it. + +The keys below are for mirrors pushed by **Canonical-only**, these mirrors +should be syncing from **`rsync.archive.ubuntu.com`** (for packages) and +**`rsync.releases.ubuntu.com`** (for CD release images). Connections from +Canonical will be originating from the IP address: `91.189.88.154` and +`185.125.188.80` for archive, and `91.189.88.156` and `185.125.188.81` for +releases - so please ensure that your firewalls allow this. + +For authenticity; these keys are signed by +[Haw Loeung](https://launchpad.net/~hloeung) with the key ID `0xD10ACEB3227B53FC`. + +If you are not being pushed by Canonical, please contact the mirror +administrator of the mirror you're being pushed from for their keys and be sure +to have your sync source set to their mirror. + +### Archive trigger key + +``` +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA384 + +Ubuntu Archive trigger SSH key for Canonical. + +SSH key fingerprint: c8:ea:0f:db:86:da:64:86:de:76:64:b8:84:33:4b:23 + +no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="~/archive-sync &",from="91.189.88.154,185.125.188.80" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAt8xHRbCVFT3Uw/B+TavIlDYRoLMxOKlN3HnBeniFUJTto5Im52FbT3ODfMszz5/BIAnXBf1baWDljHErx4huohh9MxyovZ0h8GYCmMy7dZzsrV5eYhLXd2idCOKIl6gr0BTgTlJOKOgVEoZ2YtiU9MnNzRk3gkBeCMDJrnQOCC8Sko0F0RUJnrzLXOdtvDfNu7Ff+tRNb4PwrU3inbm2YJRnOoZI9vIsv/9DwsMm9d+YIIOz/7y5jLGhZ34nXzhmI6cJO92+Ve5ubhbbpKUFQAh2L1PP6A+I7jHvoWHToSaZlt+DCN4Kg+JlZuf2FXk8MeHkEc6qWWHQTFF8/ArKew== archvsync@syowa +-----BEGIN PGP SIGNATURE----- + +iHUEARYJAB0WIQRlU8JZ1ISAlY6/xTT0TupTmP8ZlgUCYmeDUwAKCRD0TupTmP8Z +lkBLAQCbcDkL77iDOtpildPE334qHcjaKlnMimpZ/MIkI4YgEQD/cKxEv6GnVuXD +Fz7eaN0W/O7fwpAFPLrpASseraXILwE= +=fayr +-----END PGP SIGNATURE----- +``` + +### Releases trigger key + +``` +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA384 + +Ubuntu Releases trigger SSH key for Canonical. + +SSH key fingerprint: ff:79:41:eb:c7:7d:00:d4:78:34:28:d1:d2:f0:ae:90 + +no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="~/releases-sync &",from="91.189.88.156,185.125.188.81" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA7l6nWM6Z2KfR4KmwF29Fv9nAgTLwHM5H/TWhinl7DZDG+Jn+TC9kll3cuyGByhwh/mNTwbyvsyiDSXFtbglowQoPSW4rhOEVy6s+/lDjDBGTDsgk8wyBqlNJRlppODsl+kqX0IqAIc3XJ9luDl894tD5rxhiXzqXL3c8r8CuhPkGdUCCMbWU4OUAIjIAs8DClYzjrAZ54IVbk5gTjDYUtlSLNXjm1rZ788h65waKBn4/LV+8nEaFIPA9FxPZI6VLmKGO/RQqZrLPNKOzotmkofV1jV2OmQNHzIwu2seV6HGYqZc3U9jE2+Eat86C6IMYS7KPxVoQd6AnHjRMlhyb6Q== archvsync@syowa + +-----BEGIN PGP SIGNATURE----- + +iHUEARYJAB0WIQRlU8JZ1ISAlY6/xTT0TupTmP8ZlgUCYmeDvgAKCRD0TupTmP8Z +ltuCAQDnzCcZ5wTOV+Pxg+YQQihC5R64kslnjVI+CnGMzLeuzQD7B5pkoeRneVJp +JR9bXlpTBqjGCTaLVRFF4faLrLef5Qk= +=cdwz +-----END PGP SIGNATURE----- +``` + +### HTTP triggers + +While SSH triggers (above) are the preferred trigger method, it is also possible +for the master Ubuntu archive to send HTTP triggers. +HTTP triggers can be nearly any format, for example: + +``` +http://ubuntu-archive:8BBsmDsLXpjJvSjM4nZv@cctld.mirrors.example.com/trigger +http://ubuntu-releases:zT99CGf9V499RH9SQFtV@cctld.mirrors.example.com/trigger +``` + +or: + +``` +http://cctld.mirrors.example.com/ubuntu-archive/53cfa724-f75d-44a1-b9d2-a14e637887bc/trigger +http://cctld.mirrors.example.com/ubuntu-releases/f4b95974-9862-40ba-a476-f751c30c5f31/trigger +``` + +The first example uses usernames and (long random) passwords to a trigger +destination, and would decide which component to begin syncing based on the +authenticated username. The second example places the component name and (long +random) UUID in the path itself. Treat these URLs as sensitive, to avoid outside +users triggering unnecessary syncs. + +It is up to the mirror administrator to build logic into the HTTP endpoint which +authenticates the master Ubuntu archive and begins a sync. This logic varies +wildly based on your specific platform. Output format is undefined; the master +simply loads the URL and moves on without examining the output. +