Change logic for keeping ts minor versions - #1173
Merged
Merged
Conversation
keep at least 5 minor versions, but ensure compatibility with the lowest/oldest PG version (where possible)
hughcapet
added
the
bugfix
Contributor
|
👍 |
1 similar comment
Collaborator
Author
|
👍 |
patagona-anas
pushed a commit
to Patagona/spilo
that referenced
this pull request
Mar 24, 2026
keep at least 5 minor versions, but ensure compatibility with the lowest/oldest PG version (where possible)
patagona-anas
pushed a commit
to Patagona/spilo
that referenced
this pull request
Mar 24, 2026
keep at least 5 minor versions, but ensure compatibility with the lowest/oldest PG version (where possible)
patagona-anas
pushed a commit
to Patagona/spilo
that referenced
this pull request
Mar 24, 2026
keep at least 5 minor versions, but ensure compatibility with the lowest/oldest PG version (where possible)
hughcapet
added a commit
that referenced
this pull request
Apr 13, 2026
- Add PostgreSQL 18 and remove PostgreSQL 13: c432557 - Patroni 4.1.0 - WAL-G 3.0.8 - PostGIS 3.6: #1153 - Remove WAL-E and make WAL-G the default backup tool: #1143 - Enable pg_basebackup server compression: #1155 - Update bg_mon, pg_mon, pg_profile, pg_permissions refs - Use pgdg for set_user and pg_permissions Bugfixes - Allow admin selecting from hypopg views: #1162 - Fix logic for keeping timescaledb minor versions: #1173 - Properly setup clean env for ext build across major versions: #1174 - Properly format ipv6 address for rsync: #1187 - Ensure CLONE_HOST is IPv6 compatible: #1190
ldming
added a commit
to apecloud-inc/spilo
that referenced
this pull request
May 22, 2026
* Sync master with the latest changes from trigger branch (zalando#1149) - wal-g 3.0.5 (not 3.0.7, see: zalando#1133) - Support deleting old backups regardless of the current backup tool - zalando#1133, zalando#1150 - Make Spilo ipv6 compatible (on AWS) - zalando#1108, zalando#1109, zalando#1112, zalando#1113, zalando#1117, zalando#1123 - remove spilo_cmd - zalando#1147 - add roaringbitmap and pgfaceting extensions - zalando#1101, zalando#1107 - add KUBERNETES_BOOTSTRAP_LABELS env for the upcoming Patroni feature - zalando#1093, zalando#1097 - fix LOG_S3_TAGS parsing logic - zalando#1093 * Sync master with trigger 13-04-26 (zalando#1196) - Add PostgreSQL 18 and remove PostgreSQL 13: zalando@c432557 - Patroni 4.1.0 - WAL-G 3.0.8 - PostGIS 3.6: zalando#1153 - Remove WAL-E and make WAL-G the default backup tool: zalando#1143 - Enable pg_basebackup server compression: zalando#1155 - Update bg_mon, pg_mon, pg_profile, pg_permissions refs - Use pgdg for set_user and pg_permissions Bugfixes - Allow admin selecting from hypopg views: zalando#1162 - Fix logic for keeping timescaledb minor versions: zalando#1173 - Properly setup clean env for ext build across major versions: zalando#1174 - Properly format ipv6 address for rsync: zalando#1187 - Ensure CLONE_HOST is IPv6 compatible: zalando#1190 --------- Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com>
cosimomeli
added a commit
to zaphiro-technologies/spilo
that referenced
this pull request
Jun 18, 2026
* Bump wal-g and pg_profile (zalando#1013) * Update tail_postgres_logs to support any log rotation plan (zalando#955) * Patroni 3.3.3 (zalando#1027) * add extra tags for uploaded Postgres log files (zalando#1024) * enable hourly log ship mode (zalando#1025) --------- Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com> * fill s3 tags before copying to log env (zalando#1031) * fill s3 tags before copying to log env * Fix wrong bool handling of LOG_SHIP_HOURLY (zalando#1030) * ensure LOG_SHIP_HOURLY env var remains a string * only allow true or empty for LOG_SHIP_HOURLY variable Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com> --------- Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com> * do not append hour on daily uploads (zalando#1033) * Postgis 3.5 (zalando#1051) * Separate trigger and OSS pipelines (zalando#1052) - Build zalando trigger image unconditionally - Add possibility to push a pr image (staging registry) on approval - Allow building master branch images on approval - Update github actions deps * PostgreSQL 17 support (zalando#1049) - Update extensions - Adjust scripts - Remove PG 12 - Patroni 3.3.4 - Use pkgs for timescaledb --------- Co-authored-by: idanovinda <idanovinda@gmail.com> * fix: wrong regex for wal retention (zalando#1026) * Integrate Patroni 4.x (zalando#1050) * Integrate Patroni v4.x * set docs * update to latest patroni version * Fix comment in tests (zalando#1060) * [Patroni4] change no_master and master as argument (zalando#1061) * Update publish pipeline deps versions (zalando#1063) * Use timescaledb 2.18.0 (zalando#1077) * Fix timescaledb installation logic (zalando#1079) - properly delete *.so for different setups - always keep the latest compatible and transition timescaledb ver without specifying in a build arg * doc: Add callout about Docker images (zalando#1090) This commit adds a call out box to the README.rst file for clarifying lack of release policy and unavailability of publicly released Docker images. * Make leader tag value for native AWS configurable (zalando#1091) * retain multiple minor versions of timescaledb (zalando#1087) Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com> * Sync master with the latest changes from trigger branch (zalando#1149) - wal-g 3.0.5 (not 3.0.7, see: zalando#1133) - Support deleting old backups regardless of the current backup tool - zalando#1133, zalando#1150 - Make Spilo ipv6 compatible (on AWS) - zalando#1108, zalando#1109, zalando#1112, zalando#1113, zalando#1117, zalando#1123 - remove spilo_cmd - zalando#1147 - add roaringbitmap and pgfaceting extensions - zalando#1101, zalando#1107 - add KUBERNETES_BOOTSTRAP_LABELS env for the upcoming Patroni feature - zalando#1093, zalando#1097 - fix LOG_S3_TAGS parsing logic - zalando#1093 * Sync master with trigger 13-04-26 (zalando#1196) - Add PostgreSQL 18 and remove PostgreSQL 13: zalando@c432557 - Patroni 4.1.0 - WAL-G 3.0.8 - PostGIS 3.6: zalando#1153 - Remove WAL-E and make WAL-G the default backup tool: zalando#1143 - Enable pg_basebackup server compression: zalando#1155 - Update bg_mon, pg_mon, pg_profile, pg_permissions refs - Use pgdg for set_user and pg_permissions Bugfixes - Allow admin selecting from hypopg views: zalando#1162 - Fix logic for keeping timescaledb minor versions: zalando#1173 - Properly setup clean env for ext build across major versions: zalando#1174 - Properly format ipv6 address for rsync: zalando#1187 - Ensure CLONE_HOST is IPv6 compatible: zalando#1190 * Keep all TimescaleDB versions --------- Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com> Co-authored-by: annielzy <148128409+annielzy@users.noreply.github.com> Co-authored-by: Felix Kunde <felix-kunde@gmx.de> Co-authored-by: idanovinda <idanovinda@gmail.com> Co-authored-by: Rob Nickmans <nickmansrob@gmail.com> Co-authored-by: Matthias Adler <macedigital@users.noreply.github.com>
stewartshea
pushed a commit
to runwhen-contrib/spilo
that referenced
this pull request
Aug 31, 2026
* Set TIMESCALEDB env variable in the final docker image (#355)
It is used by the post_init script in order to figure out if the extension needs to be upgraded.
If the env isn't set the script was trying to upgrade it on every failover/switchover even when it is not necessary.
* Add ability to specify multiple etcd hosts (#356)
* Bump dependencies (#361)
* make set_user compatible with 9.3
* plprofiler 4.1
* wal-g 0.2.12
* plpgsql_check 1.7.5
* timescaledb 1.4.2
* use DEB_PG_SUPPORTED_VERSIONS env to make package rebuild easier
* update btree bloat query
* fix zmon function
* Extend the list of wal-e SWIFT env vars to match those of wal-e, e.g. add region, domain, project, etc. (#363)
Fixes #362
* Add IPv6 support to Spilo (#364)
Fixes zalando/postgres-operator#639
* [configure_spilo] add ALLOW_NOSSL environment variable (#366)
some apps (such as Geoserver) don't support connecting to the database over SSL. This adds an environment variable `ALLOW_NOSSL` in order to allow these applications to still connect to the database.
Fixes #367
* Fix build of spilo image (#369)
it was broken due to the release of pg12.
* Switch from supervisord to runit (#370)
It provides nearly the same functionality but the memory footprint is smaller.
In addition to that make Patroni process real-time and fix a little bug with IPv6 detection.
* Run dumb-init only after decompressing image (#371)
If the launch.sh is called with init parameter it will re-execute itself via dumb-init
* Patroni needs more environment variables (#372)
For example KUBERNETES_SERVICE_* are necessary for communication with K8s API, POD_NAME and POD_NAMESPACE and needed for callback scripts, and so on.
We can't just pass all of them due to the https://github.com/zalando/patroni/commit/d770c910fdeb4420f4fd33c7f39f3f487a5cdf2b
* Fix patroni runit script and add pgbouncer (#375)
Patroni was failing to bootstrap due to the wrong HOME directory
* Fix spilo-11 build (#377)
* don't apply pgbouncer patch
* bump wal-g version
* dump plpgsql_check version
* bump decoderbufs version
Close https://github.com/zalando/spilo/issues/376
* Fix clone with pg_upgrade (#378)
it requires PGHOME env and future Patroni requires modul import in a specific order
* grant execute on pg_stat_statements_reset() to admin (#386)
enables `admin` to reset statistics at their own discretion
* add possibility to configure ETCD via url or proxy url (#382)
This adds a possibility to configure ETCD via URL or proxy as well as permits specifying SSL certificates for ETCD connection.
Fixes #381
* handle TLS certificate rotation (#373)
When the SSL_PRIVATE_KEY_FILE is not a self-generated certificate, the file can change over time. But right now postgresql doesn't know about it and won't load the new certificate automatically.
This PR introduces a bit of a polling approach to solve that problem. Every 5 minutes it will have a look if the certificate has changed within that interval and trigger a `pg_ctl reload` if yes. There are possibly smarter approaches like listen on inotify events but this has the advantage that no additional process needs to be running and running a bash script every 5 minutes is quite lite-weight.
* Patroni 1.6.2 (#389)
and fix some bugs in configure_spilo.py
* Fix bug in maybe_pg_upgrade (#390)
the interface of Config class has changed, now it accepts the file name
* Patroni 1.6.3 (#391)
and fix quoting of connstring in scripts
* Bump wal-g version (#393)
v0.2.13 => v0.2.14
* Switch to postgres 12 by default (#410)
* Create symlink $PGHOME/etc -> /run/etc (#416)
Existing standby clusters are relying on $PGHOME/etc
* Renice without root (#418)
1. Setup the renice cron job as a postgres user if the container is running with CAP_SYS_NICE. In order to be able to set negative and rt priorities without root, we set cap_sys_nice on chrt and renice tools.
2. The SUID bit on cron daemon makes it possible to start it from non-root user, what will help to simplify #414
3. Unrelated change: added tablefunc to the extwlist.extensions
* Install pglogical package for version 12 (#415)
* Install pglogical package for version 12
* Add pg-checksums
* add TIMESCALEDB_APACHE_ONLY docker build flag (#419)
The `--build-arg TIMESCALEDB_APACHE_ONLY=false` flag enables building the community timescaledb version.
* Install Plantuner from the local tarball (#412)
The plantuner repo is http only and therefore could prevent docker build in some cases due to strict access policies.
* Handle the case when value has newlines (#421)
* `printenv -0` outputs env with null byte as delimiter
* `tr` removes all newlines
* `sed` replaces null bytes with newlines
* Add pg_mon extension (#420)
Add pg_mon extension
* allow rootless user (#414)
When not run under root (UID=0):
- "$RW_DIR" and "$PGHOME" are owned by GID 0
- "/etc/passwd" is updated to reflect assigned UID for user postgres
- "write_certificates" function in 'configure_spilo.py' obtains postgres UID and sets key ownership
- SSL CERT/KEY moves to "$RW_DIR/certs"
* enable Dockerfile build without build.sh (#423)
Get rid of build.sh and scm-source.json because it is automatically injected by CDP
* Get rid of 9.3 and 9.4 (#426)
* restore launch.sh, it was removed by mistake in the #423
* remove build.sh
* bump versions of wal-g, timescaledb, debezium, plpgsql_check
* Patroni 1.6.5 (#431)
* bump Patroni version to 1.6.5
* bump debezium to v1.1.1.Final
* fix a bug in the launch.sh
* add timescaledb 1.7.0 for postgres 12 (older postgres versions stick to 1.6.1, because 1.7.0 has some bugs)
Close https://github.com/zalando/postgres-operator/issues/929
Close https://github.com/zalando/spilo/issues/417
* Add pg mon extension (#429)
* Add pg_mon extension
* Update commitid
* Added missing reference when to basebackup_fast_xlog replica method. (#435)
Close https://github.com/zalando/spilo/pull/404
* Bump bg_mon commit (#436)
and configure it to keep 2h of aggregated metrics in memory
* Bump timescaledb version to 1.7.1 (#438)
In addition to that bump pam-oauth2 to v1.0.1, there is no real changes in the code, but it will solve https://github.com/zalando/spilo/issues/433
And last, fix build with `--build-arg PGOLDVERSIONS=""`
Close https://github.com/zalando/spilo/issues/413
* Set PGOPTIONS="-c synchronous_commit=local" in post_init.sh (#440)
refactor Dockerfile a little bit (pglogical-ticker is available for 12)
Close https://github.com/zalando/postgres-operator/issues/926
* fix permission error of PGDATA directory upon container restart (#447)
Close https://github.com/zalando/postgres-operator/issues/676
* Added pgaudit extension. (#448)
* Helper script for pgq (#454)
We want to make it possible to grant pgq_* roles by mebers of admin role
* Update dependencies (#464)
* timescaledb 1.7.2
* pg_mon cc028fdae8542ec3f5df3bf4c66f0895d87c127d
* plpgsql_check 1.11.0
* plantuner 800d81bc85da64ff3ef66e12aed1d4e1e54fc006 (pg13 support)
* Don't strip TIMESCALEDB_LEGACY env (#466)
it is used by post_init.sh script
* Don't use pspg as pager (#470)
* Enable coredumps (#475)
To facilitate troubleshooting, enable coredumps. Unfortunately by
default a coredump will take snapshot of the whole shared memory of a
process, which in case of PostgreSQL could mean gigabyres of shared
buffers. To prevent that, also configure coredump_filter to exclude
shared memory from a dump. See [1] for more details.
[1]: https://www.kernel.org/doc/html/latest/filesystems/proc.html#proc-pid-coredump-filter-core-dump-filtering-settings
* use AWS_REGION when provided (#478)
Close #477
* Make sure created certificate are owned by postgres user
Close #479
* Minor postgres releases (#481)
and install plpgsql-check from pgdg packages
* do not sum lists (#484)
We want to be able to overwrite default configuration.
* Changes configure script to use OpenStack metadata directly. (#486)
Splits AWS and OpenStack metadata fetching.
Adds OpenStack specific fetching from http://169.254.169.254/openstack/latest/meta_data.json
Like GCP OpenStack's metadata does not expose IPv4 addresses so the auto-discovered one used.
Close https://github.com/zalando/spilo/issues/485
* Patroni 2.0 (#487)
* wal-e 1.1.1
* wal-g 0.2.17
* timescaledb 1.7.3
* refactor DCS configuration (close #468)
* Housekeeping (#489)
* remove useless callback_endpoint.py
* add exponential back-off to callback_role.py
* enable linting job with GitHub actions:
1. check shell scripts with shellcheck
2. check python code with flake8
* run integration tests on GitHub actions and CDP
* Disable prefetch when restore_command is called for pg_rewind (#495)
We use the fact that `%p` parameter contains either `RECOVERYHISTORY` or `RECOVERYXLOG`, but in case of pg_rewind the file is being restored directly to `pg_wal/%f`
If it is detected, the following is done:
* export WALG_DOWNLOAD_CONCURRENCY=1 environment for wal-g
* Use -p 1 option for wal-e
In addition to that update README.rst and bump some dependencies.
* Fix problems with build (#499)
* Don't install/purge libpq with pinned version. It was required only for building pg_rewind on 9.3 and 9.4.
* Fix Patroni install
* Clean up post_init.sh
* Added Extension pg_permission (#493)
Pin the commit id (tags are inconsistent)
Co-authored-by: Alexander Kukushkin <cyberdemn@gmail.com>
* Fixed requirements in pg_view (#502)
Close https://github.com/zalando/spilo/issues/501
* Pg permission fix (#504)
* Added Extension pg_permission
* Fix docker build (#509)
Close https://github.com/zalando/spilo/issues/508
* Fix S3 performance issues (#515)
https://github.com/wal-e/wal-e/commit/485d834a18c9b0d97115d95f89e16bdc564e9a18 introduced `gevent.monkey.patch_thread()` that makes S3 performance ridiculously slow.
* Added support for sending WAL to Azure via wal-g
Close https://github.com/zalando/spilo/issues/383
* Support for additional locales (#513)
* Support for additional locales
* Fixes after review
* Add PostgreSQL 13 support and in-place major upgrade (#520)
How to trigger upgrade? This is a two-step process:
1. Update configuration (version) and rotate all pods. On start, configure_spilo will notice version mismatch and start the old version.
2. When all pods are rotated exec into the master container and call `python3 /scripts/inplace_upgrade.py N`, where N the capacity of the PostgreSQL cluster.
What `inplace_upgrade.py` does:
1. Safety checks:
* new version must be bigger than the old one
* current node must be running as a master with the leader lock
* the current number of members must match with `N`
* the cluster must not be running in maintenance mode
* all replicas must be streaming from the master with small lag
2. Prepare `data_new` by running `initdb` with matching parameters
3. Run `pg_upgrade --check`. If it fails - abort and do a cleanup.
4. Drop objects from the database which could be incompatible with the new version (e.g. pg_stat_statements wrapper, postgres_log fdw)
5. enable maintenance mode (patronictl pause --wait)
6. Do a clean shutdown of the postgres
7. Get the latest checkpoint location from pg_controldata
8. Wait for replicas to receive/apply latest checkpoint location
9. Start rsyncd, listening on port 5432 (we know that it is exposed!)
10. If all previous steps succeeded call `pg_upgrade -k`
11. If pg_upgrade succeeded we reached the point of no return!
If it failed we need to rollback previous steps.
12. Rename data directories `data -> data_old` and `data_new -> data`
13. Update configuration files (postgres.yml and wal-e envdir).
14. Call CHECKPOINT on replicas (predictable shutdown time).
15. Trigger rsync on replicas (COPY (SELECT) TO PROGRAM)
16. Wait for replicas rsync to complete. Feedback status is generated by `post-xfer exec` script. Wait timeout 300 seconds.
17. Stop rsyncd
18. Remove the initialize key from DCS (it contains old sysid)
19. Restart Patroni on the master with the new configuration
20. Start the local postgres up as the master by calling REST API `POST /restart`
21. Memorize and reset custom statistics targets.
22. Start the ANALYZE in stages in a separate thread.
23. Wait until Patroni on replicas is restarted.
24. Disable maintenance mode (patronictl resume)
25. Wait until analyze in stages finishes.
26. Restore custom statistics targets and analyze these tables
27. Call post_bootstrap script (restore dropped objects)
28. Remove `data_old`
29. Trigger creation of the new backup
Rollback:
1. Stop rsyncd if it is running
2. Disable maintenance mode (patronictl resume)
3. Remove `data_new` if it exists
Replicas upgrade with rsync
---------------------------
There are many options on how to call the script:
1. Start a separate REST API for such maintenance tasks (requires opening a new port and some changes in infrastructure)
2. Allow `pod/exec` (works only on K8s, not desirable)
3. Use COPY TO PROGRAM "hack"
The `COPY TO PROGRAM` seems to be low-hanging fruit. It requires only postgres to be up and running, which is in turn already one of the requirements for the in-place upgrade to start. When being started, the script does some sanity checks based on input parameters.
There are three parameters required: new_version, primary_ip, and PID.
* new_version - the version we are upgrading to
* primary_ip - where to rsync from
* PID - the pid of postgres backend that executed COPY TO PROGRAM.
The script must wait until the backend will exit before continuing. Also the script must check that its parent (maybe grandparent?) process has the right PID which is matching with the argument.
There are some problems with `COPY TO PROGRAM` approach. The Patroni and therefore PostgreSQL environment is cleared before start. As a result, the script started by postgres backend will not see for example `$KUBERNETES_SERVICE_HOST` and won't be able to work with DCS in all cases.
Once made sure that the client backend is gone the script will:
1. Remember the old sysid
2. Do a clean shutdown of the postgres
3. Rename data directory `data -> data_old`
4. Update configuration file (postgres.yaml and wal-e envdir). We do it before rsync because the initialize key could be cleaned up right after rsync was completed and Patroni will exit!
5. Call rsync. If it failed, rename data directory back.
6. Now we need to wait for the fact that the initialize key is removed from DCS. Since we know that it happens before the postgres on the master is started we will try to connect to the master via replication protocol and check the sysid.
7. Restart Patroni.
8. Remove `data_old`
In addition to that, implement integration tests. Mostly they are testing happy-case scenarios, like:
1. Successful in-place upgrade from 9.6 to 10
2. Successful in-place upgrade from 10 to 12
3. Major upgrade after the custom bootstrap with wal-g
4. Major upgrade after the custom bootstrap with pg_basebackup
5. Bootstrap of a new replica with wal-g
Also tests are covering a few unhappy cases, like: in-place upgrade doesn't start if pre-conditions are not meet.
* Use an IP address for Google instance metadata (#521)
* Use an IP address for Google instance metadata
Makes Spilo start even if external DNS queries from
within the container are not working.
Do not continue with the startup when /etc/service is missing.
Per https://github.com/zalando/spilo/issues/511
* Follow Python guidelines on on-line comments
Co-authored-by: Alexander Kukushkin <cyberdemn@gmail.com>
* Extra space to the shell comment for readability
Co-authored-by: Alexander Kukushkin <cyberdemn@gmail.com>
* fix clone from Azure (#523)
* Fix a few minor issues (#524)
1. Restore `ETCD_DISCOVERY_DOMAIN` -> `etcd.discovery_srv` mapping
2. Compatibility with https://github.com/zalando/patroni/pull/1788
3. `python3-requests`, `postgresql-${version}-pg-stat-kcache`, `postgresql-${version}-cron`, and `postgresql-${version}-pgq3` are required for DEMO spilo to work.
4. Don't install `pv`, `lzop`, `etcd`, and `etcdctl` in DEMO mode
5. Don't remove `pg_recvlogical` binary, it might be useful for playng with logical replication/decoding.
* Postgis 3.1, timescale 2.0 and a few optimizations in relinking of extensions/contribs (#535)
1. Fix build broken by postgis 3.1 release
- use postgis 3.1 as main version
- use postgis 3.0 as legacy version (postgres 9.5)
- fix relinking code to support legacy postgis and optimize it to support relinking between all major versions, not only the current one.
2. Bump version of timescaledb to 2.0
- change cmake requirement from 3.11 to 3.11 in CMakeLists.txt. The old version available in ubuntu 18.04 perfectly works.
- call bootstrap with -DWARNINGS_AS_ERRORS=OFF
- 9.6 and 10 will continue using 1.7.4, therefore optimized relinking code in the commit updating postgis becomes very handy.
3. A bit unrelated, but since this PR already does a lot of housekeeping, optimize the code defining/cleaning locales in order to reduce the number of repetitions.
Close: #528, #529, #534, #533
* Fix metadata service for OpenStack (#526)
Close #525
* Try to parse response to ensure it's parsable. If not, assume local provider. (#543)
Close #542
* Set AWS_REGION whenever USE_WALG_BACKUP is set to true (#546)
Since wal-g supports having both AWS_ENDPOINT (or WALE_S3_ENDPOINT) and AWS_REGION and that we cannot determine region from endpoint in the event of an onpremise deployment, this PR allows setting AWS_REGION even if AWS_ENDPOINT (or WALE_S3_ENDPOINT) is specified.
This way using an onpremise deploy with a custom region (http://someminioinstance:9000/ and my-place, ie) or using an endpoint that does not match the regex engine (https://somebucket.s3.nl-ams.scw.cloud, ie)
Close: https://github.com/zalando/spilo/issues/539
* Run CHECKPOINT on replicas in parallel with pg_upgrade (#545)
Major upgrade with replica rsync requires all replicas being up-to-date with the primary and shutdown cleanly. It is possible to make the shutdown process faster by executing the CHECKPOINT immediately before running `pg_ctl stop -m fast`.
Before this commit, the CHECKPOINT and shutdown on replicas were performed only after the pg_upgrade was successfully executed.
The time for CHECKPOINT on replicas and therefore time for the shutdown is quite measurable and comparable with the time which is required to execute the pg_upgrade.
By doing these processes (pg_upgrade and CHECKPOINT on replicase) in parallel we reduce the downtime that is required for pg_upgrade + rsync.
The old diagram:
```
+-> r1 CHECKPOINT + +-> r1 rsync +
SHUTDOWN PRIMARY -> pg_upgrade -|-> r2 CHECKPOINT |->-|-> r2 rsync |->
+-> r3 CHECKPOINT + +-> r3 rsync +
```
The new diagram:
```
+-> pg_upgrade +
|-> r1 CHECKPOINT | +-> r1 rsync +
SHUTDOWN PRIMARY -|-> r2 CHECKPOINT |->-|-> r2 rsync |->
+-> r3 CHECKPOINT + +-> r3 rsync +
```
In addition to that fix corner-case with upgrading a single-node cluster.
Also, the `registry.opensource.zalan.do/acid/spilo-cdp-13` and `registry.opensource.zalan.do/acid/spilo-13` become based on `registry.opensource.zalan.do/library/ubuntu-18.04. If someone concerned, the last one is mostly a mirror of ubuntu:18.04 from the dockerhub with a couple of layers on top. One of them updates packages and another adds zalando-marker file containing some "random" string. The FROM image is overridden in the delivery.yaml. If someone builds the image without build args supplied, the default ubuntu:18.04 is used.
* Bump the Wal-G version to 0.2.19 and add WALG SSH support (#549)
Bump the Wal-G version to 0.2.19 and add variables for the WALG_SSH_PREFIX for backups
Not adding SSH_PASSWORD as password authentication is generally not recommended.
The SSH transport functions using sftp://, meaning that the server side could
be set up in a completely restricted manner in sshd_config:
Match User spilo:
ChrootDirectory /srv/backup/spilo
ForceCommand internal-sftp
AuthenticationMethods publickey
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no
With the matching setup in the spilo container could be:
SSH_PRIVATE_KEY_PATH=/etc/patroni/ssh_key
SSH_USER=patroni
WALG_SSH_PREFIX=sftp://backup.example.com/spilo
Fixes: zalando/spilo#548
* Reverted uid/gid detection to pwd rather than home directory ownership (see #551) (#555)
Fixes an issue (see #551) with Spilo running in OpenShift when the UID is a random number picked by OpenShift. This PR reverts the method of determining what uid/gid to use back to checking the `/etc/passwd` file rather than the ownership of the `PGHOME` directory.
Verification that it works:
* Tests all pass
* `docker build -t spilo . && docker run --user=1500080000:0 spilo` is able to start the database with no issues
* Various improvements (#553)
1. Fix failing in-place major upgrade due to the amcheck_next
2. Make sure bg_mon is not in the shared_preload_libraries during pg_upgrade.
3. Bump bg_mon commitid
4. Run renice as root (when possible)
5. Change self-signed certificate domain from dummy.org to example.org
6. Bump Timescaledb version to 2.0.1
7. Silence annoying NOTICE messages about deprecated set_user GUCs
8. Fully revert wal-e [commit](https://github.com/wal-e/wal-e/commit/485d834a18c9b0d97115d95f89e16bdc564e9a18) causing performance issues with S3
9. Analyze all databases after promote
10. Patroni 2.0.1
11. Last, but not least - new PostgreSQL minor releases
* Fix bug with major upgrade after clone (#557)
When working on the code removing bg_mon from shared_preload_libraries before executing pg_upgrade a little bug was introduced. Specifically, shared_preload_libraries for the old version was overwritten by the value taken from the new version. As a result, the old cluster was failing to start due to missing (not existing) libraries and upgrade was failing.
This commit is fixing the wrong behavior and improves tests to catch similar issues in the future.
* Improve analyze and update dependencies (#558)
* Run analyze in all databases in the cluster in parallel
* Bump timescaledb to 1.7.5 and 2.1.0 and enable it for postgres 13
* Bump decoderbufs version
* Bump Patroni version
* Fix issue with analyze after upgrade (#559)
ensure we don't divide by zero
* Bump bg_mon commit id (#560)
It contains fixes of wal and pressure stats.
* Follow symlink for certificate renewal check (#561)
Particularly kubernetes is fond of having a symlink when mounting from a secret or config-map. This is a very common pattern with for example certmanager.
The `-L` flag tells stat to follow symlinks.
A typical file structure could look like this
```
postgres@gitlab-postgresql-cluster-0:~$ ls -l /tls/
total 0
lrwxrwxrwx 1 root root 13 Feb 2 08:24 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 14 Feb 2 08:24 tls.crt -> ..data/tls.crt
lrwxrwxrwx 1 root root 14 Feb 2 08:24 tls.key -> ..data/tls.key
postgres@gitlab-postgresql-cluster-0:~$ ls -l /tls/..data
lrwxrwxrwx 1 root postgres 31 Feb 28 02:15 /tls/..data -> ..2021_02_28_02_15_21.040592773
postgres@gitlab-postgresql-cluster-0:~$ ls -l /tls/..data/
total 12
-rw-r----- 1 root postgres 1922 Feb 28 02:15 ca.crt
-rw-r----- 1 root postgres 3646 Feb 28 02:15 tls.crt
-rw-r----- 1 root postgres 1675 Feb 28 02:15 tls.key
```
Without the `-L` flag stat would show "Feb 2 08:24" from the symlink instead of "Feb 28 02:15" from the updated certificate. This in essence makes the cert check useless for any configuration where the cert is mounted as a kubernetes secret or configMap. As far as I can see at least.
* Update postgres_backup.sh usage (#572)
Minor update to the usage string in the postgres_backup.sh script.
In this [commit](https://github.com/zalando/spilo/commit/84668553b9cda6ba9da199ff3b3fe9fd89cf5aec#diff-12f469d988840b9eb6a93e31e87d9160d606abcdca4e47d6c9224e3df86335f0R14) the way that `DAYS_TO_RETAIN` is set was changed from command line argument to reading from the env var `BACKUP_NUM_TO_RETAIN`.
However, the usage string for the script was not updated to reflect the change, this commit fixes that.
* Add backend_type column to postgres_log if it is missing on postgres 13 (#573)
Due to the bug in the upgrade script postgres_log table wasn't dropped before pg_upgrade. As a result it wasn't recreated with the new structure.
In order to fix it, the post_init.sh running against Postgres 13 script will always try to add a column with NOT EXISTS.
At the same time we simplify the upgrade script. We don't need to drop the postgres_log foreign table before the upgrade.
* Bump dependencies (#579)
* timescaledb 2.2.0
* latest bg_mon commit id
* Propagate WALG_S3_CA_CERT_FILE (#578)
Close #577
* Make it possible to run cron without a root (#574)
* Don't set suid bit on `/usr/sbin/cron` (even if it could be used there is no way to safely create the root cronjob, but it also means that the `sys_nice` capability which is set on `chrt` and `renice` could be used).
* Use `LD_PRELOAD` trick for hooking `seteuid()` and `initgroups()` functions when starting cron deamon without a root.
* Enhance scheduling (or not scheduling) the renice cronjob depending on the initial conditions (root, sys_nice, no-new-privileges).
* Disable unwanted cronjobs (`/etc/crontab`, `/etc/cron.{hourly,daily,...}`), they produce "scary" logs when cron is running without a root.
Close: https://github.com/zalando/spilo/issues/562
* Fix clone_with_wale on pg13+ when recovery_time in future (#580)
PostgreSQL 13 changed its behavior when the recovery target is unreachable. While old versions were finishing the recovery and
promoting Postgres, starting from 13 Postgres complains "recovery ended before configured recovery target was reached" in logs and exits. As a result, the clone completely fails.
In order to retain the same behavior as with old versions, the `maybe_pg_upgrade.py` will always start point-in-time recovery (even if there is no major upgrade required).
If it failed, for PostgreSQL 13+ it will check a few last log lines and if they contain the error about the recovery target not reached,
it will reset all recovery targets and start the postgres up one more time. The next failure is fatal.
* Add support for WAL-E/WAL-G with custom PGPORT (#582)
Close #571
* Bump timescaledb (#583)
and prepare the new minor release
* Update StatefulSet apiVersion in Kubernetes manifest (#581)
- Update deprecated apps/v1beta1 API.
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#statefulset-v116
- Add a simple test for installation on Kubernetes.
- Format affinity JSON.
* Fix issue with PITR (#584)
if the PITR performed by the maybe_pg_upgrade.py script and not followed by the major upgrade the archive_mode was set to off.
* Patroni 2.1.0 (#595)
and bump a few other dependencies
* Compatibility with Patroni 2.1 (#596)
* Calculate the shutdown checkpoint location in the inplace_upgrade script
* Make sure we use the pg_controldata from the correct major version
* Provide a better example for ETCD_HOSTS (#601)
* Fix wrong envvar name for AZURE_STORAGE_ACCESS_KEY (#598)
Regarding the wal-g documentation [1], the envvar for Azure storage access key is AZURE_STORAGE_ACCESS_KEY
[1] https://github.com/wal-g/wal-g/blob/908e18f8b1f3ea319130b69508a10c5dbf8d7b01/docs/STORAGES.md#azure
* #585 certificate rotation based on file hash, not modified time (#599)
Close https://github.com/zalando/spilo/issues/585
* Add timescaledb 2.4.0 (#607)
* build all possible latest major versions of timescaledb for every major version of Postgres (sometimes they are required for major upgrade)
* change post_init.sh script to not rely on TIMESCALEDB* env variables for determining whether the timescaledb extension should be updated
* Grant execute on pg_switch_wal to admin (#600)
To enable members of `admin` call this function after they have stopped writes to their DB but before cloning or promotion of a standby leader takes place.
* Bump up pg_auth_mon (#608)
* Prepare new release (#612)
* Prepare new release
* bump Patroni version
* don't rebuild pgbouncer
* Bump set_user
* Timescaledb 2.4.1
* Bump bg_mon commit id (#613)
for compatibility with latest minor releases
* A couple of minor features/fixes (#614)
* Use unix sockets for local replication connections
* Do not try to truncate anything that is not a normal unlogged table
* Update WAL-G to v1.1 (#588)
WAL-G has a new stable release that game out a few days ago.
https://github.com/wal-g/wal-g/releases/tag/v1.1
This commit does the following:
* Updates the Dockerfile to use the new v1.1 release.
* Adds the new Azure functionality provided in this release.
* Adds compatibility with v1.0+ to clone_with_wale.py
* Fix wale-restore script to handle wal-g 1.1 header (#615)
* Updates the regex to support either `modified` or `last_modified` in the header returned by `$WAL_E backup-list`. Ref https://github.com/zalando/spilo/pull/588#issuecomment-911319812
* Adds explicit google-crc32c version 1.1.2 to work around broken 1.1.3
* Stick to the old minio (#628)
wal-g hangs with new releases
* Document SSL_CERTIFICATE and SSL_PRIVATE_KEY and add SSL_CA and SSL_CRL environment variables (#625)
Co-authored-by: Salvatore D'Angelo <sdangelo@it.ibm.com>
* Add Patroni REST Api SSL certificates environment variables (#629)
* Fixed postgres_backup.sh script for wal-g 1.1 (#632)
Fixed the `sed` command in the `postgres_backup.sh` script to handle wal-g 1.1 headers
* docs: update with full list of wal-g env vars (#635)
close #531
* Define RESTAPI_CONNECT_ADDRESS environment variable (#640)
Spilo by default uses IP for the property ```restapi.connect_address```. This is a problem when Patroni REST API is configured in SSL with client authentication because hostname validation is performed. Due to the dynamic nature of Pod, it's impossible to create SSL certificates with the right IP into SAN.
This PR solves the problem with the introduction of the property: REST_API_CONNECT_ADDRESS.
You can configure it, for example, like this:
```
- name: RESTAPI_CONNECT_ADDRESS
value: "$(POD_NAME).<service name>"
```
so that it uses the <pod name>.<service name> stored in CoreDNS.
* Postgresql 14 (#644)
- removed PostgreSQL v9.5
- removed a couple of extensions installed from pgxn (they fail to compile on 11+ anyway)
* Drop pg_repack extension before major upgrade (#646)
and enable wal_compression by default (it is a must when running with page checksums)
* Install pg-repack from package repository (#651)
Fixes #650
* Add support for wal-g encryption environment variables (#649)
Close #636
* Add more extensions to PostgreSQL v14 (#655)
- pglogical
- pldebugger
- timescaledb 2.5.0
- latest bg_mon
- debezium v1.7.0
In addition to that fix postgres_log_* foreign tables, pg14 added two more fields to csv: leader_pid and query_id
* Next minor releases (#659)
and fix pgextwlist compile issues
Close https://github.com/zalando/spilo/issues/658
* Fix typo in function name: append_extensions (#665)
Typos are making it harder when looking for relevant pieces of code.
* change build repos (#662)
per agreement with @RafiaSabih
* Add wal-g proxy support (#661)
Add HTTP(S) proxy support for `wal-g` allowing access to public S3 buckets.
Signed-off-by: Stephan Austermühle <au@hcsd.de>
Closes https://github.com/zalando/spilo/issues/660
* docs: WAL_S3_BUCKET is not s3:// path (#669)
We will add "s3://" to the *_PREFIX \[1], so it should not appear in WAL_S3_BUCKET.
Besides, "Spilo will add /spilo/scope/wal to that path" is not true now (we added PGVERSION). So just remove it.
\[1]: https://github.com/zalando/spilo/blob/9f37ab473caea1e4420d8f4da8773a4938d9310b/postgres-appliance/scripts/configure_spilo.py#L830
* set AWS_S3_FORCE_PATH_STYLE accordingly (#670)
WAL-G will not work if we use our own endpoint that only supports path convention but AWS_S3_FORCE_PATH_STYLE is not set. This patch ensures that WAL-G and WAL-E will use the same convention.
Also add support to WALE_S3_ENDPOINT with convention other than path.
* Patroni 2.1.2 and fix small oversights with pg_cron and timescaledb (#667)
* Patroni 2.1.2
* Timescaledb 2.5.1
* Add timescaledb to shared_preload_libraries on v14
* Rename cron.schedule(text, text, text) to schedule_in_database. Extension introduced the new function with exactly the same signature.
* Grant/revoke permissions on new functions introduced by pg_cront
* Adjust cron_job_run_details_policy RLS
* Add build support on non-amd64 platforms (#664)
the biggest problem was the lack of non-amd64 binaries of wal-g which we solve by introducing yet another stage.
So far it was tested only on arm64.
* bump PostGIS to 3.2 (#675)
* Various fixes (#679)
1. Ensure files copied from git repo are not world-writable. The problem seems to be very specific to Zalando CPD, that possibly does git clone with umask=0000
2. Read-only rootfs: create symlinks only to corresponding runit directories instead of individual `run` and `finish` files.
3. Fix bug with the cron daemon running from the postgres user. While the daemon was running since https://github.com/zalando/spilo/commit/c91248e26e2ea910304d04a3acbeda1e965e2e42#diff-dd039299d1b892e68ab0ea5e22d533814e1c349a290b5cc46b9d823578f6b741, individual jobs were failing to start because
PAM was failing to read /etc/shadow.
4. Run "arm64" build on the "large" instance. Having more CPU cores available significantly speeds up the build.
* Add SSH_PORT environment vairable for wal-g (#676)
* Add first-last-agg extension (#680)
fixes #672
* Try to get ip from OpenStack EC2-compatible API (#684)
OpenStack metadata service supports two sets of APIs: an OpenStack metadata API and an EC2-compatible API, The metadata service has an API that is compatible with version 2009-04-04 of the Amazon EC2 metadata service. We should try get instance ip from it first.
* Support AZURE_ENVIRONMENT_NAME for Azure Gov (#688)
Need to persist this env var `AZURE_ENVIRONMENT_NAME` for Azure gov to work. It helps the azure sdk select the correct endpoints.
* Install decoderbufs from pgdg package (#689)
and add a few more interesting extensions:
- pgq_node
- hll
- dirtyread
- extra-window-functions
* Implement non-exclusive backups with wal-e (#686)
It makes it redundant to remove the backup_label file from the configure_spilo.py
* Make sure the initialize key with the wrong sysid is removed (#696)
As a part of major upgrade we remove the initialize key from DCS because the upgraded cluster always gets the new system identifier.
But, there is a chance that Patroni could restore a previous value before it was restarted. Therefore, the upgrade script removes the initialize key the second time right after Patroni was restarted. In most cases (and for the most DCS) it worked fine. The only exception is K8s (and maybe Etcd3), because the delete happens based on the cached state, which is updated asynchronously from watch events.
We can't completely eliminate the race condition, but at least we can significantly reduce the chance of getting the cluster to a broken state by doing yet another attempt of removing the initialize key before resume.
Last, but not least, we also reduce chances of using the stale cache by tuning tcp timeouts and keepalives.
* add the cron_admin role (#699)
And bump Patroni version.
* Adding the WALG_LIBSODIUM_KEY_TRANSFORM environment variable (#706)
The `WALG_LIBSODIUM_KEY_TRANSFORM` [variable](https://github.com/wal-g/wal-g#encryption) wasn't passed through Spilo's wal-g env directory.
Extends https://github.com/zalando/spilo/pull/649
* Fix different files permissions (#707)
- allow read/execute by everyone for runit run files
- change owner to root for etcd and etcdctl
In addition to that bump Timescaledb to 2.6.0 and allow custom settings of `WALE_DISABLE_S3_SSE` and `WALG_DISABLE_S3_SSE`
Close https://github.com/zalando/spilo/issues/697
Close https://github.com/zalando/spilo/issues/701
Close https://github.com/zalando/spilo/issues/704
* etcd, tar xz: --no-same-owner. For "rootless" Jenkins job to work. (#711)
Close https://github.com/zalando/spilo/issues/708
* Bump timescaledb to 2.6.1 (#717)
It has many bugfixes and compatibility with the old cmake.
Ref https://github.com/zalando/spilo/issues/714
* Fix demo build (#721)
It needs at least one deb file to copy and it seems that pg_permission repo was renamed to pg_permissions (plural)
* Make it possible to specify timeout for the restore_command (#723)
* Add timeline_id parameter for restore (#724)
TIMELINE_ID is added as a dynamic parameter to enable selection of timeline id for recovery.
Close: https://github.com/zalando/spilo/issues/712
* Bump walg to v2.0.0 (#729)
It is worth noting that v2.0 Postgres delta backups are not backward-compatible
with the pre-2.0 versions.
* Bump bg_mon and timescaledb (#732)
fix build by limiting protobuf version, it recently stopped supporting python 3.6
* Patroni 2.1.4 (#737)
- ensure all packages are upgraded to the latest version (apt-get -y upgrade)
- allow starting etcd on non-x86 platforms
- refactor `get_dcs_config()` function
Close https://github.com/zalando/spilo/issues/736, https://github.com/zalando/spilo/issues/726
* Bump bg_mon commitid (#751)
and add Polina as the maintainer
* Change source repos for the dependencies maintained by Zalando (#756)
* Added pg_profile extension (#752)
* Disable wal_archiving during major upgrade (#762)
We don't want WAL files to be archived to the location used by the old cluster (old version).
And bump bg_mon commit id.
* Bump timescaledb to 2.7.2 (#766)
and build images with latest postgres minor releases
* Make it possible to set a slot for standby cluster (#754) (#755)
* Bump wal-g to v2.0.1 (#767)
* Pass AWS EKS IRSA environment variables to WAL-G (#769)
* Drop pg_repack before upgrade --check (#773)
* Change ubuntu version in tests workflow, fix delivery.yaml (#774)
- Change jobs.tests.runs-on ubuntu version
- Use vm_config.type: linux
- Adjust scripts for a more strict shellcheck version
* bump up pg_auth_mon (#778)
* Support Service Principal for wal-g backup to Azure (#779)
Support Service Principal for wal-g backup to Azure
Signed-off-by: Stephan Austermühle <au@hcsd.de>
* Bump timescaledb to 2.8.0 (#780)
* Update configure_spilo.py (#784)
* Remove pgextwlist extension patching (#786)
* Bump timescaledb to 2.8.1 (#785)
* Pass AWS STS regional endpoints environment variable to WAL-G (#789)
* adds workflow for multiarch builds (linux/amd64, linux/arm64) being published to ghcr.io (#790)
* Add multiarch ghcr builds
* Update postgres-appliance/tests/README
* [WIP] Adapt test_spilo.sh for podman (#744)
Create an alias to docker
* Bump github actions to the latest versions (#797)
* Install pgextwlist unconditionally (#800)
* PostgreSQL 15 & Ubuntu 22.04 (#794)
* Add PostgreSQL 15 support
* Remove PostgreSQL v9.6
* Enable build and push of spilo-cdp-15 (#742)
* Split main Dockerfile into .sh scripts (#746)
* Migrate to Ubuntu 22.04 (#770)
Co-authored-by: Alexander Kukushkin <akukushkin@microsoft.com>
* Partially revert dev changes in delivery.yaml (#809)
* Update pg_mon commit hash (#813)
* Bump plprofiler version (#816)
* Update timescaledb version (#820)
timescaledb 2.9.1
* Return accidentally removed pg_profile (#822)
* Return accidentally removed pg_profile
* Update pg_profile version
* Add USE_OLD_LOCALES test (#821)
The newly added test verifies that with the old locales (Ubuntu
18.04) kept postgres sorting order is actually preserved on Ubuntu 22.04.
The test is only run on push to master branch.
* Update Patroni version to 2.1.5 (#823)
* PyYAML 6.0 compatibility (#825)
Switch to safe_load(), as load() is unsafe and since pyyaml 6.0 always requires loader to be specified
* Enforce search_path=pg_catalog for post_init and wale_restore scripts (#826)
* Add extension timescaledb-toolkit (#818)
* add function to list last status of active cron jobs (#824)
add function to list last status of active cron jobs
* Don't try building timescaledb for PG15 (#828)
* Bump Patroni version to 3.0.0 (#833)
* Bump timescaledb to 2.9.2 (#834)
* Move active_cronjobs helper function and view to zmon_utils schema (#832)
* Bump Patroni version to 3.0.1 (#847)
* Bump timescaledb to 2.9.3 (#846)
* Use relevant params in pgqd ini file (#842)
* Build custom plprofiler (#852)
* Bump timescaledb to 2.10.0 (#851)
* Bump timescaledb to 2.10.0
* Revert "Don't try building timescaledb for PG15 (#828)"
* CDP multi-arch image build (#830)
* Change vm type
* Add an optional step to build arm64 image
* Increase cdp pipeline timeout (#859)
* Install pg_cron in pg_catalog schema, exit on error in post_init script (#863)
* Patroni 3.0.2 and timescaledb 2.10.1 (#865)
* Install a patched pg_cron version (#866)
* Install a patched pg_cron version (update commit) (#867)
* Add support for S3 compatible backend to logsender (#765)
Add support for S3 compatible backend to logsender
- `LOG_S3_ENDPOINT` env variable added to placeholders.
- `Multipart_upload` supported natively by boto3 and custom code is no longer needed.
Chunk size passed to the library via `TransferConfig`.
- `AWS_REGION` populated as `LOG_AWS_REGION` and will be passed to boto3.
* Timescaledb 2.10.2 (#875)
* Full support of timescaledb for pg15 (#873)
* Bump timescaledb to 2.10.3 (#878)
* pg_profile 4.2, pg_cron from pgdg (#884)
* pg_profile 4.2
* Install pg_cron from pgdg packages
* Timescaledb 2.11.0 (#885)
* Update a couple of 3rd party things (#893)
- Patroni 3.0.4
- TimescaleDB 2.11.1
- set_user 4.0.1
- latest bg_mon commit (16beta1 support)
* Add pgvector extension (#896)
* Don't reset defaults in pgq_ticker.ini (#901)
* Refactor cdp pipeline (#903)
* All tests are in gh actions
* Only push to ecr and tag in the cdp pipeline (with human approval)
* cdp pipeline is only run on push to the master branch
* add new code owner (#910)
Co-authored-by: Jociele Padilha <jociele.padilha@zalando.de>
* Bump PostGIS to 3.4 (#904)
* Bump PostGIS to 3.4
* fix PostGiS file relinking
---------
Co-authored-by: idanovinda <idanovinda@gmail.com>
* Change maintainer to team ACID (#911)
* Tests housekeeping (#845)
* Various tests fixes
- Try both gnu and non-gnu 'date' command syntax
- Cleanup all test containers
- Fix scope name
* Fix flaky tests:
- wait backup before upgrade
- drop table with OIDs in upgrade container
Co-authored-by: Max Neverov <neverov.max@gmail.com>
---------
Co-authored-by: Max Neverov <neverov.max@gmail.com>
* Latest bg_mon commit to fix cgroup stats values (#917)
* Fix tests cleanup script (#916)
* Setup default image when cleaning up containers (#918)
* setup default image when cleaning up containers
* update readme to run test
* Add md extension to tests README
---------
Co-authored-by: Polina Bungina <bungina@gmail.com>
* Update Patroni and timescaledb versions (#912)
* Patroni 3.1.0, timescaledb 2.11.2
* Align with the new Patroni code API
* Patroni 3.1.2 (#924)
* Return pod_ip param to k8s config in Patroni (#931)
* add replay-lag function (#932)
Co-authored-by: Sergey Dudoladov <sergey.dudoladov@gmail.com>
* Refactor tests (#933)
- wait_all_streaming/wait_zero_lag with configurable replica count
- create table with oids using special func, not in the initial schema
- add pg_repack extension to the initial schema
- separate func for creating timescaledb objects
- set WALE_BACKUP_THRESHOLD_PERCENTAGE to a higher value to ensure wal-e
replica bootsrap method in test suite 5
- single verify_clone_upgrade func
- adjust comments and formatting
- ensure test suite 4 actually restores latest backup from PG10 and then upgrades
* Add toggle to enable grouping log by date/instance in s3 (#941)
* add toggle to enable grouping log by date
* add information in environment.rst
* Update ENVIRONMENT.rst
Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com>
---------
Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com>
* Remove redundant code (#948)
* No pg_partman update from <5.X during major upgrade (#962)
Due to the noticeable number of backward-incompatible changes in
pg_parman 5.0, we better force users to prepare and check everything
before actually upgrading the extension. So if performing major upgrade
with pg_partman pre 5.X installed, we skip automatic update of it
* add function to monitor sequences (#940)
* Ensure correct bin_dir for pg params configuration (#966)
Make in-place upgrade code use the bin_dir of the proper version
throught the whole process, so that
ConfigHandler.write_postgresql_conf() validates provided params using
the proper PG version.
* Add errorhandling for ulimit (#943)
* Update Patroni to 3.2.2 (#971)
* PostgreSQL 16 support (#992)
- remove PG10
- remove pg_tm_aux, plantuner
- timescaledb 2.14.X
---------
Co-authored-by: Ida Novindasari <idanovinda@gmail.com>
Co-authored-by: Jociele Padilha <jociele.padilha@zalando.de>
* Spilo build is failing fix dependencies.sh due to error at WAL-G (#998)
* Bump wal-g version to v3.0.0 (#987)
* Fix regex in wale_restore script (#1000)
* Remove PG11 (#999)
* Patroni 3.3.1 (#1001)
* Update Patroni to 3.3.2 (#1004)
* Install wal-g from a precompiled bin (#1006)
- decrease build time
- fix the problem of missing brotli compression type after upgrade to WAL-G v3
* Bump wal-g and pg_profile (#1013)
* Update tail_postgres_logs to support any log rotation plan (#955)
* Patroni 3.3.3 (#1027)
* add extra tags for uploaded Postgres log files (#1024)
* enable hourly log ship mode (#1025)
---------
Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com>
* fill s3 tags before copying to log env (#1031)
* fill s3 tags before copying to log env
* Fix wrong bool handling of LOG_SHIP_HOURLY (#1030)
* ensure LOG_SHIP_HOURLY env var remains a string
* only allow true or empty for LOG_SHIP_HOURLY variable
Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com>
---------
Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com>
* do not append hour on daily uploads (#1033)
* Postgis 3.5 (#1051)
* Separate trigger and OSS pipelines (#1052)
- Build zalando trigger image unconditionally
- Add possibility to push a pr image (staging registry) on approval
- Allow building master branch images on approval
- Update github actions deps
* PostgreSQL 17 support (#1049)
- Update extensions
- Adjust scripts
- Remove PG 12
- Patroni 3.3.4
- Use pkgs for timescaledb
---------
Co-authored-by: idanovinda <idanovinda@gmail.com>
* fix: wrong regex for wal retention (#1026)
* Integrate Patroni 4.x (#1050)
* Integrate Patroni v4.x
* set docs
* update to latest patroni version
* Fix comment in tests (#1060)
* [Patroni4] change no_master and master as argument (#1061)
* Update publish pipeline deps versions (#1063)
* Use timescaledb 2.18.0 (#1077)
* Fix timescaledb installation logic (#1079)
- properly delete *.so for different setups
- always keep the latest compatible and transition timescaledb ver
without specifying in a build arg
* doc: Add callout about Docker images (#1090)
This commit adds a call out box to the README.rst file for clarifying lack of release policy and unavailability of publicly released Docker images.
* Make leader tag value for native AWS configurable (#1091)
* retain multiple minor versions of timescaledb (#1087)
Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com>
* Sync master with the latest changes from trigger branch (#1149)
- wal-g 3.0.5 (not 3.0.7, see: https://github.com/zalando/spilo/pull/1133)
- Support deleting old backups regardless of the current backup tool - https://github.com/zalando/spilo/pull/1133, https://github.com/zalando/spilo/pull/1150
- Make Spilo ipv6 compatible (on AWS) - https://github.com/zalando/spilo/pull/1108, https://github.com/zalando/spilo/pull/1109, https://github.com/zalando/spilo/pull/1112, https://github.com/zalando/spilo/pull/1113, https://github.com/zalando/spilo/pull/1117, https://github.com/zalando/spilo/pull/1123
- remove spilo_cmd - https://github.com/zalando/spilo/pull/1147
- add roaringbitmap and pgfaceting extensions - https://github.com/zalando/spilo/pull/1101, https://github.com/zalando/spilo/pull/1107
- add KUBERNETES_BOOTSTRAP_LABELS env for the upcoming Patroni feature - https://github.com/zalando/spilo/pull/1093, https://github.com/zalando/spilo/pull/1097
- fix LOG_S3_TAGS parsing logic - https://github.com/zalando/spilo/pull/1093
* Sync master with trigger 13-04-26 (#1196)
- Add PostgreSQL 18 and remove PostgreSQL 13: https://github.com/zalando/spilo/commit/c43255794e7cba31b9e68c9c0bd0d1d71970e235
- Patroni 4.1.0
- WAL-G 3.0.8
- PostGIS 3.6: https://github.com/zalando/spilo/pull/1153
- Remove WAL-E and make WAL-G the default backup tool: https://github.com/zalando/spilo/pull/1143
- Enable pg_basebackup server compression: https://github.com/zalando/spilo/pull/1155
- Update bg_mon, pg_mon, pg_profile, pg_permissions refs
- Use pgdg for set_user and pg_permissions
Bugfixes
- Allow admin selecting from hypopg views: https://github.com/zalando/spilo/pull/1162
- Fix logic for keeping timescaledb minor versions: https://github.com/zalando/spilo/pull/1173
- Properly setup clean env for ext build across major versions: https://github.com/zalando/spilo/pull/1174
- Properly format ipv6 address for rsync: https://github.com/zalando/spilo/pull/1187
- Ensure CLONE_HOST is IPv6 compatible: https://github.com/zalando/spilo/pull/1190
* chore: drop upstream's CI and README brought in by the sync
Three of the files the merge added are GitHub workflows that would EXECUTE
in this org:
publish-ghcr-container.yaml on: push tags '*', packages: write, and
IMAGE_NAME: ${{ github.repository }}. It would
build Spilo upstream's way — WITHOUT our
CVE-pinned walg-builder stage — and publish to
ghcr.io/runwhen-contrib/spilo. It shares the
tag trigger with our own build-push.yaml, so
the two would race on every tag.
tests.yaml on: pull_request — runs on every PR here
kubernetes-test.yaml on: pull_request for kubernetes/**, pinned to
the retired ubuntu-20.04 runner
Also removed: .zappr.yaml (their PR-approval bot), delivery.yaml and
contrib/cloudbuild-*.yaml (their delivery pipeline), an empty .gitmodules,
and README.rst — we keep our own README.md.
Kept, as legitimate upstream content: LICENSE, docs/, kubernetes/,
CONTRIBUTING.rst, ENVIRONMENT.rst, BUGS.md, mkdocs.yml, tox.ini.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(ci): unbreak the sync workflow and keep upstream CI out of future syncs
Three fixes. The first two are why it never ran; the third is why deleting
upstream's CI once is not enough.
1. No git identity, so every commit died with
`fatal: empty ident name (for <runner@...>) not allowed`. Every sync is a
non-fast-forward merge, so this is needed on the clean path too, not only
for conflicts.
2. The conflict path could commit an unresolved index. `git checkout --ours
README.md` resolves nothing unless README.md is actually one of the
conflicted paths — the failing logs show it as "Updated 0 paths from the
index" — and `git commit` then ran on a conflicted tree. Bail out with the
conflicting file list instead. The README.md handling itself is kept:
not taking upstream's README is deliberate.
3. Upstream's workflow files are absent here, so a sync brings them in as a
clean ADD. They never surface as a conflict and the block above never
sees them — the next sync would restore every file the previous commit
removed. After a successful merge, drop anything under .github/ that is
not one of ours.
Listing what we own rather than what upstream currently has also covers
workflows they add later; naming their present three would not, and they
have added three over the repo's history.
Verified against a simulated sync carrying their three plus an invented
fourth: all four dropped, our five untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Signed-off-by: Stephan Austermühle <au@hcsd.de>
Co-authored-by: Alexander Kukushkin <cyberdemn@gmail.com>
Co-authored-by: nicodimi <nico.dimisianos@bmw.de>
Co-authored-by: Jawher Moussa <jawher.moussa@gmail.com>
Co-authored-by: Weilu Jia <optix2000@users.noreply.github.com>
Co-authored-by: Alice <31040183+alice-sawatzky@users.noreply.github.com>
Co-authored-by: Sergey Dudoladov <dss-2009@yandex.ru>
Co-authored-by: sulphur <piotr.kedziora27@gmail.com>
Co-authored-by: zimbatm <zimbatm@zimbatm.com>
Co-authored-by: Andras Vaczi <andras.vaczi@zalando.de>
Co-authored-by: db-adrian <43702457+db-adrian@users.noreply.github.com>
Co-authored-by: Gert van Dijk <github@gertvandijk.net>
Co-authored-by: Rafia Sabih <rafia.sabih@gmail.com>
Co-authored-by: Tomas <tomas+github@libre.nz>
Co-authored-by: ReSearchITEng <ReSearchITEng@users.noreply.github.com>
Co-authored-by: Armin Nesiren <armin@nesiren.com>
Co-authored-by: stoetti <31430612+stoetti@users.noreply.github.com>
Co-authored-by: Dmitry Dolgov <9erthalion6@gmail.com>
Co-authored-by: sweetbadger <69196555+sweetbadger@users.noreply.github.com>
Co-authored-by: Marco Giovannini <usernkey@gmail.com>
Co-authored-by: Igor Yanchenko <1504692+yanchenko-igor@users.noreply.github.com>
Co-authored-by: Christopher Bayliss <github@baylisscg.baylisscg.org>
Co-authored-by: SanjeevChoubey <31409038+SanjeevChoubey@users.noreply.github.com>
Co-authored-by: Steve Singer <steve@ssinger.info>
Co-authored-by: Marcin Frankiewicz <3199047+fraanek@users.noreply.github.com>
Co-authored-by: Oleksii Kliukin <alexk@hintbits.com>
Co-authored-by: Kiryl <74899928+kirillka415@users.noreply.github.com>
Co-authored-by: Max Rosin <git@hackrid.de>
Co-authored-by: Kieran Evans <keyz182@users.noreply.github.com>
Co-authored-by: François Van Ingelgom <vfrans@users.noreply.github.com>
Co-authored-by: D. Spindel <spider@skuggor.se>
Co-authored-by: Django <39873218+djcass44@users.noreply.github.com>
Co-authored-by: Petter Sælen <petter.saelen@gmail.com>
Co-authored-by: Olle Larsson <larsson1995@hotmail.com>
Co-authored-by: Stephan Austermühle <stephan2012@users.noreply.github.com>
Co-authored-by: Karl Herler <hi@karlherler.com>
Co-authored-by: Nozomu Ohki <38576286+ohkinozomu@users.noreply.github.com>
Co-authored-by: Salvatore D'Angelo <sasadangelo@gmail.com>
Co-authored-by: Remo Wenger <privat@remowenger.ch>
Co-authored-by: irishandyb <irishandyb@gmail.com>
Co-authored-by: Aaron Peschel <aaron.peschel@gmail.com>
Co-authored-by: James McDonald <james@jamesmcdonald.com>
Co-authored-by: Salvatore D'Angelo <sdangelo@it.ibm.com>
Co-authored-by: gandalfmagic <gandalfmagic@libero.it>
Co-authored-by: Mohamed Sohail <sohailsameja@gmail.com>
Co-authored-by: Daniel Degasperi <mrstackoverflow@hotmail.com>
Co-authored-by: stephan-breitrainer <87771094+stephan-breitrainer@users.noreply.github.com>
Co-authored-by: a1exsh <a1exsh@users.noreply.github.com>
Co-authored-by: 胡玮文 <huww98@outlook.com>
Co-authored-by: Felix Kunde <felix-kunde@gmx.de>
Co-authored-by: allen12921 <11305195+allen12921@users.noreply.github.com>
Co-authored-by: Tobias Florek <me@ibotty.net>
Co-authored-by: Ahmed Kamal <email.ahmedkamal@googlemail.com>
Co-authored-by: Alexander Kukushkin <alexander.kukushkin@zalando.de>
Co-authored-by: gc-jro <99796349+gc-jro@users.noreply.github.com>
Co-authored-by: Samusername <44224323+Samusername@users.noreply.github.com>
Co-authored-by: haslersn <sebastian.hasler@gmx.net>
Co-authored-by: Sudeepta92 <90620618+Sudeepta92@users.noreply.github.com>
Co-authored-by: Jasper Ben Orschulko <jasper@fancydomain.eu>
Co-authored-by: Polina Bungina <bungina@gmail.com>
Co-authored-by: Philip Haberkern <59010269+thedatabaseme@users.noreply.github.com>
Co-authored-by: machine424 <ayoubmrini424@gmail.com>
Co-authored-by: Polina Bungina <polina.bungina@zalando.de>
Co-authored-by: Anatoliy Basov <demndell@gmail.com>
Co-authored-by: ndomar <omar.noshy@student.guc.edu.eg>
Co-authored-by: Thomas Kosiewski <thoma471@googlemail.com>
Co-authored-by: m² <2553753+mmoscher@users.noreply.github.com>
Co-authored-by: Alexander Kukushkin <akukushkin@microsoft.com>
Co-authored-by: ShutterQuick <daniel@vassdal.org>
Co-authored-by: Polina Bungina <27892524+hughcapet@users.noreply.github.com>
Co-authored-by: Vladimir Kuznichenkov <5330267+kuzaxak@users.noreply.github.com>
Co-authored-by: idanovinda <idanovinda@gmail.com>
Co-authored-by: Yang Wu <yang.wu@daocloud.io>
Co-authored-by: Jociele Padilha <45459238+jopadi@users.noreply.github.com>
Co-authored-by: Jociele Padilha <jociele.padilha@zalando.de>
Co-authored-by: Max Neverov <neverov.max@gmail.com>
Co-authored-by: Sergey Dudoladov <sergey.dudoladov@gmail.com>
Co-authored-by: Anas Anjaria <499739+anasanjaria@users.noreply.github.com>
Co-authored-by: Glenn Sommer <glemsom@gmail.com>
Co-authored-by: Staticle <55652600+staticle@users.noreply.github.com>
Co-authored-by: annielzy <148128409+annielzy@users.noreply.github.com>
Co-authored-by: Rob Nickmans <nickmansrob@gmail.com>
Co-authored-by: Matthias Adler <macedigital@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
keep at least 5 minor versions, but ensure compatibility with the lowest/oldest PG version (where possible)
(copied from dev-18 branch)