Fix pool timeout overrides, defaults, and retain_connections_max quota bug

[vadv]

Summary

• Fix: Pool-level server_lifetime and idle_timeout overrides were silently ignored — general values were always used instead. Fixed in 6 places across 3 pool creation contexts (static pools, auth_query shared pools, dynamic pools).
• Fix: idle_timeout default was 300,000,000ms (83 hours) instead of 600,000ms (10 minutes), effectively disabling idle connection cleanup.
• Fix: server_lifetime default changed from 5 minutes to 20 minutes. The old value (5 min) was shorter than idle_timeout (10 min), making idle_timeout permanently inert — server_lifetime killed every connection before it could be idle for 10 minutes.
• Fix: retain_connections_max quota exhaustion caused remaining=0 which was treated as "unlimited" by retain_oldest_first(), causing pools processed after quota exhaustion to lose ALL idle connections in a single retain cycle.
• Fix: idle_timeout = 0 closed connections after ~1ms instead of disabling idle timeout. Now 0 means disabled, consistent with server_lifetime = 0 and PgBouncer semantics.
• Fix: idle_timeout had no per-connection jitter — all connections in a pool expired simultaneously after a traffic burst, causing synchronized mass closures. Now applies the same ±20% jitter as server_lifetime.
• Fix: retain_connections_max quota was unfairly distributed — HashMap iteration order is deterministic within a process, so the same pool always consumed the entire quota while other pools starved. Fixed by shuffling pool iteration order each cycle.
• Fix: Retain and replenish phases used separate get_all_pools() snapshots. If POOLS was atomically updated between them (config reload, dynamic pool GC), they operated on different pool sets. Fixed by using a single snapshot.
• Fix: Doc comment for retain_connections_max incorrectly stated default as 0 (unlimited); actual default is 3.
• Feature: default_min_pool_size for dynamic auth_query passthrough pools — minimum backend connections maintained per dynamic user pool. Prewarmed on first pool creation, replenished by retain cycle. GC skips pools with min_pool_size > 0 to prevent race condition with retain.
• Infra: make generate target for regenerating reference configs and docs. Balanced CI test partitions (@rust-4).
• Version: Bumped to 3.3.2 with changelog.

How server_lifetime and idle_timeout interact

Parameter	Default	Jitter	What it measures	When checked
server_lifetime	20 min	±20% per connection	Time since creation	At recycle (→ new connection for next client) AND at retain (→ close idle)
idle_timeout	10 min	±20% per connection	Time since last use	At retain only (→ close idle). 0 = disabled.

With the new defaults (20m / 10m), each serves its purpose:

• idle_timeout cleans up connections sitting idle after load drops (10 min ±20%)
• server_lifetime rotates long-lived connections even under constant use (20 min ±20%)

Note: idle_timeout only applies to connections that have been used at least once. Prewarmed/replenished connections that were never checked out by a client have recycled=None and are not subject to idle_timeout — they live until server_lifetime expires.

default_min_pool_size for dynamic pools

New auth_query.default_min_pool_size (default: 0) controls minimum backend connections per dynamic user pool in passthrough mode:

• Prewarm: When a dynamic pool is first created, a background task spawns default_min_pool_size connections without blocking the client.
• Retain cycle: Already iterates all pools via get_all_pools() — dynamic pools with min_pool_size are automatically replenished. No changes needed in retain.rs.
• GC protection: Dynamic pools with min_pool_size > 0 are skipped by GC. This prevents a race condition where GC could delete a pool between retain's close and replenish phases.
• Lifecycle: Prewarmed connections live until server_lifetime, then retain closes and replenishes them. Pools persist until config RELOAD or default_min_pool_size is set to 0.

Edge case	Behavior
default_min_pool_size = 0 (default)	Current behavior preserved: no prewarm, GC removes empty pools
Backend unreachable during prewarm	replenish() returns 0, warn logged. GC won't delete (min > 0). Retain retries
N users × min = many connections	100 users × min=2 = 200 backend connections. Documented in fields.yaml
RELOAD with changed value	All dynamic pools destroyed, new ones created fresh (existing logic)

Details

Pool-level timeout overrides

In src/pool/mod.rs, both ServerPool::new() (controls recycle lifetime check) and PoolSettings (controls retain idle_timeout/lifetime) always used config.general.server_lifetime / config.general.idle_timeout, completely ignoring pool-level overrides.

The fix uses pool_config.server_lifetime.unwrap_or(config.general.server_lifetime.as_millis()) — the same pattern already used in config logging.

retain_connections_max quota exhaustion

When retain_connections_max > 0 and the global counter reached the limit, saturating_sub returned 0. Since 0 means "unlimited" in retain_oldest_first(), subsequent pools got unlimited closure. HashMap iteration order is non-deterministic, so the bug manifested randomly.

Fix: early return when current_count >= max, preventing the 0-as-unlimited ambiguity.

idle_timeout jitter and 0 = disabled

idle_timeout was stored as a single pool-wide value in PoolSettings. After a traffic burst, all returned connections had the same idle start time and expired simultaneously. Now idle_timeout_ms is stored per-connection in Metrics with ±20% jitter applied at creation time — the same approach as server_lifetime.

Additionally, idle_timeout = 0 now means "disabled" (consistent with server_lifetime = 0 and PgBouncer's server_idle_timeout = 0). Previously it closed connections after ~1ms of idle time because the check was elapsed > 0ms.

Retain cycle fairness

The retain cycle iterated pools via HashMap::iter(), whose order is deterministic within a process (fixed RandomState seed). With retain_connections_max = 3, the same pool always consumed the entire quota while other pools' expired connections were never cleaned up. Now pools are shuffled each cycle via rand::seq::SliceRandom::shuffle.

Both retain and replenish phases now use a single get_all_pools() snapshot to avoid inconsistency if POOLS is atomically updated between them.

Test plan

Pool timeout overrides:

• @pool-override-server-lifetime — pool-level server_lifetime triggers recycle
• @pool-override-idle-timeout — pool-level idle_timeout closes idle connections
• @pool-override-two-pools-different-lifetime — two pools with different lifetimes behave independently
• @general-server-lifetime-baseline — general server_lifetime still works
• @general-idle-timeout-baseline — general idle_timeout still works

Dynamic pool overrides:

• @dynamic-pool-idle-timeout — auth_query dynamic pool inherits pool-level idle_timeout
• @dynamic-pool-server-lifetime — auth_query dynamic pool inherits pool-level server_lifetime
• @min-pool-size-with-pool-lifetime — min_pool_size replenishment works with pool-level server_lifetime

retain_connections_max quota:

• @retain-max-quota-two-pools — retain_connections_max quota enforced across multiple pools

End-to-end pool lifecycle:

• @e2e-dedicated-lifecycle — dedicated pool: prewarm (min_pool_size=2) → scale up (pool_size=5) → shrink to min_pool_size=2
• @e2e-auth-query-lifecycle — auth_query pool: scale up (default_pool_size=5) → shrink to 0

default_min_pool_size:

• @auth-query-min-pool-size — dynamic pool prewarm with default_min_pool_size (2 scenarios: prewarm + lifetime replenish)

Regression:

• @min-pool-size — existing min_pool_size tests (3 scenarios)
• @auth-query-passthrough — passthrough auth tests (4 scenarios)
• cargo test --lib — 259 passed (includes reference config matching)

Run:

DEBUG=1 make test-bdd TAGS=@pool-lifecycle-e2e
DEBUG=1 make test-bdd TAGS=@pool-timeout-override
DEBUG=1 make test-bdd TAGS=@pool-timeout-override-dynamic
DEBUG=1 make test-bdd TAGS=@retain-max-quota
DEBUG=1 make test-bdd TAGS=@min-pool-size
DEBUG=1 make test-bdd TAGS=@auth-query-min-pool-size

🤖 Generated with Claude Code