#292 Add sequential stopping rule for benchmark iterations

CLAUDE.md

@@ -25,7 +25,7 @@ Python, DuckDB (spatial), PostGIS on Azure Database for PostgreSQL, Apache Sedon
 - Secrets live in `.env` (gitignored) and are forwarded to ACI as `--secure-environment-variables` from `main.py`.
 - Benchmark batching: members of a batch list each other bidirectionally in `related_script_ids`. The orchestrator dedupes via `completed_experiments`, so each batch runs once as one parallel `ThreadPoolExecutor` fan-out. A batch must satisfy four constraints simultaneously: (a) same query type, (b) same `dataset_size`, (c) at most one PostGIS member (shared Azure Postgres server), (d) Databricks cluster vCPU sum ≤ 80, computed as `(workers + 1) × 4` per Sedona member on `Standard_D4s_v3`. See `README.md#pairing-and-randomization` for the full batch listing. *Why: peers must execute under the same wall-clock window for fair comparison, without contending on shared infrastructure or breaching regional quota.*
 - Adding a benchmark requires three edits in lockstep: file in `src/presentation/entrypoints/`, `case` arm in `benchmark_runner.py`, and an entry in `benchmarks.yml`. Missing any one silently breaks dispatch or orchestration.
-- Stopping rule on `@monitor`: high-frequency single-machine queries use sequential stopping (bootstrapped CI on the mean elapsed time, floors at `BENCHMARK_MIN_ITERATIONS` and `BENCHMARK_MIN_TIMED_WINDOW_SECONDS`, ceiling at the `BenchmarkIteration` value, hard timeout at `BENCHMARK_MAX_TIMED_WINDOW_SECONDS`). Long-running low-variance benchmarks (`national_scale_spatial_join_*` for Databricks, DuckDB, and PostGIS) opt out via `use_sequential_stopping=False` and run a small fixed iteration count. *Why: bootstrap CI on <10 samples is uninformative, and the per-iteration cost of those benchmarks (cluster time, shared Postgres) outweighs precision gains.* See `README.md#stopping-rule` for the full rule.
+- Stopping rule on `@monitor`: high-frequency single-machine queries use sequential stopping (bootstrapped CI on the mean elapsed time, floors at `BENCHMARK_MIN_ITERATIONS` and `BENCHMARK_MIN_TIMED_WINDOW_SECONDS`, ceiling at the `BenchmarkIteration` value, hard timeout at `BENCHMARK_MAX_TIMED_WINDOW_SECONDS`). Long-running low-variance benchmarks (`national_scale_spatial_join_*` for Databricks, DuckDB, and PostGIS) opt out via `use_sequential_stopping=False` and run a small fixed iteration count; they also override `warmup_iterations=1` since one warmup is enough on long-running queries and additional warmups dominate the wall-clock budget. *Why: bootstrap CI on <10 samples is uninformative, and the per-iteration cost of those benchmarks (cluster time, shared Postgres) outweighs precision gains.* See `README.md#stopping-rule` for the full rule.
 
 ## Commands
 
@@ -40,7 +40,7 @@ python benchmark_runner.py --script-id <id> --benchmark-run 1 --run-id dev  # on
 - Create `src/presentation/entrypoints/<name>.py`; re-export from `entrypoints/__init__.py`.
 - Add `case "<script-id>":` in `benchmark_runner.py`.
 - Append entry to `benchmarks.yml` with `id`, `image`, `cpu`, `memory_gb`, `dataset_size`, `related_script_ids` (list peers both ways; assign to an existing batch that satisfies the four constraints in the batching invariant, or create a new batch).
-- Pick the stopping rule on `@monitor`: leave the default (`use_sequential_stopping=True`) for short-iteration queries where many samples are cheap; set `use_sequential_stopping=False` when each iteration is long-running and low-variance, or the per-iteration cost (cluster time, shared Postgres) makes additional iterations expensive.
+- Pick the stopping rule on `@monitor`: leave the default (`use_sequential_stopping=True`) for short-iteration queries where many samples are cheap; set `use_sequential_stopping=False` when each iteration is long-running and low-variance, or the per-iteration cost (cluster time, shared Postgres) makes additional iterations expensive. In the long-running case, also set `warmup_iterations=1` so warmup does not dominate the wall-clock budget.
 - If a new service is needed: contract in `application/contracts/`, impl in `infra/infrastructure/services/`, provider in `containers.py`.
 </important>
 

README.md

@@ -135,7 +135,11 @@ National-scale spatial joins (`national_scale_spatial_join_databricks_*`, `natio
 fixed count (`NATIONAL_SCALE_SPATIAL_JOIN=5`). These queries are long-running and low-variance: a bootstrapped CI on
 fewer than `MIN_ITERATIONS` samples would be uninformative, and the cost of additional iterations is significant on
 Databricks (cluster runtime × workers) and on the shared Postgres instance. The wall-clock timeout is also skipped
-for this branch, so the fixed iteration count is the only upper bound on these benchmarks.
+for this branch, so the fixed iteration count is the only upper bound on these benchmarks. These same entrypoints
+override `warmup_iterations=1` on `@monitor`: one warmup is enough to prime the OS page cache, JDBC/PostGIS
+connection state, and Spark Catalyst plans on a warm Databricks cluster, while additional warmups would dominate
+the wall-clock budget (`Config.BENCHMARK_WARMUP_ITERATIONS=5` is the decorator default and applies to the
+high-frequency single-machine queries).
 
 The achieved iteration count, mean, median, bootstrapped CI half-width (both absolute seconds and as a fraction of
 the mean), and `stop_reason` (`precision`, `timeout`, `ceiling`, `fixed`, or `failed`) are persisted alongside the

src/application/common/monitor.py

@@ -29,6 +29,7 @@ def monitor(
     skip_warmup: bool = False,
     elapsed_from_result: bool = False,
     use_sequential_stopping: bool = True,
+    warmup_iterations: int | None = None,
 ):
     """
     Benchmarking decorator. Wraps a function in warmup + timed iterations, records
@@ -49,6 +50,11 @@ def monitor(
         window valid) and ``Config.BENCHMARK_MAX_TIMED_WINDOW_SECONDS`` (hard timeout).
         Set to False for Databricks national-scale runs, which use a fixed iteration count.
         Default is True.
+    :param warmup_iterations: Override the number of warmup iterations. ``None`` falls back
+        to ``Config.BENCHMARK_WARMUP_ITERATIONS``. Long-running benchmarks (national-scale
+        spatial joins) typically set this to 1 since one warmup is enough to prime the OS
+        page cache / connection / cluster state and additional warmups dominate the
+        wall-clock budget. Ignored when ``skip_warmup`` is True.
     """
 
     def decorator(func):
@@ -73,15 +79,21 @@ def wrapper(*args, **kwargs):
             failure_ended_at: datetime.datetime | None = None
             failure_partial_sample: dict | None = None
 
+            effective_warmup_iterations = (
+                warmup_iterations
+                if warmup_iterations is not None
+                else Config.BENCHMARK_WARMUP_ITERATIONS
+            )
+
             if skip_warmup:
                 logger.info(
                     f"Executing benchmark for '{query_id}' with no warmup (ceiling={ceiling})."
                 )
             else:
                 logger.info(
-                    f"Executing {Config.BENCHMARK_WARMUP_ITERATIONS} warmup runs."
+                    f"Executing {effective_warmup_iterations} warmup runs."
                 )
-                for _ in range(Config.BENCHMARK_WARMUP_ITERATIONS):
+                for _ in range(effective_warmup_iterations):
                     warmup_started_at = datetime.datetime.now(datetime.UTC)
                     (
                         _,

src/presentation/entrypoints/_databricks_benchmark_runner.py

@@ -67,6 +67,7 @@ def _build_benchmark_fn(
         skip_warmup=False,
         elapsed_from_result=True,
         use_sequential_stopping=False,
+        warmup_iterations=1,
     )
     def _benchmark(
         cluster_id: str,

src/presentation/entrypoints/national_scale_spatial_join_duckdb.py

@@ -34,6 +34,7 @@ def _build_benchmark_fn(dataset_size: DatasetSize):
         benchmark_iteration=BenchmarkIteration.NATIONAL_SCALE_SPATIAL_JOIN,
         cost_configuration=CostConfiguration(include_aci=True, include_blob_storage=True),
         use_sequential_stopping=False,
+        warmup_iterations=1,
     )
     def _benchmark(
         db_context: DuckDBPyConnection,

src/presentation/entrypoints/national_scale_spatial_join_postgis.py

@@ -68,6 +68,7 @@ def _build_benchmark_fn(dataset_size: DatasetSize):
         benchmark_iteration=BenchmarkIteration.NATIONAL_SCALE_SPATIAL_JOIN,
         cost_configuration=CostConfiguration(include_aci=True, include_postgres=True),
         use_sequential_stopping=False,
+        warmup_iterations=1,
     )
     def _benchmark(
         db_context: Engine,