Skip to content

Feature/292 create lower and upper bound for queries#307

Merged
jathavaan merged 8 commits into
mainfrom
feature/292-create-lower-and-upper-bound-for-queries
May 22, 2026
Merged

Feature/292 create lower and upper bound for queries#307
jathavaan merged 8 commits into
mainfrom
feature/292-create-lower-and-upper-bound-for-queries

Conversation

@jathavaan
Copy link
Copy Markdown
Collaborator

This pull request introduces a new, configurable stopping rule for benchmarks, allowing for more precise and efficient execution of timed iterations. The main change is the implementation of a sequential stopping rule based on bootstrapped confidence intervals for high-frequency queries, with an opt-out for long-running, low-variance benchmarks. The documentation and metadata handling have also been updated to reflect and support these changes.

Benchmark stopping rule improvements:

  • Added a sequential stopping rule to the monitor decorator in monitor.py, which stops benchmark iterations early if a bootstrapped confidence interval on the mean elapsed time meets a configured precision, subject to minimum iteration and time windows, and a ceiling based on the BenchmarkIteration value. Long-running benchmarks can opt out and use a fixed iteration count instead. The stopping reason and achieved iterations are now logged and saved as metadata. [1] [2] [3] [4]

Documentation updates:

  • Updated README.md and CLAUDE.md to document the new stopping rule, its rationale, and how to configure it for different benchmarks. This includes details on the bootstrapped CI procedure, configuration parameters, and when to use fixed vs. sequential stopping. [1] [2] [3]

Metadata and logging enhancements:

  • Modified _save_run_metadata calls in both monitor.py and monitor_cpu_and_ram.py to record additional metadata: achieved iterations, stopping reason, CI half-width, and summary statistics (mean/median elapsed time). [1] [2]

Supporting code and enum changes:

  • Added StopReason enum and updated imports throughout to support new stopping logic and metadata. [1] [2] [3]

Refactoring and code clarity:

  • Refactored the iteration loop in monitor.py for clarity, separating sequential and fixed stopping logic, and improved logging to make benchmark progress and stopping conditions more transparent. [1] [2] [3] [4] [5]

Copilot AI review requested due to automatic review settings May 22, 2026 15:36
@jathavaan jathavaan linked an issue May 22, 2026 that may be closed by this pull request
15 tasks
@jathavaan jathavaan self-assigned this May 22, 2026
@jathavaan jathavaan merged commit 57c84f3 into main May 22, 2026
30 of 31 checks passed
@jathavaan jathavaan deleted the feature/292-create-lower-and-upper-bound-for-queries branch May 22, 2026 15:39
@jathavaan jathavaan review requested due to automatic review settings May 22, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create lower and upper bound for queries

1 participant