Add maxSamples limit by PaurushGarg · Pull Request #663 · thanos-io/promql-engine

PaurushGarg · 2025-12-02T08:36:24Z

This PR implements max_samples limit enforcement to prevent OOM by tracking samples at the operator level.

Changes:

Added SampleTracker in query/ package with Add(), Remove(), and CheckLimit() methods
Added MaxSamples field to Engine.Opts (default 0 = unlimited)
Added ErrMaxSamplesExceeded error for limit violations

Tracking Implementation:

Uses delta-based tracking (similar to peak_samples pattern) in three operators:

vectorSelector: Tracks StepVector samples across batches
matrixSelector: Tracks ring buffer samples (sliding window)
subquery: Tracks ring buffer samples (sliding window)

Operator queries buffer.SampleCount(), compares to previous count, and tracks the delta using Add()/Remove().

Benchmark test results: #663 (comment)

MichaHoffmann · 2025-12-03T12:06:23Z

I think we should not support this at the engine here especially not at a logical level - if anything this needs to be pushed down into storage.

PaurushGarg · 2026-01-29T01:32:13Z

I think we should not support this at the engine here especially not at a logical level - if anything this needs to be pushed down into storage.

Thanks for the feedback! Updated to track only at storage layer (vectorSelector, matrixSelector) similar to peak_samples - at operator level where samples are decoded. Also added tracking in subquery operator since it accumulates samples in ring buffers.

harry671003 · 2026-01-31T01:46:46Z

storage/prometheus/vector_selector.go


+	if o.opts.SampleTracker != nil {
+		if totalSamplesInBatch > o.lastTrackedSamples {
+			o.opts.SampleTracker.Add(totalSamplesInBatch - o.lastTrackedSamples)


Could you help me understand why do we need to keep track of the last tracked samples? Doesn't the sample counts get reset at every time we call Next() on the vector selector?

For vector_selector and subquery, we're tracking a running total of samples, so we need lastTrackedSamples to quickly calculate the delta (just subtract the old value). Otherwise we'd have to loop through everything to recalculate the total each time we check the limit - making it a O(n) operation instead of O(1).

matrix_selector works differently—we calculate the delta for each series as we go (sampleCountAfter - sampleCountBefore) and add it to batchSamplesDelta, so the delta is already there when we need to check.

harry671003 · 2026-01-31T01:48:38Z

storage/prometheus/vector_selector.go

 					buf[currStep].AppendSampleWithSizeHint(series.signature, v, expectedSamples)
 					currStepSamples++
 				}
+				totalSamplesInBatch += currStepSamples


Should we add the check here? Imagine if we have 500k series and step batch of 10. We'll only increment and check the sample count after 500k*10 = 5M samples are loaded?

Yes makes sense. I have updated now.
Thanks.

harry671003 · 2026-01-31T01:56:34Z

Could you add unit tests to the PR?
Could you also add benchmarks before and after enabling the sample limiter just to see if there is any big impact there?

Signed-off-by: Paurush Garg <paurushg@amazon.com>

PaurushGarg · 2026-02-10T01:31:01Z

Benchmark test results:
go test ./... -bench 'BenchmarkRangeQuery' -run none -count 5 > maxsamples.out
go test ./... -bench 'BenchmarkRangeQuery' -run none -count 5 > main.out
go tool benchstat main.out maxsamples.out

goos: linux
goarch: amd64
pkg: github.com/thanos-io/promql-engine/engine
cpu: AMD EPYC 7R13 Processor
                                                                 │   main.out   │           maxsamples.out            │
                                                                 │    sec/op    │    sec/op     vs base               │
RangeQuery/vector_selector/old_engine-32                           44.55m ± ∞ ¹   43.55m ± ∞ ¹   -2.26% (p=0.008 n=5)
RangeQuery/vector_selector/new_engine-32                           23.93m ± ∞ ¹   23.94m ± ∞ ¹        ~ (p=0.841 n=5)
RangeQuery/sum/old_engine-32                                       48.71m ± ∞ ¹   48.11m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/sum/new_engine-32                                       10.65m ± ∞ ¹   10.57m ± ∞ ¹        ~ (p=0.421 n=5)
RangeQuery/sum_by_pod/old_engine-32                                56.17m ± ∞ ¹   55.97m ± ∞ ¹        ~ (p=0.310 n=5)
RangeQuery/sum_by_pod/new_engine-32                                30.89m ± ∞ ¹   30.71m ± ∞ ¹        ~ (p=0.421 n=5)
RangeQuery/topk/old_engine-32                                      51.70m ± ∞ ¹   50.44m ± ∞ ¹        ~ (p=0.151 n=5)
RangeQuery/topk/new_engine-32                                      14.16m ± ∞ ¹   14.06m ± ∞ ¹   -0.71% (p=0.008 n=5)
RangeQuery/bottomk/old_engine-32                                   51.07m ± ∞ ¹   51.53m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/bottomk/new_engine-32                                   13.97m ± ∞ ¹   14.16m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/limitk/old_engine-32                                    47.34m ± ∞ ¹   46.65m ± ∞ ¹        ~ (p=0.151 n=5)
RangeQuery/limitk/new_engine-32                                    9.126m ± ∞ ¹   9.167m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/limit_ratio/old_engine-32                               109.6m ± ∞ ¹   106.8m ± ∞ ¹        ~ (p=0.151 n=5)
RangeQuery/limit_ratio/new_engine-32                               42.91m ± ∞ ¹   44.00m ± ∞ ¹   +2.54% (p=0.032 n=5)
RangeQuery/rate/old_engine-32                                      111.2m ± ∞ ¹   107.7m ± ∞ ¹        ~ (p=0.690 n=5)
RangeQuery/rate/new_engine-32                                      23.05m ± ∞ ¹   23.92m ± ∞ ¹   +3.77% (p=0.016 n=5)
RangeQuery/rate_with_longer_window/old_engine-32                   49.09m ± ∞ ¹   48.89m ± ∞ ¹        ~ (p=0.310 n=5)
RangeQuery/rate_with_longer_window/new_engine-32                   9.910m ± ∞ ¹   9.921m ± ∞ ¹        ~ (p=0.841 n=5)
RangeQuery/subquery/old_engine-32                                  147.2m ± ∞ ¹   144.7m ± ∞ ¹   -1.70% (p=0.032 n=5)
RangeQuery/subquery/new_engine-32                                  75.07m ± ∞ ¹   75.14m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/sum_rate/old_engine-32                                  115.1m ± ∞ ¹   109.3m ± ∞ ¹   -5.02% (p=0.016 n=5)
RangeQuery/sum_rate/new_engine-32                                  16.57m ± ∞ ¹   16.61m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/sum_by_rate/old_engine-32                               121.6m ± ∞ ¹   118.0m ± ∞ ¹   -2.93% (p=0.016 n=5)
RangeQuery/sum_by_rate/new_engine-32                               31.31m ± ∞ ¹   30.60m ± ∞ ¹   -2.27% (p=0.008 n=5)
RangeQuery/quantile_with_variable_parameter/old_engine-32          175.1m ± ∞ ¹   173.1m ± ∞ ¹        ~ (p=0.421 n=5)
RangeQuery/quantile_with_variable_parameter/new_engine-32          47.99m ± ∞ ¹   47.24m ± ∞ ¹        ~ (p=0.056 n=5)
RangeQuery/binary_operation_with_one_to_one/old_engine-32          103.0m ± ∞ ¹   101.8m ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/binary_operation_with_one_to_one/new_engine-32          22.61m ± ∞ ¹   23.20m ± ∞ ¹        ~ (p=0.095 n=5)
RangeQuery/binary_operation_with_many_to_one/old_engine-32         314.9m ± ∞ ¹   316.9m ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/binary_operation_with_many_to_one/new_engine-32         53.80m ± ∞ ¹   54.03m ± ∞ ¹        ~ (p=0.310 n=5)
RangeQuery/binary_operation_with_vector_and_scalar/old_engine-32   138.3m ± ∞ ¹   135.7m ± ∞ ¹        ~ (p=0.421 n=5)
RangeQuery/binary_operation_with_vector_and_scalar/new_engine-32   34.30m ± ∞ ¹   35.79m ± ∞ ¹        ~ (p=0.151 n=5)
RangeQuery/unary_negation/old_engine-32                            45.96m ± ∞ ¹   44.80m ± ∞ ¹        ~ (p=0.056 n=5)
RangeQuery/unary_negation/new_engine-32                            23.69m ± ∞ ¹   24.07m ± ∞ ¹        ~ (p=0.095 n=5)
RangeQuery/vector_and_scalar_comparison/old_engine-32              137.4m ± ∞ ¹   135.7m ± ∞ ¹        ~ (p=0.421 n=5)
RangeQuery/vector_and_scalar_comparison/new_engine-32              34.48m ± ∞ ¹   35.33m ± ∞ ¹        ~ (p=0.056 n=5)
RangeQuery/positive_offset_vector/old_engine-32                    41.86m ± ∞ ¹   41.08m ± ∞ ¹        ~ (p=0.056 n=5)
RangeQuery/positive_offset_vector/new_engine-32                    22.13m ± ∞ ¹   22.26m ± ∞ ¹        ~ (p=0.095 n=5)
RangeQuery/at_modifier_/old_engine-32                              15.13m ± ∞ ¹   15.08m ± ∞ ¹        ~ (p=0.841 n=5)
RangeQuery/at_modifier_/new_engine-32                              14.00m ± ∞ ¹   13.80m ± ∞ ¹        ~ (p=0.151 n=5)
RangeQuery/at_modifier_with_positive_offset_vector/old_engine-32   14.83m ± ∞ ¹   14.27m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/at_modifier_with_positive_offset_vector/new_engine-32   14.00m ± ∞ ¹   13.75m ± ∞ ¹   -1.73% (p=0.032 n=5)
RangeQuery/clamp/old_engine-32                                     132.6m ± ∞ ¹   132.8m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/clamp/new_engine-32                                     31.13m ± ∞ ¹   31.05m ± ∞ ¹        ~ (p=0.690 n=5)
RangeQuery/clamp_min/old_engine-32                                 132.4m ± ∞ ¹   132.9m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/clamp_min/new_engine-32                                 28.99m ± ∞ ¹   28.56m ± ∞ ¹        ~ (p=0.690 n=5)
RangeQuery/complex_func_query/old_engine-32                        246.0m ± ∞ ¹   245.2m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/complex_func_query/new_engine-32                        42.02m ± ∞ ¹   41.99m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/func_within_func_query/old_engine-32                    75.44m ± ∞ ¹   78.10m ± ∞ ¹   +3.53% (p=0.008 n=5)
RangeQuery/func_within_func_query/new_engine-32                    11.06m ± ∞ ¹   11.45m ± ∞ ¹   +3.55% (p=0.008 n=5)
RangeQuery/aggr_within_func_query/old_engine-32                    81.46m ± ∞ ¹   78.44m ± ∞ ¹   -3.70% (p=0.032 n=5)
RangeQuery/aggr_within_func_query/new_engine-32                    10.21m ± ∞ ¹   10.50m ± ∞ ¹   +2.82% (p=0.008 n=5)
RangeQuery/histogram_quantile/old_engine-32                        571.9m ± ∞ ¹   564.2m ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/histogram_quantile/new_engine-32                        144.3m ± ∞ ¹   144.9m ± ∞ ¹        ~ (p=0.690 n=5)
RangeQuery/sort/old_engine-32                                      219.5m ± ∞ ¹   212.7m ± ∞ ¹        ~ (p=0.310 n=5)
RangeQuery/sort/new_engine-32                                      23.07m ± ∞ ¹   23.16m ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/sort_desc/old_engine-32                                 218.1m ± ∞ ¹   219.0m ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/sort_desc/new_engine-32                                 23.08m ± ∞ ¹   23.09m ± ∞ ¹        ~ (p=0.690 n=5)
RangeQuery/absent_and_exists/old_engine-32                         54.05m ± ∞ ¹   53.07m ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/absent_and_exists/new_engine-32                         8.545m ± ∞ ¹   8.612m ± ∞ ¹   +0.79% (p=0.008 n=5)
RangeQuery/absent_and_doesnt_exist/old_engine-32                   52.69µ ± ∞ ¹   52.76µ ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/absent_and_doesnt_exist/new_engine-32                   761.2µ ± ∞ ¹   716.2µ ± ∞ ¹        ~ (p=0.690 n=5)
RangeQuery/double_exponential_smoothing/old_engine-32              143.0m ± ∞ ¹   145.0m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/double_exponential_smoothing/new_engine-32              23.98m ± ∞ ¹   23.79m ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/count_over_time_5m/old_engine-32                        92.37m ± ∞ ¹   89.07m ± ∞ ¹   -3.57% (p=0.032 n=5)
RangeQuery/count_over_time_5m/new_engine-32                        24.65m ± ∞ ¹   24.11m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/count_over_time_1h/old_engine-32                        94.27m ± ∞ ¹   93.44m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/count_over_time_1h/new_engine-32                        27.96m ± ∞ ¹   29.00m ± ∞ ¹   +3.69% (p=0.008 n=5)
RangeQuery/count_over_time_6h/old_engine-32                        85.61m ± ∞ ¹   86.02m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/count_over_time_6h/new_engine-32                        30.14m ± ∞ ¹   32.04m ± ∞ ¹        ~ (p=0.151 n=5)
RangeQuery/sum_over_time_5m/old_engine-32                          111.5m ± ∞ ¹   109.0m ± ∞ ¹        ~ (p=0.310 n=5)
RangeQuery/sum_over_time_5m/new_engine-32                          25.80m ± ∞ ¹   23.18m ± ∞ ¹  -10.12% (p=0.008 n=5)
RangeQuery/sum_over_time_1h/old_engine-32                          265.4m ± ∞ ¹   261.4m ± ∞ ¹        ~ (p=0.421 n=5)
RangeQuery/sum_over_time_1h/new_engine-32                          46.77m ± ∞ ¹   48.16m ± ∞ ¹   +2.97% (p=0.008 n=5)
RangeQuery/sum_over_time_6h/old_engine-32                          312.7m ± ∞ ¹   308.0m ± ∞ ¹        ~ (p=0.095 n=5)
RangeQuery/sum_over_time_6h/new_engine-32                          56.67m ± ∞ ¹   57.57m ± ∞ ¹   +1.59% (p=0.008 n=5)
RangeQuery/avg_over_time_5m/old_engine-32                          118.5m ± ∞ ¹   118.9m ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/avg_over_time_5m/new_engine-32                          24.18m ± ∞ ¹   23.26m ± ∞ ¹   -3.82% (p=0.016 n=5)
RangeQuery/avg_over_time_1h/old_engine-32                          359.8m ± ∞ ¹   351.7m ± ∞ ¹        ~ (p=0.095 n=5)
RangeQuery/avg_over_time_1h/new_engine-32                          56.84m ± ∞ ¹   58.71m ± ∞ ¹   +3.29% (p=0.008 n=5)
RangeQuery/avg_over_time_6h/old_engine-32                          441.3m ± ∞ ¹   432.8m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/avg_over_time_6h/new_engine-32                          69.09m ± ∞ ¹   71.88m ± ∞ ¹   +4.04% (p=0.008 n=5)
RangeQuery/min_over_time_5m/old_engine-32                          111.2m ± ∞ ¹   112.2m ± ∞ ¹        ~ (p=0.690 n=5)
RangeQuery/min_over_time_5m/new_engine-32                          24.15m ± ∞ ¹   23.25m ± ∞ ¹   -3.70% (p=0.008 n=5)
RangeQuery/min_over_time_1h/old_engine-32                          269.9m ± ∞ ¹   260.2m ± ∞ ¹        ~ (p=0.095 n=5)
RangeQuery/min_over_time_1h/new_engine-32                          35.70m ± ∞ ¹   37.41m ± ∞ ¹   +4.78% (p=0.008 n=5)
RangeQuery/min_over_time_6h/old_engine-32                          309.6m ± ∞ ¹   310.2m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/min_over_time_6h/new_engine-32                          40.14m ± ∞ ¹   43.03m ± ∞ ¹   +7.18% (p=0.008 n=5)
RangeQuery/max_over_time_5m/old_engine-32                          109.3m ± ∞ ¹   110.8m ± ∞ ¹   +1.40% (p=0.008 n=5)
RangeQuery/max_over_time_5m/new_engine-32                          23.49m ± ∞ ¹   23.62m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/max_over_time_1h/old_engine-32                          248.9m ± ∞ ¹   250.7m ± ∞ ¹        ~ (p=0.151 n=5)
RangeQuery/max_over_time_1h/new_engine-32                          33.70m ± ∞ ¹   35.69m ± ∞ ¹   +5.89% (p=0.008 n=5)
RangeQuery/max_over_time_6h/old_engine-32                          294.5m ± ∞ ¹   296.6m ± ∞ ¹        ~ (p=0.151 n=5)
RangeQuery/max_over_time_6h/new_engine-32                          37.88m ± ∞ ¹   40.81m ± ∞ ¹   +7.72% (p=0.008 n=5)
RangeQuery/stddev_over_time_5m/old_engine-32                       181.8m ± ∞ ¹   184.0m ± ∞ ¹        ~ (p=0.548 n=5)
RangeQuery/stddev_over_time_5m/new_engine-32                       25.49m ± ∞ ¹   25.40m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/stddev_over_time_1h/old_engine-32                       897.1m ± ∞ ¹   892.2m ± ∞ ¹        ~ (p=1.000 n=5)
RangeQuery/stddev_over_time_1h/new_engine-32                       77.90m ± ∞ ¹   79.34m ± ∞ ¹   +1.85% (p=0.008 n=5)
RangeQuery/stddev_over_time_6h/old_engine-32                        1.153 ± ∞ ¹    1.155 ± ∞ ¹        ~ (p=0.690 n=5)
RangeQuery/stddev_over_time_6h/new_engine-32                       97.42m ± ∞ ¹   99.62m ± ∞ ¹   +2.26% (p=0.008 n=5)
RangeQuery/stdvar_over_time/old_engine-32                          178.3m ± ∞ ¹   178.9m ± ∞ ¹        ~ (p=0.841 n=5)
RangeQuery/stdvar_over_time/new_engine-32                          25.21m ± ∞ ¹   24.95m ± ∞ ¹        ~ (p=0.421 n=5)
RangeQuery/last_over_time/old_engine-32                            90.39m ± ∞ ¹   90.08m ± ∞ ¹        ~ (p=0.421 n=5)
RangeQuery/last_over_time/new_engine-32                            22.97m ± ∞ ¹   23.47m ± ∞ ¹        ~ (p=0.222 n=5)
RangeQuery/present_over_time/old_engine-32                         80.29m ± ∞ ¹   83.40m ± ∞ ¹   +3.87% (p=0.016 n=5)
RangeQuery/present_over_time/new_engine-32                         23.67m ± ∞ ¹   23.90m ± ∞ ¹        ~ (p=0.056 n=5)
geomean                                                            52.86m         52.82m         -0.06%
¹ need >= 6 samples for confidence interval at level 0.95

                                                                 │   main.out    │           maxsamples.out            │
                                                                 │     B/op      │     B/op       vs base              │
RangeQuery/vector_selector/old_engine-32                           13.51Mi ± ∞ ¹   13.51Mi ± ∞ ¹       ~ (p=0.579 n=5)
RangeQuery/vector_selector/new_engine-32                           18.03Mi ± ∞ ¹   18.03Mi ± ∞ ¹       ~ (p=0.690 n=5)
RangeQuery/sum/old_engine-32                                       2.038Mi ± ∞ ¹   2.048Mi ± ∞ ¹       ~ (p=0.548 n=5)
RangeQuery/sum/new_engine-32                                       5.836Mi ± ∞ ¹   5.837Mi ± ∞ ¹       ~ (p=0.151 n=5)
RangeQuery/sum_by_pod/old_engine-32                                6.323Mi ± ∞ ¹   6.310Mi ± ∞ ¹       ~ (p=0.476 n=5)
RangeQuery/sum_by_pod/new_engine-32                                11.48Mi ± ∞ ¹   11.48Mi ± ∞ ¹  +0.00% (p=0.016 n=5)
RangeQuery/topk/old_engine-32                                      2.460Mi ± ∞ ¹   2.456Mi ± ∞ ¹       ~ (p=0.524 n=5)
RangeQuery/topk/new_engine-32                                      6.449Mi ± ∞ ¹   6.446Mi ± ∞ ¹  -0.06% (p=0.008 n=5)
RangeQuery/bottomk/old_engine-32                                   2.450Mi ± ∞ ¹   2.463Mi ± ∞ ¹       ~ (p=0.690 n=5)
RangeQuery/bottomk/new_engine-32                                   6.446Mi ± ∞ ¹   6.446Mi ± ∞ ¹  +0.00% (p=0.008 n=5)
RangeQuery/limitk/old_engine-32                                    3.799Mi ± ∞ ¹   3.792Mi ± ∞ ¹       ~ (p=0.135 n=5)
RangeQuery/limitk/new_engine-32                                    7.795Mi ± ∞ ¹   7.779Mi ± ∞ ¹  -0.20% (p=0.008 n=5)
RangeQuery/limit_ratio/old_engine-32                               21.00Mi ± ∞ ¹   20.98Mi ± ∞ ¹       ~ (p=0.087 n=5)
RangeQuery/limit_ratio/new_engine-32                               14.63Mi ± ∞ ¹   14.61Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/rate/old_engine-32                                      14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.10% (p=0.008 n=5)
RangeQuery/rate/new_engine-32                                      20.94Mi ± ∞ ¹   20.93Mi ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/rate_with_longer_window/old_engine-32                   4.406Mi ± ∞ ¹   4.390Mi ± ∞ ¹  -0.37% (p=0.008 n=5)
RangeQuery/rate_with_longer_window/new_engine-32                   9.607Mi ± ∞ ¹   9.591Mi ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/subquery/old_engine-32                                  22.09Mi ± ∞ ¹   22.08Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/subquery/new_engine-32                                  23.77Mi ± ∞ ¹   23.76Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/sum_rate/old_engine-32                                  3.487Mi ± ∞ ¹   3.469Mi ± ∞ ¹       ~ (p=0.087 n=5)
RangeQuery/sum_rate/new_engine-32                                  8.743Mi ± ∞ ¹   8.727Mi ± ∞ ¹  -0.18% (p=0.008 n=5)
RangeQuery/sum_by_rate/old_engine-32                               7.719Mi ± ∞ ¹   7.703Mi ± ∞ ¹       ~ (p=0.389 n=5)
RangeQuery/sum_by_rate/new_engine-32                               14.39Mi ± ∞ ¹   14.37Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/quantile_with_variable_parameter/old_engine-32          93.98Mi ± ∞ ¹   93.93Mi ± ∞ ¹       ~ (p=0.095 n=5)
RangeQuery/quantile_with_variable_parameter/new_engine-32          20.55Mi ± ∞ ¹   20.52Mi ± ∞ ¹  -0.15% (p=0.008 n=5)
RangeQuery/binary_operation_with_one_to_one/old_engine-32          7.139Mi ± ∞ ¹   7.132Mi ± ∞ ¹       ~ (p=0.063 n=5)
RangeQuery/binary_operation_with_one_to_one/new_engine-32          9.696Mi ± ∞ ¹   9.689Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/binary_operation_with_many_to_one/old_engine-32         62.40Mi ± ∞ ¹   62.38Mi ± ∞ ¹       ~ (p=0.063 n=5)
RangeQuery/binary_operation_with_many_to_one/new_engine-32         23.29Mi ± ∞ ¹   23.28Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/binary_operation_with_vector_and_scalar/old_engine-32   16.00Mi ± ∞ ¹   15.99Mi ± ∞ ¹       ~ (p=0.421 n=5)
RangeQuery/binary_operation_with_vector_and_scalar/new_engine-32   20.35Mi ± ∞ ¹   20.34Mi ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/unary_negation/old_engine-32                            14.93Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/unary_negation/new_engine-32                            19.88Mi ± ∞ ¹   19.86Mi ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/vector_and_scalar_comparison/old_engine-32              15.97Mi ± ∞ ¹   15.92Mi ± ∞ ¹       ~ (p=0.246 n=5)
RangeQuery/vector_and_scalar_comparison/new_engine-32              19.97Mi ± ∞ ¹   19.95Mi ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/positive_offset_vector/old_engine-32                    14.13Mi ± ∞ ¹   14.12Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/positive_offset_vector/new_engine-32                    18.66Mi ± ∞ ¹   18.65Mi ± ∞ ¹  -0.05% (p=0.008 n=5)
RangeQuery/at_modifier_/old_engine-32                              24.99Mi ± ∞ ¹   24.98Mi ± ∞ ¹  -0.02% (p=0.008 n=5)
RangeQuery/at_modifier_/new_engine-32                              16.03Mi ± ∞ ¹   16.02Mi ± ∞ ¹  -0.03% (p=0.008 n=5)
RangeQuery/at_modifier_with_positive_offset_vector/old_engine-32   24.99Mi ± ∞ ¹   24.98Mi ± ∞ ¹  -0.02% (p=0.008 n=5)
RangeQuery/at_modifier_with_positive_offset_vector/new_engine-32   16.03Mi ± ∞ ¹   16.02Mi ± ∞ ¹  -0.03% (p=0.008 n=5)
RangeQuery/clamp/old_engine-32                                     15.98Mi ± ∞ ¹   15.96Mi ± ∞ ¹       ~ (p=0.286 n=5)
RangeQuery/clamp/new_engine-32                                     19.88Mi ± ∞ ¹   19.86Mi ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/clamp_min/old_engine-32                                 15.98Mi ± ∞ ¹   15.96Mi ± ∞ ¹       ~ (p=0.278 n=5)
RangeQuery/clamp_min/new_engine-32                                 19.88Mi ± ∞ ¹   19.86Mi ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/complex_func_query/old_engine-32                        17.10Mi ± ∞ ¹   17.07Mi ± ∞ ¹       ~ (p=0.151 n=5)
RangeQuery/complex_func_query/new_engine-32                        20.84Mi ± ∞ ¹   20.83Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/func_within_func_query/old_engine-32                    3.159Mi ± ∞ ¹   3.143Mi ± ∞ ¹  -0.51% (p=0.008 n=5)
RangeQuery/func_within_func_query/new_engine-32                    7.157Mi ± ∞ ¹   7.141Mi ± ∞ ¹  -0.22% (p=0.008 n=5)
RangeQuery/aggr_within_func_query/old_engine-32                    3.159Mi ± ∞ ¹   3.143Mi ± ∞ ¹  -0.51% (p=0.008 n=5)
RangeQuery/aggr_within_func_query/new_engine-32                    6.928Mi ± ∞ ¹   6.912Mi ± ∞ ¹  -0.23% (p=0.008 n=5)
RangeQuery/histogram_quantile/old_engine-32                        22.23Mi ± ∞ ¹   22.26Mi ± ∞ ¹       ~ (p=0.079 n=5)
RangeQuery/histogram_quantile/new_engine-32                        37.57Mi ± ∞ ¹   37.59Mi ± ∞ ¹  +0.05% (p=0.008 n=5)
RangeQuery/sort/old_engine-32                                      16.01Mi ± ∞ ¹   15.99Mi ± ∞ ¹  -0.10% (p=0.040 n=5)
RangeQuery/sort/new_engine-32                                      19.39Mi ± ∞ ¹   19.37Mi ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/sort_desc/old_engine-32                                 16.01Mi ± ∞ ¹   15.99Mi ± ∞ ¹       ~ (p=0.397 n=5)
RangeQuery/sort_desc/new_engine-32                                 19.39Mi ± ∞ ¹   19.37Mi ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/absent_and_exists/old_engine-32                         4.032Mi ± ∞ ¹   4.008Mi ± ∞ ¹       ~ (p=0.087 n=5)
RangeQuery/absent_and_exists/new_engine-32                         7.179Mi ± ∞ ¹   7.163Mi ± ∞ ¹  -0.22% (p=0.008 n=5)
RangeQuery/absent_and_doesnt_exist/old_engine-32                   66.29Ki ± ∞ ¹   66.29Ki ± ∞ ¹       ~ (p=0.905 n=5)
RangeQuery/absent_and_doesnt_exist/new_engine-32                   177.0Ki ± ∞ ¹   177.3Ki ± ∞ ¹  +0.16% (p=0.008 n=5)
RangeQuery/double_exponential_smoothing/old_engine-32              81.16Mi ± ∞ ¹   81.14Mi ± ∞ ¹  -0.02% (p=0.008 n=5)
RangeQuery/double_exponential_smoothing/new_engine-32              20.80Mi ± ∞ ¹   20.79Mi ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/count_over_time_5m/old_engine-32                        14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/count_over_time_5m/new_engine-32                        21.90Mi ± ∞ ¹   21.89Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/count_over_time_1h/old_engine-32                        14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/count_over_time_1h/new_engine-32                        41.31Mi ± ∞ ¹   41.30Mi ± ∞ ¹  -0.04% (p=0.008 n=5)
RangeQuery/count_over_time_6h/old_engine-32                        14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/count_over_time_6h/new_engine-32                        64.69Mi ± ∞ ¹   64.67Mi ± ∞ ¹  -0.02% (p=0.008 n=5)
RangeQuery/sum_over_time_5m/old_engine-32                          14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.10% (p=0.008 n=5)
RangeQuery/sum_over_time_5m/new_engine-32                          21.90Mi ± ∞ ¹   21.89Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/sum_over_time_1h/old_engine-32                          14.94Mi ± ∞ ¹   14.93Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/sum_over_time_1h/new_engine-32                          41.31Mi ± ∞ ¹   41.30Mi ± ∞ ¹  -0.04% (p=0.008 n=5)
RangeQuery/sum_over_time_6h/old_engine-32                          14.94Mi ± ∞ ¹   14.93Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/sum_over_time_6h/new_engine-32                          64.69Mi ± ∞ ¹   64.67Mi ± ∞ ¹  -0.02% (p=0.008 n=5)
RangeQuery/avg_over_time_5m/old_engine-32                          14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/avg_over_time_5m/new_engine-32                          21.90Mi ± ∞ ¹   21.89Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/avg_over_time_1h/old_engine-32                          14.95Mi ± ∞ ¹   14.93Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/avg_over_time_1h/new_engine-32                          41.31Mi ± ∞ ¹   41.30Mi ± ∞ ¹  -0.04% (p=0.008 n=5)
RangeQuery/avg_over_time_6h/old_engine-32                          14.95Mi ± ∞ ¹   14.93Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/avg_over_time_6h/new_engine-32                          64.69Mi ± ∞ ¹   64.67Mi ± ∞ ¹  -0.02% (p=0.008 n=5)
RangeQuery/min_over_time_5m/old_engine-32                          14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/min_over_time_5m/new_engine-32                          21.90Mi ± ∞ ¹   21.89Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/min_over_time_1h/old_engine-32                          14.94Mi ± ∞ ¹   14.93Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/min_over_time_1h/new_engine-32                          41.31Mi ± ∞ ¹   41.30Mi ± ∞ ¹  -0.04% (p=0.008 n=5)
RangeQuery/min_over_time_6h/old_engine-32                          14.94Mi ± ∞ ¹   14.93Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/min_over_time_6h/new_engine-32                          64.69Mi ± ∞ ¹   64.67Mi ± ∞ ¹  -0.02% (p=0.008 n=5)
RangeQuery/max_over_time_5m/old_engine-32                          14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/max_over_time_5m/new_engine-32                          21.90Mi ± ∞ ¹   21.89Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/max_over_time_1h/old_engine-32                          14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/max_over_time_1h/new_engine-32                          41.31Mi ± ∞ ¹   41.30Mi ± ∞ ¹  -0.04% (p=0.008 n=5)
RangeQuery/max_over_time_6h/old_engine-32                          14.94Mi ± ∞ ¹   14.93Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/max_over_time_6h/new_engine-32                          64.69Mi ± ∞ ¹   64.67Mi ± ∞ ¹  -0.02% (p=0.008 n=5)
RangeQuery/stddev_over_time_5m/old_engine-32                       14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/stddev_over_time_5m/new_engine-32                       21.90Mi ± ∞ ¹   21.89Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/stddev_over_time_1h/old_engine-32                       14.95Mi ± ∞ ¹   14.94Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/stddev_over_time_1h/new_engine-32                       41.31Mi ± ∞ ¹   41.30Mi ± ∞ ¹  -0.04% (p=0.008 n=5)
RangeQuery/stddev_over_time_6h/old_engine-32                       14.97Mi ± ∞ ¹   14.96Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/stddev_over_time_6h/new_engine-32                       64.69Mi ± ∞ ¹   64.67Mi ± ∞ ¹  -0.02% (p=0.008 n=5)
RangeQuery/stdvar_over_time/old_engine-32                          14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/stdvar_over_time/new_engine-32                          21.90Mi ± ∞ ¹   21.89Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/last_over_time/old_engine-32                            14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/last_over_time/new_engine-32                            21.56Mi ± ∞ ¹   21.54Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
RangeQuery/present_over_time/old_engine-32                         14.94Mi ± ∞ ¹   14.92Mi ± ∞ ¹  -0.10% (p=0.008 n=5)
RangeQuery/present_over_time/new_engine-32                         21.90Mi ± ∞ ¹   21.89Mi ± ∞ ¹  -0.07% (p=0.008 n=5)
geomean                                                            14.67Mi         14.65Mi        -0.09%
¹ need >= 6 samples for confidence interval at level 0.95

                                                                 │   main.out   │            maxsamples.out            │
                                                                 │  allocs/op   │  allocs/op    vs base                │
RangeQuery/vector_selector/old_engine-32                           24.11k ± ∞ ¹   24.11k ± ∞ ¹       ~ (p=1.000 n=5)
RangeQuery/vector_selector/new_engine-32                           41.78k ± ∞ ¹   41.78k ± ∞ ¹       ~ (p=0.516 n=5)
RangeQuery/sum/old_engine-32                                       21.14k ± ∞ ¹   21.14k ± ∞ ¹       ~ (p=0.524 n=5)
RangeQuery/sum/new_engine-32                                       38.96k ± ∞ ¹   38.96k ± ∞ ¹       ~ (p=0.484 n=5)
RangeQuery/sum_by_pod/old_engine-32                                23.17k ± ∞ ¹   23.17k ± ∞ ¹       ~ (p=0.476 n=5)
RangeQuery/sum_by_pod/new_engine-32                                57.00k ± ∞ ¹   57.00k ± ∞ ¹       ~ (p=0.103 n=5)
RangeQuery/topk/old_engine-32                                      22.36k ± ∞ ¹   22.36k ± ∞ ¹       ~ (p=0.468 n=5)
RangeQuery/topk/new_engine-32                                      39.70k ± ∞ ¹   39.70k ± ∞ ¹       ~ (p=1.000 n=5)
RangeQuery/bottomk/old_engine-32                                   22.12k ± ∞ ¹   22.12k ± ∞ ¹       ~ (p=0.476 n=5)
RangeQuery/bottomk/new_engine-32                                   39.69k ± ∞ ¹   39.70k ± ∞ ¹       ~ (p=0.190 n=5)
RangeQuery/limitk/old_engine-32                                    38.88k ± ∞ ¹   38.73k ± ∞ ¹       ~ (p=0.079 n=5)
RangeQuery/limitk/new_engine-32                                    56.46k ± ∞ ¹   56.31k ± ∞ ¹  -0.26% (p=0.008 n=5)
RangeQuery/limit_ratio/old_engine-32                               41.64k ± ∞ ¹   41.48k ± ∞ ¹       ~ (p=0.071 n=5)
RangeQuery/limit_ratio/new_engine-32                               194.2k ± ∞ ¹   194.0k ± ∞ ¹  -0.08% (p=0.008 n=5)
RangeQuery/rate/old_engine-32                                      41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/rate/new_engine-32                                      79.84k ± ∞ ¹   79.69k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/rate_with_longer_window/old_engine-32                   41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/rate_with_longer_window/new_engine-32                   78.52k ± ∞ ¹   78.37k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/subquery/old_engine-32                                  47.22k ± ∞ ¹   47.07k ± ∞ ¹  -0.32% (p=0.008 n=5)
RangeQuery/subquery/new_engine-32                                  100.5k ± ∞ ¹   100.4k ± ∞ ¹  -0.15% (p=0.008 n=5)
RangeQuery/sum_rate/old_engine-32                                  38.17k ± ∞ ¹   38.02k ± ∞ ¹       ~ (p=0.087 n=5)
RangeQuery/sum_rate/new_engine-32                                  77.02k ± ∞ ¹   76.87k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/sum_by_rate/old_engine-32                               40.20k ± ∞ ¹   40.05k ± ∞ ¹       ~ (p=0.087 n=5)
RangeQuery/sum_by_rate/new_engine-32                               95.07k ± ∞ ¹   94.92k ± ∞ ¹  -0.16% (p=0.008 n=5)
RangeQuery/quantile_with_variable_parameter/old_engine-32          1.042M ± ∞ ¹   1.042M ± ∞ ¹       ~ (p=0.071 n=5)
RangeQuery/quantile_with_variable_parameter/new_engine-32          159.9k ± ∞ ¹   159.6k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/binary_operation_with_one_to_one/old_engine-32          30.64k ± ∞ ¹   30.54k ± ∞ ¹       ~ (p=0.063 n=5)
RangeQuery/binary_operation_with_one_to_one/new_engine-32          46.10k ± ∞ ¹   46.00k ± ∞ ¹  -0.21% (p=0.008 n=5)
RangeQuery/binary_operation_with_many_to_one/old_engine-32         546.0k ± ∞ ¹   545.8k ± ∞ ¹       ~ (p=0.063 n=5)
RangeQuery/binary_operation_with_many_to_one/new_engine-32         87.44k ± ∞ ¹   87.26k ± ∞ ¹  -0.21% (p=0.008 n=5)
RangeQuery/binary_operation_with_vector_and_scalar/old_engine-32   41.17k ± ∞ ¹   41.02k ± ∞ ¹       ~ (p=0.063 n=5)
RangeQuery/binary_operation_with_vector_and_scalar/new_engine-32   67.99k ± ∞ ¹   67.84k ± ∞ ¹  -0.22% (p=0.008 n=5)
RangeQuery/unary_negation/old_engine-32                            41.14k ± ∞ ¹   40.99k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/unary_negation/new_engine-32                            67.83k ± ∞ ¹   67.68k ± ∞ ¹  -0.22% (p=0.008 n=5)
RangeQuery/vector_and_scalar_comparison/old_engine-32              41.16k ± ∞ ¹   41.00k ± ∞ ¹       ~ (p=0.103 n=5)
RangeQuery/vector_and_scalar_comparison/new_engine-32              58.98k ± ∞ ¹   58.83k ± ∞ ¹  -0.25% (p=0.008 n=5)
RangeQuery/positive_offset_vector/old_engine-32                    32.45k ± ∞ ¹   32.35k ± ∞ ¹  -0.31% (p=0.008 n=5)
RangeQuery/positive_offset_vector/new_engine-32                    50.12k ± ∞ ¹   50.02k ± ∞ ¹  -0.20% (p=0.008 n=5)
RangeQuery/at_modifier_/old_engine-32                              50.79k ± ∞ ¹   50.74k ± ∞ ¹  -0.10% (p=0.008 n=5)
RangeQuery/at_modifier_/new_engine-32                              42.30k ± ∞ ¹   42.25k ± ∞ ¹  -0.11% (p=0.008 n=5)
RangeQuery/at_modifier_with_positive_offset_vector/old_engine-32   50.80k ± ∞ ¹   50.75k ± ∞ ¹  -0.10% (p=0.008 n=5)
RangeQuery/at_modifier_with_positive_offset_vector/new_engine-32   42.30k ± ∞ ¹   42.25k ± ∞ ¹  -0.12% (p=0.008 n=5)
RangeQuery/clamp/old_engine-32                                     41.18k ± ∞ ¹   41.03k ± ∞ ¹       ~ (p=0.063 n=5)
RangeQuery/clamp/new_engine-32                                     67.89k ± ∞ ¹   67.74k ± ∞ ¹  -0.22% (p=0.008 n=5)
RangeQuery/clamp_min/old_engine-32                                 41.17k ± ∞ ¹   41.02k ± ∞ ¹       ~ (p=0.063 n=5)
RangeQuery/clamp_min/new_engine-32                                 67.88k ± ∞ ¹   67.73k ± ∞ ¹  -0.22% (p=0.008 n=5)
RangeQuery/complex_func_query/old_engine-32                        41.28k ± ∞ ¹   41.13k ± ∞ ¹       ~ (p=0.095 n=5)
RangeQuery/complex_func_query/new_engine-32                        77.12k ± ∞ ¹   76.97k ± ∞ ¹  -0.20% (p=0.008 n=5)
RangeQuery/func_within_func_query/old_engine-32                    38.24k ± ∞ ¹   38.09k ± ∞ ¹  -0.39% (p=0.008 n=5)
RangeQuery/func_within_func_query/new_engine-32                    78.66k ± ∞ ¹   78.51k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/aggr_within_func_query/old_engine-32                    38.24k ± ∞ ¹   38.09k ± ∞ ¹  -0.39% (p=0.008 n=5)
RangeQuery/aggr_within_func_query/new_engine-32                    78.66k ± ∞ ¹   78.51k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/histogram_quantile/old_engine-32                        152.7k ± ∞ ¹   153.0k ± ∞ ¹       ~ (p=0.079 n=5)
RangeQuery/histogram_quantile/new_engine-32                        277.7k ± ∞ ¹   277.9k ± ∞ ¹  +0.08% (p=0.008 n=5)
RangeQuery/sort/old_engine-32                                      41.64k ± ∞ ¹   41.49k ± ∞ ¹  -0.36% (p=0.040 n=5)
RangeQuery/sort/new_engine-32                                      58.79k ± ∞ ¹   58.65k ± ∞ ¹  -0.25% (p=0.008 n=5)
RangeQuery/sort_desc/old_engine-32                                 41.64k ± ∞ ¹   41.49k ± ∞ ¹       ~ (p=0.087 n=5)
RangeQuery/sort_desc/new_engine-32                                 58.79k ± ∞ ¹   58.64k ± ∞ ¹  -0.26% (p=0.008 n=5)
RangeQuery/absent_and_exists/old_engine-32                         38.15k ± ∞ ¹   38.00k ± ∞ ¹       ~ (p=0.087 n=5)
RangeQuery/absent_and_exists/new_engine-32                         55.81k ± ∞ ¹   55.66k ± ∞ ¹  -0.27% (p=0.008 n=5)
RangeQuery/absent_and_doesnt_exist/old_engine-32                    593.0 ± ∞ ¹    593.0 ± ∞ ¹       ~ (p=1.000 n=5) ²
RangeQuery/absent_and_doesnt_exist/new_engine-32                   1.459k ± ∞ ¹   1.460k ± ∞ ¹  +0.07% (p=0.008 n=5)
RangeQuery/double_exponential_smoothing/old_engine-32              1.487M ± ∞ ¹   1.487M ± ∞ ¹  -0.01% (p=0.008 n=5)
RangeQuery/double_exponential_smoothing/new_engine-32              73.87k ± ∞ ¹   73.72k ± ∞ ¹  -0.20% (p=0.008 n=5)
RangeQuery/count_over_time_5m/old_engine-32                        41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/count_over_time_5m/new_engine-32                        76.84k ± ∞ ¹   76.70k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/count_over_time_1h/old_engine-32                        41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/count_over_time_1h/new_engine-32                        85.85k ± ∞ ¹   85.70k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/count_over_time_6h/old_engine-32                        41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/count_over_time_6h/new_engine-32                        85.85k ± ∞ ¹   85.71k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/sum_over_time_5m/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/sum_over_time_5m/new_engine-32                          76.84k ± ∞ ¹   76.70k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/sum_over_time_1h/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/sum_over_time_1h/new_engine-32                          85.85k ± ∞ ¹   85.70k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/sum_over_time_6h/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/sum_over_time_6h/new_engine-32                          85.85k ± ∞ ¹   85.70k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/avg_over_time_5m/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/avg_over_time_5m/new_engine-32                          76.84k ± ∞ ¹   76.70k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/avg_over_time_1h/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/avg_over_time_1h/new_engine-32                          85.85k ± ∞ ¹   85.70k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/avg_over_time_6h/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/avg_over_time_6h/new_engine-32                          85.85k ± ∞ ¹   85.70k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/min_over_time_5m/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/min_over_time_5m/new_engine-32                          76.84k ± ∞ ¹   76.70k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/min_over_time_1h/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/min_over_time_1h/new_engine-32                          85.85k ± ∞ ¹   85.70k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/min_over_time_6h/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/min_over_time_6h/new_engine-32                          85.86k ± ∞ ¹   85.71k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/max_over_time_5m/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/max_over_time_5m/new_engine-32                          76.85k ± ∞ ¹   76.70k ± ∞ ¹  -0.20% (p=0.008 n=5)
RangeQuery/max_over_time_1h/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/max_over_time_1h/new_engine-32                          85.85k ± ∞ ¹   85.70k ± ∞ ¹  -0.18% (p=0.008 n=5)
RangeQuery/max_over_time_6h/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/max_over_time_6h/new_engine-32                          85.86k ± ∞ ¹   85.71k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/stddev_over_time_5m/old_engine-32                       41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/stddev_over_time_5m/new_engine-32                       76.85k ± ∞ ¹   76.70k ± ∞ ¹  -0.19% (p=0.008 n=5)
RangeQuery/stddev_over_time_1h/old_engine-32                       41.16k ± ∞ ¹   41.01k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/stddev_over_time_1h/new_engine-32                       85.85k ± ∞ ¹   85.70k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/stddev_over_time_6h/old_engine-32                       41.17k ± ∞ ¹   41.02k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/stddev_over_time_6h/new_engine-32                       85.86k ± ∞ ¹   85.70k ± ∞ ¹  -0.17% (p=0.008 n=5)
RangeQuery/stdvar_over_time/old_engine-32                          41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/stdvar_over_time/new_engine-32                          76.85k ± ∞ ¹   76.70k ± ∞ ¹  -0.20% (p=0.008 n=5)
RangeQuery/last_over_time/old_engine-32                            41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/last_over_time/new_engine-32                            67.85k ± ∞ ¹   67.70k ± ∞ ¹  -0.22% (p=0.008 n=5)
RangeQuery/present_over_time/old_engine-32                         41.15k ± ∞ ¹   41.00k ± ∞ ¹  -0.36% (p=0.008 n=5)
RangeQuery/present_over_time/new_engine-32                         76.84k ± ∞ ¹   76.70k ± ∞ ¹  -0.19% (p=0.008 n=5)
geomean                                                            54.70k         54.58k        -0.22%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal

Signed-off-by: Paurush Garg <paurushg@amazon.com>

yeya24 · 2026-02-10T19:52:12Z

storage/prometheus/vector_selector.go

 		}
+
+		if o.opts.SampleTracker != nil && (o.currentSeries+1-fromSeries)%maxSamplesCheckIntervalSeries == 0 {
+			if err := o.opts.SampleTracker.CheckLimit(); err != nil {


Maybe I missed something here. But where do we add samples to the tracker? I only see we add to the samples after looping all series.

Yes makes sense.
I updated it now.
Thanks.

yeya24 · 2026-02-10T19:59:37Z

storage/prometheus/matrix_selector.go

+		if o.opts.SampleTracker != nil && (o.currentSeries+1-firstSeries)%maxSamplesCheckIntervalSeries == 0 {
+			totalSamplesInBatch := 0
+			for i := range o.scanners {
+				totalSamplesInBatch += o.scanners[i].buffer.SampleCount()
+			}


It seems that we are counting sample count from all series not just the current series? It seems that we can just maintain a running sum for processed series instead of counting all processed series again

Yes that would make more sense.
Updated now to count sample for only current series.
Thanks.

yeya24 · 2026-02-10T20:31:10Z

execution/scan/subquery.go

+	lastTrackedSamples int
 }

+const maxSamplesCheckIntervalSteps = 100


I wonder if we have a way to infer a good sample check interval dynamically. Cardinality and number of total steps are known when we call Next(). So ideally we can check more frequently if the query has high cardinality?

I have moved the checkSampleLimit to inner loop as noted from the run on beta cell, the instant subQueries were getting checked too late. Also added dynamic interval check.
Thanks.

PaurushGarg · 2026-02-12T00:35:18Z

Could you add unit tests to the PR? Could you also add benchmarks before and after enabling the sample limiter just to see if there is any big impact there?

Thanks.
I have added unit tests.
benchmarks test results: #663 (comment)

…_selector maxSamples logic Signed-off-by: Paurush Garg <paurushg@amazon.com>

harry671003

Thanks! This is great!

yeya24

Only small comments

storage/prometheus/vector_selector.go

storage/prometheus/matrix_selector.go

Signed-off-by: Paurush Garg <paurushg@amazon.com>

yeya24

Thanks. LGTM

PaurushGarg · 2026-02-17T23:55:35Z

@MichaHoffmann requesting review.

fpetkovski · 2026-02-18T06:30:46Z

execution/scan/subquery.go

 				o.collect(vector, mint)
 			}

+			if o.opts.SampleTracker != nil {


I find it strange that we apply this limit in subqueries. They operate on existing samples and do not produce new ones.

Selectors reuse the same buffer each next, while subqueries accumulate samples in ring buffers that persist across the evaluation window.
For rate(http_requests[10m])[2d:1m], the inner selector loads samples and releases them, but the subquery keeps accumulating those in buffers for the full 2d window.

storage/prometheus/matrix_selector.go

Signed-off-by: Paurush Garg <paurushg@amazon.com>

MichaHoffmann · 2026-02-19T11:32:09Z

I reran the fuzz failure and it failed again, it seems to be real, can you dig into it please?

yeya24 · 2026-02-19T18:25:50Z

The same NH fuzz test seems failing for me in my PR as well. I wonder if it is just a flake. Someone should take a look and disable if it is flaky

PaurushGarg · 2026-02-26T00:10:59Z

I reran the fuzz failure and it failed again, it seems to be real, can you dig into it please?

@MichaHoffmann @yeya24
This appears to be a flaky test failure unrelated to the maxSamples limit implementation in this PR.
The fuzz test randomly generated this query:
(-sum without (pod) (predict_linear({__name__="http_request_duration_seconds"}[3m] @ 0.000 offset -1m45s, 0.010906995263821257)) and rate({__name__="http_request_duration_seconds"}[5m]))

This matches the known problematic pattern documented here of enginefuzz_test.go: "Thanos engine cannot correctly handle a MatrixSelector wrapped by StepInvariant". However, validateExpr didn't catch this specific variant (likely due to the offset modifier) - leading to the sample count mismatch.

MichaHoffmann · 2026-03-03T06:54:27Z

storage/prometheus/matrix_selector.go


 var ErrNativeHistogramsNotSupported = errors.New("native histograms are not supported in extended range functions")

+const sampleLimitCheckInterval = 500


Why do we have sampleLimitCheckInterval here and sampleLimitCheckPercentage in subquery.go, thats weird

PaurushGarg force-pushed the maxSamplesLimit branch 3 times, most recently from 42785ef to 54b94b6 Compare January 29, 2026 01:23

harry671003 reviewed Jan 31, 2026

View reviewed changes

PaurushGarg force-pushed the maxSamplesLimit branch from 62c38b1 to 9ac367e Compare February 9, 2026 22:45

PaurushGarg added 5 commits February 9, 2026 16:16

max samples limit

4c169c9

Signed-off-by: Paurush Garg <paurushg@amazon.com>

Update max_samples tracking to selector operators

8098487

Signed-off-by: Paurush Garg <paurushg@amazon.com>

Update to check maxSamples more frequently

1fc6e89

Signed-off-by: Paurush Garg <paurushg@amazon.com>

Updating maxSample check interval steps

734630c

Signed-off-by: Paurush Garg <paurushg@amazon.com>

Add unit tests for max_samples

1a7cf64

Signed-off-by: Paurush Garg <paurushg@amazon.com>

PaurushGarg force-pushed the maxSamplesLimit branch from 9ac367e to 1a7cf64 Compare February 10, 2026 00:16

Apply linter formatting

975cce8

Signed-off-by: Paurush Garg <paurushg@amazon.com>

PaurushGarg force-pushed the maxSamplesLimit branch from 467b1f7 to 975cce8 Compare February 10, 2026 07:24

yeya24 reviewed Feb 10, 2026

View reviewed changes

PaurushGarg force-pushed the maxSamplesLimit branch from 6ba7bab to ba9b758 Compare February 12, 2026 00:58

Update SubQuery Sample tracker to inner loop and remove update matrix…

8945224

…_selector maxSamples logic Signed-off-by: Paurush Garg <paurushg@amazon.com>

PaurushGarg force-pushed the maxSamplesLimit branch from ba9b758 to 8945224 Compare February 12, 2026 01:01

PaurushGarg requested review from harry671003 and yeya24 February 12, 2026 01:03

harry671003 approved these changes Feb 12, 2026

View reviewed changes

yeya24 reviewed Feb 12, 2026

View reviewed changes

storage/prometheus/vector_selector.go Show resolved Hide resolved

storage/prometheus/matrix_selector.go Outdated Show resolved Hide resolved

Remove Checking when delta is negative in matrix_selector

418afb2

Signed-off-by: Paurush Garg <paurushg@amazon.com>

PaurushGarg requested a review from yeya24 February 12, 2026 18:55

yeya24 approved these changes Feb 12, 2026

View reviewed changes

fpetkovski reviewed Feb 18, 2026

View reviewed changes

storage/prometheus/matrix_selector.go Outdated Show resolved Hide resolved

convert SampleTracker to interface with nop implementation

639ec0f

Signed-off-by: Paurush Garg <paurushg@amazon.com>

PaurushGarg requested a review from fpetkovski February 18, 2026 23:51

MichaHoffmann reviewed Mar 3, 2026

View reviewed changes

yeya24 mentioned this pull request Mar 16, 2026

replace deprecated label name validation with UTF8Validation #694

Merged


		var ErrNativeHistogramsNotSupported = errors.New("native histograms are not supported in extended range functions")

		const sampleLimitCheckInterval = 500

Conversation

PaurushGarg commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaHoffmann commented Dec 3, 2025

Uh oh!

PaurushGarg commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harry671003 commented Jan 31, 2026

Uh oh!

PaurushGarg commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yeya24 Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PaurushGarg commented Feb 12, 2026

Uh oh!

harry671003 left a comment

Choose a reason for hiding this comment

Uh oh!

yeya24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yeya24 left a comment

Choose a reason for hiding this comment

Uh oh!

PaurushGarg commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PaurushGarg Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MichaHoffmann commented Feb 19, 2026

Uh oh!

yeya24 commented Feb 19, 2026

Uh oh!

PaurushGarg commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PaurushGarg commented Dec 2, 2025 •

edited

Loading

PaurushGarg commented Jan 29, 2026 •

edited

Loading

PaurushGarg commented Feb 10, 2026 •

edited

Loading

yeya24 Feb 10, 2026 •

edited

Loading

PaurushGarg commented Feb 17, 2026 •

edited

Loading

PaurushGarg Feb 18, 2026 •

edited

Loading

PaurushGarg commented Feb 26, 2026 •

edited

Loading