mobo — Multi-Objective Bayesian Optimization Benchmarks

Research question: when optimizing several conflicting objectives under a tight evaluation budget, how much do parallel, batch-aware multi-objective acquisition strategies (qNEHVI, qEHVI, qNParEGO) actually buy you over random search — and how does that answer change if the surrogate model is a differentiable Gaussian Process versus a non-differentiable random-forest ensemble?

This repo runs that comparison on a synthetic, noisy, 6D-input / 3-objective test problem built on BoTorch, and reports sample efficiency (hypervolume vs. number of evaluations) and wall-clock cost per iteration.

Main result figure

Dominated hypervolume attained vs. number of function evaluations, GP surrogate, all four methods sharing one seeded initial design. Higher and to the left is better.

Key findings

• All three MOBO acquisition functions dominate random search by a wide margin. Random search's hypervolume stays flat at 0 over the full budget in the figure above — the synthetic objectives are concentrated in a small region of the input space that uniform sampling essentially never hits.
• qNEHVI and qEHVI track each other closely and outperform qNParEGO. Both directly target expected hypervolume improvement; qNParEGO's random-scalarization approach reaches a visibly lower hypervolume plateau over the same budget.
• That sample efficiency has a runtime cost. qNEHVI's noisy expected-hypervolume computation is the most expensive per iteration (tens of seconds and growing with the training set, since it integrates over the joint posterior at all observed points), qEHVI is markedly cheaper, and qNParEGO / random are near-instant — see results/runtime.png.
• The Pareto front is recovered, not just individual objectives. Circling each method's non-dominated points (results/ob_0_1.png) shows qEHVI/qNEHVI cluster tight frontiers of non-dominated solutions, while random search's non-dominated set stays sparse and scattered.

Why the problem matters

Many real optimization problems — materials design, hyperparameter tuning, engineering trade-off studies — require balancing several expensive-to-evaluate, conflicting objectives at once, where each additional evaluation is costly (a lab experiment, a simulation run, a full training job). Multi-objective Bayesian optimization is the standard tool for that regime: it uses a probabilistic surrogate to decide which candidate to evaluate next, rather than sampling blindly. This project exists to make the tradeoffs between the common acquisition strategies concrete — how much sample efficiency you gain, at what runtime cost, and how sensitive that gain is to the choice of surrogate model — rather than taking published claims on synthetic benchmarks at face value.

Methodology

• Test problem (mobo/problem.py): MyProblem maps a 6D input in [0, 1]^6 to 3 objectives, each a shifted negative-quadratic bowl over an overlapping subset of the inputs, so no single point maximizes all three simultaneously.
• Initial design (mobo/data.py): every method in a run starts from the same seeded Sobol sequence (generate_initial_data), so hypervolume curves across methods are directly comparable rather than confounded by different starting points.
• Surrogates (mobo/models.py, mobo/random_forest.py):
  • surrogate="gp" — one SingleTaskGP per objective in a ModelListGP, refit via fit_gpytorch_mll each iteration. Differentiable, so acquisition is optimized with gradients.
  • surrogate="rf" — a single multi-output RandomForestRegressor wrapped as a BoTorch EnsembleModel, treating each of 500 trees as one posterior sample. Not differentiable, so acquisition optimization falls back to a gradient-free search, and retraining every iteration makes it noticeably slower.
• Acquisition functions (mobo/acquisition.py): qLogNoisyExpectedHypervolumeImprovement (qNEHVI), qLogExpectedHypervolumeImprovement (qEHVI), and Chebyshev-scalarized qLogNoisyExpectedImprovement with random weights per candidate (qNParEGO), each optimized over the unit cube with optimize_acqf / optimize_acqf_list.
• Experiment loop (mobo/experiment.py): run_bo_experiment batches all requested methods through the same iteration count, refitting each method's surrogate and proposing batch_size new points per iteration, tracking dominated hypervolume (DominatedPartitioning) and per-iteration wall-clock runtime throughout.
• Metric: dominated hypervolume relative to a fixed reference point, computed after every batch of new observations — the standard scalar measure of multi-objective optimization progress.

Reproducing the results

git clone https://github.com/sammyschen/MOBO.git
cd MOBO

Requires Python >= 3.10. Developed and tested against torch==2.12.1, botorch==0.18.1, gpytorch==1.15.2, scikit-learn==1.9.0.

import torch
from mobo import MyProblem, BOConfig, run_bo_experiment, plotting

tkwargs = {"dtype": torch.double, "device": "cpu"}

problem = MyProblem(
    dim=6,
    bounds=torch.tensor([[0.0] * 6, [1.0] * 6], **tkwargs),
    num_objectives=3,
    ref_point=torch.tensor([-3.0, -5.0, -3.0], **tkwargs),
)

config = BOConfig(
    n_batch=10, batch_size=2, num_restarts=3, raw_samples=64, mc_samples=32,
    noise_se=torch.tensor([0.1, 0.2, 0.3], **tkwargs), seed=711,
)

result = run_bo_experiment(problem, methods=["random", "qnehvi"], config=config, surrogate="gp")

plotting.plot_hypervolume(result)
plotting.plot_objective_scatter(result, obj_idx=(1, 2), highlight_pareto=True)

The figures above come from examples/gp_all_methods.ipynb, which runs all four methods (random, qnparego, qehvi, qnehvi) on the GP surrogate and calls plotting.plot_hypervolume, plotting.plot_objective_scatter, and plotting.plot_runtime. examples/rf_all_methods.ipynb runs the same comparison on the random-forest surrogate; examples/gp_qnehvi_vs_random.ipynb and examples/rf_qnehvi_vs_random.ipynb are smaller two-method versions of each. Increase n_batch / lower noise to reproduce closer to the exact curves shown here — the committed notebooks use a smaller budget so they run quickly.

Repository structure

mobo/                importable package
  problem.py            MyProblem — the synthetic 6D -> 3-objective test function
  random_forest.py      BoTorchRandomForest — sklearn RandomForestRegressor wrapped as a BoTorch EnsembleModel
  models.py             initialize_gp_model / initialize_rf_model surrogate factories
  data.py                generate_initial_data — seeded Sobol initial design
  acquisition.py         optimize_{qehvi,qnehvi,qnparego}_and_get_observation
  experiment.py          run_bo_experiment — the generalized BO loop (BOConfig / ExperimentResult)
  plotting.py            plot_hypervolume / plot_objective_scatter / plot_runtime

examples/             thin notebooks (config + a few calls into `mobo`) for each surrogate x method-set combo
results/              figures referenced in this README
tests/                pytest smoke tests

Tests and validation

pytest

tests/test_problem.py checks MyProblem's output shape and batching behavior stays finite and well-formed. tests/test_experiment.py runs run_bo_experiment end-to-end on both surrogates with a minimal budget (n_batch=1), asserting on result shapes, NaN-free hypervolume, that all methods share an identical initial hypervolume (since they share one seeded Sobol batch), and that unknown method names are rejected. These are smoke tests for correctness and wiring, not statistical benchmarks — they run in seconds and are checked on every push via GitHub Actions (.github/workflows/tests.yml).

Limitations

• Single synthetic problem, single seed per figure. The results above are one run; no repeated-seed error bars or statistical significance testing across runs. Hypervolume curves are known to be noisy across seeds, especially early on.
• Small, fixed problem scale. 6 inputs / 3 objectives is a convenient testbed, not a scaling study — behavior on higher-dimensional or more-objective problems is not characterized here.
• Noise is synthetic and homoscedastic, applied identically regardless of input; real black-box objectives are rarely this well-behaved.
• Random-forest surrogate uncertainty is a coarse proxy. Treating each tree as one posterior "sample" gives usable but rougher uncertainty estimates than the GP's proper posterior, which likely disadvantages RF-based acquisition beyond just its slower runtime.
• qNEHVI's runtime growth is not amortized or approximated — no caching or partition approximations are used, so its cost-per-iteration climbs as the training set grows.

Future work

• Repeat each comparison across multiple seeds and report hypervolume mean ± CI.
• Extend to additional synthetic benchmarks (e.g. standard multi-objective test suites) and to problems with more objectives / higher input dimension.
• Add a noisy, heteroscedastic observation model.
• Explore additional surrogates (e.g. deep ensembles, sparse GPs) under the same harness.
• Profile and optimize qNEHVI's per-iteration cost (e.g. partition caching).

References

• Daulton, S., Balandat, M., & Bakshy, E. (2020). Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization. NeurIPS 2020. — qEHVI
• Daulton, S., Balandat, M., & Bakshy, E. (2021). Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement. NeurIPS 2021. — qNEHVI
• Knowles, J. (2006). ParEGO: A Hybrid Algorithm with On-Line Landscape Approximation for Expensive Multiobjective Optimization Problems. IEEE Transactions on Evolutionary Computation. — (q)ParEGO
• Balandat, M. et al. (2020). BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. NeurIPS 2020.