Reproducibility of the NeurIPS 2020 Black-Box Optimization Challenge

Code and experimental infrastructure for reproducing and extending the results of the NeurIPS 2020 BBO Challenge. This repository accompanies the paper "Reproducibility of the NeurIPS 2020 Black-Box Optimization Challenge: Re-evaluating Competition Submissions".

Overview

We evaluate 8 optimizers on 14 benchmarks from 5 benchmark families, using the original competition protocol (16 iterations × 8 batch size = 128 evaluations per task, 3 independent trials).

Optimizers

Optimizer	Directory	Virtual Environment
RandomSearch (baseline)	random_search/	baseline_env
HEBO	hebo_bbo/submission/	hebo_env
TuRBO	turbo_bbo/	turbo_env
Optuna (MKSTuRBO)	optuna_bbo/submissions/mksturbo/	optuna_env
JetBrains (SpaceDecay)	jetbrains_bbo/submissions/space-decay/	jetbrains_env
RAPIDS-TuRBO	rapids_bbo/example_submissions/turbo/	rapids_env
RAPIDS-SkOpt	rapids_bbo/example_submissions/skopt/	rapids_env
RAPIDS-PySOT	rapids_bbo/example_submissions/pysot/	rapids_env

14 Benchmarks

Benchmark	Family	Type
MechBench-Welded	MechBench	Engineering (4D)
MechBench-Spring	MechBench	Engineering (3D)
MechBench-Pressure	MechBench	Engineering (4D)
BBOB-20D	BBOB	Numerical optimization
BBOB-40D	BBOB	Numerical optimization
Bayesmark-SVM-digits	Bayesmark	ML HPO
Bayesmark-SVM-breast	Bayesmark	ML HPO
Bayesmark-DT-digits	Bayesmark	ML HPO
Bayesmark-DT-diabetes	Bayesmark	ML HPO
Bayesmark-RF-breast	Bayesmark	ML HPO
Bayesmark-RF-wine	Bayesmark	ML HPO
LCBench-adult	LCBench	Synthetic NN HPO
LCBench-fashion	LCBench	Synthetic NN HPO
HPO-B-5527	HPO-B	Tabular HPO

Setup

1. Create Virtual Environments

Each optimizer runs in its own isolated Python environment to avoid dependency conflicts (a key design choice for handling legacy codebases from 2020).

python setup_environments.py

This creates 9 virtual environments under venvs/:

• Optimizer envs: baseline_env, hebo_env, turbo_env, optuna_env, jetbrains_env, rapids_env
• Benchmark envs: benchmark_env, bbob_env, hpob_env

Use --force to recreate all environments without prompting.

2. HPO-B Data

The HPO-B benchmark requires tabular data in hpob-data/. The setup script will offer to download it automatically (~221MB). If skipped, the HPO-B-5527 benchmark will be unavailable.

Running Experiments

Quick smoke test

TEST_ITERS=2 TEST_BATCH=4 python3 test_all_optimizers.py

Full local run

python3 test_all_optimizers.py

SLURM cluster (as used in the paper)

sbatch run_bbo_benchmark.slurm

This runs 3 trials × 8 optimizers × 14 benchmarks × 128 evaluations = 43,008 total evaluations on the c23ms partition (4 CPUs, 16GB RAM, CPU-only).

Architecture

test_all_optimizers.py          # Main orchestrator (8 opt × 14 bench)
├── runners/run_external.py     # Optimizer ↔ Benchmark communication (JSON over stdout)
│   ├── <optimizer venv>        # Each optimizer in isolated environment
│   └── benchmarks/service.py   # Benchmark subprocess service
│       └── benchmarks/*.py     # Benchmark implementations
├── run_bbo_benchmark.slurm     # SLURM job submission
└── setup_environments.py       # Environment creation & patching

Each optimizer implements the bayesmark.AbstractOptimizer interface (suggest() / observe()). The orchestrator launches each optimizer in its own venv subprocess and communicates via JSON over stdin/stdout.

Results

Results are stored in bachelor res/trajectories_*/ and include:

• Per-trial JSON trajectory files for each optimizer-benchmark pair
• Aggregated CSV results and summary tables
• Statistical comparison analyses

Requirements

• Python 3.12+ (system), Python 3.7 and 3.6 (via pyenv, for legacy optimizer envs)
• Linux (tested on RHEL/CentOS HPC cluster)
• ~2GB disk for virtual environments
• ~221MB for HPO-B data