MF + Priors

Experiments for the paper (TODO: link).

Installation

1. Clone the repository

git clone --recursive https://github.com/automl/mf-prior-exp.git

# To use the version used for the paper
git clone --branch "vPaper-priorband" --recursive https://github.com/automl/mf-prior-exp.git

# If you forgot to use the --recursive flag or checkout the branches, you can do the following
# to manuall perform the steps
cd mf-prior-exp

# If repeating experiments from the paper
# git checkout vPaper-priorband

git submodule update --init

2. Conda, Poetry, Package, Pre-Commit

To setup tooling and install the package, follow this documentation using the environment name of your choice.

poetry install

NOTE: The version used for the paper vPaper-PriorBand requires an older version of NePS which requires Python version <3.8 so consider creating an environment with conda create -n mf-prior python=3.7.12

3. Just

To install our command runner just you can check the just documentation, or run the below command

curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to $HOME/.just

Also, to make the just command available you should add

export PATH="$HOME/.just:$PATH"

to your .zshrc / .bashrc or alternatively simply run the export manually.

4. Data

python -m mfpbench.download --data-dir data

Usage

Running a singular local experiment

just run random_search jahs_cifar10

For more options see

python -m mf_prior_experiments.run -h

and run the python command directly, e.g.,

python -m mf_prior_experiments.run algorithm=random_search benchmark=jahs_cifar10 experiment_group=debug

Running a grid of experiments on slurm

To run 10 seeds for two algorithms and benchmarks, e.g.,

just submit alg_a,alg_b bench_a,bench_b "range(0, 10)" 22-08-18_updated-priors

note the whitespace in "range(0, 10)".

For more options see

just

Analysing experiments

Contributing

Working with git submodules

See the git-scm documentation. In short:

To pull in changes for mf-prior-exp and all submodules (neps and mf-prior-bench) run

git pull --recurse-submodules

To pull in changes from one submodule, change to its directory and run

git fetch
git merge origin/main

To code in the submodule first change to its directory, then checkout the branch you want to work on, e.g.,

cd src/neps
git checkout master

then perform, commit, and push your changes to the submodule's repository

pre-commit install
git commit -m "awesome changes"
git push

next, you also need to commit the changed submodule, e.g.,

cd ../..
git add src/neps
git commit -m "update neps submodule"
git push

Managing dependencies

For how to manage dependencies see the overview on poetry.

Tooling

There is also some documentation for the tools in this repo.