Skip to content

README.md

Latest commit

 

History

History
147 lines (119 loc) · 6.73 KB

README.md

File metadata and controls

147 lines (119 loc) · 6.73 KB
Raw

Benchmark suite

The analytical / catalog sweeps behind §7 of the paper, plus the switch-driven runners for the broader catalog of ~30 problems that §7.1 points to but does not show explicitly.

Contents

benchmarks/
├── README.md                ← this file
├── requirements.txt         ← pip dependencies
├── run_olympus.py           ← Olympus real-data runner; --task switch (default photo_pce10)
├── make_convergence_figure.py ← renders the §7.4 photo_pce10 convergence-vs-budget figure
├── baselines/               ← pymoo/pycma baselines for the §7 examples (see its README)
│   ├── problem_pymoo.py     ← shared YAML+NN problem (feasibility + max_violation)
│   └── run_baselines.py     ← CMA-ES/DE/GA/PSO (SO) and NSGA-II/III/MOEA-D (MO)
├── extra_results/           ← committed result snapshots beyond §7 (see its README)
├── bbob/
│   ├── README.md            ← BBOB switch reference
│   ├── run_bbob_suites.py   ← COCO suite driver; --suite switch (default bbob-biobj-mixint, §7.2)
│   ├── run_bbob_stress.py   ← f15–f24 hard-multimodal stress test (§6.3), no COCO
│   └── results/             ← output CSVs (gitignored)
└── results/
    ├── branch_a/            ← per-problem CSVs from the catalog sweep (gitignored)
    └── olympus/             ← per-task CSVs from run_olympus.py (gitignored)

Scope. run_idc_21seeds.py + aggregate_21seeds.py ship here and sweep the single-objective photo_pce10 example over 21 seeds; make_figures.py and audit_surrogates.py also ship (figures from the committed result CSVs, and the surrogate-quality R² audit). The pymoo/pycma baselines for the three §7 example problems (photo_pce10, concrete_uci_mo, moeed13) ship in baselines/ and run on the same surrogate + YAML as the C++ IDC binaries. Only the broader ~30-problem benchmark catalog (the §7.1 catalog) and its baseline sweep are run from the authors' workspace.

What maps to which §

Script Paper § Default behavior
bbob/run_bbob_suites.py §7.2 analytical validation runs bbob-biobj-mixint (the only suite shown)
bbob/run_bbob_stress.py §6.3 limitations runs f15–f24 at n=5 and n=20
run_olympus.py §7.4 real-application SO runs photo_pce10 (the only task shown)
run_idc_21seeds.py §7.1 catalog pointer runs every catalog problem, 21 seeds

Switches for problems NOT shown in the paper

The manuscript shows one BBOB suite and one Olympus task. The runners can reproduce the rest of the catalog through switches:

# Other COCO suites (paper shows only bbob-biobj-mixint)
python bbob/run_bbob_suites.py --list-suites
python bbob/run_bbob_suites.py --suite bbob-mixint
python bbob/run_bbob_suites.py --suite bbob --dimensions 5 10 20

# Hard-multimodal stress test at extra dimensions (paper shows 5 and 20)
python bbob/run_bbob_stress.py --dimensions 5 20 40
python bbob/run_bbob_stress.py --functions 15 16 --algorithms IDC

# Other Olympus tasks (paper shows only photo_pce10)
python run_olympus.py --list-tasks
python run_olympus.py --task snar
python run_olympus.py --task alkox --objective so

Each runner prints a [note] line whenever it is asked for a problem/suite/task that is not shown explicitly in the manuscript, so it is always clear when output falls outside the paper's reported scope.

Committed result snapshots for the broader catalog (single-objective top-5% holdout over 15 problems, and the multi-objective HV table) live in extra_results/ — these are the "additional reproducible tests" the manuscript points to.

The §7.4 convergence figure is rendered from a committed trace:

python make_convergence_figure.py     # -> fig_conv_photo_pce10.pdf

It reads extra_results/photo_pce10_convergence.csv (best feasible photo-degradation vs surrogate evaluations for IDC and the SO baselines on the five top-5% holdout surrogates) and needs only matplotlib + numpy.

Reproducing the §7.2 / catalog numbers

cd benchmarks
python -m venv .venv
source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -r requirements.txt

python bbob/run_bbob_suites.py     # §7.2 analytical validation (default suite)
python bbob/run_bbob_stress.py     # §6.3 f15–f24 stress test
python run_olympus.py              # §7.4 photo_pce10 real-application SO
python make_convergence_figure.py  # §7.4 convergence figure from extra_results/

Expected runtime on commodity hardware (i7 8-core): on the order of a couple of hours, dominated by the pymoo baselines (~20 s per seed); IDC itself runs in well under a second per seed.

Optional dependencies

  • run_bbob_suites.py needs cocoex (official COCO binding). When it is missing the script prints install hints and points to run_bbob_stress.py, which needs no COCO.
  • run_bbob_stress.py needs only numpy for --algorithms IDC; cma for CMA-ES and scipy for DE.
  • run_olympus.py needs the olympus package to materialize task data, or a pre-exported CSV under ../data/olympus/<task>/.

Status

The bundled runners (run_olympus.py, bbob/run_bbob_suites.py, bbob/run_bbob_stress.py, make_convergence_figure.py), the photo_pce10 21-seed sweep (run_idc_21seeds.py + aggregate_21seeds.py), the surrogate-quality audit (audit_surrogates.py), the MO figure-regeneration (make_figures.py), and the pymoo/pycma baselines for the three §7 example problems (baselines/run_baselines.py) all run today and reproduce the results shown explicitly in the paper. Only the broader ~30-problem benchmark catalog (the §7.1 catalog) and its baseline sweep are maintained in the authors' working tree; OpenNN is pinned to the immutable tag v1.2-IDC-paper for byte-reproducibility.

Per-problem CSV schema

Every branch_a/<problem>_idc.csv produced by the full-catalog sweep follows the schema:

Column Type Description
seed int 0–20
algorithm string idc_default, idc_tight, cmaes, …
best_f float best objective value found
feasible bool True if the returned point is feasible
walltime_s float wall-clock seconds
n_evals int total surrogate evaluations
n_iters int IDC iterations (1 for non-IDC baselines)
extra per-problem columns float input coordinates of the best point, where relevant

The aggregator consumes these and produces the cross-problem summary in all_problems_21seeds.csv (both run from the authors' workspace; see the "Not bundled" note above).