doe.md

Design of Experiments

Design of Experiments (DOE) is the CheMLFlow layer for defining a set of scientific comparisons before you run them. A DOE file describes the dataset, the evaluation policy, the model and featurization axes to compare, and the output location. CheMLFlow expands that design into runtime configs, filters known-invalid combinations, and records enough metadata to audit what was run.

Use DOE when you want to compare models, featurizers, split strategies, or CV folds under one consistent policy. The point is not only to make many configs quickly; it is to make the experiment easier to trust. A good DOE keeps train, validation, and test policy consistent, prevents hand-edited config drift, keeps artifacts separated, and preserves the reasons that cases were skipped.

Minimal Workflow

Start from the example DOE file:

config/doe.example.yaml

Set at least:

• dataset.source.path
• dataset.target_column
• dataset.smiles_column for local CSV datasets
• the search_space values you want to compare
• output.dir

For real experiments, use a fresh output.dir for each DOE run. If you are rerunning the tutorial output and do not need the old generated files, clean it first:

rm -rf config/generated/flash_doe

Generate the DOE configs:

conda activate chemlflow_env
python scripts/generate_doe.py --doe config/doe.example.yaml

Inspect the generated design:

cat config/generated/flash_doe/summary.json
head -n 20 config/generated/flash_doe/manifest.jsonl
head -n 20 config/generated/flash_doe/parent_manifest.jsonl
ls config/generated/flash_doe/*.yaml

Run one generated case:

CHEMLFLOW_CONFIG=config/generated/flash_doe/<case_file>.yaml python main.py

Use the config_path values in manifest.jsonl or ls output rather than assuming a specific generated filename.

Run all valid generated cases locally:

python scripts/run_doe_local.py --doe-dir config/generated/flash_doe --max-workers 1 --resume

This writes execution_manifest.jsonl and one log per case under local_logs/. It reads valid execution children from manifest.jsonl; CV designs still run as one child config per fold/repeat.

For parallel local execution, raise --max-workers and keep --resume:

python scripts/run_doe_local.py --doe-dir config/generated/flash_doe --max-workers 4 --resume

Do not combine --max-workers > 1 with --stop-on-failure. --stop-on-failure is reserved for serial fail-fast debugging, for example:

python scripts/run_doe_local.py --doe-dir config/generated/flash_doe --limit 1 --stop-on-failure

Analyze local DOE runs without Slurm:

python analysis.py \
  --backend local \
  --doe-dir config/generated/flash_doe \
  --output-dir config/generated/flash_doe/analysis_local

Local analysis reads generated configs, each config's global.run_dir, and run_status.json/metrics files. It writes the same core audit artifacts as the Slurm path, including report.json, jobs.csv, all_runs_metrics.csv, and all_runs_metrics_by_execution.csv.

DOE YAML Shape

Top-level keys:

• version: must be 1
• dataset: fixed facts about the data
• defaults: fixed runtime settings applied to every case
• search_space: experiment axes to vary
• model_search: optional model-scoped parent-level hyperparameter expansion
• constraints: generation and artifact-safety settings
• selection: metric preference metadata
• output: generated artifact location

Typical local CSV DOE:

version: 1

dataset:
  task_type: regression
  target_column: label
  smiles_column: SMILES
  source:
    type: local_csv
    path: local_data/my_data.csv
  sample:
    fraction: 0.10
    seed: 42
    strategy: random

defaults:
  global.base_dir: data/my_doe
  global.random_state: 42
  global.runs.enabled: true
  split.mode: holdout
  split.test_size: 0.2
  split.val_size: 0.1
  train.tuning.method: fixed

search_space:
  train.model.type: [random_forest, xgboost]
  split.strategy: [random, scaffold]
  pipeline.feature_input: [featurize.rdkit, featurize.morgan]

constraints:
  isolate_case_artifacts: true

output:
  dir: config/generated/my_doe

Set dataset.sample when a DOE should run on a seeded fraction of the raw dataset before curation, featurization, split fanout, and model search. strategy: random takes a seeded row sample. strategy: stratified preserves classification class proportions; for regression it logs a warning and falls back to random sampling. Classification stratification happens before curation, so label-normalized datasets stratify on the raw label.source_column.

search_space and defaults use dotted paths. Scalar values are treated as single-value axes; lists expand into a grid.

Model Search

Use model_search when a broad DOE should also expand model-specific hyperparameter candidates. Each resolved hyperparameter point becomes a parent case; CV fold/repeat executions remain children under that parent. Generated runtime YAMLs contain concrete train.model.params values, not unresolved search definitions.

search_space:
  train.model.type: [random_forest, dl_simple, chemprop]
  split.strategy: [random, scaffold]
  pipeline.feature_input: [featurize.rdkit, smiles_native]

model_search:
  random_forest:
    method: grid
    params:
      n_estimators: [100, 300, 500]
      max_depth: [10, 20, null]

  dl_simple:
    method: optuna
    n_trials: 25
    seed: 42
    params:
      hidden_dim: {type: categorical, choices: [64, 128, 256]}
      dropout: {type: float, low: 0.0, high: 0.5}
      learning_rate: {type: float, low: 0.01, high: 0.2, log: true}
      batch_size: {type: categorical, choices: [32, 64, 128]}

  chemprop:
    method: optuna
    n_trials: 12
    seed: 42
    params:
      max_epochs: {type: categorical, choices: [20, 40]}
      batch_size: {type: categorical, choices: [32, 64]}
      max_lr: {type: float, low: 0.0005, high: 0.003, log: true}
      mp_hidden_dim: {type: categorical, choices: [128, 300]}
      ffn_hidden_dim: {type: categorical, choices: [128, 300]}
      mp_depth: {type: categorical, choices: [2, 3]}

model_search.<model_type> applies only to matching train.model.type cases, including the profile's default model when train.model.type is omitted. Unmatched model_search keys are rejected so typos cannot silently drop a hyperparameter search. model_search currently targets the standard train.model.params path; train_tdc.model.params search is not supported.

method: grid creates the full cartesian product. method: optuna converts the YAML distribution specs into Optuna distributions and asks an Optuna study for n_trials concrete parent candidates at DOE-generation time; each sampled point becomes a fixed scientific parent before CV fold/repeat fanout. In 5-fold CV, n_trials: 12 creates 12 scientific parents and 60 execution children for that matching model branch. The optuna Python package is required for this method.

This is parent-level batch trial generation. A static generate_doe.py call does not adapt later trials from completed CV metrics; implementing metric- adaptive Optuna requires an execution loop that runs each parent, aggregates its CV metric, and reports the result back to Optuna. Runtime child-level tuning routes such as train.tuning.method: train_cv, time-series train.tuning.method: optuna, and train.tuning.use_hpo: true are disabled; generated children should run fixed concrete train.model.params.

Generated Artifacts

DOE generation writes these files into output.dir:

• summary.json: deterministic counts, profile/task, selection metadata, DOE spec hash, snapshot path, and git provenance
• manifest.jsonl: one row per attempted execution child, including skipped cases and issue codes
• parent_manifest.jsonl: one row per scientific parent config
• *.yaml: one runnable runtime config per valid execution child
• doe_spec.input.yaml: the exact DOE spec snapshot used for generation

Generated runtime configs are case-isolated by default. CheMLFlow scopes global.base_dir and global.run_dir under the DOE spec hash and case_id, and sets global.runs.id to the case id. Leave constraints.isolate_case_artifacts: true unless you intentionally want shared artifacts.

Local DOE Execution

Use scripts/run_doe_local.py when a DOE should run on the current machine instead of through Slurm. The runner executes only valid execution-child records from manifest.jsonl and calls main.py with CHEMLFLOW_CONFIG set to each generated config path.

Common options:

• --max-workers N: run up to N cases concurrently.
• --resume: skip configs whose run_status.json already says status: success.
• --limit N: run a short pilot from the valid manifest records.
• --dry-run: write the execution attempt manifest without running main.py.
• --stop-on-failure: stop after the first failed case.
• --allow-shared-artifacts: allow parallel execution even when generated configs share global.run_dir or global.base_dir. Use only when you know shared artifacts cannot collide.

After a local run, use analysis.py --backend local. Do not fabricate Slurm logs or sacct output for local work; local analysis combines execution_manifest.jsonl attempt records with each case's run_status.json and metrics files.

Parent And Child Cases

DOE separates the scientific comparison from execution slices:

• A parent case is the logical scientific config, such as one model, featurizer, and split strategy.
• A child case is one concrete execution, such as a specific CV fold and repeat.
• Holdout designs usually have one child per parent.
• CV and nested designs can have many children per parent.

For split.mode: cv or nested_holdout_cv, keep split.cv.n_splits and split.cv.repeats in defaults. If fold and repeat indices are omitted, DOE expands all folds and repeats automatically. Set fold or repeat indices only when debugging or rerunning a specific execution slice.

Supported Profiles

Supported profiles are:

• reg_local_csv
• reg_local_csv_ic50
• reg_chembl_ic50
• clf_local_csv
• clf_tdc_benchmark

If dataset.profile is omitted, CheMLFlow infers it from dataset.task_type and dataset.source.type. For task_type: auto, set dataset.auto_confirmed: true.

Correctness Checks

DOE skips combinations that are known to be invalid before they reach runtime. Common examples include:

• model/task mismatches
• split strategy and split mode mismatches
• missing validation split settings
• missing target or SMILES columns
• unsupported feature input for a model
• Chemprop or Chemeleon preprocessing combinations that are not meaningful
• duplicate rendered runtime configs
• runtime schema validation errors

Skipped child cases stay in manifest.jsonl with issue codes such as DOE_MODEL_TASK_MISMATCH, DOE_SMILES_COLUMN_MISSING, or DOE_RUNTIME_SCHEMA_INVALID. This is intentional: skipped cases are part of the audit trail.

If constraints.max_cases is omitted and the expanded design exceeds the default safety limit of 10,000 execution cases, DOE generation fails fast.

Scientific Use

DOE is most useful when it protects the scientific comparison:

• Use the same split policy, random seed, target column, and metric policy across cases.
• Compare models on matched splits where possible.
• Prefer scaffold CV for chemistry generalization claims.
• Avoid picking a winner from many models on one fixed test split.
• Use repeated CV or nested holdout CV when the conclusion needs to be robust.
• Keep a final untouched holdout for final reporting, not iterative selection.

Selection metadata defaults to auc for classification and r2 for regression. Override with selection.primary_metric when the scientific question requires a different primary metric.

Practical Defaults

• clf_local_csv with non-Chemprop models defaults to pipeline.feature_input: featurize.morgan.
• chemprop and chemeleon default to pipeline.feature_input: smiles_native when that axis is omitted.
• reg_chembl_ic50 defaults global.target_column to pIC50.
• For comparisons across pipelines, strongly consider split.require_disjoint: true and split.require_full_test_coverage: true.

Best Practices

Keep one DOE file per split mode. Mixing holdout, cv, and nested_holdout_cv in one cartesian grid makes some settings meaningless for some rows.

Keep only true experiment axes in search_space; put fixed choices in defaults. Large mixed grids grow quickly and are harder to interpret.

For chemistry model comparison, prefer split.mode: cv with split.strategy: scaffold. Scaffold CV requires a usable SMILES column that can produce canonical_smiles in curated data.

Keep runtime child configs fixed. split.mode: cv or nested_holdout_cv defines the evaluation fanout; model_search defines hyperparameter candidates as fixed scientific parents that fan out across the same CV folds.

Run a small pilot before the full DOE. Confirm metrics files, split metadata, runtime, and memory behavior before spending the full compute budget.

Preserve generated configs, manifests, summary.json, and the emitted git_worktree.patch if generation happened from a dirty working tree. These files make the experiment reproducible after the run has finished.