Skip to content

Validate FitConfig parameters and fail fast on invalid ranges - #4

Merged
adrida merged 1 commit into
adrida:mainfrom
anwesh2410:fix/validate-fitconfig
Jun 8, 2026
Merged

adrida merged 1 commit into
adrida:mainfrom
anwesh2410:fix/validate-fitconfig

Conversation

@anwesh2410

Copy link
Copy Markdown
Contributor

What

Adds __post_init__ validation to FitConfig and EmbeddingConfig so invalid
values are rejected at construction time with a clear message.

Why

Today these accept out-of-range values silently:

  • FitConfig(target_teacher_agreement=1.5) / -0.2 / 0.0
  • empty frontier_targets, or values outside (0, 1]
  • min_deploy_coverage outside [0, 1], max_fit_labels=0, retrain_every=0
  • EmbeddingConfig(batch_size=0)

A bad value either silently skews the fit or surfaces much later as a confusing
error far from its cause. Since teacher-parity is the core guarantee, the knob
that controls it should fail fast and loud.

Changes

  • FitConfig.__post_init__ / EmbeddingConfig.__post_init__ raising ValueError
    with specific messages, e.g. target_teacher_agreement must be in (0, 1], got 1.5.
  • 5 tests in tests/test_fit.py covering rejection + that valid configs still build.

Compatibility

Non-breaking — every valid config (including all existing tests and the demo)
constructs exactly as before. Full suite green locally (26 passed).

FitConfig and EmbeddingConfig previously accepted out-of-range values
silently (e.g. target_teacher_agreement=1.5, empty frontier_targets,
max_fit_labels=0, batch_size=0). The bad value only surfaced — if at
all — as a confusing failure deep inside the fit.

Add __post_init__ validation that raises ValueError with a clear,
specific message at construction time, plus tests. Non-breaking: all
existing valid configs still construct fine.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@adrida adrida left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated and merging. Every bound matches the field's real default and semantics: target_teacher_agreement in (0, 1], frontier_targets non-empty with each value in (0, 1], min_deploy_coverage in [0, 1], max_fit_labels > 0, explore_rate in [0, 1], retrain_every > 0, min_labels_to_start >= 0, batch_size > 0. Neither dataclass had a pre-existing __post_init__, so there's no interaction with FitConfig.embedding's default_factory. Additive, CI green across 3.9 to 3.13, and it merges cleanly on top of the two PRs that just landed.

Reasoning for the fail-fast is right: teacher-parity is the core guarantee, so the knob that controls it should reject bad input at construction with a clear message rather than skewing a fit silently or erroring far from the cause. Thank you.

One note for the record, tied to your #6: explore_rate, retrain_every, and min_labels_to_start are the three fields that issue asks about. I'm keeping them (they parameterize the online loop, more in my reply there), so validating them here is exactly what I want.

@adrida
adrida merged commit 5eaacf2 into adrida:main Jun 8, 2026
5 checks passed
adrida added a commit that referenced this pull request Jun 8, 2026
Ships everything since 0.1.1 (0.1.2 was bumped in-tree but never tagged):
predict_batch() dim guard (#3), FitConfig/EmbeddingConfig fail-fast
validation (#4), update() fail-fast on missing all_traces.jsonl (#5),
plus the skip_candidates knob and fit() progress logging.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants