CHANGELOG.md

Changelog

All notable changes to this package will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

• Correct-class construction in DEForm. generate_clean_variants produces label-preserving variants of a clean base model (varying the seed and hyperparameters within behavior-preserving ranges), and run_one_clean_variant tests each against the base model with the same sign-flip kill test, retaining the ones that stay indistinguishable as detection_label = 0 samples. Exposed on the benchmark CLI through defaultpp-benchmark --clean-variants N.
• Seven mutation operators completing the DEForm catalog to 52, matching the published taxonomy: QHD (QKV head repartition), KRP and KMC (kernel numerical precision and memory), CDU (desynchronized cache update), EZD (zero embedding feature dimensions), NWD (weight-decay LayerNorm parameters), and RDR (residual dropout rate).
• root_cause_label_space(arch), the official Level-3 label space (40 root causes for encoders, 45 for decoders), exported from defaultplusplus.deform.
• Decoder task specs for lambada, ptb, and openwebtext in the kill-test metric registry, scored as log-perplexity.
• Label-space validation in the training loader that warns when the benchmark CSV is missing official root causes or contains root causes outside the taxonomy.

Changed

• Diagnostic-model training now uses nested grouped cross-validation (5 outer x 5 inner) grouped by the model-task pair, with the inner loop doing StratifiedGroupKFold model selection.
• Level-1 fault detection now reports binary F1 (the two classes are balanced by construction) instead of macro F1.
• The group-level FPG adjacency is built from the forward and structural mechanisms (M1, M2, M3, M4, M7) only; backward gradient coupling (M5) enters through gradient features and architecture-wide intervention (M6) through fault labels.
• The gradient feature block reports six global statistics so the optimization group stays at 21 features.
• The mutant kill rule uses strict p < alpha.

[0.4.1] - 2026-05-04

Changed

• README updated to reflect the 0.4.0 reality: pretrained weights are now shipped in the wheel under defaultplusplus/pretrained/weights/; new sections cover RuntimeNormalizer, the [viz] extra, and the defaultpp-bench-download benchmark fetcher (Zenodo DOI 10.5281/zenodo.20481557); layout tree refreshed to list the new data/, processing/, viz/, and diagnosis/model.py modules. No code changes vs. 0.4.0; this is a documentation-only release so the first PyPI surface (TestPyPI was 0.4.0) describes the package accurately.

[0.4.0] - 2026-05-04

Added

• defaultplusplus.diagnosis runtime API: load_pretrained(arch) → Predictor.predict(features) → Diagnosis(is_faulty, category, root_cause, group_importance, ...). The schema the model was trained against is bundled inside the checkpoint and validated against the live FeatureExtractor.feature_names at load time, so a model trained on one schema cannot silently consume features from another. PretrainedWeightsMissingError raised with a clear message when no .pt is on disk.
• scripts/train_diagnoser.py training driver. Reads the paper-aligned benchmark CSV (--csv) or synthesizes labels from random data (--synthetic, for development), trains a HierarchicalDiagnosisModel, and writes a v1 checkpoint via defaultplusplus.diagnosis.save_checkpoint.
• Checkpoint format v1 bundles: feature_names, category_names, category_sizes, rootcause_names, group_names, model_state_dict, scaler mean / scale, per-category prototype tensors, and the model_kwargs needed to reconstruct the model class. Format version is checked at load time; future bumps just add an if.
• DEForm mutation engine: 45 mutation operators across 12 transformer components, static + dynamic injection context managers, structural verifier, and exact one-sided sign-flip permutation test.
• Per-operator implementations for all 45 catalog entries under deform/operator_impls/ so the benchmark runner can resolve any operator by ID without a custom injector_factory callable.
• KV-cache mutation operators (CST, COB, CTR, CLK) now mutate the live DynamicCache (or legacy tuple) — truncate / shift / serve one-step-stale snapshots / cross-request leak — and demonstrably move the new cache_nll_divergence metric.
• extraction.sublayer_capture.SublayerCapture: forward hooks on each layer's attention, FFN, LayerNorm submodules plus the Q/K/V projection Linears. Promotes ffn_delta_*, residual_cos_*, ffn_var_ratio_*, ln_std_*, ln_mean_abs_*, ffn_active_dim_frac_*, ffn_out_skew_* from reconstructed to exact, and emits new qkv_alignment_qk_cos_mean / _qv_cos_mean / _kv_cos_mean direct cosines.
• cache_nll_divergence (decoder only): mean symmetric KL between fresh and cached next-token distributions, sampled at a few positions per probe step. Promoted from not_available to exact.
• defaultpp-benchmark console script and end-to-end CLI driver (benchmark.cli) that produces data/*.csv from scratch via HF Trainer for any combination of supported models / tasks / operators.
• Crash isolation in the runner: RunStatus enum + RunOutcome.status
  • discard_reason. Verifier failures, faulty-run exceptions, and non-finite test metrics each discard the configuration without affecting other runs in the batch. Discards are written to a *.discarded.jsonl log next to the dataset CSV.
• Per-task metric registry (benchmark.task_metrics.TASK_METRICS) defining the scalar that feeds the kill test for each supported task: SST-2 / QNLI / RTE / MNLI / CoLA use single metrics, MRPC / QQP use the GLUE (accuracy + F1) / 2 composite, STS-B uses (Pearson + Spearman) / 2, WikiText uses eval loss.
• Benchmark construction pipeline: configuration-grid enumeration, per-configuration runner, and CSV / Parquet shard writer.
• Feature-construction pipeline: layer / step / epoch / training-phase aggregation that produces the fixed-length feature vector consumed by the diagnostic model. Equation 7.19 dimensions pinned to 1600 (encoder) / 1705 (decoder).
• Compute Canada SLURM scripts under scripts/cc/.
• Local end-to-end dry-run harness (scripts/dry_run_local.sh and tests/test_dry_run.py).

Changed

• StructuralVerifier.verify_static now rejects silent no-op faults: if expected_param_names is non-empty but no parameter actually changed, the verifier fails. Bound-method comparison fixed via the _callable_identity helper so dynamic verification compares (__self__, __func__) rather than fresh bound-method objects.
• QSW operator rewritten to swap query.weight ↔ key.weight (and biases) within each attention block. The previous adjacent-pair positional swap silently no-op'd on standard HF models.
• Cache operators (CST, COB, CTR, CLK) wrap model.forward instead of per-layer attention so they see the whole DynamicCache once per forward and can mutate per-layer slices selectively.
• Feature-group taxonomy renamed to match the diagnostic model's twelve-encoder / thirteen-decoder schema (qkv_alignment, ffn_output, residual_stream, output, cache, representation_drift, validation_perf).
• Hierarchical loss formula reorganized into the form L = L_detect + alpha * L_cat + lambda_rc * L_rc + L_sep with L_sep = beta * L_ctr + gamma * L_pm.
• Graph aggregator implements the message-passing update H = ReLU(A_hat * H * W_msg) with row-normalized adjacency, three rounds, and a learnable matrix per round.
• Root-cause explanation reports per-group importance from the predicted vs. nearest-alternative prototype margin.
• Default training hyperparameters: 150 epochs, three message-passing rounds, early-stopping patience of 20, gamma of 0.3.

Removed

• Legacy L{layer_idx}_attention_score_var / ..._score_skew keys (log-prob proxy on attention probabilities). The exact pre_softmax_score_* family (computed from captured Q/K via the sublayer hooks) is the single score-shape signal. MAJOR bump.

Fixed

• _compute_pre_softmax_stats now reads captured Q/K from the sublayer hooks instead of recomputing the projections on the layer input (the recomputation drifted under operators that wrap attention preprocessing).
• Runner no longer aggregates a partial set of seeds when one seed crashes: any per-seed exception or non-finite metric discards the whole configuration so the n=5 kill-test guarantee is preserved.

Added (visualization, runtime normalization, distribution)

• defaultplusplus.viz package behind the [viz] extra. Nine public entry points: seven figure-returning plot functions (plot_diagnosis, plot_group_importance, plot_per_layer_heatmap, plot_training_trace, plot_attention_pattern, plot_qkv_alignment, plot_feature_anomaly) plus two HTML report writers (save_diagnosis_report, save_run_report). Reports are self-contained: embedded base64 PNGs, no external assets, no JS. VizDependencyError raised with the install hint when matplotlib is not installed.
• defaultplusplus.processing.RuntimeNormalizer with a learned clean reference (RuntimeReference: per-key median + MAD + std + count over baseline rows). encode(features, mode='raw'|'anomaly') produces a feature dict in the diagnoser's exact schema, filling missing keys with the baseline median or zeros respectively. Aliases short-form (..._l3_...) and long-form (..._layer3_...) layer names; strips the live extractor's trace__ prefix. Closes SPEC §1.3 single-run anomaly encoding.
• scripts/fit_runtime_reference.py builds a RuntimeReference from a merged trainer CSV and serializes it to .npz (no pickle).
• defaultplusplus.data.download_bench() and the defaultpp-bench-download console script. Idempotent download + SHA256 verification + safe tar extraction + per-file MANIFEST cross check. Cache lives under $DEFAULTPP_CACHE_DIR / $XDG_CACHE_HOME/defaultplusplus/ / platform default. BENCH_VERSIONS["v1"] points at Zenodo record 10.5281/zenodo.20481557.
• data/stage_release_bundle.py builds the published tarball (dist/defaultpp-bench-v1.tar.gz) with a per-file MANIFEST.sha256, a tarball-level SHA256 sidecar, and a README.
• Pretrained diagnostic-model weights now ship in the wheel: pretrained/weights/encoder.pt (val AUROC 0.9932) and pretrained/weights/decoder.pt (val AUROC 0.8735, cat acc 0.4909 on the composite-metric early-stop run). Companion encoder_reference.npz / decoder_reference.npz for RuntimeNormalizer.load(arch).

Changed (training driver, predictor)

• train_diagnoser.py now wires FeatureProcessor.fit_transform into the training loop so layer aggregation, NaN-rate drop, log1p, and CV filtering all run on the train fold before the model sees it. The fitted processor is persisted in the checkpoint under extra={"feature_processor", "raw_feature_names", "group_indices"}.
• Trainer adds: stratified train / val split with --val-split, inverse-frequency class weights on detection / categorization / per-category root-cause cross-entropy losses, early stopping with --patience / --eval-every / restoring the best-by-val state before checkpoint write, and an opt-in composite metric --early-stop-metric auroc+cat for runs where AUROC and categorization peak at different epochs (decoder benchmark).
• Predictor now applies FeatureProcessor.transform at inference when a checkpoint carries one, so the user-facing schema is the raw FeatureExtractor.finalize() keys rather than the post-aggregation column names. Legacy v1 checkpoints without extras still load via the direct vectorize → scale path.
• feature_groups.py token rules accept both short-form (attn_*) and long-form (attention_*) feature names; the component regex accepts both _l\d+_ and _layer\d+_ layer prefixes. Component map adds attention / embedding aliases.
• feature_processor.LAYER_RE accepts both layer-prefix conventions; layer aggregation runs whenever per-layer families are detected rather than gating on arch == "encoder" (the original assumption that decoder traces arrived pre-aggregated turned out wrong for offline raw CSVs).

Added (data preparation pipeline)

• data/preprocess_and_merge.py: end-to-end preprocessing + merge pipeline that takes per-task paper-aligned CSVs and produces trainer-ready encoder_merged.csv / decoder_merged.csv. Eight steps: concat → schema fixup (is_faulty, layer_idx) → synthetic-zero padding for upstream-missing groups → faulty-row dedup → high-NaN drop → median impute → log1p → constant drop → CV filter. Self-contained and idempotent.

[0.2.0] - 2026-04-29

Added

• Public feature-extraction API: FeatureExtractor (manual training loop) and DEFaultPlusCallback (HuggingFace Trainer callback).
• HF Trainer integration verified end-to-end with real DistilBERT and GPT-2 model checkpoints.
• extraction.feature_construction aggregator that converts collector output into the fixed-length diagnostic-model feature vector.
• Apache-2.0 LICENSE, MANIFEST.in, CHANGELOG.md, py.typed marker, and PyPI-ready pyproject.toml (PEP 621 metadata, dynamic version, trove classifiers, project URLs).
• Build / publish workflow under scripts/build_pypi.sh.

Fixed

• KV-cache metric module handles modern HuggingFace DynamicCache objects in addition to the legacy tuple-of-tuples shape.
• Feature-construction band-index helper no longer indexes past the end of the array when the run has fewer than three epochs / steps.

[0.1.0]

Added

• Initial research artifact: hierarchical fault-diagnosis model, ablation drivers, baseline comparisons, and the data/ mutation-dataset loader.