Skip to content

Commit 2a1ec34

Browse files
SigmaJahanSigmaJahan
committed
docs: rewrite remaining-tasks.md as a forward-looking punch list
 - Drop the "Recently completed" section: that's what the CHANGELOG is for, keeping a duplicate here just made the file longer and harder to scan. - Add task 1 (Visualization layer) as the new highest priority — it's the only fully-unblocked item and turns the package from "feature library + diagnoser" into something a user can actually inspect. Includes the function table, the headline use case as a code block, and acceptance items. - Renumber the rest down (pretrained weights -> 2, dataset distribution -> 3, Zenodo DOI -> 4, RuntimeNormalizer -> 5, T5/BART -> 6) and bump version refs to v0.3.0 / v0.4 to match the current state.
1 parent 315b079 commit 2a1ec34

1 file changed

Lines changed: 120 additions & 88 deletions

File tree

‎remaining-tasks.md‎

Lines changed: 120 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,108 @@
11
# Remaining tasks
22

3-
This is the running engineering punch list. Items move out of this
4-
file as they ship; completed work is logged in
5-
[`defaultplusplus/CHANGELOG.md`](defaultplusplus/CHANGELOG.md) and the
6-
runtime schema is in [`docs/SPEC.md`](docs/SPEC.md).
3+
Forward-looking engineering punch list. Items move out of this file
4+
as they ship; historical work is logged in
5+
[`defaultplusplus/CHANGELOG.md`](defaultplusplus/CHANGELOG.md), and
6+
the runtime schema is in [`docs/SPEC.md`](docs/SPEC.md).
77

88
---
99

10-
## 1. Pretrained diagnostic-model weights
10+
## 1. Visualization layer (`defaultplusplus.viz`)
1111

12-
**Scope:** S (the API + training driver are shipped in v0.3.0; only
13-
the actual `.pt` files remain).
12+
**Scope:** M (~1 day)
13+
**Status:** unblocked. Highest priority because it's the only task
14+
that lands without external prerequisites and turns the package from
15+
"feature library + diagnoser" into something a user can inspect.
16+
**Why:** Today a user runs `Predictor.predict(features)` and gets a
17+
`Diagnosis` dataclass plus a 2200-key feature dict. That's correct
18+
but not actionable — no one debugs by reading 2200 floats. We need
19+
visual output that fits a `.py`-first workflow: write a file, open it
20+
in a browser.
21+
**Where:** new `defaultplusplus/src/defaultplusplus/viz/` package
22+
(repurpose the reserved `ui/` namespace or add `viz/` as a sibling).
23+
Lives behind the existing `[viz]` extra so the core install stays
24+
matplotlib-free.
25+
26+
**Functions to ship**
27+
28+
| Function | Returns | Purpose |
29+
|---|---|---|
30+
| `save_diagnosis_report(diagnosis, features, path)` | `None` (writes HTML) | Main entry — debug a faulty run. Self-contained HTML with embedded base64 PNGs. No external assets, no JS. |
31+
| `save_run_report(features, path)` | `None` (writes HTML) | Sanity-check extraction without a diagnosis. Useful before pretrained weights exist. |
32+
| `plot_diagnosis(diagnosis)` | `matplotlib.Figure` | Three-stage verdict — detection / category / root-cause probabilities as a single panel. |
33+
| `plot_group_importance(diagnosis)` | `matplotlib.Figure` | Horizontal bar chart of `Diagnosis.group_importance`. Positive bars support the prediction, negative bars oppose it. |
34+
| `plot_per_layer_heatmap(features, metric)` | `matplotlib.Figure` | Heatmap of `L{i}_<metric>` across all sampled layers. |
35+
| `plot_training_trace(features, keys)` | `matplotlib.Figure` | Time series across the early / mid / late windows for a chosen list of metric keys. |
36+
| `plot_attention_pattern(features, layer)` | `matplotlib.Figure` | One layer's attention shape (entropy, sparsity, future-mass, head similarity). |
37+
| `plot_qkv_alignment(features)` | `matplotlib.Figure` | Q-K / Q-V / K-V cosines per sampled layer. |
38+
| `plot_feature_anomaly(features, baseline)` | `matplotlib.Figure` | Top N most-different feature columns between two runs. |
39+
40+
**Headline use case**
41+
42+
```python
43+
from defaultplusplus import FeatureExtractor
44+
from defaultplusplus.diagnosis import load_pretrained
45+
from defaultplusplus.viz import save_diagnosis_report
46+
47+
with FeatureExtractor(model, arch="encoder") as fx:
48+
# ... training loop calls fx.step(...) and fx.epoch_end(...) ...
49+
features = fx.finalize()
50+
51+
predictor = load_pretrained("encoder")
52+
diagnosis = predictor.predict(features)
53+
save_diagnosis_report(diagnosis, features, "run_42.html")
54+
```
55+
56+
Three lines, file on disk, attach to a Slack message or commit next
57+
to a bug report. The same plotting functions are exported
58+
individually so a user can drop a single figure into their own
59+
`fig.savefig(...)` call.
60+
61+
**Acceptance**
62+
- All 9 functions importable from `defaultplusplus.viz`.
63+
- `save_diagnosis_report` writes a valid HTML file that opens in a
64+
browser with no missing assets and no JS errors.
65+
- `save_run_report` works on a feature dict produced by
66+
`FeatureExtractor.finalize()` with no diagnosis input (sanity-check
67+
path before weights exist).
68+
- Tests render each plot to PNG via `Figure.savefig` and assert the
69+
output file is non-empty (no image-diffing — too fragile across
70+
matplotlib versions).
71+
- Plain-language one-paragraph summary at the top of the diagnosis
72+
HTML report (e.g. "This run was flagged as faulty (0.92) with the
73+
most likely cause being **zero_query** in the **qkv** subsystem.
74+
Three feature groups support this conclusion: …").
75+
76+
---
77+
78+
## 2. Pretrained diagnostic-model weights
79+
80+
**Scope:** S (the API + training driver are shipped; only the actual
81+
`.pt` files remain).
1482
**Why:** `pip install defaultplusplus` users still need real weights
1583
to call `predict()` on a real run. The synthetic-mode driver only
1684
exercises wiring, not accuracy.
17-
**Where:** run `scripts/train_diagnoser.py --csv data/<arch>.csv
18-
--output src/defaultplusplus/pretrained/weights/<arch>.pt` on the
19-
full DEFault-bench, then commit the `.pt` files (or release them
20-
separately if size matters).
21-
**Blocks:** task 4 (Single-run anomaly encoding) depends on the
22-
clean-trace corpus this training produces.
23-
**Acceptance:** `from defaultplusplus.diagnosis import load_pretrained; m = load_pretrained("encoder")`
24-
returns a `Predictor` trained on the real benchmark, not synthetic
25-
data. The paper's AUROC numbers (0.966 enc / 0.962 dec) reproduce.
26-
27-
## 2. Public dataset distribution
85+
**Where:** run
86+
87+
```bash
88+
python scripts/train_diagnoser.py \
89+
--arch encoder \
90+
--csv data/encoder_v1_killed_binary.csv \
91+
--output src/defaultplusplus/pretrained/weights/encoder.pt
92+
```
93+
94+
(and the same for decoder) on the full DEFault-bench, then commit
95+
the `.pt` files (or release them separately if size matters).
96+
**Blocks:** task 4 (`RuntimeNormalizer`) depends on the clean-trace
97+
corpus this training produces.
98+
**Acceptance:** `from defaultplusplus.diagnosis import load_pretrained;
99+
m = load_pretrained("encoder")` returns a `Predictor` trained on the
100+
real benchmark, not synthetic data. The paper's AUROC numbers
101+
(0.966 enc / 0.962 dec) reproduce.
102+
103+
---
104+
105+
## 3. Public dataset distribution
28106

29107
**Scope:** S (logistical)
30108
**Why:** `data/*.csv` is multi-gigabyte; cannot be in the wheel or
@@ -33,93 +111,47 @@ repo. The package needs a way to fetch it.
33111
that pulls from a Zenodo or HF dataset URL with checksum
34112
verification, plus a `defaultpp-bench-download` console script.
35113
**Acceptance:** `defaultpp-bench-download` downloads + verifies +
36-
extracts to `data/`.
114+
extracts to `data/`. The script idempotently re-uses an existing
115+
download when checksums match.
37116

38-
## 3. Zenodo DOI for the v0.2.0 release
117+
---
118+
119+
## 4. Zenodo DOI for the v0.3.0 release
39120

40-
**Scope:** S (15 min after task 1 + first PyPI release)
121+
**Scope:** S (15 min after task 2 + first PyPI release)
41122
**Why:** Paper review will ask for an archived artifact. PyPI is not
42123
archival; Zenodo is.
43-
**Acceptance:** DOI badge in [`defaultplusplus/README.md`](defaultplusplus/README.md)
44-
and `pyproject.toml [project.urls]`.
124+
**Acceptance:** DOI badge in
125+
[`defaultplusplus/README.md`](defaultplusplus/README.md) and
126+
`pyproject.toml [project.urls]`.
45127

46-
## 4. Single-run anomaly encoding (`RuntimeNormalizer`)
128+
---
129+
130+
## 5. Single-run anomaly encoding (`RuntimeNormalizer`)
47131

48132
**Scope:** M
49-
**Status:** blocked on task 1 (need a clean-trace corpus to fit the
133+
**Status:** blocked on task 2 (need a clean-trace corpus to fit the
50134
per-metric reference).
51135
**Why:** The offline pipeline uses paired clean / faulty deltas. The
52-
runtime requirement (per `docs/SPEC.md` §1.3) is to turn *one*
53-
user run into anomaly features against a learned clean reference.
54-
**Where:** `defaultplusplus/src/defaultplusplus/processing/` (currently a
55-
reserved namespace). Output dim must match the diagnostic model's
56-
input dim from task 1.
136+
runtime requirement (per `docs/SPEC.md` §1.3) is to turn *one* user
137+
run into anomaly features against a learned clean reference.
138+
**Where:** `defaultplusplus/src/defaultplusplus/processing/`
139+
(currently a reserved namespace). Output dim must match the
140+
diagnostic model's input dim from task 2.
57141
**Acceptance:** `RuntimeNormalizer.encode(extractor.finalize())`
58142
returns a feature dict that the diagnostic model accepts without
59143
rebuilding paired traces.
60144

61-
## 5. T5 / BART encoder-decoder support
145+
---
146+
147+
## 6. T5 / BART encoder-decoder support
62148

63-
**Scope:** L (1–2 weeks; v0.3 roadmap)
149+
**Scope:** L (1–2 weeks; v0.4 roadmap)
64150
**Why:** Paper §7.8 limitation. Adding encoder-decoder means a third
65151
arch in the schema (cross-attention metrics → new `C_INT`, new
66152
feature dim — Eq 7.19 changes), new operator scope, cross-attention
67153
capture in `inspector.py`, and retraining the diagnostic model.
68154
**Acceptance:** `FeatureExtractor(t5_model, arch="encoder-decoder")`
69155
runs end-to-end and emits a vector keyed against an updated SPEC
70-
section §2.10.
71-
72-
---
73-
74-
## Recently completed (kept for reference)
75-
76-
- **T13** — `qkv_alignment_*` direct hook → `exact` via
77-
`SublayerCapture` (2026-04-29).
78-
- **T14** — `cache_nll_divergence` → `exact` via fresh-vs-cached
79-
symmetric KL probe in `CacheMetrics` (2026-04-29).
80-
- **T15** — structural proxies (`ffn_delta_*`, `residual_cos_*`,
81-
`ffn_var_ratio_*`, `ln_std_*`, `ln_mean_abs_*`,
82-
`ffn_active_dim_frac_*`, `ffn_out_skew_*`) → `exact` via
83-
sublayer-boundary hooks (2026-04-29).
84-
- **Per-operator implementations** — all 45 catalog operators under
85-
`deform/operator_impls/` with full coverage tests; QSW fixed to
86-
swap Q↔K within each block; cache operators (CST/COB/CTR/CLK)
87-
actually mutate the live `DynamicCache`.
88-
- **`defaultpp-benchmark` CLI** — end-to-end driver that produces
89-
`data/*.csv` from scratch via HF Trainer for any combination of
90-
supported models / tasks / operators.
91-
- **Crash isolation** — `RunStatus` enum + per-config `discard_reason`
92-
+ `*.discarded.jsonl` audit log. Verifier failures, faulty-run
93-
exceptions, NaN/Inf metrics all isolate cleanly.
94-
- **Per-task metric registry** — `benchmark.task_metrics.TASK_METRICS`
95-
defines the kill-test scalar per task: GLUE composites
96-
(MRPC/QQP `(acc + F1) / 2`, STS-B `(Pearson + Spearman) / 2`),
97-
CoLA `MCC`, single-metric tasks (SST-2/QNLI/RTE/MNLI), and
98-
WikiText eval loss.
99-
- **Fixed `feature_names` schema** — `MetricCollector.feature_names`
100-
now returns a fully-determined list before any step runs; built
101-
from each metric module's `static_feature_names()`, the windowed
102-
suffixes, and every val_* key declared by the task registry.
103-
`finalize()` pads unrecorded columns to 0.0 so downstream
104-
classifiers see a stable column set across every task.
105-
`compute_window_ranges` makes early/mid/late fractional thirds of
106-
whatever epoch count the user runs.
107-
- **CI** — `.github/workflows/ci.yml` runs the test suite on Python
108-
3.10 / 3.11 / 3.12, the dry-run smoke (`scripts/dry_run_local.sh
109-
--quick`), and the PyPI build flow on every push to `main` and
110-
pull request. Build artifacts are uploaded; HF tokenizer fixtures
111-
are cached between runs.
112-
- **Diagnosis API + training driver** (v0.3.0). Public surface:
113-
`from defaultplusplus.diagnosis import load_pretrained, Predictor,
114-
Diagnosis`. The driver at `scripts/train_diagnoser.py` reads a
115-
benchmark CSV (`--csv`) or synthesizes labels (`--synthetic`),
116-
trains a `HierarchicalDiagnosisModel`, and writes the v1
117-
checkpoint format. The `Predictor` validates the live
118-
`feature_names` schema against the one bundled in the checkpoint
119-
and refuses to score across schemas. Pretrained `.pt` files are
120-
not yet shipped — task 1 above is what tracks generating them on
121-
the real benchmark.
122-
- **Schema cleanup** (v0.3.0, MAJOR bump). Removed the legacy
123-
`L{layer_idx}_attention_score_var` / `..._score_skew` log-prob
124-
proxy keys. The exact `pre_softmax_score_*` family is the only
125-
score-shape signal in the schema.
156+
section §2.10. The diagnostic model gains a third checkpoint
157+
`pretrained/weights/encoder_decoder.pt`.

0 commit comments

Comments
 (0)