Skip to content

Commit a503c30

Browse files
MendixMauclaude
andauthored
fix(routing): rewrite nine triggers the retrieval audit showed never fire (#391)
Trigger text (column 3) of nine skill-routing.tsv rows rewritten and re-rendered (README, ROUTING, six agent stubs). Every rendered cell is a superset of its base content words. Retrieval (BM25, 55 labelled phrasings, in-sample): full trigger top-3 46 -> 55, agent-visible stub text top-3 39 -> 47; no phrasing ranks worse. Review: independent review MERGE (superset verified across all 44 rendered cells; numbers reproduced; render-routing --check identical totals; routing fixtures pass; merge-tree clean vs #392/#393/#394). First commit truncated stub rows before " — " (lost "Every module before it is called done" etc.); fixed in 9bcd19d. Behaviour: probe A/B on the head pack 15/15, same as baseline. Known: bug-lookup row widened to every CE error (intended); one stub-mode over-trigger (scss phrasing ranks wiring-sweep #1, learned-css #2; base had learned-css outside top 5); agent-stub routing words are not counted by render-routing --check (mdl stub +69 words). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
1 parent 1512b09 commit a503c30

10 files changed

Lines changed: 54 additions & 53 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m
1919

2020
## Unreleased
2121

22+
- fix(routing): **nine routing triggers rewritten so the words a user or agent actually types reach the row. BM25 over the 55-phrasing audit set: full trigger text 36/55 → 47/55 top-1, 46/55 → 55/55 top-3; rendered stub text (the part before the first " — ", which on-demand stub rows show) 34/55 → 42/55 top-1, 39/55 → 47/55 top-3; no phrasing ranks lower in either.** Seven phrasings missed with every audit method: "page shows 0 to 0 of 0" (empty-widget-triage), "mx check CE0066 on my access rule" (bug-lookup), "should this be a script or a skill" and "about to write a bash checker…" (skills-over-scripts), "the test wrote a row but my DB query shows nothing" (learned-local-db-confusion), "about to go live and move the old data over" (checkpoint-cutover), "half the buttons on the page do nothing" (wiring-sweep). Two CSS phrasings sat at rank 4–5 (learned-css-that-never-applied). Each trigger now front-loads the symptom words (CLAUDE.md skill-authoring rule 1), and the new words sit before the first " — ", so the stub text is a superset of the old stub text: every content word of the old stub and of the old full trigger survives, checked per row. wiring-sweep keeps "Every module before it is called done" first. module-review ("Is the module done? … once-over") and change-intake ("it also needs Y (a field, a column, a button)") were tightened from rank 2 to rank 1. Only column 3 of `bin/lib/skill-routing.tsv` changed, then re-rendered. Rows grow +6 to +15 words, and the baseline file total is unchanged (73,522). The measurement is in-sample: the rewrites were written against the same 55 phrasings. — routing retrieval audit 2026-10-15, MendixMau
2223
- process(tests/advisory): **maintainer-side, advisory authoring audit: `tests/advisory/authoring-audit.py` reports near-duplicate rules and routing misses; it never ships and never gates.** `dupes` chunks `skills/`, `agents/`, `CLAUDE.md` and README "Baseline routing" into rule-sized pieces and ranks cross-file pairs by sparse TF-IDF cosine on 1–2-grams (ROUTING blocks, same-file and agent-to-agent pairs skipped, max 3 per file pair, `dupes-allow.tsv` seeded with the declared architect-agent ↔ workflow-structure-rules mirror and the pointer/tombstone/boilerplate pairs). `routing` scores BM25 over each `skill-routing.tsv` trigger against 55 labelled phrasings (`routing-phrasings.tsv`) and lists every top-3 miss; `--strict` exits 1 only for a `must_hit` row, and none is marked yet. Python 3 stdlib only. New non-blocking CI job `authoring-audit`; fixture `tests/wave2/test-authoring-audit.sh` (5 checks, floor top-3 ≥ 44/55); `run-all.sh` resolves `.py` subjects and `tests/advisory/`; CLAUDE.md rule 8 points at it. Field evidence, master 484e7ff: TF-IDF put 19 true duplicates in its top 40 (82.5% not false positives over 69 hand-judged pairs; 5-shingle Jaccard was worse); BM25 on trigger text hit top-3 46/55 (top-1 36/55), matching or beating an LSA stand-in; `dupes` 16 s over 7,031 chunks, `routing` < 0.1 s. — authoring audit, maintainer request
2324
- new(tests/behaviour): **a standing behaviour suite for agent-read text: arms M (base ref), T (candidate ref, or the base minus one skill for an ablation) and N (no pack), two drafting tasks with planted STOP-row requests and five retrieval probes, scored mechanically.** `tests/behaviour/build-packs.sh` builds a pack from any ref in a temporary worktree; `score.sh` adds `read_all` (the agent quoted the pack's `PACK-END` line back), `tool_calls` and `tokens`, and treats `mxcli check` as optional (NA without a binary). `RUNBOOK.md` has the triggers (every PR that changes baseline-routed text or an agent stub; N and ablations at each release cut), 5 reps x 2 tasks plus 5 probes x 3 reps, the rule (T no worse than M on every scored metric, probes T >= M - 1), and partial reads counted and re-run, never dropped. `docs/RELEASE-PROCESS.md` §4a names it as the way to run the Behaviour A/B. Field runs committed under `tests/behaviour/results/`: the 2026-10-15 stub-diet A/B (#369/#378/#380, 42 reps, 3 partial reads on the 4,822-line pack) and the first no-pack run (page trap refused 0/5 without a pack against 5/5 with either; mf passes check 1/5 against 5/5; every arm, N included, also inherited the host session's project CLAUDE.md (a generic mxcli/MDL guide), so N measures 'no toolkit rules', not 'no guidance'). Golden input: four real agent outputs, `tests/wave2/test-behaviour-score.sh`. — maintainer request (MendixMau)
2425
- process(pr-template, release-process): **every Direct-lane PR and every review fills a five-row Impact block: Process, Dependencies, Context cost, Behaviour, Runtime.** "none" or "NOT RUN: <reason>" is legal, a blank row is not. Context cost is pasted from `bin/render-routing.sh --check` on base and head, never estimated; a missing or estimated row is a review finding. The Behaviour A/B (`process/preflight-skill-baseline-2026-08-31.md`) is required for baseline-routed text and agent stubs, recommended for situational skills, NOT RUN with a reason for process docs review agents read, n/a for scripts only; Context cost may be "none" when no routed or always-loaded file is touched. `docs/RELEASE-PROCESS.md` §4a holds the rule and a bad/good example; `CONTRIBUTING.md`'s reviewer list points there. No CI check: `bin/check-pr-discipline.sh` reads the git diff, not the PR body. Field evidence: three PRs reviewed on 2026-10-15 with no process cost stated (#375 added a human step per re-baseline, #384 adds 1–11 s per exec), both found by chance. — maintainer request, MendixMau

0 commit comments

Comments
 (0)