Ask which audits a change affects, instead of remembering

Yesterday's work removed fifty keyboard tab stops and altered two control sizes,
and the audits re-run afterwards were chosen from memory -- so keyboard-audit,
wcag22-audit, row-actions-audit, tooltip-audit, wcag-audit, icon-audit and
table-audit were all skipped, every one a check measuring exactly what had
changed.

Run now, and the tab-stop removal broke nothing: keyboard-audit 146 screens at
1280 and again at 375 with no findings, axe 160 pages 0 violations, wcag22 no
failures, tooltip 653 icon-only controls 0 defects, table-audit 155 tables with
0 mixed weights.

Two audits exit non-zero and neither is from yesterday: icon-audit reports
`bi-arrow-left-right` and `bi-key` as glyphs absent from the lexicon, and
row-actions-audit reports /admin/barcode_items carrying 3 inline actions with no
menu. One genuine inconsistency surfaced: `bi-envelope` carries three labels --
"Re-send invitation" on /organization against "Re-send invite" in
admin/users/_list and partners/_partner_row -- for the same action. Pre-existing,
but it took a proper `aria-label` for the audit to see it.

bin/design/which-audits.rb is the fix, and it is deliberately not a checklist: a
checklist is memory written down, correct the day it is written and silently
wrong the day an audit is added. Omission is impossible by construction instead:

  * each audit declares its inputs in its own header, `AUDIT-READS: RENDER`, so
    the declaration cannot drift from the code it describes
  * the script enumerates bin/design itself and never carries a list of audits
  * a file that neither declares nor is named a non-audit FAILS the run, so a
    new audit cannot drop silently out of every future selection
  * six non-audits are excused with a reason each, because an unexplained
    absence reads identically to an oversight
  * it prints what it considered and did not select, and a pasteable command

Bundles are coarse on purpose. A browser audit drives real pages, so for a
render-path change the honest answer is most of them -- asked by hand about one
view and one controller I picked six and the truthful number was twenty-eight.

Three controls, in the three directions the failure can hide: a view change
selects the render audits; a docs-only change selects zero browser audits,
without which it is not selecting but saying "everything"; and a planted
undeclared file exits 2 naming the file, proven with it present and again after
removing it. Its own answer for this commit was three audits -- seam-check,
page-audit, audit-selftest -- all run, all clean.

Fixed on the way: the state.rb paragraph in onboarding.md was inside a ```bash
fence and rendering as shell. One pre-existing rubocop warning is left in
page-audit.rb, verified present at HEAD before this change.

Generalised into `audit-suite` as "after a change, do not choose the audits from
memory".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: claude

.claude/skills/audit-suite/SKILL.md

@@ -107,6 +107,52 @@ flagged element's own **ancestor**, which by definition overlaps it, so the exce
 testing could never be met. When a predicate takes a neighbourhood, the neighbourhood is a term in
 the comparison and has to be dumped like any other.
 
+## After a change, do not choose the audits from memory
+
+The suite grows past the point where you can hold it in your head, and then the question "which of
+these does my change affect?" gets answered by recall. Recall omits.
+
+A change removed **fifty keyboard tab stops** and altered two control sizes. The audits re-run
+afterwards were the ones that came to mind — and `keyboard-audit`, `wcag22-audit`,
+`row-actions-audit`, `tooltip-audit`, the axe run, `icon-audit` and `table-audit` were all skipped.
+Every one of them measures exactly what had just changed. Nothing failed; nobody was careless; the
+selection was simply made by the same faculty that had already mislaid a 20×28 button for weeks.
+
+**A checklist does not fix this**, and it is worth being clear why, because a checklist is the
+obvious answer. A checklist is memory written down: correct on the day it is written, silently
+wrong the day an audit is added, and it fails in the direction that produces silence.
+
+What fixes it is making omission impossible *by construction*:
+
+1. **Enumerate the audits from the filesystem**, never from a list inside the tool. A tool holding
+   its own list of audits has the same staleness problem one level up.
+2. **Make every audit declare what it reads**, in the audit file itself so the declaration cannot
+   drift from the code — `# AUDIT-READS: RENDER` or `// AUDIT-READS: VIEWS, DOCS`. Use a few named
+   bundles rather than per-audit path lists, so no audit restates the render path and gets it
+   subtly different.
+3. **Fail the run when a file is neither declared nor explicitly excused**, with the excuse
+   carrying a reason. This is the whole mechanism: a newly added audit cannot be quietly left out
+   of every future selection, because it cannot be silent about itself. Non-audits — the seam, the
+   generators, a one-off inspector — are named with a sentence each, because an unexplained absence
+   reads identically to an oversight.
+4. **Print what was considered and not selected.** "Three audits to run" and "three audits, having
+   considered thirty" are different claims, and only the second one is evidence.
+
+Prove it in three directions, because the failure it prevents is silence:
+
+| Control | Must |
+| --- | --- |
+| A change to a view or a script | select the audits that render pages |
+| A change to documentation only | select the doc readers and **no browser audits** — otherwise it is not selecting, it is saying "everything" |
+| A new file in the audit directory with no declaration | **fail**, naming the file |
+
+**Expect the honest answer to be "most of them" for anything on the render path**, and do not treat
+that as the tool being useless. A browser audit drives real pages, so a changed view or Stimulus
+controller genuinely can move any of them. The value is not precision; it is that the number stops
+coming from intuition. Asked by hand about a change to one view and one controller I picked six; the
+truthful answer was twenty-eight. Print a single pasteable command that runs the selected set, too —
+"it was too much effort to run them all" ends in the same place as "I forgot".
+
 **A permanently noisy check is a check whose true findings get filed as noise.** That audit had
 been reporting a real defect at every width — a control drawn at 20×28 against a documented 28×28
 — for as long as it had been flickering, and it was read as more of the flicker. Fixing the noise
@@ -233,6 +279,8 @@ the rest cost 800ms — which no amount of reasoning would have produced.
 
 ## Running them
 
+- **Ask which ones, do not remember which ones.** A `which-audits` that reads the diff and each
+  audit's own `AUDIT-READS` declaration, as above. Run it before committing, not after.
 - **Put the control harness in CI, not the audits.** The harness is fast and deterministic. The
   audits read real data and answer "is the system good today" rather than "is this change sound".
   Different question, better asked deliberately. See `reference/control-harness.md`.

bin/design/README.md

@@ -1,5 +1,33 @@
 # Design tooling
 
+## Start here: which audits does my change affect?
+
+```bash
+ruby bin/design/which-audits.rb              # the last commit, plus anything uncommitted
+ruby bin/design/which-audits.rb HEAD~3       # a range you name
+ruby bin/design/which-audits.rb --staged     # what you are about to commit
+```
+
+There are **30 audits** here, which is more than anyone holds in their head, and the failure this
+prevents is a real one: a change removed fifty keyboard tab stops and altered two control sizes,
+and the seven audits that measure exactly those things — `keyboard-audit`, `wcag22-audit`,
+`row-actions-audit`, `tooltip-audit`, `wcag-audit`, `icon-audit`, `table-audit` — were all skipped,
+because the re-run list was chosen from memory.
+
+It is **not** a checklist, deliberately. A checklist is memory written down and goes stale the day
+an audit is added. Instead every audit declares its inputs in its own header —
+`# AUDIT-READS: RENDER` — the script enumerates `bin/design/` itself rather than carrying a list,
+and **a file that neither declares nor is named as a non-audit fails the run**. That last part is
+the whole mechanism: a new audit cannot be quietly left out of every future selection.
+
+The bundles (`RENDER`, `VIEWS`, `RUBY`, `CSS`, `ROUTES`, `DOCS`, `AUDITS`) are coarse on purpose. A
+browser audit drives real pages, so a changed view or Stimulus controller really can move any of
+them — **expect "most of them" for anything on the render path**, and a much shorter list for a
+documentation change. It prints what it considered and did not select, and a single pasteable
+command to run the selected set.
+
+---
+
 `audit.js` renders a page in headless Chromium as a signed-in bank admin and reports the
 things a design migration silently gets wrong: sidebar geometry, the computed h1/card/active-nav
 tokens, leftover Bootstrap/AdminLTE class names, Font Awesome icons that render as nothing on a

bin/design/address-audit.js

@@ -1,3 +1,4 @@
+// AUDIT-READS: RENDER
 // Checks that every screen collecting an address asks for it the same way.
 //
 // Seven screens collect one, and before this they did it in five different shapes: `state` was a

bin/design/audit-selftest.js

@@ -1,3 +1,4 @@
+// AUDIT-READS: RENDER, AUDITS
 // Tests the audits, by breaking a page on purpose and by leaving it alone on purpose.
 //
 // Over two days, five checks in the WCAG audits reported failures the app did not have. Every one

bin/design/button-audit.js

@@ -1,3 +1,4 @@
+// AUDIT-READS: RENDER
 // Audits every page header against the rule design.md already settles:
 //
 //   "At most three actions, exactly one of them primary, primary last."

bin/design/citation-audit.py

@@ -1,4 +1,5 @@
 #!/usr/bin/env python3
+# AUDIT-READS: design.md, docs/design-decisions.md, bin/design/citation-baseline.json
 """Audits the industry citations in design.md and docs/design-decisions.md.
 
 Written after a real error: design.md justified the selection bar covering the filter row with

bin/design/confirm-audit.js

@@ -1,3 +1,4 @@
+// AUDIT-READS: RENDER
 // Every confirmation in the app, opened and checked for being the app's own dialog.
 //
 // The app replaces `window.confirm` with a styled `<dialog>`: `confirm_dialog_controller.js`

bin/design/copy-audit.rb

@@ -1,6 +1,8 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
+# AUDIT-READS: VIEWS, app/helpers/, app/models/, app/services/, app/controllers/, app/javascript/, config/locales/
+
 # Copy audit: inclusive language, and the WCAG success criteria that are about *words*.
 #
 # The other audits look at markup and geometry. This one looks at what the words say, which axe

bin/design/dead-code.rb

@@ -1,3 +1,4 @@
+# AUDIT-READS: RENDER
 # Code that exists and cannot run: actions no route reaches, templates nothing renders, helpers
 # nothing calls, files nothing links to.
 #

bin/design/dead-routes.rb

@@ -1,3 +1,4 @@
+# AUDIT-READS: ROUTES, VIEWS
 # Routes that cannot work: the controller or the action behind them does not exist, or an
 # earlier declaration shadows them so they never run.
 #