Give TDAD Layers 2 and 3 a dispatch path — 50 scenarios are written and never executed

[russmiles]

The 2026-08-25 AI literacy assessment places the Testing operational axis at L3. The single thing holding it there is that 50 of the 132 TDAD scenarios have no execution path.

The gap

81  tier: structural   → run in CI on every PR (tdad-tests-fast.yml)
10  tier: trigger      → no CI path
40  tier: behavioural  → no CI path
 1  tier: finding      → not a test tier

tdad-tests-fast.yml is explicit and honest about why, in its own header:

Layers 2 (trigger) and Layer 3 (behavioural) are NOT run here — they require ANTHROPIC_API_KEY and per-run cost ($0.03 per Layer 2 trigger run; $0.05–$0.20 per Layer 3 behavioural run). A separate workflow can be added if the project decides to gate Layer 2/3 nightly or behind a label.

That separate workflow was never added. The corpus has kept growing.

What it costs to close

Per full Layer 2+3 run, at the header's own figures:

	cost
One full run (10 trigger + 40 behavioural)	$2.30 – $8.30
Weekly schedule	$9.89 – $35.69 / month
Nightly schedule	$69.00 – $249.00 / month

Weekly is the obvious starting point: it costs less per month than a single Layer 3 scenario costs to write, and it converts 50 scenarios from documentation into tests.

Why this matters beyond the axis score

This is the third instance of one defect class in the repository, and the assessment names it as the single pattern explaining most of the operational shortfall:

1. 10 agent-enforced constraints — declared in HARNESS.md, and grep -rn 'harness-enforcer' .github/workflows/ returns nothing. They cannot fail a build.
2. 50 TDAD scenarios — written, tiered, and never executed.
3. The GC job's step masking — six consecutive failing runs where a failure at one step recorded every later step skipped.

All three are declared verification with no dispatch path. A scenario that never runs and a constraint that never fires are the same artefact: something that looks like assurance and provides none. That is also, almost exactly, the subject of the one rule this repository currently has in force — HDR-2026-08-25-four-mechanisms-report-the-reassuring-answer-when-they-cannot-determine-the-real-one.

Options

1. Weekly scheduled workflow — cheapest, catches drift within a week, does not gate PRs. Recommended starting point.
2. Behind a label (tdad-full) — runs on demand for PRs that touch agent or skill behaviour. Complements rather than replaces a schedule.
3. Pre-release gate — runs before a version tag. Strongest signal, slowest feedback.
4. Nightly — 7× the weekly cost for feedback most projects of this cadence would not use.

Options 1 and 2 together are probably the right shape: a weekly floor, plus on-demand for PRs that actually change behaviour.

Acceptance

• A workflow exists that executes Layer 2 and Layer 3 scenarios
• Its cadence is declared in HARNESS.md alongside the existing TDAD constraints
• A failing Layer 2/3 scenario produces a visible, attributable failure — not a skipped step behind an earlier failure, which would reproduce defect (3) in the mechanism built to fix (2)
• The assessment's Testing axis can be re-placed against evidence of actual execution

Open question this does not answer

The assessment asked whether Layers 2/3 are currently run manually on any cadence — if they are, the axis may already be L4 and this issue is about automating what exists rather than starting it. Worth answering when scoping.

Raised from the 2026-08-25 AI literacy assessment (in progress).

[russmiles]

Open question answered, and it reframes this issue.

Layers 2 and 3 are run — manually, weekly. So this is not "start executing 50 scenarios that have never run"; it is automating a discipline that already exists, which is a smaller and better-evidenced piece of work.

Two consequences:

The Testing axis is L4, not L3. The assessment's deciding criterion was whether the corpus runs on a real cadence. It does. The 2026-08-25 assessment records it at L4 — flagged asked rather than observed, because nothing in the repository evidences it.

That flag is the sharper reason to do this. A weekly manual run leaves no artifact: no log, no timestamp, no record of which scenarios passed. So:

• The next assessment cannot confirm the cadence held; it can only ask again.
• A regression caught in week 3 and fixed in week 4 leaves no trace that it was ever caught.
• If the cadence lapses, nothing notices — the same failure mode as the GC job, which ran weekly and failed for six consecutive weeks before anyone read it.

The value of the workflow is therefore less about whether the tests run and more about making a discipline that already happens legible to the mechanism — which converts an asked fact into an observed one and removes a single point of failure that happens to be a person's memory.

Revised recommendation. Weekly scheduled workflow, matching the cadence already being kept by hand. Option 2 (label-triggered) still complements it for PRs that change agent or skill behaviour. The cost figures are unchanged: $9.89–$35.69/month for the weekly schedule.

The acceptance criterion about visible, attributable failures matters more under this framing, not less: an automated run that masks its own failures would be strictly worse than the manual run it replaces, because the person doing it by hand currently sees every result.

[russmiles]

Held — decided 2026-08-25. Not proceeding for now: the recurring cost is not justified for a project of this size.

The figures, restated so the decision is legible later:

	cost
One full Layer 2+3 run	$2.30 – $8.30
Weekly schedule	$9.89 – $35.69 / month
Nightly schedule	$69.00 – $249.00 / month

For a single-maintainer open-source project that is real money against a discipline already being kept by hand every week. The tests are being run; what is missing is only the artifact proving it.

What this decision costs, recorded plainly

The 2026-08-25 assessment places the Testing operational axis at L4, flagged asked — the placement rests on the maintainer's testimony that Layers 2 and 3 run weekly, and nothing in the repository evidences it. This issue was the route from asked to observed. Declining it means:

• Testing stays asked indefinitely. The next assessment can only ask again; it cannot confirm.
• A lapse would be silent. Nothing notices if the weekly run stops — the same failure mode as the GC job, which ran weekly and failed six consecutive times before anyone read it.
• A regression caught and fixed between assessments leaves no trace that it was ever caught.

None of that makes the decision wrong. It makes the asked flag load-bearing, and the flag is doing its job: it records that a real discipline exists and that the mechanism cannot see it.

If this is ever revisited

The cost is not uniform across the two layers:

• Layer 2 (trigger) — 10 scenarios at $0.03 = $0.30 per run, about $1.30/month weekly
• Layer 3 (behavioural) — 40 scenarios at $0.05–$0.20 = $2.00–$8.00 per run, and effectively all of the cost

A Layer-2-only schedule would cost roughly a pound a month and would evidence the trigger semantics, leaving Layer 3 manual. Recorded as information for whoever picks this up, not as an argument against the decision just made.

The other lever is workflow_dispatch with no schedule: zero recurring cost, and the run leaves an artifact whenever the maintainer chooses to trigger it instead of running locally. That converts asked to observed for any week it is used, at no standing cost.

Leaving open and held rather than closed, so the Testing axis has something to point at.