You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 67a27bd
Browse filesBrowse the repository at this point in the historyBrowse files
feat: guard the contract's round trip across sessions
`add-caching-cross-session` settles a contract in one session, answers its question, then sends a
bare "continue" into a fresh session on the same workspace and judges what that session does with
it: the tree it leaves behind, the `Intent check` line it reports, and whether the saved contract
was read back off disk rather than remembered. It is the first case that changes the workspace, so
it declares `writable: true` — the delivery arms' posture, where every other case stays read-only —
and its own `timeout: 1200`.
The runner gains four case keys (a `turns` entry may now be a mapping carrying `new_session`; plus
`writable`, `timeout` and the `expect_final` block), two assertion keys (`spec_optional`,
`no_new_question`), `check_final()` for the whole-run assertions, and a `--rescore` refusal for a
transcript of a case that declares `expect_final`. `read_back` matches only the model's own words:
a tool result echoing the skill body satisfied it in the first measured run, which is the failure
mode the assertion is now tested against. The dsh channel refuses a writable case rather than run it
under the read-only posture it pins.
Measured on `dp/deepseek-flash`, because the harness-default route stalled repeatedly that day:
`evals/reports/2026-10-16-opencode.md` records 1/1 subset cases passing, and the two cases that
share its fixture still pass after the change (`2/2`). The suite is eighteen cases and the
full-suite threshold is 15 of 18.
| **Support & service triage** | ticket history, order and event logs, entitlements, the policy in force | refund vs. replace, when both are allowed and one forecloses the other | ✅ **measured** — 1 case (`support-delegate-irreversible`), [subset run](evals/reports/2026-09-23-opencode-2.md) |
410
410
| **Research & analysis** | prior notes, previous reports, the source allow-list, cached retrievals | depth vs. breadth, when the deliverable changes shape | 📋 **specified** |
411
411
| **Ops & data work** | schema, dashboards, last run's output, deploy and incident history, retention policy | may a backfill rewrite historical rows | 📋 **specified** |
@@ -498,7 +498,7 @@ itself stays English-only):
498
498
499
499
## Evals
500
500
501
-
Seventeen cases against fixture workspaces built as probe targets, run in a real harness with
501
+
Eighteen cases against fixture workspaces built as probe targets, run in a real harness with
502
502
nothing mocked. Cases assert the decision state, the counters, and that every evidence pointer
503
503
names a file that exists — a single invented citation fails the suite regardless of everything
504
504
else.
@@ -524,7 +524,8 @@ cases are the evidence for the description rewrite. Its sixth case, `git-only-fa
524
524
the failure is the sandbox's rather than the skill's: reaching a fact that lives only in version
525
525
history needs a shell that can read the repository, and on Windows the dsh sandbox cannot read the
526
526
runner's own temporary workspaces — the same case passes on dsh with confinement off. Probe ratio
527
-
is a diagnostic metric, never judged. The suite is seventeen cases,
527
+
is a diagnostic metric, never judged. The suite is eighteen cases,
528
+
which puts the full-suite threshold at 15 of 18 with zero hallucinated evidence; no full-suite
528
529
run at that size has been published yet. Every published report is in
| 8/10 full suite, probe ratio 0.56 |[`2026-09-23-opencode.md`](../../evals/reports/2026-09-23-opencode.md)|
278
279
| the `fully-specified-auto-quiet` design decision |[`evals/cases.yaml`](../../evals/cases.yaml)|
279
-
|17 cases, 14/17 threshold, five cost tiers, failure-mode split |[`evals/README.md`](../../evals/README.md)|
280
+
|18 cases, 15/18 threshold, five cost tiers, failure-mode split |[`evals/README.md`](../../evals/README.md)|
280
281
| why the Claude Code report was withdrawn |[issue #1](https://github.com/angel291592/Intent-Router/issues/1)|
281
282
| decision-state instability (~1/3 of runs miss the expected ASK; measured on the withdrawn 2026-09-22 claude-code channel, so it is itself pending clean-channel re-verification) |[issue #2](https://github.com/angel291592/Intent-Router/issues/2)|
0 commit comments