Skip to content

Commit 67a27bd

Browse files
committed
feat: guard the contract's round trip across sessions
`add-caching-cross-session` settles a contract in one session, answers its question, then sends a bare "continue" into a fresh session on the same workspace and judges what that session does with it: the tree it leaves behind, the `Intent check` line it reports, and whether the saved contract was read back off disk rather than remembered. It is the first case that changes the workspace, so it declares `writable: true` — the delivery arms' posture, where every other case stays read-only — and its own `timeout: 1200`. The runner gains four case keys (a `turns` entry may now be a mapping carrying `new_session`; plus `writable`, `timeout` and the `expect_final` block), two assertion keys (`spec_optional`, `no_new_question`), `check_final()` for the whole-run assertions, and a `--rescore` refusal for a transcript of a case that declares `expect_final`. `read_back` matches only the model's own words: a tool result echoing the skill body satisfied it in the first measured run, which is the failure mode the assertion is now tested against. The dsh channel refuses a writable case rather than run it under the read-only posture it pins. Measured on `dp/deepseek-flash`, because the harness-default route stalled repeatedly that day: `evals/reports/2026-10-16-opencode.md` records 1/1 subset cases passing, and the two cases that share its fixture still pass after the change (`2/2`). The suite is eighteen cases and the full-suite threshold is 15 of 18.
1 parent 7fa1003 commit 67a27bd

12 files changed

Lines changed: 916 additions & 60 deletions

‎CHANGELOG.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,39 @@ skill 的**:要跟随一条"只有版本历史能解释"的引用,需要一
8484
的那份 `SKILL.md` 的 SHA-256,因为未提交的工作树上"提交号"不足以定位内容。上面那些 OpenCode 数字没有针对
8585
新描述重测。
8686

87+
The suite gains a case for the contract's round trip across sessions. `add-caching-cross-session`
88+
settles a contract in one session, answers its question, then sends a bare "continue" into a
89+
**fresh** session on the same workspace, and judges what that session does with it: the tree it
90+
leaves behind, the `Intent check` line it reports, and whether the saved contract was read back off
91+
disk rather than remembered. It is the first case that changes the workspace, so it declares
92+
`writable: true` — the delivery arms' posture, where every other case stays read-only — and a
93+
`timeout: 1200` of its own, because carrying work out does not fit the 480-second default. Four case
94+
keys are new (a `turns` entry may now be a mapping carrying `new_session`; plus `writable`, `timeout`
95+
and the `expect_final` block) and two assertion keys are (`spec_optional`, because a verification
96+
turn owes a file and an `Intent check` line rather than a second fence; and `no_new_question`,
97+
because a settled question must not come back as an ASK). `--rescore` refuses a transcript of a case
98+
that declares `expect_final` rather than reporting a verdict with half its assertions missing. The
99+
suite is eighteen cases and the full-suite threshold is 15 of 18. Measured on `dp/deepseek-flash`
100+
(`evals/reports/2026-10-16-opencode.md`); the two cases that share its fixture still pass after the
101+
change. The first attempt at this case is recorded in that report too, and it is the reason the case
102+
looks the way it does: under the suite's read-only posture a run cannot save
103+
`.intent/<intent>.intent.yaml` at all, so the fresh session had nothing to read back and correctly
104+
asked again instead of guessing.
105+
/
106+
套件新增了一个守护契约跨会话往返的用例。`add-caching-cross-session` 先在会话 1 里谈定契约、回答它
107+
的问题,然后往**全新会话**发一句裸的「继续」(同一工作区),再判定那个会话拿它做了什么:留下的工作
108+
树、报出的那行 `Intent check`,以及保存的契约到底是从盘上读回来的、还是靠记忆续写的。它是套件里
109+
第一个会改动工作区的用例,因此声明 `writable: true`(交付臂那套姿态,其余用例一律保持只读)并自带
110+
`timeout: 1200`——因为"真干活"装不进 480 秒的默认上限。新增四个用例键(`turns` 的条目现在可以是带
111+
`new_session` 的映射,外加 `writable`、`timeout` 与 `expect_final` 断言块)与两个断言键
112+
(`spec_optional`:核验轮交的是一个文件加一行 `Intent check`,不是第二个围栏;`no_new_question`:
113+
已定的问题不得以 ASK 回来)。`--rescore` 遇到声明 `expect_final` 的用例的 transcript 会直接拒绝,
114+
而不是交出一份"少了一半断言"的判定。套件现为 18 例,全量阈值为 18 例中至少通过 15 例。已在
115+
`dp/deepseek-flash` 上实测(`evals/reports/2026-10-16-opencode.md`);与它共用 fixture 的两个用例
116+
在改动后仍全部通过。这份报告里也记着这个用例的第一次尝试,而它正是本用例长成现在这样的原因:在套件的
117+
只读姿态下,运行根本存不下 `.intent/<intent>.intent.yaml`,于是全新会话没有任何东西可读回,只能正确
118+
地再问一次,而不是猜。
119+
87120
## [1.3.0] — 2026-10-09
88121

89122
The saved contract becomes a two-way artifact. It gains an optional `verification_status`

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -405,7 +405,7 @@ way this project states harness compatibility — measured, specified, or neithe
405405

406406
| Domain | PROBE reaches | The question worth asking | Status |
407407
|---|---|---|---|
408-
| **Coding agents** | repo, deps, version history, tests, CI, ADRs | irreversible technical trade-offs | ✅ **measured** — [eval suite](#evals), 17 cases |
408+
| **Coding agents** | repo, deps, version history, tests, CI, ADRs | irreversible technical trade-offs | ✅ **measured** — [eval suite](#evals), 18 cases |
409409
| **Support & service triage** | ticket history, order and event logs, entitlements, the policy in force | refund vs. replace, when both are allowed and one forecloses the other | ✅ **measured** — 1 case (`support-delegate-irreversible`), [subset run](evals/reports/2026-09-23-opencode-2.md) |
410410
| **Research & analysis** | prior notes, previous reports, the source allow-list, cached retrievals | depth vs. breadth, when the deliverable changes shape | 📋 **specified** |
411411
| **Ops & data work** | schema, dashboards, last run's output, deploy and incident history, retention policy | may a backfill rewrite historical rows | 📋 **specified** |
@@ -498,7 +498,7 @@ itself stays English-only):
498498

499499
## Evals
500500

501-
Seventeen cases against fixture workspaces built as probe targets, run in a real harness with
501+
Eighteen cases against fixture workspaces built as probe targets, run in a real harness with
502502
nothing mocked. Cases assert the decision state, the counters, and that every evidence pointer
503503
names a file that exists — a single invented citation fails the suite regardless of everything
504504
else.
@@ -524,7 +524,8 @@ cases are the evidence for the description rewrite. Its sixth case, `git-only-fa
524524
the failure is the sandbox's rather than the skill's: reaching a fact that lives only in version
525525
history needs a shell that can read the repository, and on Windows the dsh sandbox cannot read the
526526
runner's own temporary workspaces — the same case passes on dsh with confinement off. Probe ratio
527-
is a diagnostic metric, never judged. The suite is seventeen cases,
527+
is a diagnostic metric, never judged. The suite is eighteen cases,
528+
which puts the full-suite threshold at 15 of 18 with zero hallucinated evidence; no full-suite
528529
run at that size has been published yet. Every published report is in
529530
[`evals/reports/`](evals/reports/).
530531

‎README.zh-CN.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -364,7 +364,7 @@ trace: [...] # 可重放
364364

365365
| 领域 | PROBE 能够到 | 值得问的那个问题 | 状态 |
366366
|---|---|---|---|
367-
| **编码 agent** | 仓库、依赖、版本历史、测试、CI、ADR | 不可逆的技术权衡 | ✅ **已实测**——[评测套件](#评测evals),17 个用例 |
367+
| **编码 agent** | 仓库、依赖、版本历史、测试、CI、ADR | 不可逆的技术权衡 | ✅ **已实测**——[评测套件](#评测evals),18 个用例 |
368368
| **客服与服务工单** | 工单历史、订单与事件日志、账户权益、当前生效的政策 | 退款还是换货——当两者都被允许、且发一个就否掉另一个 | ✅ **已实测**——1 个用例(`support-delegate-irreversible`),[子集运行](evals/reports/2026-09-23-opencode-2.md) |
369369
| **研究与分析** | 先前笔记、历史报告、在用的来源白名单、已缓存的检索结果 | 深度还是广度——当交付物的形态会因此改变 | 📋 **已写规格** |
370370
| **运维与数据作业** | schema、看板、上一次运行的输出、部署与事故历史、留存策略 | 回填能不能改写历史行 | 📋 **已写规格** |
@@ -448,7 +448,7 @@ Junie、Augment、Factory Droid
448448
449449
## 评测(Evals)
450450
451-
17 个用例,跑在专门作为探测靶子构建的 fixture 工作区上,在真实 harness 里跑,零 mock。用例断言
451+
18 个用例,跑在专门作为探测靶子构建的 fixture 工作区上,在真实 harness 里跑,零 mock。用例断言
452452
决策态、各个计数器,以及**每个 evidence 指针指向的文件必须真实存在**——只要有一处编造的引用,
453453
整个套件就算失败,其他全对也不算。
454454
@@ -469,8 +469,8 @@ skill 臂三次运行全部交付 6/6,且每次都写明了缓存失败策略
469469
运行,6 用例子集,其中四个 auto 模式用例正是 description 改写所依据的证据。第六个用例
470470
`git-only-fact` 失败,而这是沙箱的失败、不是 skill 的:要拿到"只存在于版本历史里的事实",需要一个能读
471471
仓库的 shell,而在 Windows 上 dsh 沙箱读不到 runner 自己的临时工作区;同一用例在不设限的 dsh 下通过。
472-
probe ratio 是**诊断指标**,从不参与判定。套件现为 17 例,全量阈值随之为 17 例中至少通过
473-
14 例、且零编造 evidence;这一规模下的全量运行尚无公开报告。所有公开报告都在
472+
probe ratio 是**诊断指标**,从不参与判定。套件现为 18 例,全量阈值随之为 18 例中至少通过
473+
15 例、且零编造 evidence;这一规模下的全量运行尚无公开报告。所有公开报告都在
474474
[`evals/reports/`](evals/reports/)。
475475
476476
怎么自己跑、每个用例查什么:[`evals/README.md`](evals/README.md)。

‎docs/articles/why-agents-guess-intent.en.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Why agents guess your intent: seventeen decidable cases
1+
# Why agents guess your intent: eighteen decidable cases
22

33
> Written 2026-10-03. Every number here comes from a committed report in `evals/reports/`; nothing is an
44
> estimate. Methodology: [`evals/README.md`](../../evals/README.md). Case definitions:
@@ -84,9 +84,9 @@ your judgement".
8484

8585
---
8686

87-
## 3. Seventeen cases cover four states, not four difficulty levels
87+
## 3. Eighteen cases cover four states, not four difficulty levels
8888

89-
Case count is not the metric; coverage is. The seventeen cases fall into four groups.
89+
Case count is not the metric; coverage is. The eighteen cases fall into four groups.
9090

9191
**Should ask** (`add-caching-auto`, `partially-specified-auto`, `chinese-ambiguous`, …)
9292
The request carries a decision-bearing unknown → it must ask, and ask only that.
@@ -246,6 +246,7 @@ The boundaries matter more than the conclusions, so here they are:
246246
compatibility docs (its docs say it loads standard `SKILL.md`; not exercised here). The runner
247247
supports `claude-code`, `opencode` and `dsh`; the dsh channel cannot score a case that needs to
248248
read the workspace's git history, because the Windows sandbox denies the shell that access.
249+
- **The full-suite threshold is 15/18**, but **no full-suite run at that size has been published**.
249250
The most recent committed report is a subset run.
250251
- **The delivery comparison is small** (bare N=5, skill N=3). It can show that the skill arm states a
251252
failure policy; it cannot give you a precise effect size.
@@ -276,6 +277,6 @@ measures how much was asked, and asking more is not asking better.
276277
| 105 contamination hits; bash key-order issue (36 hits); one-hop scorer fix | same report, §4 Observations |
277278
| 8/10 full suite, probe ratio 0.56 | [`2026-09-23-opencode.md`](../../evals/reports/2026-09-23-opencode.md) |
278279
| the `fully-specified-auto-quiet` design decision | [`evals/cases.yaml`](../../evals/cases.yaml) |
279-
| 17 cases, 14/17 threshold, five cost tiers, failure-mode split | [`evals/README.md`](../../evals/README.md) |
280+
| 18 cases, 15/18 threshold, five cost tiers, failure-mode split | [`evals/README.md`](../../evals/README.md) |
280281
| why the Claude Code report was withdrawn | [issue #1](https://github.com/angel291592/Intent-Router/issues/1) |
281282
| decision-state instability (~1/3 of runs miss the expected ASK; measured on the withdrawn 2026-09-22 claude-code channel, so it is itself pending clean-channel re-verification) | [issue #2](https://github.com/angel291592/Intent-Router/issues/2) |

‎docs/articles/why-agents-guess-intent.zh-CN.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -71,9 +71,9 @@
7171

7272
---
7373

74-
## 三、17 个用例,覆盖的是四种状态而不是四种难度
74+
## 三、18 个用例,覆盖的是四种状态而不是四种难度
7575

76-
用例数量不是指标,覆盖面才是。这 17 个用例分布在四类上:
76+
用例数量不是指标,覆盖面才是。这 18 个用例分布在四类上:
7777

7878
**该问的**(`add-caching-auto`、`partially-specified-auto`、`chinese-ambiguous`…)
7979
请求有决策性未知项 → 必须问,且只问该问的。
@@ -199,8 +199,8 @@
199199

200200
- **有两个 harness 出过可发布的报告**(opencode + `dp/deepseek-flash`;DeepSeek Harness(dsh)+ `opencode-go3/deepseek-v4.1-flash`,6 例子集),两者都由同一个模型家族应答。
201201
- **Claude Code 的报告被撤回了。** 2026-09-22 那次跑在一个强制关闭 `thinking`、且对英文提问用中文回答的通道上,失败无法归因于 skill 本身。报告已从仓库移除,Claude Code 降级为 `spec-compatible`。方法学和用例保持公开,**只有报告被扣下**——一个无法归因的数字,不该被发布。
202-
- **`codex` 从未被评测。** 它只在兼容性文档里被列为 `spec-compatible`(文档声称支持标准 `SKILL.md`,本项目未实测)。runner 支持的目标只有 `claude-code` 和 `opencode`。
203-
- **完整套件的门槛是 14/17**,但**这个规模的完整套件运行还没有发布过**。已提交的最新报告是子集运行。
202+
- **`codex` 从未被评测。** 它只在兼容性文档里被列为 `spec-compatible`(文档声称支持标准 `SKILL.md`,本项目未实测)。runner 支持 `claude-code`、`opencode` 与 `dsh`;dsh 通道无法给"需要读工作区 git 历史"的用例评分,因为 Windows 沙箱不允许 shell 读那里。
203+
- **完整套件的门槛是 15/18**,但**这个规模的完整套件运行还没有发布过**。已提交的最新报告是子集运行。
204204
- **交付对比的样本很小**(裸跑 N=5,技能组 N=3)。它能证明"技能组会写出失败策略",不能给出精确的效应量。
205205

206206
`probe ratio` 是诊断指标,**从不参与判定**。把它当成绩看会误导——它衡量的是"问的比例",而问得多不等于问得对。
@@ -224,6 +224,6 @@
224224
| 105 处污染命中;bash key-order 问题(36 处);评分器一跳修复 | 同上,§4 Observations |
225225
| 8/10 完整套件、probe ratio 0.56 | [`2026-09-23-opencode.md`](../../evals/reports/2026-09-23-opencode.md) |
226226
| `fully-specified-auto-quiet` 的设计决策注释 | [`evals/cases.yaml`](../../evals/cases.yaml) |
227-
| 17 用例、门槛 14/17、五档成本、失败模式分类 | [`evals/README.md`](../../evals/README.md) |
227+
| 18 用例、门槛 15/18、五档成本、失败模式分类 | [`evals/README.md`](../../evals/README.md) |
228228
| Claude Code 报告撤回的原因 | [issue #1](https://github.com/angel291592/Intent-Router/issues/1) |
229229
| 决策态不稳定(~1/3 运行漏触发 ASK,数据来自 2026-09-22 那个已被撤回的 claude-code 通道,因此本身也是 issue #2 待干净通道复核的对象) | [issue #2](https://github.com/angel291592/Intent-Router/issues/2) |

‎docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ This page is the site entry point. The authoritative sources live in the reposit
1717

1818
## Writing
1919

20-
- **[Why agents guess your intent: seventeen decidable cases](articles/why-agents-guess-intent.en.md)** —
20+
- **[Why agents guess your intent: eighteen decidable cases](articles/why-agents-guess-intent.en.md)** —
2121
how "it guessed wrong" becomes decidable, why one hallucinated citation fails the whole suite, and the
2222
contamination check that invalidated an entire run. ([中文](articles/why-agents-guess-intent.zh-CN.md))
2323

0 commit comments

Comments
 (0)