Skip to content

Commit 7fa1003

Browse files
committed
feat: measure the skill on DeepSeek Harness and fit the description to its catalog cap
dsh renders `name` plus a normalized description capped at 500 characters, so the 1019-character description lost its second half inside it, negative predicates included. It is now 491 characters with every predicate kept, and `SKILL.md` drops three passages of environment mechanics that restate what any harness already does. `evals/run.py` gains `--harness dsh`: it drives `dsh --profile headless`, parses the `--json` event stream so only assistant prose is a candidate reply, isolates every run in its own `DSH_HOME` and `DSH_AGENTS_HOME`, pins the skill surface and a read-only posture with a generated `--patch` overlay, and measures that isolation in a fourth preflight session. `--delivery` and `--rescore-delivery` refuse dsh, and `.dsh` joins the delivery diff's runner-owned paths. `references/harness-compat.md` corrects its dsh row; `README.md` gains the install path and the explicit invocation. The description is a triggering surface, so the change was measured on dsh itself: a 6-case subset covering four auto-mode cases passed 5/6 with zero hallucinated evidence (`evals/reports/2026-10-15-dsh.md`). `git-only-fact` fails there, and that failure is the Windows sandbox's rather than the skill's. The OpenCode numbers were not re-measured against the new description.
1 parent f2265f2 commit 7fa1003

13 files changed

Lines changed: 869 additions & 105 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,83 @@ This project follows semantic versioning from 1.0 onwards; 0.x releases may chan
77

88
## [Unreleased]
99

10+
The description now fits the harness that renders the smallest slice of it, and the eval runner can
11+
measure DeepSeek Harness alongside the two harnesses it already ran. A description is what a harness
12+
matches a request against, and dsh renders `name` plus a normalized description capped at 500
13+
characters — anything longer is truncated with an ellipsis, never rejected. The previous
14+
1019-character description therefore lost its second half inside dsh, including both negative
15+
predicates ("silent on a fully stated task", "not for explaining existing state") that keep
16+
auto-invocation off work that needs no convergence. It is now 491 characters and keeps every
17+
predicate, including the qualifier that lets a complete request still be premised-checked ("silent
18+
on a fully stated request its sources do not contradict"); the `ask`-budget detail it dropped lives
19+
in §4.3, where it is executed. Three passages of
20+
environment mechanics that restate what any harness already does — how to call tools, how to retry a
21+
shell command, why a reply keeps its fence — are gone from `SKILL.md`, and the evidence exclusion
22+
list gained `.dsh/`, the directory dsh installs from. The file is 493 lines.
23+
`--check-frontmatter` now fails above 500 characters, with the dsh cap as its stated reason.
24+
25+
`evals/run.py` gains `--harness dsh`, which drives `dsh --profile headless` and parses its `--json`
26+
event stream: only assistant prose is a candidate reply, so a run that correctly stays silent is
27+
never scored against the skill text its own tool output carried. Each dsh run gets an isolated
28+
`DSH_HOME` and `DSH_AGENTS_HOME` beside its workspace, a generated `--patch` overlay that pins the
29+
skill surface to the workspace and the posture to a read-only sandbox with approval `never`,
30+
credentials copied in as a file, and a fourth preflight session that measures the isolation instead
31+
of asserting it. The posture is not the shipped default because the shipped `workspace-write`
32+
sandbox cannot read the runner's temporary workspaces on Windows — every shell call is denied there
33+
and a run spends its budget diagnosing the sandbox instead of probing. `--delivery`
34+
and `--rescore-delivery` refuse dsh — their per-arm configuration and report section do not exist
35+
for it yet. Adding `.dsh` to the delivery diff's runner-owned paths keeps the installed copy and
36+
anything the model writes under it out of the delivered-work check, exactly as `.claude` and
37+
`.agents` already did.
38+
39+
`references/harness-compat.md` corrects its dsh row — explicit invocation is `/<skill>`, and the
40+
project directory is both `.dsh/skills/` and `.agents/skills/` — and its notes now record the
41+
500-character description cap, that `whenToUse` and `metadata` are parsed but never rendered, the
42+
root ranking, the `.system` exclusion, and that dsh's project root is the nearest `.git` ancestor.
43+
44+
`README.md` gains the dsh install paths and the `/intent-router` invocation. The description change
45+
is a triggering-surface change, so it was measured on dsh itself: a six-case subset covering four of
46+
the suite's auto-mode cases (including both the report scores for auto-trigger) passed 5/6, with
47+
zero hallucinated evidence (`evals/reports/2026-10-15-dsh.md`). The sixth case, `git-only-fact`,
48+
failed — and that failure is the sandbox's, not the skill's: following a reference that only version
49+
history explains needs a shell that can read the repository, and on Windows the dsh sandbox cannot
50+
read the runner's own temporary workspaces; the same case passes on dsh with confinement off, which
51+
is how the failure was localized. Reports now also record the SHA-256 of the `SKILL.md` that ran,
52+
because the commit alone does not identify an uncommitted working tree. The OpenCode numbers above
53+
were not re-measured against the new description.
54+
/
55+
描述现在适配"渲染得最少"的那个 harness,评测跑分器也能把 DeepSeek Harness 和前两个 harness 一起测
56+
了。description 是 harness 拿来做请求匹配的字段,dsh 只渲染 `name` 加一段归一化后的描述、上限 500
57+
字符——超长会被省略号截断而不是报错。旧描述 1019 字符,因此在 dsh 里**后半段整体消失**,其中包括两条
58+
否定谓词("任务已完整说明时保持沉默"、"不用于解释现状"),而它们正是防止自动触发在无需收敛的工作上开
59+
火的条件。新描述 491 字符,谓词一条不少,包括让"完整请求仍可做前提检查"的那句限定语("silent on a
60+
fully stated request **its sources do not contradict**");被它省下的提问预算细节在 §4.3 里,那里才是执行位置。
61+
`SKILL.md` 删掉三处"环境机制"交代(怎么调工具、怎么重试 shell 命令、回复为什么保留围栏),这些是任何
62+
harness 都已经在做的事;evidence 排除清单补上 `.dsh/`(dsh 的安装目录)。全文 493 行。
63+
`--check-frontmatter` 现在超过 500 字符即失败,并在输出里写明理由是 dsh 的目录渲染上限。
64+
65+
`evals/run.py` 新增 `--harness dsh`:驱动 `dsh --profile headless`,解析它的 `--json` 事件流——只有
66+
assistant 正文进入候选,因此一次"正确地保持沉默"的运行不会被它自己工具输出里的技能正文误判。每次 dsh
67+
运行都在工作区旁拿到隔离的 `DSH_HOME`/`DSH_AGENTS_HOME`、一份自动生成的 `--patch`(把技能面钉在工作区、
68+
把权限姿态钉成只读沙箱 + 审批 `never`)、以文件形式复制进来的凭据,以及第四个 preflight 会话**实测**这套
69+
隔离,而不是声称它成立。之所以不用随包默认姿态:随包的 `workspace-write` 沙箱在 Windows 上读不到 runner
70+
的临时工作区,每次 shell 调用都被拒绝,一次运行会把预算花在诊断沙箱而不是探测上。`--delivery` 与
71+
`--rescore-delivery` 对 dsh 直接拒绝:它们的每臂配置与报告段落尚不存在。把 `.dsh` 加进交付 diff 的
72+
"runner 自有路径",是为了让安装副本与模型写在 `.dsh/` 下的任何东西都不被算作交付——`.claude` 与
73+
`.agents` 早就如此。
74+
75+
`references/harness-compat.md` 修正 dsh 行——显式调用是 `/<skill>`,项目目录是 `.dsh/skills/` 与
76+
`.agents/skills/` 两处——并在注意事项里记下:500 字符描述上限、`whenToUse` 与 `metadata` 会被解析但
77+
从不渲染、技能根优先级、`.system` 例外,以及 dsh 的项目根是最近的 `.git` 祖先。
78+
79+
`README.md` 补上 dsh 的安装路径与 `/intent-router` 调用方式。description 属触发面改动,因此直接在 dsh
80+
上实测:覆盖套件中四个 auto 模式用例(含报告里计入自动触发指标的那两个)的 6 例子集通过 5/6,零编造
81+
evidence(`evals/reports/2026-10-15-dsh.md`)。第六个用例 `git-only-fact` **失败,而这是沙箱的失败、不是
82+
skill 的**:要跟随一条"只有版本历史能解释"的引用,需要一个能读仓库的 shell,而在 Windows 上 dsh 沙箱读
83+
不到 runner 自己的临时工作区;同一用例在不设限的 dsh 下通过,失败因此定位在沙箱。报告现在还会记下实际运行
84+
的那份 `SKILL.md` 的 SHA-256,因为未提交的工作树上"提交号"不足以定位内容。上面那些 OpenCode 数字没有针对
85+
新描述重测。
86+
1087
## [1.3.0] — 2026-10-09
1188

1289
The saved contract becomes a two-way artifact. It gains an optional `verification_status`

‎README.md‎

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -447,6 +447,8 @@ agent reads:
447447

448448
- `.claude/skills/` — Claude Code
449449
- `.agents/skills/` — Cursor, Codex, OpenCode, Gemini CLI, Copilot, Pi, Amp, Zed and others
450+
- `.dsh/skills/` — DeepSeek Harness (dsh); it scans one level deep and takes the nearest `.git`
451+
ancestor as the project root, so the copy belongs at `<repo>/.dsh/skills/intent-router/`
450452

451453
Add `~/` in front for a global install. Per-harness paths, including the ones that use their own
452454
directory name, are in
@@ -462,24 +464,24 @@ when you remember to invoke it. (`grill-me`, by contrast, documents that *"the a
462464
for it on its own."*) Force it when you want to:
463465

464466
```
465-
/intent-router refactor the auth module # Claude Code, Cursor, Copilot, Zed, Kiro, Augment
467+
/intent-router refactor the auth module # Claude Code, Cursor, Copilot, Zed, Kiro, Augment, dsh
466468
$intent-router refactor the auth module # Codex
467469
/skill:intent-router refactor the auth module # Pi, Kimi Code
468470
```
469471

470472
<details>
471473
<summary>Which agents it runs in</summary>
472474

473-
`SKILL.md` names no tools — it asks for *"whatever file-reading, search, or shell capability your
474-
environment provides"* — so it runs anywhere that reads the
475+
`SKILL.md` names no tools — it asks for *"the read, search and shell capability the environment
476+
has"* — so it runs anywhere that reads the
475477
[Agent Skills](https://agentskills.io/) format.
476478

477-
**verified** (run here, report in `evals/reports/`) — OpenCode
479+
**verified** (run here, report in `evals/reports/`) — OpenCode, DeepSeek Harness (dsh)
478480

479481
**spec-compatible** (its docs say it loads standard `SKILL.md`; not exercised here) — Claude Code,
480-
Codex CLI, Cursor, GitHub Copilot (CLI and VS Code), Gemini CLI, Antigravity, Windsurf, DeepSeek
481-
Harness (dsh), Pi, Qwen Code, Kimi Code CLI, Trae, Cline, Roo Code, Kilo Code, Goose, OpenHands,
482-
Amp, Zed, Warp, Kiro CLI, Junie, Augment, Factory Droid
482+
Codex CLI, Cursor, GitHub Copilot (CLI and VS Code), Gemini CLI, Antigravity, Windsurf, Pi, Qwen
483+
Code, Kimi Code CLI, Trae, Cline, Roo Code, Kilo Code, Goose, OpenHands, Amp, Zed, Warp, Kiro CLI,
484+
Junie, Augment, Factory Droid
483485

484486
**needs-adapter** (no documented skill mechanism; paste `SKILL.md` into the system prompt) —
485487
Continue
@@ -507,6 +509,7 @@ else.
507509
2026-09-24 · opencode · dp/deepseek-flash · delivery add-caching-delivery · bare 5.0/6 (N=5) · with skill 6.0/6 (N=3) · ADR trap avoided bare 5/5 vs skill 3/3
508510
2026-09-24 · opencode · (harness default) · 4/4 subset cases · probe ratio 0.25 · 0 over-asks · 0 hallucinated evidence
509511
2026-09-27 · opencode · dp/deepseek-flash · 6/6 subset cases · probe ratio 0.44 · 0 over-asks · 0 hallucinated evidence
512+
2026-10-15 · dsh · opencode-go3/deepseek-v4.1-flash · 5/6 subset cases · probe ratio 0.35 · 0 over-asks · 0 hallucinated evidence
510513
<!-- evals:end -->
511514

512515
Line by line: line 1 is a full-suite run of the suite as it stood on 2026-09-23, which was ten
@@ -516,8 +519,12 @@ project's result metric — the skill arm delivered 6/6 on all three runs and ev
516519
cache failure policy, where none of the 5 bare runs did; line 4 is the v1.1.0 release
517520
verification, a 4-case subset run; line 5 is a subset run verifying the three request-defect
518521
cases (`ambiguous`, `conflict`, `premise`) added for v1.2.0 alongside the three cases that change
519-
could disturb. Probe ratio is a diagnostic metric, never judged. The suite is seventeen cases,
520-
which puts the full-suite threshold at 14 of 17 with zero hallucinated evidence; no full-suite
522+
could disturb; line 6 is the first DeepSeek Harness (dsh) run, a 6-case subset whose four auto-mode
523+
cases are the evidence for the description rewrite. Its sixth case, `git-only-fact`, failed, and
524+
the failure is the sandbox's rather than the skill's: reaching a fact that lives only in version
525+
history needs a shell that can read the repository, and on Windows the dsh sandbox cannot read the
526+
runner's own temporary workspaces — the same case passes on dsh with confinement off. Probe ratio
527+
is a diagnostic metric, never judged. The suite is seventeen cases,
521528
run at that size has been published yet. Every published report is in
522529
[`evals/reports/`](evals/reports/).
523530

‎README.zh-CN.md‎

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,8 @@ skill 没自动触发的 session 里也被读到:*"若 `.intent/` 里有未结
403403

404404
- `.claude/skills/`——Claude Code
405405
- `.agents/skills/`——Cursor、Codex、OpenCode、Gemini CLI、Copilot、Pi、Amp、Zed 等
406+
- `.dsh/skills/`——DeepSeek Harness(dsh);它只扫一层,并以最近的 `.git` 祖先为项目根,
407+
所以副本要放在 `<仓库>/.dsh/skills/intent-router/`
406408

407409
前面加 `~/` 就是全局安装。逐个 harness 的路径(包括那些用自己专属目录名的)在
408410
[`references/harness-compat.md`](skills/intent-router/references/harness-compat.md)。
@@ -415,23 +417,23 @@ system prompt 即可。`references/` 是按需加载的,可以一起粘,也
415417
(相比之下 `grill-me` 自己的文档写着 *"agent 不会主动去用它。"*)想强制触发就:
416418

417419
```
418-
/intent-router refactor the auth module # Claude Code、Cursor、Copilot、Zed、Kiro、Augment
420+
/intent-router refactor the auth module # Claude Code、Cursor、Copilot、Zed、Kiro、Augment、dsh
419421
$intent-router refactor the auth module # Codex
420422
/skill:intent-router refactor the auth module # Pi、Kimi Code
421423
```
422424
423425
<details>
424426
<summary>它能在哪些 agent 里跑</summary>
425427
426-
`SKILL.md` 不点名任何工具——它只要求 *"你的环境提供的任何读文件、搜索或 shell 能力"*——所以凡是能读
428+
`SKILL.md` 不点名任何工具——它只要求 *"环境具备的读取、搜索与 shell 能力"*——所以凡是能读
427429
[Agent Skills](https://agentskills.io/) 格式的地方它都能跑。
428430
429-
**verified**(在这里跑过,报告在 `evals/reports/`)——OpenCode
431+
**verified**(在这里跑过,报告在 `evals/reports/`)——OpenCode、DeepSeek Harness(dsh)
430432
431433
**spec-compatible**(其文档声称支持标准 `SKILL.md`;本项目未实测)——Claude Code、Codex CLI、
432-
Cursor、GitHub Copilot(CLI 与 VS Code)、Gemini CLI、Antigravity、Windsurf、DeepSeek Harness
433-
(dsh)、Pi、Qwen Code、Kimi Code CLI、Trae、Cline、Roo Code、Kilo Code、Goose、OpenHands、Amp、
434-
Zed、Warp、Kiro CLI、Junie、Augment、Factory Droid
434+
Cursor、GitHub Copilot(CLI 与 VS Code)、Gemini CLI、Antigravity、Windsurf、Pi、Qwen Code、
435+
Kimi Code CLI、Trae、Cline、Roo Code、Kilo Code、Goose、OpenHands、Amp、Zed、Warp、Kiro CLI、
436+
Junie、Augment、Factory Droid
435437
436438
**needs-adapter**(没有公开的 skill 机制;把 `SKILL.md` 粘进 system prompt)——Continue
437439
@@ -456,15 +458,20 @@ Zed、Warp、Kiro CLI、Junie、Augment、Factory Droid
456458
2026-09-24 · opencode · dp/deepseek-flash · delivery add-caching-delivery · bare 5.0/6 (N=5) · with skill 6.0/6 (N=3) · ADR trap avoided bare 5/5 vs skill 3/3
457459
2026-09-24 · opencode · (harness default) · 4/4 subset cases · probe ratio 0.25 · 0 over-asks · 0 hallucinated evidence
458460
2026-09-27 · opencode · dp/deepseek-flash · 6/6 subset cases · probe ratio 0.44 · 0 over-asks · 0 hallucinated evidence
461+
2026-10-15 · dsh · opencode-go3/deepseek-v4.1-flash · 5/6 subset cases · probe ratio 0.35 · 0 over-asks · 0 hallucinated evidence
459462
<!-- evals:end -->
460463
461464
逐行说明:第 1 行是 2026-09-23 当日套件规模(10 例)下的全量单次运行;第 2 行是当天提示词强化后
462465
对 8 用例子集的单次重测——子集口径,与全量数字分开陈述;第 3 行是交付对照,本项目的**结果指标**——
463466
skill 臂三次运行全部交付 6/6,且每次都写明了缓存失败策略,不装的 5 次一次都没有;第 4 行是 v1.1.0
464467
发版核验,4 用例子集运行;第 5 行是 v1.2.0 新增的三个请求缺陷用例(`ambiguous`、`conflict`、
465-
`premise`)连同该改动可能影响的三个回归用例一起的子集运行。probe ratio 是**诊断指标**,从不参与
466-
判定。套件现为 17 例,全量阈值随之为 17 例中至少通过 14 例、且零编造 evidence;这一规模下的全量
467-
运行尚无公开报告。所有公开报告都在 [`evals/reports/`](evals/reports/)。
468+
`premise`)连同该改动可能影响的三个回归用例一起的子集运行;第 6 行是 DeepSeek Harness(dsh)的首次
469+
运行,6 用例子集,其中四个 auto 模式用例正是 description 改写所依据的证据。第六个用例
470+
`git-only-fact` 失败,而这是沙箱的失败、不是 skill 的:要拿到"只存在于版本历史里的事实",需要一个能读
471+
仓库的 shell,而在 Windows 上 dsh 沙箱读不到 runner 自己的临时工作区;同一用例在不设限的 dsh 下通过。
472+
probe ratio 是**诊断指标**,从不参与判定。套件现为 17 例,全量阈值随之为 17 例中至少通过
473+
14 例、且零编造 evidence;这一规模下的全量运行尚无公开报告。所有公开报告都在
474+
[`evals/reports/`](evals/reports/)。
468475
469476
怎么自己跑、每个用例查什么:[`evals/README.md`](evals/README.md)。
470477

‎docs/articles/why-agents-guess-intent.en.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -235,16 +235,17 @@ itself as a finding.
235235

236236
The boundaries matter more than the conclusions, so here they are:
237237

238-
- **One model, one harness have a publishable report** (opencode + `dp/deepseek-flash`).
238+
- **Two harnesses have publishable reports** (opencode + `dp/deepseek-flash`; DeepSeek Harness
239+
(dsh) + `opencode-go3/deepseek-v4.1-flash`, a 6-case subset), and one model serves both.
239240
- **The Claude Code report was withdrawn.** The 2026-09-22 run went through a channel that forced
240241
`thinking` off and answered English prompts in Chinese, so its failures cannot be attributed to the
241242
skill. The report was removed from the repo and Claude Code dropped to `spec-compatible`. The
242243
methodology and cases stay public; **only the report is held back** — a number nobody can attribute
243244
should not be published.
244245
- **`codex` has never been evaluated.** It appears only as a `spec-compatible` target in the
245246
compatibility docs (its docs say it loads standard `SKILL.md`; not exercised here). The runner
246-
supports `claude-code` and `opencode` only.
247-
- **The full-suite threshold is 14/17**, but **no full-suite run at that size has been published**.
247+
supports `claude-code`, `opencode` and `dsh`; the dsh channel cannot score a case that needs to
248+
read the workspace's git history, because the Windows sandbox denies the shell that access.
248249
The most recent committed report is a subset run.
249250
- **The delivery comparison is small** (bare N=5, skill N=3). It can show that the skill arm states a
250251
failure policy; it cannot give you a precise effect size.

‎docs/articles/why-agents-guess-intent.zh-CN.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# agent 为什么会猜错意图:17 个可判定的用例
1+
# agent 为什么会猜错意图:18 个可判定的用例
22

33
> 写于 2026-10-03。本文所有数字都来自 `evals/reports/` 里已提交的报告,没有一处估计值。
44
> 方法论见 [`../evals/README.md`](../../evals/README.md),用例定义见 [`../../evals/cases.yaml`](../../evals/cases.yaml)。
@@ -197,7 +197,7 @@
197197

198198
我把边界写清楚,因为这些边界比结论更影响你怎么用它:
199199

200-
- **只有一个模型、一个 harness 出过可发布的报告**(opencode + `dp/deepseek-flash`)。
200+
- **有两个 harness 出过可发布的报告**(opencode + `dp/deepseek-flash`;DeepSeek Harness(dsh)+ `opencode-go3/deepseek-v4.1-flash`,6 例子集),两者都由同一个模型家族应答。
201201
- **Claude Code 的报告被撤回了。** 2026-09-22 那次跑在一个强制关闭 `thinking`、且对英文提问用中文回答的通道上,失败无法归因于 skill 本身。报告已从仓库移除,Claude Code 降级为 `spec-compatible`。方法学和用例保持公开,**只有报告被扣下**——一个无法归因的数字,不该被发布。
202202
- **`codex` 从未被评测。** 它只在兼容性文档里被列为 `spec-compatible`(文档声称支持标准 `SKILL.md`,本项目未实测)。runner 支持的目标只有 `claude-code` 和 `opencode`。
203203
- **完整套件的门槛是 14/17**,但**这个规模的完整套件运行还没有发布过**。已提交的最新报告是子集运行。

0 commit comments

Comments
 (0)