Repository navigation
Commit 5fe0b96
skills(objectstack-data): an eval asks for a hook's
Fixes #26834
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:`objectstack-data` 技能的 eval 文件
`evals/hooks-security-seeds-datasources.json` 新增第 7 题:多语言应用里一个 hook
拒绝保存并给用户看一句话,期望答案在 hook 上声明 `refusals: { CREDIT_LIMIT: { message } }`、沙箱
`js` body 只抛 `{ code: 'CREDIT_LIMIT' }`、中文句子放进翻译包的 `hooks` 组;禁止 body
里抛英文句子(`new Error('`)和用 `handler:`(没有 `js` body 的目录会在保存时被拒)。同文件先删掉 5
句只是复述自身 `must_not_contain` 断言的话(453 字节 = 114 token),再按实测把该文件的 token 上限
1404 → 1573(净 +169,席位预算 ≤ +180);`@objectstack/skills` 补一条 patch
changeset。
- 为什么改:PR #26831 给 hook 加了 `refusals` 目录并在 `references/data-hooks.md`
教了它,但没有任何 eval 检查 AI 作者真会用它;照旧习惯直接 `throw new Error('...')` 的英文句子会原样到达
zh-CN 用户,而且没有东西在测,技能会悄悄漂移。
- 风险与回滚:只改一个已发布 eval 夹具、棘轮脚本的一行上限(及其注释)和一条 changeset,不动 schema、键或运行时;回滚即
revert 本 PR(上限随之回落)。删掉的 5 句话每一句都仍由同一题的 `must_not_contain`
机械断言覆盖,事实与断言一个不少。
- 先看:①
`skills/objectstack-data/evals/hooks-security-seeds-datasources.json` 第
7 题(题面、期望、六条 must_contain、两条 must_not_contain)与第 1–5 题的删句;②
`scripts/check-skills-token-ratchet.mjs` 该行注释里的算术;③
`.changeset/26834-hook-refusals-eval.md`。
- 两分钟验证:`node scripts/check-skills-token-ratchet.mjs`,预期看到
`hooks-security-seeds-datasources.json is 1573 tokens (ceiling 1573;
headroom 0)` 与末行 `82 authored bundle file(s) within their
ceilings`;`node -e
"JSON.parse(require('fs').readFileSync('skills/objectstack-data/evals/hooks-security-seeds-datasources.json','utf8'))"`
无输出即解析通过。
- 需要你做的:批准本 PR(`skills/**` 受管面,Tier H;席位不 ready / 不 arm / 不入队)。
## What changes
**`skills/objectstack-data/evals/hooks-security-seeds-datasources.json`
— case 7 joins the six.** Prompt: an app serving English and Chinese
users; an `order` whose `amount` exceeds its account's `credit_limit`
must not be saved; API clients branch on the code `CREDIT_LIMIT`; a
zh-CN rep reads `超出账户信用额度`. Expected: a `beforeInsert` / `beforeUpdate`
hook on `order` declaring `refusals: { CREDIT_LIMIT: { message: 'Over
the account credit limit.' } }`, a sandboxed `body: { language: 'js',
source }` that reads the account through `ctx.api` and refuses by code
only — `throw Object.assign(new Error(), { code: 'CREDIT_LIMIT' })`, the
body's own text never shown — and the zh-CN sentence under the bundle's
nested `hooks: { HOOK: { refusals: { CREDIT_LIMIT: { message } } } }`,
served whole in the caller's locale; a catalogue on a hook without a
`js` body is refused at save.
- `must_contain`: `refusals:`, `CREDIT_LIMIT: {`, `code:
'CREDIT_LIMIT'`, `language: 'js'`, `hooks: {`, `超出账户信用额度`.
- `must_not_contain`: `new Error('` (a sentence on the throw — the old
habit, `throw new Error('Email is required')` as
`references/data-hooks.md` still spells its pre-catalogue examples — and
also the half-habit `Object.assign(new Error('text'), { code })`, whose
text is dead), `handler:` (an in-process handler cannot carry a
catalogue; `refusals` without a `js` body is refused at save).
Every spelling asserted is the spec's, read on `origin/main` `f708cf22`:
`HookSchema.refusals` is `z.record(SCREAMING_SNAKE code,
HookRefusalSchema { message, status? })`
(`packages/spec/src/data/hook.zod.ts`), answered only on a `language:
'js'` body; the body's literal `code: 'X'` / `e.code = 'X'` throws are
read at save by `hookRefusalCodesThrownIn` (`hook-refusal-codes.ts`);
the translation key is `hookRefusalMessageKey(hook, code)` =
`hooks.HOOK.refusals.CODE.message` (`system/i18n-resolver.ts`), and the
bundle declares it as a nested `hooks` record
(`system/translation.zod.ts`) — which is why the eval pins the nested
`hooks: {` and not a dotted path a correct bundle would never contain
(the stack's own `hooks:` is an array, so `hooks: {` cannot match it).
The teaching in `references/data-hooks.md` agrees with the spec on every
spelling; no disagreement to report.
**Paid down first (the standing ruling, 「A:先削重复,余额按实测提高」).** Five
sentences in cases 1–5 that restated their own `must_not_contain`
assertions are gone — the shape the landed objectstack-formula precedent
cut: case 1's "Never a dotted path (… `400 INVALID_FIELD`) and never a
`formula` mirror (…)", case 2's "(an object-level `permissions` key is
rejected)", case 3's "— there is no `field.columnName` on an external
object", case 4's three "never" clauses (its one non-asserted clause,
"Parents before children in the exported array", stays as its own
sentence), case 5's "No validation rule and no `readonlyWhen`, …". Each
cut spelling remains a mechanical `must_not_contain` on the same case;
no assertion changed. 453 bytes = 114 units cut; case 7 adds 1129 bytes
= 283 units.
**`scripts/check-skills-token-ratchet.mjs`** — the one row for this file
moves 1404 → 1573, pinned AT the measurement (zero headroom), with the
arithmetic and the ruling citation beside it as its siblings carry:
triage 6450886908 (room inside the package first, no raise) read
together with the maintainer's standing ruling 2026-10-27 on the
decision card 「skills 修复卡的教学和 eval 都撞 token 天花板,怎么付?」, tap A, verbatim
and untranslated: 「A:先削重复,余额按实测提高」. Strictest first: the room was made
inside the file; the raise is only the balance trimming could not cover.
No other row moved (the open PR #26739 edits other rows; `origin/main`
was merged before this PR opened and the row region is disjoint).
`evals/README.md` untouched (142 / 143, headroom 1 — a mention of
refusals does not fit, see Acceptance notes).
**`.changeset/26834-hook-refusals-eval.md`** — `'@objectstack/skills':
patch`, both declaration lines. Why a changeset and not
`skip-changeset`: the two latest `skills/**`-only PRs on this eval path
(bb60b86 #26416, 7c70f91 #26393) carried `skip-changeset`, but the
three token-ceiling precedents this card names (b1b153a #26774,
40264be #26810, d7dd47a #26852) each added a `@objectstack/skills`
patch changeset, and the changeset gate's own text in
`.github/workflows/pr-automation.yml` says `skills/**` "ships in
'@objectstack/skills', whose build copies it into the 'dist' that
package's 'files' field publishes … never the label". Measured here:
`packages/skills/scripts/sync-catalog.mjs` copied 93 files into
`dist/skills/**` byte-identical;
`dist/skills/objectstack-data/evals/hooks-security-seeds-datasources.json`
carries the new `CREDIT_LIMIT` (3 hits; positive control
`membershipKeys` 2 hits; absent-symbol control 0). The eval publishes,
so the published-face rule owes the changeset.
### Two readings for a `skills/**` diff
| reading | before (`origin/main` f708cf2) | after (this head) |
|:--|--:|--:|
| `evals/hooks-security-seeds-datasources.json` — bytes | 5616 | 6292 |
| same file — tokens (gate convention, `ceil(bytes / 4)`) | 1404 /
ceiling 1404 | 1573 / ceiling 1573 |
| same file — lines | 65 | 75 |
| `skills/objectstack-data/**` — sum of the 17 priced files the gate
lists | 43289 | 43458 (+169) |
| catalog-wide — sum of every priced `SKILL.md` | 56410 | 56410 (no
`SKILL.md` touched) |
`ceil($(wc -c <
skills/objectstack-data/evals/hooks-security-seeds-datasources.json) /
4) == ceil(6292 / 4) == 1573`.
## Tests
Head measured: `1a0fb94a`.
<details><summary>Token ratchet — before, mid (the unpaid reading that
proves the gate read the file), after</summary>
```
before (origin/main f708cf2):
✓ check-skills-token-ratchet: skills/objectstack-data/evals/hooks-security-seeds-datasources.json is 1404 tokens (ceiling 1404; headroom 0).
✓ check-skills-token-ratchet: 82 authored bundle file(s) within their ceilings; 10 generator-owned file(s) measured, not ratcheted. exit 0
mid (case 7 applied, row not yet moved):
✗ check-skills-token-ratchet: skills/objectstack-data/evals/hooks-security-seeds-datasources.json is 1573 tokens; the ratchet ceiling is 1404 (over by 169). … exit 1
after (row 1404 -> 1573):
✓ check-skills-token-ratchet: skills/objectstack-data/evals/hooks-security-seeds-datasources.json is 1573 tokens (ceiling 1573; headroom 0).
✓ check-skills-token-ratchet: 82 authored bundle file(s) within their ceilings; 10 generator-owned file(s) measured, not ratcheted. exit 0
node scripts/check-skills-token-ratchet.mjs --self-test → ✓ check-skills-token-ratchet self-test: 79 cases pass. exit 0
```
</details>
<details><summary>Published-face probe — `@objectstack/skills`
build</summary>
```
node packages/skills/scripts/sync-catalog.mjs --self-test → ✓ sync-catalog self-test: 9 cases pass. exit 0
node packages/skills/scripts/sync-catalog.mjs → ✓ @objectstack/skills: 93 file(s) across 11 skill(s) copied from skills/** into dist/skills and read back byte-identical. exit 0
grep -c CREDIT_LIMIT dist/skills/objectstack-data/evals/hooks-security-seeds-datasources.json → 3
grep -c membershipKeys dist/…/hooks-security-seeds-datasources.json → 2 (positive control)
grep -c NOT_A_SYMBOL_XYZ dist/…/hooks-security-seeds-datasources.json → 0 (absent control)
cmp dist copy vs source → byte-identical
```
</details>
<details><summary>Derived local gate families — `dispatch-gates
--commands` (60 commands at this head), each exit captured before any
pipe, `--ran` reconciled</summary>
```
node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack → 60 commands (derived from the tree at 1a0fb94; change set: 3 paths, 52 changed lines vs the 3000 human-merge threshold)
runner: eval "$cmd" > log 2>&1; status=$?; printf '%s :: exit %d\n' → ran.list (57 × exit 0, 3 × exit 3)
node scripts/pm/dispatch-gates.mjs --ran ran.list --repo objectstack-ai/objectstack
✓ dispatch-gates --ran: 60 derived famil(ies) accounted for — 57 run, 3 NOT-MEASURED (3 DERIVED from a recorded exit 3). exit 0
NOT-MEASURED (exit 3 = EXIT_PREREQUISITE_NOT_MET, an unbuilt dist/; the diff touches no workspace package, so no build closure is owed — CI measures them on its fresh build):
pnpm --filter @objectstack/lint run check:doc-formula-expressions (PREREQUISITE NOT MET — @objectstack/formula is not built)
pnpm check:dual-build-cjs-loads (self-test 93 cases pass; gate body: some package has no dist/)
pnpm check:lean-entry-closure (packages/objectql/dist absent)
exit 0 (57): check:adr-0087-registration (+ --self-test), check:changeset-no-major, check:closing-keyword-parity (+ --self-test), check:comment-mask-corpus, check:declaration-mirrors (+ --self-test), check:doc-route-spelling --advisory (+ --self-test), check:dts-emitted --self-test, check:empty-changeset (+ --self-test), check:self-test-wired (+ --self-test), check:self-test-workflow-commands (+ --self-test), check:skills-token-ratchet (+ --self-test), check:whole-set-label-write (+ --self-test), ci/assert-turbo-remote-cache --self-test, ci/plan-test-core-shards --self-test, ci/prune-turbo-cache --self-test, pm/bare-root-worklist --self-test, pm/release-rehearsal-clone --self-test, release-pending-publish --self-test, check:agent-test-spelling, check:bash32-floor, check:changeset-gate-self-tests, check:ci-filter-parity, check:cli-command-ids, check:closed-ceremony-text, check:corpus-claim-drift, check:cross-package-test-inputs, check:doc-authoring, check:driver-memory-census, check:dts-closure, check:entry-guard, check:gitlink-declared, check:nul-bytes, check:parse-guard, check:pm-changeset-deadline-census, check:pm-dispatch-gates, check:pm-governed-merges, check:pnpm-filter-targets, check:pnpm-flag-spellings, check:published-files, check:ratchet-remedy-authority, check:refd-timer-probe, check:role-word, check:scripts-symbol-anchors, check:skill-compatibility, check:skill-frame-sync, check:skill-identifier-liveness, check:sourcemap-no-sources-content, check:watch-hint-literal
```
</details>
Edited gate script's own suite (clause ⑤):
`scripts/check-skills-token-ratchet.mjs` has no `*.test.*` naming it
(`git grep -l check-skills-token-ratchet -- '*.test.*'` is empty); its
suite is its `--self-test`, run above (79 cases pass). The local-only
families `check:dev-prereqs` / `check:skill-frame-freshness` are not
owed: no `packages/spec` build and no four-axis frame file touched.
`test:repo` is CI's (`Test Core`); not run locally — the diff touches no
vitest `repo` project test.
Every family CI runs on the PR is cited from the PR's check-runs by the
seat, not re-run here; at report time they are `in_progress`.
No package build closure (①) is owed: the diff touches no workspace
package source (`skills/**` at the repo root, one `scripts/` row, one
changeset). No ablation: no tested runtime behaviour changed (eval data
and a ratchet constant). No dogfood battery: no `!` in the title and no
`(narrowing)` arm.
## Acceptance notes
- **Observation, not filed — `skills/objectstack-i18n` names no `hooks`
translation group.** `content/docs/protocol/kernel/i18n-standard.mdx`
lists `hooks.{hook}.refusals.{code}` among the translation targets (PR
#26831 added it), and `translation.zod.ts` declares the nested `hooks`
record, but `skills/objectstack-i18n/SKILL.md`'s roster of top-level
groups ("apps, messages, globalActions, picklists, dashboards, datasets,
reports, pages, flows, positions, permissions, metadataForms,
settingsCommon") does not name `hooks`, and nothing under that skill
mentions it (`git grep -c hooks origin/main -- skills/objectstack-i18n/`
is empty). An AI author writing the zh-CN bundle from that skill alone
would not know where the refusal sentence goes. A coverage gap in a
published skill, no runtime defect; the skills seat decides whether it
rides a follow-up (a `SKILL.md` row at its own ceiling, same ratchet
shape). Out of this card's file surface.
- `evals/README.md` still describes this file as "search mirrors via
hooks, permission-set scopes, external datasources, seeds (…)"; a
mention of the refusal case does not fit its headroom of 1 token and the
seat set README at +0. Noted, not changed.
- `references/data-hooks.md` still spells its three pre-catalogue
examples as `throw new Error('Email is required')` etc. (lines 163, 780,
785) beside the new `refusals` section — the file is out of this card's
surface (the teaching landed with #26831) and the examples are legal
where no catalogue is declared ("A hook that declares no `refusals` is
untouched"); the eval now scores the habit where a user-visible,
translated message is asked for.
## Verification
- `node scripts/check-skills-token-ratchet.mjs` →
`hooks-security-seeds-datasources.json is 1573 tokens (ceiling 1573;
headroom 0)`, last line `82 authored bundle file(s) within their
ceilings`.
- `node -e "const
f=require('./skills/objectstack-data/evals/hooks-security-seeds-datasources.json');
console.log(f.evals.length, f.evals[6].assertions)"` → `7` and the two
assertion arrays above.
- `ceil($(wc -c <
skills/objectstack-data/evals/hooks-security-seeds-datasources.json) /
4)` → 1573.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01QSeKYmhU5KEj1nukYhp1Tg)_
Co-authored-by: Claude <noreply@anthropic.com>refusals catalogue instead of a thrown English sentence (#27088)1 parent 6e5f909 commit 5fe0b96
3 files changed
Lines changed: 46 additions & 6 deletions
File tree
- .changeset
- scripts
- skills/objectstack-data/evals
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
933 | 933 | | |
934 | 934 | | |
935 | 935 | | |
936 | | - | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
937 | 955 | | |
938 | 956 | | |
939 | 957 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
63 | 73 | | |
64 | 74 | | |
65 | 75 | | |
0 commit comments