test(skills): the platform skill's evals ask for push notifications on os serve (#30101)

domain:skills#1 · huangyiirene · session_01J8VVAnMUWA9rufXzFaEh4J

<!-- claude-projects-attribution: {"github_login":"huangyiirene"} -->
_Requested by **Irene** · [project
thread](https://claude.ai/code/project/chan_01TJXtQ1tVD26ecLDZxFS67x?thread=cmsg_01TJXtQ1tVD26ecLDZxFS67x22jqGxEAKY3gdWfxgAUJpp)_

Part of #29969

Clause-②: no
Metadata-protocol: none

The maintainer waived the model-tier mandate for `skills/**` while that
tier had no quota; this PR was developed below it.

## 维护者速读

- 改了什么:`skills/objectstack-platform/evals/config-plugins-ops.json` 新增第
12 个 eval:提示要求给 `os serve` 部署开启推送;期望答案在栈根声明 `requires: ['push']`、写顶层
`push:` 键,且不内联任何密钥(APNs .p8 与 Firebase 服务账号 JSON 都按引用命名)。另加一个
`'@objectstack/skills': patch` changeset。
- 为什么改:PR #29968 让 `requires: ['push']` 与栈的 `push` 键在 `os serve` 下挂上
`PushServicePlugin`,其 changeset 记下了本卡这一评测欠账。
- 风险与回滚:只加评测夹具与 changeset;eval 夹具只计量、不受 token 棘轮约束,无上限变动;回滚即 revert。
- 先看:新 case 12 的 `assertions`。断言只覆盖 `SKILL.md` 现在教的内容(L42 顶层 `push`
键、L310 `push` token),不断言 transport / apns / fcm / relay,因为 `SKILL.md`
还没教 push 块的形状。卡片 Wanted (2)(钉住 `transport: 'direct'` 与被拒的 relay)移到
#30092:先在技能里教 push 块形状,再扩展本 case;故本 PR 是 `Part of #29969`。
- 需要你做的:Tier H 受管面(`skills/**`),需要 `GOVERNED_APPROVERS` 的批准;PR 保持 draft。

## What changes

`skills/objectstack-platform/evals/config-plugins-ops.json` gains case
12, in the same `prompt` / `expected_output` / `files` / `assertions`
layout as cases 1–11:

- **Prompt:** turn on push notifications for an `os serve` deployment
whose iOS and Android apps are signed with the team's own APNs key and
Firebase project.
- **Expected output:** `requires: ['push']` on the stack root, which
mounts `@objectstack/service-push`, plus a top-level `push:` key. There
is no host code and no `PushServicePlugin` in `plugins:`. The APNs
signing key and the service-account JSON stay in the secrets layer and
are named by reference (`credentialRef`). Nothing is written inline.
- **must_contain:** `requires`, `'push'`, `push:`.
- **must_not_contain:** `BEGIN PRIVATE KEY`, `privateKey:`,
`"private_key"`.

**Why the case stays generic.** `skills/objectstack-platform/SKILL.md`
teaches only two things about push today: `push` among the stack's
top-level keys (L42) and the `push` token in the `requires` table
(L310). It does not yet teach the push block's shape: `transport`, the
`apns` / `fcm` vendor blocks, `credentialRef`, or the boot refusing the
hosted relay. So the case asserts the stack-level shape the skill
teaches and the no-inline-secret rule, and does not yet pin the
transport: as written it still passes a `push: {}` or `transport:
'relay'` answer, which the boot refuses. `SKILL.md` is not touched here:
it is at 6000 of its 6011-token ceiling, and teaching the push block's
shape there is its own change.

**What moves to #30092.** The card's Wanted (2), pinning `transport:
'direct'` and the refused relay, moves to #30092, which first teaches
the push block's shape in the skill and then extends this case. Hence
`Part of #29969`, not `Fixes`.

The facts were read on `main` from PR #29968:
`packages/spec/src/system/push-config.zod.ts`
(`PushServiceConfigSchema`, with `credentialRef` a secrets-layer
reference and the inline `privateKey` / `key` / `serviceAccount` keys
refused) and `packages/services/service-push/src/capability-arg.ts`.

`.changeset/29969-platform-push-eval.md`: `'@objectstack/skills':
patch`, with both declaration lines.

## Verification

Head `66bfe707`, three commits on `61b487b2`. Its tree is identical to
`44939f51`, the head the contract review read: `e871e3cd` pinned the
transport in the case and `66bfe707` reverts it, because that pin moves
to #30092.

- The fixture parses as JSON (12 cases). For every case, each
`must_contain` string appears in its `expected_output` and no
`must_not_contain` string does.
- `node scripts/check-skills-token-ratchet.mjs`: exit 0. Eval fixtures
are measured, not ratcheted, so no ceiling row moved.

<details><summary>Local gates on 44939f5 and again on 66bfe70: 39
derived, 39 run with exit 0, 0 NOT MEASURED each time (`dispatch-gates
--ran`)</summary>

- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derived 39 commands for the two changed
files. All of them ran after `pnpm install --frozen-lockfile` and a full
`pnpm build` (exit 0). Each had a 900 s timeout, and its exit code was
captured before any pipe.
- All 39 exited 0. They include `check-adr-0087-registration.mjs --base
origin/main`, `check-changeset-no-major.mjs`,
`check-empty-changeset.mjs`, `check:changeset-gate-self-tests`,
`check-comment-mask-corpus.mjs`, `check:doc-authoring`,
`check:skill-compatibility`, `check:skill-frame-sync`,
`check:skill-identifier-liveness`, `check:dts-closure` and
`check:dual-build-cjs-loads`.
- The artifact rosters `pnpm check:changeset-fixed` and `pnpm
check:skill-top-level-keys` both exited 0.
- `--ran` printed: `39 derived famil(ies) accounted for — 39 run, 0
NOT-MEASURED`.
- `node scripts/check-commit-card-trailers.mjs --range
origin/main..HEAD`: exit 0 on both heads (three commits on 66bfe70).

</details>

## Human Minimum

- [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic.
(Not touched.)
- [x] 2. Every name I added uses the casing for its kind.
- [x] 3. No formatter ran. (`pnpm lint` was not run: the diff is one
JSON fixture and one changeset, no code.)
- [x] 4. One issue, one branch off an up-to-date `main`, one pull
request.
- [x] 5. A change that publishes carries a changeset, and both
declaration lines are filled in here and in the changeset.
- [x] 6. `packages/spec` touched: its generated artifacts are
regenerated and committed. (Not touched.)
- [x] 7. No hand edits to `content/docs/references/`,
`packages/*/CHANGELOG.md` or `content/docs/releases/`.
- [x] 8. Title, body and comments are in English.
- [x] The *Before Submitting* list in CONTRIBUTING.md is done.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01J8VVAjFpbQxLtYHuw3di1T

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: objectstack-fleet[bot]

.changeset/29969-platform-push-eval.md

@@ -0,0 +1,10 @@
+---
+'@objectstack/skills': patch
+---
+
+test(skills): the platform skill's evals ask for push notifications on an `os serve` deployment
+
+Clause-②: no
+Metadata-protocol: none
+
+`skills/objectstack-platform/evals/config-plugins-ops.json` gains case 12: the prompt asks to turn on push notifications for an `os serve` deployment, and the expected answer declares `requires: ['push']` with a top-level `push:` key on the stack definition and writes no secret inline (the APNs signing key and the Firebase service-account JSON are named by reference). The assertions stay with what `objectstack-platform/SKILL.md` teaches today, the `push` token and the top-level key; the push block's transport and vendor shape are not asserted until the skill teaches them.

skills/objectstack-platform/evals/config-plugins-ops.json

@@ -197,6 +197,24 @@
           "PUT /api/v1/meta/permission/crm_sales_rep"
         ]
       }
+    },
+    {
+      "id": 12,
+      "prompt": "Turn on push notifications for our `os serve` deployment: our own iOS and Android apps, signed with our APNs key and our Firebase project.",
+      "expected_output": "`requires: ['push']` on the stack root (the token mounts `@objectstack/service-push`) plus a top-level `push:` key on the stack definition for the transport; no host code, no `PushServicePlugin` in `plugins:`. The APNs signing key and the Firebase service-account JSON stay in the secrets layer (an environment variable in the open tier) and the config names them by reference (`credentialRef`): no `.p8` key, `privateKey` or service-account JSON written inline.",
+      "files": [],
+      "assertions": {
+        "must_contain": [
+          "requires",
+          "'push'",
+          "push:"
+        ],
+        "must_not_contain": [
+          "BEGIN PRIVATE KEY",
+          "privateKey:",
+          "\"private_key\""
+        ]
+      }
     }
   ]
 }