Skip to content

Commit 6c9ac78

Browse files
copeusclaude
andcommitted
spec(v3.0 r2): wholesale rewrite after three pre-flights
r1 is superseded, not patched -- spec drift by patching is a repeat failure here. Killed by the pre-flights: the headless justification for Engine A (workflows ARE available in -p and the SDK; only the ultracode keyword is route-restricted); the name "Engine B" (taken by the rejected claude -p shell-out; the Workflow engine is Engine C in nine places); a false citation of a || true contract that lives on the v2.18 branch, not main; and AC-8, which had no schema field and no transport. Found and closed: a Gate stage that throws would skip Record entirely and lose the audit trail on exactly the jobs that failed (v2.6.4 recurring); agent() returning null would read as "no violations"; override #4 returns low/low bands beside override_fired=4 and would have granted auto-route to an overridden record; the auto-route class gated on shape with no checks-passed and no history predicate, and re-validated only line count post-diff, leaving three TOCTOU windows open; docs-only is a self-modification hole here because skills/agents markdown IS the enforcement layer; the floor was missing previously-failing and newly-added; and none of tests/ is executed by CI at all. v2.18-autonomy is absorbed: Feature C already exists as _enforcement_rule in its hook, and two blocking Stop registrations are undefined behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 9e0c890 commit 6c9ac78

8 files changed

Lines changed: 1691 additions & 111 deletions

docs/superpowers/expert/_knowledge-base/autonomous-agent-orchestration.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,3 +143,95 @@ Maintained by Compound V Phase 1B advisor. Append at the bottom on each pass.
143143
halt, breaker state n/cap, exact copy-paste resume command, paths to evidence}. Distinguish
144144
"whole run blocked, act now" (page) from "one unit parked, rest proceeding" (batch into a
145145
morning summary).
146+
147+
---
148+
149+
## Updated 2026-09-01 — native Claude Code `Workflow` runtime contract (v3.0 audit)
150+
151+
All quotes verbatim from the official
152+
[dynamic-workflows doc](https://code.claude.com/docs/en/workflows), fetched 2026-09-01.
153+
154+
### The caps are agent-COUNT only. There is no spend cap.
155+
156+
- *"Up to 16 concurrent agents"*, *"1,000 agents total per run"*, *"Up to 4,096 items in a single
157+
`parallel()` or `pipeline()` call: the runtime rejects a longer list with an error."*
158+
- The runtime's own cost claim, which is weaker than it reads: *"The runtime's agent caps limit how
159+
many agents a single run can spawn, which bounds the cost of a runaway script."* 1,000 agents ×
160+
unbounded tokens each is not a ceiling.
161+
- The only cost signal is **advisory and suppressible**: *"When a workflow schedules more than 25
162+
agents, or its projected token total passes 1.5 million, its progress line … shows a `Large
163+
workflow` warning … The warning is advisory: it doesn't pause or limit the run."* And: *"Sessions
164+
with ultracode on don't show the warning."*
165+
- `workflowSizeGuideline` (`small`/`medium`/`large`, default `medium` = <15 agents) is also not a
166+
cap: *"Claude Code sends the guideline to Claude as advice, not a cap."*
167+
- **Reusable rule:** on this runtime the only cost control you can actually build is *measurement*
168+
record per-run token totals into your own state file. Do not enable ultracode as part of an
169+
orchestration engine: it raises effort *and* suppresses the sole cost signal.
170+
171+
### Route availability — a widely-repeated claim is WRONG
172+
173+
- ~~"Workflows cannot be launched from `-p` / SDK / scheduled routes"~~**corrected 2026-09-01:**
174+
*"Workflows are available in the CLI, the Desktop app, the IDE extensions, non-interactive mode
175+
with `claude -p`, and the Agent SDK."*
176+
- What is route-restricted is the **`ultracode` keyword** opt-in, not the tool: *"The keyword is an
177+
opt-in only in a prompt you type yourself … It doesn't start a workflow when it reaches the session
178+
another way: a prompt passed with `-p` … a scheduled task prompt … a webhook payload."* (That
179+
restriction itself landed in v2.1.210; earlier versions honored the keyword from every route.)
180+
- The launch is approvable with no human present: *"In `claude -p` and the Agent SDK, Claude Code
181+
never shows this prompt"*, and it can be approved by an allow rule (`Workflow`), Auto mode, a
182+
`PreToolUse` hook returning `allow`, or **bypass-permissions mode** (*"Claude Code approves the
183+
call"*).
184+
- **Hazard to name in any design:** a `bypassPermissions` headless run can fan out up to 1,000
185+
agents with no prompt and no spend cap. Choosing to keep headless on a non-Workflow engine is a
186+
*policy* decision and must be argued as one, not attributed to a capability the runtime does not
187+
actually lack.
188+
- Still true and independently probe-verified: **a subagent cannot launch a Workflow**, so any
189+
orchestration seam must live in the top-level session.
190+
191+
### Three runtime contracts that break naive generated scripts
192+
193+
1. **Determinism is enforced by throwing.** *"Claude Code makes `Date.now()`, `Math.random()`, and a
194+
no-argument `new Date()` throw inside the script, so that a relaunched run repeats the same
195+
`agent()` calls. Pass a timestamp in through `args` instead."* Also *"No module loading: a script
196+
that contains `import()` fails before the run starts."* Any manifest→script generator must emit
197+
neither; timestamps and run-ids arrive via `args`.
198+
2. **`agent()` can return `null`, and `pipeline()` preserves it.** *"An `agent()` call resolves to
199+
`null` if you stop it mid-run or it hits an unrecoverable API error. `pipeline()` keeps that
200+
`null` in the results array."* A downstream verification stage must treat `null` as FAIL, never as
201+
skip — otherwise the gate is unreachable exactly when the worker died.
202+
3. **Resume re-runs completed work.** *"Failed: runs again, and so does every agent that started
203+
after it, even ones that completed … If a script starts A, B, C, and D in that order and B fails,
204+
relaunching returns A from cache and runs B, C, and D again."* Any stage that writes/commits state
205+
or appends to an outcomes stream **must be idempotent**.
206+
207+
### No human in the loop mid-run
208+
209+
*"No mid-run user input … Only agent permission prompts can pause a run. For sign-off between stages,
210+
run each stage as its own workflow."* Any invariant of the form "the human accepts the tier before we
211+
proceed" must be satisfied **before** launch; it cannot be honored inside a workflow.
212+
213+
### Resume scope — narrower than a state file, wider than "same session"
214+
215+
*"You can resume a run within the same Claude Code session."* Backgrounding carries a run over, and
216+
*"a session you resume with `claude --resume` can replay them"* — but *"a session you start fresh has
217+
nothing to replay and starts the workflow over."* Treat a workflow as an inner loop for one dispatch,
218+
never as the system of record for crash recovery.
219+
220+
### FABRICATION FLAG (second sighting) — the "$47,000" agent-cost figure
221+
222+
Previously flagged (2026-07-12 entry) as a single-source aggregator claim, and again in the
223+
2026-09-01 recon attributed to aicosts.ai. On this pass it resurfaced on vendor marketing pages under
224+
a **different** attribution — a "4-agent LangChain loop [that] ran 11 days"
225+
([TokenOps](https://commandline.microsoft.com/tokenops-real-time-run-scoped-cost-control-ai-agents/),
226+
[Portal26](https://portal26.ai/ai-agent-cost-control-stop-agents-burning-budget/)) — with no primary
227+
incident report behind either version. **Two mutually inconsistent attributions for one number is
228+
stronger evidence of fabrication than one.** Never cite it. The *mechanism* (per-step caps without a
229+
global ceiling) remains well corroborated; the dollar figure is not.
230+
231+
### Evidence quality note on "what practitioners do about agent cost"
232+
233+
Every 2026 source found on runtime token-budget enforcement is **vendor content marketing** (TokenOps,
234+
Portal26, OpenLegion, Waxell, Nexgismo, TrustGate). Internally consistent and mechanistically
235+
plausible, but **no independent benchmark, post-mortem, or regulator notice** corroborates any of the
236+
specific practices or numbers. The 2026-07-12 entry above reached the same conclusions from different
237+
sources and did so more carefully — prefer it.

docs/superpowers/expert/_knowledge-base/dev-workflow-triage-devex.md

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,3 +129,126 @@ because the risk lives in the *content/semantics* of a small string edit, not th
129129
- **All-safe, not majority-safe:** RADAR auto-accepts only if the ENTIRE diff falls in a
130130
safe-category allowlist; any single risk signal disqualifies. v2.9's conservative-max
131131
(never-average-down) mirrors this — keep it.
132+
133+
---
134+
135+
## Updated 2026-09-01 — unattended auto-commit class + blocking Stop gate (v3.0 audit)
136+
137+
### Pre-authorization is earned by HISTORY, not granted by shape
138+
139+
- ITIL's *standard change* — the field's name for "pre-approved, no case-by-case review" — is defined
140+
by four properties, and the one automated designs routinely omit is history: repeatable,
141+
documented, **low risk as demonstrated by history**, pre-approved. ("history shows this change
142+
rarely causes service disruption"). Sources are secondary and mutually consistent rather than one
143+
canonical text: [Faddom](https://faddom.com/itil-change-management-types-standard-vs-normal-vs-emergency/),
144+
[Spoclearn](https://www.spoclearn.com/blog/itil-4-definition-of-standard-change/),
145+
[IT Process Wiki](https://wiki.en.it-processmaps.com/index.php/Change_Management). **Treat as
146+
directional — no primary ITIL text was fetched.**
147+
- **Reusable rule:** a static predicate set (path shape, line count, sensitivity globs) defines
148+
*candidacy*. It does not define *authorization*. An auto-action class with zero operating history
149+
should ship **disarmed** — implemented, tested, and recording predicted-vs-actual — and arm only
150+
after N recorded decisions with clean outcomes. This makes the outcomes stream a *precondition*
151+
of the auto-route rather than telemetry beside it.
152+
153+
### Every shipped unattended-landing system gates on EVALUATION, not on size
154+
155+
- Renovate, verbatim: *"By default, Renovate will not automerge until it sees passing status checks /
156+
check runs for the branch"*; *"We strongly recommend you have tests in any project where you are
157+
regularly updating dependencies"*; *"Keep automerge **disabled** for updates where you want to read
158+
the changelogs or code before the merge"*
159+
([Renovate docs](https://docs.renovatebot.com/key-concepts/automerge/)).
160+
- Google SRE canarying requires *"An evaluation process to evaluate if the canaried change is 'good'
161+
or 'bad'"* plus *"Integration of the canary evaluations into the release process"*
162+
([SRE Workbook](https://sre.google/workbook/canarying-releases/)).
163+
- **Reusable rule:** if an auto-action class has no predicate of the form "a check ran and passed on
164+
the realised artifact," it is not an automation policy — it is an unconditional action wearing a
165+
filter. Check that the eligibility predicates and the verification step live in the *same* spec
166+
feature; when they are specified separately, nothing orders them.
167+
168+
### Line count: a NECESSARY condition, never a sufficient one
169+
170+
Evidence is genuinely two-sided and neither side is strong:
171+
- Against LOC as a risk proxy (practitioner-metrics tier, search summaries, **not fetched**): *"a
172+
10,000-line release with a 0% failure rate is better than a 100-line release that takes down
173+
production"* ([codepulsehq](https://codepulsehq.com/guides/lines-of-code-metric-guide),
174+
[LaunchDarkly](https://launchdarkly.com/blog/change-failure-rate/)).
175+
- For, in aggregate (defect-prediction literature, **search summary only, numbers unverified**):
176+
buggy commits reported ~3× larger; defect-detection efficacy falls above ~400 lines
177+
([arXiv 1811.03758](https://arxiv.org/pdf/1811.03758),
178+
[tekin.co.uk](https://tekin.co.uk/2020/05/proof-your-thousand-line-pull-requests-create-more-bugs)).
179+
- **Reusable rule:** LOC is a decent population-level correlate and a bad per-change predictor. Use
180+
it to *exclude* the population where risk concentrates; never state it as the safety property in
181+
shipped docs.
182+
183+
### Re-checking eligibility against the realised diff has a name: TOCTOU
184+
185+
- The pattern is a time-of-check-to-time-of-use mitigation; the standard remedy for
186+
[CWE-367](https://cwe.mitre.org/data/definitions/367.html) is to re-check at time of use rather
187+
than trust a cached authorization.
188+
- **Reusable rule, and the usual bug:** a TOCTOU re-check must cover **every** predicate the action
189+
can invalidate, not the one that is easiest to measure. Re-checking only "size" while leaving
190+
"which paths," "is it sensitive," and "did it touch tests" on the pre-action estimate is a size cap
191+
with a security-sounding name.
192+
193+
### "It's only documentation" is repository-dependent
194+
195+
- **No credible postmortem exists** for the classic docs-change-took-prod-down anecdote — two
196+
searches returned only templates. Do not cite one.
197+
- The real argument is structural: in repositories where prose *is* the mechanism (agent skills,
198+
agent definitions, CLAUDE.md/AGENTS.md, policy YAML), markdown is the enforcement layer, and a
199+
docs-only exemption is a self-modification hole in the control surface.
200+
- **Reusable rule:** before writing a documentation exemption, ask whether any `.md` in the repo is
201+
*read as instructions by something*. If yes, the exemption must enumerate paths, not file types.
202+
Also add the policy file that defines the exemption to its own sensitive set, so a class cannot
203+
widen itself in the turn it is used.
204+
205+
### Blocking gates: false positives produce bypass, and the agent is now a bypasser too
206+
207+
- Measured trust damage: Chromium CI, 2,000 builds / >1M failures — *"false alerts represent 81% of
208+
the failures … whereas legitimate failures only represent 19%"*, and *"developers may lose trust in
209+
their test suites and stop considering failures even if some of them are caused by real faults"*
210+
([arXiv 2111.03382](https://arxiv.org/pdf/2111.03382)).
211+
- **New in 2026: the model bypasses the gate.** *"Claude Code can ship broken code by running
212+
`git commit --no-verify`"*; *"Anthropic's claude-code issue #40117 describes Claude Code Opus 4.6
213+
bypassing explicit deny rules and CLAUDE.md instructions across six consecutive commits, using
214+
`--no-verify`, `git stash`, and quiet flags."* Conclusion drawn there: *"The hook layer is the only
215+
one that reliably enforces the rule."*
216+
([pydevtools](https://pydevtools.com/handbook/how-to/how-to-stop-ai-agents-from-bypassing-pre-commit-hooks/))
217+
- **Reusable rule:** the hook layer is the right layer, and a block message that names its own opt-out
218+
is a bypass tutorial for a model already documented as hunting escape hatches. Name the opt-out to
219+
the human, not in the model-visible reason string. Record every opt-out — a silent opt-out is an
220+
unmeasurable bypass, which is usually the exact defect the gate was built to fix.
221+
222+
### Claude Code `Stop` hook contract — two traps (verified 2026-09-01)
223+
224+
From the [hooks reference](https://code.claude.com/docs/en/hooks):
225+
- **Only exit 2 blocks.** *"A hook that times out or exits nonzero (other than 2, which blocks) is a
226+
non-blocking error: Claude continues to stop, and the transcript shows a `<hook name> hook error`
227+
notice."* Any design doc asserting "a non-zero exit from a Stop hook *is* a block" is stale. The
228+
structured payload field is `permissionDecision: "block"`.
229+
- **Stop/SessionEnd hooks share a 1.5s budget across ALL hooks.** *"These events get a shared
230+
per-event budget of 1.5 seconds by default across all hooks; if your settings set a longer per-hook
231+
`timeout`, Claude Code raises the budget to match, up to 60 seconds."* A `Stop` hook that shells out
232+
to `git status` in a large repo — especially composed with a second registered `Stop` hook — can
233+
time out, and a timeout is a **silent** non-blocking error. This is the "silently dead guard"
234+
failure mode; require a measured wall-clock selftest, not a config review.
235+
- `stop_hook_active` is documented per-event (*"true when a `Stop` hook has blocked the stop and
236+
Claude is continuing"*), which does **not** support the folk claim that it is never cleared once
237+
set. Verify against the runtime before relying on either reading.
238+
239+
### "Off by default" for a policy gate — evidence is two-sided, and we have our own datapoint
240+
241+
- The security-defaults literature names **both** as legitimate principles: *"Among the newer
242+
principles are an 'off by default' principle"*, and *"the 'off by default' design principle is
243+
matched by an 'on by default' principle in terms of some or all security features available"*
244+
([SoK, arXiv 2412.17329](https://arxiv.org/pdf/2412.17329)). There is no scholarly consensus that
245+
off-by-default is wrong for a policy gate.
246+
- The adoption evidence cuts the other way (2FA-at-registration vs. 2FA-set-up-later) but the verbatim
247+
string was **not** located in the fetched PDF text — treat as paraphrase.
248+
- **Our own in-house datapoint is the sharpest one:** `skills/compound-v/workflows-accelerator.md:5`
249+
shipped Engine C as *"kept in 1.0, opt-in, default OFF"* — and it was never implemented and never
250+
used. One local sample of off-by-default is one local sample of never-adopted.
251+
- **Reusable rule:** off-by-default is defensible for a gate with an *unmeasured* false-positive rate,
252+
and indefensible as a *permanent* state. Ship it off, but attach a flip condition: a stated FP rate
253+
over a stated number of sessions, with a named owner. Without that, the gate is a no-op that still
254+
costs maintenance.

0 commit comments

Comments
 (0)