Skip to content

Add Codex as an advisory red-team audit engine - #1263

Merged
nedtwigg merged 2 commits into
mainfrom
codex-advisory-engine
Oct 21, 2026
Merged

nedtwigg merged 2 commits into
mainfrom
codex-advisory-engine

Conversation

@nedtwigg

Copy link
Copy Markdown
Member

Summary

  • scripts/security-audit-local.sh --engine codex runs the same .github/audit/ prompts on gpt-6-sol with --cyber-access-program daybreak_blue, sandboxed (workspace-write + network). canary --key <run ID> seeds what that CI canary run seeded; --claude-scorecard (the run's decrypted scorecard.json) adds the "caught by either" count. It prints the gh workflow run codex-audit-report.yaml … command and files nothing.
  • .github/workflows/codex-audit-report.yaml: workflow_dispatch with only non-secret inputs (commit, model, verdicts, canary counts). security-audit-public-body.mjs --engine codex validates every input, requires the commit on main and an admin dispatcher, and checks the counts against the CI canary run's own canary-recall record and commit. It then comments once on the status issue and uploads a codex-recall artifact. Only issues: write on its one job (S45 exception added to security-ci.md and workflow-lint.mjs).
  • Canary graph: three lines (Claude, Codex, caught by either) with a legend. Identity is carried by dash and marker shape as well as color, since the colors come from the theme. The generator pairs each admin-dispatched codex-recall record with its canary run and fails the snapshot on an impossible pairing.
  • security-audit.md: S8's Claude split stays the release gate. A new Red team section sets the cadence (quarterly, plus before releases that change remote control or Hosted) and adds S275. The embargo table lists the new public sinks, and rationale entries are added.

Test plan

  • pnpm test:checks (spec, md, public-docs, workflow lints and their tests)
  • pnpm test:audit: 180 pass, including new S275 workflow/builder tests and the local Codex engine and canary-union tests
  • website pnpm test: typecheck + 456 vitest
  • actionlint on the new workflow
  • Rendered /security with a patched snapshot to check the three-line graph
  • zizmor (runs in CI)
  • A live --engine codex run: not exercised here

🤖 Generated with Claude Code

`scripts/security-audit-local.sh --engine codex` runs the same audit
prompts on gpt-6-sol under the daybreak_blue Cyber access program,
sandboxed to the working tree. Its canary takes a CI canary run's seed
key (`--key <run ID>`) and that run's decrypted scorecard to count the
seeds either engine caught, then prints the publishing command.

`.github/workflows/codex-audit-report.yaml` publishes from non-secret
dispatch inputs only (commit, model, verdicts, canary counts). The
public-body builder checks each input, the commit against main, the
dispatcher's admin role, and the counts against the CI canary run's own
record, then comments on the status issue and uploads `codex-recall`.

The recall graph on /security draws three lines: Claude, Codex, and
caught by either. security-audit.md keeps S8's Claude split as the
release gate and adds a "Red team" section with its cadence (quarterly,
plus before releases that change remote control or Hosted) and S275.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Deploying mouseterm with  Cloudflare Pages  Cloudflare Pages

Latest commit: 60c8467
Status: ✅  Deploy successful!
Preview URL: https://bbfd2dc2.mouseterm.pages.dev
Branch Preview URL: https://codex-advisory-engine.mouseterm.pages.dev

View logs

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The local Codex canary seeds the checkout's HEAD, not the commit of the run its --key names. In run_canary, git archive HEAD | tar -x -C "$tree" builds the tree whatever --key says. The Red team section says to key it by "the latest CI canary run on main". That run is weekly and main usually moves past it, so a maintainer who runs from an up-to-date main gets the following:

  • union passes, because both scorecards hold the same seed IDs from the same pool.
  • The script prints a dispatch command with commit=<local HEAD>.
  • checkAgainstClaude then rejects it ("seeded X, not Y"), after a full five-domain Codex canary has already run.

Publishing stays correct; the problem is that the quarterly run is wasted, and the requirement to be at that commit appears only in a spec sentence ("when run at its commit"). One fix makes this correct by construction: when --key is a run ID, read its head_sha (gh api repos/$GITHUB_REPOSITORY/actions/runs/$key --jq .head_sha), git archive that commit, and pass it to print_codex_report as the commit. That would also drop the uncommitted-changes warning print_codex_report prints on this path, which doesn't apply to an archived tree.

listCodexRuns's second admin check has no trigger (generate-canary-recall.js). The query already keeps only status=success runs, and publishCodex throws for any non-admin GITHUB_TRIGGERING_ACTOR. So a successful run exists only if an admin triggered it, re-runs included. The only case the re-check changes is one where nothing was bypassed: an admin who is later demoted loses their records from the published graph. It also costs an API call per actor and a spec clause ("counts only from an admin's dispatch"). Dropping the role filter, and that clause, keeps the same guarantee with less code.

`--key <run ID>` now reads that run's `head_sha` and archives it, so a
maintainer on an up-to-date `main` seeds what the CI run seeded and the
printed dispatch names the commit `codex-audit-report.yaml` will accept.
The recall generator drops its second admin check: only an admin's
dispatch of the report workflow can succeed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nedtwigg

Copy link
Copy Markdown
Member Author

Both addressed in 60c8467:

  • canary --key <run ID> now reads the run's head_sha (gh api …/actions/runs/<id> --jq .head_sha), archives that commit (erroring if the clone lacks it), and passes it to the printed dispatch; the uncommitted-changes warning moved to the verdicts path only. The canary test now runs from a commit past the run's and fails if that later file is seeded.
  • Dropped listCodexRuns's role filter and the spec's admin clause for the snapshot; the builder's admin refusal (S275) remains the one gate.

@dormouse-bot

Copy link
Copy Markdown
Collaborator

Both look resolved in 60c8467. With --key <run ID>, run_canary now archives the run's head_sha and passes that commit to print_codex_report, so the printed dispatch names the commit checkAgainstClaude will accept. It also fails early if the clone doesn't have that commit, and the uncommitted-tree warning now only appears on the verdicts path. listCodexRuns now leaves the admin check to the workflow, which is the only place it's enforced.

One small asymmetry, and it fails safe: run_key is only set from --key. So a numeric CANARY_KEY in the environment still seeds HEAD and prints "keyed by no CI run", and adding --claude-scorecard with it errors with "needs the --key". The header comment says "--key (or CANARY_KEY) picks the seeds", so either derive run_key from the final key, or say there that only --key names a CI run.

@nedtwigg
nedtwigg merged commit 33804cb into main Oct 21, 2026
14 checks passed
@nedtwigg
nedtwigg deleted the codex-advisory-engine branch October 21, 2026 00:38

This branch was successfully deployed

No deployments
hosted-preview — 60c8467d Deployed Oct 21, 2026 by nedtwigg via cleanup #1714
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants