Skip to content

Commit d613147

Browse files
jamesmontemagnoCopilot
jamesmontemagno
and
Copilot
authored
Phase 2: submission-gate, risk tiers, and PR status state machine (#4184) (#4190)
* Add submission-gate check, merge-risk tiers, and PR status state machine (#4184) Phase 2 (enforcement) of #4184: - submission-gate aggregate required check (reader) with infra vs contribution failure classification - merge-risk:low/medium/high classification from .github/risk-tiers.yml with tier approval policy - Submission Gate Writer maintaining state labels and a persistent status comment - /rerun-checks and /request-review PR commands - validate-skills and validate-submission-gate workflows, unit tests, and maintainer docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix codespell findings and add spelling to submission gate checks Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden submission gate per review feedback - Publish the submission-gate check from the trusted writer via the Checks API and flag any other submission-gate check run as tampering - Fail closed on truncated file lists, unscannable diffs, permission lookup errors, and skipped required checks - Discard evaluations when the PR head or reviews change mid-run - Low tier requires a resource owner approval - Commands must start the comment, matching the workflow filter Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address #4190 review: command reader/writer, pools, paths, staleness - Split /rerun-checks and /request-review into a read-only issue_comment reader and a workflow_run writer that re-reads the comment, PR, and permission before writing. - Drop the workflow-security pool; agentic workflows and hooks are content (medium unless they add scripts or hook commands); the core pool is core-maintainers. - Include each check's workflow file and eng scripts in its paths. - Discard evaluations when the base branch or risk labels change; run the gate on PR edits (retargeting). - Correct copies of the writer's check that claim success while the evaluation fails. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fail closed on lost contributor signal; publish gate check before labels - A contributor check whose pr-check job succeeded but whose result artifact is missing or unreadable is now an infrastructure failure, so a lost HIGH signal can't lower the approval tier. - Publish the submission-gate check right after stale-state revalidation; label and comment sync are independent and reported afterwards. - Revert the self-path added to validate-agentic-workflows-pr.yml's trigger: that workflow rejects any .github change. Exempt it in the drift test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fail closed on bad routing config; advisory checks never hold the gate - loadGateConfig: a missing review-routing.yml still falls back, but a file that can't be parsed (or whose pools isn't a mapping) now throws instead of silently relaxing approvals. - Pending required: false checks no longer keep the PR in awaiting-automation or the gate check in progress, and the polling loop doesn't wait on them. They render as pending (advisory). - PR commands: 👀 marks a command as claimed; 🚀 is written only after the reply is posted and is the only replay guard, so a run that failed partway can be retried. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix codespell: unparseable -> unparsable Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 6c5a964 commit d613147

18 files changed

Lines changed: 3550 additions & 0 deletions

‎.github/risk-tiers.yml‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# Merge risk tiers.
2+
#
3+
# Automation (eng/submission-gate.mjs) applies exactly one of `merge-risk:low`,
4+
# `merge-risk:medium`, or `merge-risk:high` to every open PR and the
5+
# `submission-gate` check enforces the approvals each tier requires.
6+
# See docs/maintainers/submission-gate.md.
7+
#
8+
# Evaluation order:
9+
# 1. high if any changed file matches `high.paths`, any added line matches a
10+
# `high.capabilities` trigger, a `high.labels` label is present, or the
11+
# changes can't be fully scanned (GitHub truncated the file list, or a
12+
# text file in a capability's scope has no diff to scan).
13+
# 2. low if every changed file matches `low.paths`, or the PR only modifies
14+
# existing resource files (no added/removed/renamed files) and the total
15+
# change is at most `low.small_update.max_changed_lines`.
16+
# 3. medium otherwise.
17+
#
18+
# This file is a review-policy file: changing it places a PR in merge-risk:high.
19+
20+
high:
21+
description: Privileged execution, automation, or review-policy change
22+
paths:
23+
# Review policy and automation
24+
- ".github/**"
25+
- "CODEOWNERS"
26+
- "docs/CODEOWNERS"
27+
# Build, release, and repository scripts
28+
- "eng/**"
29+
- "scripts/**"
30+
- "package.json"
31+
- "package-lock.json"
32+
# MCP server configuration
33+
- "**/mcp.json"
34+
- "**/.mcp.json"
35+
# Bundled executable scripts (hooks, skills, and plugins). Agentic workflow
36+
# sources (`workflows/**`) and hook metadata (`hooks/**`) are content; a hook
37+
# command in hooks.json is caught by the `mcp-server-command` capability.
38+
- "**/*.sh"
39+
- "**/*.bash"
40+
- "**/*.ps1"
41+
- "**/*.psm1"
42+
- "**/*.bat"
43+
- "**/*.cmd"
44+
- "**/*.py"
45+
- "skills/**/scripts/**"
46+
- "plugins/**/scripts/**"
47+
# Plugin marketplace sources for externally hosted code
48+
- "plugins/external.json"
49+
# Generated output that lives under a high-risk path but is safe to regenerate.
50+
exclude_paths:
51+
- ".github/plugin/marketplace.json"
52+
# Regexes matched against added lines of the diff. `files` limits where each applies.
53+
capabilities:
54+
- id: process-execution
55+
description: Spawns processes or evaluates code
56+
files: ["extensions/**", "plugins/**", "skills/**", "website/**"]
57+
pattern: "child_process|\\bexecSync\\(|\\bexecFile(Sync)?\\(|\\bspawn(Sync)?\\(|\\beval\\(|new Function\\(|node:vm|require\\(['\"]vm['\"]\\)"
58+
- id: remote-script-execution
59+
description: Pipes a downloaded script into a shell
60+
pattern: "(curl|wget)[^\\n|]*\\|\\s*(sudo\\s+)?(ba|z)?sh\\b|(iwr|irm|Invoke-WebRequest|Invoke-RestMethod)[^\\n|]*\\|\\s*(iex|Invoke-Expression)|Invoke-Expression"
61+
- id: mcp-server-command
62+
description: Declares an MCP server or hook command
63+
files: ["plugins/**/*.json", "extensions/**/*.json", "skills/**/*.json", "hooks/**/*.json"]
64+
pattern: "\"(mcpServers|command|bash|powershell)\"\\s*:"
65+
# Files whose missing diff is acceptable: binaries GitHub never diffs, and generated
66+
# documentation whose sources are scanned. Other files in a capability's scope without
67+
# a diff (binary or too large) fail closed to high because their added lines can't be scanned.
68+
unscanned_paths:
69+
- "docs/**"
70+
- "README.md"
71+
- ".github/plugin/marketplace.json"
72+
- "**/*.png"
73+
- "**/*.jpg"
74+
- "**/*.jpeg"
75+
- "**/*.gif"
76+
- "**/*.webp"
77+
- "**/*.ico"
78+
- "**/*.pdf"
79+
- "**/*.woff"
80+
- "**/*.woff2"
81+
- "**/*.ttf"
82+
- "**/*.otf"
83+
- "**/*.mp4"
84+
- "**/*.webm"
85+
- "**/*.zip"
86+
labels:
87+
# Applied by the Contributor Reputation Check writer.
88+
- "needs-review:HIGH"
89+
approvals:
90+
required: 2
91+
require_core: true
92+
93+
medium:
94+
description: New or substantially changed resource without privileged execution
95+
approvals:
96+
required: 1
97+
require_domain: true
98+
99+
low:
100+
description: Documentation, metadata, generated output, or a small update to an existing resource
101+
paths:
102+
- "docs/**"
103+
- "README.md"
104+
- "CONTRIBUTING.md"
105+
- "CODE_OF_CONDUCT.md"
106+
- "SECURITY.md"
107+
- "SUPPORT.md"
108+
- "LICENSE"
109+
- ".all-contributorsrc"
110+
- ".github/plugin/marketplace.json"
111+
- "**/*.png"
112+
- "**/*.jpg"
113+
- "**/*.jpeg"
114+
- "**/*.gif"
115+
- "**/*.svg"
116+
- "**/*.webp"
117+
small_update:
118+
max_changed_lines: 40
119+
# Existing resources that may be updated in a small change and stay low risk.
120+
resource_paths:
121+
- "agents/**"
122+
- "instructions/**"
123+
- "skills/**"
124+
- "plugins/**"
125+
- "extensions/**"
126+
approvals:
127+
# One approval from an owner of the changed resource (its domain pool, or the core
128+
# pools for files outside every domain). Until pools are staffed, any approver with
129+
# write access counts.
130+
required: 1
131+
require_owner: true
132+
133+
# Areas used to pick the domain reviewer pool from .github/review-routing.yml.
134+
# `pools` are routing pool keys. When a matching pool has no reviewers yet, any
135+
# approver with write access satisfies the domain requirement.
136+
domains:
137+
canvas:
138+
paths: ["extensions/**"]
139+
pools: ["canvas"]
140+
plugin:
141+
paths: ["plugins/**"]
142+
pools: ["plugin"]
143+
content:
144+
paths: ["agents/**", "instructions/**", "skills/**", "workflows/**", "hooks/**"]
145+
pools: ["content"]
146+
# Files outside every domain (repository automation such as `.github/workflows/**`,
147+
# `eng/**`, review policy, and docs) are owned by the core pools.
148+
149+
# Routing pools whose members count as core maintainers for high-risk approvals.
150+
# Until those pools are staffed, users with admin or maintain permission on the
151+
# repository count instead.
152+
core_pools: ["core-maintainers"]

‎.github/submission-gate.yml‎

Lines changed: 234 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
# Submission gate configuration.
2+
#
3+
# The submission gate aggregates the checks below on the PR head commit into one
4+
# required `submission-gate` check. The read-only Submission Gate workflow waits for
5+
# them; the trusted Submission Gate Writer publishes the check.
6+
# Logic lives in eng/submission-gate.mjs. See docs/maintainers/submission-gate.md.
7+
#
8+
# This file is a review-policy file: changing it places a PR in merge-risk:high.
9+
#
10+
# Check fields:
11+
# id Stable identifier used in the status comment.
12+
# title Human-readable name.
13+
# workflow Workflow file (under .github/workflows/) that reports the check.
14+
# Omit and set `check_name` for checks matched by check-run name only.
15+
# check_name Match a check run by name instead of by workflow (any workflow).
16+
# branches Base branches the check runs for (mirrors the workflow trigger).
17+
# paths Path globs that make the check applicable (mirrors the workflow's
18+
# `paths` filter). Omit to apply to every PR. Kept in sync with the
19+
# workflow triggers by eng/submission-gate.test.mjs.
20+
# required true: failures block the gate. false: failures are shown as warnings.
21+
# optional true: skip silently when the check never reports (pluggable slot).
22+
# allow_skip true: a run its workflow skipped counts as skipped. Otherwise a skipped
23+
# required check is an infrastructure failure, because it validated nothing.
24+
# failure_kind auto (default): classify a failed run by its failing step.
25+
# infrastructure: every failure of this check is an infrastructure failure.
26+
# contribution_steps Step-name patterns (regex) that mean the contribution failed.
27+
# hint Fix guidance shown for contribution failures.
28+
29+
wait:
30+
# Maximum time the gate polls for pending checks before reporting them as incomplete.
31+
timeout_minutes: 40
32+
interval_seconds: 30
33+
# A check that has not appeared after this long (while everything else finished)
34+
# is reported as "did not report".
35+
report_grace_minutes: 8
36+
37+
# Failed steps that match these patterns are infrastructure failures (runner, network,
38+
# dependency install, artifact transfer) rather than problems with the contribution.
39+
infrastructure_steps:
40+
- "^Set up job$"
41+
- "^Complete job$"
42+
- "^Post "
43+
- "(Checkout|checkout)"
44+
- "^Setup (Node|Python)"
45+
- "^Set up (Node|Python)"
46+
- "(Install|install) (dependencies|gh-aw)"
47+
- "^Fetch AGT"
48+
- "(Upload|Download) .*artifact"
49+
50+
checks:
51+
- id: line-endings
52+
title: Line endings
53+
workflow: check-line-endings.yml
54+
branches: [main]
55+
required: true
56+
contribution_steps: ["CRLF"]
57+
hint: Run `bash eng/fix-line-endings.sh` and commit the result.
58+
59+
- id: spelling
60+
title: Spelling
61+
workflow: codespell.yml
62+
branches: [main]
63+
required: true
64+
contribution_steps: ["^Check spelling with codespell$"]
65+
hint: Fix the misspellings reported by codespell in the job log.
66+
67+
- id: readme
68+
title: Generated README consistency
69+
workflow: validate-readme.yml
70+
branches: [main]
71+
paths:
72+
- "instructions/**"
73+
- "prompts/**"
74+
- "agents/**"
75+
- "plugins/**"
76+
- "workflows/**"
77+
- "*.js"
78+
- "README.md"
79+
- "docs/**"
80+
- "skills/**"
81+
- "eng/update-readme.mjs"
82+
- "eng/generate-marketplace.mjs"
83+
- "eng/validate-plugins.mjs"
84+
- ".github/workflows/validate-readme.yml"
85+
required: true
86+
contribution_steps: ["^Validate plugins$", "^Update README", "^Fail workflow if files need updating$"]
87+
hint: Run `npm start` locally and commit the regenerated files.
88+
89+
- id: plugin-validation
90+
title: Plugin and extension validation
91+
workflow: validate-plugins.yml
92+
branches: [main]
93+
paths:
94+
- "plugins/**"
95+
- "extensions/**"
96+
- "eng/validate-plugins.mjs"
97+
- ".github/workflows/validate-plugins.yml"
98+
required: true
99+
contribution_steps: ["^Validate plugins and extensions$"]
100+
hint: Run `npm run plugin:validate` locally and fix the reported errors.
101+
102+
- id: canvas-extension-validation
103+
title: Canvas extension validation
104+
workflow: validate-canvas-extensions.yml
105+
branches: [main]
106+
paths:
107+
- "extensions/**"
108+
- "eng/validate-plugins.mjs"
109+
- ".github/workflows/validate-canvas-extensions.yml"
110+
required: true
111+
contribution_steps: ["^Validate changed extensions$"]
112+
hint: Run `npm run plugin:validate` locally and fix the reported errors.
113+
114+
- id: plugin-structure
115+
title: Plugin structure
116+
workflow: check-plugin-structure.yml
117+
branches: [main]
118+
paths:
119+
- "plugins/**"
120+
- ".github/workflows/check-plugin-structure.yml"
121+
required: true
122+
contribution_steps: ["materialized files"]
123+
hint: Remove materialized or symlinked files from the plugin directory.
124+
125+
- id: skill-validation
126+
title: Skill validation
127+
workflow: validate-skills.yml
128+
branches: [main]
129+
paths:
130+
- "skills/**"
131+
- "eng/validate-skills.mjs"
132+
- "eng/yaml-parser.mjs"
133+
- ".github/workflows/validate-skills.yml"
134+
required: true
135+
contribution_steps: ["^Validate skills$"]
136+
hint: Run `npm run skill:validate` locally and fix the reported errors.
137+
138+
- id: skill-lint
139+
title: Skill lint (vally)
140+
workflow: skill-check.yml
141+
branches: [main]
142+
paths:
143+
- "skills/**"
144+
- "agents/**"
145+
- "plugins/**/skills/**"
146+
- "plugins/**/agents/**"
147+
- ".github/workflows/skill-check.yml"
148+
required: true
149+
failure_kind: infrastructure
150+
151+
- id: submission-gate-validation
152+
title: Submission gate tests
153+
workflow: validate-submission-gate.yml
154+
branches: [main]
155+
paths:
156+
- ".github/submission-gate.yml"
157+
- ".github/risk-tiers.yml"
158+
- ".github/workflows/**"
159+
- "eng/submission-gate.mjs"
160+
- "eng/submission-gate.test.mjs"
161+
required: true
162+
contribution_steps: ["^Test submission gate$"]
163+
hint: Run `node --test eng/submission-gate.test.mjs` and keep `.github/submission-gate.yml` in sync with workflow triggers.
164+
165+
- id: agentic-workflow-validation
166+
title: Agentic workflow validation
167+
workflow: validate-agentic-workflows-pr.yml
168+
branches: [main]
169+
paths:
170+
- "workflows/**"
171+
required: true
172+
contribution_steps: ["^Check for forbidden files$", "^Compile workflow files$"]
173+
hint: Only add `.md` sources under `workflows/` and make sure `gh aw compile --validate` passes.
174+
175+
- id: risk-scan
176+
title: Risk scan
177+
workflow: pr-risk-scan.yml
178+
branches: [main]
179+
paths:
180+
- "skills/**"
181+
- "agents/**"
182+
- "workflows/**"
183+
- "plugins/**"
184+
- "hooks/**"
185+
- "instructions/**"
186+
- "eng/pr-risk-scan.mjs"
187+
- ".github/workflows/pr-risk-scan.yml"
188+
required: true
189+
failure_kind: infrastructure
190+
191+
- id: contributor-reputation
192+
title: Contributor reputation
193+
workflow: contributor-check.yml
194+
required: true
195+
# Its jobs skip for bot authors (Dependabot, github-actions, Copilot coding agent).
196+
allow_skip: true
197+
failure_kind: infrastructure
198+
199+
# AI-assisted advisory reviews. Their findings are posted as comments and never fail
200+
# the run, so a failed run is always an infrastructure failure (for example the
201+
# intermittent Copilot inference 401). They are non-blocking by design.
202+
- id: duplicate-scan
203+
title: Duplicate resource scan
204+
workflow: pr-duplicate-check.lock.yml
205+
required: false
206+
failure_kind: infrastructure
207+
208+
- id: quality-signal
209+
title: PR quality signal
210+
workflow: pr-quality-signal.lock.yml
211+
required: false
212+
failure_kind: infrastructure
213+
214+
# Pluggable slot for the canvas/plugin materialization and install smoke test.
215+
# Any workflow can satisfy it by reporting a job named `canvas-smoke-test`.
216+
- id: canvas-smoke-test
217+
title: Canvas/plugin smoke test
218+
check_name: canvas-smoke-test
219+
branches: [main]
220+
paths:
221+
- "extensions/**"
222+
- "plugins/**"
223+
required: true
224+
optional: true
225+
contribution_steps: ["smoke", "(Materialize|materialize|Materialization|materialization)", "(Install|install) plugin"]
226+
hint: See the smoke-test job log for the failing plugin or extension.
227+
228+
commands:
229+
request_review:
230+
label: needs-reviewer
231+
# Reviewer routing workflow dispatched after the label is added (labels added with
232+
# GITHUB_TOKEN do not trigger `labeled` workflows). Ignored if the file is absent.
233+
dispatch_workflow: review-routing.yml
234+
dispatch_ref: main

‎.github/workflows/check-plugin-structure.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ on:
55
branches: [main]
66
paths:
77
- "plugins/**"
8+
- ".github/workflows/check-plugin-structure.yml"
89

910
permissions:
1011
contents: read

0 commit comments

Comments
 (0)