Skip to content

Commit 8182417

Browse files
authored
Merge pull request #5046 from github/aaronpowell-pr-review-knowledge-uplift
Assess knowledge uplift in the code-review skill and PR Review board
2 parents 9861787 + 17ffd2e commit 8182417

3 files changed

Lines changed: 66 additions & 3 deletions

File tree

‎.github/extensions/open-pr-board/lib/review-prompt.mjs‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,11 @@ export const REVIEW_FIELDS_SCHEMA = {
2525
summary: { type: "string", description: "What the pull request actually changes." },
2626
repoFit: { type: "string", description: "Repository fit and meaningful value for Copilot users." },
2727
compliance: { type: "string", description: "Deterministic checks: frontmatter, naming, README regeneration, manifests." },
28-
differentiation: { type: "string", description: "How it differs from existing resources and native model capability." },
28+
differentiation: {
29+
type: "string",
30+
description:
31+
"Knowledge-uplift verdict (high / partial / none) with the specific content a capable model would not produce unprompted, plus overlap with existing resources.",
32+
},
2933
risks: { type: "string", description: "Correctness, security, licensing, or harmful-content concerns." },
3034
validation: { type: "string", description: "Evidence the contributor tested or validated the change." },
3135
confidence: { type: "string", enum: ["low", "medium", "high"], description: "Confidence in the recommendation." },
@@ -60,6 +64,16 @@ function sharedRules(repo, guidancePath) {
6064
" - `gh pr diff <number> -R " + repo + "` for the actual change (use `--name-only` first if the diff is large).",
6165
" - `gh pr checks <number> -R " + repo + "` when checks are failing, to see which ones and why.",
6266
"Judge the submitted result, not assumptions about the tool that produced it. A title ending in 🤖🤖🤖 is a disclosure, not a defect.",
67+
"",
68+
knowledgeUpliftRules(),
69+
].join("\n");
70+
}
71+
72+
function knowledgeUpliftRules() {
73+
return [
74+
"Knowledge uplift — apply the \"Knowledge uplift\" test from the code-review skill to every agent, instruction, skill, prompt, hook, or workflow the PR adds or substantially changes. It is the criterion most often under-applied, so do not skip it for well-formatted PRs:",
75+
" - Always start the `differentiation` field with the `high` / `partial` / `none` classification and the quoted lines that justify it, then note any overlap with existing resources.",
76+
" - `none` must not be `approve`: use `needs-insight`, or `close` when it is also a near-duplicate. `partial` is usually `request-changes`; name the generic sections to cut in `suggestedComment`.",
6377
].join("\n");
6478
}
6579

‎.github/extensions/open-pr-board/review-guidance.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,10 @@ For every pull request:
4545
reason for `request-changes`.
4646
- **Duplication** — search the repo for an existing agent/instruction/skill covering the
4747
same ground. Near-duplicates are `needs-insight` unless the overlap is total.
48+
- **Knowledge uplift** — the most common way a well-formatted PR still fails. Apply the
49+
"Knowledge uplift" test from `.github/skills/code-review/SKILL.md` to every content
50+
resource. `none` is never `approve` (use `needs-insight`, or `close` if it is also a
51+
near-duplicate); `partial` is usually `request-changes`.
4852
- **Provenance** — content copied from elsewhere without attribution, or obviously
4953
machine-generated filler with no evidence it was tested, is `needs-insight` at best.
5054
- **Checks and conflicts** — failing CI or a conflicting branch is `request-changes`,
@@ -57,7 +61,9 @@ For every pull request:
5761
- `rationale` is one sentence shown on the card. Lead with the deciding factor.
5862
- `summary` describes what the PR actually changes, in the reviewer's own words.
5963
- `repoFit`, `compliance`, `differentiation`, `risks`, and `validation` map to the
60-
SKILL.md criteria — fill in the ones that mattered, leave the rest out.
64+
SKILL.md criteria — fill in the ones that mattered, leave the rest out. The exception
65+
is `differentiation`: always fill it for content PRs, leading with the knowledge-uplift
66+
classification and its evidence.
6167
- `checkNext` is for the maintainer: the one thing to verify before acting.
6268
- `suggestedComment` is prefilled into the board's comment box, so write it as a comment
6369
addressed to the contributor. Be specific, name files and line numbers, and keep the

‎.github/skills/code-review/SKILL.md‎

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ Confirm that a submission addresses a specific GitHub Copilot workflow,
3131
technology, domain constraint, or user problem. Flag contributions that:
3232

3333
- provide generic advice that current models already handle well without
34-
meaningful uplift
34+
meaningful uplift (see [Knowledge uplift](#knowledge-uplift))
3535
- restate an existing resource without a clear differentiator
3636
- use broad claims such as doing everything for every project
3737
- lack concrete instructions, constraints, examples, or expected outcomes
@@ -41,6 +41,49 @@ Paid or commercial services are not automatically unsuitable. Evaluate whether
4141
the contribution provides standalone user value and follows the repository's
4242
guidance for paid-service submissions.
4343

44+
## Knowledge uplift
45+
46+
A resource is only valuable if it changes what the model does. Well-formatted,
47+
compliant content can still add nothing. Apply this test to every agent,
48+
instruction, skill, prompt, hook, or workflow that a PR adds or substantially
49+
changes:
50+
51+
1. Picture a current frontier model given the same task with no resource
52+
loaded. What would it do?
53+
2. Identify what this resource makes it do differently. The value is that
54+
delta, not the content itself.
55+
56+
Count as real uplift:
57+
58+
- facts the model is unlikely to know or get right, such as recent or niche
59+
APIs, version-specific breaking changes, and non-obvious gotchas
60+
- internal, proprietary, or organization-specific tools and conventions
61+
- constraints that deliberately override sensible defaults
62+
- bundled scripts, templates, or data that produce deterministic results
63+
- multi-step workflows that wire specific tools or MCP servers together
64+
- concrete examples of non-obvious expected output
65+
66+
Do not count as uplift:
67+
68+
- generic best practices such as clean code, SOLID, DRY, meaningful names,
69+
writing tests, or handling errors
70+
- restatements of well-known public documentation for mainstream languages and
71+
frameworks
72+
- persona framing such as "You are an expert X" without new constraints
73+
- checklists of behavior models already exhibit by default
74+
- long prose that a one-line user request would replace
75+
76+
Length, polish, and structure are not evidence of value. Quote or cite the
77+
specific lines that carry the uplift; if you cannot find any, say so plainly.
78+
Classify the result as:
79+
80+
- **high** — most of the content is non-obvious to a capable model
81+
- **partial** — a useful core surrounded by generic filler
82+
- **none** — a capable model would behave essentially the same without it
83+
84+
Do not recommend approval when uplift is **none**. For **partial**, ask the
85+
author to cut the named generic sections and expand the specific ones.
86+
4487
## AI-authored submissions
4588

4689
A PR title ending in `🤖🤖🤖` is an intentional AI-authorship disclosure from

0 commit comments

Comments
 (0)