Skip to content
Closed
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,5 +47,6 @@ Every reproduced race requires a failing-before and passing-after regression. As
- In evidence records, label cited commits as baselines, intermediate checkpoints, or validated heads. Keep final exact-head results in a place that can name the resulting commit, such as the PR body or CI record.
- Reproduce summary-only review concerns or turn them into a concrete follow-up issue. Do not repeatedly patch vague wording without a failure case.
- A validation fixture for a summary-only concern must assert the disputed intermediate representation or state before using a downstream outcome as evidence that the concern was exercised.
- Manual assignment evidence must include a reproducible per-segment manifest with stable choice ID, path, operation, assigned item, and final scope; aggregate counts alone are insufficient.
- For each review round, record what changed, what was declined and why, and the regression evidence. Re-request review until a round returns no new findings.
- Treat review-lesson extraction as a merge gate. Before invoking merge, classify every review finding in the PR body as: covered by an existing rule (cite it), captured by a new rule in this branch (cite it), or one-off (record why). Do not merge until this audit is complete and every required `AGENTS.md` update is included in the reviewed head. Omit rules that merely repeat existing guidance.
18 changes: 18 additions & 0 deletions docs/experiments/manual-assignment.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"base": "5a2685cc709ca8f9f384e35e4d733012c7290d69",
"head": "442147d1bc9a33edeca1a605c9de3ada13f397f1",
"segments": [
{ "choice_id": "96c76cee6518b177f272d8dba1f8db83314d0c09b62f036c6d6f41de40227d5c", "kind": "text", "operation": "+", "path": "AGENTS.md", "before_row": "Unplanned", "item": "P4", "scope": "in-scope" },
{ "choice_id": "6000e022cb5f2c7fb602ba9dbd6b590cdf0672b1bdc984c036bf563423a86afd", "kind": "text", "operation": "+", "path": "AGENTS.md", "before_row": "Unplanned", "item": "P4", "scope": "in-scope" },
{ "choice_id": "aa5dbe642c11d5f5f6f1630c0f979006b8e0e3182da3e0aeee8485fa997784d1", "kind": "file", "operation": null, "path": "docs/experiments/review-summary-edge-cases.md", "before_row": "Unplanned", "item": "P3", "scope": "in-scope" },
{ "choice_id": "dfcf7097d83b6d55df0e4959f111370a96976b1c25f327e9d51838fc8224ceaf", "kind": "text", "operation": "+", "path": "docs/experiments/review-summary-edge-cases.md", "before_row": "Unplanned", "item": "P3", "scope": "in-scope" },
{ "choice_id": "7a5227f9a6be94d6c16d1ea058ae3a5d956a5b53f93333a15d3b778222a90218", "kind": "text", "operation": "-", "path": "test/plant.test.ts", "before_row": "Unplanned", "item": "P2", "scope": "in-scope" },
{ "choice_id": "142cf07bee22dbe7cf281b0d97efa49f9a502fed62701de8ec0c94500a94c788", "kind": "text", "operation": "+", "path": "test/plant.test.ts", "before_row": "Unplanned", "item": "P2", "scope": "in-scope" },
{ "choice_id": "36cb67aa87587c9159f19ce0bac78a4d67f1348f55108f99e1fed99e5273a414", "kind": "text", "operation": "+", "path": "test/plant.test.ts", "before_row": "Unplanned", "item": "P2", "scope": "in-scope" },
{ "choice_id": "88fada2e5b9a307f6db4fa6ccdbbf9e5bfbf00f26cbcacb33ac5bc141d74b85e", "kind": "text", "operation": "+", "path": "test/plant.test.ts", "before_row": "Unplanned", "item": "P2", "scope": "in-scope" },
{ "choice_id": "9f010f7ae66fd80d72c5d78d45d4effbcdac76ebb6dfadc4efd4e66ac4af9f86", "kind": "text", "operation": "-", "path": "test/review.test.ts", "before_row": "Unplanned", "item": "P1", "scope": "in-scope" },
{ "choice_id": "987cbec380909ee199bbe4df04a7d700c6f2a1f284c52f08120c159f8e92c35d", "kind": "text", "operation": "+", "path": "test/review.test.ts", "before_row": "Unplanned", "item": "P1", "scope": "in-scope" },
{ "choice_id": "6b276597aa7294f44e5a549e38d423876266fe6a9a502bb5be089db2223a8976", "kind": "text", "operation": "+", "path": "test/review.test.ts", "before_row": "Unplanned", "item": "P1", "scope": "in-scope" },
{ "choice_id": "c9379939cb69513a37cbb385440217283975294a2f8b340839d5f9d09ea126df", "kind": "text", "operation": "+", "path": "test/review.test.ts", "before_row": "Unplanned", "item": "P1", "scope": "in-scope" }
]
}
19 changes: 19 additions & 0 deletions docs/experiments/manual-assignment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Manual real-issue assignment

This prerequisite exercise uses real issue [#10](https://github.com/codeabovelab/codeboost/issues/10) and merged [PR #17](https://github.com/codeabovelab/codeboost/pull/17). It is an attribution exercise, not a timed trial and not evidence for the go/no-go thresholds.

- Base: `5a2685cc709ca8f9f384e35e4d733012c7290d69`
- Head: `442147d1bc9a33edeca1a605c9de3ada13f397f1`
- Ledger setup: the squash commit was intentionally recorded as foreign so every resulting segment required an explicit manual assignment.
- Plan: P1 rename-reassignment regression, P2 literal-path regression, P3 evidence record, and P4 durable review rules.

| Plan item | Declared file | Assigned segments | Final attribution | Final scope |
|---|---|---:|---|---|
| P1 | `test/review.test.ts` | 4 text | Attributed | In scope |
| P2 | `test/plant.test.ts` | 4 text | Attributed | In scope |
| P3 | `docs/experiments/review-summary-edge-cases.md` | 1 file + 1 text | Attributed | In scope |
| P4 | `AGENTS.md` | 2 text | Attributed | In scope |

Codeboost exposed 12 segments before assignment, all in the Unplanned row. Each segment mapped to exactly one item through that item's declared file contract. After the assignments, the per-item segment counts were 4, 4, 2, and 2; all four rows reported `✓ Attributed` and `✓ In scope`. No segment was unclear or multiply owned, so this exercise produced no ambiguity fixture. The earlier rename and literal-path concerns already have focused fixtures in `test/review.test.ts` and `test/plant.test.ts`.

The assignment was rerun from a detached worktree at the exact head using a fresh SQLite review store. The stored choices, rather than a hand-written diff classification alone, produced the final attribution and scope results above. `manual-assignment.json` records every stable choice ID with its kind, operation, path, original row, assigned item, and final scope so the 12 assignments can be checked individually.
22 changes: 14 additions & 8 deletions docs/experiments/review-protocol.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,25 @@
# Plan-indexed review: go/no-go protocol

Status: **not started; issue pairs and human reviewer still required**. This document does not claim a passed gate. Complete the pair table and commit its exact revision before any timed review. No pair or threshold may change once the first review begins.
Status: **pair selection frozen; pair 1 prepared; reviews not started**. Max Hwang (`mchwang`) is the blinded reviewer and Codex is the experiment operator. This document does not claim a passed gate. Complete each row's PR and SHA fields and commit that exact revision before the first timed review. No issue, assignment, order, or threshold may change once the first review begins.

## Pair selection and order

Choose four pairs of comparable, small real issues in a repository owned by the reviewer. Match each pair on scope, language, and estimated review complexity before implementing either issue. Record issue URLs, PR URLs, base/head SHAs, and which review method each gets. Do not reuse the demo fixture as an experimental issue.

| Pair | Real issue A / PR | Real issue B / PR | A method | B method |
All eight issues are in `codeabovelab/guardyx-gstack`. A/B placement within each pair was randomized once before implementation; the method pattern remains the alternating pattern approved in the protocol.

| Pair | Real issue A / PR / base..head | Real issue B / PR / base..head | A method | B method |
|---|---|---|---|---|
| 1 | Pending selection | Pending selection | Raw GitHub diff | codeboost |
| 2 | Pending selection | Pending selection | codeboost | Raw GitHub diff |
| 3 | Pending selection | Pending selection | Raw GitHub diff | codeboost |
| 4 | Pending selection | Pending selection | codeboost | Raw GitHub diff |
| 1 — companion documentation | [#493](https://github.com/codeabovelab/guardyx-gstack/issues/493) / [PR #600](https://github.com/codeabovelab/guardyx-gstack/pull/600) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..6a273dab42005e01a6f4c2ea92297f40ad29e0d7` | [#176](https://github.com/codeabovelab/guardyx-gstack/issues/176) / [PR #601](https://github.com/codeabovelab/guardyx-gstack/pull/601) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..f61a793ebe4e7352f7ebf74238adfad3c18750d0` | Raw GitHub diff | codeboost |
| 2 — agent-key security | [#506](https://github.com/codeabovelab/guardyx-gstack/issues/506) / PR pending / SHA pending | [#443](https://github.com/codeabovelab/guardyx-gstack/issues/443) / PR pending / SHA pending | codeboost | Raw GitHub diff |
| 3 — asynchronous notifications | [#13](https://github.com/codeabovelab/guardyx-gstack/issues/13) / PR pending / SHA pending | [#429](https://github.com/codeabovelab/guardyx-gstack/issues/429) / PR pending / SHA pending | Raw GitHub diff | codeboost |
| 4 — compliance evidence | [#504](https://github.com/codeabovelab/guardyx-gstack/issues/504) / PR pending / SHA pending | [#507](https://github.com/codeabovelab/guardyx-gstack/issues/507) / PR pending / SHA pending | codeboost | Raw GitHub diff |

The pairs are matched before implementation on repository, primary implementation language, scope, and expected review surface. Pair 1 changes operational/compliance documentation without runtime behavior; pair 2 changes agent-key lifecycle across the API and UI; pair 3 adds medium asynchronous notification delivery across backend jobs and consumer-facing interfaces; pair 4 adds compliance evidence across stored invocation data, exports, mapping data, and PDF surfaces. If implementation discovery makes a pair materially unequal or blocked, stop before the first timed review and record a new frozen protocol revision rather than silently substituting an issue.

Selection history: the first frozen draft paired #554 with #556. Before either implementation began, their required baseline check found that both depend on live 1,400-action/session telemetry unavailable from the operator's environment. A proposed replacement included #335, but inspect-before-build then proved that `main` already contains its requested in-flight guard even though the issue remains open. A current-tree audit also excluded other stale open issues whose requested work has shipped. The table above contains only work confirmed absent from current `main`; every A/B placement was randomized after that audit. No timed review, experimental implementation, or plant had started before this final selection.

The reviewer must not implement the paired changes or inspect the plants before deciding on the PR. An operator prepares plans and commits by hand, records trusted ledger ownership, and supplies both the plan and code to each review method. The earlier one-issue manual assignment also remains required: record how every change maps to the attribution table and turn unclear cases into fixtures.
The reviewer must not implement the paired changes or inspect the plants before deciding on the PR. The operator must not expose plant inputs, `sealed.json`, mappings, planted locations, or unblinded implementation notes in the PR, review package, task messages, or protocol updates. An operator prepares plans and commits by hand, records trusted ledger ownership, and supplies both the plan and code to each review method. The earlier one-issue manual assignment is recorded in `manual-assignment.md`; all 12 segments were assigned and in scope, with no unclear case requiring another fixture.
Comment thread
mchwang marked this conversation as resolved.
Resolved

## Planting

Expand All @@ -37,4 +43,4 @@ Report declared-file catch rate for both methods without a pass threshold. Stop

## Current evidence

Engineering tests prove browser interactions, persistence, attribution, and planting mechanics. They do **not** establish human review speed or catch rates. No real issue pairs, timed decisions, or catch-rate results exist yet.
Engineering tests prove browser interactions, persistence, attribution, and planting mechanics. The separate manual assignment proves that one real merged issue can be completely classified through stored review choices. Neither establishes human review speed or catch rates. Pair 1 is prepared at the exact PR revisions above and passed the repository preflight; the other three pairs remain in preparation. No timed decision or catch-rate result exists yet.
2 changes: 1 addition & 1 deletion docs/experiments/review-summary-edge-cases.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ Result: no defect reproduced. The summary supplied no expected color, state toke

## Gate result

All three concerns are resolved as validated behavior. No issue supplied a failing case that justified a production patch. Issue #3's manual assignment and paired human go/no-go experiment remain pending and must use the frozen protocol in `review-protocol.md`.
All three concerns are resolved as validated behavior. No issue supplied a failing case that justified a production patch. Issue #3's manual assignment is recorded in `manual-assignment.md`; its paired human go/no-go experiment remains pending and must use the frozen protocol in `review-protocol.md`.
2 changes: 1 addition & 1 deletion docs/implementation/read-only-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Store schema v2 adds the review counter and per-item notes through a transaction

- No agent answers, test execution, AI findings ingestion, send-to-agent action, or merge control is included. These belong after the go/no-go gate. Four checks distinguish unavailable evidence from success.
- File cards show mode/path/object IDs and blob byte sizes. PNG/JPEG/GIF/WebP previews are bounded to 1 MiB each and 4 MiB across a history; unsupported/oversized images say unavailable. Gitlink byte sizes are not applicable. SVG/HTML is never embedded.
- The protocol at `docs/experiments/review-protocol.md` must be filled with the real pairs and committed before the first timed review. The manual assignment, real paired PRs, human timing, and final result are pending. Do not mark issue #3 closed or claim the gate passed.
- The real issue pairs are frozen in `docs/experiments/review-protocol.md`, and the completed manual assignment is recorded in `docs/experiments/manual-assignment.md`. Pair 1 is prepared at its exact PR revisions; pairs 2–4, human timing, and the final result remain pending. Do not mark issue #3 closed or claim the gate passed.
- The planting helper is intentionally limited to disposable clones and supported regular top-level paths; it never publishes PRs.

## Validation
Expand Down
Loading