docs(design): coordination design for gating marketplace skill PRs on eval results (doc 110) - #227
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a design-only coordination document outlining a strategy for gating marketplace skill pull requests based on behavioral evaluation results. The proposed approach utilizes an advisory-first lane to ensure reliability before transitioning to a required check, maintaining the integrity of the existing CI/CD pipeline. No actual CI changes are implemented in this PR; it serves as a foundational study and coordination artifact for future integration. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
WHAT: replaces the raw bead ID with the bead's plain-English title (fixes the typos gate's 'thi' finding AND complies with the bead-naming rule that autogen IDs are command handles, never quoted in docs), rewords 'de-noiser' -> 'de-noising mechanism', and prettier --write on doc 110 + 000-INDEX.md. WHY: targeted fix-up for the two real CI findings on PR #227 (typos, prettier). Vale failures are in pre-existing docs 073/086 (advisory lane) and the pytest-coverage failure is the pre-existing research/phase-a-0-baseline 43%<60% floor — both unrelated to this docs-only diff, left untouched per minimal-impact. VERIFIED: local `prettier --check` clean on both files; grep confirms no bead-ID string remains in the doc.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates issue tracking beads, adds new encrypted environment variables, and introduces a new coordination design document (110-AT-SPEC) for gating marketplace skill PRs on evaluation results. The review feedback highlights two key issues in the design document: first, a violation of Repository Style Guide Rule 6, as the design allows probabilistic LLM judge verdicts to be signed or gate merges without a deterministic wall; second, a contradiction between the path-filtered trigger in Option (a) and the 'always-report' requirement in Option (b) that could lead to blocked PRs.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| - `databricks-cluster-forensics`: 0/11 advisory → **11/11 pass** | ||
| - `coreweave-gpu-cost-leak-hunter`: advisory → **fail (7/10)** — in rollout-gate terms, the verdict swung from would-SHIP-territory to BLOCK with zero input change. | ||
|
|
||
| That flip is exactly why the flap sidecar exists (j-rig PR #220): _"a permanent signature must not ride a noisy verdict."_ The same principle transfers verbatim to gating: **never require (and never sign) on a flapping verdict.** Handling, in order: |
There was a problem hiding this comment.
According to the Repository Style Guide (Rule 6: Determinism / probabilistic boundary), an LLM must never author a signed artifact or close a deterministic signal without a deterministic wall (such as a substring pre-check + corpus post-check).
The current design notes that the nightly roster signs gate-result/v1 rows directly from the probabilistic j-rig eval output, and Option (b) proposes graduating this to a required check (a deterministic merge gate). As documented in this section, the probabilistic judge verdicts can flip (flap) with zero input changes (e.g., coreweave-gpu-cost-leak-hunter swinging from pass to fail).
To comply with Rule 6:
- For Signed Artifacts: The nightly roster must not sign probabilistic LLM judge verdicts directly without a deterministic wall.
- For Gating (Required Checks): A probabilistic LLM verdict must never directly close the deterministic PR merge gate. Only the deterministic layers (such as the golden-set correctness) should be allowed to block the merge or be signed, keeping the probabilistic LLM-panel layer strictly advisory.
References
- An LLM must never author a signed artifact or close a deterministic signal without a deterministic wall (substring pre-check + corpus post-check). (link)
| 2. **Flap rate below threshold on the affected skills:** zero unexplained gate-decision flips for the spec-covered skills over the trailing 14 nightly flap reports (`flap-report.json` sidecar), and no PR-lane verdict that contradicts the same-night nightly verdict for an unchanged skill. A flapping verdict is disqualifying by construction (§ 4). | ||
| 3. **Operator sign-off: Jeremy approves the flip.** Not the CCPI session, not this platform. | ||
|
|
||
| **Mechanics of the flip when (and only when) approved:** the eval lane becomes its **own** branch-protection status context — it is _never_ folded into `ci-required`'s `needs:`, because a path-scoped, provider-dependent job violates `ci-required`'s no-undesigned-skip invariant (a provider outage would read as a skip and green the aggregate). As its own context it must always-report (run on every PR, internally no-op-green when no spec-covered skill changed) — the same lesson `ci-required` itself encodes from #778. Escape hatch mirrors the precedent: revert one workflow file + remove one branch-protection context. |
There was a problem hiding this comment.
There is a design contradiction between the trigger defined in Option (a) and the graduation requirements in Option (b):
- Option (a) specifies the trigger as:
pull_requestonplugins/**paths. - Option (b) states that as a required check, the workflow must "always-report (run on every PR, internally no-op-green when no spec-covered skill changed)".
If the workflow trigger is restricted to plugins/** paths, any PR that does not touch these paths (e.g., dependency updates, documentation, or core package changes) will never trigger the workflow. Consequently, the required status check will remain in an "Expected" state on GitHub, blocking the PR indefinitely.
To resolve this, the design should explicitly state that either:
- The path filtering must be removed from the workflow trigger upon graduation to a required check, or
- The workflow should trigger on all PRs from the beginning, with path detection handled internally within the job steps.
…ll PRs on eval results (doc 110) WHAT: adds 000-docs/110-AT-SPEC-marketplace-skill-pr-eval-gating- coordination-design-2026-07-16.md — a design-only coordination doc (no CI changes anywhere) for wiring j-rig behavioral eval results into the marketplace (CCPI) skill-PR flow — and indexes it in 000-INDEX.md (also backfilling the drifted 107-109 governed-judgment rows). WHY: skill PRs merge on static validation only (ci-required + gitleaks) while behavioral evidence from the nightly j-rig roster gates nothing there; CCPI's CI is bespoke and load-bearing, so the standing rule is study + coordinate, never bolt on. Chose an advisory-first PR lane (minimax-review.yml shape) over an immediate required check because the 2026-07-17 nightly flap record (cost-leak-hunter advisory->fail, cluster-forensics 0/11->11/11 on identical pinned inputs) disqualifies requiring on a possibly-flapping verdict; graduation mirrors the audit-harness rollout-gate-dogfood advisory->enforcing precedent (intent-audit-harness#128) and is gated on Jeremy's explicit sign-off. NOTE: branch rewritten once pre-merge — the first push accidentally swept in two pre-staged, out-of-scope files from the shared working tree (.beads/issues.jsonl, .env.sops — another session's in-flight state); this commit contains ONLY the doc + index. Their working-tree modifications remain locally untouched for their owning session. VERIFIED: partner-name guard grep over the changed files returns zero hits (the partner-name-guard.yml CI gate re-checks); prettier@3.8.4 --check clean (the doc-quality.yml pinned version); typos gate clean after replacing the raw bead ID with its plain-English title; index table rows render. UNFINISHED: the doc authorizes nothing — next action is the CCPI session/owner handoff (doc section 6); @j-rig/pr-comment publish is a platform-side prerequisite tracked there. Refs jeremylongshore/intent-audit-harness#127
884f90d to
1d236d7
Compare
WHAT: two clarifying additions to doc 110 — (1) section 3(b) now states explicitly that graduation REMOVES option (a)'s paths filter and moves skill detection inside the always-reporting job (the two trigger shapes are sequential stages, not a contradiction); (2) section 4 gains a 'where the deterministic wall sits' paragraph: binary per-criterion verdicts + 5-sample majority feed the deterministic decideRollout() contract, signing stays out of the PR lane, and the nightly's signing of judge-derived rows is pre-existing platform behavior this design does not modify. WHY: targeted fix-up for the two gemini-code-assist review findings on PR #227 (high: probabilistic-boundary Rule 6; medium: option-a/option-b trigger contradiction). Chose amending the doc over reply-only because future readers of the filed design get the resolution, not just PR archaeology. VERIFIED: prettier@3.8.4 --check clean on the doc; partner-name grep still zero hits.
…e bbb-seed-v1 docs WHAT: prettier-formatted the three new markdown files (table alignment, emphasis markers) and gave two fenced code blocks an explicit `text` language. WHY: CI on PR #230 flagged both — `prettier --check "**/*.md"` failed on ALLOWLIST.md, README.md, and corpus/PROVENANCE.md, and markdownlint-cli2 raised MD040/fenced-code-language on ALLOWLIST.md:12 and README.md:34. Both are repo-wide gates that every markdown file here has to clear; the new files were hand-authored and had not been run through either. VERIFICATION: `markdownlint-cli2 "evals/bbb-seed-v1/**/*.md"` -> "0 issues in 0 files" (360 files linted); `prettier --check "evals/bbb-seed-v1/**/*.md"` -> "All matched files use Prettier code style!". No prose meaning changed — the diff is whitespace, table padding, `*`->`_` emphasis, and two code-fence language tags. NOTE — two OTHER failures on #230 are pre-existing and NOT from this branch: `Vale (prose lint, advisory)` scans only `000-docs/`, `README.md`, `CLAUDE.md`, and `AGENTS.md` (none touched here), and `pytest + coverage (research/phase-a-0-baseline)` reports "75 passed, 4 skipped, 1 xfailed" then exits 2 in the coverage step, against Python this branch does not touch. Both fail identically on already-merged PRs #229, #227, and #226. Out of scope here; worth their own fix. Refs #230
…e bbb-seed-v1 docs WHAT: prettier-formatted the three new markdown files (table alignment, emphasis markers) and gave two fenced code blocks an explicit `text` language. WHY: CI on PR #230 flagged both — `prettier --check "**/*.md"` failed on ALLOWLIST.md, README.md, and corpus/PROVENANCE.md, and markdownlint-cli2 raised MD040/fenced-code-language on ALLOWLIST.md:12 and README.md:34. Both are repo-wide gates that every markdown file here has to clear; the new files were hand-authored and had not been run through either. VERIFICATION: `markdownlint-cli2 "evals/bbb-seed-v1/**/*.md"` -> "0 issues in 0 files" (360 files linted); `prettier --check "evals/bbb-seed-v1/**/*.md"` -> "All matched files use Prettier code style!". No prose meaning changed — the diff is whitespace, table padding, `*`->`_` emphasis, and two code-fence language tags. NOTE — two OTHER failures on #230 are pre-existing and NOT from this branch: `Vale (prose lint, advisory)` scans only `000-docs/`, `README.md`, `CLAUDE.md`, and `AGENTS.md` (none touched here), and `pytest + coverage (research/phase-a-0-baseline)` reports "75 passed, 4 skipped, 1 xfailed" then exits 2 in the coverage step, against Python this branch does not touch. Both fail identically on already-merged PRs #229, #227, and #226. Out of scope here; worth their own fix. Refs #230
1 participant
What
Files
000-docs/110-AT-SPEC-marketplace-skill-pr-eval-gating-coordination-design-2026-07-16.md— a design-only coordination doc for gating marketplace (CCPI) skill PRs on j-rig behavioral eval results — and updates000-docs/000-INDEX.md(new row 110, plus backfilling the drifted 107-109 governed-judgment rows). No CI changes anywhere; the doc explicitly authorizes none.Why + decision rationale
Skill PRs merge into the marketplace on static validation only (
ci-required+gitleaks); the behavioral evidence from the nightly j-rig roster (8 spec-covered skills, MiniMax-M3, 5-sample majority, signed gate-result/v1 rows) gates nothing there. CCPI's CI is bespoke and load-bearing — standing rule: study + coordinate, never bolt on. This doc is that study + coordination artifact.Recommendation: advisory-first PR eval lane (structural sibling of CCPI's
minimax-review.yml: same-repo guard, kill-switch variable, sticky comment, never inci-required's needs), with pre-agreed graduation criteria to a required check mirroring the audit-harnessrollout-gate-dogfood.ymladvisory-to-enforcing precedent, and a spec-coverage rule (no eval-spec = exempt + labeled). Chose advisory-first over an immediate required check because the 2026-07-17 nightly flap record (runs 29545349026 vs 29549429404, identical pinned inputs:coreweave-gpu-cost-leak-hunteradvisory→fail 7/10;databricks-cluster-forensics0/11→11/11) disqualifies requiring — or signing — on a possibly-flapping verdict.Layers touched
Lab
000-docs/only (methodology/spec layer). No workflow, schema, or code changes; no invariant changes.How it works
The doc maps the CCPI CI constraints (ci-required aggregate no-undesigned-skip invariant, auto-bump
[skip auto-bump]gotcha #985, scan-allowlist waiver pattern, fork-PR secret-safety pattern, existingMINIMAX_API_KEYsecret), specifies the advisory lane shape + real cost/latency data (~26 min for the full 8-skill nightly incl. build/sign, ~3 min/skill amortized, single-digit cents/skill), hard boundaries (never weaken ci-required/gitleaks; additive-only; keys never reach fork PRs; no PR-lane signing), and decision rights (CCPI session/owner executes the workflow addition; this platform provides CLI + decision lib + evidence contract; Jeremy alone approves required-check graduation).Verification & evidence
partner-name-guard.ymlCI gate re-checks on this PR).claude-code-plugins-plus-skillsworkflows (validate-plugins.ymlci-required block,auto-bump-on-pr.yml,minimax-review.yml,emit-evidence.yml,scripts/scan-allowlist.txt),j-rig-skill-binary-eval/.github/workflows/nightly-skill-evals.yml+eval-roster/roster.json+ PR chore(beads): commit lab bead-state export left uncommitted by prior sessions #220,intent-audit-harness/.github/workflows/rollout-gate-dogfood.yml, and GH run 29549429404 timing.Risk assessment
Docs-only: zero runtime risk. Residual risk is design-level — if CCPI CI changes before the handoff, the CCPI session amends at step 2 of the sequencing table (the doc says it owns that call).
Operational impact
None now. When implemented per the doc: one additive CCPI workflow, reuses the existing
MINIMAX_API_KEYrepo secret (no new secret for the recommended provider),DEEPSEEK_API_KEYonly if funded.Follow-up & deferred
@j-rig/pr-commentis workspace-internal, not on npm — publish under@intentsolutions/*or bundle into@intentsolutions/jrig-cli(j-rig repo).Governance links
Bead: "Draft the coordination design for gating marketplace skill PRs on eval results without disturbing the bespoke CCPI CI" (epic: give the rollout gate its first real enforcement teeth). Precedent: intent-audit-harness#128 (enforcing flip). Flap instrumentation: j-rig-skill-binary-eval#220.
Refs jeremylongshore/intent-audit-harness#127
intentsolutions.io