diff --git a/AGENTS.md b/AGENTS.md index b0aa5671a..53d375c6e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -79,9 +79,10 @@ Feedback: paths are affected, including checking existing APIs before accepting a new one. The workflow or coordinator owns mechanical validation and review bookkeeping. -- Start Fable reviews without waiting for CI on `main` or the reviewed branch. - Main health and required checks remain merge gates; a review pass waives - neither. +- Start internal feedback reviews without waiting for CI. Start final Fable + delivery review only after the operator submits the exact draft with internal + feedback complete and required PR CI green. Main health and required checks + remain merge gates. - When a required Fable review is unavailable (including exhausted quota), use Codex Astra (`gpt-6-astra`) with the same independence from the Implementor, scope, reasoning-effort policy and correction limits. Reconcile any in-flight diff --git a/examples/adversarial-implementation/README.md b/examples/adversarial-implementation/README.md index eaf55140d..48430e091 100644 --- a/examples/adversarial-implementation/README.md +++ b/examples/adversarial-implementation/README.md @@ -248,11 +248,12 @@ xmd run --allow-absolute /absolute/path/to/workflow/examples/adversarial-impleme --include /absolute/path/to/workflow/examples/adversarial-implementation/components --raw ``` -`status` starts no model. It reads the saved joined review, the current PR and -GitHub's CI workflow runs. Its `nextAction` is computed from the same facts as -its `status` and `remaining` fields. When it requests `review`, select -`--props-action review` on that command to perform the independent review with -the provider, model, effort and session recorded at launch. The review uses +Read `status` to see the current heads, submitted revision, required gates, +and next action. After exact-SHA internal feedback and green required CI, +run the same command with `--props-action submit --props-revision `. +Submission pins the clean checkout's commit on the matching open draft PR +against main. Interim pushes, feedback, CI retries and status reads leave the +draft waiting for submission. The review uses normal coding-agent permissions in a provider scope of its own. It receives the original captured acceptance text and may inspect source and run focused checks; it receives no implementation, delegation, posting or merge task. @@ -291,14 +292,15 @@ again. `status`, `closeout` and `reconcile` ask nothing and start no reviewer. ```text feedback pending -> continue original feedback -feedback ready -> independent review pending -> joined exact-commit verdict - | Fable unavailable/refused -> one Astra review (review) +feedback + green required CI -> explicit submit -> joined exact-commit verdict + | Fable unavailable/refused -> one Astra review (submit) | no reviewer completed -> review_held: answer via review | incomplete/uncertain -> reconcile original call - | blocking finding -> request approved repair - | pass + required CI + main -> finish PR metadata - | all passed + PR ready -> await authorized merge - | confirmed merge -> delivered + | comment pending -> publish-review + | published changes required -> request approved repair + | published pass + checks -> ready + | all passed + PR ready -> await authorized merge + | confirmed merge -> delivered ``` Independent review summaries allow 1500 characters; detailed observations belong @@ -340,6 +342,28 @@ and saves a raw reply the execution joined but never wrote. It never shortens te a model call or overwrites a contradictory receipt. Missing or uncertain execution continues to hold. Status uses this same validation boundary to read receipts. +`submit` and `reconcile` publish each validated verdict as a comment on the +current pull request, including findings that require repair. The comment +names the exact reviewed commit and the reviewer who supplied the result. +Delivery uses `` to reconcile and verify the comment. With +the native GitHub provider, include the repository URL in the host's +`XMD_WORKFLOW_GITHUB_PULL_REQUESTS` allowlist. For a host that reads one repository: + +```sh +export XMD_WORKFLOW_GITHUB_PULL_REQUESTS='{"allowed":["https://github.com/owner/repository"]}' +``` + +Once GitHub returns the matching comment, the workflow saves its URL with the +reviewed SHA. A new head has its own comment. + +If `nextAction` is `publish_review`, use `--props-action publish-review` with +the same record and PR number. It checks for the matching comment before any +write and adopts it if GitHub already accepted it. Concurrent calls with the +same delivery record share one write reservation per PR and reviewed SHA. +If a write could have reached GitHub and no matching comment is visible, the +reservation stays in place for reconciliation. `status` and `closeout` only +read GitHub. A pending comment holds readiness and the repair handoff. + Reconciliation is limited to independent review from a work item launched under the same response contract. Its assignment retains the exact response schema; a missing or different schema refuses rather than reinterpreting an old reply. It validates only corrections recorded under the original review assignment. @@ -377,8 +401,27 @@ Main health reuses the repository's authoritative selection rule for a push to main's exact current head. Pending, cancelled, skipped and failed runs cannot pass. The maintainer-applied `ci-main-red-fix` exception keeps the repository's CI contract: it substitutes composability for the main lookup -inside CI. A draft PR remains `publication_pending`; only a confirmed merge with -a merge commit reports `delivered`. Refresh status before acting on readiness. +inside CI. Run `--props-action ready` when status reports `ready_pending`. +It verifies the existing PASS comment, then refreshes the source and checks. +The result gives the maintainer the exact reviewed SHA to compare with the +current GitHub head before marking the draft ready. Repeating the command +returns the same comment on the same revision. The GitHub readiness mutation +accepts the PR identity and has no expected-head field. Status observes the +maintainer's transition and reports `ready_for_merge` while the exact gates +still pass. Merge requires separate authorization. A confirmed merge with a +merge commit reports `delivered`. + +If the local or remote head moves, status retains the submitted SHA and +original attempt and reports both observed heads. New review attempts record +the local HEAD reflog before and after the reviewer runs. A change in its +history holds the joined reply for reconciliation, even if the checkout has +returned to the submitted SHA. A disrupted primary attempt also holds before +it can select a replacement reviewer. Restore the submitted +revision for publication or readiness, or use `reconcile` to recover the +original attempt. Supply `--props-revision ` when selecting among +older attempts. A joined changes-required verdict with a verified comment +allows an approved repair to be submitted after fresh exact feedback and +green required CI. Submit the repair explicitly with its own SHA. The saved `status.json` is the last observation, not a second execution state. Use the command for a current answer. Local records are trusted workflow @@ -782,8 +825,8 @@ including whether existing APIs suffice. Mechanical validation and review bookkeeping stay with their existing workflow or coordinator owners. This guide adds no workflow phase or review stage. -Start required Fable reviews when the exact feedback commit and focused evidence -are ready, without waiting for CI on `main` or the reviewed branch. When Fable +Start final Fable review when the operator submits the exact draft with +completed internal feedback and green required PR CI. When Fable is unavailable, including exhausted Claude quota, use an independent Codex Astra (`gpt-6-astra`) reviewer under the same scope, effort policy and correction limits. Reconcile any in-flight call and preserve its findings before fallback. diff --git a/examples/adversarial-implementation/components/Delivery/Closeout.md b/examples/adversarial-implementation/components/Delivery/Closeout.md index 7afb51765..dada20555 100644 --- a/examples/adversarial-implementation/components/Delivery/Closeout.md +++ b/examples/adversarial-implementation/components/Delivery/Closeout.md @@ -21,6 +21,12 @@ Status: {status.status} Revision: {reportRevision} + +Local head: {status.local.head} + +Pull request head: {status.remote.observedRevision ?? status.remote.revision} + + Next action: {status.nextAction} @@ -30,6 +36,10 @@ Next action: {status.nextAction} Independent review: {status.review.result.verdict} + +Published review comment {status.reviewPublication.url} + + {status.review.result.summary} diff --git a/examples/adversarial-implementation/components/Delivery/Dispatch.md b/examples/adversarial-implementation/components/Delivery/Dispatch.md index b11fb96b1..f6eb38892 100644 --- a/examples/adversarial-implementation/components/Delivery/Dispatch.md +++ b/examples/adversarial-implementation/components/Delivery/Dispatch.md @@ -7,6 +7,7 @@ props: selection: {type: object} position: {type: integer, minimum: 0, maximum: 1} attempt: {type: string, pattern: '^/'} + branch: {type: string, minLength: 1} required: [delivery, selection, position, attempt] additionalProperties: false --- @@ -15,12 +16,24 @@ props: A release that fails ends the run and leaves the attempt reserved, so it holds for reconciliation and can never select a replacement. + + + + - + + + + + - + + + + + diff --git a/examples/adversarial-implementation/components/Delivery/Gates.md b/examples/adversarial-implementation/components/Delivery/Gates.md new file mode 100644 index 000000000..172c2ff6f --- /dev/null +++ b/examples/adversarial-implementation/components/Delivery/Gates.md @@ -0,0 +1,26 @@ +--- +description: 'Read exact feedback, checkout and GitHub gates for delivery. Use .' +props: + type: object + properties: + delivery: {type: object} + number: {type: integer, minimum: 1} + required: [delivery, number] + additionalProperties: false +returns: {type: object} +--- + + + + + + + +{text} + + + + + + + diff --git a/examples/adversarial-implementation/components/Delivery/Legacy.md b/examples/adversarial-implementation/components/Delivery/Legacy.md new file mode 100644 index 000000000..02a53d3a9 --- /dev/null +++ b/examples/adversarial-implementation/components/Delivery/Legacy.md @@ -0,0 +1,18 @@ +--- +description: 'Find original attempts available for recovery. Use .' +props: + type: object + properties: + delivery: {type: object} + required: [delivery] + additionalProperties: false +returns: {type: array} +--- + + +```bash exec as="listing" +printf '%s\n' review-* +``` + + + /^review-[0-9a-f]{40}$/.test(path)).map(path => path.slice(7))} /> diff --git a/examples/adversarial-implementation/components/Delivery/Pin.md b/examples/adversarial-implementation/components/Delivery/Pin.md new file mode 100644 index 000000000..7e0df00ce --- /dev/null +++ b/examples/adversarial-implementation/components/Delivery/Pin.md @@ -0,0 +1,45 @@ +--- +description: 'Read the admitted candidate and its successor slot. Use .' +props: + type: object + properties: + delivery: {type: object} + number: {type: integer, minimum: 1} + required: [delivery, number] + additionalProperties: false +returns: {type: object} +--- + + + + + + + + + + + + + +{text} + + + + + + + + + + + + + + + + +```bash exec as="occupied" +test -e {quoted} -o -L {quoted} +``` + diff --git a/examples/adversarial-implementation/components/Delivery/Ready.md b/examples/adversarial-implementation/components/Delivery/Ready.md new file mode 100644 index 000000000..a44a0ab27 --- /dev/null +++ b/examples/adversarial-implementation/components/Delivery/Ready.md @@ -0,0 +1,15 @@ +--- +description: 'Verify the exact reviewed draft for maintainer readiness. Use .' +props: + type: object + properties: + delivery: {type: object} + number: {type: integer, minimum: 1} + required: [delivery, number] + additionalProperties: false +returns: {type: object} +--- + + + + diff --git a/examples/adversarial-implementation/components/Delivery/Reconcile.md b/examples/adversarial-implementation/components/Delivery/Reconcile.md index bf5daaf8a..c08ba36b8 100644 --- a/examples/adversarial-implementation/components/Delivery/Reconcile.md +++ b/examples/adversarial-implementation/components/Delivery/Reconcile.md @@ -26,11 +26,23 @@ completion turn can supply the review. {assignmentText} + + + + + + + +{beforeText} +{afterText} + + + + {executionText} {outcomeText} - {returned.turn.reply} @@ -50,8 +62,10 @@ completion turn can supply the review. + + - + diff --git a/examples/adversarial-implementation/components/Delivery/Remote.md b/examples/adversarial-implementation/components/Delivery/Remote.md index 5bab60570..1438dfeb4 100644 --- a/examples/adversarial-implementation/components/Delivery/Remote.md +++ b/examples/adversarial-implementation/components/Delivery/Remote.md @@ -58,4 +58,4 @@ gh api repos/{props.repository}/pulls/{props.number} {mainAgainRead.stdout} {prAgainRead.stdout} - label.name === "ci-main-red-fix")}} /> + label.name === "ci-main-red-fix")}} /> diff --git a/examples/adversarial-implementation/components/Delivery/Review.md b/examples/adversarial-implementation/components/Delivery/Review.md index daae17bd9..1e274622b 100644 --- a/examples/adversarial-implementation/components/Delivery/Review.md +++ b/examples/adversarial-implementation/components/Delivery/Review.md @@ -5,6 +5,7 @@ props: properties: delivery: {type: object} revision: {type: string, pattern: '^[0-9a-f]{40}$'} + branch: {type: string, minLength: 1} required: [delivery, revision] additionalProperties: false returns: {type: object} @@ -26,7 +27,7 @@ mkdir -- {quoted} ``` - + @@ -37,7 +38,7 @@ mkdir -- {quotedReplacement} ``` - + diff --git a/examples/adversarial-implementation/components/Delivery/ReviewComment.ts b/examples/adversarial-implementation/components/Delivery/ReviewComment.ts new file mode 100644 index 000000000..4fff38377 --- /dev/null +++ b/examples/adversarial-implementation/components/Delivery/ReviewComment.ts @@ -0,0 +1,256 @@ +import { rm } from "@effectionx/fs"; +import { API, cwd, exec, parseFilesFailure } from "@executablemd/runtime"; +import { ensure, scoped, type Operation } from "effection"; +import { z } from "zod"; +import { createDirectory, digest } from "../Ticket/native-node.ts"; +import { clear, write } from "../Ticket/text-storage.ts"; +import { reviewerSchema } from "./records.ts"; + +export const props = { + type: "object", + properties: { + target: { type: "object" }, + review: { type: "object" }, + publish: { type: "boolean" }, + allowPost: { type: "boolean" }, + comments: { type: "array" }, + }, + required: ["target", "review", "publish", "allowPost", "comments"], + additionalProperties: false, +}; + +const sha = z.string().regex(/^[0-9a-f]{40}$/); +const targetSchema = z.strictObject({ + directory: z.string().startsWith("/"), + repository: z.string().regex(/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/), + number: z.number().int().positive().max(Number.MAX_SAFE_INTEGER), + branch: z.string().min(1), + revision: sha, +}); +const input = z.strictObject({ + target: targetSchema, + review: z.object({ + reviewer: reviewerSchema, + model: z.string().min(1).nullable(), + result: z.object({ + revision: sha, + verdict: z.enum(["pass", "changes_required", "blocked"]), + summary: z.string().min(1).max(1500), + findings: z.array(z.object({ blocking: z.boolean(), description: z.string().min(1) })), + }), + }), + publish: z.boolean(), + allowPost: z.boolean(), + comments: z.array( + z.object({ + kind: z.enum(["conversation", "review"]), + id: z.string(), + url: z.string(), + body: z.string(), + }), + ), +}); +const pulls = z.object({ + number: targetSchema.shape.number, + base: z.object({ repo: z.object({ full_name: z.string() }) }), + head: z.object({ ref: z.string(), sha }), +}); +const saved = z.strictObject({ + revision: sha, + bodySha256: z.string().regex(/^[0-9a-f]{64}$/), + id: z.number().int().positive().max(Number.MAX_SAFE_INTEGER), + url: z.string(), +}); + +type Target = z.infer; +type Review = z.infer["review"]; +type Verified = z.infer; + +function pending(reason: string) { + return { state: "pending", reason }; +} + +function quote(text: string) { + return text + .replace(/\r\n?/g, "\n") + .split("\n") + .map((line) => " " + line) + .join("\n"); +} + +function render(marker: string, review: Review, model: string) { + const { revision, verdict, summary, findings } = review.result; + const titles = { pass: "PASS", changes_required: "CHANGES REQUIRED", blocked: "BLOCKED" }; + return [ + marker, + "## Independent review", + "Reviewed commit `" + revision + "`", + "Verdict **" + titles[verdict] + "**", + "Reviewer\n\n" + quote(review.reviewer.provider + " / " + model), + "### Summary\n\n" + quote(summary), + "### Findings", + ...(findings.length === 0 + ? ["None."] + : findings.map( + ({ blocking, description: findingText }) => + (blocking ? "**Blocking**" : "**Non-blocking**") + "\n\n" + quote(findingText), + )), + ].join("\n\n"); +} + +function* github(...args: string[]) { + return yield* exec({ + command: ["gh", "api", "--hostname", "github.com", ...args], + cwd: yield* cwd(), + timeout: 30_000, + }); +} + +function* readTarget(target: Target) { + const { exitCode, stdout } = yield* github(`repos/${target.repository}/pulls/${target.number}`); + if (exitCode !== 0) { + return false; + } + const pr = pulls.parse(JSON.parse(stdout)); + return ( + pr.number === target.number && + pr.base.repo.full_name.toLowerCase() === target.repository.toLowerCase() && + pr.head.ref === target.branch && + pr.head.sha === target.revision + ); +} + +function findComment( + target: Target, + comments: z.infer["comments"], + marker: string, + body: string, +): Verified | null { + const matches = comments.filter( + (item) => item.kind === "conversation" && item.body.split("\n", 1)[0] === marker, + ); + if (matches.length > 1) { + throw new Error("Multiple GitHub comments claim the same review revision."); + } + const [value] = matches; + if (value === undefined) { + return null; + } + const { repository, number, revision } = target; + const id = Number(value.id); + const urls = ["issues", "pull"].map((kind) => + `https://github.com/${repository}/${kind}/${number}#issuecomment-${id}`.toLowerCase(), + ); + if ( + !Number.isSafeInteger(id) || + id <= 0 || + value.id !== String(id) || + value.body !== body || + !urls.includes(value.url.toLowerCase()) + ) { + throw new Error("The GitHub comment does not match the saved review."); + } + return { revision, bodySha256: digest(new TextEncoder().encode(body)), id, url: value.url }; +} + +function* readSaved(directory: string) { + const result = yield* API.Files.operations.readTextFile({ cwd: directory, path: "comment.json" }); + if (result.ok) { + return saved.parse(JSON.parse(result.value)); + } + if (parseFilesFailure(result.error)?.reason === "missing") { + return null; + } + throw result.error; +} + +function* post(target: Target, reservation: string, body: string): Operation { + try { + // A reservation survives interruption so a retry can adopt the remote comment. + yield* createDirectory(reservation); + } catch (error) { + if (z.object({ code: z.literal("EEXIST") }).safeParse(error).success) { + return; + } + throw error; + } + yield* scoped(function* () { + let release = true; + yield* ensure(function* () { + if (release) { + yield* rm(reservation, { recursive: true, force: true }); + } + }); + const payload = reservation + "/request.json"; + yield* write(target.directory, payload, JSON.stringify({ body })); + if (!(yield* readTarget(target))) { + return; + } + release = false; + const { exitCode, stdout } = yield* github( + "--include", + "--method", + "POST", + `repos/${target.repository}/issues/${target.number}/comments`, + "--input", + payload, + ); + // An explicit rejection permits another attempt. Transport and server errors leave the reservation in place. + release = exitCode !== 0 && /^HTTP\/\d(?:\.\d)? (400|401|403|404|422|429)\b/.test(stdout); + }); +} + +export default function* ReviewComment(value: unknown) { + const { target, review, publish, allowPost, comments } = input.parse(value); + if (review.result.revision !== target.revision) { + return pending("The retained review names a different revision."); + } + const marker = ``; + const model = review.model ?? review.reviewer.model; + const body = render(marker, review, model); + const record = `${target.directory}/review-${target.revision}/publication-${target.number}`; + try { + const previous = yield* readSaved(record); + if (!publish && previous === null) { + return pending("Publish the joined review and verify the GitHub comment."); + } + if (body.length > 65_536) { + return pending("The rendered review exceeds the GitHub comment limit."); + } + yield* clear( + [ + model, + review.result.summary, + ...review.result.findings.map((finding) => finding.description), + ].join("\n"), + ); + yield* clear(body); + if (publish && !(yield* readTarget(target))) { + return pending("The current pull request does not match the reviewed revision."); + } + + const observed = findComment(target, comments, marker, body); + if (observed === null && publish && allowPost && previous === null) { + yield* post(target, record, body); + return { state: "readback" }; + } + if (observed === null) { + return pending( + "The current review comment could not be verified. Retry publication from the saved receipt.", + ); + } + if (previous !== null && JSON.stringify(previous) !== JSON.stringify(observed)) { + return pending("The GitHub comment differs from its recorded identity."); + } + if (publish && previous === null) { + // A matching comment may already exist even if no local request survived. + yield* API.Fs.operations.ensureDir(record); + yield* write(target.directory, record + "/comment.json", JSON.stringify(observed)); + } + return { state: "verified", ...observed }; + } catch { + return pending( + "The current review comment could not be verified. Check publication access and reconcile the saved receipt.", + ); + } +} diff --git a/examples/adversarial-implementation/components/Delivery/ReviewPublication.md b/examples/adversarial-implementation/components/Delivery/ReviewPublication.md new file mode 100644 index 000000000..29e5e522b --- /dev/null +++ b/examples/adversarial-implementation/components/Delivery/ReviewPublication.md @@ -0,0 +1,26 @@ +--- +description: 'Verify a saved review comment or publish it from its validated receipt. Use .' +props: + type: object + properties: + target: {type: object} + review: {type: object} + publish: {type: boolean} + required: [target, review, publish] + additionalProperties: false +returns: {type: object} +--- + + + + + + + + + + + + + + diff --git a/examples/adversarial-implementation/components/Delivery/ReviewState.ts b/examples/adversarial-implementation/components/Delivery/ReviewState.ts index 11ba99468..c193bd092 100644 --- a/examples/adversarial-implementation/components/Delivery/ReviewState.ts +++ b/examples/adversarial-implementation/components/Delivery/ReviewState.ts @@ -34,6 +34,13 @@ type Selection = z.infer; type Reviewer = z.infer; type Attempt = z.infer; +const sourceObservation = z.object({ + head: z.string(), + branch: z.string().nullable(), + clean: z.literal(true), + headHistory: z.string().regex(/^[0-9a-f]{40}$/), +}); + function* read(cwd: string, path: string): Operation { const result = yield* API.Files.operations.readTextFile({ cwd, path }); if (result.ok) { @@ -65,6 +72,36 @@ function same(left: unknown, right: unknown): boolean { return JSON.stringify(left) === JSON.stringify(right); } +function* holdMovedPrimary( + directory: string, + revision: string, + primary: Attempt, +): Operation { + const beforePath = "source-before.json"; + const afterPath = "source-after.json"; + const beforeText = yield* read(directory, beforePath); + const afterText = yield* read(directory, afterPath); + if (beforeText === null && afterText === null) { + // Attempts created before source tracking have no observations. + return primary; + } + const before = parsed(sourceObservation, beforeText); + const after = parsed(sourceObservation, afterText); + if (before?.head === revision && same(before, after)) { + return primary; + } + return { + ...primary, + state: "held", + reason: "The source observations do not prove a stable checkout during the primary review.", + evidence: [ + ...primary.evidence, + ...(beforeText === null ? [] : [directory + "/" + beforePath]), + ...(afterText === null ? [] : [directory + "/" + afterPath]), + ], + }; +} + type Outcome = z.infer; function isFableUsageLimit( @@ -238,13 +275,16 @@ function* observe( if (first === undefined) { return { ...empty, next: "uncertain", hold: null }; } - const primary = yield* attempt( + let primary = yield* attempt( directory, 0, first.reviewer, assignmentFor(expected, first.reviewer), expected.chain === "fable-astra" ? first.choices : null, ); + if (second !== undefined && primary.state === "eligible") { + primary = yield* holdMovedPrimary(directory, expected.revision, primary); + } const attempts: Attempt[] = [primary]; let candidate: { position: number; directory: string; reviewer: Reviewer } | null = { position: 0, diff --git a/examples/adversarial-implementation/components/Delivery/Source.md b/examples/adversarial-implementation/components/Delivery/Source.md new file mode 100644 index 000000000..68821d147 --- /dev/null +++ b/examples/adversarial-implementation/components/Delivery/Source.md @@ -0,0 +1,44 @@ +--- +description: 'Observe the current local branch, commit and worktree. Use .' +props: + type: object + properties: + repository: {type: string, pattern: '^/'} + trackHead: {type: boolean, default: false} + required: [repository] + additionalProperties: false +returns: {type: object} +--- + + +```bash exec as="worktree" +git --no-optional-locks status --porcelain=v2 --branch --untracked-files=normal --ignore-submodules=none +``` + + + + + + line.startsWith(headPrefix))?.slice(headPrefix.length) ?? null} /> + line.startsWith(branchPrefix))?.slice(branchPrefix.length) ?? null} /> + + +```bash exec as="symbolic" +git symbolic-ref --quiet --short HEAD +``` + + + + line.startsWith("#"))}} /> + + +```bash exec as="history" +set -e -o pipefail +git reflog exists HEAD +git reflog show --no-color --no-notes --no-show-signature --format='%H %gD %gs' --date=raw HEAD | git hash-object --stdin +``` + + + + + diff --git a/examples/adversarial-implementation/components/Delivery/Status.md b/examples/adversarial-implementation/components/Delivery/Status.md index 5db424e04..36754a2e8 100644 --- a/examples/adversarial-implementation/components/Delivery/Status.md +++ b/examples/adversarial-implementation/components/Delivery/Status.md @@ -5,26 +5,37 @@ props: properties: delivery: {type: object} number: {type: integer, minimum: 1} + revision: {type: string, pattern: '^[0-9a-f]{40}$'} required: [delivery, number] additionalProperties: false returns: {type: object} --- - + + + + + + - - + + ({gate,state:"pending"}))} /> + - -{feedbackText} - - - + + + + + + + + + attempt + "/" + file)} /> @@ -34,7 +45,8 @@ returns: {type: object} -{reading.output} +{reading.output} + @@ -42,22 +54,51 @@ returns: {type: object} - - - -```bash exec as="branchRead" -git symbolic-ref --quiet --short HEAD -``` - - + - - - + + + + + + + + + + + + + + + + + + gate.state !== "passed")} /> - - gate.state === "failed") ? "checks_failed" : remaining.length !== 0 ? "checks_pending" : remote.merged && /^[0-9a-f]{40}$/.test(remote.mergeRevision) ? "delivered" : remote.state !== "open" ? "closed_unmerged" : remote.draft ? "publication_pending" : "ready_for_merge"} /> - - + + + 1}> + + + + + + + + + + + + + + + + + + + + + + diff --git a/examples/adversarial-implementation/components/Delivery/Submit.md b/examples/adversarial-implementation/components/Delivery/Submit.md new file mode 100644 index 000000000..70367c222 --- /dev/null +++ b/examples/adversarial-implementation/components/Delivery/Submit.md @@ -0,0 +1,66 @@ +--- +description: 'Submit one exact draft candidate for independent review. Use .' +props: + type: object + properties: + delivery: {type: object} + number: {type: integer, minimum: 1} + revision: {type: string, pattern: '^[0-9a-f]{40}$'} + required: [delivery, number, revision] + additionalProperties: false +returns: {type: object} +--- + + + + + + + + + + + + + + + + + + + +```bash exec as="reserved" +mkdir -- {quoted} +``` + + + + + + + + + + + + + + +{execution.output} + + + + + + + + + + + + + + + + + diff --git a/examples/adversarial-implementation/delivery.md b/examples/adversarial-implementation/delivery.md index ea366dadc..74d7ecfe4 100644 --- a/examples/adversarial-implementation/delivery.md +++ b/examples/adversarial-implementation/delivery.md @@ -4,32 +4,57 @@ props: properties: record: {type: string, minLength: 1} number: {type: integer, minimum: 1} - action: {enum: [status, review, reconcile, closeout], default: status} + action: {enum: [status, submit, review, reconcile, publish-review, ready, closeout], default: status} + revision: {type: string, pattern: '^[0-9a-f]{40}$'} required: [record, number] additionalProperties: false --- -Read the original delivery record and perform only its computed next action. -A status read starts no agent. An explicit progress reply gets one continuation -in the same conversation. An unavailable Fable reviewer is replaced once by -Codex Astra. The workflow preserves each attempt and asks about a review it -cannot complete. Readiness permits no merge. +Submit the exact draft candidate after internal feedback and green required CI. +Status reads the pinned review and current heads. Recovery reads the original +attempt. Ready verifies the passing review, comment and current checks and +reports the exact draft revision for the maintainer to mark ready. {recordText} - + + + + + + + + + + + + + - - - + + + - - - + + + + + + + + + + + + + + + + diff --git a/examples/adversarial-implementation/tests/Delivery.test.md b/examples/adversarial-implementation/tests/Delivery.test.md index 364cbd7b9..2c87275e2 100644 --- a/examples/adversarial-implementation/tests/Delivery.test.md +++ b/examples/adversarial-implementation/tests/Delivery.test.md @@ -35,25 +35,26 @@ printf '%s' "$XMD_EXAMPLE_COMMAND" - + + - -```bash exec as="identity" -git rev-parse HEAD -git symbolic-ref --short HEAD -``` - - - - - - - - - - - + + + + + + + + + + + + + + + + diff --git a/examples/adversarial-implementation/tests/DeliveryCase.md b/examples/adversarial-implementation/tests/DeliveryCase.md index 15460023b..d7d571d02 100644 --- a/examples/adversarial-implementation/tests/DeliveryCase.md +++ b/examples/adversarial-implementation/tests/DeliveryCase.md @@ -92,7 +92,7 @@ git symbolic-ref --short HEAD {change:{stable:false},status:"source_changed"}, {change:{revision:"0".repeat(40)},status:"source_changed"}, {change:{branch:"another-branch"},status:"source_changed"}, - {change:{draft:true},status:"publication_pending"}, + {change:{draft:true},status:"review_pending"}, {change:{state:"closed",merged:true,mergeRevision:"1".repeat(40)},status:"delivered"} ]} let="change"> @@ -108,6 +108,28 @@ git symbolic-ref --short HEAD + +{originalReceipt} + + + + + + + + + +```bash exec as="adoptedEntry" +{input.runner} {entryArguments} --props-action submit --props-revision {revision} +``` + + +{adoptedEntry.stdout} + + + + + A changed source after review. ```bash exec as="changedSource" diff --git a/examples/adversarial-implementation/tests/DeliveryPublication.test.md b/examples/adversarial-implementation/tests/DeliveryPublication.test.md new file mode 100644 index 000000000..dc23bd130 --- /dev/null +++ b/examples/adversarial-implementation/tests/DeliveryPublication.test.md @@ -0,0 +1,32 @@ +```bash exec as="location" +pwd -P +``` + +```bash exec as="commandSetting" +printf '%s' "$XMD_EXAMPLE_COMMAND" +``` + + + + + + + +```bash exec as="identity" +git rev-parse HEAD +git symbolic-ref --short HEAD +``` + + + + + + + + + + + + + + diff --git a/examples/adversarial-implementation/tests/acp-overlap/github-comments.ts b/examples/adversarial-implementation/tests/acp-overlap/github-comments.ts new file mode 100644 index 000000000..712f1e5f4 --- /dev/null +++ b/examples/adversarial-implementation/tests/acp-overlap/github-comments.ts @@ -0,0 +1,97 @@ +import { exists, readTextFile, writeTextFile } from "@effectionx/fs"; +import { PullRequestAPI } from "@executablemd/git/api"; +import { API } from "@executablemd/runtime"; +import { z } from "zod"; +import { waitForFile } from "./receipts.ts"; + +const repository = "taras/executable.md"; +const pr = `repos/${repository}/pulls/7`; +const comments = `repos/${repository}/issues/7/comments`; +const url = `https://github.com/${repository}/pull/7`; +const evidence = z.strictObject({ + kind: z.literal("conversation"), + id: z.string(), + url: z.string(), + author: z.string(), + body: z.string(), + createdAt: z.string(), + updatedAt: z.string(), +}); +const reply = (stdout: string, exitCode = 0) => ({ stdout, stderr: "", exitCode }); + +export function* installGitHub(directory: string, checkout: string, actor: string) { + let lost = false; + const remote = z.object({ revision: z.string(), branch: z.string() }); + const payload = z.strictObject({ body: z.string() }); + const tracked = `${directory}/comment.json`; + + yield* PullRequestAPI.around({ + *read([target, options]) { + if (target !== url || options.kind !== "comments") { + throw new Error("Unexpected pull request evidence read."); + } + if (lost || (yield* exists(`${directory}/comments-unavailable`))) { + throw new Error("The pull request comments are unavailable."); + } + const items = (yield* exists(tracked)) + ? [evidence.parse(JSON.parse(yield* readTextFile(tracked)))] + : []; + return { kind: "comments", items }; + }, + }); + yield* API.Process.around({ + *exec([options], next) { + const [tool, ...args] = options.command; + if (tool !== "gh") { + return yield* next(options); + } + if (args.slice(0, 3).join(" ") !== "api --hostname github.com") { + throw new Error("Unexpected GitHub CLI boundary."); + } + const rest = args.slice(3); + if (rest.join(" ") === pr) { + const { revision, branch } = remote.parse( + JSON.parse(yield* readTextFile(`${checkout}/remote.json`)), + ); + const stale = yield* exists(`${directory}/stale`); + return reply( + JSON.stringify({ + number: 7, + base: { repo: { full_name: repository } }, + head: { ref: branch, sha: stale ? "0".repeat(40) : revision }, + }), + ); + } + if ( + rest.length !== 6 || + rest.slice(0, 5).join(" ") !== `--include --method POST ${comments} --input` + ) { + throw new Error("Unexpected GitHub comment command."); + } + yield* writeTextFile(`${directory}/${actor}-posted`, ""); + const body = payload.parse(JSON.parse(yield* readTextFile(rest[5]))).body; + const delayed = yield* exists(`${directory}/delay-post`); + if (delayed && actor === "winner") { + yield* writeTextFile(`${directory}/post-ready`, ""); + if ((yield* waitForFile(`${directory}/release-post`, 45_000)).timeout) { + throw new Error("Timed out waiting to release the GitHub write."); + } + } + if (yield* exists(tracked)) { + throw new Error("GitHub received a duplicate review write."); + } + const saved = evidence.parse({ + kind: "conversation", + id: "42", + url: `${url}#issuecomment-42`, + author: "workflow", + createdAt: "2026-11-15T00:00:00Z", + updatedAt: "2026-11-15T00:00:00Z", + body, + }); + yield* writeTextFile(tracked, JSON.stringify(saved)); + lost = delayed; + return delayed ? reply("", 1) : reply("HTTP/2.0 201 Created\r\n\r\n" + JSON.stringify(saved)); + }, + }); +} diff --git a/examples/adversarial-implementation/tests/acp-overlap/local-workers.ts b/examples/adversarial-implementation/tests/acp-overlap/local-workers.ts index 2b826cf95..2361ec425 100644 --- a/examples/adversarial-implementation/tests/acp-overlap/local-workers.ts +++ b/examples/adversarial-implementation/tests/acp-overlap/local-workers.ts @@ -1,17 +1,18 @@ import { createAcpRuntime, createAcpxProvider, createAcpxSessionStore } from "@executablemd/acp"; import { AgentProviders, registerAgentProvider } from "@executablemd/core"; import { Component, Plugin } from "@executablemd/core/api"; -import { API, command, env, readTextFile } from "@executablemd/runtime"; +import { API, command, cwd, env, readTextFile } from "@executablemd/runtime"; import { useTestAgentController } from "@executablemd/test-agent"; import { writeTextFile } from "@effectionx/fs"; import { ensure } from "effection"; +import { installGitHub } from "./github-comments.ts"; import { actorEnv, actorSchema, fableFailure, overlapEnv, waitForFile } from "./receipts.ts"; import type { PromptReceipt } from "./receipts.ts"; export default Plugin({ name: "delivery-acp-overlap", - *install(request) { - if (request.command !== "run") { + *install(invocation) { + if (invocation.command !== "run") { throw new Error("Delivery ACP workers require a run invocation."); } const directory = yield* env(overlapEnv); @@ -19,6 +20,7 @@ export default Plugin({ if (!directory?.startsWith("/")) { throw new Error("Delivery ACP workers require an absolute fixture directory."); } + yield* installGitHub(directory, yield* cwd(), actor); const receipts: PromptReceipt[] = []; yield* ensure(() => writeTextFile(`${directory}/${actor}-prompts.json`, JSON.stringify(receipts)), @@ -89,10 +91,10 @@ export default Plugin({ return [...worker, "--connect", scenario.route]; }, }, - createRuntime(options) { + createRuntime(runtimeOptions) { const pending = new Map(); return createAcpRuntime({ - ...options, + ...runtimeOptions, onAcpMessage(direction, message) { if ( direction === "outbound" && diff --git a/examples/adversarial-implementation/tests/components/Delivery/Remote.md b/examples/adversarial-implementation/tests/components/Delivery/Remote.md index 326c8f2b9..26fc01fec 100644 --- a/examples/adversarial-implementation/tests/components/Delivery/Remote.md +++ b/examples/adversarial-implementation/tests/components/Delivery/Remote.md @@ -4,4 +4,10 @@ returns: {type: object} --- {text} - + + + +{next} + + + diff --git a/examples/adversarial-implementation/tests/components/Delivery/ReviewPublication.md b/examples/adversarial-implementation/tests/components/Delivery/ReviewPublication.md new file mode 100644 index 000000000..a35e56ce4 --- /dev/null +++ b/examples/adversarial-implementation/tests/components/Delivery/ReviewPublication.md @@ -0,0 +1,31 @@ +--- +props: {type: object, properties: {target: {type: object}, review: {type: object}, publish: {type: boolean}}, required: [target, review, publish], additionalProperties: false} +returns: {type: object} +--- + + + + + +{remoteText} + + + + + + + + + + + + + + + + + + + +{moved} + diff --git a/examples/adversarial-implementation/tests/components/Delivery/Role.md b/examples/adversarial-implementation/tests/components/Delivery/Role.md index fd2943828..13d0e91a7 100644 --- a/examples/adversarial-implementation/tests/components/Delivery/Role.md +++ b/examples/adversarial-implementation/tests/components/Delivery/Role.md @@ -3,10 +3,15 @@ props: {type: object, properties: {delivery: {type: object}, revision: {type: st --- - + - - clause.id),findings:["repair","false-pass"].includes(work) ? [{blocking:true,description:"A blocking regression remains."}] : []}} /> + + + + + + + clause.id),findings:(repairRequested || ["repair","false-pass"].includes(work)) ? [{blocking:true,description:"A blocking regression remains."}] : []}} /> @@ -23,6 +28,17 @@ props: {type: object, properties: {delivery: {type: object}, revision: {type: st {calledText} + + +```bash exec as="joinedGate" +for attempt in $(seq 1 400); do + if ! test -e {gate}/hold || test -e {gate}/release; then exit 0; fi + sleep 0.05 +done +exit 1 +``` + + diff --git a/examples/adversarial-implementation/tests/components/Fixture/DeliveryJourney.ts b/examples/adversarial-implementation/tests/components/Fixture/DeliveryJourney.ts new file mode 100644 index 000000000..37a1668ed --- /dev/null +++ b/examples/adversarial-implementation/tests/components/Fixture/DeliveryJourney.ts @@ -0,0 +1,436 @@ +import { ensureDir, exists, readTextFile, rm, writeTextFile } from "@effectionx/fs"; +import { runShell, shellQuote } from "@executablemd/test-support/launch"; +import { ensure, race, scoped, spawn } from "effection"; +import { z } from "zod"; +import { waitForFile } from "../../acp-overlap/receipts.ts"; + +export const props = { + type: "object", + properties: { + fixture: { type: "object" }, + command: { type: "string" }, + root: { type: "string" }, + mode: { enum: ["repair", "remote", "local", "roundtrip", "fallback-roundtrip"] }, + }, + required: ["fixture", "command", "root", "mode"], + additionalProperties: false, +}; + +const input = z.object({ + fixture: z.object({ repository: z.string(), directory: z.string() }), + command: z.string(), + root: z.string(), + mode: z.enum(["repair", "remote", "local", "roundtrip", "fallback-roundtrip"]), +}); + +const statusSchema = z.object({ + revision: z.string().nullable().optional(), + status: z.string(), + nextAction: z.string(), + pin: z.object({ revision: z.string(), previous: z.string().nullable() }).nullable(), + local: z.object({ head: z.string() }).optional(), + remote: z.object({ revision: z.string(), draft: z.boolean() }).optional(), + review: z + .object({ result: z.object({ verdict: z.string() }) }) + .nullable() + .optional(), + reviewPublication: z + .object({ state: z.string(), url: z.string().optional() }) + .nullable() + .optional(), +}); + +function check(condition: boolean, message: string): asserts condition { + if (!condition) { + throw new Error(message); + } +} + +export default function* DeliveryJourney(value: unknown) { + const { fixture, command, root, mode } = input.parse(value); + const fallback = mode === "fallback-roundtrip"; + const cwd = fixture.repository; + const directory = fixture.directory + "/delivery"; + const example = root + "/examples/adversarial-implementation"; + const launch = (script: string) => runShell(script, { cwd, timeout: 20_000 }).join(); + + function* shell(script: string) { + const result = yield* launch(script); + check(result.code === 0, result.stderr + result.stdout); + return result.stdout.trim(); + } + + const args = [ + example + "/delivery.md", + "--props-record", + directory + "/requirements.json", + "--props-number", + "7", + "--raw", + "--include", + example + "/tests/components", + "--include", + example + "/components", + ] + .map(shellQuote) + .join(" "); + const entry = (action: string, revision?: string) => + launch( + `export PATH="$PWD/bin:$PATH"; exec ${command} ${args} --props-action ${shellQuote(action)}${revision ? " --props-revision " + shellQuote(revision) : ""}`, + ); + + function* observe(action = "status", revision?: string) { + const result = yield* entry(action, revision); + check(result.code === 0, result.stderr + result.stdout); + return statusSchema.parse(JSON.parse(result.stdout)); + } + + function* refuse(action: string, revision?: string) { + const result = yield* entry(action, revision); + check(result.code !== 0, `${action} unexpectedly succeeded: ${result.stdout}`); + } + + function* json(path: string, data: unknown) { + yield* writeTextFile(path, JSON.stringify(data)); + } + + const called = directory + "/called.json"; + function* prompts() { + if (!(yield* exists(called))) { + return []; + } + return z + .array(z.object({ revision: z.string(), prompted: z.literal(true) })) + .parse(JSON.parse(yield* readTextFile(called))) + .map((call) => call.revision); + } + + yield* ensureDir(directory); + yield* ensureDir(cwd + "/bin"); + yield* writeTextFile( + cwd + "/bin/gh", + `#!/bin/bash +set -eu +if [[ "$*" != "pr ready 7 --repo taras/executable.md" ]]; then exit 70; fi +printf 'ready\\n' >> ready-count.txt +exit 1 +`, + ); + yield* shell("chmod +x bin/gh"); + const branch = yield* shell("git symbolic-ref --short HEAD"); + const base = yield* shell("git rev-parse HEAD"); + const initial = yield* shell("git commit --allow-empty -qm interim && git rev-parse HEAD"); + const green = { status: "completed", conclusion: "success" }; + const remote = { + revision: initial, + branch, + base: "main", + state: "open", + draft: true, + stable: true, + merged: false, + mergeRevision: null, + ciGreen: true, + ci: green, + mainRepair: false, + main: { revision: base, run: green }, + }; + const remotePath = cwd + "/remote.json"; + const feedback = (revision: string) => + json(directory + "/feedback.json", { status: "feedback_ready", revision }); + const readyCount = cwd + "/ready-count.txt"; + yield* writeTextFile(directory + "/journey", ""); + yield* json(directory + "/requirements.json", { + work: fallback ? "fallback-roundtrip" : "delivery-journey", + repository: cwd, + directory, + policy: { + repository: "taras/executable.md", + reviewer: { + provider: "claude", + model: fallback ? "fable" : "fixture", + effort: null, + session: "independent", + }, + acceptance: [{ id: "behavior", source: "accepted.md", requirement: "Keep the behavior." }], + }, + inputs: [], + required: ["independent-review", "required-ci", "main-health"], + }); + + yield* json(remotePath, remote); + if (mode === "repair") { + yield* refuse("ready"); + yield* refuse("submit", initial); + yield* feedback(base); + yield* refuse("submit", initial); + yield* feedback(initial); + for (const ci of [ + { status: "in_progress", conclusion: null }, + { status: "completed", conclusion: "failure" }, + ]) { + yield* json(remotePath, { ...remote, ci }); + yield* refuse("submit", initial); + } + yield* json(remotePath, { ...remote, revision: base }); + yield* refuse("submit", initial); + yield* json(remotePath, remote); + yield* writeTextFile(cwd + "/untracked.fixture", "uncommitted source"); + yield* refuse("submit", initial); + yield* rm(cwd + "/untracked.fixture"); + check((yield* observe()).nextAction === "submit", "Green retry must await submission."); + yield* refuse("ready"); + check( + (yield* prompts()).length === 0 && !(yield* exists(readyCount)), + "An interim action dispatched or marked ready.", + ); + } else { + yield* feedback(initial); + } + + if (mode !== "repair") { + const source = cwd + "/legacy.fixture"; + const submittedText = yield* readTextFile(source); + yield* writeTextFile(source, "Different source during review.\n"); + const newer = yield* shell( + "git add -- legacy.fixture && git commit -qm moved && git rev-parse HEAD", + ); + yield* shell(`git reset --hard ${shellQuote(initial)}`); + const release = directory + "/release"; + yield* writeTextFile(directory + "/hold", ""); + const result = yield* scoped(function* () { + const first = yield* spawn(() => entry("submit", initial)); + yield* ensure(function* () { + yield* writeTextFile(release, ""); + if (!fallback) { + yield* first; + } + }); + const held = yield* race([ + waitForFile(called, 12_000), + (function* () { + throw new Error( + "Submission completed before reaching the held reviewer: " + + JSON.stringify(yield* first), + ); + })(), + ]); + check(!held.timeout, "Initial submission did not reach the reviewer."); + if (mode === "roundtrip" || fallback) { + yield* shell(`git reset --hard ${shellQuote(newer)}`); + check( + (yield* readTextFile(source)) !== submittedText, + "The live reviewer had no different source to read.", + ); + yield* shell(`git reset --hard ${shellQuote(initial)}`); + check( + (yield* readTextFile(source)) === submittedText, + "The original tree was not restored before join.", + ); + } else { + yield* refuse("submit", initial); + if (mode === "local") { + yield* shell(`git reset --hard ${shellQuote(newer)}`); + yield* feedback(newer); + } + yield* json(remotePath, { ...remote, revision: newer }); + yield* refuse("submit", newer); + yield* refuse("ready"); + } + yield* writeTextFile(release, ""); + if (fallback) { + const recorded = yield* race([ + waitForFile(directory + "/review-" + initial + "/hold.json", 12_000), + (function* () { + throw new Error( + "The moved primary completed without a hold: " + JSON.stringify(yield* first), + ); + })(), + ]); + check(!recorded.timeout, "The moved primary did not record its hold."); + yield* first.halt(); + return null; + } + return yield* first; + }); + if (fallback) { + const retained = yield* observe(); + check( + retained.status === "review_held" && + retained.pin?.revision === initial && + retained.review === null, + "The moved primary did not hold against its submitted SHA.", + ); + const attempt = directory + "/review-" + initial; + const hold = z + .object({ attempts: z.array(z.object({ reason: z.string() })) }) + .parse(JSON.parse(yield* readTextFile(attempt + "/hold.json"))); + check( + hold.attempts[0]?.reason.includes("stable checkout") === true, + "The hold did not identify the missing stable source evidence.", + ); + yield* refuse("submit", initial); + yield* refuse("publish-review"); + yield* refuse("ready"); + check((yield* prompts()).join() === initial, "The moved attempt prompted a replacement."); + for (const path of [ + attempt + "/replacement", + attempt + "/receipt.json", + directory + "/comment-" + initial + ".json", + readyCount, + ]) { + check(!(yield* exists(path)), "The moved attempt created " + path); + } + return; + } + check(result !== null, "The submission did not return its delivery result."); + if (mode === "roundtrip") { + check( + result.code !== 0 && result.stderr.includes("Source changed during review"), + "The restored checkout supplied a verdict: " + result.stderr + result.stdout, + ); + const retained = yield* observe(); + check( + retained.status === "reconciliation_required" && + retained.pin?.revision === initial && + retained.local?.head === initial && + retained.remote?.revision === initial && + retained.review === null, + "The round trip did not hold the attempt against the submitted SHA.", + ); + const attempt = directory + "/review-" + initial; + const reply = z.object({ revision: z.literal(initial), verdict: z.literal("pass") }); + const outcome = z + .object({ turn: z.object({ state: z.literal("reply"), reply: z.string() }) }) + .parse(JSON.parse(yield* readTextFile(attempt + "/outcome.json"))); + reply.parse(JSON.parse(outcome.turn.reply)); + z.object({ status: z.literal("completed") }).parse( + JSON.parse(yield* readTextFile(attempt + "/execution.json")), + ); + const reconciliation = yield* entry("reconcile"); + check( + reconciliation.code !== 0 && reconciliation.stderr.includes("Source changed during review"), + "Reconciliation accepted or lost the source-movement evidence.", + ); + yield* refuse("publish-review"); + yield* refuse("ready"); + yield* refuse("submit", initial); + for (const path of [ + attempt + "/receipt.json", + directory + "/comment-" + initial + ".json", + readyCount, + ]) { + check(!(yield* exists(path)), "The moved review created " + path); + } + check( + (yield* prompts()).join() === initial, + "Recovering the moved attempt repeated the prompt.", + ); + return; + } + check(result.code === 0, result.stderr + result.stdout); + const joined = statusSchema.parse(JSON.parse(result.stdout)); + check( + joined.pin?.revision === initial && joined.remote?.revision === newer, + "Observed head displaced the pin.", + ); + check( + joined.local?.head === (mode === "remote" ? initial : newer), + "Wrong local head observation.", + ); + check((yield* prompts()).join() === initial, "Overlap launched another reviewer."); + check( + !(yield* exists(directory + "/comment-" + initial + ".json")) && !(yield* exists(readyCount)), + "Moved head received an old verdict or readiness mutation.", + ); + if (mode === "local") { + yield* shell(`git reset --hard ${shellQuote(initial)}`); + yield* feedback(initial); + yield* json(remotePath, remote); + yield* refuse("reconcile"); + check( + (yield* observe()).status === "reconciliation_required", + "Restoring source certified a moved checkout.", + ); + } else { + check( + joined.review?.result.verdict === "pass", + "Remote movement discarded the joined verdict.", + ); + } + return; + } + + yield* writeTextFile(directory + "/changes-" + initial, ""); + const changes = yield* observe("submit", initial); + check( + changes.review?.result.verdict === "changes_required" && + changes.reviewPublication?.state === "verified", + "Repair verdict or comment missing.", + ); + const repair = yield* shell("git commit --allow-empty -qm repair && git rev-parse HEAD"); + yield* json(remotePath, { ...remote, revision: repair, ci: { status: "in_progress" } }); + yield* refuse("submit", repair); + yield* feedback(repair); + yield* refuse("submit", repair); + yield* json(remotePath, { ...remote, revision: repair }); + yield* feedback(initial); + yield* refuse("submit", repair); + check((yield* prompts()).join() === initial, "Ineligible repair launched a reviewer."); + yield* feedback(repair); + const pass = yield* observe("submit", repair); + check( + pass.pin?.previous === initial && + pass.pin.revision === repair && + pass.reviewPublication?.state === "verified", + "Repair did not retain both admissions and publication.", + ); + check( + yield* exists(directory + "/comment-" + initial + ".json"), + "Repair discarded the original comment.", + ); + yield* writeTextFile(directory + "/comments-unavailable", ""); + yield* refuse("ready"); + yield* rm(directory + "/comments-unavailable"); + yield* json(cwd + "/remote-next.json", { ...remote, revision: repair, ciGreen: false }); + yield* refuse("ready"); + check(!(yield* exists(readyCount)), "Readiness used stale CI or comment evidence."); + yield* json(remotePath, { ...remote, revision: repair }); + const ready = yield* observe("ready"); + const repeated = yield* observe("ready"); + check( + ready.status === "ready_pending" && + ready.nextAction === "mark_ready_manually" && + ready.remote?.draft === true && + ready.revision === repair && + repeated.revision === repair && + repeated.reviewPublication?.url === ready.reviewPublication?.url, + "Ready did not return the same verified draft candidate.", + ); + check(!(yield* exists(readyCount)), "Readiness changed the remote draft."); + yield* json(directory + "/remote-during-comment.json", { ...remote, revision: initial }); + yield* refuse("ready"); + check((yield* observe()).status === "source_changed", "Ready missed movement during readback."); + yield* json(remotePath, { ...remote, revision: repair, draft: false }); + check( + (yield* observe()).status === "ready_for_merge", + "Status missed the maintainer transition.", + ); + yield* json(directory + "/remote-during-comment.json", { + ...remote, + revision: initial, + draft: false, + }); + const moved = yield* observe(); + check( + moved.status === "source_changed" && + moved.remote?.revision === initial && + moved.pin?.revision === repair && + moved.reviewPublication?.url === ready.reviewPublication?.url, + "Status reported readiness for the former head or lost its verified comment.", + ); + check( + (yield* prompts()).join() === [initial, repair].join(), + "Unexpected review prompts across repair and readiness.", + ); +} diff --git a/examples/adversarial-implementation/tests/components/Fixture/Fallback.md b/examples/adversarial-implementation/tests/components/Fixture/Fallback.md index a5d9852d7..677b596ef 100644 --- a/examples/adversarial-implementation/tests/components/Fixture/Fallback.md +++ b/examples/adversarial-implementation/tests/components/Fixture/Fallback.md @@ -41,7 +41,7 @@ git symbolic-ref --short HEAD }[scenario]} /> - + @@ -268,15 +268,18 @@ rm -r -- {quotedReplacement} ``` - + + +{readyRemote} + ```bash exec as="resumedEntry" -{props.runner} {entry} --props-action review +{props.runner} {entry} --props-action submit --props-revision {revision} ``` call.position)} expected={[0,1,1]} /> - + !path.includes("/replacement/")).map(path => afterCut[path])} expected={Object.keys(beforeCut).filter(path => !path.includes("/replacement/")).map(path => beforeCut[path])} /> diff --git a/examples/adversarial-implementation/tests/components/Fixture/Repository.md b/examples/adversarial-implementation/tests/components/Fixture/Repository.md index b5fe67d9e..07271dddb 100644 --- a/examples/adversarial-implementation/tests/components/Fixture/Repository.md +++ b/examples/adversarial-implementation/tests/components/Fixture/Repository.md @@ -10,6 +10,7 @@ ssh-keygen -q -t ed25519 -N '' -f signer printf 'workflow-test ' > allowed-signers cat signer.pub >> allowed-signers git -C repository init -q +printf '%s\n' /bin/gh /remote.json /remote-next.json /ready-count.txt /interrupt-input.json /interrupt.out /interrupt.err /status.out /status.err /closeout.out /closeout.err /reconcile.out /reconcile.err /review.out /review.err >> repository/.git/info/exclude git -C repository config user.name Workflow git -C repository config user.email workflow@example.invalid git -C repository config commit.gpgsign true diff --git a/examples/adversarial-implementation/tests/components/Fixture/ReviewOverlap.ts b/examples/adversarial-implementation/tests/components/Fixture/ReviewOverlap.ts index 988493a82..b1b524358 100644 --- a/examples/adversarial-implementation/tests/components/Fixture/ReviewOverlap.ts +++ b/examples/adversarial-implementation/tests/components/Fixture/ReviewOverlap.ts @@ -1,4 +1,4 @@ -import { copyFile, ensureDir, exists, readTextFile, writeTextFile } from "@effectionx/fs"; +import { copyFile, ensureDir, exists, readTextFile, rm, writeTextFile } from "@effectionx/fs"; import { cwd } from "@executablemd/runtime"; import { runShell, shellQuote } from "@executablemd/test-support/launch"; import { ensure, race, scoped, spawn } from "effection"; @@ -20,8 +20,9 @@ export const props = { command: { type: "string" }, directory: { type: "string" }, revision: { type: "string" }, + scenario: { enum: ["pass", "changes_required", "lost"] }, }, - required: ["command", "directory", "revision"], + required: ["command", "directory", "revision", "scenario"], additionalProperties: false, }; @@ -29,6 +30,7 @@ const input = z.strictObject({ command: z.string(), directory: z.string().startsWith("/"), revision: z.string(), + scenario: z.enum(["pass", "changes_required", "lost"]), }); const production = new URL("../../../", import.meta.url); const fixtures = new URL("../../acp-overlap/", import.meta.url); @@ -45,6 +47,12 @@ function check(condition: boolean, message: string): asserts condition { const statusSchema = z.object({ status: z.string(), + nextAction: z.string(), + reviewPublication: z.object({ + state: z.string(), + revision: z.string().optional(), + url: z.string().optional(), + }), review: z.object({ attempt: z.string(), result: z.object({ @@ -66,16 +74,28 @@ const outcomeSchema = z.object({ }); export default function* ReviewOverlap(value: unknown) { - const { command, directory, revision } = input.parse(value); + const { command, directory, revision, scenario } = input.parse(value); const repository = yield* cwd(); const evidence = `${directory}/overlap`; const attempt = `${directory}/review-${revision}`; const replacement = `${attempt}/replacement`; const release = `${evidence}/release`; yield* ensureDir(evidence); - for (const file of ["Fable.md", "Astra.md", "Gate.md"]) { + for (const file of ["Fable.md", "Gate.md"]) { yield* copyFile(new URL(file, fixtures), `${evidence}/${file}`); } + const astra = yield* readTextFile(new URL("Astra.md", fixtures)); + yield* writeTextFile( + `${evidence}/Astra.md`, + scenario !== "changes_required" + ? astra + : astra + .replace('verdict:"pass"', 'verdict:"changes_required"') + .replace( + "findings:[]", + 'findings:[{blocking:true,description:"Queue drops a waiting message."}]', + ), + ); const args = [ fileURLToPath(new URL("delivery.md", production)), @@ -83,8 +103,6 @@ export default function* ReviewOverlap(value: unknown) { `${directory}/requirements.json`, "--props-number", "7", - "--props-action", - "review", "--include", fileURLToPath(new URL("components", fixtures)), "--include", @@ -98,13 +116,16 @@ export default function* ReviewOverlap(value: unknown) { .map(shellQuote) .join(" "); - const review = (actor: z.infer, timeout: number) => - runShell(`exec ${command} ${args}`, { - cwd: repository, - env: { [overlapEnv]: evidence, [actorEnv]: actor }, - inheritEnv: true, - timeout, - }).join(); + const review = (actor: z.infer, timeout: number, action = "submit") => + runShell( + `exec ${command} ${args} --props-action ${shellQuote(action)}${action === "submit" ? " --props-revision " + shellQuote(revision) : ""}`, + { + cwd: repository, + env: { [overlapEnv]: evidence, [actorEnv]: actor }, + inheritEnv: true, + timeout, + }, + ).join(); function* held() { return ( @@ -163,11 +184,14 @@ export default function* ReviewOverlap(value: unknown) { check( contender.code === 1 && contender.stderr.includes( - "The next action is reconcile_original_review; do not start another review.", + "is already reserved. Reconcile its original invocation; do not start another review.", ), `Unexpected contender result (${contender.code}): ${contender.stderr} ${contender.stdout}`, ); + if (scenario === "lost") { + yield* writeTextFile(`${evidence}/stale`, ""); + } yield* writeTextFile(release, ""); released = true; return yield* first; @@ -185,16 +209,120 @@ export default function* ReviewOverlap(value: unknown) { check(winner.code === 0, `Winner exited ${winner.code}: ${winner.stderr} ${winner.stdout}`); const status = statusSchema.parse(JSON.parse(winner.stdout)); check( - status.status === "ready_for_merge" && + status.status === + (scenario === "pass" + ? "ready_pending" + : scenario === "changes_required" + ? "repair_required" + : "publication_pending") && status.review.attempt === replacement && status.review.result.revision === revision && - status.review.result.verdict === "pass" && + status.review.result.verdict === + (scenario === "changes_required" ? "changes_required" : "pass") && status.review.result.acceptance.length === 1 && status.review.result.acceptance[0] === "behavior", `Winner did not retain its exact-head Astra pass: ${JSON.stringify(status)}`, ); - const later = yield* review("later", 25_000); + if (scenario === "lost") { + check(!(yield* exists(`${evidence}/winner-posted`)), "A stale PR head received a POST."); + check( + status.nextAction === "publish_review", + "A missing publication must take precedence over merge readiness.", + ); + yield* rm(`${evidence}/stale`); + yield* writeTextFile(`${evidence}/comments-unavailable`, ""); + const unavailable = yield* review("later", 25_000, "publish-review"); + check( + unavailable.code === 0 && + statusSchema.parse(JSON.parse(unavailable.stdout)).nextAction === "publish_review", + "Unavailable native comments granted readiness.", + ); + check( + !(yield* exists(`${evidence}/later-posted`)), + "Unavailable native comments allowed a POST.", + ); + yield* rm(`${evidence}/comments-unavailable`); + yield* rm(`${replacement}/receipt.json`); + yield* writeTextFile(`${evidence}/delay-post`, ""); + yield* scoped(function* () { + const first = yield* spawn(() => review("winner", 75_000, "reconcile")); + yield* ensure(function* () { + yield* writeTextFile(`${evidence}/release-post`, ""); + yield* first; + }); + function* exitedEarly(): Operation { + const result = yield* first; + throw new Error( + `Reconcile exited before comment publication (${result.code}): ${result.stderr} ${result.stdout}`, + ); + } + const ready = yield* race([waitForFile(`${evidence}/post-ready`, 35_000), exitedEarly()]); + check(!ready.timeout, "Reconciliation did not attempt to publish the joined result."); + const overlapping = yield* review("contender", 25_000, "publish-review"); + check(overlapping.code === 0, `Concurrent publication failed: ${overlapping.stderr}`); + const waiting = statusSchema.parse(JSON.parse(overlapping.stdout)); + check( + waiting.nextAction === "publish_review", + "Concurrent publish did not wait for readback.", + ); + check( + !(yield* exists(`${evidence}/contender-posted`)), + "Concurrent publication sent a second POST.", + ); + yield* writeTextFile(`${evidence}/release-post`, ""); + const result = yield* first; + check( + result.code === 0, + `Reconciliation after lost acknowledgement failed: ${result.stderr}`, + ); + const unconfirmed = statusSchema.parse(JSON.parse(result.stdout)); + check(unconfirmed.nextAction === "publish_review", "Lost acknowledgement granted readiness."); + }); + const report = yield* review("later", 25_000, "closeout"); + check( + report.code === 0 && + statusSchema.parse(JSON.parse(report.stdout)).nextAction === "publish_review", + "Closeout must leave unconfirmed publication pending.", + ); + check(!(yield* exists(`${evidence}/later-posted`)), "Closeout attempted a GitHub write."); + } + const published = yield* review("later", 25_000, "publish-review"); + check(published.code === 0, `Publication retry failed: ${published.stderr} ${published.stdout}`); + const verified = statusSchema.parse(JSON.parse(published.stdout)); + check( + verified.status === (scenario === "changes_required" ? "repair_required" : "ready_pending") && + verified.reviewPublication.state === "verified" && + verified.reviewPublication.revision === revision && + verified.reviewPublication.url === + "https://github.com/taras/executable.md/pull/7#issuecomment-42", + `Publication was not verified: ${JSON.stringify(verified)}`, + ); + const comment = z + .object({ body: z.string() }) + .parse(JSON.parse(yield* readTextFile(`${evidence}/comment.json`))); + check( + comment.body.startsWith(``) && + comment.body.includes("codex / gpt-6-astra") && + comment.body.includes("Reviewed the assigned behavior.") && + comment.body.includes( + scenario === "changes_required" ? "**CHANGES REQUIRED**" : "**PASS**", + ) && + (scenario !== "changes_required" || comment.body.includes("Queue drops a waiting message.")), + `The published comment does not describe the validated Astra review: ${comment.body}`, + ); + for (const actor of ["contender", "later"] as const) { + check(!(yield* exists(`${evidence}/${actor}-posted`)), `${actor} sent an extra POST.`); + check( + (yield* readPrompts(evidence, actor)).length === 0, + `${actor} contacted ACP during publication.`, + ); + } + if (scenario === "lost") { + check((yield* readPrompts(evidence, "winner")).length === 0, "Reconciliation contacted ACP."); + } + + const later = yield* review("later", 25_000, "review"); const laterPrompts = yield* readPrompts(evidence, "later"); check( laterPrompts.length === 0, @@ -203,7 +331,7 @@ export default function* ReviewOverlap(value: unknown) { check( later.code === 1 && later.stderr.includes( - "The next action is await_authorized_merge; do not start another review.", + `The next action is ${scenario === "changes_required" ? "request_approved_repair" : "mark_ready_manually"}; do not start another review.`, ), `Unexpected later result (${later.code}): ${later.stderr} ${later.stdout}`, ); diff --git a/packages/cli/tests/adversarial-session-continuity.test.ts b/packages/cli/tests/adversarial-session-continuity.test.ts index 39a35bee8..0cb0d27a4 100644 --- a/packages/cli/tests/adversarial-session-continuity.test.ts +++ b/packages/cli/tests/adversarial-session-continuity.test.ts @@ -35,6 +35,7 @@ import type { ReplSession, ReplSessionOptions } from "../src/repl/session.ts"; import type { ReplQuestion } from "../src/repl/elicitation.ts"; import { createFakeAcp, makeStore } from "./support/fake-acp.ts"; import type { FakeAcp, FakeStore } from "./support/fake-acp.ts"; +import { runGit } from "./support/fixture-git.ts"; import { AGENT_SOURCE, STANDARD_AGENT_SOURCE, world } from "../../acp/tests/released-peer.ts"; import PhaseSession, { props as phaseSessionProps, @@ -380,6 +381,7 @@ const captured = z.strictObject({ interface ReviewWorld { root: string; + revision: string; store: FakeStore; delivery: Record; asked: unknown[]; @@ -388,13 +390,17 @@ interface ReviewWorld { function* reviewWorld(fake: FakeAcp, create = fake.create): Operation { const root = join(tmpdir(), `xmd-delivery-review-${randomUUID()}`); - yield* ensureDir(join(root, "repository")); + const repository = join(root, "repository"); + yield* ensureDir(repository); + yield* runGit(repository, ["init", "--quiet", "--initial-branch=main"]); + yield* runGit(repository, ["commit", "--quiet", "--allow-empty", "-m", "Review fixture"]); + const revision = yield* runGit(repository, ["rev-parse", "HEAD"]); yield* ensureDir(join(root, "delivery")); const asked: unknown[] = []; const store = makeStore(); const delivery = { work: "production-review", - repository: join(root, "repository"), + repository, directory: join(root, "delivery"), policy: { repository: "taras/executable.md", @@ -452,7 +458,7 @@ function* reviewWorld(fake: FakeAcp, create = fake.create): Operation harness.run(reviewSource, { delivery: harness.delivery, revision }), ); @@ -796,6 +803,7 @@ describe( fake.script({ failure: "The Astra provider turn failed." }); const gate = closeGate(fake); const harness = yield* reviewWorld(fake, gate.create); + const { revision } = harness; const task = yield* spawn(() => harness.run(reviewSource, { delivery: harness.delivery, revision }), ); @@ -855,6 +863,7 @@ describe( fake.configuration(catalog(["fable", "gpt-6-astra"])); fake.script({ reply: "", cancelled: true }); const harness = yield* reviewWorld(fake); + const { revision } = harness; const ran = yield* harness.run(reviewSource, { delivery: harness.delivery, revision }); expect(ran.ok).toBe(true); expect(fake.turns.map((_, index) => agentOf(fake, index))).toEqual(["claude"]); @@ -938,6 +947,7 @@ describe( }); fake.closeFailure = new Error("the review provider would not close"); const harness = yield* reviewWorld(fake); + const { revision } = harness; const ran = yield* harness.run(reviewSource, { delivery: harness.delivery, revision }); expect(ran.ok).toBe(false); expect(ran.text).toContain("agent provider teardown failed"); @@ -1111,6 +1121,7 @@ describe( fake.script({ failure: "The Fable provider turn failed." }); fake.script({ failure: "The Astra provider turn failed." }); const harness = yield* reviewWorld(fake); + const { revision } = harness; const attempt = join(harness.delivery.directory as string, `review-${revision}`); const source = ``; const stream = new InMemoryStream(); diff --git a/scripts/lib/runtime-tests.ts b/scripts/lib/runtime-tests.ts index 0ba358bca..d882be7b4 100644 --- a/scripts/lib/runtime-tests.ts +++ b/scripts/lib/runtime-tests.ts @@ -106,7 +106,7 @@ export const MARKDOWN_DEADLINES: Record = { ArtifactReturn: 120_000, ArtifactReturnRecovery: 120_000, DecisionResolution: 300_000, - Delivery: 300_000, + Delivery: 420_000, Friction: 180_000, LaunchPolicy: 120_000, Preparation: 420_000, diff --git a/scripts/tests/runtime-exclusions.test.ts b/scripts/tests/runtime-exclusions.test.ts index 125123dd9..19585e43a 100644 --- a/scripts/tests/runtime-exclusions.test.ts +++ b/scripts/tests/runtime-exclusions.test.ts @@ -179,6 +179,7 @@ describe("the applicable corpus", () => { "ArtifactReturnRecovery", "DecisionResolution", "Delivery", + "DeliveryPublication", "Friction", "Implementation", "LaunchPolicy", @@ -384,7 +385,7 @@ describe("the one-file Markdown commands", () => { ArtifactReturn: 120_000, ArtifactReturnRecovery: 120_000, DecisionResolution: 300_000, - Delivery: 300_000, + Delivery: 420_000, Friction: 180_000, LaunchPolicy: 120_000, Preparation: 420_000, diff --git a/specs/testing-spec.md b/specs/testing-spec.md index bffc6eb6b..a0d8d2e2c 100644 --- a/specs/testing-spec.md +++ b/specs/testing-spec.md @@ -970,9 +970,9 @@ source host relaunches itself as a TestAgent worker, including workers for isolated run-profile children. Each adversarial suite but `Implementation`, whose named groups carry their -own deadlines, runs under a whole-launch deadline. `Preparation` has 420 -seconds, `Friction` has 180, `DecisionResolution` and `Delivery` have 300, and -the other adversarial suites have 120. A suite that +own deadlines, runs under a whole-launch deadline. `Preparation` and `Delivery` +have 420 seconds. `Friction` has 180, `DecisionResolution` has 300, and the +other adversarial suites have 120. A suite that reaches it is stopped and its cleanup joined, its path and deadline are reported, and it fails with status 1. The files after it still run. `SessionConfiguration` has a 120-second whole-launch deadline under each