Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 32 additions & 8 deletions examples/adversarial-implementation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,8 +248,8 @@ xmd run --allow-absolute /absolute/path/to/workflow/examples/adversarial-impleme
--include /absolute/path/to/workflow/examples/adversarial-implementation/components --raw
```

`status` starts no model. It reads the saved joined review, the current PR and
GitHub's CI workflow runs. Its `nextAction` is computed from the same facts as
`status` starts no model. It reads the saved joined review, its published comment,
the current PR and GitHub's CI workflow runs. Its `nextAction` is computed from the same facts as
its `status` and `remaining` fields. When it requests `review`, select
`--props-action review` on that command to perform the independent review with
the provider, model, effort and session recorded at launch. The review uses
Expand Down Expand Up @@ -295,10 +295,11 @@ feedback ready -> independent review pending -> joined exact-commit verdict
| Fable unavailable/refused -> one Astra review (review)
| no reviewer completed -> review_held: answer via review
| incomplete/uncertain -> reconcile original call
| blocking finding -> request approved repair
| pass + required CI + main -> finish PR metadata
| all passed + PR ready -> await authorized merge
| confirmed merge -> delivered
| comment pending -> publish-review
| published changes required -> request approved repair
| published pass + checks -> finish PR metadata
| all passed + PR ready -> await authorized merge
| confirmed merge -> delivered
```

Independent review summaries allow 1500 characters; detailed observations belong
Expand Down Expand Up @@ -340,6 +341,28 @@ and saves a raw reply the execution joined but never wrote. It never shortens te
a model call or overwrites a contradictory receipt. Missing or uncertain execution
continues to hold. Status uses this same validation boundary to read receipts.

`review` and `reconcile` publish each validated verdict as a comment on the
current pull request, including findings that require repair. The comment
names the exact reviewed commit and the reviewer who supplied the result.
Delivery uses `<PullRequest.Comments>` to reconcile and verify the comment. With
the native GitHub provider, include the repository URL in the host's
`XMD_WORKFLOW_GITHUB_PULL_REQUESTS` allowlist. For a host that reads one repository:

```sh
export XMD_WORKFLOW_GITHUB_PULL_REQUESTS='{"allowed":["https://github.com/owner/repository"]}'
```

Once GitHub returns the matching comment, the workflow saves its URL with the
reviewed SHA. A new head has its own comment.

If `nextAction` is `publish_review`, use `--props-action publish-review` with
the same record and PR number. It checks for the matching comment before any
write and adopts it if GitHub already accepted it. Concurrent calls with the
same delivery record share one write reservation per PR and reviewed SHA.
If a write could have reached GitHub and no matching comment is visible, the
reservation stays in place for reconciliation. `status` and `closeout` only
read GitHub. A pending comment holds readiness and the repair handoff.

Reconciliation is limited to independent review from a work item launched under
the same response contract. Its assignment retains the exact response schema; a
missing or different schema refuses rather than reinterpreting an old reply. It validates only corrections recorded under the original review assignment.
Expand Down Expand Up @@ -377,8 +400,9 @@ Main health reuses the repository's authoritative selection
rule for a push to main's exact current head. Pending, cancelled, skipped and
failed runs cannot pass. The maintainer-applied `ci-main-red-fix` exception keeps
the repository's CI contract: it substitutes composability for the main lookup
inside CI. A draft PR remains `publication_pending`; only a confirmed merge with
a merge commit reports `delivered`. Refresh status before acting on readiness.
inside CI. A draft with a verified passing review and passing checks reports
`publication_pending` with `finish_pr_metadata`. A confirmed merge with a
merge commit reports `delivered`. Refresh status before acting on readiness.

The saved `status.json` is the last observation, not a second execution state.
Use the command for a current answer. Local records are trusted workflow
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ Next action: {status.nextAction}
<If condition={status.review != null}>
Independent review: {status.review.result.verdict}

<If condition={status.reviewPublication?.state === "verified"}>
Published review comment {status.reviewPublication.url}
</If>

{status.review.result.summary}

<Each in={status.review.result.findings} let="finding">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,4 +54,4 @@ completion turn can supply the review.
<If condition={checkout.head !== props.revision || !checkout.clean}><Fail message="Source changed since the review. Preserve its result; the current source needs its own evidence." /></If>
<If condition={!derived.includes("receipt.json")}><Dir path={attempt}><File path="receipt.json"><Json value={receipt} /></File></Dir></If>
</If>
<Return value={receipt} />
<Return value={{...receipt,model:read.state.attempts[read.state.candidate.position].model}} />
Original file line number Diff line number Diff line change
@@ -0,0 +1,256 @@
import { rm } from "@effectionx/fs";
import { API, cwd, exec, parseFilesFailure } from "@executablemd/runtime";
import { ensure, scoped, type Operation } from "effection";
import { z } from "zod";
import { createDirectory, digest } from "../Ticket/native-node.ts";
import { clear, write } from "../Ticket/text-storage.ts";
import { reviewerSchema } from "./records.ts";

export const props = {
type: "object",
properties: {
target: { type: "object" },
review: { type: "object" },
publish: { type: "boolean" },
allowPost: { type: "boolean" },
comments: { type: "array" },
},
required: ["target", "review", "publish", "allowPost", "comments"],
additionalProperties: false,
};

const sha = z.string().regex(/^[0-9a-f]{40}$/);
const targetSchema = z.strictObject({
directory: z.string().startsWith("/"),
repository: z.string().regex(/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/),
number: z.number().int().positive().max(Number.MAX_SAFE_INTEGER),
branch: z.string().min(1),
revision: sha,
});
const input = z.strictObject({
target: targetSchema,
review: z.object({
reviewer: reviewerSchema,
model: z.string().min(1).nullable(),
result: z.object({
revision: sha,
verdict: z.enum(["pass", "changes_required", "blocked"]),
summary: z.string().min(1).max(1500),
findings: z.array(z.object({ blocking: z.boolean(), description: z.string().min(1) })),
}),
}),
publish: z.boolean(),
allowPost: z.boolean(),
comments: z.array(
z.object({
kind: z.enum(["conversation", "review"]),
id: z.string(),
url: z.string(),
body: z.string(),
}),
),
});
const pulls = z.object({
number: targetSchema.shape.number,
base: z.object({ repo: z.object({ full_name: z.string() }) }),
head: z.object({ ref: z.string(), sha }),
});
const saved = z.strictObject({
revision: sha,
bodySha256: z.string().regex(/^[0-9a-f]{64}$/),
id: z.number().int().positive().max(Number.MAX_SAFE_INTEGER),
url: z.string(),
});

type Target = z.infer<typeof targetSchema>;
type Review = z.infer<typeof input>["review"];
type Verified = z.infer<typeof saved>;

function pending(reason: string) {
return { state: "pending", reason };
}

function quote(text: string) {
return text
.replace(/\r\n?/g, "\n")
.split("\n")
.map((line) => " " + line)
.join("\n");
}

function render(marker: string, review: Review, model: string) {
const { revision, verdict, summary, findings } = review.result;
const titles = { pass: "PASS", changes_required: "CHANGES REQUIRED", blocked: "BLOCKED" };
return [
marker,
"## Independent review",
"Reviewed commit `" + revision + "`",
"Verdict **" + titles[verdict] + "**",
"Reviewer\n\n" + quote(review.reviewer.provider + " / " + model),
"### Summary\n\n" + quote(summary),
"### Findings",
...(findings.length === 0
? ["None."]
: findings.map(
({ blocking, description: findingText }) =>
(blocking ? "**Blocking**" : "**Non-blocking**") + "\n\n" + quote(findingText),
)),
].join("\n\n");
}

function* github(...args: string[]) {
return yield* exec({
command: ["gh", "api", "--hostname", "github.com", ...args],
cwd: yield* cwd(),
timeout: 30_000,
});
}

function* readTarget(target: Target) {
const { exitCode, stdout } = yield* github(`repos/${target.repository}/pulls/${target.number}`);
if (exitCode !== 0) {
return false;
}
const pr = pulls.parse(JSON.parse(stdout));
return (
pr.number === target.number &&
pr.base.repo.full_name.toLowerCase() === target.repository.toLowerCase() &&
pr.head.ref === target.branch &&
pr.head.sha === target.revision
);
}

function findComment(
target: Target,
comments: z.infer<typeof input>["comments"],
marker: string,
body: string,
): Verified | null {
const matches = comments.filter(
(item) => item.kind === "conversation" && item.body.split("\n", 1)[0] === marker,
);
if (matches.length > 1) {
throw new Error("Multiple GitHub comments claim the same review revision.");
}
const [value] = matches;
if (value === undefined) {
return null;
}
const { repository, number, revision } = target;
const id = Number(value.id);
const urls = ["issues", "pull"].map((kind) =>
`https://github.com/${repository}/${kind}/${number}#issuecomment-${id}`.toLowerCase(),
);
if (
!Number.isSafeInteger(id) ||
id <= 0 ||
value.id !== String(id) ||
value.body !== body ||
!urls.includes(value.url.toLowerCase())
) {
throw new Error("The GitHub comment does not match the saved review.");
}
return { revision, bodySha256: digest(new TextEncoder().encode(body)), id, url: value.url };
}

function* readSaved(directory: string) {
const result = yield* API.Files.operations.readTextFile({ cwd: directory, path: "comment.json" });
if (result.ok) {
return saved.parse(JSON.parse(result.value));
}
if (parseFilesFailure(result.error)?.reason === "missing") {
return null;
}
throw result.error;
}

function* post(target: Target, reservation: string, body: string): Operation<void> {
try {
// A reservation survives interruption so a retry can adopt the remote comment.
yield* createDirectory(reservation);
} catch (error) {
if (z.object({ code: z.literal("EEXIST") }).safeParse(error).success) {
return;
}
throw error;
}
yield* scoped(function* () {
let release = true;
yield* ensure(function* () {
if (release) {
yield* rm(reservation, { recursive: true, force: true });
}
});
const payload = reservation + "/request.json";
yield* write(target.directory, payload, JSON.stringify({ body }));
if (!(yield* readTarget(target))) {
return;
}
release = false;
const { exitCode, stdout } = yield* github(
"--include",
"--method",
"POST",
`repos/${target.repository}/issues/${target.number}/comments`,
"--input",
payload,
);
// An explicit rejection permits another attempt. Transport and server errors leave the reservation in place.
release = exitCode !== 0 && /^HTTP\/\d(?:\.\d)? (400|401|403|404|422|429)\b/.test(stdout);
});
}

export default function* ReviewComment(value: unknown) {
const { target, review, publish, allowPost, comments } = input.parse(value);
if (review.result.revision !== target.revision) {
return pending("The retained review names a different revision.");
}
const marker = `<!-- xmd-independent-review:v1 ${target.revision} -->`;
const model = review.model ?? review.reviewer.model;
const body = render(marker, review, model);
const record = `${target.directory}/review-${target.revision}/publication-${target.number}`;
try {
const previous = yield* readSaved(record);
if (!publish && previous === null) {
return pending("Publish the joined review and verify the GitHub comment.");
}
if (body.length > 65_536) {
return pending("The rendered review exceeds the GitHub comment limit.");
}
yield* clear(
[
model,
review.result.summary,
...review.result.findings.map((finding) => finding.description),
].join("\n"),
);
yield* clear(body);
if (!(yield* readTarget(target))) {
return pending("The current pull request does not match the reviewed revision.");
}

const observed = findComment(target, comments, marker, body);
if (observed === null && publish && allowPost && previous === null) {
yield* post(target, record, body);
return { state: "readback" };
}
if (observed === null) {
return pending(
"The current review comment could not be verified. Retry publication from the saved receipt.",
);
}
if (previous !== null && JSON.stringify(previous) !== JSON.stringify(observed)) {
return pending("The GitHub comment differs from its recorded identity.");
}
if (publish && previous === null) {
// A matching comment may already exist even if no local request survived.
yield* API.Fs.operations.ensureDir(record);
yield* write(target.directory, record + "/comment.json", JSON.stringify(observed));
}
return { state: "verified", ...observed };
} catch {
return pending(
"The current review comment could not be verified. Check publication access and reconcile the saved receipt.",
);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
description: 'Verify a saved review comment or publish it from its validated receipt. Use <Delivery.ReviewPublication target={target} review={review} publish={false} as="publication" />.'
props:
type: object
properties:
target: {type: object}
review: {type: object}
publish: {type: boolean}
required: [target, review, publish]
additionalProperties: false
returns: {type: object}
---

<Let as="url" value={"https://github.com/" + props.target.repository + "/pull/" + props.target.number} />
<Try>
<PullRequest.Comments url={url} as="comments" />
<Delivery.ReviewComment target={props.target} review={props.review} publish={props.publish} allowPost={props.publish} comments={comments} as="publication" />
<If condition={publication.state === "readback"}>
<PullRequest.Comments url={url} as="readback" />
<Delivery.ReviewComment target={props.target} review={props.review} publish={true} allowPost={false} comments={readback} as="publication" />
</If>
<Return value={publication} />
<Catch>
<Return value={{state:"pending",reason:"Could not read pull request comments. Check the GitHub host allowlist and retry from the saved receipt."}} />
</Catch>
</Try>
Loading
Loading