Document complete development graph

Author: rainerhahnekamp

AGENTS.md

@@ -0,0 +1,37 @@
+# Software Development Graph
+
+This repository implements a durable, local-LLM software-development workflow orchestrated with Mastra. It is not merely a GitHub webhook receiver. Preserve the complete graph described in `docs/architecture.md` whenever changing the implementation.
+
+## Operating model
+
+The production graph starts from a GitHub event and first checks whether the issue contains enough information. It may ask a human for clarification or propose child issues before implementation. One write-capable Codex worker then implements the accepted issue in an isolated worktree using a persistent goal and explicit acceptance criteria.
+
+After deterministic checks, independent read-only reviewers run in parallel. The required reviewer perspectives are security, architecture, technology/framework, performance, tests and acceptance criteria, accessibility, code quality, bug detection, and visual design consistency. A manager node consolidates their evidence, removes duplicates, resolves compatible recommendations, and identifies genuine conflicts. It may send repair work back to implementation. When competing recommendations depend on product priorities or cannot be resolved from the recorded criteria, it suspends the run and presents the arguments to a human.
+
+Successful runs may proceed to a pull request and human approval. All steps emit evidence to the recorder. Production runs never rewrite their own prompts, skills, or graph.
+
+The self-improvement graph is a separate scheduled process. It reads newly recorded runs after `lastAnalysedRunId`, distils recurring lessons, proposes versioned candidate changes, and evaluates each candidate against a frozen baseline using replay cases. A candidate is promoted only when it improves the agreed metrics without violating regression or safety gates and a human approves it.
+
+## Validation boundaries
+
+Keep these concepts separate in code and documentation:
+
+1. Inner self-correction checks whether the agent's latest action worked.
+2. Codex goal validation checks whether the full stated outcome and acceptance criteria are satisfied.
+3. Deterministic checks execute externally defined tests, builds, linters, type checks, and policy checks.
+4. Independent reviewers assess concerns not established by the implementation worker's own tests.
+5. Human approval resolves product decisions and accepts high-impact changes.
+
+Agent-written tests are useful implementation artifacts, but they are not independent proof. Acceptance criteria, existing tests, externally defined checks, review evidence, and human evaluation form the independent verification boundary.
+
+## Safety and concurrency
+
+GitHub delivery IDs are idempotency keys. Persist an event before acknowledging it. An event for an active issue joins that run's inbox and is consumed only at a safe boundary; it must not interrupt a model turn. Different issues may have separate durable runs, but only one write-capable worker may use a given worktree at a time. Reviewers are read-only and may run concurrently after the implementation snapshot is fixed.
+
+Do not let the system review or merge its own bot-generated GitHub events. Version prompts, skills, rubrics, graph definitions, models, and acceptance criteria on every recorded run. Never auto-promote a learned candidate merely because it is frequent or recent.
+
+## Current implementation status
+
+The repository currently implements GitHub ingestion, duplicate suppression, same-issue correlation, a local implementation-slot queue, Mastra persistence, and human suspension/resumption. The implementation action is still simulated. The readiness/decomposition node, real Codex goal worker, reviewer fan-out, manager, recorder schema, pull-request integration, and scheduled learning graph remain to be implemented. Do not describe those pieces as complete until code and tests exist.
+
+When adding a node, update `docs/architecture.md`, add durable state for its inputs and outputs, and test success, retry, suspension, and duplicate-delivery behavior where applicable.

README.md

@@ -1,16 +1,26 @@
-# Durable GitHub → Mastra loop
+# Local-LLM software-development graph
 
-This is the small control plane we discussed for the webinar. GitHub events are
-recorded in SQLite first and acknowledged quickly. A coordinator then gives
-Mastra at most one implementation slot by default. Consequently, an event that
-arrives while Mastra is working is never used to interrupt the current model
-turn.
+The intended system is a complete development graph: issue readiness and
+decomposition, Codex goal-based implementation, deterministic checks, parallel
+specialist reviewers, manager consolidation and human escalation, recording,
+and a separate scheduled self-improvement graph. The durable GitHub → Mastra
+loop below is the first implemented slice, not the complete workflow.
 
-For an event concerning the same issue, the coordinator attaches it to the
-existing run's inbox. For a different issue it creates another run, which waits
-for the local implementation slot. GitHub delivery IDs make retries harmless,
-and messages produced by the configured bot are ignored to prevent feedback
-loops.
+Future agent sessions should begin with `AGENTS.md`. The full design, reviewer
+contract, conflict handling, visual-design evidence, recorder, and learning
+boundary are documented in `docs/architecture.md`. Machine-readable node status
+and the required reviewer set live in `src/graph-definition.ts`.
+
+## Implemented control-plane slice
+
+GitHub events are recorded in SQLite first and acknowledged quickly. A
+coordinator gives Mastra at most one implementation slot by default. An event
+that arrives while Mastra is working never interrupts the current model turn.
+
+For the same issue, the coordinator attaches the event to the existing run's
+inbox. For a different issue it creates another run, which waits for the local
+implementation slot. GitHub delivery IDs make retries harmless, and messages
+produced by the configured bot are ignored to prevent feedback loops.
 
 ```mermaid
 flowchart LR
@@ -38,32 +48,29 @@ pnpm start
 
 The service listens only on `127.0.0.1:4317`. `GET /health` shows whether an
 implementation occupies the slot, and `GET /runs` shows the durable run state.
-The event database and Mastra's workflow snapshots live separately under
-`.data/`.
+The event database and Mastra workflow snapshots live separately under `.data/`.
 
 The included [GitHub Actions workflow](.github/workflows/local-loop.yml) uses a
 self-hosted runner labelled `local-llm`. That runner opens an outbound
 connection to GitHub, receives the job, and posts the event to the loop service
-on the same machine. Therefore the local machine does not need a public inbound
-port. For a direct GitHub webhook instead, expose the receiver through a secure
-tunnel and set `GITHUB_WEBHOOK_SECRET`; the receiver validates
-`x-hub-signature-256` against the raw body.
+on the same machine. The local machine therefore needs no public inbound port.
+For a direct GitHub webhook, expose the receiver through a secure tunnel and set
+`GITHUB_WEBHOOK_SECRET`; the receiver validates `x-hub-signature-256`.
 
 ## Human decisions
 
-Add the label `needs-human` to an issue to demonstrate suspension. Mastra stores
-the suspended workflow snapshot and the control database marks the run as
-`waiting_human`. A subsequent human issue comment is correlated with that issue
-and resumes the exact Mastra run. Other issues may proceed while this one waits.
+Add the label `needs-human` to demonstrate suspension. Mastra stores the
+suspended workflow snapshot and the control database marks the run as
+`waiting_human`. A subsequent human issue comment resumes the exact run. Other
+issues may proceed while this one waits.
 
-## Where Codex fits
+## Where Codex fits today
 
 The current implementation step deliberately waits for
-`SIMULATED_IMPLEMENTATION_MS`; this makes concurrency behavior deterministic in
-the demo and tests. Replace that delay inside `src/workflow.ts` with the Codex
-goal invocation. The durable inbox, one-writer rule, suspension, and event
-routing stay unchanged. Review agents can then be added after implementation as
-parallel Mastra branches, followed by the manager/consolidation node we designed.
+`SIMULATED_IMPLEMENTATION_MS`; this makes concurrency deterministic in the demo
+and tests. It still needs to be replaced by the real Codex goal worker. The
+readiness node, reviewers, manager, recorder, pull-request integration, and
+self-improvement graph are specified but explicitly marked as planned.
 
 ## Verify
 

docs/architecture.md

@@ -0,0 +1,64 @@
+# Intended software-development graph
+
+The webhook receiver is the entrance to the system, not the system itself. The target is a durable graph in which Codex performs one bounded implementation, independent specialists review the resulting snapshot, a manager reconciles their findings, and humans decide questions that cannot be derived from acceptance criteria or evidence.
+
+```mermaid
+flowchart TD
+  E["GitHub event"] --> I["Durable inbox"]
+  I --> R["Readiness and decomposition"]
+  R -->|missing information| H1["Ask human and suspend"]
+  H1 --> R
+  R -->|split needed| S["Propose child issues"]
+  S --> H1
+  R -->|ready| C["Codex implementation with goal"]
+  C --> D["Deterministic checks"]
+  D -->|failure| C
+  D -->|pass| F["Parallel specialist reviews"]
+  F --> M["Review manager"]
+  M -->|repairable findings| C
+  M -->|unresolved trade-off| H2["Human decision"]
+  H2 --> C
+  M -->|accepted| P["Pull request and human approval"]
+  P --> O["Recorder / experience store"]
+```
+
+Readiness is explicit because implementation must not compensate for an underspecified issue by inventing product decisions. It checks the definition of done, constraints, affected behaviour, required visual references, and available repository context. If the change contains multiple independently deliverable outcomes, it proposes child issues with separate acceptance criteria. Creating them is a human-approved side effect.
+
+The implementation node has exclusive write access to an isolated worktree. Codex uses a persistent goal containing the accepted outcome, constraints, and verification criteria. Its internal corrections and goal completion are useful but do not count as independent review. Repository-owned tests, builds, linting, type checks, and policy checks run before reviewers receive an immutable commit or diff.
+
+## Review fan-out and manager
+
+The required reviewer set lives in `src/graph-definition.ts`: security, architecture, technology/framework, performance, tests and acceptance criteria, accessibility, code quality, bug detection, and visual design. Every reviewer gets the same issue, acceptance criteria, base and candidate revisions, diff, deterministic evidence, and repository guidance. Reviewers are read-only and return structured, evidenced findings rather than modifying code.
+
+The visual-design reviewer must inspect rendered evidence rather than infer appearance from source. Its inputs include baseline and candidate screenshots at agreed viewports, relevant design tokens or component examples, and important interaction states. It judges hierarchy, spacing, typography, colour, responsiveness, visual regressions, and consistency with the surrounding application. Accessibility remains separate because a coherent-looking screen can still be unusable with a keyboard or assistive technology.
+
+The manager does not decide by majority vote. It groups duplicate findings, checks evidence, applies explicit project priorities, and produces one repair brief. If an architecture recommendation conflicts with a performance recommendation, measurements and existing constraints may resolve it. If the choice depends on an unstated product priority, the manager suspends the run and presents both arguments, evidence, costs, and the decision required to a human. That answer becomes durable input to the same run.
+
+Repair loops are bounded. Exceeding the maximum implementation/review iterations escalates to a human instead of creating an infinite loop.
+
+## Events and concurrency
+
+Every delivery is persisted before acknowledgement. Duplicate delivery IDs are no-ops. An event for an active issue joins its inbox and is read at a safe boundary; it never interrupts a model turn. Different issues retain independent durable state, while a lease prevents two writers from using the same worktree. Reviewers may run in parallel because they are read-only. Bot-authored events and run markers prevent self-triggering.
+
+The current code implements this control-plane slice and generic Mastra suspension/resumption. `src/graph-definition.ts` records which nodes are implemented, partial, or planned so future sessions cannot mistake the design for completed behaviour.
+
+## Recorder and separate self-improvement graph
+
+Every production node will emit an append-only record containing its inputs, outputs, timestamps, tool calls, command results, changed files, deterministic results, reviewer findings, manager resolutions, human decisions, merge/revert/regression outcome, and exact model, prompt, skill, rubric, and graph versions. The recorder observes production; it never changes the graph that generated the evidence.
+
+```mermaid
+flowchart LR
+  P["Production runs"] --> X["Append-only experience store"]
+  X --> N["Select after lastAnalysedRunId"]
+  N --> L["Distil recurring lessons"]
+  L --> C["Versioned candidate"]
+  C --> B["Baseline vs candidate replay"]
+  B --> G{"Regression and safety gates"}
+  G -->|fail| Q["Reject with evidence"]
+  G -->|pass| A["Human promotion approval"]
+  A --> V["New version for future runs"]
+```
+
+The learning graph is scheduled separately from production. `lastAnalysedRunId` makes discovery incremental, while older runs remain as replay and regression cases. Baseline and candidate run the same representative cases. Quality, completion, regressions, safety, cost, latency, iteration count, and escalation rate are compared. Frequency or plausible wording is not proof of improvement.
+
+Promotion is versioned and human-approved, affects only future runs, and retains the previous version for rollback. This boundary makes the system self-improving without allowing it to silently rewrite its own rules during active development.

src/graph-definition.ts

@@ -0,0 +1,57 @@
+/** Machine-readable contract for the graph we are building. */
+export type NodeStatus = "implemented" | "partial" | "planned";
+
+export interface GraphNodeDefinition {
+  id: string;
+  responsibility: string;
+  kind: "deterministic" | "agent" | "parallel" | "human-gate" | "side-effect";
+  status: NodeStatus;
+}
+
+export interface ReviewerDefinition {
+  id:
+    | "security"
+    | "architecture"
+    | "technology"
+    | "performance"
+    | "tests"
+    | "accessibility"
+    | "code-quality"
+    | "bugs"
+    | "visual-design";
+  question: string;
+  requiresVisualEvidence: boolean;
+}
+
+export const reviewers: readonly ReviewerDefinition[] = [
+  { id: "security", question: "Does the change introduce exploitable behavior, unsafe trust boundaries, or leaked secrets?", requiresVisualEvidence: false },
+  { id: "architecture", question: "Does the change preserve the application's boundaries, ownership, and long-term design?", requiresVisualEvidence: false },
+  { id: "technology", question: "Does it follow the current framework, language, and repository-specific practices?", requiresVisualEvidence: false },
+  { id: "performance", question: "Does it create unacceptable latency, memory, network, bundle-size, or scaling costs?", requiresVisualEvidence: false },
+  { id: "tests", question: "Are the acceptance criteria independently covered, including important failure paths?", requiresVisualEvidence: false },
+  { id: "accessibility", question: "Can people using keyboards and assistive technology perceive and operate the result?", requiresVisualEvidence: true },
+  { id: "code-quality", question: "Is the change understandable, maintainable, cohesive, and appropriately simple?", requiresVisualEvidence: false },
+  { id: "bugs", question: "What incorrect behavior, edge cases, races, or regressions remain?", requiresVisualEvidence: false },
+  { id: "visual-design", question: "Is the rendered result visually appealing and consistent with the rest of the application?", requiresVisualEvidence: true },
+] as const;
+
+export const productionGraph: readonly GraphNodeDefinition[] = [
+  { id: "github-ingest", responsibility: "Persist, deduplicate, correlate, and acknowledge GitHub events.", kind: "deterministic", status: "implemented" },
+  { id: "readiness-and-decomposition", responsibility: "Verify sufficient context and decide whether the issue should be split into child issues.", kind: "agent", status: "planned" },
+  { id: "human-clarification", responsibility: "Suspend and resume when missing information or a product decision requires a human.", kind: "human-gate", status: "partial" },
+  { id: "codex-goal-implementation", responsibility: "Implement one accepted issue in an isolated worktree against explicit acceptance criteria.", kind: "agent", status: "planned" },
+  { id: "deterministic-checks", responsibility: "Run repository-owned tests, builds, linting, type checks, and policy checks.", kind: "deterministic", status: "planned" },
+  { id: "specialist-reviewers", responsibility: "Run the independent reviewer set in parallel against one fixed implementation snapshot.", kind: "parallel", status: "planned" },
+  { id: "review-manager", responsibility: "Consolidate evidence, resolve compatible findings, route repairs, and expose true conflicts.", kind: "agent", status: "planned" },
+  { id: "human-conflict-decision", responsibility: "Choose between irreconcilable recommendations using their evidence and trade-offs.", kind: "human-gate", status: "planned" },
+  { id: "pull-request", responsibility: "Publish an approved, verified change and its evidence exactly once.", kind: "side-effect", status: "planned" },
+  { id: "recorder", responsibility: "Append reconstructable run evidence without modifying the active graph.", kind: "deterministic", status: "planned" },
+] as const;
+
+export const selfImprovementGraph: readonly GraphNodeDefinition[] = [
+  { id: "select-new-experiences", responsibility: "Read production evidence after lastAnalysedRunId while retaining older replay cases.", kind: "deterministic", status: "planned" },
+  { id: "distil-lessons", responsibility: "Identify recurring failures and propose bounded prompt, skill, rubric, or graph changes.", kind: "agent", status: "planned" },
+  { id: "baseline-versus-candidate-replay", responsibility: "Run identical representative cases against the frozen baseline and each candidate.", kind: "parallel", status: "planned" },
+  { id: "improvement-gates", responsibility: "Reject regressions, safety violations, semantic drift, and unsupported gains.", kind: "deterministic", status: "planned" },
+  { id: "human-promotion", responsibility: "Approve a versioned candidate for future runs; never mutate active runs.", kind: "human-gate", status: "planned" },
+] as const;

test/graph-definition.test.ts

@@ -0,0 +1,38 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import {
+  productionGraph,
+  reviewers,
+  selfImprovementGraph,
+} from "../src/graph-definition.js";
+
+test("the graph contract retains every agreed independent reviewer", () => {
+  assert.deepEqual(
+    reviewers.map(({ id }) => id),
+    [
+      "security",
+      "architecture",
+      "technology",
+      "performance",
+      "tests",
+      "accessibility",
+      "code-quality",
+      "bugs",
+      "visual-design",
+    ],
+  );
+});
+
+test("production and learning remain separate graphs with explicit status", () => {
+  assert.ok(productionGraph.some(({ id }) => id === "recorder"));
+  assert.ok(selfImprovementGraph.some(({ id }) => id === "human-promotion"));
+  assert.equal(
+    productionGraph.find(({ id }) => id === "codex-goal-implementation")?.status,
+    "planned",
+  );
+  assert.equal(
+    selfImprovementGraph.find(({ id }) => id === "baseline-versus-candidate-replay")
+      ?.status,
+    "planned",
+  );
+});