[Marc-André Moreau (mamoreau-devolutions)]

Why

The first run of the implement-ticket agent (run 39783250637) produced a plausible change. But:

• it never used the feature it built;
• its screenshots caught a page still loading;
• it spent most of its time waiting on builds;
• nothing checked its claims;
• the same ticket could produce a different design each run.

This turns the workflow into a pipeline around the agent.

What changes

The agent's own run

• Scripts and prompt:
  • Stop-UniGetUIDev.ps1 replaces a stop command that Copilot CLI refuses to run.
  • Wait-UniGetUIIdle.ps1 waits for the page to finish loading (through UI Automation) before a screenshot.
  • PROMPT.md asks for acceptance criteria, before/after screenshot pairs that look visibly different, and one formatter pass. It also bans dotnet publish.
• Faster builds: dev builds keep the shared compiler server and skip restore while the project files and their restore results are unchanged.
• Avalonia docs: the agent gets Avalonia's Build MCP (mcp-config.json).

The pipeline (agent-implement-ticket.yml)

• resolve (no agent): turns the trigger into a request and acknowledges it on the issue or PR (Resolve-AgentRequest.ps1). It handles:

  • a manual run,
  • an agent:spec or agent:implement label,
  • /agent spec or /agent implement [guidance] on an issue,
  • /agent <feedback> on an agent PR.

  Labels and comments only start a run for users with write access.

• spec (Linux): an agent writes a spec with behavior, an ASCII UI sketch, acceptance criteria and the alternatives it considered (SPEC_PROMPT.md). spec-publish posts it on the issue; /agent implement (or spec_run_id) implements the approved spec.

• implement (Windows desktop): the agent, then:

  • checks the workflow runs itself (Test-AgentChange.ps1): no .github/ changes, the build, the tests of projects that reference the change, formatting of the changed files, and new strings missing from lang_en.json;
  • screenshot evidence (New-ScreenshotEvidence.ps1): crops the taskbar, outlines in red what changed between after-Na/after-Nb, and flags pairs that are nearly identical.

• review and fix round (on the same Windows desktop, inside the implement job):

  • A second agent on a different model (claude-sonnet-5.5, then claude-opus-5.5, gpt-6-luna) reviews the change in place (Invoke-AgentReview.ps1, REVIEW_PROMPT.md).
  • Its findings are structured (id, severity, file, line, fix) and flagged implementable when the fix is local, has one clear resolution and needs no design or spec decision.
  • If the reviewer touches the working tree, the implementer's change is put back.
  • Select-AgentFixes.ps1 picks, without an agent, the failed checks and the implementable blocker, major and minor findings.
  • One fix round (FIX_PROMPT.md) fixes only those, reports each one fixed or not, and updates the description. The checks and evidence then run again; the first round is kept in before-fix/.
  • There's no second review or fix round: whatever is left goes to the maintainer.

• report (New-AgentReport.ps1):

  • writes the checks, the evidence, the review and the run metrics into the PR body, the job summary and an artifact;
  • opens a draft PR, or pushes to the agent PR it iterates on;
  • comments the result on the issue.

• Metrics: Invoke-CopilotAgent.ps1 runs every agent with JSON output. It records the model auto picked, the AI credits, the tool calls, the builds, tests, waits, clicks and screenshots, and the MCP calls. The report adds the time spent in each phase.

• Older starting commits: a run from an older commit (an eval case, a PR iteration) uses this workflow's own .github/agent and .github/actions, committed locally so the agent's patch leaves them out.

Eval (agent-eval.yml)

• Replays merged PRs (.github/agent/eval/cases.json: Let the package details action button grow with its translated label #5561, Count failed operations in the operations panel header #5567 and Ask for confirmation before ignoring the updates of several packages #5556) from the commit before each one, with only the problem as the ticket.
• Scores each case: checks, review verdict, how much the files changed overlap with the human PR's, screenshots, minutes, AI credits and MCP calls.

Desktop session

• Faster module install: AwakeCoding.PSRemoting is installed with ModuleFast.
• Devolutions Agent (Devolutions.Agent 2026.3.1.0) is installed with its RDP extension (F.Session).
• NOW channel: with the now-channel input, the action negotiates NOW right after connecting and checks that a PowerShell command runs in the desktop session over NOW. Invoke-DesktopNowCommand exposes that to the agent.
• IronRDP pin: ironrdp-ref points at the merge of Devolutions/IronRDP#2209 (10199db3). That fixes the DVC pipe proxy, which handed its NOW messages to the client as channel 0 and dropped the RDP session. ironrdp-patches/ can carry upstream fixes ahead of a merge; it's empty now.
• NOW check: the command is retried up to 3 times; if NOW keeps failing, the job carries on without it (DESKTOP_SESSION_NOW=0) and warns. One run failed with "NOW operation stream closed before a terminal event" right after the handshake.
• Cache use:
  • the Rust build cache is gone, since it evicted the other caches from the repository's 10 GB;
  • the agent workflow restores NuGet packages from main without saving its own copy.

Tested in CI

Run	What it showed
39893205057 spec	Spec agent (gpt-6-luna, 93s, 1.33 credits) wrote a spec with an ASCII sketch and 8 acceptance criteria.
39893563369 implement from that spec	Desktop session and NOW: NOW negotiated, and "DevolutionsSession (pid 8356) ran a PowerShell command in desktop session 2 over NOW". Agent: covered all 8 criteria with 41 waits, 39 clicks and 0 sleeps. Checks: caught 4 strings missing from lang_en.json. Review: found 4 real major bugs (updates hidden for undiscovered sources, an error state that can't appear, "Select all" leaving a manager indeterminate, the missing strings). Evidence: flagged a nearly identical pair.
39899544626 continue that run with feedback	Iteration: run B's change became the base, and the new diff is only the feedback (2 files). The Sonnet review approved with minor findings. The report artifact has the phase timings (setup 1m13s, base build 1m42s, agent 9m20s, checks 7m20s).
39960837528 implement from the spec, with the fix round	NOW ran its check command on the first attempt. The Sonnet review approved with 5 minor findings, 3 marked implementable. The fix round fixed all 3 in 7m18s (including 6 new unit tests: 873 → 879). R1 (a design choice) and R5 (missing evidence for the loading, empty and error states) were left for the maintainer. The checks passed before and after; the reviewer guard didn't trigger.
39969885850 and 39971646154 (desktop-smoke), IronRDP 10199db3 without the patch	ironrdp-agent built from the merge commit. NOW negotiated, and the check command ran on the first attempt. desktop-smoke launched UniGetUI and took clean screenshots.
39893339409 eval, case 5561	Reusable-workflow path end to end. The checks passed (870 and 225 tests). The review requested changes (a real split-button height bug). The agent changed the same single file as the human PR. The scorer wrote the report table.

Locally:

• the resolver was run on nine sample events (labels, comments, outsiders, non-agent PRs);
• the checks script was run on deliberately broken changes;
• the evidence script was run on earlier screenshots;
• the restore check was run across a Release restore;
• Build MCP calls were made through --additional-mcp-config.

Not tested yet

• Label and comment triggers: GitHub only runs issues and issue_comment workflows from the default branch, so they start working after merge. The resolver logic is tested locally, and its dispatch equivalents (spec_run_id, continue_run_id) ran in CI.
  • Not exercised: posting the spec on an issue, reading an approved spec back from an issue, and pushing a commit to an existing agent PR.
  • agent-eval.yml can only be started by hand from the default branch. Its CI run used a temporary push trigger that has since been removed.
• No real pull request opened: every test run had create_pr off.
• Labels: the agent:spec and agent:implement labels don't exist in the repository yet.
• TUI tests: the checks skip UniGetUI.Tui.Tests unless the TUI changed (its test host crashed once on the runner). That rule was tested locally, not in CI.

🤖 Generated with Claude Code