[feat] Score a finished session on demand #1178

Parent: #1137

Goal

Scorers judge workflow steps (#1085). Most agent work happens in ordinary sessions, which no scorer can reach today. Let a member judge a finished session of Claude Code, Codex or OpenCode with a team scorer, keep the reasoning on their machine, and publish only the label.

This issue adds the on-demand path: what a session ran as, how its judge runs, and where its labels go. Scoring sessions in the background is #1179.

What a session ran as

A session's label carries the agent the session ran as (#1176): the tool and its version, model and effort, the team revision delivered to the checkout and whether it was edited, and the project commit. Most of that has to be read when the session starts, because a later pull changes it.

• Capture at session start. The session-start hook writes <dataHome>/sessions/captures/<platform>/<sessionId>/<captureId>.json: { captureId, platform, sessionId, capturedAt, agent }, where agent comes from [feat] Record the agent each workflow step ran as #1176's builder with driver: "session" and captureId is a random UUID.
• Before the pull. Today the hook dispatcher starts the background handlers, the pull among them, before the foreground ones (hook-dispatch-cli.ts:493-506). Write the capture's pull-record part before that pull starts. The tool version can be added after.
• A pull during the session. When the pull a session started delivers a new revision to that checkout, it marks the session's captures teamChanged: true. The session may have loaded some assets from each revision.
• A resume writes another capture, since the model or the team config can change between the two starts. A SessionStart for compaction, where the tool reports it as such, writes none.
• No capture is written when TEAMAI_LAUNCHED_BY is set. A workflow step is scored as a workflow subject ([feat] Add workflow scorers and recorded quality labels #1085), and a judge is never scored.
• A capture that fails is logged at debug level and never fails the hook.

A label is about the whole session. Its agent is the captured observation only when every capture of the session holds the same observation and none is marked teamChanged. With teamChanged, agent keeps everything but team and fingerprint. With different observations, or none, the label has no agent. model.reported is set only when the session's messages name one model (metadata.model in the session format, ir.ts).

The command

teamai session save --session-id <id> --source <platform> --scorer <name> [--push], on the existing command.

• --source takes the platform names the other session commands use (session migrate --source). It can be left out when the captures name exactly one platform for that id. Otherwise the command fails and lists the platforms it found.
• <name> is a scorer in the team's scorers/ as cached by pull ([feat] Add workflow scorers and recorded quality labels #1085). An unknown or invalid name fails with the list of valid scorers.
• The session is read with AgentSession.find, redacted, and given to the judge as AgentSession.transcript entries (feat(session): safe session reads and copies for workflows, drivers, scorers, and benchmarks (#593 follow-up) #1135). Scoring does not depend on dashboard events: a session TeamAI never saw events for can still be scored. session save still records its summary when it has one, and prints the summary and the scoring outcome separately.
• samplingRate is ignored: an explicit request always scores, including with samplingRate: 0.
• Evidence longer than 200,000 characters keeps the first and last 100,000, and the prompt says how much was cut.
• The command prints the label, the score, whether it passed, and the path of the reasoning file.
• --push publishes every pending label, as below. The --force gate of the summary does not apply to labels.
• session save already honors the global --dry-run (dry-run-guard.ts). Scoring has no preview: with --scorer, --dry-run fails before anything runs, with Session scoring has no --dry-run preview; nothing was run. Without --scorer, --dry-run works as today.

The judge

The judge runs on the member's machine and can do nothing but answer.

• Profiles. This issue keeps an ordered list of judge profiles, one per tool, each used only for installed versions it was verified on. A tool without a verified profile cannot judge. With none available, the command fails: No installed tool can run a scorer judge with every tool turned off. Install a supported version of Claude Code, Codex or OpenCode. Find executables the way the workflow adapter does, not through the private candidate list in utils/ai-client.ts, which has no OpenCode.
• What a profile turns off. Running commands, editing and reading files, web access, subagents, MCP servers, and hooks or plugins from the member's or the project's config. The judge gets the prepared, redacted evidence and nothing else. It keeps only the login the tool needs to reach its model. An empty working directory and a cleared environment help, but they do not meet this on their own.
• Starting points, to be verified by criterion 6:
  • Claude: -p --output-format json --tools "" --strict-mcp-config, with built-in tools and MCP turned off separately.
  • Codex: exec --json --sandbox read-only with approvals off and no MCP servers. Read-only limits the filesystem and does not turn tools off by itself.
  • OpenCode 2: run --format json --standalone with every permission denied, shell and subagents included.
• It runs in a new empty temp directory, removed afterwards, with TEAMAI_LAUNCHED_BY=scorer, and without GITHUB_TOKEN, GH_TOKEN, GITLAB_TOKEN, GL_TOKEN, CI_JOB_TOKEN, SSH_AUTH_SOCK, GIT_ASKPASS and SSH_ASKPASS.
• Model. The scorer's model, resolved for that tool with the alias lookup [feat] Add workflow scorers and recorded quality labels #1085 uses. Without one, the tool's default.
• The answer. TeamAI reads the judge's stdout and writes the label. The judge writes no file. An answer that names no label of the scorer fails the command with a non-zero exit and writes no label. The raw answer is kept in a separate file for failed attempts.

Reuse of #1085

This issue builds on what #1085 shipped. If the scorer parser, sampling, the judge prompt or the answer parsing live in workflow code, move them out here. Session labels carry rubric: { sourceHash, teamRevision? } and judge whether or not #1085 adopted them for workflow labels. Workflow labels written by #1085 are not changed. Nothing here asks a workflow issue to change or wait.

Labels

The label is #1085's, with source: "session":

• subject: { platform, sessionId }.
• agent: as described above, or absent.
• rubric: { sourceHash, teamRevision? }, sourceHash being the hash of the scorer file.
• judge: { tool, model?, usage? }, with usage (inputTokens, outputTokens, costUsd) only as far as the tool reports it.
• The logical key is (scorer, rubric.sourceHash, subject). Scoring the same session with the same rubric again replaces the label. An edited rubric gives a new one.

Where they are kept:

• Locally. <dataHome>/scores/labels.jsonl, one line per label, the newest per key winning when read. The judge's full answer goes to <dataHome>/scores/reasoning/<labelId>.md, where labelId is the sha256 of the canonical logical key. A replacement that fails keeps the earlier label and its reasoning.
• Pending publication. <dataHome>/scores/pending/<labelId>.json, the same outbox pattern as run records (pendingDir and publishPendingRecords in workflow/records.ts). After a push, remove a pending file only if it still holds the label that was sent. A rescore written during the push stays pending.
• Published. scores/<member>/<YYYY-MM>.jsonl on teamai-reports, one JSON line per label, <member> being the username stats use and the month taken from scoredAt in UTC. Write with updateReports and read the base with readReportsOriginFile (utils/reports-branch.ts). Merge by key against origin, keeping the newest scoredAt, so a member's second machine cannot overwrite a newer label with an older one. Readers keep the newest scoredAt per key across all files.
• Opt-out. When usage reporting is off (isUsageReportEnabled), --push publishes nothing and prints Usage reporting is off, so labels stay on this machine. The labels stay pending.

Never published: the transcript, the prompt, the judge's answer and its reasoning.

Acceptance criteria

1. Starting a Claude session in a pulled project writes one capture whose agent.tool.name is claude and whose agent.team.revision is the full commit of that checkout's revision before the session's own pull. Resuming it writes a second capture. Compaction writes none.
2. A session the local workflow driver starts writes no capture.
3. teamai session save --session-id <id> --source claude-code --scorer <name> prints a label, adds one line to labels.jsonl and one reasoning file, and changes nothing on teamai-reports. It works for a native Codex session TeamAI has no dashboard events for.
4. With --push, scores/<member>/<YYYY-MM>.jsonl on origin has that label. A phrase taken from the session's transcript does not appear anywhere on the reports branch.
5. Scoring the same session again with the same scorer leaves one line for that key on origin, with the newer scoredAt. After editing the scorer's prompt, the next score adds a second line.
6. For each profile, the effective configuration has every capability above turned off. A fixture transcript that tells the judge to run a command, read a file outside the evidence, edit a file and call an MCP server or subagent gets each attempt denied, and no private content appears in the answer. An installed version that does not honor the profile is reported unavailable. A judge that merely chose not to call a tool does not pass this criterion.
7. With no profiled tool available, the command fails with the message above and writes no label.
8. With usage reporting off, --push prints the message above and publishes nothing.
9. A session whose two captures differ in model gives a label with no agent. A session whose pull delivered a new revision mid-session gives a label with agent but no team and no fingerprint.
10. When the push fails, for example offline, the label stays pending, and the next --push publishes it. Two clones of the same member pushing labels for the same month keep the newest per key. A label written during a push stays pending.
11. Session labels work against workflow labels in the shape [feat] Add workflow scorers and recorded quality labels #1085 shipped before its label comment was adopted, without changing them.
12. teamai --dry-run session save --session-id <id> --source claude-code --scorer <name> --push exits non-zero with the message above, launches no judge, writes no summary, label, reasoning or pending file, and publishes nothing.

Test plan

• Unit tests for capture writing, skipping and marking, the agent rule across captures, the logical key, the merge against origin and the outbox.
• Judge launch tests with a fake executable that records argv and environment, and checks of each profile's effective configuration.
• E2E with a fixture session and a fake tool for criteria 3 to 5, 8, 10 and 11.
• Real-CLI runs for criterion 6 on each tool available locally, recorded in the PR. Say which tools were not run.

Watch out for

• The session-start hook must stay fast. Writing the capture's pull-record part first must not wait on --version.
• docs/guide/ and docs/guide/zh-CN/: add a section on scoring a session to the scorer guide [feat] Add workflow scorers and recorded quality labels #1085 adds. Regenerate skill-data/core/references/commands.md with npx vitest run commands-reference -u.

Out of scope

Scoring part of a session, background scoring (#1179), scoring subagents, showing labels (#1182), and a spending budget beyond #1179's daily limit.

Dependencies

• Blocked by: [feat] Add workflow scorers and recorded quality labels #1085 (scorer files, sampling, the judge prompt and answer parsing, the label type), [feat] Record the agent each workflow step ran as #1176 (the agent observation)
• Unblocks: [feat] Score finished sessions in the background #1179, [feat] Show scorer results in the dashboard #1182