Repository navigation
[feat] Record the agent each workflow step ran as #1176
Description
Parent: #818
Goal
A run record says a step ran as reviewer. That is the name of the agent definition the step selected (records.ts:173), not the agent that did the work. The work came from a composition: the definition, if any, and how the tool applied it; the tool and its version; the model and effort; the team config delivered to that checkout; and the project commit. Two passes of the same step can differ in any of these, and today the record cannot tell them apart.
Record that composition for every pass, as observed, so a result can be traced to what produced it and two results can be compared.
From this issue on, agent in run records, labels and stats means this composition. The YAML key agent: on a step keeps its meaning: it selects a definition.
Shape
/** What one pass of a step ran as. Every field is best effort: absent when not observed, never inferred. */
type AgentObservation = {
v: 1;
definition?: { name: string; applied?: 'main' | 'prompt' | 'subagent' };
tool: { name: string; version?: string };
model?: { requested?: string; reported?: string; alias?: string };
effort?: { requested?: string; reported?: string };
team?: { revision: string; localChanges: boolean };
project?: { commit: string; dirty: boolean };
teamai: string;
driver: string;
fingerprint?: string;
};| Field | Source |
|---|---|
definition.name |
The step's agent:. Absent on a step without one. |
definition.applied |
How the local driver passed the definition: main for Claude and OpenCode (--agent), prompt for Codex, where the instructions are prepended to the prompt (local-tool-adapter.ts). subagent is for drivers that hand the step to a subagent (#1086). |
tool.name |
The tool the driver launched. |
tool.version |
getAgentVersion (src/agent-version.ts) for the tools it has a detector for. Absent for the others. Add a detector only where a criterion below needs it, and never report one binary's version under another's name. |
model.requested, effort.requested, model.alias |
What the launch asked the tool for. For Codex, the --model the adapter passes. For Claude and OpenCode, which read the deployed agent file themselves, the resolution in lastPullByWorkspace[<checkout>].agentModels[<stem>][<tool>] (types.ts), used only while that agent file still has its delivered hash. If the member edited the file, leave these absent. Effort is absent where the launch does not pass one. |
model.reported, effort.reported |
The tool's own output after it exits (result.model today). Absent when the tool does not report it. |
team.revision |
rev of the lastPullByWorkspace entry for the checkout the tool runs in, keyed with checkoutKey (pull.ts:789), resolved in the team clone to the full commit with getHeadCommit (utils/git.ts:1427). The record stores the short hash getHeadRev returns (utils/git.ts:283), and published records name a team commit in full. When it does not resolve, leave team out. |
team.localChanges |
true when any path in that entry's delivered no longer has the recorded sha256. |
project |
getHeadState(cwd) (utils/git.ts:1446): commit is its head. dirty covers tracked files only. |
teamai |
getCurrentVersion() (package-info.ts). |
driver |
The driver kind: local, or the kind an external driver leases with. |
fingerprint |
See below. |
When team is left out. A pull record does not always name one revision. A partial or inherited delivery keeps the earlier rev and adds the newer one to pushBaseRevs while updating delivered (pull.ts). Leave team out when the entry is missing, its rev is empty, it has no delivered, or its pushBaseRevs is not empty.
Fingerprint. The sha256 of the canonical JSON (sorted keys, no whitespace) of { v, definition, tool, model, effort, team } as recorded, so a requested-only model and a reported one never hash alike. Write it only when tool.name, a model and team are present and team.localChanges is false. It claims only that these recorded parts are equal, not that the whole environment was: the project's own config, the member's own config and hook context are outside it. project, teamai and driver are not hashed. The project commit changes with every unrelated change, so it is published next to the fingerprint for readers who need to split by it.
What gets published. The observation crosses into public records. Keep only the keys above, and drop any tool.version, model or effort value that is not shaped like a name, the way MODEL_NAME already filters models (records.ts:40-45). A path or free text sent by a driver is dropped, not published.
Scope
- Add
src/agent-observation.tswith the type, its Zod schema, a function that builds the observation for a launch, the fingerprint, and a reader that turns an older stringagentinto{ definition: { name } }. That reader returns a partial value, not a full observation. The module lives outsidesrc/workflow/because [Proposal] Score agent sessions and run benchmarks as workflows #1137 captures sessions with the same function. - In the local driver, build the observation right before
adapter.run(local-driver.ts:293), after the definition is loaded, and addmodel.reportedandeffort.reportedfrom the result. A failed attempt reports its observation too. A step that launches no AI tool gets none. - Add
agentto the fields areportcarries (step-state.ts,report), persisted with the pass liketoolandmodel. Passes moved topassesorrun.archive(feat(workflow): bounded loops, parallel for_each and list expressions #1148) keep theirs. - Accept
agentfrom any driver'sreport, filtered as above. The engine setsdefinition.name,teamaianddriveritself and ignores a driver's values for them. - In the run record, each step entry's
agentbecomes the observation of that pass. Keep the flattoolandmodelon the entry for current readers. - Stop mixing requested and reported in the flat
model:local-driver.ts:316falls back fromresult.modelto the definition's model. Report only what the tool reported. Updateworkflow-local-driver.test.ts:126-155, which asserts the fallback. - Stop the background pull from changing a prepared step's checkout while it runs. A tool the local driver launches starts its own session, and its session-start hook runs
teamai pull(pullHandler,hook-handlers.ts:113). SetTEAMAI_LAUNCHED_BY=workflowin the tool's environment, next toTEAMAI_WORKFLOW_RESULT(local-tool-adapter.ts:272). When it is set,pullHandlerskips the pull only if the checkout's pull record shows a complete delivery for the tool being launched: a non-emptyrev, adeliveredbaseline, nopushBaseRevs, andtargetsthat include the tool. Otherwise it pulls as today, root seeding included, so a new worktree or a newly enabled tool still gets the team's assets, and the observation has noteam. A newer commit on origin does not count as incomplete. [Proposal] Score agent sessions and run benchmarks as workflows #1137's judge sets the same variable toscorer, which always skips the pull. docs/guide/workflow.mdanddocs/guide/zh-CN/workflow.md: describeagentin the run records list.
Acceptance criteria
teamai workflow start <name> --driver localwith a prompt stepagent: revieweron Claude publishes a run record whose entry for that pass hasagent.definition{ name: "reviewer", applied: "main" },agent.tool.nameclaude,agent.tool.versionequal to the numberclaude --versionprints,agent.team.revisionequal to the full commit of the checkout's last pulled revision, andagent.model.requestedequal to themodelin the deployed agent file.- The same step on Codex has
applied: "prompt"andmodel.requestedequal to the--modelvalue the adapter passed. - A prompt step without
agent:has nodefinition. - A tool output that names no model gives an entry with no
agent.model.reportedand no flatmodel, whileagent.model.requestedis still there. - After editing a delivered skill file in the checkout, the next run records
team.localChanges: trueand nofingerprint. After editing the deployed agent file'smodel,model.requestedis absent. - A pull record with a non-empty
pushBaseRevs, or withoutdelivered, gives noteamand nofingerprint. - Two runs of the same step, with the same tool, model and team revision and no local changes, have the same
fingerprint. Changing the model gives a different one. A checkout with no pull record gives none. - With a new commit on the team repo's origin, a run in a checkout with a complete delivery record for its tool leaves
lastPullRevand the delivered files unchanged. A worktree with no record, a record left incomplete by a pull, and a record whosetargetsomit the tool still go through the existing delivery when the tool starts, and their observation has noteamorfingerprint. - A step a loop ran three times has three entries, each with its own
agent. - A run record published before this change, with a string
agent, still merges, and itsagentreads as{ definition: { name } }. - An external driver's
reportwhoseagentsetsteamai, or atool.versionthat is a file path, publishes the engine'steamaiand no version. - A record written by this change is merged by the previous release's reports merge (
mergeRecordkeeps unknown values as they are,records.ts:227-230). Check it once by hand withnpx teamai-cli@<previous>against the same reports branch, and note it in the PR. Resuming a run with an older CLI is not covered: its state schema drops unknown fields.
Test plan
- Unit tests for the observation builder with a temp checkout, a real
state.jsonpull record and a fake tool executable that prints a version. - Fingerprint tests: key order does not matter, each missing input removes it, requested and reported hash differently.
- Extend
workflow-local-driver.test.tsfor criteria 1 to 6 and 9, and the record tests for 10 and 11. - E2E: extend
workflow-records-reports-branch-1083.test.tsto readagentfrom the record on origin. - One real-CLI run for the PR body with Claude:
teamai pull,teamai workflow start <name> --driver local, then criterion 1 checked on the published record.
Watch out for
- [feat] Define the team revision once for run records, labels and snapshots #1163 and this issue both edit
records.ts. Rebase onto whichever merges first. - [feat] Tell a driver which payload fields next and report ignored #1164 lists the payload fields a driver sent and the engine ignored. Whichever of the two merges second adds
agentto the acceptedreportfields. - [feat] Add workflow workspaces, branches, and session continuation #1080 runs steps in worktrees. If its manager prepares each worktree through pull before the tool starts, as its comment on the worktree hook suggests, that worktree has a complete record and the tool's own pull is skipped. The mere existence of a record is not preparation. Either issue can merge first.
- [feat] Publish and resume complete workflow state across machines #1084 publishes state snapshots. The observation has no paths and nothing about the member's own config, so it can be published as it is.
Out of scope
- The member's own config (the tool's user instruction file and user MCP config). It is private, and nothing reads it yet.
- The text context hooks add at session start.
- Sessions outside workflows and showing the data ([Proposal] Score agent sessions and run benchmarks as workflows #1137).
- Telling a driver which model to use ([feat] Let a workflow choose the model of its prompt steps #1177).
Dependencies
- Blocked by: none
- Unblocks: [feat] Score a finished session on demand #1178, [feat] Run benchmark suites as workflows #1180
Activity
feat(workflow): tell whether a local tool started and read the model …
feat(workflow): record the agent each pass ran as
feat(workflow): observe the launch of each local prompt step
feat(hooks): keep a workflow step's checkout as delivered while its t…
test(workflow): read a prompt step's agent from the record on origin (T…
docs(workflow): describe the agent a run record entry carries (Tencen…
SaulMoro commentedon Oct 16, 2026
Two things from #1177 (#1192) that this record needs to get right.
Claude never applies an agent file's
effort:under--agent. Measured on Claude Code 2.1.301, reading each request body through a localANTHROPIC_BASE_URL:modelsentoutput_config.effortsentmodel: haiku,effort: low--agentclaude-haiku-5-5mediummodel: haiku, no effort--agentclaude-haiku-5-5mediummodel: opus,effort: max--agentclaude-opus-5-5highmodel: opus,effort: low--agentclaude-opus-5-5high--model opusclaude-opus-5-5highmodel: opus,effort: max--agent --effort lowclaude-opus-5-5lowobserveLaunchin #1184 takeseffort.requestedfrom the pull record'sagentModelsfor amainlaunch. On Claude that names an effort the session did not run at, and it goes into the fingerprint. For Claude,effort.requestedshould come only from an--effortthe launch passed.A step's
modelreplaces the definition's model and effort.promptReportgets them fromchooseStepModel(src/workflow/step-model.ts) aschosen.model,{ name, effort? }, and passes them on argv. When that is set,model.requestedandeffort.requestedcome from it, not from the pull record. An alias with no effort for the tool runs at the tool's default effort, soeffort.requestedis absent then, even when the agent file sets one.feat(workflow): add the agent observation of a step pass
feat(workflow): tell whether a local tool started and read the model …
feat(workflow): record the agent each pass ran as
feat(workflow): observe the launch of each local prompt step
feat(hooks): keep a workflow step's checkout as delivered while its t…
test(workflow): read a prompt step's agent from the record on origin (T…
docs(workflow): describe the agent a run record entry carries (Tencen…
SaulMoro commentedon Oct 16, 2026
The effort part is fixed in #1184:
observeLaunchno longer records an effort for any local launch. Besides the Claude measurement above, OpenCode 2 takes an effort only as the model's#variant, while the agent file teamai writes for it carries a separatevariant:key, so that value is not confirmed to apply either. With no launch passing an effort on argv today, a local pass has noeffort, and the fingerprint no longer splits on a value the session did not run at.The step model part needs
chooseStepModel, which only #1192 has. Whichever of #1184 and #1192 merges second handsobserveLaunchthe chosen{ name, effort? }, so thatmodel.requestedandeffort.requestedcome from it when it is set. For Claude, that effort is the--effortthe launch passes.fix(workflow): record no definition handoff or file model for an Open…
feat(workflow): record the agent each step pass ran as (#1184)