Skip to content

[feat] Record the agent each workflow step ran as #1176

Closed

Description

@SaulMoro

Parent: #818

Goal

A run record says a step ran as reviewer. That is the name of the agent definition the step selected (records.ts:173), not the agent that did the work. The work came from a composition: the definition, if any, and how the tool applied it; the tool and its version; the model and effort; the team config delivered to that checkout; and the project commit. Two passes of the same step can differ in any of these, and today the record cannot tell them apart.

Record that composition for every pass, as observed, so a result can be traced to what produced it and two results can be compared.

From this issue on, agent in run records, labels and stats means this composition. The YAML key agent: on a step keeps its meaning: it selects a definition.

Shape

/** What one pass of a step ran as. Every field is best effort: absent when not observed, never inferred. */
type AgentObservation = {
  v: 1;
  definition?: { name: string; applied?: 'main' | 'prompt' | 'subagent' };
  tool: { name: string; version?: string };
  model?: { requested?: string; reported?: string; alias?: string };
  effort?: { requested?: string; reported?: string };
  team?: { revision: string; localChanges: boolean };
  project?: { commit: string; dirty: boolean };
  teamai: string;
  driver: string;
  fingerprint?: string;
};
Field Source
definition.name The step's agent:. Absent on a step without one.
definition.applied How the local driver passed the definition: main for Claude and OpenCode (--agent), prompt for Codex, where the instructions are prepended to the prompt (local-tool-adapter.ts). subagent is for drivers that hand the step to a subagent (#1086).
tool.name The tool the driver launched.
tool.version getAgentVersion (src/agent-version.ts) for the tools it has a detector for. Absent for the others. Add a detector only where a criterion below needs it, and never report one binary's version under another's name.
model.requested, effort.requested, model.alias What the launch asked the tool for. For Codex, the --model the adapter passes. For Claude and OpenCode, which read the deployed agent file themselves, the resolution in lastPullByWorkspace[<checkout>].agentModels[<stem>][<tool>] (types.ts), used only while that agent file still has its delivered hash. If the member edited the file, leave these absent. Effort is absent where the launch does not pass one.
model.reported, effort.reported The tool's own output after it exits (result.model today). Absent when the tool does not report it.
team.revision rev of the lastPullByWorkspace entry for the checkout the tool runs in, keyed with checkoutKey (pull.ts:789), resolved in the team clone to the full commit with getHeadCommit (utils/git.ts:1427). The record stores the short hash getHeadRev returns (utils/git.ts:283), and published records name a team commit in full. When it does not resolve, leave team out.
team.localChanges true when any path in that entry's delivered no longer has the recorded sha256.
project getHeadState(cwd) (utils/git.ts:1446): commit is its head. dirty covers tracked files only.
teamai getCurrentVersion() (package-info.ts).
driver The driver kind: local, or the kind an external driver leases with.
fingerprint See below.

When team is left out. A pull record does not always name one revision. A partial or inherited delivery keeps the earlier rev and adds the newer one to pushBaseRevs while updating delivered (pull.ts). Leave team out when the entry is missing, its rev is empty, it has no delivered, or its pushBaseRevs is not empty.

Fingerprint. The sha256 of the canonical JSON (sorted keys, no whitespace) of { v, definition, tool, model, effort, team } as recorded, so a requested-only model and a reported one never hash alike. Write it only when tool.name, a model and team are present and team.localChanges is false. It claims only that these recorded parts are equal, not that the whole environment was: the project's own config, the member's own config and hook context are outside it. project, teamai and driver are not hashed. The project commit changes with every unrelated change, so it is published next to the fingerprint for readers who need to split by it.

What gets published. The observation crosses into public records. Keep only the keys above, and drop any tool.version, model or effort value that is not shaped like a name, the way MODEL_NAME already filters models (records.ts:40-45). A path or free text sent by a driver is dropped, not published.

Scope

  • Add src/agent-observation.ts with the type, its Zod schema, a function that builds the observation for a launch, the fingerprint, and a reader that turns an older string agent into { definition: { name } }. That reader returns a partial value, not a full observation. The module lives outside src/workflow/ because [Proposal] Score agent sessions and run benchmarks as workflows #1137 captures sessions with the same function.
  • In the local driver, build the observation right before adapter.run (local-driver.ts:293), after the definition is loaded, and add model.reported and effort.reported from the result. A failed attempt reports its observation too. A step that launches no AI tool gets none.
  • Add agent to the fields a report carries (step-state.ts, report), persisted with the pass like tool and model. Passes moved to passes or run.archive (feat(workflow): bounded loops, parallel for_each and list expressions #1148) keep theirs.
  • Accept agent from any driver's report, filtered as above. The engine sets definition.name, teamai and driver itself and ignores a driver's values for them.
  • In the run record, each step entry's agent becomes the observation of that pass. Keep the flat tool and model on the entry for current readers.
  • Stop mixing requested and reported in the flat model: local-driver.ts:316 falls back from result.model to the definition's model. Report only what the tool reported. Update workflow-local-driver.test.ts:126-155, which asserts the fallback.
  • Stop the background pull from changing a prepared step's checkout while it runs. A tool the local driver launches starts its own session, and its session-start hook runs teamai pull (pullHandler, hook-handlers.ts:113). Set TEAMAI_LAUNCHED_BY=workflow in the tool's environment, next to TEAMAI_WORKFLOW_RESULT (local-tool-adapter.ts:272). When it is set, pullHandler skips the pull only if the checkout's pull record shows a complete delivery for the tool being launched: a non-empty rev, a delivered baseline, no pushBaseRevs, and targets that include the tool. Otherwise it pulls as today, root seeding included, so a new worktree or a newly enabled tool still gets the team's assets, and the observation has no team. A newer commit on origin does not count as incomplete. [Proposal] Score agent sessions and run benchmarks as workflows #1137's judge sets the same variable to scorer, which always skips the pull.
  • docs/guide/workflow.md and docs/guide/zh-CN/workflow.md: describe agent in the run records list.

Acceptance criteria

  1. teamai workflow start <name> --driver local with a prompt step agent: reviewer on Claude publishes a run record whose entry for that pass has agent.definition { name: "reviewer", applied: "main" }, agent.tool.name claude, agent.tool.version equal to the number claude --version prints, agent.team.revision equal to the full commit of the checkout's last pulled revision, and agent.model.requested equal to the model in the deployed agent file.
  2. The same step on Codex has applied: "prompt" and model.requested equal to the --model value the adapter passed.
  3. A prompt step without agent: has no definition.
  4. A tool output that names no model gives an entry with no agent.model.reported and no flat model, while agent.model.requested is still there.
  5. After editing a delivered skill file in the checkout, the next run records team.localChanges: true and no fingerprint. After editing the deployed agent file's model, model.requested is absent.
  6. A pull record with a non-empty pushBaseRevs, or without delivered, gives no team and no fingerprint.
  7. Two runs of the same step, with the same tool, model and team revision and no local changes, have the same fingerprint. Changing the model gives a different one. A checkout with no pull record gives none.
  8. With a new commit on the team repo's origin, a run in a checkout with a complete delivery record for its tool leaves lastPullRev and the delivered files unchanged. A worktree with no record, a record left incomplete by a pull, and a record whose targets omit the tool still go through the existing delivery when the tool starts, and their observation has no team or fingerprint.
  9. A step a loop ran three times has three entries, each with its own agent.
  10. A run record published before this change, with a string agent, still merges, and its agent reads as { definition: { name } }.
  11. An external driver's report whose agent sets teamai, or a tool.version that is a file path, publishes the engine's teamai and no version.
  12. A record written by this change is merged by the previous release's reports merge (mergeRecord keeps unknown values as they are, records.ts:227-230). Check it once by hand with npx teamai-cli@<previous> against the same reports branch, and note it in the PR. Resuming a run with an older CLI is not covered: its state schema drops unknown fields.

Test plan

  • Unit tests for the observation builder with a temp checkout, a real state.json pull record and a fake tool executable that prints a version.
  • Fingerprint tests: key order does not matter, each missing input removes it, requested and reported hash differently.
  • Extend workflow-local-driver.test.ts for criteria 1 to 6 and 9, and the record tests for 10 and 11.
  • E2E: extend workflow-records-reports-branch-1083.test.ts to read agent from the record on origin.
  • One real-CLI run for the PR body with Claude: teamai pull, teamai workflow start <name> --driver local, then criterion 1 checked on the published record.

Watch out for

Out of scope

Dependencies

Activity

  1. self-assigned this
    on Oct 16, 2026
  2. SaulMoro commented on Oct 16, 2026

    @SaulMoro
    CollaboratorAuthor

    Two things from #1177 (#1192) that this record needs to get right.

    Claude never applies an agent file's effort: under --agent. Measured on Claude Code 2.1.301, reading each request body through a local ANTHROPIC_BASE_URL:

    Agent file Flags model sent output_config.effort sent
    model: haiku, effort: low --agent claude-haiku-5-5 medium
    model: haiku, no effort --agent claude-haiku-5-5 medium
    model: opus, effort: max --agent claude-opus-5-5 high
    model: opus, effort: low --agent claude-opus-5-5 high
    no agent --model opus claude-opus-5-5 high
    model: opus, effort: max --agent --effort low claude-opus-5-5 low

    observeLaunch in #1184 takes effort.requested from the pull record's agentModels for a main launch. On Claude that names an effort the session did not run at, and it goes into the fingerprint. For Claude, effort.requested should come only from an --effort the launch passed.

    A step's model replaces the definition's model and effort. promptReport gets them from chooseStepModel (src/workflow/step-model.ts) as chosen.model, { name, effort? }, and passes them on argv. When that is set, model.requested and effort.requested come from it, not from the pull record. An alias with no effort for the tool runs at the tool's default effort, so effort.requested is absent then, even when the agent file sets one.

  3. SaulMoro commented on Oct 16, 2026

    @SaulMoro
    CollaboratorAuthor

    The effort part is fixed in #1184: observeLaunch no longer records an effort for any local launch. Besides the Claude measurement above, OpenCode 2 takes an effort only as the model's #variant, while the agent file teamai writes for it carries a separate variant: key, so that value is not confirmed to apply either. With no launch passing an effort on argv today, a local pass has no effort, and the fingerprint no longer splits on a value the session did not run at.

    The step model part needs chooseStepModel, which only #1192 has. Whichever of #1184 and #1192 merges second hands observeLaunch the chosen { name, effort? }, so that model.requested and effort.requested come from it when it is set. For Claude, that effort is the --effort the launch passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions