Skip to content

feat(workflow): record the agent each step pass ran as - #1184

Merged
jeff-r2026 merged 9 commits into
Tencent:mainfrom
SaulMoro:feat/1176-agent-observation
Oct 17, 2026
Merged

jeff-r2026 merged 9 commits into
Tencent:mainfrom
SaulMoro:feat/1176-agent-observation

Conversation

@SaulMoro

@SaulMoro SaulMoro commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Closes #1176.

A run record entry's agent named the definition a step selected. It now holds what the pass ran as. The local driver observes it at launch, and the engine filters it the same way for every driver.

 promptReport                       local-driver.ts
+  observeLaunch                    definition and how the tool applied it, tool version,
+                                   requested model, team commit, project commit
+  workflowLaunch                   TEAMAI_LAUNCHED_* in the tool's environment
   adapter.run
-    claude -p --output-format json
+    claude -p --output-format stream-json --verbose
-  report.model = result.model ?? the definition's model
+  report.model = the model the tool reported, else none
+  report.agent = the observation, when the tool started
 reportAttempt                      engine.ts
+  settleObservation                drops keys and values not shaped like names, sets teamai,
+                                   driver and definition.name, writes the fingerprint
 buildEndRecord                     records.ts
-  steps[].agent = "reviewer"
+  steps[].agent = { v: 1, definition, tool, model, effort, team, project, teamai, driver, fingerprint }

 session-start hook of the launched tool, or of a session it starts in the step's checkout
   pullHandler                      hook-handlers.ts
+    skip when launchHoldsCheckout  the checkout holds one complete delivery for the launched tool

Calls the issue left open, and one place this departs from it:

  • Team commit. A pushBaseRevs entry that resolves to the same commit as rev is not a second revision. A pending full sync counts as an incomplete delivery, so it gives no team.
  • Claude's reported model. --output-format json prints no model, only modelUsage, which keys every model the session billed. A real session that ran sonnet billed a haiku side call beside it. The driver now reads stream-json, whose init event names the session's model. With no init event the pass has no reported model. The result, tokens, session id and the result file fallback are read as before.
  • Requested effort. None is recorded for a local launch, which departs from the issue's table. Claude does not apply an agent file's effort: under --agent (measured through the request body on 2.1.301), and OpenCode 2 takes an effort only as the model's #variant. The file's value is not what ran, and it would split the fingerprint of identical runs. An effort the launch passes on argv can be recorded once a launch passes one.
  • OpenCode. A pass on OpenCode records the definition's name, with no applied and no model from the file. OpenCode 1 runs its default agent in place of a mode: subagent file, which is how pull deploys agents, and OpenCode 2 runs the file's prompt without its model or variant ([bug] A workflow step on OpenCode does not run as its agent #1193). Neither is what the file says. feat(workflow): let a prompt step choose its model (#1177) #1192 makes OpenCode run the definition, and the PR that merges second records how.
  • Which sessions skip the pull. While the checkout holds a complete delivery for the launched tool, that tool and every session it starts in the step's checkout skip the pull. Otherwise a nested tool's pull moves the step's files to a newer team commit, which a real run showed. A session started outside the step still pulls.

Evidence

  • Before (main, read from the code): the entry is "agent": "reviewer", and its flat model falls back to the definition's model when the tool names none.

    After (real CLI with Claude, a build of this branch): teamai pull, then teamai workflow start review --driver local. The review entry on teamai-reports:

    {
      "step": "review",
      "pass": 1,
      "agent": {
        "v": 1,
        "definition": { "name": "reviewer", "applied": "main" },
        "tool": { "name": "claude", "version": "2.1.301" },
        "model": { "requested": "claude-sonnet-5-5", "reported": "claude-sonnet-5-5" },
        "team": { "revision": "6ef313dab8792aabe51030c9a155d7a14bec0dac", "localChanges": false },
        "project": { "commit": "136a1791b31671ab4a842d763c0206d56d1c5094", "dirty": false },
        "teamai": "0.22.0",
        "driver": "local",
        "fingerprint": "1ed3d45f5c8f72706b76925e1181617afcb2361a515e0ced50a65f10b5c2d4e9"
      },
      "tool": "claude",
      "model": "claude-sonnet-5-5",
      "outcome": "success",
      "tokens": { "input": 6, "output": 409, "cachedInput": 51326 }
    }

    claude --version prints 2.1.301. In the team clone, git rev-parse 6ef313d, the pull record's rev, gives team.revision. The deployed reviewer.md has model: claude-sonnet-5-5. The session also billed a haiku side call, and the entry still names sonnet and only its tokens. The step's output came through the stream launch: the run state holds {"result": "OK"}, the output the session wrote to result.json. teamai is 0.22.0 because a source build reads the repository's package.json.

  • Criterion 8, real CLI. With a newer commit on origin, a workflow run left the checkout record's rev and the delivered rule unchanged. A plain claude -p in the same checkout pulled the new commit. A codex session-start hook under the step's launch also left rev and the rule unchanged, and the same hook without the launch pulled.

  • Requested effort, real CLI. A team alias wrote model: claude-haiku-5-5 and effort: low into the deployed reviewer.md, and agentModels records both. The published entry has no effort, and its agent.model is { "requested": "claude-haiku-5-5", "reported": "claude-haiku-5-5", "alias": "smart" }.

  • OpenCode, real CLI. OpenCode 1.18.23 ran against a local server posing as the Anthropic API, which logs each request body. The team alias wrote mode: subagent and model: anthropic/claude-sonnet-4-5 into the deployed reviewer.md.

    Both runs sent only the configured default claude-haiku-4-5, plus OpenCode's own title request, and no request carried the reviewer's prompt. OpenCode's session store names its default agent build. The step fails because the fake API answers with an error, and the record is still published.

    Before, with the previous commit of this branch, the record gave "definition": { "name": "reviewer", "applied": "main" }, "model": { "requested": "anthropic/claude-sonnet-4-5", "alias": "smart" } and a fingerprint. After:

    "agent": {
      "v": 1,
      "definition": { "name": "reviewer" },
      "tool": { "name": "opencode" },
      "team": { "revision": "2f0944dd7261cdfdcffaf0ba5fd25000ff125186", "localChanges": false },
      "project": { "commit": "136a1791b31671ab4a842d763c0206d56d1c5094", "dirty": false },
      "teamai": "0.22.0",
      "driver": "local"
    }

    tool.version is absent because teamai has no OpenCode version detector, as the issue specifies. OpenCode 1 waits for stdin to close, and the launch leaves it open, as it does on main. These runs went through a wrapper that gives opencode /dev/null. feat(workflow): let a prompt step choose its model (#1177) #1192 fixes the launch.

  • Criterion 12. npx teamai-cli@0.28.0-beta.5 pull drained a record this change had queued and merged it into teamai-reports. Each step's agent is identical in the queued and the merged file (jq -S, then diff). 0.27.0, the npm latest, has no workflow command, so 0.28.0-beta.5 is the previous release with run records.

  • Tests. Criteria 1 to 11 are covered in workflow-local-driver.test.ts, agent-observation.test.ts, team-delivery.test.ts, launch-context.test.ts, hook-dispatch-cli.test.ts, workflow-engine.test.ts and workflow-records.test.ts. The e2e workflow-records-reports-branch-1083.test.ts reads the review entry's agent from teamai-reports on the bare origin.

    npx vitest run: 463 files and 10897 tests pass. The e2e passes. npx tsc --noEmit and npm run lint exit 0.

  • Not covered locally. Real runs on Codex, CodeBuddy and OpenCode 2, an OpenCode run against a real model, the gitlab and github providers, and Windows. Left to CI.

Known limits:

  • A session in the same checkout that the workflow did not launch still pulls.
  • teamai's git hooks still pull. A prompt step that triggers them changes the repository and fails as read-only.
  • A nested tool the checkout was never delivered to gets its first delivery at its next session outside the step.
  • A skipped pull leaves queued learnings for the next pull.
  • No local launch requests an effort and no current tool reports one, so a local pass has no effort.
  • An OpenCode pass has no applied, no requested model, and so no fingerprint unless OpenCode reports a model, until OpenCode runs the definition ([bug] A workflow step on OpenCode does not run as its agent #1193).
  • feat(workflow): add workspace branches #1175 runs steps in worktrees with their own pull record. Whichever of the two merges second passes the worktree's config to loadAgent, observeLaunch and workflowLaunch, or a worktree step records the member checkout's delivery (details on [feat] Add workflow workspaces, branches, and session continuation #1080).
  • localChanges hashes each delivered file once per pass. I did not measure that on a large delivery.

Merge Danger

Door: two-way

steps[].agent changes from a string to an object. This change reads an older string as { definition: { name } }, and the previous release merges the object as it is. A revert writes strings again. Records already published keep their objects.

Blast Radius: moderate

Every local Claude prompt step now launches with --output-format stream-json --verbose, and a Claude build that rejects those flags fails the step. A reader of the reports branch that expects steps[].agent to be a string gets an object. A session the step's tool starts no longer pulls when its checkout is fully delivered.

…for a tool

The session-start pull skip and the recorded team revision of a workflow pass
both hang on this one predicate (Tencent#1176).
@github-actions

Copy link
Copy Markdown

No confident findings in the specified PR diff.

The PR description satisfies the testing requirement: it documents a representative real-CLI run with Claude, session-start pull verification, and previous-release record compatibility. Additional provider and agent coverage is explicitly deferred to CI.

Review was read-only; I did not run, build, or install anything from the PR.

@SaulMoro
SaulMoro force-pushed the feat/1176-agent-observation branch from 35b9dc5 to 253f509 Compare October 16, 2026 14:23
@github-actions

Copy link
Copy Markdown

No confident findings in the specified PR diff. The earlier review raised no findings, so there are none to mark resolved.

The PR description satisfies the testing requirement: it documents a representative real-CLI Claude run, session-start pull verification, and previous-release record compatibility. Additional provider and agent coverage is explicitly deferred to CI.

Review was read-only. I did not run, build, or install anything from the PR.

One module holds the shape, the filter that keeps only name-shaped values,
the fingerprint and the reader of older string agents (Tencent#1176). The model
name rule moves here so the record and the observation share it.
…a Claude session ran

claude -p --output-format json prints no model, only modelUsage, which keys
every model the session billed: a side call or a subagent shows up beside the
model that ran. The adapter now reads stream-json, whose init event names the
session's model (Tencent#1176). It also says whether the tool spawned and how a
launch hands it its definition, which the agent observation of a pass needs.
A driver's report may carry agent, which the engine settles with what it
owns (definition name, CLI version, driver kind) and the name filter before
saving it with the pass. A run record's step entry publishes it, and an
entry nothing observed keeps the definition's name (Tencent#1176).
The local driver observes the tool, the definition handoff, the team
delivery and the project head before it starts the tool, and reports the
observation of a pass whose tool started. The flat model is now only what
the tool reported: the definition's model is what the launch asked for, and
the observation records it as requested (Tencent#1176).
…ool runs

A tool the local driver launches starts a session whose hook pulls. The
driver now launches it with TEAMAI_LAUNCHED_BY=workflow, bound to the tool
and the checkout, and the dispatcher forwards that launch to the background
pull in its payload, which also reaches a child started through WMI. While
the checkout holds one complete team delivery for the launched tool, every
session that tool starts there skips the pull, so the step keeps the team
commit it records. A new worktree, an incomplete delivery, or a launched
tool the checkout was not delivered to still pulls. A nested tool the
checkout was never delivered to gets its first delivery at its next session
outside the step (Tencent#1176).
@SaulMoro
SaulMoro force-pushed the feat/1176-agent-observation branch from 253f509 to 5a3281b Compare October 16, 2026 23:01
@github-actions

Copy link
Copy Markdown

No confident findings in the specified PR diff. Earlier reviews raised no findings, so there are none to mark resolved.

The PR description satisfies the testing requirement: it documents a representative real-CLI Claude run, session-start pull verification, and previous-release record compatibility. Additional provider and agent coverage is explicitly deferred to CI.

Review was read-only. I did not run, build, or install anything from the PR.

…Code step (Tencent#1176)

OpenCode 1 runs its default agent in place of a `mode: subagent` file,
which is how pull deploys agents, and OpenCode 2 runs the file's prompt
without its model or variant (Tencent#1193). A pass on OpenCode recorded
`definition.applied: main` and the file's model as `model.requested`,
which did not describe what ran. It now records neither.
@github-actions

Copy link
Copy Markdown

No confident findings in the specified PR diff. Earlier reviews raised no findings, so there are none to mark resolved.

The PR description satisfies the testing requirement: it documents a representative real-CLI Claude run, session-start pull verification, and previous-release record compatibility. Additional provider and agent coverage is explicitly deferred to CI.

Review was read-only. I did not run, build, or install anything from the PR.

@jeff-r2026
jeff-r2026 merged commit 3281eda into Tencent:main Oct 17, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feat] Record the agent each workflow step ran as

2 participants