Skip to content

Write negotiation as a standard tool-use agent: one toolkit, native function calling, MCP for host CLIs - #106

Merged
faj-design5260 merged 9 commits into
mainfrom
feat/native-tool-reasoners
Oct 15, 2026
Merged

faj-design5260 merged 9 commits into
mainfrom
feat/native-tool-reasoners

Conversation

@liruihan000

@liruihan000 liruihan000 commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Why

Write negotiation now uses the industry-standard tool-use agent pattern — the way current LLMs are trained, evaluated and deployed, and the way to get the most out of them. Every major agent stack (the Anthropic and OpenAI tool-use APIs, Claude Code, Codex, the agent SDKs, MCP) has converged on the same mechanics: the model gets typed tools, decides what to call and how often, sees each result as a structured tool message, and stops when it judges it is done. Our Write loop worked against every one of those:

  • A hand-written tool protocol. Looks and final operations were JSON lines inside free text, parsed by the library, with a format retry for when the model got the syntax wrong. Native tool calling gives typed arguments and structured results.
  • Flattened history. Each round re-rendered the whole transcript into one user prompt, quoting the model's own earlier replies back to it as ## Your reply. The model never saw a real assistant/tool turn structure.
  • Limits written around the model instead of trusting it. A round cap, a forced last round, an extension round and a format retry bounded the exchange. The model could not decide for itself when it had seen enough.
  • A crippled agent inside a hand-rolled agent. Claude Code and Codex are themselves tool-use agents. We ran them one turn at a time with their tools off and their coding-agent system prompt on, started a cold process for every round, and wrapped our own loop around them.

PR #105's benchmark located where the cost actually was. Keeping history in a Codex thread (resume) cut client prompt characters by two thirds but model input tokens by under a tenth, and ran slower — the model sees the full history every turn whoever holds it, so "sharing context" is not the lever. The large gains in its app-server arm came from one process per batch and from removing the host's coding-agent prompt and tools. The standard pattern delivers exactly those, on both Claude Code and Codex, without depending on Codex's experimental app-server protocol. This PR therefore supersedes the session-reuse transports in #105.

What changed

  • One Write toolkit (core): recall, read, submit_operations. It is the only definition of tool semantics — handle extension from tool results, active-only reads, submission validation. Tools never write the Store; an accepted submission goes through the single write path.
  • Endpoint reasoner: a standard function-calling loop — system + user messages, tools, one tool message per call, ends when the model stops calling tools.
  • Claude Code and Codex: one headless run per batch. The toolkit is served by a per-batch stdio MCP adapter (agent_memory.mcp.write_session); posture is tools-only (Claude: --strict-mcp-config, --tools "", only the toolkit allowed; Codex: user config and rules ignored, read-only sandbox, server required and pre-approved, local execution features disabled); the distillation instructions replace the host's system prompt.
  • No look budget, turn cap, forced last turn or format retry on the native paths. Request and process timeouts remain as transport-failure detection only.
  • Submission is validated inside the run — first against the reconcile sheet, then by the Store's own write preparation. Store now splits a write into prepare and persist; check_many runs prepare and persists nothing, so its verdicts are the write's. Refusals come back as tool errors with per-operation reasons and the model resubmits; the latest accepted submission is the answer. No accepted submission → nothing written, backlog and Watermark untouched (same guarantee as Keep archived backlog pending when executor output is incomplete #80). Only sibling collisions or concurrent Store changes still reach the single-shot repair.
  • Prompt split into instructions (system) and batch material (user), constant material first so prefixes stay cacheable.
  • Hermes and Muse Code keep the text protocol with a byte-identical prompt; their looks now go through the same toolkit.
  • Fix found while verifying: tools-off Claude Code runs (text protocol, Write repair, Manage review) loaded every MCP server from the user's Claude config. On this machine their tool definitions alone were ~230k tokens and every single-shot call failed with "Prompt is too long" on Haiku. They now run with --strict-mcp-config.
  • Docs: new docs/design/write-negotiation.md; host-adapters.md, design index, plan, README and docs/TODO.md updated.

Invariants kept: 2 (single write path), 5 (no model client in core — the endpoint loop lives in the executor), 8 (one toolkit; transports carry no tool semantics), 9 (Recall unchanged).

Verification

  • uv run ruff check ., uv run mypy, uv run pytest — 845 passed, coverage 91%.
  • New tests: toolkit relationships; Store check gives the same verdicts as the write and writes nothing; endpoint loop against a fake chat endpoint (multi and parallel calls, malformed arguments, no turn cap, natural stop); MCP adapter framing; Claude/Codex tools-only command shapes; system test in which fake claude and codex binaries start the real MCP adapter from their own flags and distill a batch end to end; red-team for injection through recalled bodies, handles named only inside recalled text, foreign e-mail addresses, malformed arguments, and an adapter that never writes the Store.
  • Live, paired, integration-class run (5 pairs per host, one fixed supersede scenario, fresh Store per arm, same Recall, arm order alternated, host retries off). Not a benchmark score.
Host Arm Superseded Left pending Median s (range) Host processes
Claude Code 2.1.286 / Haiku 4.5 text 3/5 2/5 32.6 (29.8–68.0) 2–3
Claude Code 2.1.286 / Haiku 4.5 native 5/5 0/5 14.0 (12.2–19.2) 1
Codex 0.153.0 / gpt-5.6-sol text 5/5 0/5 21.0 (16.7–22.9) 2
Codex 0.153.0 / gpt-5.6-sol native 5/5 0/5 25.3 (19.3–33.1) 1
  • Native Haiku corrected a refused submission inside the run in 4/5 batches; text Haiku left two batches pending after its single-shot repair failed.
  • Native Codex is ~4 s slower here: it chose to recall and read before every submission and corrected a refused submission in 4/5 batches. These are turns the model chose; the caps that used to prevent them are removed by design. Text Codex answered in one round without looking and then needed a second process for repair every time.
  • An earlier run on the same scenario, before submissions were checked by the Store, had native Haiku supersede only 1/5 (the Store refused future valid_from dates after the sheet check had accepted them). That is why the Store check is part of this PR.
  • Records, stores and write-up: experiments/runs/native-tools-latency/ in the main working tree (the experiment tree is untracked by design).
  • Endpoint path, live over OpenRouter (same scenario, 5 pairs per model, revision 894769e; text = the endpoint as a plain prompt-in/text-out callable):
Model Arm Superseded Left pending Median s (range) Requests
anthropic/claude-haiku-4.5 text 4/5 1/5 4.6 (4.3–5.6) 3
anthropic/claude-haiku-4.5 native 5/5 0/5 7.0 (5.2–8.6) 3–5
google/gemini-2.5-flash text 5/5 0/5 2.8 (2.0–5.8) 2–3
google/gemini-2.5-flash native 5/5 0/5 5.0 (2.2–6.8) 2–3
  • Native function calling works through OpenRouter for both Anthropic and Google models.
  • On an endpoint native is 2–3 s slower: there is no process start to save, and the model spends extra turns correcting refused submissions.
  • Refusal probes: most refused first submissions set valid_from to the announced Saturday, later than the cited messages (the Store refuses that). Rewording the schema description did not change it (second run, revision 34d0000), so that change was reverted. The second run also hit a pre-existing Store naming edge where Gemini looped seven refusals and ended without a submission — nothing was written and the backlog stayed, as designed. Both are tracked in docs/TODO.md.
  • Vertex returned 403 for any call from this environment, so the endpoint was tested through OpenRouter instead.

Trade-offs and follow-ups

  • No caps is deliberate. A runaway model is bounded only by request and process timeouts.
  • Operations that collide with each other inside one submission (e.g. a decision and a fact with the same key fields) are only caught at apply time; the pre-existing collision itself is tracked in docs/TODO.md.
  • Harness Claude runs with tools enabled still load user MCP servers (tracked; changing it alters experiment conditions, so it needs a ledger entry).
  • Two Store behaviours cost the reasoner turns and are tracked rather than changed here: announced future effective dates are refused as valid_from, and a supersede into another group with the same subject derives the predecessor's own name.

🤖 Generated with Claude Code

liruihan000 and others added 9 commits October 14, 2026 12:59
Plan and design contract: one Write toolkit (recall, read, submit operations),
a standard endpoint loop and a host-run loop over a per-batch MCP adapter,
with no look budget or turn cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Write toolkit (recall, read, submit_operations) is the single definition of
what a reasoner may do during a batch. The endpoint reasoner runs a standard
function-calling loop over it; Claude Code and Codex run their own loop with the
toolkit served by a per-batch stdio MCP adapter and their system prompt replaced.
No look budget, turn cap, forced last turn or format retry. Hermes and Muse keep
the text protocol, which now looks through the same toolkit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se Write loop

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without --strict-mcp-config every MCP server in the user's Claude config loads
into the headless reasoner. Observed live: their tool definitions alone came to
~230k tokens and every single-shot call (text-protocol Write, repair, Manage
review) failed with 'Prompt is too long' on Haiku.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Store's write is split into prepare and persist; a check runs prepare and
persists nothing, so its verdicts are the write's. submit_operations uses it,
so refusals such as a valid_from later than the cited evidence or an in-place
rewrite of a changed fact return to the model inside the same run instead of
surfacing afterwards through the single-shot text repair.

Found by the first paired live run: Haiku submissions passed the sheet check,
then the Store refused them and the text repair could not recover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every refused first submission in the OpenRouter probes (Haiku 4.5, Gemini 2.5
Flash) set valid_from to an announced future effective date, which the Store
refuses. The schema now says what the Store accepts, so the model can get it
right on the first submission instead of spending a turn on the refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reworded description did not change behaviour: in the follow-up OpenRouter
probe Haiku 4.5 and Gemini 2.5 Flash still set valid_from to the announced
future date in 2 of 3 runs each. Wording is an experiment variable; a change
not shown to help does not stay.

This reverts commit 34d0000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@faj-design5260
faj-design5260 merged commit c395acc into main Oct 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants