Skip to content

feat(toolsets): one text object per tool, and ModelRetry for a fixable mistake - #109

Merged
DEENUU1 merged 3 commits into
mainfrom
feat/tool-text
Aug 22, 2026
Merged

feat(toolsets): one text object per tool, and ModelRetry for a fixable mistake#109
DEENUU1 merged 3 commits into
mainfrom
feat/tool-text

Conversation

@DEENUU1

@DEENUU1 DEENUU1 commented Aug 22, 2026

Copy link
Copy Markdown
Member

Two commits, both about what a tool says to the model.

1. One text object per tool, with a Returns:feat(toolsets): give each console tool one text object

Each tool's text is now one ToolTextsummary, usage, coding, args, returns — instead of a description constant in one file and an argument docstring beside the function in another. The description is rendered from it; args becomes the per-argument text in the JSON schema.

The prose was rewritten against the house standard: what it does, when to use it and when to use another tool, every argument, and what comes back. That last part existed nowhere. Nothing said that grep answers in three shapes depending on output_mode, that glob lists 100 paths and grep 50 before summarising the rest as ... and N more, that read_file's offset counts from 0 while the printed line numbers count from 1, or that a failed execute still returns its output under Command failed (exit code N).

Also: profile="agent" drops the repository-specific guidance execute carried on every request (about 240 tokens across the seven tools a typical host registers), and descriptions now takes a ToolText as well as a string, with an unknown key raising UserError instead of being ignored.

2. ModelRetry for a mistake the model can fix — feat(toolsets): raise ModelRetry

Every failure used to be a returned string, including the ones the model had plainly got wrong: an old_string matching three places, a file edited between the read and the edit, a path that does not exist. It was told in a sentence indistinguishable from a real answer and left to notice.

Raises ModelRetry — a missing file, an offset past the end, an absent or ambiguous old_string, a stale read, a hashline hash that no longer matches.

Stays a returned string, deliberately — a non-zero exit from execute is a result to reason about rather than a malformed call; grep finding nothing is an answer; a dropped connection is not something different arguments would fix; and a permission refusal must never be a retry, since a retry prompt invites the model to look for a way around the rule. PERMISSION_DENIED_PREFIX is one constant PermissionGuard writes and toolsets/_failures.py reads, with a test holding both ends.

max_retries becomes a floor as well as a ceiling. ModelRetry past the budget ends the whole run with UnexpectedModelBehavior — so without the floor, a model that mistyped an old_string twice would kill a run that used to carry on: a regression hiding inside an improvement. On a tool's last attempt the message is returned rather than raised, so the worst case is exactly the old behaviour, and max_retries=0 reproduces it entirely.

tests/test_console_retry_budget.py proves that in a real run rather than by assertion: a FunctionModel sends the same ambiguous edit twice, and the message history is exactly one retry prompt followed by one tool return.

Verified

1713 tests, coverage 100%, pyright and mypy strict clean, mkdocs --strict builds, and the suite passes against the declared pydantic-ai floor 1.74.0 as well as 1.80.0 — RunContext.last_attempt and the __doc__ assignment are the two things that had to hold on both.

Not verified: no eval against a real model. Whether the new text changes behaviour gets measured in agenticos.

Next, not in this PR

agenticos takes profile="agent", uses ToolText.summary for the Builder's catalogue, bumps its pin, and gets the same failure taxonomy written down and applied to its own capabilities — most already raise ModelRetry; channel_tools returns strings for a documented and correct reason; code_execution.run_python returning Execution failed: ... for a SyntaxError in code the model itself wrote is the one clear mismatch.

A tool definition is a prompt, and these were written the way Claude
Code's are: shouty (NEVER, ALWAYS, MUST and IMPORTANT sixteen times
between them), thin on arguments, and silent on the one thing a model
cannot infer - the shape of the answer.

Nothing in the old text said that grep replies in three different shapes
depending on output_mode, that glob lists 100 paths and grep 50 before
summarising the rest as "... and N more", that read_file's offset counts
from 0 while the line numbers it prints count from 1, that a failed
execute still returns its output under "Command failed (exit code N)",
or that a timeout answers "Error: Command timed out" with code 124. All
of that is in the descriptions now, under a Returns paragraph.

The mechanism: each tool's text is a ToolText - summary, usage, coding,
args, returns - and the description is rendered from it while `args`
becomes the per-argument text in the JSON schema, assigned to the
function's __doc__ at registration because that is the only route into
the schema. Before, the description was a constant and the argument text
was a docstring beside the function: a host could override one and not
the other, and docs/api/toolsets.md held a third copy that had gone
stale.

Two additions fall out of it. `profile="agent"` drops the guidance
written for an agent working in a repository - git, package managers,
what to do after three failed attempts - which execute carried on every
request whether or not the workspace was a checkout: 961 characters,
about 240 tokens, across the seven tools a typical host registers. And
`descriptions` now takes a ToolText as well as a string, so a host can
reach the argument text too, while an unknown key raises UserError
instead of being ignored - a misspelled or renamed key used to mean an
override that silently reached nothing.

Verified: 1696 tests pass, coverage 100%, pyright and mypy strict clean,
mkdocs --strict builds, and the suite passes against the declared
pydantic-ai floor (1.74.0) as well as 1.80.0 - the __doc__ assignment is
the part that had to hold on both. tests/test_tool_text.py asserts every
argument of every tool in both edit formats carries a description, and
that TOOL_TEXT names exactly the arguments each function takes.

Not verified: no eval against a real model yet. The claim that better
text changes behaviour is measured in agenticos, next, by running the
sandbox tasks before and after.
Every failure in this toolset was a returned string, including the ones
the model had plainly got wrong - an old_string matching three places, a
file edited between the read and the edit, a path that does not exist.
It was told in a sentence indistinguishable from a real answer and left
to notice. Those now come back as a retry prompt: a missing file, an
offset past the end, an absent or ambiguous old_string, a stale read, a
hashline hash that no longer matches.

What stays a returned string is as deliberate. A non-zero exit from
execute is a result to reason about, not a malformed call; grep finding
nothing is an answer; a dropped connection is not something different
arguments would fix; and a permission refusal must never be a retry,
because a retry prompt invites the model to look for a way around the
rule. PERMISSION_DENIED_PREFIX is now one constant PermissionGuard
writes and toolsets/_failures.py reads, with a test holding both ends.

max_retries becomes a floor as much as a ceiling. ModelRetry past the
budget ends the whole run with UnexpectedModelBehavior, so without a
floor a model that mistyped an old_string twice would kill a run that
used to carry on - a regression hiding inside an improvement. On the
last attempt the message is returned rather than raised, so the worst
case is exactly the old behaviour, and max_retries=0 reproduces it.

Verified: 1712 tests pass, coverage 100%, pyright and mypy strict clean,
mkdocs --strict builds. tests/test_console_failures.py covers each
retryable case, the last-attempt floor, the refusal that must not
retry, and the three shapes that are results rather than mistakes.
`RunContext.last_attempt` exists on the declared pydantic-ai floor
(1.74.0), which is what this rests on.
The floor in `_failures.steer` only matters in a real run: `max_retries`
reaches the tool from the toolset, and `UnexpectedModelBehavior` is
raised by pydantic-ai rather than by anything in this repository, so a
unit test calling the tool function directly cannot show that the run
would have died.

A `FunctionModel` that sends the same ambiguous `edit_file` twice and
then answers proves both halves at once: the message history is exactly
one retry prompt followed by one tool return. No retry prompt would mean
nothing steers the model; no tool return would mean the run died on the
second attempt.
@DEENUU1 DEENUU1 changed the title feat(toolsets): give each console tool one text object, with returns feat(toolsets): one text object per tool, and ModelRetry for a fixable mistake Aug 22, 2026
@DEENUU1
DEENUU1 merged commit 8a6d31e into main Aug 22, 2026
14 checks passed
@DEENUU1
DEENUU1 deleted the feat/tool-text branch August 22, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

1 participant