feat(toolsets): one text object per tool, and ModelRetry for a fixable mistake - #109
Merged
Conversation
A tool definition is a prompt, and these were written the way Claude Code's are: shouty (NEVER, ALWAYS, MUST and IMPORTANT sixteen times between them), thin on arguments, and silent on the one thing a model cannot infer - the shape of the answer. Nothing in the old text said that grep replies in three different shapes depending on output_mode, that glob lists 100 paths and grep 50 before summarising the rest as "... and N more", that read_file's offset counts from 0 while the line numbers it prints count from 1, that a failed execute still returns its output under "Command failed (exit code N)", or that a timeout answers "Error: Command timed out" with code 124. All of that is in the descriptions now, under a Returns paragraph. The mechanism: each tool's text is a ToolText - summary, usage, coding, args, returns - and the description is rendered from it while `args` becomes the per-argument text in the JSON schema, assigned to the function's __doc__ at registration because that is the only route into the schema. Before, the description was a constant and the argument text was a docstring beside the function: a host could override one and not the other, and docs/api/toolsets.md held a third copy that had gone stale. Two additions fall out of it. `profile="agent"` drops the guidance written for an agent working in a repository - git, package managers, what to do after three failed attempts - which execute carried on every request whether or not the workspace was a checkout: 961 characters, about 240 tokens, across the seven tools a typical host registers. And `descriptions` now takes a ToolText as well as a string, so a host can reach the argument text too, while an unknown key raises UserError instead of being ignored - a misspelled or renamed key used to mean an override that silently reached nothing. Verified: 1696 tests pass, coverage 100%, pyright and mypy strict clean, mkdocs --strict builds, and the suite passes against the declared pydantic-ai floor (1.74.0) as well as 1.80.0 - the __doc__ assignment is the part that had to hold on both. tests/test_tool_text.py asserts every argument of every tool in both edit formats carries a description, and that TOOL_TEXT names exactly the arguments each function takes. Not verified: no eval against a real model yet. The claim that better text changes behaviour is measured in agenticos, next, by running the sandbox tasks before and after.
Every failure in this toolset was a returned string, including the ones the model had plainly got wrong - an old_string matching three places, a file edited between the read and the edit, a path that does not exist. It was told in a sentence indistinguishable from a real answer and left to notice. Those now come back as a retry prompt: a missing file, an offset past the end, an absent or ambiguous old_string, a stale read, a hashline hash that no longer matches. What stays a returned string is as deliberate. A non-zero exit from execute is a result to reason about, not a malformed call; grep finding nothing is an answer; a dropped connection is not something different arguments would fix; and a permission refusal must never be a retry, because a retry prompt invites the model to look for a way around the rule. PERMISSION_DENIED_PREFIX is now one constant PermissionGuard writes and toolsets/_failures.py reads, with a test holding both ends. max_retries becomes a floor as much as a ceiling. ModelRetry past the budget ends the whole run with UnexpectedModelBehavior, so without a floor a model that mistyped an old_string twice would kill a run that used to carry on - a regression hiding inside an improvement. On the last attempt the message is returned rather than raised, so the worst case is exactly the old behaviour, and max_retries=0 reproduces it. Verified: 1712 tests pass, coverage 100%, pyright and mypy strict clean, mkdocs --strict builds. tests/test_console_failures.py covers each retryable case, the last-attempt floor, the refusal that must not retry, and the three shapes that are results rather than mistakes. `RunContext.last_attempt` exists on the declared pydantic-ai floor (1.74.0), which is what this rests on.
The floor in `_failures.steer` only matters in a real run: `max_retries` reaches the tool from the toolset, and `UnexpectedModelBehavior` is raised by pydantic-ai rather than by anything in this repository, so a unit test calling the tool function directly cannot show that the run would have died. A `FunctionModel` that sends the same ambiguous `edit_file` twice and then answers proves both halves at once: the message history is exactly one retry prompt followed by one tool return. No retry prompt would mean nothing steers the model; no tool return would mean the run died on the second attempt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits, both about what a tool says to the model.
1. One text object per tool, with a
Returns:—feat(toolsets): give each console tool one text objectEach tool's text is now one
ToolText—summary,usage,coding,args,returns— instead of a description constant in one file and an argument docstring beside the function in another. The description is rendered from it;argsbecomes the per-argument text in the JSON schema.The prose was rewritten against the house standard: what it does, when to use it and when to use another tool, every argument, and what comes back. That last part existed nowhere. Nothing said that
grepanswers in three shapes depending onoutput_mode, thatgloblists 100 paths andgrep50 before summarising the rest as... and N more, thatread_file'soffsetcounts from 0 while the printed line numbers count from 1, or that a failedexecutestill returns its output underCommand failed (exit code N).Also:
profile="agent"drops the repository-specific guidanceexecutecarried on every request (about 240 tokens across the seven tools a typical host registers), anddescriptionsnow takes aToolTextas well as a string, with an unknown key raisingUserErrorinstead of being ignored.2.
ModelRetryfor a mistake the model can fix —feat(toolsets): raise ModelRetryEvery failure used to be a returned string, including the ones the model had plainly got wrong: an
old_stringmatching three places, a file edited between the read and the edit, a path that does not exist. It was told in a sentence indistinguishable from a real answer and left to notice.Raises
ModelRetry— a missing file, an offset past the end, an absent or ambiguousold_string, a stale read, a hashline hash that no longer matches.Stays a returned string, deliberately — a non-zero exit from
executeis a result to reason about rather than a malformed call;grepfinding nothing is an answer; a dropped connection is not something different arguments would fix; and a permission refusal must never be a retry, since a retry prompt invites the model to look for a way around the rule.PERMISSION_DENIED_PREFIXis one constantPermissionGuardwrites andtoolsets/_failures.pyreads, with a test holding both ends.max_retriesbecomes a floor as well as a ceiling.ModelRetrypast the budget ends the whole run withUnexpectedModelBehavior— so without the floor, a model that mistyped anold_stringtwice would kill a run that used to carry on: a regression hiding inside an improvement. On a tool's last attempt the message is returned rather than raised, so the worst case is exactly the old behaviour, andmax_retries=0reproduces it entirely.tests/test_console_retry_budget.pyproves that in a real run rather than by assertion: aFunctionModelsends the same ambiguous edit twice, and the message history is exactly one retry prompt followed by one tool return.Verified
1713 tests, coverage 100%, pyright and mypy strict clean,
mkdocs --strictbuilds, and the suite passes against the declared pydantic-ai floor 1.74.0 as well as 1.80.0 —RunContext.last_attemptand the__doc__assignment are the two things that had to hold on both.Not verified: no eval against a real model. Whether the new text changes behaviour gets measured in agenticos.
Next, not in this PR
agenticos takes
profile="agent", usesToolText.summaryfor the Builder's catalogue, bumps its pin, and gets the same failure taxonomy written down and applied to its own capabilities — most already raiseModelRetry;channel_toolsreturns strings for a documented and correct reason;code_execution.run_pythonreturningExecution failed: ...for aSyntaxErrorin code the model itself wrote is the one clear mismatch.