|
| 1 | +# Backend Agent Tools |
| 2 | + |
| 3 | +## Purpose |
| 4 | + |
| 5 | +The **Backend Agent Tools** module provides the low-level, agent-facing "hands" that CodeWiki's |
| 6 | +LLM-driven documentation agents use to actually read code and write documentation files. It is |
| 7 | +intentionally small and dependency-light: it does not decide *what* documentation to write (that |
| 8 | +is the job of the orchestration layer in |
| 9 | +[Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)), it only exposes |
| 10 | +safe, well-defined **tool functions** that an LLM agent can call, plus the **shared dependency |
| 11 | +bag** (`CodeWikiDeps`) that carries per-run context (paths, component registry, module tree, |
| 12 | +config, etc.) into every tool invocation. |
| 13 | + |
| 14 | +Concretely, this module contains: |
| 15 | + |
| 16 | +- `CodeWikiDeps` — a dataclass bundling all contextual state a tool needs to run safely for a |
| 17 | + given documentation-generation session/module. |
| 18 | +- `EditTool` (+ helpers `Filemap`, `WindowExpander`) and the `str_replace_editor` tool function — |
| 19 | + a sandboxed file-viewing/editing tool (modeled after SWE-agent's editor) that lets an agent |
| 20 | + inspect repository source code (read-only) and create/edit documentation markdown files |
| 21 | + (read-write), with automatic Mermaid diagram validation on every markdown write. |
| 22 | + |
| 23 | +This module is consumed by two different agent runtime backends: |
| 24 | + |
| 25 | +- The **pydantic-ai** backend (`PydanticAIBackend` in |
| 26 | + [Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)), which registers |
| 27 | + `str_replace_editor_tool` (a `pydantic_ai.Tool` wrapper around the `str_replace_editor` |
| 28 | + function) directly. |
| 29 | +- The **caw** backend (`CawBackend` / `CawToolKit`, also in |
| 30 | + [Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)), whose |
| 31 | + `CawToolKit.str_replace_editor` MCP tool re-uses the very same `EditTool` implementation and |
| 32 | + Mermaid validator from this module, adding extra path-confinement checks appropriate for an |
| 33 | + MCP tool-server context. |
| 34 | + |
| 35 | +## Architecture Overview |
| 36 | + |
| 37 | +```mermaid |
| 38 | +flowchart TB |
| 39 | + subgraph Orchestration["Backend LLM & Documentation Services"] |
| 40 | + PAB["PydanticAIBackend"] |
| 41 | + CB["CawBackend / CawToolKit"] |
| 42 | + DG["DocumentationGenerator"] |
| 43 | + end |
| 44 | +
|
| 45 | + subgraph AgentTools["Backend_Agent_Tools (this module)"] |
| 46 | + Deps["CodeWikiDeps\n(deps.py)"] |
| 47 | + SRE["str_replace_editor()\n+ str_replace_editor_tool\n(str_replace_editor.py)"] |
| 48 | + ET["EditTool"] |
| 49 | + FM["Filemap"] |
| 50 | + WE["WindowExpander"] |
| 51 | + SRE --> ET |
| 52 | + ET --> FM |
| 53 | + ET --> WE |
| 54 | + end |
| 55 | +
|
| 56 | + subgraph Support["Supporting modules"] |
| 57 | + Node["Node\n(Dependency_Analyzer_Core)"] |
| 58 | + Config["Config\n(Core_Config_&_Utils)"] |
| 59 | + MermaidVal["validate_mermaid_diagrams()\n(be/utils.py)"] |
| 60 | + end |
| 61 | +
|
| 62 | + DG --> PAB |
| 63 | + DG --> CB |
| 64 | + PAB -->|registers Tool| SRE |
| 65 | + CB -->|delegates to| ET |
| 66 | + Deps -.uses.-> Node |
| 67 | + Deps -.uses.-> Config |
| 68 | + SRE -->|on .md write| MermaidVal |
| 69 | +
|
| 70 | + PAB -- "RunContext[CodeWikiDeps]" --> Deps |
| 71 | + CB -- "self._deps: CodeWikiDeps" --> Deps |
| 72 | +``` |
| 73 | + |
| 74 | +### Two working directories, one tool |
| 75 | + |
| 76 | +The editor tool always operates against exactly one of two root directories, selected per call via |
| 77 | +the `working_dir` parameter: |
| 78 | + |
| 79 | +| `working_dir` | Root path | Allowed commands | Purpose | |
| 80 | +|----------------|-----------------------------------|-----------------------------|--------------------------------------------| |
| 81 | +| `"repo"` | `CodeWikiDeps.absolute_repo_path` | `view` only | Inspect the target repository's source code | |
| 82 | +| `"docs"` | `CodeWikiDeps.absolute_docs_path` | `view`, `create`, `str_replace`, `insert`, `undo_edit` | Read/write the generated documentation tree | |
| 83 | + |
| 84 | +This split enforces a hard invariant: **agents can never mutate the source repository**, only the |
| 85 | +documentation output. Any attempt to run a mutating command against `"repo"` is rejected before |
| 86 | +the underlying `EditTool` is even invoked. |
| 87 | + |
| 88 | +```mermaid |
| 89 | +sequenceDiagram |
| 90 | + participant Agent as LLM Agent |
| 91 | + participant Tool as str_replace_editor() |
| 92 | + participant Edit as EditTool |
| 93 | + participant FS as Filesystem |
| 94 | + participant Val as validate_mermaid_diagrams() |
| 95 | +
|
| 96 | + Agent->>Tool: command, working_dir, path, ... |
| 97 | + Tool->>Tool: resolve absolute_path via CodeWikiDeps |
| 98 | + alt working_dir == "repo" and command != "view" |
| 99 | + Tool-->>Agent: error: only `view` allowed on repo |
| 100 | + else valid request |
| 101 | + Tool->>Edit: validate_path(command, path) |
| 102 | + Edit->>FS: read/write/list as needed |
| 103 | + Edit-->>Tool: log messages (result/errors) |
| 104 | + opt command != "view" and path endswith .md |
| 105 | + Tool->>Val: validate_mermaid_diagrams(path) |
| 106 | + Val-->>Tool: validation report |
| 107 | + end |
| 108 | + Tool-->>Agent: combined result string |
| 109 | + end |
| 110 | +``` |
| 111 | + |
| 112 | +## Core Components |
| 113 | + |
| 114 | +### `CodeWikiDeps` (`deps.py`) |
| 115 | + |
| 116 | +A plain dataclass acting as the **dependency-injection context** passed to every tool call. It is |
| 117 | +constructed once per module-documentation task and threaded through the agent runtime as |
| 118 | +`RunContext[CodeWikiDeps].deps` (pydantic-ai) or `self._deps` (caw's `CawToolKit`). |
| 119 | + |
| 120 | +| Field | Type | Description | |
| 121 | +|---|---|---| |
| 122 | +| `absolute_docs_path` | `str` | Root directory for generated documentation (the `"docs"` working dir root). | |
| 123 | +| `absolute_repo_path` | `str` | Root directory of the analyzed repository (the `"repo"` working dir root). | |
| 124 | +| `registry` | `dict` | A mutable, per-session key/value store (e.g. used by `EditTool` to persist `file_history` for `undo_edit`). | |
| 125 | +| `components` | `dict[str, Node]` | Map of component id → `Node` (from [Dependency_Analyzer_Core](Dependency_Analyzer_Core.md)), the source-of-truth for `read_code_components`. | |
| 126 | +| `path_to_current_module` | `list[str]` | Breadcrumb path in the module tree identifying the module currently being documented (used for recursive sub-module generation). | |
| 127 | +| `current_module_name` | `str` | Name of the module currently being documented. | |
| 128 | +| `module_tree` | `dict[str, any]` | The in-memory hierarchical module tree being built up as documentation is generated. | |
| 129 | +| `max_depth` / `current_depth` | `int` | Guards recursion depth for sub-module documentation generation. | |
| 130 | +| `config` | `Config` | LLM/runtime configuration (see [Core_Config_&_Utils](Core_Config_&_Utils.md)). | |
| 131 | +| `custom_instructions` | `str \| None` | Optional user-supplied custom instructions injected into agent prompts. | |
| 132 | + |
| 133 | +`CodeWikiDeps` itself contains no logic — it is intentionally inert so that both the pydantic-ai |
| 134 | +and caw backends can share the exact same context shape. |
| 135 | + |
| 136 | +### `str_replace_editor` tool function & `EditTool` (`str_replace_editor.py`) |
| 137 | + |
| 138 | +This is the heart of the module: a sandboxed file editor exposing the classic |
| 139 | +view/create/str_replace/insert/undo_edit command set (a design borrowed from |
| 140 | +[SWE-agent's editor tool](https://github.com/SWE-agent/SWE-agent/blob/main/tools/edit_anthropic/bin/str_replace_editor)). |
| 141 | + |
| 142 | +**Public entry points:** |
| 143 | + |
| 144 | +- `str_replace_editor(ctx, working_dir, command, path, ...)` — the async tool function registered |
| 145 | + with pydantic-ai. Resolves `working_dir` + `path` into an absolute path scoped to either the repo |
| 146 | + or docs root, enforces the repo-is-read-only rule, delegates to `EditTool`, and — for any |
| 147 | + non-`view` write to a `.md` file — runs `validate_mermaid_diagrams` on the result and appends the |
| 148 | + validation report to the tool output so the agent immediately sees Mermaid syntax errors. |
| 149 | +- `str_replace_editor_tool` — a `pydantic_ai.Tool` wrapping the function above, registered by |
| 150 | + `PydanticAIBackend`. |
| 151 | +- `EditTool` — the actual command implementation, reusable outside of pydantic-ai (this is exactly |
| 152 | + what `CawToolKit.str_replace_editor` does, adding its own path-confinement checks for the MCP |
| 153 | + context — see [Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)). |
| 154 | + |
| 155 | +**Commands implemented by `EditTool`:** |
| 156 | + |
| 157 | +| Command | Behavior | |
| 158 | +|---|---| |
| 159 | +| `view` | For directories: lists non-hidden entries up to 2 levels deep. For files: shows `cat -n`-style numbered output, optionally scoped to a `view_range`, with the viewport auto-expanded to whole function/class boundaries via `WindowExpander`. Long files can be summarized with `Filemap` (currently gated by `USE_FILEMAP`). | |
| 160 | +| `create` | Creates a new file; fails if the file already exists. | |
| 161 | +| `str_replace` | Replaces a unique occurrence of `old_str` with `new_str`; fails loudly if `old_str` is missing or ambiguous (multiple matches). Optionally runs `flake8` before/after the edit and reports newly introduced lint errors (gated by `USE_LINTER`). | |
| 162 | +| `insert` | Inserts `new_str` after a given 1-based `insert_line`. | |
| 163 | +| `undo_edit` | Pops and restores the previous version of a file from an in-`registry` edit history. | |
| 164 | + |
| 165 | +All results are accumulated as human-readable log strings in `EditTool.logs`, which the tool |
| 166 | +function joins and returns to the agent — there are no raised exceptions for expected error |
| 167 | +conditions (invalid path, ambiguous replace, etc.), keeping the agent loop resilient to bad tool |
| 168 | +calls. |
| 169 | + |
| 170 | +**Helper classes:** |
| 171 | + |
| 172 | +- **`Filemap`** — uses `tree_sitter_languages` to parse Python source and produce an "abbreviated" |
| 173 | + view of a file where long function bodies (≥ 5 lines) are elided with a placeholder line, |
| 174 | + keeping large-file previews compact. Only used when a file exceeds `MAX_RESPONSE_LEN` and |
| 175 | + `USE_FILEMAP` is enabled. |
| 176 | +- **`WindowExpander`** — given a requested `[start, stop]` line range, tries to "snap" the viewport |
| 177 | + boundaries to natural breakpoints (blank lines, `def`/`class`/decorator lines for `.py` files, or |
| 178 | + file start/end) so that `view` and edit-confirmation snippets show whole logical blocks instead |
| 179 | + of arbitrarily truncated code. Expansion amount is bounded by module-level constants |
| 180 | + (`MAX_WINDOW_EXPANSION_VIEW`, `MAX_WINDOW_EXPANSION_EDIT_CONFIRM`), both currently set to `0` |
| 181 | + (i.e. expansion is disabled by default but the mechanism remains in place). |
| 182 | + |
| 183 | +**Safety/robustness features baked into the tool:** |
| 184 | + |
| 185 | +- Strict path validation (`validate_path`): must be absolute, must exist unless creating, cannot |
| 186 | + overwrite an existing file with `create`, only `view` is allowed on directories. |
| 187 | +- `_coerce_json_string` (via a Pydantic `BeforeValidator`) transparently parses list/int arguments |
| 188 | + that arrive as JSON-encoded strings — a compatibility shim for OpenAI-compatible endpoints |
| 189 | + (LiteLLM, vLLM, Ollama, etc.) that don't always emit strictly-typed tool arguments the way the |
| 190 | + native Anthropic API does. |
| 191 | +- Output truncation (`maybe_truncate` / `MAX_RESPONSE_LEN` / `TRUNCATED_MESSAGE`) prevents any |
| 192 | + single tool call from flooding the agent's context window; the truncation message explicitly |
| 193 | + instructs the agent to retry with `grep -n` + `view_range`. |
| 194 | +- Robust file I/O: `read_file` tries a sequence of encodings (`None` → `utf-8` → `latin-1` → |
| 195 | + `utf-8` with `errors="replace"`) to gracefully handle non-UTF-8 source files (e.g. Windows/GBK |
| 196 | + paths), and `view`/`flake8` subprocess output decoding also uses `errors="replace"`. |
| 197 | +- Automatic **Mermaid diagram validation**: any non-`view` write to a `.md` file triggers |
| 198 | + `validate_mermaid_diagrams` (from `codewiki/src/be/utils.py`), and its report — including |
| 199 | + precise error locations — is appended to the tool result so the documentation agent can |
| 200 | + self-correct malformed diagrams in the same turn. |
| 201 | + |
| 202 | +## Relationship to Other Modules |
| 203 | + |
| 204 | +```mermaid |
| 205 | +flowchart LR |
| 206 | + Deps["CodeWikiDeps"] --> Node["Node\n(Dependency_Analyzer_Core)"] |
| 207 | + Deps --> Config["Config\n(Core_Config_&_Utils)"] |
| 208 | +
|
| 209 | + subgraph BE["Backend LLM & Documentation Services"] |
| 210 | + PAB["PydanticAIBackend"] |
| 211 | + CB["CawBackend"] |
| 212 | + Toolkit["CawToolKit"] |
| 213 | + end |
| 214 | +
|
| 215 | + PAB --> SRE_Tool["str_replace_editor_tool"] |
| 216 | + SRE_Tool --> SRE_Fn["str_replace_editor()"] |
| 217 | + SRE_Fn --> Deps |
| 218 | +
|
| 219 | + Toolkit --> ET["EditTool (reused)"] |
| 220 | + Toolkit --> Deps |
| 221 | + ET --> Deps |
| 222 | +``` |
| 223 | + |
| 224 | +- **[Dependency_Analyzer_Core](Dependency_Analyzer_Core.md)**: Supplies the `Node` model that |
| 225 | + populates `CodeWikiDeps.components` — the registry that both `read_code_components` (in |
| 226 | + `CawToolKit`) and the editor's repo-view path ultimately reason about. |
| 227 | +- **[Core_Config_&_Utils](Core_Config_&_Utils.md)**: Supplies the `Config` object stored in |
| 228 | + `CodeWikiDeps.config`, carrying LLM/model/provider settings and doc-generation options |
| 229 | + (`max_depth`, `agent_instructions`, etc.) that flow into agent prompts alongside these tools. |
| 230 | + `FileManager` in the same module handles broader file-system utilities used elsewhere in the |
| 231 | + pipeline. |
| 232 | +- **[Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)**: The primary |
| 233 | + consumer of this module. `PydanticAIBackend` registers `str_replace_editor_tool` for its agent |
| 234 | + runs; `CawBackend`/`CawToolKit` reuse `EditTool` and the Mermaid validator directly inside an MCP |
| 235 | + tool server, and drive the recursive sub-module documentation flow (`generate_sub_module_documentation`) |
| 236 | + that ultimately writes files through this same editor. |
| 237 | +- **[MCP_Session_Management](MCP_Session_Management.md)**: Manages the session/workspace lifecycle |
| 238 | + (`SessionStore`, `SessionWorkspace`) within which `CawToolKit`'s tool calls — including the reused |
| 239 | + `EditTool` — execute over MCP. |
0 commit comments