Skip to content

Commit a7f7223

Browse files
authored
Merge pull request #84 from FSoft-AI4Code/docs/regenerated-self-docs
Regenerate self-documentation with current CodeWiki
2 parents 3c4b4c2 + bcfcf3f commit a7f7223

63 files changed

Lines changed: 25819 additions & 23452 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/Backend_Agent_Tools.md‎

Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
# Backend Agent Tools
2+
3+
## Purpose
4+
5+
The **Backend Agent Tools** module provides the low-level, agent-facing "hands" that CodeWiki's
6+
LLM-driven documentation agents use to actually read code and write documentation files. It is
7+
intentionally small and dependency-light: it does not decide *what* documentation to write (that
8+
is the job of the orchestration layer in
9+
[Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)), it only exposes
10+
safe, well-defined **tool functions** that an LLM agent can call, plus the **shared dependency
11+
bag** (`CodeWikiDeps`) that carries per-run context (paths, component registry, module tree,
12+
config, etc.) into every tool invocation.
13+
14+
Concretely, this module contains:
15+
16+
- `CodeWikiDeps` — a dataclass bundling all contextual state a tool needs to run safely for a
17+
given documentation-generation session/module.
18+
- `EditTool` (+ helpers `Filemap`, `WindowExpander`) and the `str_replace_editor` tool function —
19+
a sandboxed file-viewing/editing tool (modeled after SWE-agent's editor) that lets an agent
20+
inspect repository source code (read-only) and create/edit documentation markdown files
21+
(read-write), with automatic Mermaid diagram validation on every markdown write.
22+
23+
This module is consumed by two different agent runtime backends:
24+
25+
- The **pydantic-ai** backend (`PydanticAIBackend` in
26+
[Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)), which registers
27+
`str_replace_editor_tool` (a `pydantic_ai.Tool` wrapper around the `str_replace_editor`
28+
function) directly.
29+
- The **caw** backend (`CawBackend` / `CawToolKit`, also in
30+
[Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)), whose
31+
`CawToolKit.str_replace_editor` MCP tool re-uses the very same `EditTool` implementation and
32+
Mermaid validator from this module, adding extra path-confinement checks appropriate for an
33+
MCP tool-server context.
34+
35+
## Architecture Overview
36+
37+
```mermaid
38+
flowchart TB
39+
subgraph Orchestration["Backend LLM & Documentation Services"]
40+
PAB["PydanticAIBackend"]
41+
CB["CawBackend / CawToolKit"]
42+
DG["DocumentationGenerator"]
43+
end
44+
45+
subgraph AgentTools["Backend_Agent_Tools (this module)"]
46+
Deps["CodeWikiDeps\n(deps.py)"]
47+
SRE["str_replace_editor()\n+ str_replace_editor_tool\n(str_replace_editor.py)"]
48+
ET["EditTool"]
49+
FM["Filemap"]
50+
WE["WindowExpander"]
51+
SRE --> ET
52+
ET --> FM
53+
ET --> WE
54+
end
55+
56+
subgraph Support["Supporting modules"]
57+
Node["Node\n(Dependency_Analyzer_Core)"]
58+
Config["Config\n(Core_Config_&_Utils)"]
59+
MermaidVal["validate_mermaid_diagrams()\n(be/utils.py)"]
60+
end
61+
62+
DG --> PAB
63+
DG --> CB
64+
PAB -->|registers Tool| SRE
65+
CB -->|delegates to| ET
66+
Deps -.uses.-> Node
67+
Deps -.uses.-> Config
68+
SRE -->|on .md write| MermaidVal
69+
70+
PAB -- "RunContext[CodeWikiDeps]" --> Deps
71+
CB -- "self._deps: CodeWikiDeps" --> Deps
72+
```
73+
74+
### Two working directories, one tool
75+
76+
The editor tool always operates against exactly one of two root directories, selected per call via
77+
the `working_dir` parameter:
78+
79+
| `working_dir` | Root path | Allowed commands | Purpose |
80+
|----------------|-----------------------------------|-----------------------------|--------------------------------------------|
81+
| `"repo"` | `CodeWikiDeps.absolute_repo_path` | `view` only | Inspect the target repository's source code |
82+
| `"docs"` | `CodeWikiDeps.absolute_docs_path` | `view`, `create`, `str_replace`, `insert`, `undo_edit` | Read/write the generated documentation tree |
83+
84+
This split enforces a hard invariant: **agents can never mutate the source repository**, only the
85+
documentation output. Any attempt to run a mutating command against `"repo"` is rejected before
86+
the underlying `EditTool` is even invoked.
87+
88+
```mermaid
89+
sequenceDiagram
90+
participant Agent as LLM Agent
91+
participant Tool as str_replace_editor()
92+
participant Edit as EditTool
93+
participant FS as Filesystem
94+
participant Val as validate_mermaid_diagrams()
95+
96+
Agent->>Tool: command, working_dir, path, ...
97+
Tool->>Tool: resolve absolute_path via CodeWikiDeps
98+
alt working_dir == "repo" and command != "view"
99+
Tool-->>Agent: error: only `view` allowed on repo
100+
else valid request
101+
Tool->>Edit: validate_path(command, path)
102+
Edit->>FS: read/write/list as needed
103+
Edit-->>Tool: log messages (result/errors)
104+
opt command != "view" and path endswith .md
105+
Tool->>Val: validate_mermaid_diagrams(path)
106+
Val-->>Tool: validation report
107+
end
108+
Tool-->>Agent: combined result string
109+
end
110+
```
111+
112+
## Core Components
113+
114+
### `CodeWikiDeps` (`deps.py`)
115+
116+
A plain dataclass acting as the **dependency-injection context** passed to every tool call. It is
117+
constructed once per module-documentation task and threaded through the agent runtime as
118+
`RunContext[CodeWikiDeps].deps` (pydantic-ai) or `self._deps` (caw's `CawToolKit`).
119+
120+
| Field | Type | Description |
121+
|---|---|---|
122+
| `absolute_docs_path` | `str` | Root directory for generated documentation (the `"docs"` working dir root). |
123+
| `absolute_repo_path` | `str` | Root directory of the analyzed repository (the `"repo"` working dir root). |
124+
| `registry` | `dict` | A mutable, per-session key/value store (e.g. used by `EditTool` to persist `file_history` for `undo_edit`). |
125+
| `components` | `dict[str, Node]` | Map of component id → `Node` (from [Dependency_Analyzer_Core](Dependency_Analyzer_Core.md)), the source-of-truth for `read_code_components`. |
126+
| `path_to_current_module` | `list[str]` | Breadcrumb path in the module tree identifying the module currently being documented (used for recursive sub-module generation). |
127+
| `current_module_name` | `str` | Name of the module currently being documented. |
128+
| `module_tree` | `dict[str, any]` | The in-memory hierarchical module tree being built up as documentation is generated. |
129+
| `max_depth` / `current_depth` | `int` | Guards recursion depth for sub-module documentation generation. |
130+
| `config` | `Config` | LLM/runtime configuration (see [Core_Config_&_Utils](Core_Config_&_Utils.md)). |
131+
| `custom_instructions` | `str \| None` | Optional user-supplied custom instructions injected into agent prompts. |
132+
133+
`CodeWikiDeps` itself contains no logic — it is intentionally inert so that both the pydantic-ai
134+
and caw backends can share the exact same context shape.
135+
136+
### `str_replace_editor` tool function & `EditTool` (`str_replace_editor.py`)
137+
138+
This is the heart of the module: a sandboxed file editor exposing the classic
139+
view/create/str_replace/insert/undo_edit command set (a design borrowed from
140+
[SWE-agent's editor tool](https://github.com/SWE-agent/SWE-agent/blob/main/tools/edit_anthropic/bin/str_replace_editor)).
141+
142+
**Public entry points:**
143+
144+
- `str_replace_editor(ctx, working_dir, command, path, ...)` — the async tool function registered
145+
with pydantic-ai. Resolves `working_dir` + `path` into an absolute path scoped to either the repo
146+
or docs root, enforces the repo-is-read-only rule, delegates to `EditTool`, and — for any
147+
non-`view` write to a `.md` file — runs `validate_mermaid_diagrams` on the result and appends the
148+
validation report to the tool output so the agent immediately sees Mermaid syntax errors.
149+
- `str_replace_editor_tool` — a `pydantic_ai.Tool` wrapping the function above, registered by
150+
`PydanticAIBackend`.
151+
- `EditTool` — the actual command implementation, reusable outside of pydantic-ai (this is exactly
152+
what `CawToolKit.str_replace_editor` does, adding its own path-confinement checks for the MCP
153+
context — see [Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)).
154+
155+
**Commands implemented by `EditTool`:**
156+
157+
| Command | Behavior |
158+
|---|---|
159+
| `view` | For directories: lists non-hidden entries up to 2 levels deep. For files: shows `cat -n`-style numbered output, optionally scoped to a `view_range`, with the viewport auto-expanded to whole function/class boundaries via `WindowExpander`. Long files can be summarized with `Filemap` (currently gated by `USE_FILEMAP`). |
160+
| `create` | Creates a new file; fails if the file already exists. |
161+
| `str_replace` | Replaces a unique occurrence of `old_str` with `new_str`; fails loudly if `old_str` is missing or ambiguous (multiple matches). Optionally runs `flake8` before/after the edit and reports newly introduced lint errors (gated by `USE_LINTER`). |
162+
| `insert` | Inserts `new_str` after a given 1-based `insert_line`. |
163+
| `undo_edit` | Pops and restores the previous version of a file from an in-`registry` edit history. |
164+
165+
All results are accumulated as human-readable log strings in `EditTool.logs`, which the tool
166+
function joins and returns to the agent — there are no raised exceptions for expected error
167+
conditions (invalid path, ambiguous replace, etc.), keeping the agent loop resilient to bad tool
168+
calls.
169+
170+
**Helper classes:**
171+
172+
- **`Filemap`** — uses `tree_sitter_languages` to parse Python source and produce an "abbreviated"
173+
view of a file where long function bodies (≥ 5 lines) are elided with a placeholder line,
174+
keeping large-file previews compact. Only used when a file exceeds `MAX_RESPONSE_LEN` and
175+
`USE_FILEMAP` is enabled.
176+
- **`WindowExpander`** — given a requested `[start, stop]` line range, tries to "snap" the viewport
177+
boundaries to natural breakpoints (blank lines, `def`/`class`/decorator lines for `.py` files, or
178+
file start/end) so that `view` and edit-confirmation snippets show whole logical blocks instead
179+
of arbitrarily truncated code. Expansion amount is bounded by module-level constants
180+
(`MAX_WINDOW_EXPANSION_VIEW`, `MAX_WINDOW_EXPANSION_EDIT_CONFIRM`), both currently set to `0`
181+
(i.e. expansion is disabled by default but the mechanism remains in place).
182+
183+
**Safety/robustness features baked into the tool:**
184+
185+
- Strict path validation (`validate_path`): must be absolute, must exist unless creating, cannot
186+
overwrite an existing file with `create`, only `view` is allowed on directories.
187+
- `_coerce_json_string` (via a Pydantic `BeforeValidator`) transparently parses list/int arguments
188+
that arrive as JSON-encoded strings — a compatibility shim for OpenAI-compatible endpoints
189+
(LiteLLM, vLLM, Ollama, etc.) that don't always emit strictly-typed tool arguments the way the
190+
native Anthropic API does.
191+
- Output truncation (`maybe_truncate` / `MAX_RESPONSE_LEN` / `TRUNCATED_MESSAGE`) prevents any
192+
single tool call from flooding the agent's context window; the truncation message explicitly
193+
instructs the agent to retry with `grep -n` + `view_range`.
194+
- Robust file I/O: `read_file` tries a sequence of encodings (`None` → `utf-8` → `latin-1` →
195+
`utf-8` with `errors="replace"`) to gracefully handle non-UTF-8 source files (e.g. Windows/GBK
196+
paths), and `view`/`flake8` subprocess output decoding also uses `errors="replace"`.
197+
- Automatic **Mermaid diagram validation**: any non-`view` write to a `.md` file triggers
198+
`validate_mermaid_diagrams` (from `codewiki/src/be/utils.py`), and its report — including
199+
precise error locations — is appended to the tool result so the documentation agent can
200+
self-correct malformed diagrams in the same turn.
201+
202+
## Relationship to Other Modules
203+
204+
```mermaid
205+
flowchart LR
206+
Deps["CodeWikiDeps"] --> Node["Node\n(Dependency_Analyzer_Core)"]
207+
Deps --> Config["Config\n(Core_Config_&_Utils)"]
208+
209+
subgraph BE["Backend LLM & Documentation Services"]
210+
PAB["PydanticAIBackend"]
211+
CB["CawBackend"]
212+
Toolkit["CawToolKit"]
213+
end
214+
215+
PAB --> SRE_Tool["str_replace_editor_tool"]
216+
SRE_Tool --> SRE_Fn["str_replace_editor()"]
217+
SRE_Fn --> Deps
218+
219+
Toolkit --> ET["EditTool (reused)"]
220+
Toolkit --> Deps
221+
ET --> Deps
222+
```
223+
224+
- **[Dependency_Analyzer_Core](Dependency_Analyzer_Core.md)**: Supplies the `Node` model that
225+
populates `CodeWikiDeps.components` — the registry that both `read_code_components` (in
226+
`CawToolKit`) and the editor's repo-view path ultimately reason about.
227+
- **[Core_Config_&_Utils](Core_Config_&_Utils.md)**: Supplies the `Config` object stored in
228+
`CodeWikiDeps.config`, carrying LLM/model/provider settings and doc-generation options
229+
(`max_depth`, `agent_instructions`, etc.) that flow into agent prompts alongside these tools.
230+
`FileManager` in the same module handles broader file-system utilities used elsewhere in the
231+
pipeline.
232+
- **[Backend LLM & Documentation Services](Backend_LLM_&_Documentation_Services.md)**: The primary
233+
consumer of this module. `PydanticAIBackend` registers `str_replace_editor_tool` for its agent
234+
runs; `CawBackend`/`CawToolKit` reuse `EditTool` and the Mermaid validator directly inside an MCP
235+
tool server, and drive the recursive sub-module documentation flow (`generate_sub_module_documentation`)
236+
that ultimately writes files through this same editor.
237+
- **[MCP_Session_Management](MCP_Session_Management.md)**: Manages the session/workspace lifecycle
238+
(`SessionStore`, `SessionWorkspace`) within which `CawToolKit`'s tool calls — including the reused
239+
`EditTool` — execute over MCP.

0 commit comments

Comments
 (0)