|
1 | 1 | # TinyAgent |
2 | 2 |
|
3 | | -TinyAgent routes one query to one registered Python tool through a local language |
4 | | -model, with at most `1 + max_repairs` model calls. Its architecture and tradeoffs |
5 | | -are documented in [ARCHITECTURE.md](ARCHITECTURE.md). |
| 3 | +TinyAgent is a zero-dependency Python library (stdlib only, Python >= 3.10) that |
| 4 | +lets a 7B/8B local language model, served by Ollama or llama.cpp, reliably call |
| 5 | +your Python functions. |
6 | 6 |
|
7 | | -Aisha's reflection, registry, prompt, and frozen contract modules are integrated. |
8 | | -The transport, parser, and executor modules are owned by the other engineers |
9 | | -and have not yet landed. The CLI and full evaluation require the parser and |
10 | | -executor. The evaluation uses the PRD-04 fixture signatures in |
11 | | -`examples/demo_tools.py`; Aisha's `tests/fixtures_tools.py` has different |
12 | | -`search` and `set_alarm` signatures. |
| 7 | +Agent frameworks built for frontier cloud models assume the model can follow |
| 8 | +long system prompts, emit valid JSON and recover from its own errors. A local |
| 9 | +8B model does not, which leads to context exhaustion, JSON parse crashes and |
| 10 | +retry loops that never end. TinyAgent instead treats the model as a |
| 11 | +**deterministic router**: each query becomes one bounded decision, either call |
| 12 | +one registered tool with arguments, or return a final answer. A query uses at |
| 13 | +most `1 + max_repairs` model calls (`max_repairs` is 0, 1 or 2), so infinite |
| 14 | +loops are structurally impossible. The full design and tradeoffs are in |
| 15 | +[ARCHITECTURE.md](ARCHITECTURE.md). |
| 16 | + |
| 17 | +## Install |
| 18 | + |
| 19 | +```bash |
| 20 | +pip install . |
| 21 | +``` |
| 22 | + |
| 23 | +There are no runtime dependencies. For live use, run a local |
| 24 | +[Ollama](https://ollama.com) or llama.cpp server. |
13 | 25 |
|
14 | 26 | ## Quickstart |
15 | 27 |
|
16 | 28 | ```python |
17 | | -from examples.demo_tools import build_registry |
18 | | -from tinyagent import TinyAgent |
19 | | -from tinyagent.backends.simulator import SimulatedBackend |
| 29 | +from tinyagent import TinyAgent, ToolRegistry, tiny_tool |
| 30 | +from tinyagent.backends import OllamaBackend |
20 | 31 |
|
21 | | -backend = SimulatedBackend(['ACTION: add(a="2", b="3")']) |
22 | | -agent = TinyAgent(backend, build_registry()) |
| 32 | +registry = ToolRegistry() |
| 33 | + |
| 34 | +@tiny_tool(registry=registry) |
| 35 | +def add(a: int, b: int) -> int: |
| 36 | + """Add two integers.""" |
| 37 | + return a + b |
| 38 | + |
| 39 | +agent = TinyAgent(OllamaBackend("mistral:latest"), registry) |
23 | 40 | result = agent.run("What is 2+3?") |
24 | | -print(f"{result.text} (provenance: {result.provenance})") |
| 41 | +print(f"{result.text} (provenance: {result.provenance})") # 5 (provenance: tool) |
25 | 42 | ``` |
26 | 43 |
|
27 | | -After the other modules land, run `python -m tinyagent --backend sim`. |
28 | | -The CLI labels this mode `[SIMULATED]` and prints provenance next to every |
29 | | -answer. A `FINAL:` answer has `provenance=model`; it is not verified. |
| 44 | +Without a model, use the scripted backend: |
| 45 | +`python examples/quickstart.py --backend sim`. |
| 46 | + |
| 47 | +### Command line |
| 48 | + |
| 49 | +```bash |
| 50 | +tinyagent --backend sim # scripted, prints [SIMULATED] |
| 51 | +tinyagent --backend ollama --model mistral:latest "What's the weather in Pune?" |
| 52 | +tinyagent --backend llamacpp --host http://127.0.0.1:8080 "Add 4 and 9" --verbose |
| 53 | +``` |
| 54 | + |
| 55 | +`python -m tinyagent` is equivalent. The CLI routes to the demo tools in |
| 56 | +`tinyagent/demo_tools.py`; exit code is 1 when `status` is `error`. |
| 57 | + |
| 58 | +## Reading results |
| 59 | + |
| 60 | +`agent.run()` returns an `AgentResult`. Always show `provenance` next to the |
| 61 | +answer: |
| 62 | + |
| 63 | +| `status` | `provenance` | `text` | Meaning | |
| 64 | +|---|---|---|---| |
| 65 | +| `tool` | `tool` | Tool output, unmodified | A registered tool ran with validated arguments | |
| 66 | +| `final` | `model` | The model's own answer | No tool was used. **This text is not verified.** Pass `require_tool=True` to disallow it | |
| 67 | +| `error` | `none` | `CODE: detail` | See `error_code`; `run()` does not raise for model, backend or tool failures | |
| 68 | + |
| 69 | +`run()` raises only `ValueError`, for an empty query or invalid configuration. |
| 70 | + |
| 71 | +Error codes: parse (`E_EMPTY`, `E_NO_PREFIX`, `E_BAD_SYNTAX`) and argument |
| 72 | +(`E_UNKNOWN_TOOL`, `E_UNEXPECTED_ARG`, `E_MISSING_ARG`, `E_TYPE_COERCION`) |
| 73 | +failures trigger a repair prompt; if repairs run out the result is |
| 74 | +`E_RETRY_EXHAUSTED`. Backend failures (`E_BACKEND_*`) and tool failures |
| 75 | +(`E_TOOL_EXCEPTION`, `E_TOOL_TIMEOUT`) are never retried, because tools may |
| 76 | +have side effects. |
30 | 77 |
|
31 | 78 | ## Supported runtimes |
32 | 79 |
|
33 | | -The planned live backends are Ollama and llama.cpp server. Tested runtime |
34 | | -versions will be added after Deepak's transport handoff and a live run. |
35 | | -Use `python -m tinyagent --backend ollama --model llama3:8b "What is 2+3?"` |
36 | | -or `--backend llamacpp` when those modules are available. |
| 80 | +| Runtime | Endpoint | Tested | |
| 81 | +|---|---|---| |
| 82 | +| Ollama | `POST /api/generate` | Ollama 0.34.0, `mistral:latest` (7.2B, Q4_K_M), Linux x86_64, NVIDIA RTX 5070 | |
| 83 | +| llama.cpp server | `POST /completion` | Contract-tested against a local stub only | |
| 84 | + |
| 85 | +Both backends use greedy, seeded decoding (`temperature 0`, `top_k 1`). They |
| 86 | +warn when the host is not loopback, because prompts would leave the machine. |
| 87 | + |
| 88 | +## Writing tools |
| 89 | + |
| 90 | +- Parameters may be `str`, `int`, `float`, `bool` or `Optional[...]` of those. |
| 91 | + `*args`, `**kwargs`, positional-only and complex types are rejected at |
| 92 | + decoration time. |
| 93 | +- The first docstring line (max 80 chars) becomes the tool description the |
| 94 | + model sees. Clear descriptions are the main lever on tool-selection accuracy. |
| 95 | +- Arguments are coerced strictly: `"2.5"` is not an `int`, `"nan"` is not a |
| 96 | + `float`, and booleans accept only `true/false/yes/no/1/0/on/off`. |
| 97 | +- Tool output is returned as data. It is never parsed again or executed. |
| 98 | + |
| 99 | +## Security model |
| 100 | + |
| 101 | +- Model text never becomes code: no `eval`, `exec`, `compile`, `pickle` or |
| 102 | + `importlib` in the package (enforced by an AST gate in CI). |
| 103 | +- Only registered tools can run, by exact name. There is no fuzzy matching. |
| 104 | +- Ambiguous output (unterminated quotes, duplicate arguments, non-integral |
| 105 | + integers) is rejected, never guessed. |
| 106 | +- **Prompt injection:** a user query can still persuade the model to call any |
| 107 | + registered tool. Tools that change state must enforce their own authorization. |
| 108 | +- **Timeouts:** tools run in a daemon thread with a timeout (default 10 s). |
| 109 | + Python cannot kill threads, so a timed-out tool keeps running in the |
| 110 | + background. Tools that call external systems must enforce their own timeouts. |
| 111 | +- Error details are single-line and capped; prompts are redacted from backend |
| 112 | + errors and file paths from tool exceptions. |
37 | 113 |
|
38 | 114 | ## Evaluation |
39 | 115 |
|
40 | | -`python -m eval.run_eval --mode sim` runs the seeded fault corpus without a GPU |
41 | | -once the parser and executor are available. The latest |
42 | | -[evaluation report](eval/report.md) currently marks both simulated and live |
43 | | -runs as not executed. No live accuracy is claimed until a live run is recorded. |
| 116 | +| Run | Command | Result | |
| 117 | +|---|---|---| |
| 118 | +| Simulated (no model, CI) | `python -m eval.run_eval --mode sim` | 795 fault-injection cases: 280/280 recoverable faults recovered on the first call, **0 false executions**, max 2 model calls (ceiling 2), median framework overhead 0.05 ms — [report](eval/report.md) | |
| 119 | +| Live | `python -m eval.run_eval --mode live --backend ollama --model mistral:latest --record eval/fixtures/live-mistral.jsonl --out eval/report_live.json` | 50 queries: **100% tool selection, 100% argument accuracy**, 0 repairs, p50 130 ms, p95 184 ms — [report](eval/report_live.md) | |
| 120 | +| Replay | `python -m eval.run_eval --mode replay --fixture eval/fixtures/live-mistral.jsonl` | Re-runs the recorded live session exactly; also run in CI by `tests/test_replay.py` | |
| 121 | + |
| 122 | +Live numbers are for one machine and model as listed above. They are |
| 123 | +measurements, not guarantees. Other models, quantizations or hardware can |
| 124 | +differ. |
| 125 | + |
| 126 | +## Development |
| 127 | + |
| 128 | +```bash |
| 129 | +python -m unittest discover -s tests -t . # unit, integration, gate and replay tests |
| 130 | +python -m eval.run_eval --mode sim # simulated evaluation (exit 1 on any NFR failure) |
| 131 | +python -m bench.bench_parser # parser p99 latency (NFR-8) |
| 132 | +``` |
| 133 | + |
| 134 | +CI runs all three on Python 3.10, 3.12 and 3.13, plus a clean-install smoke test. |
| 135 | + |
| 136 | +## Known limitations |
44 | 137 |
|
45 | | -Known limits: one tool per query; no conversation memory, tool chaining, or |
46 | | -complex argument types. A valid but wrong tool choice cannot be detected by the |
47 | | -parser. Registered tools must enforce their own authorization and timeouts; |
48 | | -the executor cannot kill a tool thread after timeout. |
| 138 | +- One tool per query; no conversation memory, tool chaining, async API or |
| 139 | + streaming. |
| 140 | +- No complex argument types (`list`, `dict`, nested objects). |
| 141 | +- A valid but wrong tool choice cannot be detected by the parser; it is |
| 142 | + returned with `provenance="tool"`. The live evaluation measures how often this |
| 143 | + happens. |
| 144 | +- Temperature 0 is not bit-identical across hardware and runtime versions; use |
| 145 | + record/replay for reproducible tests. |
| 146 | +- No OS-level sandboxing: TinyAgent controls *which* registered function runs |
| 147 | + and *with what* validated arguments, not what the function does. |
0 commit comments