|
| 1 | +# Terminal-Bench (macOS) |
| 2 | + |
| 3 | +Runs Terminal-Bench 2.0 through Harbor and mini-SWE-agent against the local |
| 4 | +ExecuTorch LLM server. |
| 5 | + |
| 6 | +## Prerequisites |
| 7 | + |
| 8 | +Prepare a self-contained `.pte`, matching tokenizer/chat template, and a worker |
| 9 | +implementing this checkout's [JSONL protocol](../../cpp/worker_loop.h). The |
| 10 | +worker must support the export's backend and input/output signatures. |
| 11 | + |
| 12 | +Follow your model's artifact and worker build instructions. MLX requires Apple |
| 13 | +silicon, full Xcode, and the Metal toolchain. |
| 14 | + |
| 15 | +## Run |
| 16 | + |
| 17 | +From the repository root: |
| 18 | + |
| 19 | +```bash |
| 20 | +cd examples/llm_server/evals/terminal_bench |
| 21 | +bash setup.sh |
| 22 | +cp ../configs/terminal-bench.example.toml ../configs/terminal-bench.local.toml |
| 23 | +# Adapt the example to your model, server module, and artifact paths. |
| 24 | +bash run.sh --config ../configs/terminal-bench.local.toml --dry-run |
| 25 | +bash run.sh --config ../configs/terminal-bench.local.toml |
| 26 | +``` |
| 27 | + |
| 28 | +Setup creates separate Harbor and server environments under |
| 29 | +`~/.cache/executorch-evals/`. It reuses Docker or installs Colima through Homebrew. |
| 30 | +Model export and worker builds are separate. Set `EXECUTORCH_EVAL_CACHE` before |
| 31 | +setup and run to relocate environments and results. |
| 32 | + |
| 33 | +Configuration details: |
| 34 | + |
| 35 | +- `[server]` selects the Python interpreter, server module, and its CLI options. |
| 36 | + The interpreter needs [`python/requirements.txt`](../../python/requirements.txt) |
| 37 | + plus any module-specific dependencies. |
| 38 | +- `[terminal_bench]` controls tasks, attempts, step limit, output token limit, and |
| 39 | + temperature. Prompt plus output must fit `[server].max_context`. |
| 40 | +- Paths support `~`, but not shell variables. Prefix relative `hf_tokenizer` |
| 41 | + directories with `./` to distinguish them from Hub IDs. |
| 42 | +- The server defaults to `0.0.0.0:8000` with no authentication. `agent_host` |
| 43 | + defaults to `host.lima.internal` for Colima and `host.docker.internal` otherwise. |
| 44 | +- Optional `session_affinity` sends an `x-session-affinity` header for KV reuse. |
| 45 | + It requires worker support for named sessions; sequential trials share the ID. |
| 46 | + |
| 47 | +`--dry-run` writes resolved commands to `run.json`. Use `--task NAME` to override |
| 48 | +the task list and `--output DIR` to select a new results directory. Each run owns |
| 49 | +its server process. |
| 50 | + |
| 51 | +## Outputs |
| 52 | + |
| 53 | +| File | Contents | |
| 54 | +| --- | --- | |
| 55 | +| `run.json`, `source.diff`, `agent.yaml` | Resolved configuration, checkout changes, and agent settings | |
| 56 | +| `server.log` | Worker and server logs, generation timing | |
| 57 | +| `connection.json`, `connection.log` | Preflight response and Docker/curl diagnostics | |
| 58 | +| `harbor.log`, `harbor/` | Evaluation logs, rewards, and trajectories | |
| 59 | +| `metrics.json` | Token counts and timings, excluding preflight | |
| 60 | + |
| 61 | +Model revisions, export/build commands, and dependency versions are not captured; |
| 62 | +retain them alongside results. |
0 commit comments