Skip to content

Commit 07e1b50

Browse files
authored
[LLM] Add Terminal-Bench evaluation scripts (pytorch#23384)
Add setup and run scripts for evaluating the LLM server with Harbor and mini-SWE-agent. TOML files under evals/configs select the model and task settings. Harbor handles task downloads, attempts, scores, and trajectories; the launcher starts the server, checks container connectivity, and records token counts and timings. Validated with launcher dry-run, Harbor command validation, Lintrunner, and ShellCheck. Authored with assistance from OpenAI Codex.
1 parent 238506a commit 07e1b50

11 files changed

Lines changed: 663 additions & 2 deletions

File tree

‎examples/llm_server/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ examples/llm_server/
88
spec/ # language-neutral OpenAI contract ExecuTorch targets
99
conformance/ # one test suite every language server must pass
1010
python/ # Python server implementation (current)
11+
evals/ # task accuracy and performance through the server
1112
# cpp/ # future: no-Python single-binary server
1213
```
1314

@@ -106,3 +107,8 @@ Reliability guidance:
106107
`tools` were included in the request.
107108
- If a request fails with `unsupported_parameter`, remove or disable that
108109
OpenAI knob in your pi/client config.
110+
111+
## Evaluate with Terminal-Bench
112+
113+
See [Terminal-Bench](evals/terminal_bench/README.md) for setup, configuration,
114+
and evaluation commands on macOS.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.local.toml
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
[server]
2+
python = "/path/to/model-env/bin/python"
3+
module = "executorch.examples.models.muse_glimmer.serving.serve"
4+
worker_bin = "/path/to/executorch/cmake-out/examples/models/muse-glimmer/muse_glimmer_worker"
5+
model_path = "/path/to/muse-glimmer-k-quant-17G-128K-text-dflash-metal.pte"
6+
# Keep tokenizer.json, tokenizer_config.json, and chat_template.jinja together.
7+
tokenizer_path = "/path/to/tokenizer/tokenizer.json"
8+
hf_tokenizer = "/path/to/tokenizer"
9+
model_id = "muse-glimmer-30b"
10+
max_context = 131072
11+
max_sessions = 2 # One scratch session and one named session for cross-turn KV reuse.
12+
tool_parser = "atem"
13+
artifact_mode = "dflash"
14+
# host = "0.0.0.0" # Exposes the server on all network interfaces.
15+
# port = 8000
16+
17+
[terminal_bench]
18+
tasks = ["fix-git"]
19+
max_output_tokens = 4096
20+
temperature = 0.8
21+
step_limit = 100
22+
attempts = 1
23+
session_affinity = "glimmer-terminal-bench"
24+
# agent_host = "host.lima.internal" # Override automatic Docker host detection.
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Terminal-Bench (macOS)
2+
3+
Runs Terminal-Bench 2.0 through Harbor and mini-SWE-agent against the local
4+
ExecuTorch LLM server.
5+
6+
## Prerequisites
7+
8+
Prepare a self-contained `.pte`, matching tokenizer/chat template, and a worker
9+
implementing this checkout's [JSONL protocol](../../cpp/worker_loop.h). The
10+
worker must support the export's backend and input/output signatures.
11+
12+
Follow your model's artifact and worker build instructions. MLX requires Apple
13+
silicon, full Xcode, and the Metal toolchain.
14+
15+
## Run
16+
17+
From the repository root:
18+
19+
```bash
20+
cd examples/llm_server/evals/terminal_bench
21+
bash setup.sh
22+
cp ../configs/terminal-bench.example.toml ../configs/terminal-bench.local.toml
23+
# Adapt the example to your model, server module, and artifact paths.
24+
bash run.sh --config ../configs/terminal-bench.local.toml --dry-run
25+
bash run.sh --config ../configs/terminal-bench.local.toml
26+
```
27+
28+
Setup creates separate Harbor and server environments under
29+
`~/.cache/executorch-evals/`. It reuses Docker or installs Colima through Homebrew.
30+
Model export and worker builds are separate. Set `EXECUTORCH_EVAL_CACHE` before
31+
setup and run to relocate environments and results.
32+
33+
Configuration details:
34+
35+
- `[server]` selects the Python interpreter, server module, and its CLI options.
36+
The interpreter needs [`python/requirements.txt`](../../python/requirements.txt)
37+
plus any module-specific dependencies.
38+
- `[terminal_bench]` controls tasks, attempts, step limit, output token limit, and
39+
temperature. Prompt plus output must fit `[server].max_context`.
40+
- Paths support `~`, but not shell variables. Prefix relative `hf_tokenizer`
41+
directories with `./` to distinguish them from Hub IDs.
42+
- The server defaults to `0.0.0.0:8000` with no authentication. `agent_host`
43+
defaults to `host.lima.internal` for Colima and `host.docker.internal` otherwise.
44+
- Optional `session_affinity` sends an `x-session-affinity` header for KV reuse.
45+
It requires worker support for named sessions; sequential trials share the ID.
46+
47+
`--dry-run` writes resolved commands to `run.json`. Use `--task NAME` to override
48+
the task list and `--output DIR` to select a new results directory. Each run owns
49+
its server process.
50+
51+
## Outputs
52+
53+
| File | Contents |
54+
| --- | --- |
55+
| `run.json`, `source.diff`, `agent.yaml` | Resolved configuration, checkout changes, and agent settings |
56+
| `server.log` | Worker and server logs, generation timing |
57+
| `connection.json`, `connection.log` | Preflight response and Docker/curl diagnostics |
58+
| `harbor.log`, `harbor/` | Evaluation logs, rewards, and trajectories |
59+
| `metrics.json` | Token counts and timings, excluding preflight |
60+
61+
Model revisions, export/build commands, and dependency versions are not captured;
62+
retain them alongside results.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Harbor runs separately from the model server and requires Python 3.12+.
2+
harbor==0.22.0
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/usr/bin/env bash
2+
# Copyright (c) Meta Platforms, Inc. and affiliates.
3+
# All rights reserved.
4+
#
5+
# This source code is licensed under the BSD-style license found in the
6+
# LICENSE file in the root directory of this source tree.
7+
set -euo pipefail
8+
9+
evals_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)
10+
repo_dir=$(cd -- "${evals_dir}/../../.." && pwd)
11+
export EXECUTORCH_EVAL_CACHE="${EXECUTORCH_EVAL_CACHE:-${HOME}/.cache/executorch-evals}"
12+
13+
eval_python="${EXECUTORCH_EVAL_CACHE}/terminal-bench/venv/bin/python"
14+
if [[ ! -x "${eval_python}" ]]; then
15+
echo "Run bash ${evals_dir}/terminal_bench/setup.sh first." >&2
16+
exit 2
17+
fi
18+
export PYTHONPATH="${repo_dir}/src${PYTHONPATH:+:${PYTHONPATH}}"
19+
exec "${eval_python}" -m "executorch.examples.llm_server.evals.terminal_bench.runner" "$@"

0 commit comments

Comments
 (0)