You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 90d45b7
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: BLUEPRINT.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ This document says **what** gitagent is built to do and **why** each part exists
6
6
7
7
## 0. Goal in one paragraph
8
8
9
-
gitagent 3 is a single static Rust binary that runs **agents defined as git repositories**. An agent is a folder of plain files (identity, rules, memory, skills, tools, hooks, schedules, sub-agents), so its behaviour is versioned, reviewed and diffed like code. One loop runs every agent, the same way in a terminal, behind an SDK, in a fresh clone of a repository, on a timetable, or inside a benchmark container. That loop is built to finish real work: it shapes what the model reads, keeps the provider's cache warm, guards against loops and running out of time, and refuses "done" until the work has been verified. The proof is public: 93.0% ±0.8 on Terminal-Bench 2.1 with Claude Opus 5.5, above every published harness on that benchmark.
9
+
gitagent 3 is a single static Rust binary that runs **agents defined as git repositories**. An agent is a folder of plain files (identity, rules, memory, skills, tools, hooks, schedules, sub-agents), so its behaviour is versioned, reviewed and diffed like code. One loop runs every agent, the same way in a terminal, behind an SDK, in a fresh clone of a repository, on a timetable, or inside a benchmark container. That loop is built to finish real work: it shapes what the model reads, keeps the provider's cache warm, guards against loops and running out of time, and refuses "done" until the work has been verified. The proof is public: 93.03% ±0.8 on Terminal-Bench 2.1 with Claude Opus 5.5, above every published harness on that benchmark.
10
10
11
11
---
12
12
@@ -43,7 +43,7 @@ gitagent 3 is a single static Rust binary that runs **agents defined as git repo
43
43
44
44
gitagent is measured on Terminal-Bench 2.1, and Terminal-Bench runs through **Harbor**, an open framework for running agents against containerised tasks. Harbor matters for four reasons.
45
45
46
-
1.**It is the same yardstick as the published results.** The leaderboards for Terminal-Bench 2.1 were produced with Harbor: the same task images, the same time limits, the same hidden tests, the same scoring. Running gitagent through it is what makes 93.0% ±0.8 comparable with the published 87.6% for Terminus 2 on the same model, rather than a number from a home-made runner.
46
+
1.**It is the same yardstick as the published results.** The leaderboards for Terminal-Bench 2.1 were produced with Harbor: the same task images, the same time limits, the same hidden tests, the same scoring. Running gitagent through it is what makes 93.03% ±0.8 comparable with the published 87.6% for Terminus 2 on the same model, rather than a number from a home-made runner.
47
47
2.**Grading is independent of the agent.** Harbor starts each task's container, hands the agent the instruction, stops it at the task's time limit, and when the agent has finished it runs the task's hidden tests itself and records the reward. gitagent never sees, reads or writes the tests. The result is a pass or a fail decided outside the agent, which is the only kind worth publishing.
48
48
3.**It fits a static binary.** Harbor's "installed agent" model uploads the agent into the task container and runs it there. gitagent is one binary with no runtime, so installing it is a single file copy, and nothing in the container has to change. The shim that does this, `gitagent_harbor/agent.py`, is about two hundred lines.
49
49
4.**It produces evidence others can check.** Harbor runs many trials in parallel, resumes trials that failed for reasons outside the agent (a revoked key, a full Docker network), keeps every trajectory and verifier log, and uploads a job publicly. The headline run's nine jobs, one per time limit, are public, so anyone can read every trial behind the number.
<imgsrc="./docs/media/console.gif"alt="A gitagent session in the terminal"width="820" />
33
33
</p>
34
34
35
-
**Terminal-Bench 2.1: 93.0% ±0.8** with Claude Opus 5.5, all 89 tasks, five runs each, code frozen at the published commit; the ± is the run-to-run variation, as on the official leaderboard. The published harnesses on the same benchmark: Terminus 2 with the same model 87.6%, Codex CLI with GPT-6 Astra 87.4%, and the highest published entry, Artificial Analysis's own harness with Claude Fable 5.1 at maximum effort, 91.4%. The numbers are under [Results](#results); the reason is under [How gitagent works](#how-gitagent-works).
35
+
**Terminal-Bench 2.1: 93.03% ±0.8** with Claude Opus 5.5, all 89 tasks, five runs each, code frozen at the published commit; the ± is the run-to-run variation, as on the official leaderboard. The published harnesses on the same benchmark: Terminus 2 with the same model 87.6%, Codex CLI with GPT-6 Astra 87.4%, and the highest published entry, Artificial Analysis's own harness with Claude Fable 5.1 at maximum effort, 91.4%. The numbers are under [Results](#results); the reason is under [How gitagent works](#how-gitagent-works).
36
36
37
37
## Why gitagent
38
38
@@ -49,7 +49,7 @@ Most agent frameworks treat the agent as code scattered across your application.
49
49
50
50
Fork an agent. Branch a personality. `git log` its memory. Diff its rules. Review a change to its behaviour like a code change. Every result traces back to a commit.
51
51
52
-
**gitagent 3** is the Rust rewrite: one static binary with no runtime to install, the harness that scored 93.0% on Terminal-Bench 2.1, a console in your terminal, parallel sub-agents with a live task board, and SDKs for Python, TypeScript and any other language over a simple line protocol. Agents, hooks, schedules and SDK code written for the TypeScript gitagent (1.x and 2.x) run unchanged, and agent folders in the OpenGAP format load too. See [Coming from the TypeScript gitagent](#coming-from-the-typescript-gitagent).
52
+
**gitagent 3** is the Rust rewrite: one static binary with no runtime to install, the harness that scored 93.03% on Terminal-Bench 2.1, a console in your terminal, parallel sub-agents with a live task board, and SDKs for Python, TypeScript and any other language over a simple line protocol. Agents, hooks, schedules and SDK code written for the TypeScript gitagent (1.x and 2.x) run unchanged, and agent folders in the OpenGAP format load too. See [Coming from the TypeScript gitagent](#coming-from-the-typescript-gitagent).
53
53
54
54
## Install
55
55
@@ -222,7 +222,7 @@ The headline run's logs are public: every trial of it, with its trajectory and v
222
222
223
223
| Harness | Model | Terminal-Bench 2.1 | Runs per task | Source |
224
224
|---|---|---|---|---|
225
-
|**gitagent 3**| Claude Opus 5.5 |**93.0% ±0.8** (414/445; 75 of 89 tasks passed every run) | 5 |[job logs on the Harbor hub](#the-same-run-by-time-limit)|
225
+
|**gitagent 3**| Claude Opus 5.5 |**93.03% ±0.8** (414/445; 75 of 89 tasks passed every run) | 5 |[job logs on the Harbor hub](#the-same-run-by-time-limit)|
226
226
| Terminus 2 | Claude Opus 5.5 | 87.6% | 1 |[Vals.ai](https://www.vals.ai/benchmarks/terminal-bench-2-1)|
227
227
| Codex CLI | GPT-6 Astra | 87.4% ±0.9 | — |[Snorkel, the official leaderboard](https://snorkel.ai/leaderboard/terminal-bench-2-1/)|
228
228
| Artificial Analysis's own harness | Claude Fable 5.1, max effort | 91.4% | — |[Artificial Analysis](https://artificialanalysis.ai/evaluations/terminalbench-2-1)|
@@ -242,7 +242,7 @@ Pass@k is the chance that at least one of k runs passes:
| 3 h 20 min | 5 | 100.0% | 100.0% | 100.0% |[job](https://hub.harborframework.com/jobs/1d8e0bdb-7ae5-4ec7-8e54-329007590e9d)|
245
-
|**all**|**445**|**93.0% ±0.8**||||
245
+
|**all**|**445**|**93.03% ±0.8**||||
246
246
247
247
Two of the 89 tasks start a virtual machine inside the task container, which this host cannot do, and no harness has passed them here; without those two the mean is 95.2%. The ±0.8 is the standard error across the five runs (93.3, 95.5, 92.1, 92.1 and 92.1%), the measure the official leaderboard shows; across tasks, the 95% interval is 88.8–96.9%, which answers the wider question of how the score would hold on a different set of tasks. It cost $112.85, about $0.25 per run, with 92% of the input tokens served from the prompt cache.
0 commit comments