Skip to content

Commit 90d45b7

Browse files
Show the score as 93.03% ±0.8
1 parent 7ee3592 commit 90d45b7

3 files changed

Lines changed: 8 additions & 8 deletions

File tree

‎BLUEPRINT.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ This document says **what** gitagent is built to do and **why** each part exists
66

77
## 0. Goal in one paragraph
88

9-
gitagent 3 is a single static Rust binary that runs **agents defined as git repositories**. An agent is a folder of plain files (identity, rules, memory, skills, tools, hooks, schedules, sub-agents), so its behaviour is versioned, reviewed and diffed like code. One loop runs every agent, the same way in a terminal, behind an SDK, in a fresh clone of a repository, on a timetable, or inside a benchmark container. That loop is built to finish real work: it shapes what the model reads, keeps the provider's cache warm, guards against loops and running out of time, and refuses "done" until the work has been verified. The proof is public: 93.0% ±0.8 on Terminal-Bench 2.1 with Claude Opus 5.5, above every published harness on that benchmark.
9+
gitagent 3 is a single static Rust binary that runs **agents defined as git repositories**. An agent is a folder of plain files (identity, rules, memory, skills, tools, hooks, schedules, sub-agents), so its behaviour is versioned, reviewed and diffed like code. One loop runs every agent, the same way in a terminal, behind an SDK, in a fresh clone of a repository, on a timetable, or inside a benchmark container. That loop is built to finish real work: it shapes what the model reads, keeps the provider's cache warm, guards against loops and running out of time, and refuses "done" until the work has been verified. The proof is public: 93.03% ±0.8 on Terminal-Bench 2.1 with Claude Opus 5.5, above every published harness on that benchmark.
1010

1111
---
1212

@@ -43,7 +43,7 @@ gitagent 3 is a single static Rust binary that runs **agents defined as git repo
4343

4444
gitagent is measured on Terminal-Bench 2.1, and Terminal-Bench runs through **Harbor**, an open framework for running agents against containerised tasks. Harbor matters for four reasons.
4545

46-
1. **It is the same yardstick as the published results.** The leaderboards for Terminal-Bench 2.1 were produced with Harbor: the same task images, the same time limits, the same hidden tests, the same scoring. Running gitagent through it is what makes 93.0% ±0.8 comparable with the published 87.6% for Terminus 2 on the same model, rather than a number from a home-made runner.
46+
1. **It is the same yardstick as the published results.** The leaderboards for Terminal-Bench 2.1 were produced with Harbor: the same task images, the same time limits, the same hidden tests, the same scoring. Running gitagent through it is what makes 93.03% ±0.8 comparable with the published 87.6% for Terminus 2 on the same model, rather than a number from a home-made runner.
4747
2. **Grading is independent of the agent.** Harbor starts each task's container, hands the agent the instruction, stops it at the task's time limit, and when the agent has finished it runs the task's hidden tests itself and records the reward. gitagent never sees, reads or writes the tests. The result is a pass or a fail decided outside the agent, which is the only kind worth publishing.
4848
3. **It fits a static binary.** Harbor's "installed agent" model uploads the agent into the task container and runs it there. gitagent is one binary with no runtime, so installing it is a single file copy, and nothing in the container has to change. The shim that does this, `gitagent_harbor/agent.py`, is about two hundred lines.
4949
4. **It produces evidence others can check.** Harbor runs many trials in parallel, resumes trials that failed for reasons outside the agent (a revoked key, a full Docker network), keeps every trajectory and verifier log, and uploads a job publicly. The headline run's nine jobs, one per time limit, are public, so anyone can read every trial behind the number.

‎README.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
</p>
1111

1212
<p align="center">
13-
<img src="https://img.shields.io/badge/Terminal--Bench_2.1-93.0%25_%C2%B10.8-e05a44?style=flat-square" alt="Terminal-Bench 2.1: 93.0% ±0.8" />
13+
<img src="https://img.shields.io/badge/Terminal--Bench_2.1-93.03%25_%C2%B10.8-e05a44?style=flat-square" alt="Terminal-Bench 2.1: 93.03% ±0.8" />
1414
<img src="https://img.shields.io/badge/version-3.0.0-2a1f18?style=flat-square" alt="version 3.0.0" />
1515
<img src="https://img.shields.io/badge/built_with-Rust-2a1f18?style=flat-square" alt="Rust" />
1616
<img src="https://img.shields.io/github/license/open-gitagent/gitagent?style=flat-square&color=2a1f18" alt="license" />
@@ -32,7 +32,7 @@
3232
<img src="./docs/media/console.gif" alt="A gitagent session in the terminal" width="820" />
3333
</p>
3434

35-
**Terminal-Bench 2.1: 93.0% ±0.8** with Claude Opus 5.5, all 89 tasks, five runs each, code frozen at the published commit; the ± is the run-to-run variation, as on the official leaderboard. The published harnesses on the same benchmark: Terminus 2 with the same model 87.6%, Codex CLI with GPT-6 Astra 87.4%, and the highest published entry, Artificial Analysis's own harness with Claude Fable 5.1 at maximum effort, 91.4%. The numbers are under [Results](#results); the reason is under [How gitagent works](#how-gitagent-works).
35+
**Terminal-Bench 2.1: 93.03% ±0.8** with Claude Opus 5.5, all 89 tasks, five runs each, code frozen at the published commit; the ± is the run-to-run variation, as on the official leaderboard. The published harnesses on the same benchmark: Terminus 2 with the same model 87.6%, Codex CLI with GPT-6 Astra 87.4%, and the highest published entry, Artificial Analysis's own harness with Claude Fable 5.1 at maximum effort, 91.4%. The numbers are under [Results](#results); the reason is under [How gitagent works](#how-gitagent-works).
3636

3737
## Why gitagent
3838

@@ -49,7 +49,7 @@ Most agent frameworks treat the agent as code scattered across your application.
4949

5050
Fork an agent. Branch a personality. `git log` its memory. Diff its rules. Review a change to its behaviour like a code change. Every result traces back to a commit.
5151

52-
**gitagent 3** is the Rust rewrite: one static binary with no runtime to install, the harness that scored 93.0% on Terminal-Bench 2.1, a console in your terminal, parallel sub-agents with a live task board, and SDKs for Python, TypeScript and any other language over a simple line protocol. Agents, hooks, schedules and SDK code written for the TypeScript gitagent (1.x and 2.x) run unchanged, and agent folders in the OpenGAP format load too. See [Coming from the TypeScript gitagent](#coming-from-the-typescript-gitagent).
52+
**gitagent 3** is the Rust rewrite: one static binary with no runtime to install, the harness that scored 93.03% on Terminal-Bench 2.1, a console in your terminal, parallel sub-agents with a live task board, and SDKs for Python, TypeScript and any other language over a simple line protocol. Agents, hooks, schedules and SDK code written for the TypeScript gitagent (1.x and 2.x) run unchanged, and agent folders in the OpenGAP format load too. See [Coming from the TypeScript gitagent](#coming-from-the-typescript-gitagent).
5353

5454
## Install
5555

@@ -222,7 +222,7 @@ The headline run's logs are public: every trial of it, with its trajectory and v
222222

223223
| Harness | Model | Terminal-Bench 2.1 | Runs per task | Source |
224224
|---|---|---|---|---|
225-
| **gitagent 3** | Claude Opus 5.5 | **93.0% ±0.8** (414/445; 75 of 89 tasks passed every run) | 5 | [job logs on the Harbor hub](#the-same-run-by-time-limit) |
225+
| **gitagent 3** | Claude Opus 5.5 | **93.03% ±0.8** (414/445; 75 of 89 tasks passed every run) | 5 | [job logs on the Harbor hub](#the-same-run-by-time-limit) |
226226
| Terminus 2 | Claude Opus 5.5 | 87.6% | 1 | [Vals.ai](https://www.vals.ai/benchmarks/terminal-bench-2-1) |
227227
| Codex CLI | GPT-6 Astra | 87.4% ±0.9 | — | [Snorkel, the official leaderboard](https://snorkel.ai/leaderboard/terminal-bench-2-1/) |
228228
| Artificial Analysis's own harness | Claude Fable 5.1, max effort | 91.4% | — | [Artificial Analysis](https://artificialanalysis.ai/evaluations/terminalbench-2-1) |
@@ -242,7 +242,7 @@ Pass@k is the chance that at least one of k runs passes:
242242
| 60 min | 65 | 95.4% | 97.7% | 100.0% | [job](https://hub.harborframework.com/jobs/e1ea8dbd-7a55-4fa3-95de-b9338a3c1223) |
243243
| 2 h | 5 | 100.0% | 100.0% | 100.0% | [job](https://hub.harborframework.com/jobs/127445ee-63ec-44c7-8206-189eb22cd912) |
244244
| 3 h 20 min | 5 | 100.0% | 100.0% | 100.0% | [job](https://hub.harborframework.com/jobs/1d8e0bdb-7ae5-4ec7-8e54-329007590e9d) |
245-
| **all** | **445** | **93.0% ±0.8** | | | |
245+
| **all** | **445** | **93.03% ±0.8** | | | |
246246

247247
Two of the 89 tasks start a virtual machine inside the task container, which this host cannot do, and no harness has passed them here; without those two the mean is 95.2%. The ±0.8 is the standard error across the five runs (93.3, 95.5, 92.1, 92.1 and 92.1%), the measure the official leaderboard shows; across tasks, the 95% interval is 88.8–96.9%, which answers the wider question of how the score would hold on a different set of tasks. It cost $112.85, about $0.25 per run, with 92% of the input tokens served from the prompt cache.
248248

‎docs/media/results.svg‎

Lines changed: 1 addition & 1 deletion
Loading

0 commit comments

Comments
 (0)