You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2b37bf3
Browse filesBrowse the repository at this point in the historyBrowse files
docs: acknowledge published scoring 1.2 Sep 6 board (#65)
README ELI5 and Benchmark Results still treated the 1.2 harness×model
matrix as unpublished and only pasted the June 1.1 felix table. Point
at the live spread-2026-09-06 board on rle.appsprout.dev / HF, keep
the 1.1 rows labeled historical and non-comparable, and include the
N=1 caveat.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
Copy file name to clipboardExpand all lines: README.md
+19-4Lines changed: 19 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
## ELI5
4
4
5
-
RLE is a public research benchmark — not a product — where AI agent setups try to keep a RimWorld colony alive. The published leaderboard is one 11-model Crashlanded run (2026-06-11, N=1, seed 42, 10 ticks) on the `felix` harness at scoring 1.1 — Grok 4.3 leads at mean composite 0.836; it is content-first and not statistically valid. Winners advance to N=4; N=4 is not published. Those rows are not comparable to scoring 1.2. Harness × model and scoring 1.2 are how the code works now; that matrix is not a published result yet. Full table: [Benchmark Results](#benchmark-results).
5
+
RLE is a public research benchmark — not a product — where AI agent setups try to keep a RimWorld colony alive. The live published board is [spread-2026-09-06](https://huggingface.co/datasets/AppSprout/rle-benchmarks) on [rle.appsprout.dev](https://rle.appsprout.dev): scoring 1.2, harness × model, Crashlanded, N=1, seed 42, 10 ticks. Grok Build (ACP) leads at mean composite 0.811 (final 0.833). N=1 is content-first and not statistically valid. Winners advance to N=4; N=4 is not published. The June 2026 scoring 1.1 `felix` 11-model table is historical and **not comparable** (different scoring era). Full table: [Benchmark Results](#benchmark-results).
6
6
7
7
A **harness × model** benchmark: swappable agent harnesses manage a RimWorld colony under uncertainty and are scored on the same footing against an unmanaged baseline. Think [FLE](https://github.com/chenhao-wang/FLE) (Factorio Learning Environment) but stochastic, multi-agent-capable, and with the *harness* — not just the model — as a first-class variable.
**11-model spread, `felix` harness, scoring 1.1** — Crashlanded, 10 ticks, seed 42, 2026-06-11. **N=1, content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks)and [rle.appsprout.dev](https://rle.appsprout.dev) (same `site_data.json` payload).
221
+
**Live board — `spread-2026-09-06`, scoring 1.2, harness × model** — Crashlanded, 10 ticks, seed 42, N=1. **Content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on [rle.appsprout.dev](https://rle.appsprout.dev) and the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks)(`runs/spread-2026-09-06`).
222
222
223
-
These rows predate the harness axis and scoring 1.2: they were all produced by the Felix harness and include the since-removed `coordination` / `communication_efficiency` metrics, so they are not comparable to 1.2 runs. The next published spread will be a harness × model matrix at scoring 1.2.
223
+
`felix` here is a technical harness name (the original 7-agent stack), not a product.
224
+
225
+
| # | Harness / model | Mean | Final | vs baseline | Cost |
Measured against a pinned no-agent baseline (4 seeds, mean time-to-end 8.0 days). **0 of 4 harness/model rows beat the unmanaged baseline.** Highest mean composite is not the same as beating baseline.
233
+
234
+
Costs match the live site: Felix $0.86 is OpenRouter billed; OpenCode $1.59* is estimated from OpenRouter grok-4.6 tokens×rates (796917 tokens; OSS harness still billed via OpenRouter); Grok Build ACP $7.00* is console. raw-grok cost is unknown and omitted from the cost frontier. `*` = estimated / console. Total spend $9.45.
235
+
236
+
### Historical — scoring 1.1 (non-comparable)
237
+
238
+
**11-model spread, `felix` harness, scoring 1.1** — Crashlanded, 10 ticks, seed 42, 2026-06-11 (`spread-2026-06-11`). **N=1, content-first — not statistically valid.** These rows predate the harness axis and scoring 1.2: they were all produced by the Felix harness and include the since-removed `coordination` / `communication_efficiency` metrics, so they are **not comparable** to the 1.2 board above.
224
239
225
240
| # | Model | Mean | Final | vs baseline | Cost |
226
241
|---|-------|------|-------|-------------|------|
@@ -236,7 +251,7 @@ These rows predate the harness axis and scoring 1.2: they were all produced by t
Measured against a pinned no-agent baseline (4 seeds, mean time-to-end 8.0 days). **1 of 11 models beat the no-agent baseline** (GLM-5.1). Highest mean composite is not the same as beating baseline. Costs marked `~` are token-count estimates (subscription-billed); unmarked costs are OpenRouter billed.
254
+
**1 of 11 models beat the no-agent baseline** (GLM-5.1). Costs marked `~` are token-count estimates (subscription-billed); unmarked costs are OpenRouter billed.
0 commit comments