|
| 1 | +# Public-proof challenge packs |
| 2 | + |
| 3 | +A public RLE proof is three printed artifacts: |
| 4 | + |
| 5 | +1. a **challenge file** (`challenge.yaml` or `challenge.yml`) |
| 6 | +2. an **expected judgment name** |
| 7 | +3. a **pass/fail** result of a deterministic predicate over a **printed receipt** |
| 8 | + |
| 9 | +That is the whole proof. The receipt is one of `summary.json`, an events JSONL file, or a preflight-adapter JSON file. If those three artifacts are absent, the claim does not count. |
| 10 | + |
| 11 | +This memo defines the pack. It does not change scoring, the game loop, or any published board number. The live cite stays `spread-2026-09-07` (Crashlanded, 10 ticks, seed 42, N=1, scoring 1.2). That board is content-first: one observation per cell, no within-cell σ. See [bench-design-n-ticks.md](bench-design-n-ticks.md). |
| 12 | + |
| 13 | +## Chain-of-thought is not a receipt |
| 14 | + |
| 15 | +A private chain-of-thought, an abridged `reasoning_trace` / `reasoning_traces` dump, or a narrative math writeup is a story about a run. It is not an RLE proof. |
| 16 | + |
| 17 | +Predicates read receipt fields only. They do not read reasoning text. A receipt object that carries `reasoning_trace` or `reasoning_traces` fails closed: those keys are narrative, and packing them into the proof artifact does not satisfy a judgment. The same keys on an events line make that line ineligible. OpenAI-style abridged reasoning traces stay in that category. |
| 18 | + |
| 19 | +## Pack shape |
| 20 | + |
| 21 | +One directory per pack. `challenge.yaml` is the challenge. `receipts/pass/` and `receipts/fail/` hold the printed files the predicates name. Fixture packs check both directories in unit tests. A harness pack checked into the tree uses the same shape; producing the receipt still means printing `summary.json`, events, or a preflight adapter — not a transcript. |
| 22 | + |
| 23 | +```yaml |
| 24 | +schema_version: 1 |
| 25 | +id: research_bench_present # matches the directory name |
| 26 | +scenario: Crashlanded Survival # scenarios/definitions name |
| 27 | +seed: 42 # RLE-side seed recorded on the summary |
| 28 | +tick_budget: 10 # ticks the receipt may consume |
| 29 | +mode: fixture # fixture | harness |
| 30 | +harness: fixture # harness name; "fixture" when mode is fixture |
| 31 | +model: fixture # model id; "fixture" when mode is fixture |
| 32 | +scoring_version: "1.2" # SCORING_VERSION the receipt was scored under |
| 33 | +expected: # judgment names only — no prose, no rates |
| 34 | + - research_bench_present |
| 35 | +pass_fail: |
| 36 | + - judgment: research_bench_present |
| 37 | + receipt: preflight.json # file under receipts/<case>/ |
| 38 | + match: object # object (JSON) | any (JSONL line) |
| 39 | + all: |
| 40 | + - path: research_bench_present |
| 41 | + eq: true |
| 42 | + - path: structures.0.def_name |
| 43 | + eq: SimpleResearchBench |
| 44 | + absent: # top-level keys that fail the judgment |
| 45 | + - reasoning_trace |
| 46 | + - reasoning_traces |
| 47 | +``` |
| 48 | +
|
| 49 | +`mode: harness` uses the same fields with `harness` and `model` set to the pair that printed the receipt. Naming a pair records provenance. It does not add a leaderboard row and it does not authorize an N>1 matrix. |
| 50 | + |
| 51 | +| Field | Rule | |
| 52 | +|-------|------| |
| 53 | +| `schema_version` | `1` | |
| 54 | +| `id` | Directory name | |
| 55 | +| `scenario` | Scenario `name` from YAML. These packs use `Crashlanded Survival`. | |
| 56 | +| `seed` | Integer. Compared to `summary.json` `random_seed` when that judgment reads the summary. | |
| 57 | +| `tick_budget` | Positive integer. `ticks_run` on a summary receipt must be ≤ this budget. | |
| 58 | +| `mode` | `fixture` (checked-in receipts, no live game, no billed model) or `harness` (a printed run). | |
| 59 | +| `harness`, `model` | Both `fixture` when `mode` is `fixture`. Both non-empty and not `fixture` when `mode` is `harness`. | |
| 60 | +| `scoring_version` | `"1.2"` (`SCORING_VERSION` in `tracking/metadata.py`). | |
| 61 | +| `expected` | List of judgment names from the catalog below. | |
| 62 | +| `pass_fail` | One predicate per expected name. `judgment` must appear in `expected`. | |
| 63 | + |
| 64 | +## Expected names |
| 65 | + |
| 66 | +`expected` is a list of typed names. A name is the judgment. It is not a score, a rate, or a sentence. |
| 67 | + |
| 68 | +| Name | What the predicate reads | |
| 69 | +|------|--------------------------| |
| 70 | +| `research_bench_present` | Brief flag from `research_bench_present()`: true when a structure `def_name` matches `is_research_bench_def` (`SimpleResearchBench`, `HiTechResearchBench`, and the same aliases). | |
| 71 | +| `research_adapters` | `RimAPIClient._adapt_research` fields `current_project`, `progress`, `completed`, `available`, plus `research_target_status` (`available`, `current`, `finished`, `locked`). The failure shape is the tech-level bag the adapter refuses: `completed: ["Medieval"]`, empty `available`, status `locked`. | |
| 72 | +| `ensure_live_save` | `LiveSaveStatus` fields `save_name`, `live_save_sha256`, `pinned_sha256`, `copied`. Pin-match: the two hashes are equal and `error` is absent. Stale AppData that cannot be staged raises `LiveSavePinError` and fails closed (`scenarios/loader.py`). | |
| 73 | +| `stockpile_delete_quarantine` | `action_exec` event for `stockpile_delete`: `success` false and `error` containing `quarantined`. `QUARANTINED_WRITES` hides the endpoint; the executor refuses it before HTTP. | |
| 74 | +| `preflight_contract` | The five dispatch gates in `orchestration/preflight.py`: growing-zone already covered, `work_priority` normalized to a WorkTypeDef, `research_target_status`, a resolved tend pair, and `stockpile_delete` quarantined. | |
| 75 | +| `composite_receipt` | Per-run `summary.json` from `scripts/run_scenario.py`: `scoring_version`, `scenario`, `random_seed`, `harness`, `model`, `ticks_run`, `final_score`, `outcome`. | |
| 76 | + |
| 77 | +Names outside this catalog are not judgments. Adding a name is a doc change plus a fixture, not a new board number. |
| 78 | + |
| 79 | +## Pass/fail |
| 80 | + |
| 81 | +Each predicate names one receipt file and a list of clauses. A case passes when every clause holds. `match: object` applies clauses to one JSON object. `match: any` applies them to at least one JSONL line after `where` (a map of dotted-path equalities, usually `event_type: action_exec`). |
| 82 | + |
| 83 | +A clause has one operator: |
| 84 | + |
| 85 | +| Operator | Holds when | |
| 86 | +|----------|------------| |
| 87 | +| `eq`, `ne` | The value at `path` equals / differs. | |
| 88 | +| `lte`, `gte` | Numeric comparison. Booleans are not numbers. | |
| 89 | +| `contains`, `not_contains` | Substring, or list membership. | |
| 90 | +| `type` | `number`, `string`, `bool`, `list`, or `object`. | |
| 91 | +| `present` | Path exists and is not null. | |
| 92 | +| `absent` | Path is missing or null. On the predicate itself, `absent:` is a list of top-level keys. | |
| 93 | +| `eq_path` | Value at `path` equals the value at another dotted path (`live_save_sha256` vs `pinned_sha256`). | |
| 94 | + |
| 95 | +Paths are dotted. A numeric segment indexes a list (`structures.0.def_name`). |
| 96 | + |
| 97 | +`final_score` is checked only as a number on the summary. The predicate does not compare it to a published composite. A fixture may carry any in-range shape value; that value is not a result. |
| 98 | + |
| 99 | +## Receipts |
| 100 | + |
| 101 | +**`summary.json`** — the object `_build_run_summary` writes (`collect_metadata` plus scenario, harness, model, `ticks_run`, `outcome`, `final_score`, `live_save_sha256`, `live_save_copied`). One file is one run. |
| 102 | + |
| 103 | +**Events JSONL** — `Event` lines from `tracking/event_log.py` (`event_type`, `tick`, `data`). `action_exec` lines carry `action_type`, `success`, `error`, `parameters`. |
| 104 | + |
| 105 | +**`preflight.json`** — a printed adapter or gate receipt: the research-bench flag and structure defs, the research-adapter object, a `LiveSaveStatus` / `LiveSavePinError` record, or the preflight gate fields. It is not a model transcript. |
| 106 | + |
| 107 | +## N stays on the claim surface |
| 108 | + |
| 109 | +Per-run `summary.json` has no `n_runs` field and no pass rate. `n_runs` exists on `BaselineReference` and `BaselinePoint` (how many no-agent runs were aggregated into the pinned baseline sidecar). A `composite_receipt` predicate rejects `n_runs` and `pass_rate` on the summary so a single receipt cannot pose as a matrix cell. |
| 110 | + |
| 111 | +The published N=1 caveat is unchanged: `spread-2026-09-07` is one observation per cell. This packing spec does not size a matrix, does not report a new pass rate, and does not move the 25-tick horizon lock. |
0 commit comments