Skip to content

Commit 77c5fb8

Browse files
JarbasAlclaude
andcommitted
docs(spec): make invariants normative and tuned constants advisory, and close four underspecified clauses
R5b named MAX_AUTO_WEIGHT_PER_PAIR and its 0.25 auto-vote weight as normative text; restate it as the invariant (a symmetric cap, proportional scaling that preserves the observed win rate, dataset size never a lever, deterministic replay) with the tuned constant pointed at arena/assembler.py and docs/methodology.md instead of duplicated inline. R1 did not say what happens when a dataset publishes a sample-set manifest; add that both predictions must fall inside the managed set, matching arena.predictions.group_rows. R12 described tally as fetching every issue and replaying "from scratch" every run, which reads as contradicting the append-only vote record in §6; restate it as fetch-to-detect-new, append once, then replay the full committed record. §3.1 left trained_on's coverage of a republished corpus and label_set's ranking floor unstated; both are now spelled out, with label_set pointing at the per-league floor in leagues.md. The dataset lang field now states that a bare ISO 639-3 primary subtag is accepted only from registry/schemas.py's closed allow-list. Verified: MAX_AUTO_WEIGHT_PER_PAIR = 5.0 (arena/assembler.py:61) and BARE_PRIMARY_SUBTAGS = {"arb", "kab"} (registry/schemas.py:52) match the text referencing them. tests/test_spec_coverage.py (3 tests) passes unchanged since R1/R5b/R12 keep their existing R-numbers. ste_lint.py PASS on both docs files; the style grep for "currently" returns the same 3 pre-existing hits as origin/dev, no new ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent e633a7b commit 77c5fb8

1 file changed

Lines changed: 57 additions & 24 deletions

File tree

docs/SPECIFICATION.md

Lines changed: 57 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,13 @@ Padatious × Adapt).
7878
An offline fighter MUST declare the `label_set` its artefact was trained on
7979
— the dataset_ids whose labels it can emit. It is benchmarked only on those
8080
corpora; anywhere else its answers measure a label-space mismatch rather than
81-
the engine.
81+
the engine. The declared claim is checked, not trusted: before scoring, the
82+
runner intersects the loaded model's own class list with the corpus's
83+
labels, and a fighter whose model classes reach no overlap with a corpus
84+
in its own declared `label_set` is unranked there rather than scored on a
85+
board of zeros; the reachable coverage floor is per-league and documented
86+
in `docs/leagues.md`, alongside how a board records the class overlap a
87+
fighter actually measured.
8288

8389
The per-league task definitions and the exact metric formulas (what each
8490
benchmark board ranks by and what seeds ELO) are specified in
@@ -131,18 +137,29 @@ plugin class it instantiates), `types` (architecture tags: `GOFAI`,
131137
`fuzzy-match`, `neural-net`, `template-match`, `keyword-match`, `embedding`,
132138
`LLM`, `ensemble`), `description`, `model`, `links`.
133139

134-
`trained_on` lists dataset ids whose recordings were in this competitor's
135-
training data — the deny-list mirror of `train_datasets` below: a fighter
136-
stays registered and competes normally everywhere else, but every scoring
137-
point (the media/wake-word bench, `arena.predictions.group_rows`, the
138-
autorun scheduler) skips the `(fighter, dataset)` pair instead of grading
139-
it. `validate_registry` rejects a `trained_on` id that does not resolve to
140-
a registered dataset for the competitor's own modality.
140+
`trained_on` lists every dataset id whose recordings were in this
141+
competitor's training data — the deny-list mirror of `train_datasets`
142+
below: a fighter stays registered and competes normally everywhere else,
143+
but every scoring point (the media/wake-word bench,
144+
`arena.predictions.group_rows`, the autorun scheduler) skips the
145+
`(fighter, dataset)` pair instead of grading it, so a fighter is never
146+
scored on data it was trained on. A corpus republished under a different
147+
dataset id is a different id as far as this list is concerned — it is not
148+
covered by an entry naming the original id, so the registrant MUST list
149+
every id the training recordings were published under, including
150+
republications. `validate_registry` rejects a `trained_on` id that does not
151+
resolve to a registered dataset for the competitor's own modality.
141152

142153
**Datasets** (`registry/datasets/<modality>/<id>.json`): one corpus per
143154
entry, source (HF id + revision + split or per-lang `file_pattern`),
144155
`reference_fields` (the datashape contract), license, `lang` (or
145-
`lang: multi` plus a `langs` list), and a `role`. `display_name` is the
156+
`lang: multi` plus a `langs` list), and a `role`. Every `lang` tag MUST be a
157+
full BCP-47 lang-REGION tag; a bare primary subtag is accepted only for a
158+
language on `registry/schemas.py`'s allow-list, because none of the regions
159+
it is spoken in is a dialect the corpora distinguish. That allow-list is
160+
closed: adding a language to it is an owner ruling recorded in the schema
161+
file itself, not a convenience a registrant can reach for by leaving off a
162+
region. `display_name` is the
146163
human-readable corpus name the leaderboard shows in place of the code-name
147164
`dataset_id`, and `summary` is the plain-language paragraph that tells a
148165
visitor where the data comes from, what one row is and what changes how to
@@ -278,7 +295,13 @@ Voting options MUST include: candidate A, candidate B, tie, both-wrong.
278295
## 4. Matchmaking rules
279296

280297
- **R1, Same stimulus.** A battle pairs two predictions for the *same*
281-
`sample_id` from the same dataset, by two different competitors.
298+
`sample_id` from the same dataset, by two different competitors. Where the
299+
dataset publishes a sample-set manifest (a `sample_policy`-capped
300+
dataset's `sample_sets/<lang>.json`, see `docs/runner.md`), both
301+
predictions MUST also fall inside that manifest's id set — a sample
302+
either competitor answered outside the published manifest carries no
303+
signal about the managed comparison and MUST NOT seed a battle
304+
(`arena.predictions.group_rows`).
282305
- **R2, Identical outputs are never battled** (no signal for a voter).
283306
- **R3, Prefer discriminative samples.** Within each competitor pair,
284307
both-wrong samples sort first, then one-wrong disagreements. Battle pools
@@ -298,21 +321,31 @@ Voting options MUST include: candidate A, candidate B, tie, both-wrong.
298321
statistically indistinguishable is benchmark noise and MUST NOT seed the
299322
rating.
300323
- **R5b, Weight cap.** A pair's total Bradley-Terry auto-vote weight is
301-
capped at `MAX_AUTO_WEIGHT_PER_PAIR` (5 human-vote-equivalents),
302-
proportionally scaled to preserve the observed win rate, dataset size
303-
MUST NOT be a lever on how much the auto-vote seed can move a pair's
304-
rating. `ovos-arena audit-seeds` reports every pair's weight and whether
305-
it sits at the cap.
306-
- **R12, Full-history replay.** `tally` MUST fetch every `vote`-labelled
307-
issue (open and closed), not only issues opened since the previous run,
308-
the vote log is the complete issue history (§6), and every tally run
309-
replays it from scratch. An issue absent from the vote record is
310-
recorded exactly once, with the battle context it was cast on; an issue
311-
already in the record MUST NOT be re-parsed, so a later title edit
312-
changes no rating. Already-closed issues MUST NOT be re-commented on or
324+
capped, in human-vote-equivalent units, at a fixed ceiling applied
325+
symmetrically to both members of the pair; the weight below the cap is
326+
scaled proportionally so it preserves the pair's observed auto-battle win
327+
rate rather than distorting it. Dataset size MUST NOT be a lever on how
328+
much the auto-vote seed can move a pair's rating — a benchmark corpus with
329+
ten thousand samples must not outweigh a hand-voted pair the way an
330+
uncapped weight would let it. Replay MUST reproduce the capped weight
331+
deterministically from the published predictions alone, same as any other
332+
seed value (§2 P5). The tuned ceiling is the `MAX_AUTO_WEIGHT_PER_PAIR`
333+
constant in `arena/assembler.py`, worked through alongside the per-round
334+
scaling in `docs/methodology.md`. `ovos-arena audit-seeds` reports
335+
every pair's weight and whether it sits at the cap.
336+
- **R12, Full-history replay.** The vote record (§6) is the complete history
337+
of votes as they were publicly cast, one committed line per `vote`-labelled
338+
issue, appended once and never rewritten. `tally` fetches every
339+
`vote`-labelled issue (open and closed) so it can detect any one not yet
340+
in the record, records that issue exactly once with the battle context it
341+
was cast on, and then rebuilds every leaderboard by replaying the FULL
342+
record from the start — not only the newly appended lines — so the rating
343+
never depends on which run first observed a given vote. An issue already
344+
in the record MUST NOT be re-parsed, so a later title edit changes no
345+
rating, and an already-closed issue MUST NOT be re-commented on or
313346
re-closed. `assemble`, `tally` and `verify-replay` MUST build every
314-
leaderboard through the same replay of that record, so what `assemble`
315-
publishes is what `verify-replay` reproduces.
347+
leaderboard through that same replay of the committed record, so what
348+
`assemble` publishes is what `verify-replay` reproduces.
316349
- **R13, Vote fraud rules** (`arena/fraud.py`, pure functions of the vote
317350
log, see `docs/methodology.md` for the full rationale):
318351
- one vote per (voter, battle), R1's battle identity dedupe. - a per-voter, per-league, per-UTC-day cap (`DAILY_VOTE_CAP = 50`). - an account-age gate (`NEW_ACCOUNT_MIN_DAYS = 7`) using a creation-date

0 commit comments

Comments
 (0)