You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(spec): make invariants normative and tuned constants advisory, and close four underspecified clauses
R5b named MAX_AUTO_WEIGHT_PER_PAIR and its 0.25 auto-vote weight as normative
text; restate it as the invariant (a symmetric cap, proportional scaling that
preserves the observed win rate, dataset size never a lever, deterministic
replay) with the tuned constant pointed at arena/assembler.py and
docs/methodology.md instead of duplicated inline. R1 did not say what happens
when a dataset publishes a sample-set manifest; add that both predictions
must fall inside the managed set, matching arena.predictions.group_rows. R12
described tally as fetching every issue and replaying "from scratch" every
run, which reads as contradicting the append-only vote record in §6; restate
it as fetch-to-detect-new, append once, then replay the full committed
record. §3.1 left trained_on's coverage of a republished corpus and
label_set's ranking floor unstated; both are now spelled out, with label_set
pointing at the per-league floor in leagues.md. The dataset lang field
now states that a bare ISO 639-3 primary subtag is accepted only from
registry/schemas.py's closed allow-list.
Verified: MAX_AUTO_WEIGHT_PER_PAIR = 5.0 (arena/assembler.py:61) and
BARE_PRIMARY_SUBTAGS = {"arb", "kab"} (registry/schemas.py:52) match the
text referencing them. tests/test_spec_coverage.py (3 tests) passes
unchanged since R1/R5b/R12 keep their existing R-numbers. ste_lint.py
PASS on both docs files; the style grep for "currently" returns the
same 3 pre-existing hits as origin/dev, no new ones.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`trained_on` lists dataset ids whose recordings were in this competitor's
135
-
training data — the deny-list mirror of `train_datasets` below: a fighter
136
-
stays registered and competes normally everywhere else, but every scoring
137
-
point (the media/wake-word bench, `arena.predictions.group_rows`, the
138
-
autorun scheduler) skips the `(fighter, dataset)` pair instead of grading
139
-
it. `validate_registry` rejects a `trained_on` id that does not resolve to
140
-
a registered dataset for the competitor's own modality.
140
+
`trained_on` lists every dataset id whose recordings were in this
141
+
competitor's training data — the deny-list mirror of `train_datasets`
142
+
below: a fighter stays registered and competes normally everywhere else,
143
+
but every scoring point (the media/wake-word bench,
144
+
`arena.predictions.group_rows`, the autorun scheduler) skips the
145
+
`(fighter, dataset)` pair instead of grading it, so a fighter is never
146
+
scored on data it was trained on. A corpus republished under a different
147
+
dataset id is a different id as far as this list is concerned — it is not
148
+
covered by an entry naming the original id, so the registrant MUST list
149
+
every id the training recordings were published under, including
150
+
republications. `validate_registry` rejects a `trained_on` id that does not
151
+
resolve to a registered dataset for the competitor's own modality.
141
152
142
153
**Datasets** (`registry/datasets/<modality>/<id>.json`): one corpus per
143
154
entry, source (HF id + revision + split or per-lang `file_pattern`),
144
155
`reference_fields` (the datashape contract), license, `lang` (or
145
-
`lang: multi` plus a `langs` list), and a `role`. `display_name` is the
156
+
`lang: multi` plus a `langs` list), and a `role`. Every `lang` tag MUST be a
157
+
full BCP-47 lang-REGION tag; a bare primary subtag is accepted only for a
158
+
language on `registry/schemas.py`'s allow-list, because none of the regions
159
+
it is spoken in is a dialect the corpora distinguish. That allow-list is
160
+
closed: adding a language to it is an owner ruling recorded in the schema
161
+
file itself, not a convenience a registrant can reach for by leaving off a
162
+
region. `display_name` is the
146
163
human-readable corpus name the leaderboard shows in place of the code-name
147
164
`dataset_id`, and `summary` is the plain-language paragraph that tells a
148
165
visitor where the data comes from, what one row is and what changes how to
@@ -278,7 +295,13 @@ Voting options MUST include: candidate A, candidate B, tie, both-wrong.
278
295
## 4. Matchmaking rules
279
296
280
297
-**R1, Same stimulus.** A battle pairs two predictions for the *same*
281
-
`sample_id` from the same dataset, by two different competitors.
298
+
`sample_id` from the same dataset, by two different competitors. Where the
299
+
dataset publishes a sample-set manifest (a `sample_policy`-capped
300
+
dataset's `sample_sets/<lang>.json`, see `docs/runner.md`), both
301
+
predictions MUST also fall inside that manifest's id set — a sample
302
+
either competitor answered outside the published manifest carries no
303
+
signal about the managed comparison and MUST NOT seed a battle
304
+
(`arena.predictions.group_rows`).
282
305
-**R2, Identical outputs are never battled** (no signal for a voter).
283
306
-**R3, Prefer discriminative samples.** Within each competitor pair,
284
307
both-wrong samples sort first, then one-wrong disagreements. Battle pools
@@ -298,21 +321,31 @@ Voting options MUST include: candidate A, candidate B, tie, both-wrong.
298
321
statistically indistinguishable is benchmark noise and MUST NOT seed the
299
322
rating.
300
323
-**R5b, Weight cap.** A pair's total Bradley-Terry auto-vote weight is
301
-
capped at `MAX_AUTO_WEIGHT_PER_PAIR` (5 human-vote-equivalents),
302
-
proportionally scaled to preserve the observed win rate, dataset size
303
-
MUST NOT be a lever on how much the auto-vote seed can move a pair's
304
-
rating. `ovos-arena audit-seeds` reports every pair's weight and whether
305
-
it sits at the cap.
306
-
-**R12, Full-history replay.**`tally` MUST fetch every `vote`-labelled
307
-
issue (open and closed), not only issues opened since the previous run,
308
-
the vote log is the complete issue history (§6), and every tally run
309
-
replays it from scratch. An issue absent from the vote record is
310
-
recorded exactly once, with the battle context it was cast on; an issue
311
-
already in the record MUST NOT be re-parsed, so a later title edit
312
-
changes no rating. Already-closed issues MUST NOT be re-commented on or
324
+
capped, in human-vote-equivalent units, at a fixed ceiling applied
325
+
symmetrically to both members of the pair; the weight below the cap is
326
+
scaled proportionally so it preserves the pair's observed auto-battle win
327
+
rate rather than distorting it. Dataset size MUST NOT be a lever on how
328
+
much the auto-vote seed can move a pair's rating — a benchmark corpus with
329
+
ten thousand samples must not outweigh a hand-voted pair the way an
330
+
uncapped weight would let it. Replay MUST reproduce the capped weight
331
+
deterministically from the published predictions alone, same as any other
332
+
seed value (§2 P5). The tuned ceiling is the `MAX_AUTO_WEIGHT_PER_PAIR`
333
+
constant in `arena/assembler.py`, worked through alongside the per-round
334
+
scaling in `docs/methodology.md`. `ovos-arena audit-seeds` reports
335
+
every pair's weight and whether it sits at the cap.
336
+
-**R12, Full-history replay.** The vote record (§6) is the complete history
337
+
of votes as they were publicly cast, one committed line per `vote`-labelled
338
+
issue, appended once and never rewritten. `tally` fetches every
339
+
`vote`-labelled issue (open and closed) so it can detect any one not yet
340
+
in the record, records that issue exactly once with the battle context it
341
+
was cast on, and then rebuilds every leaderboard by replaying the FULL
342
+
record from the start — not only the newly appended lines — so the rating
343
+
never depends on which run first observed a given vote. An issue already
344
+
in the record MUST NOT be re-parsed, so a later title edit changes no
345
+
rating, and an already-closed issue MUST NOT be re-commented on or
313
346
re-closed. `assemble`, `tally` and `verify-replay` MUST build every
314
-
leaderboard through the same replay of that record, so what`assemble`
315
-
publishes is what `verify-replay` reproduces.
347
+
leaderboard through that same replay of the committed record, so what
348
+
`assemble`publishes is what `verify-replay` reproduces.
316
349
-**R13, Vote fraud rules** (`arena/fraud.py`, pure functions of the vote
317
350
log, see `docs/methodology.md` for the full rationale):
318
351
- one vote per (voter, battle), R1's battle identity dedupe. - a per-voter, per-league, per-UTC-day cap (`DAILY_VOTE_CAP = 50`). - an account-age gate (`NEW_ACCOUNT_MIN_DAYS = 7`) using a creation-date
0 commit comments