State the control draw in the adjudication report - #1
Merged
Conversation
The report's header described all 80 labelled pairs as one stratified sample of 50-99 x50, 100-499 x20, 500+ x5. Those strata sum to 75, not 80, and they did not match the band table printed below them, which reads 53 / 20 / 7. Five of the 80 are controls, drawn separately from prose pairs scoring 500 or above plus a pool of pairs whose shared text is a known generic formula, then deduplicated against the stratified draw. That draw was not disclosed here. The strata line was a hardcoded constant printed as fact. It is now computed from the labelled rows, so the header and the band table cannot disagree again, and a new section states the composition: - Three controls are identifiable, a008, a054 and a074, recovered by joining the labelled pairs against the generic-formula pool. Both annotators labelled all three generic. - The other two came from the same score >= 500 prose set the top stratum drew from, so no shipped field separates them from the five stratified pairs there. - Every marginal in the report therefore mixes controls with stratified pairs. observations/README.md carries the same disclosure beside the schema. No labels, scores or counts change; adjudication-2026-08-06.jsonl is untouched and the rendered JSON is byte-identical. 264 tests pass.
Formatting only. The rendered report is unchanged and 264 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The adjudication report's header described all 80 labelled pairs as one stratified sample of
50-99 x50, 100-499 x20, 500+ x5. Those strata sum to 75, not 80, and they did not match the band table printed 20 lines below them, which reads 53 / 20 / 7. A reader could falsify the sentence without leaving the page.Five of the 80 are controls, drawn separately from prose pairs scoring 500 or above plus a pool of pairs whose shared text is a known generic formula, then deduplicated against the stratified draw. That draw was not disclosed anywhere in the published artifact.
What changed
The strata line was a hardcoded constant printed as fact. It is now computed from the labelled rows, so the header and the band table cannot disagree again, and a new section states the composition:
a008,a054,a074, recovered by joining the labelled pairs against the generic-formula pool. Both annotators labelled all three generic, which is the control returning what it was built to return.score >= 500prose set the top stratum drew from, so no shipped field separates them from the five stratified pairs there.observations/README.mdcarries the same disclosure beside the schema.What did not change
No labels, scores or counts.
observations/adjudication-2026-08-06.jsonlis untouched, the renderedadjudication-2026-08-22.jsonis byte-identical, and re-running the renderer is idempotent. 264 tests pass.