feat(review): extend debate to Stages 14/18 and the tournament to Stage 9 - #318
Open
Jacky628 wants to merge 2 commits into
Open
feat(review): extend debate to Stages 14/18 and the tournament to Stage 9#318Jacky628 wants to merge 2 commits into
Jacky628 wants to merge 2 commits into
Conversation
Follow-up to aiming-lab#316: opt-in features that aren't advertised tend to stay unused. Documents the nine settings in the example config, with the reviewer fields written out explicitly (empty = current behaviour) and the debate/tournament ones commented out. Scope is limited to what is actually wired up on main, which is narrower than the engines can do: debate and the tournament are documented for Stage 8 only, and there is no reasoning_effort setting. Also records the non-obvious parts a user cannot infer from field names — that the tournament takes priority over debate on Stage 8, that a debate panel needs >= 2 distinct models after deduplication or it silently falls back to the single-model path, and the four ARC_* env overrides. Verified by parsing the example config through RCConfig: reviewer_model empty, debate_enabled and tournament_enabled False — the documented defaults match what the loader produces. No code changes. Full suite: 2951 passed, 0 failed, 56 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ge 9 aiming-lab#316 added the engines but wired them into Stage 8 only. This connects the remaining stages where the same argument applies, on the same opt-in terms: with debate_enabled / tournament_enabled off, every one of these stages takes exactly the path it takes today. - Stage 14 (result analysis): the optimist / skeptic / methodologist roles now argue on distinct models. This is the stage that decides what the results mean, so a single model reading its own experiment is the case where agreement is least informative. - Stage 18 (peer review): three reviewer roles — methodology, domain, rigor — each bound to a different panel model, so the three reviews are not one model's opinion restated three times. Adds the review_synthesize sub-prompt, which upstream did not have. - Stage 9 (experiment design): best-of-N over three fixed stances (ambitious / robust / compute-efficient), judged and reduced to one winning plan before any YAML parsing or normalization runs. Stage 18 deliberately passes no `synthesizer`, so the independent reviewer writes the report itself. Handing the write-up back to the author model would put it back in the loop and defeat the stage — this is the same decision documented in aiming-lab#316, applied to the multi-role path. The example config is updated in the same commit rather than left to drift: the debate/tournament sections now name the real stage sets, and the "primary_model writes the final text" line is qualified, since it is not true of Stage 18. Full suite: 2951 passed, 0 failed, 56 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#316 added the debate and tournament engines but wired them into Stage 8 only. This connects the remaining stages where the same argument applies, on identical opt-in terms: with
debate_enabled/tournament_enabledoff, every stage below takes exactly the path it takes today.Stage 14 — result analysis (debate)
The optimist / skeptic / methodologist roles now argue on distinct models. This is the stage that decides what the results mean, which makes it the case where a single model agreeing with itself is least informative — an over-eager reading of a null result is exactly what a skeptic on a different model is there to catch.
Stage 18 — peer review (debate)
Three reviewer roles — methodology, domain, rigor — each bound to a different panel model, so the three reviews are not one model's opinion restated three times. Adds the
review_synthesizesub-prompt, which upstream did not have.No
synthesizeris passed here, on purpose. The independent reviewer writes the report itself; handing the write-up back to the author model would put it back in the loop and defeat the stage. This is the same decision you highlighted in #316, applied to the multi-role path.Stage 9 — experiment design (tournament)
Best-of-N over three fixed stances (ambitious / robust / compute-efficient), judged and reduced to one winning plan before any YAML parsing, normalization or capping runs — so everything downstream still sees exactly one plan.
Example config
Updated in the same commit rather than left to drift. The debate/tournament sections now name the real stage sets, and the "primary_model writes the final text" line is qualified, because it is not true of Stage 18.
Testing
Also verified that the three new call sites sit inside the opt-in branches, and that Stage 8/14 pass a
synthesizerwhile Stage 18 does not.🤖 Generated with Claude Code