Skip to content

Commit 8d7f063

Browse files
committed
Remove unimportant MCTS adapter references
1 parent 4f2589a commit 8d7f063

8 files changed

Lines changed: 10 additions & 55 deletions

File tree

docs/benchmark/CCERL_LEADERBOARD.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Status: current release-candidate snapshot for `CCERL-2P10-v2`.
44

5-
This is the current local calibration snapshot for the public Chinese Checkers engine benchmark. It is engine-relative, anchored at `random = 0`, and should not be interpreted as human Elo. Source-style open-source adapters are included directly in the all-pairs ladder, not projected from a single match. `svjayanthi-mcts-fast` is excluded from the public table because it lost every useful comparison game in screening.
5+
This is the current local calibration snapshot for the public Chinese Checkers engine benchmark. It is engine-relative, anchored at `random = 0`, and should not be interpreted as human Elo. Source-style open-source adapters are included directly in the all-pairs ladder, not projected from a single match.
66

77
## Fixed-Simulation Baseline Ladder
88

@@ -66,4 +66,4 @@ Interpretation: `JumpStar_60` remains the public benchmark champion for now. `Ju
6666

6767
## Open-Source Adapter Check
6868

69-
Source-style adapters for representative public Chinese Checkers bots are included in the main all-pairs ladder above. The retained adapters are `marblefish-ab`, `harryz-rule`, and `zedrichu-minimax`; `svjayanthi-mcts-fast` was removed from the official ladder because it produced an uninformative all-loss profile.
69+
Source-style adapters for representative public Chinese Checkers bots are included in the main all-pairs ladder above. The retained adapters are `marblefish-ab`, `harryz-rule`, and `zedrichu-minimax`.

docs/benchmark/CHANGELOG_2P10_V1_TO_V2.md

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ during the open-source bot screening:
2727
| `harryz-rule` (deterministic 1-ply greedy) | 172 | 0-101-71 | 101 / 172 (**59%**) | 1470.6 | -441.7 |
2828
| `marblefish-ab` | 172 | 4-11-157 | 0 / 172 | 1129.8 | 985.6 |
2929
| `zedrichu-minimax` | 172 | 5-7-160 | 0 / 172 | 1109.5 | 1021.0 |
30-
| `svjayanthi-mcts-fast` | 172 | 0-5-167 | 0 / 172 | 890.6 | -585.1 |
3130

3231
`harryz-rule` never beat the champion, yet earned an Elo only ~150 points
3332
below it because 59% of its games hit the ply cap with a 0.5-point draw. The
@@ -106,8 +105,6 @@ exceptions, both inflated by the v1 loophole:
106105
- `harryz-rule`: expected to drop from ~1471 toward the
107106
decisive-only estimate of ~-441 (still above strict random because some
108107
of its draws against weaker engines remain legitimate halves).
109-
- `svjayanthi-mcts-fast`: smaller correction, since most of its outcomes
110-
were already decisive losses. Expected drop from ~891.
111108

112109
Numbers will be updated on the public benchmark page after the v2 re-run
113110
completes.

docs/benchmark/OPEN_SOURCE_BOT_SCREENING.md

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ open-source projects. The most useful candidates for a first comparison were:
99

1010
| Project | Public description | Fit for CCERL |
1111
|---|---|---|
12-
| `SVJayanthi/ChineseCheckersMCTS` | Java Chinese Checkers player using Monte Carlo Tree Search. | Good algorithmic reference; GUI/original-code integration is brittle, so v1 uses a CCERL MCTS-style adapter. |
1312
| `arthur-x/MarbleFish` | Browser-playable Chinese Checkers AI using alpha-beta search and beam-style pruning. | Good algorithmic reference; v1 ports the published alpha-beta scoring/search style to CCERL. |
1413
| `Zedrichu/Chinese-Checkers-AI` | Python project with random, non-repeating-random, and minimax players. | Useful minimax/heuristic reference, but upstream uses `triangle_size=3`, so direct binary comparison would not be the same game. |
1514
| `HarryZalessky/Chinese-Checkers` | Java graph-based 121-cell, 10-piece game with a rule-based "perfect game" AI. | Good rules/heuristic reference; v1 ports the rule-path style to CCERL. |
@@ -19,7 +18,6 @@ open-source projects. The most useful candidates for a first comparison were:
1918
Sources:
2019

2120
- GitHub topic page: https://github.com/topics/chinese-checkers
22-
- SVJayanthi MCTS project: https://github.com/SVJayanthi/ChineseCheckersMCTS
2321
- MarbleFish project: https://github.com/arthur-x/MarbleFish
2422
- Zedrichu minimax project: https://github.com/Zedrichu/Chinese-Checkers-AI
2523
- HarryZalessky Java project: https://github.com/HarryZalessky/Chinese-Checkers
@@ -36,7 +34,6 @@ adapters:
3634
| `harryz-rule` | HarryZalessky rule/path AI | deterministic forward-progress, jump, and goal-entry heuristic |
3735
| `zedrichu-minimax` | Zedrichu minimax heuristic players | depth-limited alpha-beta over goal occupancy and distance terms |
3836
| `marblefish-ab` | MarbleFish alpha-beta worker | beam-pruned alpha-beta with vertical-progress and centrality scoring |
39-
| `svjayanthi-mcts-fast` | SVJayanthi MCTS project | small UCT/rollout adapter over CCERL legal moves |
4037

4138
These are not vendored upstream engines. They are source-style CCERL ports:
4239
the trusted CCERL referee owns legality, terminal detection, and the official
@@ -57,7 +54,7 @@ Protocol:
5754
- Ruleset: `CCERL-2P10-v2`
5855
- Position suite: all 86 rows of `official_elo_v2`
5956
- Pairing: color-swapped games from every row
60-
- Games: 172 per adapter, 688 total
57+
- Games: 172 per adapter, 516 total
6158
- Anchor model: `iter063`, 64 simulations per move
6259
- Runner: `tools/ccp_referee.py`
6360
- Direct Elo: from the adapter's perspective relative to JumpStar, draws scored as 0.5
@@ -70,14 +67,12 @@ Evaluation profile:
7067
| `harryz-rule` | deterministic rule heuristic |
7168
| `marblefish-ab` | depth 2, beam 4 |
7269
| `zedrichu-minimax` | depth 2, beam 4 |
73-
| `svjayanthi-mcts-fast` | 6 iterations, rollout depth 6 |
7470

7571
| Opponent adapter | Games | Adapter W-D-L | Adapter score | Score CI95 | Elo vs JumpStar | Terminations |
7672
|---|---:|---:|---:|---:|---:|---|
7773
| `harryz-rule` | 172 | 0-97-75 | 0.282 | 0.245 - 0.319 | -161 | all_pieces_in_goal: 7, anti_block_goal_full: 68, max_ply: 97 |
7874
| `marblefish-ab` | 172 | 8-41-123 | 0.166 | 0.124 - 0.208 | -278 | all_pieces_in_goal: 107, anti_block_goal_full: 24, equal_turn_goal_draw: 4, max_ply: 37 |
7975
| `zedrichu-minimax` | 172 | 3-6-163 | 0.035 | 0.011 - 0.059 | -563 | all_pieces_in_goal: 157, anti_block_goal_full: 9, max_ply: 6 |
80-
| `svjayanthi-mcts-fast` | 172 | 0-0-172 | 0.000 | 0.000 - 0.000 | -1015 | anti_block_goal_full: 172 |
8176

8277
The full-suite run preserves the main screening conclusion: even from the adapters' own perspective, JumpStar
8378
reduced to 64 simulations per move is far ahead of the first source-style
@@ -100,7 +95,6 @@ Protocol:
10095

10196
| Opponent adapter | Games | Adapter W-D-L | Adapter score | Score CI95 approx | Elo vs Iter60 raw | Elo vs Iter60 adjusted | Terminations |
10297
|---|---:|---:|---:|---:|---:|---:|---|
103-
| `svjayanthi-mcts-fast` | 16 | 0-0-16 | 0.000 | 0.000 - 0.194 | -inf | -607 | anti_block_goal_full:16 |
10498
| `marblefish-ab` | 16 | 1-2-13 | 0.125 | 0.035 - 0.360 | -338 | -305 | all_pieces_in_goal:10, anti_block_goal_full:4, equal_turn_goal_draw:2 |
10599
| `zedrichu-minimax` | 16 | 2-0-14 | 0.125 | 0.035 - 0.360 | -338 | -305 | all_pieces_in_goal:14, anti_block_goal_full:2 |
106100
| `harryz-rule` | 16 | 0-10-6 | 0.312 | 0.142 - 0.556 | -137 | -128 | all_pieces_in_goal:1, anti_block_goal_full:5, max_ply:10 |
@@ -120,7 +114,7 @@ The current evidence should remain bounded:
120114
- This is a source-style adapter evaluation, not the official public ladder.
121115
- The adapters are CCERL-rule ports, not exact upstream binaries.
122116
- Stronger exact integrations should target `alexicanesse/ChineseCheckers`,
123-
`SVJayanthi/ChineseCheckersMCTS`, and `MarbleFish` first.
117+
`MarbleFish`, or other actively maintained public engines first.
124118
- Any exact upstream binary benchmark must document rule differences, build
125119
steps, commit hashes, and coordinate conversion tests before its result is
126120
mixed into CCERL.

docs/benchmark/PUBLIC_BENCHMARK.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ The current native/CCP all-pairs ladder is:
7070
experiments/public_benchmark/ladder_v2_latest_champions_s64/baseline_ladder.json
7171
```
7272

73-
It uses all 86 audited v2 positions, paired color swaps, all 171 displayed-engine pairs, 29,412 games total, 64-simulation neural search, the existing native/CCP logs plus latest-champion incremental runs, and `random = 0`. Open-source adapter rows are fitted in the same all-pairs graph as every other displayed engine; `svjayanthi-mcts-fast` is excluded because it produced no useful comparison signal.
73+
It uses all 86 audited v2 positions, paired color swaps, all 171 displayed-engine pairs, 29,412 games total, 64-simulation neural search, the existing native/CCP logs plus latest-champion incremental runs, and `random = 0`. Open-source adapter rows are fitted in the same all-pairs graph as every other displayed engine.
7474

7575
| Rank | Baseline | Elo | Games | Score | vs JumpStar_60 | vs JumpStar_82 |
7676
|---:|---|---:|---:|---:|---:|---:|

experiments/public_benchmark/ladder_v2_latest_champions_s64/baseline_ladder.json

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -413,9 +413,7 @@
413413
}
414414
]
415415
},
416-
"excluded_labels": [
417-
"svjayanthi-mcts-fast"
418-
],
416+
"excluded_labels": [],
419417
"pairs": [
420418
{
421419
"a": "greedy",

experiments/public_benchmark/open_source_v2/elo_half.json

Lines changed: 3 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"anchor": "iter063_s64",
33
"anchor_elo": 0.0,
44
"draw_policy": "half",
5-
"games": 688,
5+
"games": 516,
66
"log_likelihood": -205.56415383438255,
7-
"players": 5,
7+
"players": 4,
88
"ratings": [
99
{
1010
"ci95": [
@@ -13,7 +13,7 @@
1313
],
1414
"draws": 144,
1515
"elo": 0.0,
16-
"games": 688,
16+
"games": 516,
1717
"losses": 11,
1818
"player": "iter063_s64",
1919
"score_rate": 0.8793604651162791,
@@ -61,20 +61,6 @@
6161
"score_rate": 0.03488372093023256,
6262
"standard_error": 72.00292606070057,
6363
"wins": 3
64-
},
65-
{
66-
"ci95": [
67-
-4156.888254968699,
68-
-269.2489590306452
69-
],
70-
"draws": 0,
71-
"elo": -2213.068606999672,
72-
"games": 172,
73-
"losses": 172,
74-
"player": "svjayanthi-mcts-fast",
75-
"score_rate": 0.0,
76-
"standard_error": 991.7447183515442,
77-
"wins": 0
7864
}
7965
]
8066
}

experiments/public_benchmark/open_source_v2/results.json

Lines changed: 0 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -62,24 +62,6 @@
6262
"anti_block_goal_full": 9,
6363
"max_ply": 6
6464
}
65-
},
66-
{
67-
"anchor_losses": 0,
68-
"anchor_score": 1.0,
69-
"anchor_wins": 172,
70-
"direct_elo_delta": null,
71-
"direct_elo_delta_adjusted": 1015.127638029308,
72-
"draws": 0,
73-
"games": 172,
74-
"log": "experiments/public_benchmark/open_source_v2/svjayanthi-mcts-fast_vs_iter063_s64.jsonl",
75-
"opponent": "svjayanthi-mcts-fast",
76-
"score_ci95_normal": [
77-
1.0,
78-
1.0
79-
],
80-
"terminations": {
81-
"anti_block_goal_full": 172
82-
}
8365
}
8466
],
8567
"elo_half": "experiments/public_benchmark/open_source_v2/elo_half.json",

tools/ccp_open_source_style_bot.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -471,16 +471,14 @@ def choose_move(args: argparse.Namespace, cells: str, player: int, rng: random.R
471471
return choose_alphabeta(cells, player, args.depth, marblefish_score, beam=args.beam)
472472
if args.style == "harryz-rule":
473473
return choose_rule_path(cells, player)
474-
if args.style in {"svjayanthi-mcts", "svjayanthi-mcts-fast"}:
475-
return choose_mcts(cells, player, rng, args.iterations, args.rollout_depth)
476474
raise RuntimeError(f"unknown style: {args.style}")
477475

478476

479477
def main() -> int:
480478
parser = argparse.ArgumentParser(description=__doc__)
481479
parser.add_argument(
482480
"--style",
483-
choices=("zedrichu-minimax", "marblefish-ab", "harryz-rule", "svjayanthi-mcts", "svjayanthi-mcts-fast"),
481+
choices=("zedrichu-minimax", "marblefish-ab", "harryz-rule"),
484482
required=True,
485483
)
486484
parser.add_argument("--name")

0 commit comments

Comments
 (0)