A running record of how this engine was built: what was added, what broke, what was measured, and what got thrown away. Written as I went rather than reconstructed at the end.
Status markers used below: [shipped], [reverted] for anything measured and then removed, [in progress].
PyCheckmate is a chess engine written from scratch in Python, with three faces: a Pygame desktop game, a post-game review screen in the style of chess.com, and a UCI adapter that runs the same engine as a bot on Lichess.
The stack:
| Layer | Choice | Why |
|---|---|---|
| Language | Python 3.14 | A learning project, so clarity beat raw speed |
| Engine core | Pure stdlib, zero dependencies | This is what lets the engine run under PyPy |
| Runtime accel | PyPy 3.11 subprocess | The JIT gives a large speedup on the hot search loop. Auto-detected, never required |
| GUI | pygame-ce | Desktop game plus review screen |
| Tooling | uv, pytest, mypy (strict), ruff | A reproducible environment. 159 tests, a clean type check and a clean lint gate every change |
| Deployment | lichess-bot bridge, macOS | The engine speaks UCI, so a standard bridge can host it |
| Measurement | Stockfish as referee, fastchess for SPRT | An engine cannot grade itself, because it misjudges a position identically when playing and when grading |
The goal was to learn search and evaluation by building them instead of reading about them, then prove the result was real by making it play rated games against strangers. A second goal turned out to matter just as much: learning to measure engine changes honestly, because most of the interesting failures here were measurement failures rather than coding failures.
Scale is about 14,600 lines of Python, roughly 8,500 of it in engine/.
search.py at 1,341 lines and board.py at 1,311 are the core, and there are
about 2,500 lines of tests. Roughly a third of engine/ is measurement tooling
rather than chess. (search.py and board.py were called move_finder.py and
chess_engine.py until the package split on 2026-07-20, so earlier entries below
use the old names.)
Where it ended up: about 2300 Elo calibrated against Stockfish, and 2187 in Lichess rapid across 130 rated games, with every game graded automatically by an independent engine. Rapid is the number to quote. Blitz sits 137 points lower, and averaging the two pools describes neither of them. Key Takeaway 9 explains why.
The board is a list[list[int]] of small integer piece codes, where 0 is
empty, 1-6 is White and 7-12 is Black. It began as 'wP' and '--' strings,
and migrating to ints took string comparison out of the hot loops. The string
codes survive only at the FEN, SAN/UCI and GUI-image boundaries.
The central design decision is that there are two parallel move pipelines:
- The UI path.
get_valid_moves()returns richMoveobjects, andmake_move()maintains the move log, the state log, repetition counts, and a full Zobrist recompute. - The AI path.
get_valid_moves(for_ai=True)returns bare 5-tuples, andmake_ai_move()skips all logging, updates Zobrist incrementally, and returns a 4-tuple undo package.
The hot loop cannot afford the bookkeeping the UI needs, and the UI cannot work
without it. What makes both of them fast and correct is keeping them separate and
converting between them at exactly one seam, Move.from_ai_tuple.
Built up piece by piece, each one measured before it was kept:
| Technique | Purpose |
|---|---|
| Negamax + alpha-beta | The base search |
| Iterative deepening | Anytime search, so there is a usable move whenever the clock stops |
| Transposition table (Zobrist-keyed) | Reuse work across transpositions; depth-preferred replacement with aging |
| Quiescence search | Fixes the horizon effect by never evaluating mid-capture |
| MVV-LVA + killers + history | Move ordering; alpha-beta's benefit depends almost entirely on trying good moves first |
| Static exchange evaluation (SEE) | Prune losing captures in quiescence instead of searching them |
| Null-move pruning | Skip a turn; if still winning, the position is not worth full depth |
| PVS (null-window scouts) | Assume the first move is best, verify siblings cheaply |
| Late move reductions | Search unpromising late moves shallower, re-search if one beats alpha |
| Aspiration windows | Start each iteration in a narrow window around the last score |
| Check extension | Never stop the search inside a forced sequence |
| Futility / reverse-futility pruning | Skip nodes that cannot reach alpha |
Material plus piece-square tables, tapered between middlegame and endgame by material phase. A king belongs in the corner at move 20 and in the centre at move 60, and one static table cannot say both. On top of that sit mobility, rook activity and pawn structure terms. Everything is computed from White's perspective and memoized by Zobrist key.
A two-bound design, soft and hard. time_limit is the target and hard_limit
funds a panic extension when the score collapses. The budget divides the
remaining clock by an estimate of how many moves are still to play, with two
emergency tiers below 25s and 10s. The 2026-07-19 entries cover this, and it is
the part of the project with the most interesting measurement story.
analysis.py does chess.com-style move grading, the blunder to mistake to
brilliant ladder, scored by win-percentage loss. pgn.py imports SAN by matching
against legal moves rather than re-implementing the rules. uci.py and
uci_client.py are the engine-as-a-process pair. book.py is the opening book,
which is a single Zobrist-keyed dict probe.
Under engine/tools/ sit bench.py, sprt.py, calibrate.py, selfplay.py,
sf_review.py, sf_watch.py, tune.py and build_book.py. That is the
measurement toolkit, and it grew this large for good reason. An earlier
generation of it, abtest.py, tm_replay.py and tm_allocate.py, was retired
on 2026-07-20 once sprt.py and sf_watch.py superseded it. The entries below
were written while it was still live.
- 07-01. First commit. Board state, piece rendering,
GameState. - 07-04 to 07-06. Move generation, then legality: check detection, pin handling, checkmate and stalemate.
- 07-07. Castling and pawn promotion.
- 07-08. En passant, including the hidden horizontal pin. Capturing en passant removes two pawns from a rank at once, which can expose a king sideways.
- 07-10. The 50-move rule, threefold repetition, and the GUI extracted into
gui/.
- 07-16. Zobrist hashing and the first real search, in
move_finder.py. Alpha-beta, move ordering, iterative deepening. - 07-16. UCI adapter written, so the engine became hostable as a bot.
- 07-17. [shipped] Perft bug found and fixed, covered in Challenges below. Perft suite added.
- 07-17. PyPy UCI subprocess hosting, and the engine confirmed dependency-free.
- 07-17. Game review screen: evaluation bar, move grading, variations, FEN and PGN import.
Ten measured stages, A through J, each committed separately so any one of them could be reverted alone.
- [shipped] Quiescence, a persistent transposition table, extra evaluation terms.
- [shipped] Integer piece codes. Depth-5 search went from 1.35s to 0.28s.
- [shipped] Evaluation cache keyed by Zobrist. Depth-6 search 35% faster under PyPy, 8% under CPython.
- [shipped] PVS, quiescence transposition table with delta pruning, log-scaled LMR, tapered evaluation, mobility.
- [reverted] Late move pruning, at 60 Elo worse, and still 24 worse after fixing a broken test.
- [reverted] Countermove history plus internal iterative reductions. Never measured on its own, and unproven code goes.
- 07-18. Deployed to Lichess. Self-play smoke test added.
- [shipped] Node-count methodology adopted. Score-neutral changes are now verified by exact node equality instead of by overnight matches.
- [shipped] Ray tables precomputed, for 10.4% faster overall and
_mobility43% faster. Proven safe by identical node counts, and verified in about 30 seconds. - [shipped]
tm_replay.py, which grades the bot's 16 real online games to find where it actually loses. - [shipped] Time budget reshaped around moves-to-go plus an overhead reserve. Middlegame thinking time up 46%.
- [reverted] Best-move stability, which measured worse at 1.11x against 1.34x.
- [reverted] The 400-game clock-mode gate, abandoned twice as unusable. The absolute hoarding thresholds relocate when the test clock is scaled, so a 90+0 match measured a partially self-cancelling version of the change and read 45 Elo worse. Recorded as void rather than as evidence.
- [shipped] The 1/43 record explained. All 47 Lichess games had been casual, so the rating never left the provisional 3000 and matchmaking kept pairing the bot with opponents around 2930. Not an engine result at all.
- [shipped] First real strength number, about 2133 Elo, from three Stockfish
UCI_Elolevels agreeing within 47 points viaengine/tools/calibrate.py. - [shipped] Bitboard rewrite considered and rejected on measurement. python-chess and its bitboards benchmark slower than our mailbox engine in Python.
- [shipped] UCI
infooutput added. The engine had been unobservable in production, and this immediately exposed a depth cap bug. - [shipped] Clock utilisation from 58% to 101%, because aborted iterations are no longer discarded.
- [shipped] Hoarding thresholds made relative to the starting clock, which retired the trap that voided two overnight gates.
- [shipped] SPRT harness,
engine/tools/sprt.pyover fastchess, replacing fixed-size matches. - [shipped] Automated per-game Stockfish analysis in
engine/tools/sf_watch.py, running in the bot's idle time and stamped with the engine build. - [shipped] Redeployed rated, with opponent bounds set around the calibrated rating.
Move generation looked correct and passed every hand-written test I had.
Perft is what found it. Count leaf nodes at depth N, compare against published values. Depth 5 produced 4,865,607 nodes where the canonical answer is 4,865,609. Two nodes wrong out of 4.8 million, which is a bug no amount of playing would ever surface and no amount of eyeballing would ever find.
The cause was in _is_square_attacked, which let sliding attack rays pass
through an adjacent enemy pawn whenever that pawn did not itself attack the
probed square. A queen could therefore "attack" a square straight through her own
blocking pawn, illegally restricting where the enemy king could go.
The fix was one blocking condition, plus perft(4), Kiwipete perft(1) through (3), and a direct regression test. The lesson I took from it: a test that produces one exact number beats any number of tests that produce "looks right".
The search was too slow, and everyone knows attack detection dominates a chess engine.
cProfile at depth 5 disagreed. The hotspots were evaluate() and move
generation, not _is_square_attacked, which is where I would have spent the day.
A later depth-6 profile put evaluate at 31% of runtime and _mobility alone at
14% across 121,789 calls.
So I optimized what the profiler pointed at instead. The _mobility cost turned
out to be its bookkeeping rather than its logic: walking a ray re-tested
0 <= r < 8 and 0 <= c < 8 at every single step. Board geometry never changes,
so every ray can be enumerated once at import. That came out 43% faster in
isolation and 10.4% faster overall.
Search stages F through J ran overnight self-play matches and returned roughly 0 Elo, most of it unresolvable noise. A whole program of machine time bought no knowledge at all.
The instrument was wrong for the question. A 100-game match resolves nothing finer than 70 Elo either way, and 400 games gets that to 35. Real search improvements are worth 10 to 30 Elo. A 100-game verdict on a 20-Elo change is noise wearing a number.
The fix is the most valuable thing I learned in this project, which is to match the instrument to the change.
For score-neutral changes like a faster evaluation or cheaper move generation, use node count. It is exactly deterministic, so identical node totals prove the search made every same decision and the change cannot have altered play. That turns an overnight match into a 30-second check, and it is stronger evidence besides, because it is a proof rather than a statistic. For behaviour changes, self-play with 400 games minimum, and never two matches running at once.
There is a catch. Node determinism only holds because the bench seeds the root
RNG. The root shuffle changes how much the search prunes, so unseeded counts do
not reproduce and the method silently stops working.
test_node_count_is_reproducible_for_a_seeded_search guards that now.
Self-play used a depth cap of 6 as a safety net, on the assumption that the clock would always stop the search first.
I measured it instead of assuming. At the 0.2s budget, depth 6 completed naturally in 4 of 7 realistic positions. The cap was binding, not the clock.
That matters more than it sounds. A faster engine in a cap-bound position has nowhere to spend the speed, so every speed optimization would measure as 0 Elo no matter how much it actually helped. The measuring instrument had a hard ceiling and was quietly reporting that ceiling as a result.
Raising DEPTH to 12 fixed it, since that is above anything the budget can
reach, which leaves the clock as the only binding constraint.
The bot kept blundering in positions where it had time to spare.
I built tm_replay.py to replay all 16 real online games, parse the %clk
annotations for time actually spent, and grade every move. Two things came out of
it. The engine finished sixty-move games with 28 to 48% of its clock unspent. And
73% of its blunders, mistakes and missed wins fell in moves 21 to 40, at error
rates of 8% and 7%, against 1% in moves 1 to 20 where the opening book answers
anyway.
The cause was that clock_move_budget divided the remaining clock by a constant
30. Dividing by a constant decays geometrically, so the clock is never actually
spent, it just asymptotes toward zero. Time was being hoarded forever and left on
the table at the end of the game.
Dividing by an estimate of the moves remaining fixed it. The middlegame budget moved like this:
| Time control | Moves 21 to 40, before | After | Change |
|---|---|---|---|
| 5+0 | 3.7s | 5.4s | up 46% |
| 10+2 | 9.3s | 12.9s | up 39% |
The opening got damped in exchange, with move 1 of a 5+0 game going from 10.0s to 5.7s, and unspent clock after a 45-move game fell from about 25% to 18%.
One near miss is worth recording. The first draft engaged its safety guard below 60s, which in a 5+0 game means hoarding from around move 35, precisely inside the band where the errors happen. A simulation caught it flagging at move 77. The version I shipped uses 25s and 10s tiers and survives a marathon identically to the old rule, with both reaching about move 124.
This was meant to be the headline feature: know when to think longer. The standard idea is that if the root best move keeps changing between iterations, the position is sharp, so extend.
The first attempt measured a uniform 1.01x, because the mechanism was not firing at all. I had counted every best-move change, but depths 1 through 3 always flip, so the real signal drowned in them. A decaying accumulator fixed that.
The second attempt fired, and fired backwards, at 0.64x. Tracing it iteration by iteration showed the fault was in my test labels rather than in the code. I had hand-picked Kiwipete as the hard position and a rook endgame as the easy one. In fact the engine picks the same move at every depth in Kiwipete, which makes it calm, and flaps all search long in the endgame, which makes it unstable. My labels were exactly inverted.
So I rebuilt the instrument as tm_allocate.py, on real blunders from real
games, those being the only labels my guesses could not contaminate. Against
those:
| Rule | Critical to routine time ratio |
|---|---|
| The existing panic rule, a binary score drop | 1.34x |
| Plus continuous response and the stability signal | 1.11x |
| Plus continuous response only | 1.36x, a wash |
The planned feature was worse than the crude rule it was supposed to replace. I reverted it and committed only the negative result, as a comment where the next person would look.
Stability backfires here for a specific reason. In quiet endgames the root best move flaps between moves of identical score, so it reads as maximally unstable exactly where there is least to think about.
The part that matters most is what a self-play match would have done with this. It would have called the whole change neutral after a full night, with no way to tell "did not work" apart from "never fired". Both failure modes were caught in minutes by an instrument that watched the mechanism directly.
The bot appeared online and accepted challenges without ever playing them, and
the error log was full of RateLimitedError.
Lichess rate-limits /api/stream/event. Restarting the bot repeatedly while
iterating on config tripped a 429 the bridge could not recover from, because it
retried faster than the roughly 60s cooldown and so could never escape. Orphaned
multiprocessing children from previous runs made it worse by holding streams
open where a normal process check could not see them.
The fix is to sweep the orphans explicitly with
ps aux | grep "lichess-bot/.venv", then start once and wait in silence. The
operational rule I adopted afterwards: batch all config edits, then do exactly
one restart cycle.
The bot went 1/43 against Lichess opponents averaging 2928. Read at face value that is a damning number, and it triggered a full "the engine is over-engineered, rebuild it from scratch" review, including a proposal to migrate to bitboards.
The API told a different story:
"count": { "all": 47, "rated": 0, "win": 5, "loss": 42 }
"perfs": { "blitz": { "games": 0, "rating": 3000, "rd": 500, "prov": true } }
Look at rated: 0. Every game had been played casual, so the rating never moved
off Lichess's provisional 3000, and matchmaking used that 3000 to choose
opponents. It is a closed loop with no exit. Casual games mean the rating never
updates, so it stays at 3000, so the bot gets paired with 2900-rated bots, so it
loses everything, and there is still no rating feedback.
The setting was mine, chosen as a safety measure when I first deployed. I never asked what it did to the measurement.
Two independent instruments then showed the engine was not the problem. Stockfish
at depth 14 over all 43 games put our blunder rate at 0.4%, or 5 in 1289 moves.
That is not an engine that hangs pieces. It is an engine consistently slightly
worse than a much stronger opponent. A calibration ladder against Stockfish under
UCI_LimitStrength returned:
| Stockfish level | Our score | Implied rating |
|---|---|---|
| 1600 | 19.0/20 | ~2112 |
| 2000 | 13.5/20 | ~2127 |
| 2400 | 4.0/20 | ~2159 |
Three levels agreeing within 47 Elo. Against opposition rated 2928, a 2133 engine is expected to score 0.44 points in 43 games. It scored 1.0, which is better than its rating predicted.
What that cost me was the entire time-management program, which had been tuned against those games. It was not wrong work, but its evidence base could not support any conclusion, because an 1800 engine and a 2400 engine both score around 0% at that gap. The sample was structurally incapable of telling them apart.
The general lesson is the one worth keeping. I had built three increasingly
sophisticated analysis tools on top of a dataset without ever checking that the
dataset could express the thing being measured, and no amount of sophistication
downstream recovers a signal that was never sampled. The fix was one config line,
challenge_mode: rated. Finding it meant reading the raw API response instead of
the scoreboard.
The natural next move after "the engine is too complex for its performance" was to rewrite the board representation as bitboards, which is the standard answer in chess programming and the one every reference recommends.
Measured first, same perft, make/unmake at every ply:
| Engine | Representation | Host | Nodes/sec |
|---|---|---|---|
| python-chess | bitboard | CPython | 335,510 |
| PyCheckmate | mailbox + piece-sets | CPython | 392,994 |
| PyCheckmate | mailbox + piece-sets | PyPy | 958,410 |
Our mailbox engine is already 17% faster than a mature bitboard library on
CPython. The reason is that the bitboard advantage is a C advantage.
attacks & ~own is one machine instruction across 64 squares, but in Python
every & allocates a heap integer, and Python has no uint64, so bit 63
silently promotes into arbitrary-precision arithmetic. Mailbox with piece-sets,
meanwhile, iterates only the 16 or so pieces that actually exist.
Profiling closed the argument. Move generation is 37% of search time, so by Amdahl's law even an infinitely fast generator caps out at a 1.6x speedup, which is roughly half a ply. That would have been weeks of work on the most correctness-critical code in the project, in a direction the benchmark said was slower.
The instinct behind the request was right and the mechanism was not. The real levers for depth are searching fewer nodes, which means ordering, and incremental evaluation. Not visiting nodes faster.
The UCI adapter emitted no info lines at all. No depth, no score, no nodes.
Lichess showed nothing, and there was no record anywhere of what depth a real
game reached. Every diagnostic in this document had to be reconstructed offline
from finished PGNs, which cannot see what the engine was actually thinking at the
time.
Adding them took an hour, and the first run exposed a bug they would have caught
weeks earlier. go movetime 4000 returned after 452ms, having stopped at depth
5. The clock path set an unlimited depth, while the movetime path fell through
to DEFAULT_DEPTH = 5. Lichess itself was unaffected, since the bridge sends
wtime and btime, but every movetime-driven test I had run was measuring a
depth-capped engine.
Fixing it changed the move played on the test position.
Observability is not a nice-to-have on a system you are trying to measure. The whole project had been running instruments against a black box.
Once the info lines existed, the search turned out to be spending 58% of a 3s
budget and 69% of a 5s one.
Two rules combined to cause it. _search_root discarded a timed-out iteration
wholesale, and because starting an iteration you could not finish was therefore
pure waste, the soft-stop gate refused to begin one past 45% of the budget. The
second rule was a rational response to the first.
The first one is the real defect. Root moves are ordered best-first from the previous iteration, so a partial pass has already examined the most promising candidates. It either confirms the previous best or replaces it with a move that outscored it a ply deeper, and both of those are strictly better information than stopping early. Keeping the partial result let the gate move out to 0.9:
| Budget | Before | After |
|---|---|---|
| 3.0s | 1.73s (58%) | 3.03s (101%) |
| 5.0s | 3.44s (69%) | 5.01s (100%) |
That exposed a second-order problem underneath it. The clock was sampled every 2048 nodes, or about 58ms, and that error is near-constant in time, so it only bites when the budget is small. A 0.1s budget overran by 41%, which is exactly the situation a nearly-flagged clock produces. Sampling every 512 nodes quartered the overrun and cost 0.6% on the benchmark, with the node count unchanged, and that unchanged count is the proof the search itself was untouched.
Best-move stability, meaning narrowing the time gate when successive search
iterations keep returning the same root move, was built, measured and deleted
months earlier. It scored 1.11x against the crude panic rule's 1.34x on the
question of whether the engine thinks longer on positions it actually blundered.
A comment was left in front of PANIC_SCORE_DROP recording both the verdict and
the reason:
in quiet endgames the root best move flaps between moves of identical score, so it reads as maximally unstable exactly where there is least to think about
When the Phase 1 plan called for building it again, that comment was a reason to
check rather than a reason to skip. The check said the premise was gone. Ties can
no longer displace the incumbent, because _search_root compares with a strict
score > best_score and searches the previous iteration's best move first, so an
equal-scoring rival gets examined second and loses. Measured over 12 positions and
84 iteration transitions:
| positions | best-move change rate | equal-score changes |
|---|---|---|
| quiet | 2% | 0 |
| sharp | 24% | 0 |
The flapping the note describes is now literally zero, and the signal discriminates 12x in the direction it was supposed to. Nobody set out to fix this. It fell out of an unrelated change that reordered root moves best-first so aborted iterations could be kept.
Rebuilt on that basis, the gate now slides from 0.9 down to 0.5 of the budget, one step per consecutive stable iteration, with panic overriding it outright. Share of a 3s budget actually spent:
| off | on | |
|---|---|---|
| quiet | 100.3% | 71.9% |
| sharp | 100.4% | 95.5% |
It takes clock away from positions with nothing left to find and leaves sharp ones alone, which is the behavior the first attempt was aiming at and got backwards.
The lesson here is about the shape of the record rather than about chess. A measured negative result is only valid while the code it measured still exists, and nothing announces when that stops being true. What made this expiry detectable is that the note recorded the mechanism instead of the verdict. Had it said "tried best-move stability, measured worse, don't rebuild", skipping would have been correct on the evidence available and wrong in fact, and the error would have been invisible, because a thing not built produces no symptom.
One caveat I kept deliberately: this measures clock reallocation, not Elo. The original experiment's 1.36x against 1.34x is a fair warning that a signal working does not mean the change is worth anything. Returning time to the endgame only pays if the endgame spends it well, and only the rated games can say.
docs/ENGINE_V2_PLAN.md §5.3 listed staged generation: yield the TT move, then
captures, then killers, then quiets, so a cutoff on the TT move never pays for
generating quiets at all. The plan called it a node-count win and a time win.
Instrumenting the search first turned that slogan into numbers, over 13,534
generating nodes:
| share of generating nodes | |
|---|---|
| any beta cutoff | 71.0% |
| cutoff on the first move tried | 62.8% |
| cutoff on the TT move specifically | 18.2% |
| cutoff before any quiet was needed | 46.0% |
| quiet share of generated moves | 93.3% |
The TT-move framing undersells it by 2.5x, because only 25% of nodes have a TT move at all, and most first-move cutoffs come from MVV-LVA and killers rather than from the table.
Two measurements then redirected the work and a third killed it.
Staged generation is worth little here. A noisy-only pass costs 78 to 80% of a
full pass in typical positions, because the generator walks every piece and every
ray either way, and captures_only merely declines to append. Skipping quiets
therefore saves about 20% of generation, which works out to
0.46 x 0.20 x 0.37, or roughly 3% of search time, before paying for a second
stage on the other 54% of nodes.
Ordering looked like the better target. Scoring and sorting costs 35 to 41% of combined generate-and-order time, and unlike generation it is entirely skippable for moves that never get searched. I estimated about 9%. So the build became staged ordering: partition by a board lookup per move, score and sort only what outranks a killer, and score the remaining 93% lazily.
It measured neutral. Best-of-5 back to back, CPython at 3.29 to 3.33s against a 3.27 to 3.59s baseline, PyPy at 2.59 to 2.68s against 2.60 to 2.88s. Overlapping ranges on both interpreters, so by this project's own rule there is nothing here to believe. Reverted.
Why the 9% never appeared is the part worth keeping, because it tells the next attempt what to fix. The saving lands on 46% of nodes while the machinery is paid on 100% of them: a generator's per-yield cost, a partition pass over every move, and a scorer promoted from a closure to a six-argument module function so both paths could share one definition. Python call overhead on every node cancelled out the scoring skipped on some of them. A version that keeps the closure and avoids the generator might still win. This one put its cost in exactly the place its benefit was supposed to come from.
One sub-result is solid and worth reusing. Staged yielding with eager scoring
reproduces the old order exactly, 59,410 nodes against 59,410, which proves the
band partition is sound, with the first stage at 900,000 or above and losing
captures held back below killers at 300_000 + SEE. The node count only moved
once scoring went lazy, down to 59,092, because stage-2 quiets get scored against
a history table that stage-1 subtrees have already updated. That is fresher
information and 0.5% fewer nodes, and it is a reminder that lazy scoring is not
automatically behavior-neutral even when lazy ordering is.
Mistake M7 is that ~34 evaluation constants were hand-picked and never fitted,
and §7.5 put Texel tuning at 50 to 100 Elo, making it the highest-value strength
item never attempted. engine/tools/tune.py now implements it: replay the bot's
PGNs into positions labeled by game result, fit the sigmoid constant K, then run
coordinate descent on the weights against mean squared error.
Two safeguards went in from the start rather than after a bad result, and they are the reason this entry exists. Split by game and not by position, because positions within one game share pawn structure, material and result, so a position-level split leaks the answer across it. And keep only what improves error on games the fit never saw, because training error always falls. That is what fitting means.
Run to convergence, it overfits outright. Training error fell 33%, from 0.0871 to 0.0587, while held-out error bottomed at round 3 and then climbed to 0.1401, which is worse than not tuning at all. The converged weights were chess nonsense: bishop below knight, knight outposts penalized, isolated pawns rewarded, and every passed-pawn bonus negative.
Early stopping rescued a plausible-looking 5.9% held-out improvement, and that turned out to be noise as well. Repeating the entire fit across four different game splits:
| split | start valid | best valid | change |
|---|---|---|---|
| 1 | 0.0938 | 0.0909 | -3.1% |
| 2 | 0.1089 | 0.1022 | -6.2% |
| 3 | 0.1285 | 0.1285 | +0.0% |
| 4 | 0.0661 | 0.0543 | -17.8% |
The answer ranges from nothing to 18% depending only on which 32 games are held out, and the starting error varies 2x across the same splits. The 5.9% was one draw from that distribution.
The cause is volume: 127 usable games give 3,231 quiet positions, about 111 per parameter, where the method wants thousands. The tuned values were not applied.
This is the 70-Elo-instrument mistake wearing a new costume: a number that looks like a result because an optimizer produced it, on a sample too small to carry it. The difference this time is that the check existed before the number did, so it cost one afternoon instead of a night of matches. The tool stays and re-runs cheaply. The gate for believing it is that the split-to-split spread above collapses.
The entry above concluded "re-run it once the bot has played several hundred more games", and the plan duly carried that as one of three reasons the validation run needed about 600 games. Then Lucas asked the obvious question, which was why we were not using an existing dataset, and the answer was that nobody had checked.
load_positions() reads PGNs out of the bot's game_records. Nothing else about
the method requires that. It was simply the input format the tool was born with,
and it quietly turned "we need more data" into "we need to play more games".
quiet-labeled.epd is 725,000 already-quiet labeled positions in a 39 MB
download that has been public for years. Texel's own paper used 8.8M positions
from 64,000 games.
The arithmetic that should have been done first: 600 bot games is roughly 15,000 positions, still ~50x short of what the method wants. The eight days would have bought a second noisy fit.
What it actually took, all inside engine/tools/tune.py with engine/
untouched:
GameState.from_fen on a 4-field EPD FEN |
already worked, since it accepts 4 or more fields and defaults the clock |
| new code | load_epd(), about 60 lines, plus a per-position result label so both loaders feed one splitter |
uncached evaluate() |
20.1 us CPython against 2.25 us PyPy, or 1.6 s per pass over 725k |
| memory | 6.6 KB per GameState, so 725k would need about 4.6 GB. The default samples 200,000, or about 1.3 GB |
Sampling takes every Nth block of 512 lines rather than every Nth line, which keeps both properties that matter: blocks stay contiguous, so neighbouring positions from one game do not straddle the train/validation cut, and the blocks taken are spread through the file instead of being a prefix of it.
Two cores go to running different --seed values side by side rather than into
a worker pool inside one fit. The verification needs 3-4 seeds either way, so
both spend the same hour, and seed-level parallelism costs no code and loses
only one result if something crashes.
Nothing is applied yet. Applying the values is a behavior change, and
PIECE_VALUES is shared with the search's SEE, so it moves move ordering as well
as scoring. That needs an SPRT, after the current validation run. The fit itself
waits for the bot to be down, because 30 minutes of a saturated core would starve
the engine in the rated games that run exists to collect.
The lesson here is not about chess. When a tool appears to need data you do not have, check what its loader can already read before going out to collect any. The measurement discipline in this repo is aimed at not believing bad numbers and it worked, but it had nothing to say about a question nobody asked.
Four seeds, 200,000 sampled positions each, 5,155 training positions per
parameter against 106 before. Logs kept in
~/.local/share/pycheckmate/tune-runs/.
| seed | start valid | best valid | change |
|---|---|---|---|
| 1 | 0.065585 | 0.063511 | -3.16% |
| 2 | 0.066302 | 0.064384 | -2.89% |
| 3 | 0.066325 | 0.064223 | -3.17% |
| 4 | 0.064771 | 0.063169 | -2.47% |
This was the gate. The previous run answered -3.1%, -6.2%, +0.0% and -17.8% depending only on which games were held out, with the starting error varying by a factor of two, so the answer was a draw from a distribution rather than a measurement. The spread is now 0.7 points and the starting errors agree within 2.4%. Volume was the blocker, exactly as suspected, but the fix was the loader rather than more games.
Two results matter more than the error does.
PIECE_VALUES did not move in any seed. The hand-picked 320, 330, 500 and 900
were already at the fit's optimum. That also removes the change's main risk,
since the search shares PIECE_VALUES with static exchange evaluation, so a
shift there would have moved move ordering as well as scoring. It does not.
The values are also coherent as chess, which the last run's were not. Passed pawns are the clearest case, and they read as nonsense until you check the indexing, because index 1 is one step from promotion rather than one step advanced:
- Endgame passers keep their weight (
_END[1]180 -> 156-172). - Middlegame passers lose most of theirs (
[1]120 -> 40-72), and the ranks furthest from promotion go slightly negative ([4..6]30/20/10 -> about -18/-12/-22).
That is the evaluation's own comment, fitted: with heavy pieces still on, the same passer is often just a target. The hand-picked column never expressed it. Compare the previous run, where every passed-pawn bonus went negative including the endgame column. That was noise. This is a shape.
Two terms should be treated as open questions rather than results.
ROOK_ON_SEVENTH_BONUS went from 20 to 4 with all four seeds agreeing, and a
rook on the seventh is a real and well-known asset, so either the dataset's quiet
filter removes the positions where it pays or the term overlaps something already
scored. Agreement across seeds means the fit is consistent, not that it is right.
And MOBILITY_BONUS[2] for the knight went from 4 to 1 while the rook went from
2 to 4 and the queen from 1 to 4, which is coherent as a redistribution toward
sliding pieces but is the other place the fit contradicts received wisdom.
Nothing is applied. Held-out error is not Elo. It says the evaluation predicts game outcomes better, not that the engine plays better, and those two come apart when a term also feeds search behavior. Applying these needs an SPRT, and the machine cannot run one while the bot plays.
A correction from the same day: all four seeds read the same biased 82% of the
file. A cold review of the new loader found that blocks[::stride][:wanted] does
not thin a file out, because integer division always over-strides, so the
truncation discards exactly the tail the stride existed to reach. At the 200k
default the last 17.6% of quiet-labeled.epd was unreachable, and --seed could
not rescue it, because the seed reshuffles blocks after selection. The four seeds
above are therefore four splits of one biased sample, which weakens the
spread-collapse argument they were run to make. The values are parked unapplied on
branch texel-tuned, and the fit was re-run on the fixed sampler before anything
was measured. Worth stating plainly: the check that would have caught this is one
the test file already had. It asserted the right property with the single input
size that cannot expose the bug, at 6 blocks and 2 wanted, where the stride
happens to come out exact.
Four seeds again, this time reading the whole file.
| starting-error spread | answer spread | |
|---|---|---|
| original fit, 3.2k positions from our PGNs | 2x | 17.8 points |
| EPD fit, biased to 82% of the file | 2.4% | 0.7 points |
| EPD fit, whole file | 1.0% | 0.38 points |
Held-out error fell 2.73%, 2.57%, 2.92% and 2.95%. Seventeen of the 24 moved
parameters have three or more seeds agreeing exactly, and PIECE_VALUES again
did not move in any seed, which is what keeps this change out of the search,
since static exchange evaluation shares those values.
The sampling bias turned out not to matter. The values land essentially on top of
the biased fit, with ROOK_ON_SEVENTH_BONUS at 20 to 8 in all four seeds, the
same mobility redistribution toward sliding pieces, and the same passed-pawn
shape. The file was homogeneous enough that the missing 17.6% carried no distinct
signal.
That is worth being careful about rather than relieved by. The fix was still necessary, because without it there was no way to know. "The tail probably looks like the rest of the file" is an assumption about someone else's dataset, and the point of the exercise was to stop assuming things about the input. A bug that would have mattered and a bug that happened not to are indistinguishable until you remove it. What the agreement does buy is confidence in the earlier run's conclusion, which was drawn before anyone knew the sampler was broken.
Applied as 94efc2d on branch texel-tuned and left unmerged, because held-out
error is not Elo. The SPRT decides.
The rule in CLAUDE.md is explicit and predates all of this: the bot and any
timing instrument cannot run together, because sprt, calibrate and bench
timings all need the machine to themselves. I broke it anyway, during the very
session that added the memory-budget rule it exists to protect.
While an SPRT was live at concurrency 2, the following ran alongside it: two
bench invocations, several depth-7 PyPy search probes with 30-second limits,
and repeated test suites. Each probe holds its own transposition table and
evaluation cache, so this is the concurrency-4 memory failure again, arriving
through a side door.
From the match log, the damage was 22 time forfeits out of 436 games, none of
them before game 201 and most clustered between 384 and 417, which is exactly the
measurement window. Overruns ran to 261s, 540s and 925s on a 60-second clock,
which is the swap signature and not any plausible search behaviour. It was also
asymmetric: new lost 15 of them against base's 7, so the corruption biased
the result against the change under test.
Recomputing from the PGN over the 416 games that finished normally:
| games | score | Elo | |
|---|---|---|---|
| as reported by fastchess (all games) | 436 | 54.17% | +29.0 ± 29.5 |
| excluding time forfeits | 416 | 55.05% | +35.2, CI [+1.9, +69.2] |
Two things follow, and the second matters more than the first.
The filtered number is evidence rather than a verdict. Dropping games after seeing them is exactly how analyses get biased, even when the exclusion criterion is environmental rather than chess-related, as it is here. I quote it as a sanity check on direction, and the SPRT's own LLR, which still contains the corrupted games, remains the thing that decides.
A conservative contamination is survivable where the reverse would not be. The bias runs against the change, so if the SPRT still concludes H1, that conclusion is safe. Had it run the other way, the whole run would have to be discarded. That asymmetry is luck rather than design, and the rule exists precisely because you do not get to choose the direction of contamination.
The general lesson is one this log keeps relearning in new costumes: an instrument sharing a machine is not an instrument. Twice now the harness has quietly changed what it measured, first with the clock-fidelity trap and then with the concurrency-4 memory failure, and this is the third. It differs only in that the interference came from the person watching the experiment.
With the evaluation's optimization space measured out, the remaining lever is searching fewer nodes rather than making each node cheaper, which means move ordering. The standard metric is the share of beta cutoffs found on the first move tried, since that is what alpha-beta needs in order to prune a whole subtree.
The baseline is 88.4% over 22,681 cutoffs across three depth-7 searches, with 97.2% inside the first three and a mean rank of 0.34. Strong engines target 90 to 95%, so there were a few points of headroom. Splitting it by move type says exactly where:
| cutoff move | on 1st try | later | first-move rate |
|---|---|---|---|
| captures | 15,302 | 449 | 97.2% |
| quiet | 4,739 | 2,191 | 68.4% |
Capture ordering is already near-perfect. The TT move plus MVV-LVA plus SEE
demotion leaves almost nothing on the table, so capture history, the item the
plan carried from ENGINE_V2_PLAN.md §7.5, would buy essentially nothing. 83% of
all late cutoffs are quiet moves. That is a plan item retired by five minutes of
instrumentation.
So the change I built instead was for the quiet table. History gravity makes scores saturate toward a ceiling rather than accumulating forever, and history malus treats quiet moves searched before the cutoff move as evidence against themselves. Those had previously been ignored entirely, since only the winner was rewarded, so refuted moves kept whatever stale score they arrived with.
It measured worse and was reverted.
| position | baseline | with malus | |
|---|---|---|---|
| opening | 34,528 | 35,976 | +4.2% |
| middlegame | 23,423 | 30,627 | +30.8% |
| tactical | 33,872 | 34,016 | +0.4% |
| endgame | 19,705 | 15,585 | -20.9% |
| total | 111,528 | 116,204 | +4.2% |
The first-move rate moved from 88.4% to 88.6%, which is to say not at all. Node
counts are exact here rather than noisy, because the bench seeds _root_rng.
Two mechanisms are worth recording, since a future attempt will meet both.
HISTORY_MAX at 16,384 against a bonus of depth * depth, which is 49 or less
at these depths, makes the gravity term almost inert, because
current * bonus / 16384 rounds to nothing until a score is already huge. So
what actually got measured was the malus on its own. And the malus as written
debits every earlier quiet move by the full bonus, which punishes moves for
having been ordered early rather than for being bad, and killers and
previously-good moves are exactly the ones sitting in that list. The endgame's
21% improvement suggests the idea is sound where the quiet move count is small,
and the middlegame's 31% regression that it is actively harmful where that count
is large.
I stopped here rather than tuning the malus, on a stop rule agreed before the work started: ship at 90% or better, revert otherwise. The alternative is an open-ended search over a metric that is a diagnostic rather than a deliverable.
The plan's headline remaining item was an incremental evaluation accumulator:
track which pieces a move changed and adjust a running score, instead of
recomputing evaluate() at every node. The claim was that it kills the 22%
evaluation slice, makes the 1M-entry _EVAL_CACHE deletable, and buys about 0.3
of a ply.
Decomposing evaluate() under PyPy over four representative positions, by
stubbing one term at a time:
| slice | cost | share | decomposable per piece? |
|---|---|---|---|
| mobility (ray walking) | 0.87 µs | 43.2% | no |
| material + PST | 0.12 µs | 5.8% | yes |
| pawns / rooks / king / outposts | 1.03 µs | 50.9% | no |
| total | 2.02 µs |
An accumulator reaches 5.8% of the evaluation, which is about 1.3% of search time. Not 22%.
The reason is structural rather than incidental. Material and piece-square values are the only terms that are a pure sum over pieces, and the loop that collects them has to run anyway, because mobility needs it and so do the pawn lists and the rook and knight lists. The accumulator does not remove a loop. It removes two array lookups from inside a loop that keeps running. Everything costly depends on the whole board configuration, since moving one piece changes the mobility of every sliding piece whose ray crosses the vacated or occupied square.
All three original claims fail. It is not 22%, _EVAL_CACHE stays plainly worth
having against an evaluation of about 1.9 µs, and 0.3 of a ply does not follow
from 1.3%.
The obvious next candidate was a pawn hash, the classic answer for pawn
structure, which depends only on pawn placement, and pawns move far less often
than pieces do. Measured rather than assumed, it does not rescue this either. The
pawn-structure hit rate is 92.4% over 49,546 evaluations across three depth-6
searches, so the cache itself would work. But the post-loop terms it could serve
come to only 0.32 µs, or 16% of evaluate(), since stopping the evaluation right
after the two piece loops costs 1.70 µs against the full 2.02 µs. And it cannot
skip the loops, because they are fused: the same pass that collects pawn files
and square colours also does mobility and the rook, knight and bishop lists. In
engines where a pawn hash pays, the pawn evaluation is a separate pass that can
be skipped whole.
The ceiling on a pawn hash here is roughly 0.3 µs, about 15% of the evaluation and 3% of search time. Better than the accumulator, and still not worth delicate new caching machinery.
The honest conclusion is that evaluate() has no large safe win left in it. Its
dominant cost is mobility at 43%, which sits inside the loop and depends on
whole-board occupancy, so it is cacheable only by something keyed on the entire
position, and that is what _EVAL_CACHE already is. The architecture-level
optimization was already done.
That redirects the effort rather than ending it. The remaining lever is searching fewer nodes rather than making each node cheaper, which means move ordering, and it is verified by exactly the same 8-second node count.
The part worth transferring is the sequencing. This took one measurement of about thirty lines, and it retired a feature that would have meant days of delicate make/unmake plumbing, in the exact code path where the 50-move clock bug hid for months. Estimate the ceiling of an optimization before building it, because the ceiling is usually cheaper to measure than the optimization is to write.
MIN_MOVES_TO_GO was raised 18 → 28 to push thinking time into the endgame,
where 32% of our moves and 47% of our blunders were being played on 8-12%
of the clock. The plan's way of checking that was ~600 rated games: eight days,
±28 Elo.
But "did the clock move where we aimed it" is a mechanism question rather than a
strength question, and the two have wildly different costs. A win rate is one bit
per game, so it needs hundreds of games. A clock allocation is one number per
move, about 60 observations per game, and every game we have ever played already
carries per-move timings in game_analysis.jsonl, lifted from the PGN's %clk
tags. engine/tools/clock_replay.py recomputes the allocation for any value of
the constant, over games that already happened, in about a second.
On the 41 games of the current build:
| phase | actual | MMTG=18 | MMTG=28 | MMTG=40 |
|---|---|---|---|---|
| 1-15 | 20.2% | 33.4% | 35.2% | 38.2% |
| 16-30 | 41.7% | 32.8% | 33.5% | 30.2% |
| 31-45 | 25.4% | 23.0% | 19.1% | 17.9% |
| 46+ | 12.7% | 10.8% | 12.2% | 13.7% |
The change does what it claims. Moves 46 and later gain 1.4 points of clock share, going from 2.80s to 3.00s per move, and it is smaller than the plan implied. That estimate came from simulating a 90-move 5+0 game, while our games run about 62 moves on 600s, and an absolute move-count floor binds much later there.
The first version of the tool was wrong, and the error is the interesting part. It reconstructed the clock from what the games actually spent, which pins every setting to the same trajectory, and that breaks the exact feedback loop the change relies on, since a bigger divisor is a thinner slice now but leaves a larger clock alive later. Measured against a fixed history, a bigger divisor can only look worse at every phase, by construction. The tool duly reported that raising the constant hurt the endgame by 4.1 points.
It was caught only because that answer contradicted a prediction with a mechanism
behind it. An instrument disagreeing with a mechanistic prediction is a reason to
audit the instrument first rather than the prediction, and here the instrument
was simulating a counterfactual while holding fixed the very quantity the
counterfactual changes.
test_a_higher_floor_moves_time_into_the_endgame pins the direction now.
Two things it deliberately will not judge: whether the reallocation helps, and anything about the best-move stability exit, whose gate depends on a stability count the records do not carry.
The first attempt to SPRT anything failed in a way that looked like an engine bug and was not. Every game ended with both engines overrunning the 60s clock, by 131s to 212s, and forfeiting, and the overrun grew with the length of the game. fastchess's trace ended on "Engine base is thinking" and never came back.
Ruled out one at a time, all negative: the tuned values (the untouched baseline hangs too), the interpreter (PyPy resolves correctly), time management (clocks stepped 60s down to 50ms, every reply under 0.94s), and the opening book (12 sampled positions, all ~1.1s).
The cause is memory. Each engine holds a transposition table of up to
uci.TT_MAX_ENTRIES = 2,000,000 entries plus eval._EVAL_CACHE at 1,000,000,
and sampling RSS through a real game shows 460-625 MB per process partway
through 114 plies, still climbing. Concurrency N runs 2N engines, so the
default of 4 put eight of them on this machine's 8 GB. It swapped; every
search slowed by orders of magnitude; the clock is simply what noticed first.
And because the caches grow as a game goes on, so did the overrun, which means
the one symptom that had looked most like a search bug is the plainest evidence
for memory.
Verified at concurrency 1: two complete games, 45 moves each, longest search
1.26s, both terminated normally with no forfeit. sprt.py now defaults to 2
(four processes, ~2.5 GB), which also sits under the CPU limit already
recorded here.
The lesson is that the instrument's own resource budget is part of the measurement design. This project has now lost time twice to a measurement harness that quietly changed what it was measuring, first with the clock-fidelity trap, where scaled test clocks moved the hoarding tiers to a different point in the game, and now with this. Both presented as engine defects. Neither was.
The fitted constants finally got their SPRT. texel-tuned (24 fitted eval
constants) vs master, 60+0, concurrency 2, run unattended to a **pre-committed
1,300 games, with the target fixed in advance precisely so the run could not be
stopped the moment the interval first cleared zero. Optional stopping is the bias
a fixed target exists to defeat.
The result was +19.5 ± 17.0 Elo at LOS 98.8%, pentanomial [92, 92, 240, 103, 123], which is 123 double-wins against 92 double-losses. As pre-committed, this is a fixed-size match rather than an SPRT pass. The LLR was 1.03 against the ±2.94 bound at 1,300 games and would have needed the full 2,000-game cap to resolve formally.
Then the clean recompute. A stretch of the early games was contaminated by running bench and PyPy probes alongside the live match (documented above); parsing the PGN and dropping the 28 time forfeits plus one illegal move leaves 1,287 clean games reading +20.0 Elo. The contamination biased against the change, since the forfeits fell on the tuned side, so the raw match was conservative and the clean number is the one to quote: +20 Elo, real.
The calibration that followed is the more interesting finding. Running
calibrate.py on the merged engine bracketed the 50% crossover at Stockfish
2400, with 75% at 2200, 62.5% at 2300 and 52.5% at 2400. That is far above the
2133 the project measured in July, and a +20-Elo tune cannot account for a +270
jump, so the gap demanded an explanation. The two hypotheses were real engine
improvement against instrument drift, meaning a newer Stockfish playing
differently at a given UCI_Elo.
The decisive test holds the instrument fixed and swaps only the engine, so I re-calibrated the July v1 engine under today's Stockfish. It read about 2180, with 72.5%, 42.5% and 27.5% at 2000, 2200 and 2400. That is the same engine on the same ruler as its original 2133, consistent within the ±150 that 20 games per level allows. Drift is ruled out and the instrument is stable, so the jump is real. Going from v1 at 2180 to tuned at 2400 on the same day is about +220 Elo, of which the tune is +20 and the remaining +200 or so is the Phase 1 bundle, chiefly the 50-move-rule-in-search and mop-up fixes, which stop the engine scoring won-but-drawn endgames as wins it never converts. Those four changes shipped unvalidated in July, and this measurement retroactively vindicates them.
Two honest limits on that number. The +200 is a bundle of four changes measured
together, so it names no single culprit, and it is read at n=20, so "about +220"
means "large and real" rather than a precise figure. And UCI_Elo is Stockfish's
own soft ruler rather than a portable Lichess rating. What is solid is that the
endgame work was worth far more than the tune, and that the classical evaluation
is now genuinely at 2300 or better against a foreign opponent our own evaluation
cannot flatter.
The plan's next candidate was king safety, and for once it was pre-justified by
data. The post-Phase-1 blunder hotspot had moved onto the middlegame, moves 16 to
32, carrying 41.5% of blunders, and it was not a clock problem, because that
phase gets the most time per move and still blundered most. That points at an
evaluation blind spot rather than a shallow search. The cheap de-risk before
building anything was to reconstruct the seven worst middlegame blunders from the
game record and read Stockfish's refutations. Six of the seven were our own king:
four clean attacks on a castled king (Rxg7+, a Qg4+ hunt, Bh6 with Ng5, Qh6) and
two an uncastled king caught in the centre. The engine kept walking its queen off
to grab a pawn while an enemy queen, minor piece and rook massed on the king's
ring, and evaluate() had a pawn shelter term but nothing at all for enemy
attackers. So the term to build was not shelter. It was king danger by attacker
count.
The term counts the squares ringing each king that the enemy attacks, penalises
that count squared, because a mating attack is a threshold rather than a sum of
independent threats, and gates the whole thing on the enemy still having a queen.
The first cut tested each of the eight ring squares with is_square_attacked,
which re-walks eight ray directions per square, and a profile put that at about
26% of the whole search and 34% of NPS. Rewriting it as a single pass over the
attacker list held it to about 17% of NPS. Each piece marks the ring squares it
hits, with an alignment pre-filter that rejects any slider not sharing a rank,
file or diagonal with the 3x3 king box, and the result was verified
byte-identical to the naive version across 240 random positions. I also tried a
cheaper approximate metric, attackers on the king's own rays, and it measured 0
on every real blunder position, because the attacks come from perpendicular lines
the king's own rays miss. The exact ring scan is required. The weight was
Texel-fit in isolation using a new tune.py --only flag, so the
already-validated constants stayed fixed, and it went from 3 to 7 with held-out
error falling, so the term carries real signal even on quiet positions.
Paying the NPS bill came first. A 17% evaluation tax wants a faster search to
spend it against, so a whole-repo performance audit ran before the SPRT. Its top
find was that board and movegen, the two modules the search touches at every
node, never got the precomputed-ray treatment eval and search already had,
and every call re-tested 0 <= r < 8 at each step of each ray. Moving that trick
down into board as _ATTACK_RAYS, _SLIDE_RAYS and _KNIGHT_ATTACKS, all
built once at import, plus three micro-optimizations (skip the castle-rights
repack when rights cannot have changed, replace abs(x) < T with range
comparisons, use a defaultdict for the get-or-zero counters) measured about 16%
faster. Being provably score-neutral, it was verified by identical bench node
counts with no SPRT spent, which is the node-count check earning its keep again:
a 30-second proof standing in for an overnight match.
The king term against the sped-up master at 60+0 came out at +30.5 ± 31.5 Elo with LOS 97.2% over 338 clean games. I accepted it pragmatically at 97% rather than driving it to the formal +2.94 LLR, which had only reached 0.45, because it is coherent with every prior and the +30 is net of the 17% NPS cost. The king-safety knowledge clears the slowdown and gains on top of it. Merged.
The operational lesson is recorded because it cost the evening. The SPRT died and
restarted three times before one survived, and each cause was different and each
masqueraded as another. Overruns of 100 to 160s looked like the documented
memory-versus-concurrency swap, but were actually macOS throttling the engine
processes when the display slept. caffeinate fixes that, and an 856s freeze on
a two-engine game, which is impossible from memory pressure, is what finally
named it. A run that died at game 338 after four clean hours looked like the
engine, and was the harness reaping its own tracked background task. What ended
the cycle was launching the match fully detached with nohup, re-parented to
launchd, so no supervisor can reap it. setsid, the obvious tool, silently does
not exist on macOS, which is why the first two detach attempts failed without a
word. The general shape of it: when a long unattended job keeps dying, suspect
the environment running it, meaning sleep, the supervisor and the OS, before you
suspect the job.
The engine had no opening theory at all. From the start position it just searched, which is the slowest place to search, because the tree is at its widest, and the least rewarding, because the lines are known theory that every strong opponent has memorised. A book fixes both at once.
engine/tools/build_book.py reads the Lichess Elite Database (2400+ rated
games), replays the first 16 plies of each, and tallies how often each move was
played from each position. Positions are keyed by the **Zobrist key the search
already maintains, so the runtime lookup is one dict probe with no new position
representation and no parsing at play time. Two filters keep it small and sound:
a position is kept only if at least 8 games reached it (MIN_POS), and within a
kept position any move seen fewer than 2 times is dropped as noise (MIN_MOVE).
80,000 games produce 291k raw positions, of which 13,230 are kept, at 700 KB of
JSON. That is small enough to commit, so the engine ships self-contained. The
build runs in 26 seconds under PyPy.
At runtime (engine/book.py, probed by handle_go in uci.py) a booked
position returns a move weighted at random by how often the corpus played it
favouring the mainline while varying our openings so a repeat opponent cannot
memorise our replies. It does that with zero search, banking the clock for the
middlegame where depth actually changes results. Off book, book_move returns
None and the search runs exactly as before. The probe checks that the book move
is legal in the exact position first, so a stale book can never make us forfeit
on an illegal bestmove.
It is worth being clear about what this is not. It does not make the search or
the evaluation any stronger, because it is a lookup table bolted on the front.
Its value is clock economy, not fumbling known theory, since our naive opening
evaluation is weakest exactly where the book is strongest, and unpredictability.
It is also invisible to calibrate.py, which forces a random 8-ply opening on
every game for variance reduction, and from those junk positions the elite book
never fires. The book's payoff therefore shows only in real start-position games,
meaning the bot, while calibration measures the engine without it. That is a
property of the instrument rather than a defect of the book.
Recalibration (2026-07-24), the reason the book work was paired with it. A fresh bracket confirms the engine sits at ~2300 Elo, holding the post king-safety gains with no regression from the new code path:
| Stockfish level | Score | Implied rating |
|---|---|---|
| 2100 | 16.5/20 (82.5%) | ~2369 |
| 2300 | 10.0/20 (50.0%) | ~2300 (the clean anchor) |
| 2500 | 9.0/20 (45.0%) | ~2465 |
20 games per level is noisy, and the three points span about 165 Elo, but the 2300 level landing on exactly 50% is the tightest read and pins the estimate at 2300, consistent with what was measured on 2026-07-23. The book contributed nothing to these numbers by construction, because of the forced openings. This run's job was to prove the engine was unregressed and to refresh a stale figure, and it did both.
Measured with engine.tools.bench (4 positions: opening / middlegame / tactical /
endgame), seeded RNG, best-of-5 back-to-back runs.
| Date | Change | Metric | Before | After | Gain |
|---|---|---|---|---|---|
| 07-18 | Integer piece codes + search cuts | Depth-5 search | 1.35s | 0.28s | ~4.8x |
| 07-18 | Zobrist eval cache (PyPy) | Depth-6 search | n/a | n/a | 35% faster |
| 07-18 | Zobrist eval cache (CPython) | Depth-6 search | n/a | n/a | 8% faster |
| 07-19 | Precomputed ray tables | Bench total (best-of-5) | 3.471s | 3.111s | 10.4% |
| 07-19 | Precomputed ray tables | _mobility isolated (56k calls) |
0.053s | 0.030s | 43% |
The safety proof for the ray-table change is that both versions visited exactly 113,218 nodes. Identical node counts across all four positions mean every search decision was unchanged, so the optimization provably cannot have altered play. The sample ranges were disjoint, at 3.471-3.499 against 3.111-3.210, so the timing result is real rather than the 29% run-to-run noise this machine shows.
Time allocation (tm_replay over 16 real games):
| Metric | Before | After |
|---|---|---|
| Moves 21-40 budget, 5+0 | 3.7s | 5.4s (+46%) |
| Moves 21-40 budget, 10+2 | 9.3s | 12.9s (+39%) |
| Clock unspent after 45-move game | ~25% | 18% |
| Opening move 1 budget, 5+0 | 10.0s | 5.7s (damped) |
Not yet measured: nodes/second, and depth-reached-in-fixed-time over the project's history. Both would strengthen this table; neither was recorded early enough to reconstruct honestly.
Perft from the start position, make/unmake at every ply:
| Engine | Representation | Host | Nodes/sec |
|---|---|---|---|
| python-chess | bitboard | CPython | 335,510 |
| PyCheckmate | mailbox + piece-sets | CPython | 392,994 |
| PyCheckmate | mailbox + piece-sets | PyPy | 958,410 |
| Component | Share |
|---|---|
| Move generation | 37% |
| Evaluation | 22% |
| Ordering, SEE, make/unmake, overhead | 41% |
| Budget | Before | After |
|---|---|---|
| 3.0s | 1.73s (58%) | 3.03s (101%) |
| 5.0s | 3.44s (69%) | 5.01s (100%) |
Overrun at a 0.1s budget: 141% → 111% (clock sampled every 512 nodes instead of 2048; benchmark cost 0.6%, node count unchanged).
| Moves | ACPL | Blunders |
|---|---|---|
| 1-15 | 13.3 | 1 |
| 16-30 | 30.2 | 5 |
| 31-45 | 28.1 | 3 |
| 46+ | 24.1 | 1 |
The opening is clean, because the polyglot book is doing its job there. Play degrades sharply the moment it runs out while the position is still complex.
159 tests, strict mypy across 43 source files, and a clean ruff run; all three gate every change.
| Method | What it proves | Result |
|---|---|---|
| Perft, depths 1-4 at 20 / 400 / 8,902 / 197,281, plus Kiwipete | Move generation is exactly correct | Byte-exact. Caught the 2-node ray bug at depth 5 |
| Random-walk make/unmake | white_pieces and black_pieces sets stay exact through thousands of random moves |
Passing, and it guards the subtlest class of bug in the codebase |
| Node-count reproducibility | The measurement method itself still works | Passing, and it protects the seeded-RNG invariant |
| Self-play smoke test | Whole games hold together over the real UCI round trip, and the referee rejects illegal moves | Passing |
A/B self-play matches, abtest.py, retired 2026-07-20 for sprt.py |
Strength changes, where 400 games gets to about ±35 Elo | Used to reject LMP at 60 and then 24 Elo worse |
Blunder replay, tm_replay.py, retired for sf_watch.py |
Where the engine actually loses, from real games | 73% of errors in moves 21-40 |
Allocation probe, tm_allocate.py, retired once the question closed on 2026-07-25 |
Whether a time rule spends more where games were lost | Rejected the stability heuristic at 1.11x against 1.34x |
| Live Lichess play | Everything, against real opponents | Deployed and playing |
The measurement instruments took the longest to get right:
| Instrument | Question it answers |
|---|---|
engine/tools/bench.py |
Did a score-neutral change stay score-neutral? Node counts must be identical, which makes it a proof rather than a p-value |
engine/tools/sprt.py |
Is version B stronger than version A? A sequential test over fastchess that stops when decisive and declines when neutral |
engine/tools/calibrate.py |
How strong are we in absolute terms? Stockfish pinned by UCI_LimitStrength |
engine/tools/sf_review.py |
Where did we go wrong in these games? Independent Stockfish grading |
engine/tools/sf_watch.py |
The same thing, automatically, in the bot's idle time, stamped with the engine build |
| perft suite | Is move generation still exactly correct? |
Two design rules govern all of them, and both were learned the hard way. One change in flight at a time, because five features landing together with a net of zero teaches nothing about any of them. And never average across engine versions, because a mean over two engines describes neither of them.
Known limitations:
- No endgame tablebases.
- Single-threaded search, so no SMP.
- The opening book covers only lines that at least 8 elite games reached, to a depth of 16 plies. Off that path, which happens fast against unusual bot openings, it contributes nothing.
- 34 hand-picked search and evaluation constants, none of them ever fitted to data.
- Self-play at a scaled clock is no longer used as a verdict at all, for the reasons in the clock-fidelity trap. The historical caveat here applies to the abandoned gate, which used self-play games averaging 72 moves per side while real online games average 45, so it exercises the time schedule's tail harder than deployment does.
- Zero flagged games observed in the gate so far, which means the overspend-protection half of time management is still untested by self-play.
Deployed on Lichess as PyCheckmate, playing blitz 5+0 and rapid 10+2, with
bullet disabled because move overhead is too tight to be safe there yet. All
games are rated, against opponents bracketed to 1800-2500.
Strength is about 2300 or better on Stockfish's UCI_LimitStrength ruler, from
2026-07-23, with the crossover bracketed at 2400, up from about 2133 in July.
The bracket:
| Stockfish level | Score | Implied rating |
|---|---|---|
| 2000 | 15.0/20 (75.0%) | n/a |
| 2200 | 15.0/20 (75.0%) | n/a |
| 2300 | 12.5/20 (62.5%) | n/a |
| 2400 | 10.5/20 (52.5%) | near the crossover |
The July figure of 2133 was not wrong. Re-running that same v1 engine under
today's Stockfish still reads about 2180, so the ruler is stable and the gain is
real engine improvement: roughly +200 Elo from the Phase 1 endgame bundle, being
the 50-move rule in search and the mop-up term, plus +20 from the Texel fit. The
2026-07-23 entry above has the full decomposition and its caveats, chiefly that
UCI_Elo is a soft internal ruler rather than a portable Lichess rating.
Quality graded by Stockfish at depth 14 over 43 games: ACPL 19.5 and a blunder rate of 0.4%, or 5 in 1289 moves. For scale, the 2930-rated opponents in that set averaged 6.8 ACPL.
Lichess rating, settled 2026-08-05 from api/user/PyCheckmate. Both pools are
established at RD 50, well past provisional.
| Pool | Rating | Games |
|---|---|---|
| Rapid (10+2) | 2187 | 130 |
| Blitz (5+0) | 2050 | 114 |
| Bullet / Classical | n/a | not played |
Rated record: 105 W / 134 L / 30 D over 269 games, or 44.6%. That number comes with this project's own rule attached, because it spans several engine versions and both rating pools, so it describes no single engine. The per-pool ratings are the quotable ones, and rapid is the honest one. The 137-point gap down to blitz is the depth-starvation finding in Key Takeaway 9, not two different engines.
The other 47 games of the 315 total were the casual run that produced the 1/43 record. Challenge 8 covers why that sample could not measure anything.
Profile: https://lichess.org/@/PyCheckmate
Stockfish's UCI_Elo ruler puts the engine at about 2300 while Lichess rapid
says 2187. Those are different scales rather than a contradiction: UCI_Elo is
Stockfish's own internal ruler and was never portable. They agree on the part
that matters, which is that the engine plays in the low 2200s against real
opposition.
What is left, with docs/FUTURE_PLAN.md carrying the full version:
-
Texel tuning of the evaluation constants.Done on 2026-07-23, at +20 Elo by SPRT, merged. 24 constants fitted on 725k public labeled positions. -
King safety in the evaluation.Done on 2026-07-23, at +30 Elo in self-play with LOS 97%, merged. The de-risk confirmed the theme, since 6 of the 7 worst middlegame blunders were our own king, so the term counts attackers on the king ring rather than measuring pawn shelter. Verified net of its 17% NPS cost, and covered in the 2026-07-23 entry. -
Endgame evaluation, meaning passed-pawn races and when to trade into a pawn ending. This is a live strength weakness rather than a bug, and rated game
JvxRHGJaon 2026-07-24, lost to GNUPassant, surfaced it. From5Q2/8/2p1q1k1/2P5/3P2pp/8/1r4PK/5R2 wthe engine playedQd6, trading queens into a rook and pawn ending. Stockfish at depth 24 scores that at -2.52 against -0.41 for keeping queens on withQf4, a two-pawn swing that lost the game. The engine still choosesQd6at full depth, so this is the static evaluation and not the horizon effect. It overvalues its own blockadable c and d passers and undervalues Black's connected, far-advanced g and h passers sitting next to White's king. The later moves that looked like blunders,59.Ra1and70.Ra1hanging the rook, were already-lost positions plus in-game time pressure, and at a clean 3s search the engine finds Stockfish's top move in both. The candidate fix is a passed-pawn term scaling with rank and king proximity, plus a signal discouraging trades into a worse pawn structure, validated by SPRT the way king safety was rather than merged on intuition. -
Incremental evaluation.Measured on 2026-07-22 and dropped. It reaches 5.8% ofevaluate(), or about 1.3% of search time, against a claimed 22%. -
Collapsing the dual move pipeline, meaning
for_aiandmake_ai_movealongsideMoveandmake_move, across 64 references with every invariant maintained twice. -
Opening book owned by the engine rather than the bridge.Done on 2026-07-24:books/book.json, 13,230 positions from the Lichess Elite Database, Zobrist-keyed and weighted-random. Covered in the 2026-07-24 entry. -
Let the engine's own book actually play, because it has still never been tested. Found on 2026-07-25 while capturing the bot's config into
deploy/. The book was built, committed and shipped, and then measured nothing, because the bridge answers first. lichess-bot'sconfig.ymlsetpolyglot.enabled: truewithengines/komodo.binandmax_depth: 20, andlib/engine_wrapper.pycallsget_book_moveat the top of its search, so if that returns a move the engine is never asked.max_depth: 20is 39 plies, so for the entire opening of every rated game the Komodo book chose the move andbooks/book.jsonwas never probed.This is the same class of error as the casual-games trap and the scaled test clock, and the third time it has appeared: a measurement that silently never exercised the thing it was supposed to measure. The 2026-07-24 entry closed by saying the book's payoff would show in real games rather than on the ruler, and that was wrong in a way nothing in the game records could reveal, because a shadowed book and a useless book produce identical evidence. What would have caught it is one question asked before collecting anything: by what path does this code get executed in the thing I am about to measure? The book was verified end to end under
python -m engine.uci, which is exactly the path lichess-bot bypasses.Flipped to
polyglot.enabled: falseindeploy/config.ymlon 2026-08-05, as part of closing the project out. It changes what the bot plays, so whoever brings the bot up next should take a fresh version cut, and the book finally gets its own answer. Documented indeploy/README.md.
What "done" looks like: the engine plays rated blitz unattended without crashing, flagging, or hanging pieces to the horizon effect, at a stable rating I can actually quote. That is met. The strength number exists at about 2300 on Stockfish's ruler, up from 2133, and the rated deployment produced the public Lichess ratings above to match it.
-
The hardest problem was measurement rather than chess. Writing alpha-beta is a weekend. Knowing whether your change helped is the actual discipline. I burned a full night of self-play on search stages F through J for roughly 0 Elo of unresolvable noise, because I was asking a 70-Elo instrument about a 20-Elo change.
-
Deterministic checks beat statistical ones whenever you can get them. Node count is exact, so identical totals prove a change did not alter play. That replaced an overnight match with a 30-second check for the ray-table optimization, and it is stronger evidence besides, since it is a proof rather than a p-value. The lesson generalizes past chess: prefer the check that can only have one answer.
-
My intuitions about my own code were wrong more often than they were right. I expected attack detection to be the hotspot and it was evaluation. I expected best-move stability to identify hard positions, and at the time it identified quiet endgames flapping between equal moves, though it does identify hard positions now, as takeaway 4 explains. I hand-labelled two test positions and got both backwards. Measuring caught every one of those, and reading the code would have caught none of them.
-
Negative results are worth committing, and they expire. Three features were built, measured and deleted: LMP at 60 Elo worse, countermove plus IIR as unproven, and best-move stability at 1.11x against the crude rule's 1.34x. Each left behind a comment explaining why it failed, and that why is the load-bearing part. In July 2026 best-move stability was rebuilt and worked, because an unrelated change had removed the exact mechanism the old comment blamed, and the comment naming that mechanism is the only reason anyone noticed. A note saying "tried it, measured worse, don't rebuild" would have been obeyed, correctly on the evidence and wrongly in fact. A negative result is a statement about code that existed at the time and not a law, and a thing not built produces no symptom to alert you.
-
Check that your data can answer your question before analysing it. I built three increasingly sophisticated analysis tools on top of 43 games that were structurally incapable of measuring anything, because every one was played casual, so matchmaking paired a 2133 engine against 2928 opponents, where an 1800 and a 2400 engine both score around 0%. No amount of downstream sophistication recovers a signal that was never sampled. The fix was one config line, and finding it meant reading the raw API response instead of the scoreboard.
-
"Rewrite it in the faster representation" is a hypothesis rather than a plan. The obvious fix for a slow Python engine is bitboards. Measured, our mailbox engine is 17% faster than python-chess's bitboards on CPython, because the bitboard win is a C win and in Python every bitwise operation allocates. Profiling then capped the entire theoretical gain at 1.6x anyway. The instinct behind the request was right and the mechanism would have made things worse.
-
You cannot debug what you cannot see. The engine shipped without a single UCI
infoline. Adding them took an hour and immediately surfaced a bug that had been silently capping every movetime-based test at depth 5, and then a second one where the search was discarding a third of its clock. Both had been invisible for weeks while I ran ever more elaborate offline analysis against a black box. -
The bug that mattered was invisible to playing the game. Two nodes wrong out of 4.8 million: undetectable by watching games, unhittable by hand-written tests, and found instantly by one number that had a known correct value. Since then every subsystem got an exact-value test wherever one existed.
-
The ceiling is the language, and it shows up as a time-control gap. 113 rated games on
c2f0117score 0.513 overall, but Lichess rates blitz and rapid in separate pools and the two describe different engines. Rapid performs at 2214 to 2235, blitz at 2027 to 2084. That is a gap of about 160 Elo between 10+x and 5+x, roughly double the normal cost of halving a time control, and it is concentrated in the middlegame, where blitz ACPL runs 28.3 against 22.3 and the blunder rate 1.4% against 0.6%. That is the phase where the branching factor is widest and depth is worth the most.The tempting read is a time-management bug, and it is wrong. Four checks across 7,163 graded moves all came back negative. The clock is never in trouble, with a median minimum remaining of 72s in a 300s game and one time forfeit in 113. The blunder rate is flat against the fraction of budget spent, at 1.3, 1.0, 0.5, 1.2 and 1.1% across quintiles. The 52 critical blunders used 85% of their budget against 68% on clean moves, so they were the opposite of rushed. And games containing one score 0.486 against 0.474 for games without, which is no predictive power at all. The late-game moves that look instant are mate-score breaks in already-decided positions, and in contested late positions the sub-second moves blunder at 0.6%.
So the engine is not misplaying its clock. It is depth-starved. Measured on 2026-07-25 on this machine, the search sustains 33,777 nodes per second under CPython and 42,917 under PyPy, which is the interpreter that actually hosts it in deployment, where a C engine does millions. PyPy earns its place, since raw move generation goes from 467k to 1.07M nodes per second on perft, but it narrows the gap rather than closing it, because the search is dominated by evaluation rather than by move generation. Every Elo the search wins has to be bought back out of nodes. This is the same wall takeaway 6 hit from the other side, where the obvious bitboard rewrite turned out to be a C win that Python cannot collect. For the README, then: the honest framing of this project's strength is a rapid-time-control number, and the language is the binding constraint on everything else.
The last open question in this log, and the shortest entry in it. books/book.json
was built on 2026-07-24, committed, shipped, and then measured nothing for twelve
days, because lichess-bot's polyglot book answered the whole opening before the
engine was ever asked. Flipping polyglot.enabled to false and bringing the
bot up settled it in one game.
Game uwuPNwac, rated rapid, White against HynobiusChess at 1967:
| Moves chosen by our book | 3, being e2e4, g1f3 and f3e5 |
| Moves chosen by the polyglot book | 0 |
| Result | Win by checkmate, 117 plies |
| Rating | 2184 to 2187 |
The log line to look for is info string book e2e4 followed immediately by
bestmove e2e4, with no info depth in between, which is the book returning
without a search. d2d4 on move 4 is the first move the engine actually
searched, at depth 8 in 3.2s. Every move in the game reports Source: Engine,
so the bridge's book never answered once.
The line it produced was 1.e4 e5 2.Nf3 Nf6 3.Nxe5, which Lichess labels
Petrov's Defense. That matters more than the win does: the book is reproducing
real theory out of the Lichess Elite corpus rather than emitting whatever a
thin sample happened to contain, which was the risk of a book built from 80,000
games with a MIN_POS of 8.
One game proves the plumbing works and proves nothing about strength. The book's actual value, which is clock economy plus not fumbling known theory, still needs the dozens of games this project no longer intends to play. But the specific failure recorded above, a shipped feature that produced no evidence either way, is closed.
Last updated: 2026-08-05. Development stopped here. Lichess ratings settled at
rapid 2187 and blitz 2050, both established, against about 2300 on Stockfish's
ruler. Two loose ends were closed rather than left hanging: the rating
placeholder above now holds real numbers, and polyglot.enabled went to false
in deploy/config.yml, which let the engine's own opening book choose a move in
a rated game for the first time. The entry directly above records what happened.
On the ceiling: every cheap performance lever was measured and came back empty.
The 2026-07-22 entries have the numbers, with the accumulator at 1.3%, the pawn
hash near 3%, staged ordering neutral, and move ordering already at 88.4%. What
is left is a JIT-compiled hot path and a neural evaluation, and this architecture
cannot absorb either. docs/FUTURE_PLAN.md carries that analysis along with the
smaller items that would still fit here.