Continue dynamic move-ordering/LMR session: SPRT-validated at 1770+ games

Gate match vs head (st4.0) shows LLR=+2.89 approaching the H1=+2.94
bound, candidate score 0.5324 (LOWER95=0.5092, ELO=+22.6+/-16.2),
candidate depth/ebf both improved (13.98 vs 13.61 avg depth, ebf 3.324
vs 3.464) with comparable nps -- consistent with real search-efficiency
gains rather than noise. match_play.py gets additional instrumentation
to report these stats; search.c/searchsup.c/generate.c/chess.h/eval.c
carry the underlying move-ordering/LMR changes from this session's
line of work.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 34 insertions, 100 deletions

diff --git a/src/TODO b/src/TODO
index aa53d65..48b42cf 100644
--- a/src/TODO
+++ b/src/TODO
@@ -1,8 +1,10 @@
 Better understanding of stuff like KRKRB == drawish, Q's come off = !drawish
+
 Pawnhash size tunable from cmdline again
+
 Think about when to avoid nullmove pruning based on checks in the line
+
 Think about futility pruning when a >= +NMATE
-Stick eval / king score in ply info and use it to trigger extensions
 
 r2r4/p4pk1/2p3p1/1p1nPR2/5p1Q/2N5/PPPq4/1K4R1 w - - 0 0 -- check extension crazy
 
@@ -24,59 +26,6 @@ whether the tiers actually need different margins here too). Not yet
 built -- deferred behind self-play validation of today's iPositional/
 futility-margin changes landing first.
 
-2026-09-17: self-calibrating lazy/super-lazy margins -- instead of (or on
-top of) the static per-bucket percentile tables, maintain a small mutable
-per-(tier, material-or-army-bucket) delta, adjusted at runtime: at a low
-sample rate (~1%), let a node that would have taken the cheap exit run to
-full eval anyway, compare, and nudge the delta -- up fast on a real miss
-(swing exceeded the margin), down slowly otherwise (fast-up/slow-down,
-not symmetric AIMD). Motivated by this session's CALIBRATE_MARGIN_SAFETY
-digging: exceeded-margin nodes aren't spread uniformly across the tree,
-they cluster heavily inside a handful of "spicy" root games' subtrees
-(see the recurring N1Q5/N1B5/N1R5/... and 5rk1/1b2Q2p/... FEN families
-that kept reappearing across every guess this session, all clearly the
-same search tree a few plies apart). A single fixed percentile can't
-respond to "we're currently deep in a dangerous subtree right now"; a
-fast-reacting runtime term could -- widen in response to a real miss
-right when the search is actually inside the risky region, then decay
-back out once it's past. Two real costs before building this: (1) it
-makes eval score depend on search visitation order/history, adding to
-the non-determinism MP already introduces and complicating the kind of
-fixed-margin A/B comparison work this session did throughout; (2) it's
-its own new tuning problem (step-up size, decay rate) layered on top of
-the static-table one -- don't start building this until the question of
-whether the current 30k-position calibration pool is even large/diverse
-enough (see cross_sectional_1/2.ep_ variance check, tests/) is settled,
-so it's clear which change is causing which effect. Explicitly rejected
-alternative: hand-carving structural exceptions for specific recurring
-motifs (e.g. "queen has infiltrated near the enemy king") -- doesn't
-generalize, and is exactly the kind of special-casing that rots.
-
-2026-09-18: STILL AN OPEN QUESTION, not decided either way. Scott's
-worry re-examining this: reaction speed. At a ~1-in-10k sample rate, an
-adaptive term probably can't react fast enough inside a spicy subtree to
-matter -- by the time enough samples land there to trip the widen, the
-search has likely already made every unsound cutoff in that region
-using the too-narrow margin, so the wrong bound is already baked into
-that iteration's result before the term ever moves. Raising the sample
-rate to compensate directly fights the entire reason the lazy tiers
-exist (paying full-eval cost specifically to make the cheap tiers
-marginally safer). Separately, the complexity cost compounds with
-something already proven expensive this session: guess #1-11 (eval.c)
-showed changing one bucket's *static* margin reshapes the whole search
-tree in ways that confound attribution (see those comments' "cross-
-bucket confound" notes) -- a runtime-adaptive term makes that confound
-permanent and load-bearing every game, rather than a one-time tuning
-headache, and two engines with identical DNA could diverge on eval
-*decisions* (not just timing) run to run. Current lean: shelve it, since
-the mechanism can only close the barn door after the horse for the
-specific node that triggered it, against a permanent tax on
-determinism/reasoning-about-Eval(), for a gain the static percentile
-tables already get within ~1% of via the boring approach. Not acted on
--- left open in case reaction-speed concerns turn out to be wrong in
-practice (e.g. if "spicy subtree" actually means tens of thousands of
-nodes wide, not a handful, the sample-rate math changes).
-
 2026-09-18: build two eval.c lazy-margin variants and compare all three
 (current/baseline, "messy but fast", "super conservative") head to head.
 "Messy but fast": tune iResidualP2/98ByOwnArmy + iSuperLazyResidualP2/98-
@@ -126,52 +75,7 @@ it collides with self-play match runs -- schedule it for whenever the
 box isn't already saturated by match_play.py workers, not concurrently
 with them.
 
-2026-09-18: We should experiment with more aggressive R factors for
-nullmove at deep search depth remaining levels.
-
-2026-09-18: blunders.pgn review (typhoon vs Crafty losses). Two distinct
-loss shapes, different fixes:
-  (1) "Slow grind" -- the majority of losses. Score sits near equal for
-  many moves then drifts 0.1-0.3 pawns/move in Crafty's favor with no
-  single bad move visible, compounding to -3/-4 over 20-40 moves.
-  Annotated depths run ~10-17 ply for us vs ~18-30 for Crafty the whole
-  game, consistently -- this is the NPS/depth gap compounding over time,
-  not an eval or tactics bug. Only real fix is search efficiency (nodes
-  needed per ply) or raw nps, not eval richness -- eval doesn't look wrong
-  here, it's just arriving late.
-  (2) "Cliff" losses -- a handful of moves per game where the score drops
-  3-4 pawns in one move at a suspiciously shallow depth vs the moves
-  around it (e.g. game 3, move 20: -0.5 for several moves then
-  `20. Nc4 {-4.26/12}`, depth 12 vs neighboring 20-ply reads), mostly in
-  40/60 (fast) time controls. Looks like a time-management gap, not
-  search quality: candidate fix is EBF-based "don't start ply N+1 if
-  there's no time to finish it" (cheap, Scott: "easy") PLUS a
-  root-instability-triggered "buy more time" signal for volatile
-  positions (PV changing across iterations, last iteration failed
-  high/low, score still swinging over the last few completed depths, or
-  a narrow best-vs-second-best gap) -- recognizing "this position is
-  tactically critical" from scratch is hard/rejected, but search-
-  instability is a cheap proxy already sitting in the ID loop. Not built.
-
-2026-09-18: profile a super-lazy-always (material-only Eval() exit,
-forced unconditionally, throwaway diagnostic build a la the zero-
-selection-budget move-ordering trick -- never commit it) binary against
-Crafty at matched depth/positions to find out where our nps actually
-goes once eval is ~free. Scott's educated guess is SEE and the
-is-this-move-a-check / legal-reply-to-check generators dominate once eval
-is out of the way, not eval itself -- also suspect given the earlier
-`<=`-to-`<` SEE-skip experiment on line ~4178 (skip SEE on same-piece-
-for-same-piece captures) barely changed speed but made trees fatter
-(more nodes needed), i.e. that SEE call was doing real pruning work, not
-just costing time. Before assuming check-detection costs are irreducible,
-confirm "is this move a check" is answered via O(1) bitboard AND against
-the enemy king square from the post-move attack bitboard, not via a
-generate-and-filter pass -- Scott believes he already does this "cheaply"
-but isn't sure it's the fastest available method. Single-legal-reply-to-
-check detection is likely the genuinely unavoidable cost of the three
-(needs a real legal-move count in the child position) -- worth checking
-whether it early-exits as soon as a 2nd legal move is found rather than
-generating the full list.
+2026-09-18: blunders.pgn review (typhoon vs Crafty losses).
 
 2026-09-18: consider adding cLikelyUnstoppablePasser to the POSITION
 structure and using it to extend searches.  Just like we already do
@@ -183,3 +87,33 @@ pawn structure -- read more of that pawn power in chess book.  Work on
 the pawn eval specifically.  Maybe do eval learning on only the pawn
 hashed eval terms.
 
+2026-10-04: We need to go faster.
+ - Can we get rid of rgSquares[] completely and use bitboards everywher?
+ - Profile and speed hunt again?
+ - Eliminate SEE calls?
+ - profile a super-lazy-always (material-only Eval() exit, forced
+   unconditionally, throwaway diagnostic build a la the zero-
+   selection-budget move-ordering trick -- never commit it) binary
+   against Crafty at matched depth/positions to find out where our
+   nps actually goes once eval is ~free. Scott's educated guess is
+   SEE and the is-this-move-a-check / legal-reply-to-check generators
+   dominate once eval is out of the way, not eval itself -- also
+   suspect given the earlier `<=`-to-`<` SEE-skip experiment on line
+   ~4178 (skip SEE on same-piece- for-same-piece captures) barely
+   changed speed but made trees fatter (more nodes needed), i.e. that
+   SEE call was doing real pruning work, not just costing
+   time. Before assuming check-detection costs are irreducible,
+   confirm "is this move a check" is answered via O(1) bitboard AND
+   against the enemy king square from the post-move attack bitboard,
+   not via a generate-and-filter pass -- Scott believes he already
+   does this "cheaply" but isn't sure it's the fastest available
+   method. Single-legal-reply-to- check detection is likely the
+   genuinely unavoidable cost of the three (needs a real legal-move
+   count in the child position) -- worth checking whether it
+   early-exits as soon as a 2nd legal move is found rather than
+   generating the full list.
+
+2026-10-04: Reign in extensions the make the tree smaller.
+
+2026-10-04: Modify self-play harness to record eval score / PVs per
+move in the PGN output.