diff options
| author | Scott Gasch <[email protected]> | 2026-10-15 13:11:22 -0700 |
|---|---|---|
| committer | Scott Gasch <[email protected]> | 2026-10-15 13:11:22 -0700 |
| commit | 993b6eeb2a9ac7b659c473b6883fa53af1ad3a0b (patch) | |
| tree | c5388824dec15a7691ce9952a6eb4af66e837127 /src/docs | |
| parent | 53b380c658f828c4c0e096be2763e6b012c23abd (diff) | |
Phase A NNUE: feature-indexing spec, annotation/training pipeline, dump nnuefeatures
Builds out the NNUE Phase A prototype per docs/NNUE.md:
- src/nnuefeatures.c: standalone HalfKAv2_hm feature-index reference
implementation (new `dump nnuefeatures` command), sourced from
Stockfish's half_ka_v2_hm.{h,cpp} (sf_16 tag), including the 0x88->64
square conversion typhoon's own board representation needs.
- annotate_pgns/: sharded TWIC/self-play annotation harness
(annotate.py), position extraction, quiet-position filtering (flat
and game-shaped, plus an upstream --prefilter-quiet option),
train/holdout game-level stratified split, WDL-sigmoid K fitting,
the Python HalfKAv2_hm reference encoder (nnue_feature_spec.py) and
its bit-for-bit cross-checker against the C side
(cross_check_features.py), and a first PyTorch training pipeline
(nnue_model.py, nnue_train.py, build_training_tensors.py) -- a
256x2-32-32-1-ish net trained locally on CPU against ~1.85M quiet
TWIC positions as a pipeline/framework validation pass.
- Finish an in-progress InitMagic/InitLMRTable/InitEval ->
InitializeBitboardMagic/InitializeLMRTable/InitializeEval rename:
chess.h's prototypes hadn't been updated to match the already-renamed
definitions/call sites, leaving 4 implicit-function-declaration
warnings.
- tests/arasan.epd: new curated EPD suite.
- docs/: add the project's design docs (previously untracked).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Diffstat (limited to 'src/docs')
| -rw-r--r-- | src/docs/ANALYZE_GAME.md | 254 | ||||
| -rw-r--r-- | src/docs/EFP_LMP_MERGE.md | 234 | ||||
| -rw-r--r-- | src/docs/EXTENDED_PRUNING_DEPTH.md | 493 | ||||
| -rw-r--r-- | src/docs/MOBILITY_BITBOARD_REWRITE.md | 150 | ||||
| -rw-r--r-- | src/docs/MOVE_GUESSER.md | 285 | ||||
| -rw-r--r-- | src/docs/MOVE_ORDERING_IDEAS.md | 226 | ||||
| -rw-r--r-- | src/docs/NNUE.md | 1153 | ||||
| -rw-r--r-- | src/docs/NULLMOVE_SKIP_CALIBRATION.md | 213 | ||||
| -rw-r--r-- | src/docs/PROFILE.md | 222 | ||||
| -rw-r--r-- | src/docs/TESTING_PLAN.md | 298 | ||||
| -rw-r--r-- | src/docs/UNIFY_SEARCH.md | 406 |
11 files changed, 3934 insertions, 0 deletions
diff --git a/src/docs/ANALYZE_GAME.md b/src/docs/ANALYZE_GAME.md new file mode 100644 index 0000000..2576a1e --- /dev/null +++ b/src/docs/ANALYZE_GAME.md @@ -0,0 +1,254 @@ +# Analyzing a game from blunders.pgn + +A working playbook for pulling a specific game out of `blunders.pgn` and +figuring out *where the eval/search actually went wrong*, as opposed to +just noticing that the final result was bad. Written after a session spent +doing exactly this on the 2026-10-06 `typhoon vs Crafty-25.2` loss (a +KRP-vs-KQ endgame that looked like it had a scoring bug but turned out to +be legitimately lost, plus two real engine bugs found along the way). + +## 0. What's actually in blunders.pgn + +Each game's move comments are **only** `{score/depth time}`, e.g. +`{-5.09/34 6}` -- score in pawns from White's perspective (not side-to-move; +confirmed by the sign trending consistently negative as White loses, across +both White's and Black's own move comments), search depth reached, and +wall-clock seconds spent on that move. **There is no PV text and no engine +tag in the PGN at all** -- not `<REC>`, not `<EGTB>`, not `<#>`, nothing. +Whatever wrapper plays these self-play/vs-Crafty games strips the PV down to +just the eval number before writing the comment. If you want to see *why* +a move was chosen -- the actual PV, or whether a recognizer/tablebase hit +produced the score -- you have to replay the position through a live engine +yourself; the PGN alone can't tell you. + +Don't go looking for `<EGTB>`/`<#>` tags in old PGNs for another reason too: +those sentinels (`EGTBMOVE` plus the `RECOGN_EGTB` wiring) are new, added in +the 2026-10-06 session that produced this document. Any game recorded before +that commit landed couldn't have produced them even if the PGN format did +carry PV text. + +## 1. Pulling the game and exact FENs with python-chess + +`python3 -c "import chess"` works in this environment (python-chess is +installed) -- use it rather than hand-parsing SAN, especially for games deep +enough that move numbers and ply counts diverge from intuition. + +```python +import chess.pgn +f = open('blunders.pgn') +games = [] +g = chess.pgn.read_game(f) +while g: + games.append(g) + g = chess.pgn.read_game(f) +last = games[-1] # or games[N] for a specific game +board = last.board() +moves = list(last.mainline_moves()) +``` + +**Getting a FEN at a specific PGN move number**: ply count and PGN move +number are NOT the same thing and it's easy to get this wrong (did exactly +this once already this session -- cost a wasted round-trip). The mapping: + +```python +def ply(movenum, color): # color: 'w' or 'b' + return 2*(movenum-1) + (1 if color == 'w' else 2) +``` + +`moves[ply(69, 'w') - 1]` is White's 69th move (0-indexed list, 1-indexed +ply). Push moves up through the target ply and read `board.fen()` -- +`board.halfmove_clock` is directly available too and matters a lot (see +"fifty-move-rule proximity" below). + +To scan a whole stretch and watch halfmove_clock / material change move by +move (useful for finding exactly where a pawn/capture resets the fifty-move +counter, or where material transitions from one tablebase signature to +another): + +```python +for i, m in enumerate(moves, start=1): + before = board.copy() + board.push(m) + if lo <= i <= hi: + print(i, before.san(m), 'halfmove_clock=', board.halfmove_clock, + 'fen=', board.fen()) +``` + +## 2. Replaying a position through typhoon + +Use `--batch --command` with `setboard`, not interactive stdin piping (per +CLAUDE.md, piping commands over stdin to a live REPL doesn't reliably +complete a multi-step script). Always pass `--egtbpath` explicitly if you +want to be sure which EGTB directory is in play (the hardcoded default is +`/zscratch/egtb`, which currently has full WDL+DTZ coverage through 6-man -- +see CLAUDE.md -- but don't assume that silently when writing up findings; +state it). + +```sh +./typhoon --cpus 1 --hash 256m --egtbpath /zscratch/egtb \ + --logfile /tmp/typhoon/<name>.log \ + --batch --command "force; setboard <fen>; eval; sd <depth>; go" +``` + +- `eval` alone (no search) gives the raw static eval -- useful as a sanity + baseline before trusting a search score. +- `sd <depth>` fixes search depth; prefer this over `st <seconds>` for + reproducibility (machine load varies a lot on this shared box -- see + CLAUDE.md's ECM section). +- Watch stdout for these diagnostic lines, which tell you *how* the move was + chosen, not just what it was: + - `ONLY MOVE --> stop searching now` -- forced, no real search happened. + - `TABLEBASE MOVE --> stop searching now` -- root EGTB (DTZ) root-move + short-circuit; score will show as `PV=<EGTB>` and a flat `±NMATE`-ish + "mate" display that is **not** a real mate distance (see section 4). + - `FORCED MATE --> stop searching now` -- the iterative loop detected a + mate score at the current depth and is stopping; the printed PV comes + from `szLastPV`, which on a **root fail-high** is just the bare move + (`MoveToSan`), not a real PV walk -- see section 5, don't expect `<#>` + here. + - `DEPTH LIMIT --> stop searching now` -- ordinary completion, nothing + special. +- Grep the output/log for `tellothers` to get the final + `score, time, nps, PV=...` summary line in one place. + +## 3. Cross-checking with Crafty + +`/usr/local/bin/crafty` is available and gives a genuinely independent +second opinion -- different engine, different (or absent) EGTB usage, +different search. Use it exactly the way you'd use typhoon: + +```sh +echo "setboard <fen> +sd <depth> +go +quit" | timeout 20 /usr/local/bin/crafty 2>&1 | tail -30 +``` + +Crafty's output is far chattier (full eval component breakdown, PV with +move count, nodes/nps) -- look for the `depth ... score ... variation` +block and the `note: scores are for the white side` line (confirms sign +convention, which is directly comparable to typhoon's White-POV scores). + +**Why this matters**: if typhoon reports something wild (a huge mate-ish +score with a PV that goes nowhere, or a score frozen for 20+ plies in a row) +and Crafty reports an ordinary, non-mate score with a PV that's actually +converging, that's your signal the *typhoon* number is suspect, not that the +position itself is unusual. This exact comparison is what caught the +`ProbeEGTBRoot` DTZ-as-mate-distance bug (section 4) -- typhoon said +`-MATE4`, Crafty's independent 16-ply search said `-7.38` with no mate +anywhere in a 13-move PV. + +## 4. Things that look like bugs but are fifty-move-rule artifacts (and one that was a real bug) + +**Watch `halfmove_clock` whenever a long endgame grind produces a +suspicious score.** Syzygy tables report **WDL** (win/draw/loss) and +**DTZ** (distance to the next fifty-move-rule-*zeroing* move -- a capture or +pawn push), not DTM (distance to mate). This project's EGTB directory has +**zero `.rtbm` files** -- DTM tables were never downloaded, and Fathom's DTM +code (`probe_dtm_table`, `TB_probe_dtm`, etc.) is present in the source but +dead weight without data behind it. So: + +- A genuinely won endgame can sit at a **small DTZ** while still being many + plies from actual mate, especially once `halfmove_clock` is climbing + toward 100 (a reset becomes forced soon regardless of how far mate + actually is). +- A score that stays flat for many consecutive moves, or a score that + jumps to a huge "mate-ish" magnitude with a PV that doesn't resolve + anywhere close to that many plies later, is the signature of this -- + **not** evidence of missing/corrupt tablebase files by itself. Check + `ls /zscratch/egtb | grep <material-signature>` to rule out a genuine + coverage gap first, but if the files are present, suspect the + DTZ-vs-mate-distance conflation next. +- This was a **real, fixed bug** as of 2026-10-06: `probe.c`'s + `ProbeEGTBRoot` used to compute `*piScore = INFINITY_SCORE ∓ uDtz` and + report it through the mate-score display path -- i.e. it fabricated a + "mate in N" number out of a quantity that was never a mate distance. Fixed + to report a flat `+NMATE`/`-NMATE` (same convention `ProbeEGTB`/ + `RecognLookup` already use at interior nodes) instead. If you're + diagnosing a *pre-fix* game (anything before this fix landed -- check + `git log probe.c`), expect to see exactly this artifact and don't chase + it as a new finding. +- The **move** Fathom picks at the root is unaffected by any of this -- + `tb_probe_root()` does its own internal DTZ-based ranking before + returning, so the chosen move is already DTZ-optimal regardless of + whether the *score* we derived from it was sane. Don't second-guess the + move choice because the displayed score looked wrong; check the score + logic and the move logic as separate questions. + +## 5. Reading PV sentinels (and when you won't see them) + +`WalkPV` (`util.c`) terminates a printed PV early and appends a tag when it +hits one of five sentinel "moves" (`data.c`): `HASHMOVE` → `<TT>`, +`RECOGNMOVE` → `<REC>`, `DRAWMOVE` → `<=>`, `MATEMOVE` → `<#>`, `EGTBMOVE` → +`<EGTB>`. These mark *why the PV stopped*, not what the move was: + +- `<REC>` / `<EGTB>`: an interior-node `RecognLookup` hit resolved the + score outright. `RECOGN_EGTB` (→ `<EGTB>`) means an actual on-disk Syzygy + WDL probe succeeded; `RECOGN_EXACT` (→ `<REC>`) means one of `recogn.c`'s + hand-written classic recognizers (KPK, KBKP, etc.) fired instead. Before + 2026-10-06 these were indistinguishable in the PV (both reported + `RECOGNMOVE`) -- `RecognLookup` had an `RECOGN_EGTB` enum value sitting + unused the whole time, always returning `RECOGN_EXACT` even on a genuine + Fathom hit. If you're looking at an old log/build, `<REC>` could mean + either; `<EGTB>` specifically didn't exist yet. +- `<#>`: search found an actual forced checkmate (zero legal moves + in + check) at a PV leaf. +- `<=>`: a drawn leaf (zero legal moves, not in check == stalemate; this + sentinel is also used for recognized/forced draws elsewhere). +- **Root tablebase moves get `<EGTB>` in `szLastPV` directly** (as a literal + string, not via `WalkPV` -- `ProbeEGTBRoot`'s success path short-circuits + the whole iterative loop and never calls `WalkPV` at all for that move). + +**The big gotcha**: `PrintPV` (`util.c`) only calls the real `WalkPV` walk +-- the one that can surface any of these tags -- when the reported score is +a genuine PV node, `iAlpha < iScore < iBeta`. On a **root fail-high** +(`iScore > iBeta`, i.e. the aspiration window is still widening upward), +it falls back to printing just the bare best move via `MoveToSan`, with no +PV walk and therefore no sentinel, ever -- even if the position is a +dead-obvious mate-in-1 whose leaf genuinely wrote `MATEMOVE` two plies down. +So: **don't read "no `<#>` in the output" as "no mate was found"** -- check +whether the line says `FORCED MATE` (a fail-high-driven stop) vs. a normal +depth-complete report before concluding anything about whether the engine +actually resolved the mating line as an exact PV. + +## 6. The timer-elapsed-time gotcha + +If you see `Searched for -<huge number> seconds` in output, that's not a +real timing anomaly to chase -- or rather, it *was* a real bug (fixed +2026-10-06): `Iterate()` (`root.c`) used to stamp +`g_MoveTimer.dEndTime = SystemTimeStamp()` partway through the function, +*before* the shared `end:` label that every early-exit path (`ONLY MOVE`, +`TABLEBASE MOVE`, pre-move terminal states) jumps to via `goto end`. Those +paths skipped the stamp entirely, leaving `dEndTime` stale (0, or whatever +the previous unrelated search last set it to), which produces exactly this +kind of nonsense negative "elapsed time" when the end-of-search report +computes `dEndTime - dStartTime`. Fixed by moving the stamp to right after +the `end:` label itself. If you're diagnosing a pre-fix build and see this, +it's cosmetic (doesn't affect move choice or score), but it's also a tell +that you're looking at a root short-circuit path (tablebase or only-move), +which is useful context for *why* no real iterative search happened at that +node. + +## 7. General workflow summary + +1. Identify the game (usually "last game in the file") and get the exact + FEN at the move(s) you care about via python-chess -- don't eyeball SAN + move numbers against ply counts by hand. +2. Check `board.halfmove_clock` at that FEN before anything else if it's an + endgame -- rules out/in the fifty-move-rule class of artifact immediately. +3. Replay the FEN through `./typhoon --batch --command "force; setboard + <fen>; sd <depth>; go"`, read the diagnostic `--> stop searching now` + line to understand *which* code path produced the move, and check + `tellothers`'s `PV=` field. +4. Get a second opinion from `/usr/local/bin/crafty` on the same FEN at + comparable depth. Large disagreement (especially "my engine claims near- + mate, Crafty's PV goes nowhere near that far") is the signal to dig into + the score-computation code, not the position. +5. If something still looks wrong, read the actual source for the code path + the diagnostic line pointed at (`root.c`'s `Iterate`, `probe.c`'s + `ProbeEGTB`/`ProbeEGTBRoot`, `recogn.c`'s `RecognLookup`) rather than + guessing from behavior alone -- every bug found this way so far + (`ProbeEGTBRoot`'s DTZ-as-mate-score, the `dEndTime` placement, the + dead `RECOGN_EGTB` branch) was a straightforward, small, findable-in- + minutes fix once the actual code was in front of us; none of them + required deep speculation. diff --git a/src/docs/EFP_LMP_MERGE.md b/src/docs/EFP_LMP_MERGE.md new file mode 100644 index 0000000..e2bc27a --- /dev/null +++ b/src/docs/EFP_LMP_MERGE.md @@ -0,0 +1,234 @@ +# Retiring EFP in favor of LMP + +Status: **plan only, not started.** Supersedes item 4 ("EFP-vs-LMP +substitution experiment") in `EXTENDED_PRUNING_DEPTH.md`, which flagged this +as the next step but didn't have the concrete numbers below. Do not start +until the current uncommitted working-tree changes (presently being +evaluated against HEAD via `match_play.py`, score trending positive) are +either committed or reverted -- this work should start from a clean, +committed baseline, not stack on top of an in-flight, unresolved candidate. + +## Motivation + +EFP (`search.c`'s `fTryEFPHere`/`fThisMoveEFPPruned`, ~line 1440-1522) and +LMP (`ShouldLatePrune`, `searchsup.c`) are two separate forward-pruning +mechanisms that fire on heavily overlapping populations at remaining depth +0-2 ply, maintained independently, with no shared tuning discipline between +them. EFP uses a hand-picked margin (Heinz's paper: `VALUE_KNIGHT`/ +`VALUE_ROOK` by 2-tier depth) plus two fail-high-history vetoes (static +table, dynamic per-move). LMP uses the same underlying 5D joint fail-high +table (`g_uJointFHThousandths`, `data.c`) directly, with no margin test and +its own, differently-calibrated pair of vetoes. Goal: retire EFP entirely, +let LMP's table-driven decision govern that depth range on its own, without +losing EFP's pruning volume (tree size must not grow) and without adopting +Crafty's `break`-the-whole-node semantics (every mechanism here stays +per-move, `continue`-based -- rejected explicitly, see "What we are +deliberately not doing" below). + +## What's already been verified (no code changes made yet) + +1. **The joint table has no sentinels.** `g_uJointFHThousandths` (`data.c`) + is fully backoff-smoothed (Wilson-upper-bound, shrinkage to coarser + marginals) as of the mid-run splice documented in + `../dual_table_run/README.md` -- "No 0xffff sentinels remain." Both + `ShouldLatePrune`'s and EFP's sentinel checks are currently dead code, + not a live concern. + +2. **The table's collection was censored by EFP/LMP themselves, and that's + expected, not a flaw.** `search.c`'s `FH_TABLE_COLLECTION` instrumentation + sits textually *after* the `if (fThisMoveEFPPruned || fThisMoveLMPPruned) + { ...; continue; }` block (~line 1550-1613) during the collection run + that built this table. A move either mechanism decided to prune never + reached the collection counters. This means the table is accurate for + the population it actually saw (eligible moves that survived both + mechanisms) but has **zero data** on the population EFP has been + discarding -- not biased data, no data. There is no way to query "what + was the real FH% of EFP's historically-pruned moves" from existing logs; + it was never measured. (Corrects an earlier, imprecise claim in this + investigation that bucket[0]/[1] were "thin" there -- they aren't thin, + they're simply about a different population.) + +3. **EFP's own margin, read through our own table, is already far more + conservative than its nominal static-veto bar.** Looking up EFP's tier-1 + margin (`VALUE_KNIGHT`≈325cp, depth bucket 1, gap bucket 4) in the live + table gives a measured FH% of 0.01-0.05% across lateness buckets -- + roughly 10x *under* EFP's own `uPredictedFH < 300` (0.3%) static veto. + That veto has essentially never been the operative gate; the margin test + alone already does the real filtering, more conservatively than its own + stated threshold. + +4. **LMP's existing static ceiling already has slack to cover EFP's old + population, unmodified.** Comparing EFP's old population's real + table-measured FH% against LMP's current `uLMPStaticMaxPctThousandths[]` + ceiling by depth bucket: + + | depth bucket | LMP's current ceiling | EFP's-old-population measured FH% | + |---|---|---| + | 0 | 0.200% | 0.010% | + | 1 | 0.200% | 0.020% | + | 2 | 0.150% | 0.036% | + | 3 | 0.075% | 0.063% | + | 4 | 0.050% | 0.071% (now exceeds ceiling -- correct, by design) | + + At depth buckets 0-2 (EFP's actual operating range), LMP's own ceiling + already permits 5-20x more risk than EFP's population actually carries. + The static side of LMP should, on its own, already prune *at least* as + much as EFP did there. By depth bucket 3+, LMP's own schedule correctly + starts refusing -- that tightening-with-depth behavior is intentional, + not a gap to fix. + +5. **The real asymmetry is the DYNAMIC veto, not the static one, and this + is the mechanism most likely to cause a tree-size regression on naive + retirement.** Both EFP and LMP gate on real, per-move-identity observed + fail-high history in addition to the static table, but at wildly + different strictness: + + | | min samples | max tolerated dynamic FH% | + |---|---|---| + | EFP (`search.c`) | 5 | **10%** (`EFP_FH_PRUNE_THRESHOLD_PCT_THOUSANDTHS=10000`) | + | LMP (`searchsup.c`) | 1 | **0.001%** (`LMP_DYNAMIC_MAX_PCT_THOUSANDTHS=1`) | + + A 10,000x gap. `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS=1` is, in practice, a + "this move identity has never once failed high, for any nonzero sample + count" gate -- not a calibrated percentage. Any move identity with real, + nonzero fail-high history anywhere in the tree (common for moves tried + more than a handful of times) gets exempted from LMP pruning even when + the static table is fully confident. EFP rarely hit this problem because + its own dynamic bar (10%) almost never tripped. **Predicted outcome if + EFP is retired without also touching this constant: LMP's static side + alone is not sufficient to match EFP's old pruning volume, because this + gate silently vetoes a meaningful fraction of what the static table + would otherwise allow.** This is the paper's central, falsifiable + prediction -- see Plan step 2. + +6. **The apparent "Crafty's ply-2 margin predicts a LOWER fail-high rate + than ply-3, despite a smaller margin" oddity is a bucket-resolution + artifact, not a real inversion.** Crafty's 150cp (remaining ply 2) and + 200cp (remaining ply 3) both land in our 100cp-wide gap bucket 2. + Holding gap bucket fixed and varying depth shows the expected + monotonic-with-depth increase (0.039% -> 0.053% -> 0.089% -> 0.176% ... + at gap bucket 2, late bucket 4, dumb=0, improve=neutral) -- "more depth + remaining = more chance to recover" holds throughout; our 100cp gap + bucket is just too coarse to resolve Crafty's 50cp margin steps at the + shallow end. Not something this plan needs to fix to proceed, but worth + remembering if a future finer-grained gap axis is ever considered. + +7. **Reference points from Crafty/Stockfish, translated into our table + (dumb=0, improve=neutral, several lateness buckets), for calibration + context only -- not adopted as targets:** + - Crafty's `FP_margin[]` at remaining ply 1 (100cp) implies **0.2-3.2%** + FH% in our table depending on lateness -- the single largest + "implies more room exists" data point of this whole investigation, + concentrated at the shallowest depth and earliest move numbers. + Crafty's deeper tiers (150-400cp, ply 2-6) land close to our own + existing 0.05-0.3% range -- roughly agreeing with us already. + - Stockfish's quiet-move futility margin translates to **0.01-0.13%** + throughout -- more conservative than Crafty in our terms, not less; + consistent with SF leaning on other layered mechanisms (move-count + quiet-move skip, continuation-history pruning, SEE-based pruning) to + do work Crafty's single margin does alone. SF's own margin test is + therefore not a fair apples-to-apples number by itself. + - Neither engine's number is being proposed as a direct target -- + they're priors for where real engines have found a workable + risk/reward point, used to sanity-check, not dictate, what we test. + +## What we are deliberately not doing + +**No `break`.** Crafty `break`s its entire move loop on either its +futility or LMP check -- abandons every remaining move at that node, not +just the one under consideration. Stockfish's futility tests are `continue` +(per-move, like ours); its one break-like mechanism (`skip_quiet_moves()`) +is narrower still -- it only stops trying further *quiet* moves, captures +and checks still get considered afterward. Both EFP and LMP already use +`continue`, evaluating every move's prune/no-prune decision independently, +and that is staying exactly as-is. Nothing in this plan should introduce a +node-wide bailout of any kind. + +## Plan + +1. **Baseline.** Confirm `head_reference/` is current at a clean, committed + HEAD (rebuild if the in-flight uncommitted changes mentioned above get + committed first). Capture `sd 10` and `sn 5000000` logs for all three + curated suites (`ecm_ringers`, `ecm_confident_quick`, `ecm_hard_quick`) + if not already fresh. + +2. **Two-variant node-count experiment, before touching `match_play.py` at + all.** Build and compare two variants against the baseline, fixed `sd`, + all three curated suites, node count + first-move-cutoff-rate as the + signal (per CLAUDE.md's existing move-ordering-experiment methodology -- + measure the pick, not the game): + - **Variant A**: delete EFP (`fTryEFPHere`, `fThisMoveEFPPruned`, and its + contribution to the final prune-or-continue check) entirely. + `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS` left untouched at `1`. + **Predicted result**: tree size grows relative to baseline, confirming + finding 5 above -- the dynamic veto asymmetry is real and costs + pruning volume LMP's static side alone doesn't recover. + - **Variant B**: same EFP deletion, plus raise + `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS` toward EFP's old bar (start at + `10000` to match EFP exactly, as the cleanest single-variable swap; + narrower values can be swept later if this overshoots). + **Predicted result**: tree size lands close to or tighter than + baseline, since finding 4 already showed the static side has headroom + once the dynamic gate stops over-vetoing. + - If both predictions hold, that's strong, cheap (no `match_play.py` + games needed yet) confirmation of the mechanism before spending any + self-play budget. If either prediction is wrong, stop and re-diagnose + before proceeding -- a wrong prediction here means the mental model of + which gate is load-bearing is incomplete, and tuning further on top of + a wrong model wastes the expensive match-play stage. + +3. **`match_play.py` gate on Variant B only** (not A -- A is a diagnostic, + not a candidate; it's not expected to be an improvement on its own). + No cheap/fast elimination stage exists in this project's tooling -- + there's no shortcut cheaper than a full run, so don't plan around one. + Use step 2's node-count + fail-high-rate-by-bucket diagnostic as the + only pre-filter (lean on it harder than a minimal single-value check -- + compare across all three suites and several buckets, not just + aggregate node count, since it's the last free signal before paying + for self-play). Then commit to **one full-length, long-run + `match_play.py` game count** (matching the run already used for the + current unrelated candidate) directly on Variant B at + `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS=10000` -- don't pre-plan a sweep. + Gate on `LOWER95 >= 0.5`, not raw score, per existing project + convention. Only spend a second full run sweeping a narrower + `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS` value if this first run comes back + close-but-not-passing; if it misses badly, that's a signal the + dynamic-veto model itself is off (go to step 5), not that a nearby + value would have worked. + +4. **If Variant B wins**: this becomes the EFP-retirement commit. Remove + the now-dead EFP code paths, `EFP_FH_MIN_SAMPLES`/ + `EFP_FH_PRUNE_THRESHOLD_PCT_THOUSANDTHS` macros, and update + `ShouldLatePrune`'s docstring again to drop the "second gate, applied + after EFP" framing (it's no longer applied after anything -- it's the + only gate). Rebuild `head_reference/` as the new baseline. + +5. **If Variant B does not win** (tree size recovers per the prediction but + playing strength doesn't hold, or the dynamic-veto fix overshoots into + real tactical misses): fall back to sweeping + `LMP_DYNAMIC_MAX_PCT_THOUSANDTHS` between `1` and `10000` rather than + either extreme, same one-variable-at-a-time discipline, before + concluding EFP's margin test was catching something the table's axes + genuinely can't see (in which case EFP's own tiers should be recalibrated + against measured surprise rate instead of retired, per + `EXTENDED_PRUNING_DEPTH.md`'s original fallback plan). + +## Open items, not part of this plan's critical path + +- **Horizon-tier loosening beyond matching EFP** (bucket 0/1's static + ceiling raised past its current 200/1000, toward Crafty's own + ~1-3%-at-early-lateness risk tolerance) is a separate, later experiment -- + only worth running once EFP retirement itself is confirmed at parity. + Don't conflate "retire EFP without regressing" with "also get more + aggressive than EFP ever was" in the same test; they have different, + separable risk profiles and should be gated independently. +- **`split.c` has neither EFP nor LMP today** (confirmed: no + `fTryEFPHere`/`fThisMoveEFPPruned`/`ShouldLatePrune` references in + `split.c`). This plan doesn't change that; `UNIFY_SEARCH.md` already + tracks bringing LMP (and, after this plan, LMP-without-EFP) to the split + path as its own separate work. +- **Gap-bucket resolution** (currently 100cp-wide, per finding 6) is coarser + than Crafty's own margin schedule moves in. Not a blocker for this plan, + but flagged for whoever eventually tackles the horizon-tier-loosening + follow-on above, since a coarse bucket there would blur exactly the + region that follow-on cares about most. diff --git a/src/docs/EXTENDED_PRUNING_DEPTH.md b/src/docs/EXTENDED_PRUNING_DEPTH.md new file mode 100644 index 0000000..f916715 --- /dev/null +++ b/src/docs/EXTENDED_PRUNING_DEPTH.md @@ -0,0 +1,493 @@ +# Extended pruning depth: closing the ebf gap to Crafty/Stockfish + +## Background + +Typhoon's effective branching factor is worse than Crafty's and Stockfish's +despite having the same core forward-pruning toolkit (EFP, LMP, LMR, +nullmove, same R factor). An experiment that forced the main search loop to +abandon all remaining moves the instant one looked prune-worthy (`continue` +-> `goto end`) cut node counts at low depth but made deep searches *worse* +-- the single most informative result driving this plan, since it shows +the gap isn't about move ordering (already measured as competitive, see +the 2026-08-29 dynamic-move-ordering session) and isn't fixed by recklessness +alone. It's about margin size and depth-extent of existing mechanisms, and +possibly a mechanism we don't have at all. + +Reading both reference engines' actual source settled several questions +that would otherwise have been guesswork: + +- `~/programming/chess/crafty-25.2/search.c` + `data.c`: Crafty's + `FP_margin[16] = {0,100,150,200,250,300,400,500,...}` with `FP_depth = 7` + prunes with a flat, hand-picked margin schedule out to **6 plies of + remaining depth**, and on a prune it executes `break` -- abandons every + remaining move at that node, not just the one under consideration. No + reverse/static-null mechanism exists in Crafty; what looked like it might + be one is actually "razoring" (Kent et al.), which drops to qsearch when + *eval is far below alpha*, the opposite side of the board from what's + relevant here. Margins are tuned by whole-game cluster testing, not + per-node surprise-rate measurement. +- `~/programming/chess/stockfish/src/search.cpp`: Stockfish has **both** + mechanisms, cleanly separated, and both use `continue`/return-based + pruning rather than Crafty's `break`: + - "Step 9. Futility pruning: child node" (`eval - futilityMargin >= beta`, + no nullmove verification, depth < 19) -- a true reverse-futility/ + static-null check, evaluated once per node before the move loop even + starts. + - "Futility pruning: parent node" (`futilityValue <= alpha`, gated on + `lmrDepth < 12` where `lmrDepth` is itself adjusted by that move's own + history score) -- the same idea as Typhoon's EFP, but extended much + deeper and with a per-move, history-sensitive margin instead of a flat + depth-indexed one. + +Typhoon's own EFP (`search.c` ~line 1310) is exactly 2 tiers (remaining +depth <= 1.75 ply: margin `VALUE_KNIGHT`; <= 2.75 ply: margin `VALUE_ROOK`), +sourced from Ernst Heinz's "Scalable Search in Computer Chess" and +calibrated against measured surprise rates via `CALIBRATE_QSEARCH_FUTILITY` +(QSearch only -- this calibration harness does not currently cover main +search's EFP). It already uses `continue`, matching Stockfish's safer +choice, not Crafty's `break`. + +## Alpha-side vs. beta-side pruning: two mechanisms, not one + +Despite feeling like the same idea, EFP and reverse futility pruning (RFP) +differ enough mechanically to warrant separate experiments, even though +they are related by a satisfying piece of algebra. + +**The algebra.** In negamax, a parent's alpha is the negation of the +child's beta (`alpha_parent = -beta_child`). EFP's test at the parent -- +skip move `m` if `iEstimatedMoveValue + margin <= iAlpha_parent` -- uses +`iEstimatedMoveValue` as an approximation of `-StaticEval(child)` (the +mover's-eye view of the position after `m`, without actually making it). +Substituting: + +``` +-StaticEval(child) + margin <= iAlpha_parent + StaticEval(child) - margin >= -iAlpha_parent + StaticEval(child) - margin >= beta_child +``` + +...which is exactly Stockfish's RFP test, evaluated one ply later with a +real number in place of an estimate. So EFP-at-the-parent-with-an-estimate +and RFP-at-the-child-with-ground-truth are the same inequality, viewed from +opposite sides of one ply. That's a useful mental model, but not a reason +to implement only one of them: + +| | EFP (alpha-side) | RFP (beta-side) | +|---|---|---| +| Granularity | Per move, inside the move loop | Per node, before move generation | +| Input | Estimated child eval (`iEval + iImprovement - iHangCp`) -- approximate, no Make() | Current node's own already-computed static eval -- exact, no estimation error | +| Cost when it fires | Skips one move; loop continues | Skips the entire node, including move generation | +| What it's betting on | "Even optimistically, this specific move can't reach alpha" | "This position is already so good that no single reply drags it back below beta" | +| Typhoon status | 2 tiers, Heinz margins, `continue` | Does not exist | +| Crafty | 6 tiers, flat small margins, `break` | Does not exist (razoring is alpha-side, not this) | +| Stockfish | ~12-ply-equivalent (history-adjusted), `continue` | Exists, depth < 19, dynamic margin | + +Both are in scope. They prune different populations and at different +points in the per-node sequence (RFP can save move-generation cost +entirely; EFP cannot, since it needs the move to estimate against). + +## Why this is dangerous and needs a safety net + +Both mechanisms are forward pruning: skipping real search based on a +static/approximate signal. Typhoon's EFP margins were chosen from Heinz's +paper and calibrated after the fact against measured surprise rates +(`CALIBRATE_QSEARCH_FUTILITY`, QSearch-side only). Crafty's much more +aggressive schedule was validated only by whole-game cluster testing, which +answers "does this help in aggregate" but not "how often, and at what +cost, is a specific prune wrong." Any extension of EFP's tiers, and any new +RFP mechanism, should ship with the same evidence-calibration discipline +already used elsewhere in this codebase (QSearch futility calibration, the +2026-08-29 move-ordering fail-high-rate-by-evidence-bucket methodology): +measure the surprise rate at the proposed margin/tier/depth *before* +trusting it live, not just end-to-end solve counts on a 90-position suite +(too noisy for a single knob, per that session's own finding). + +## Eval variance and the shape of the tree + +Does the variance/distribution of `Eval()`'s output materially shape the +tree built above it, independent of whether `Eval()` is accurate on +average? Yes, and the mechanism is stronger than "noisy input, noisy +output" -- minimax search doesn't just pass leaf noise through, it +amplifies it. + +**Why.** If `Eval()` returns the true value plus independent noise at +each leaf, backing that up through alternating min/max nodes doesn't +average the noise away -- at each max node the search picks the most +favorably-noisy branch, at each min node the most unfavorably-noisy one. +The backed-up value is systematically biased relative to the true value, +and the distortion grows with branching factor and with noise variance +(the "minimax pathology" result -- Beal, Pearl, Nau). Searching deeper +helps for two separable reasons that are usually bundled together as one: +it resolves real tactics (moves bias toward truth), *and* it averages over +more terminal positions (shrinks the noise-driven distortion). The second +effect is exactly the mechanistic version of "a more accurate eval shapes +a smaller, more correct tree" floated earlier about NNUE -- not just +closer to the true value on average (bias), but plausibly smoother across +similar positions too (variance), and the two have different +consequences for search quality. + +**Where this bites every mechanism in this plan, concretely.** EFP's +margin test, LMP's gap axis, the proposed RFP test, and `iImprovement` +are all built directly on `iEval`. `iImprovement` specifically (`iEval - +eval_two_plies_ago`) is a *difference* of two noisy reads -- if the +per-call noise is roughly independent, the difference is noisier than +either input, which argues for some skepticism before trusting it as a +calibration axis (as proposed above for the new beta-side table) without +checking its own signal-to-noise ratio first. + +More importantly, the joint FH% table's own "gap" axis is computed from +`iEval` -- if `iEval` is noisy, a move's measured gap bucket is partly +noise, not signal. That's a textbook attenuation-bias setup: a noisy +covariate systematically shrinks the measured strength of a true +relationship toward zero. Which reframes a question from earlier in this +plan -- **Typhoon's margins may need to be wider than Stockfish's/ +Crafty's not because of a different risk-tolerance philosophy, but +because the thing the margin is being compared against is measured less +precisely.** That's a mechanistic, testable claim, not an editorial one. + +It also touches an assumption the "approximate the beta-side table from +the existing one" idea leaned on and that got ruled out on bucketing +grounds (see above) -- "move 1, by move ordering, is probably the best +move." Move ordering itself depends partly on eval-derived heuristics; a +noisier eval weakens that assumption independent of the bucketing problem +that already killed the shortcut. + +**What to actually measure, using instrumentation that mostly already +exists:** + +- **Eval-tier disagreement, position held fixed.** `eval.c`'s margin- + safety-swing tracking (`g_uSuperLazyMarginSwing*`, built for the lazy/ + super-lazy calibration) already measures how much the eval disagrees + with itself across tiers, bucketed by army -- a clean noise measurement + since only the eval *tier* varies, not the position. Re-read this data + with the variance question in mind rather than building new collection. +- **Sibling-eval spread at a node.** A full-move-list population analysis + (evaluate every legal move's resulting position, look at the spread) is + messier -- it conflates genuine positional differences between moves + with eval noise -- but it's informative in its own right: a node where + every move evals similarly is exactly the "tactics can equalize + anything" case flagged as the real risk in pruning deep, independent of + noise. Worth collecting, but don't conflate it with the tier- + disagreement measurement above when interpreting results. +- **Correlate either signal against measured EFP/LMP/RFP surprise rate.** + If noisier positions (by either measure) show higher surprise rates at + a fixed margin, margins should be a function of (depth, gap, noise/ + variance bucket), not just depth and gap -- a genuinely new axis + candidate for the tables in this plan, pending that correlation holding + up. + +**A genuinely new idea this surfaces, not yet anywhere in the tuning +pipeline:** `eval_tune/`'s DNA tuner fits constants to minimize squared +error against game outcomes -- a pure bias-reduction objective. Nothing in +that objective rewards a parameterization with *lower score variance +across similar positions* at equal average error, and two settings of the +same constants could plausibly achieve the same training loss with very +different volatility. If eval variance is costing tree quality the way +this reasoning suggests, the fix isn't "tune harder" on the current +objective, it's checking whether the tuner is unknowingly indifferent +between a low-variance and a high-variance solution, and adding an +explicit variance penalty to the objective if so. Lower priority than the +RFP/EFP work above -- a tuning-pipeline change, not a search change -- but +worth tracking as a follow-on, since it would improve every downstream +mechanism in this plan rather than just one of them. + +## Plan + +1. **Baseline audit.** Confirm current EFP tier count/margins and absence + of RFP in the working tree; snapshot `head_reference/` at the pre-change + commit if not already current, both `sd 10` and `sn 5000000` logs for + all three curated suites. + +2. **Map the beta-side (inverted-gap) half of the 4D joint table.** + `g_uJointFHThousandths` has only ever been populated/queried for the + alpha-side case (gap-to-alpha positive, used by EFP/LMP pruning + decisions). The beta-side case (`iEval` already above `iBeta`) is + unmapped. See "The 4D FH% table and static nullmove" below for the + dimensional analysis -- the existing table's `lateness` and `hung + material` axes are not meaningful for this question, so this is a new, + smaller table, not a re-slice of the existing one. + +3. **Reverse futility pruning (RFP) experiment.** Add a Stockfish-style + static cutoff (`iEval - margin >= iBeta`, no nullmove verification, + evaluated once per node before move generation) at shallow-to-moderate + remaining depth. Margin sourced from the new 2D table where sample + confidence supports it, a Heinz/Stockfish-style formula as fallback + elsewhere. Add a `CALIBRATE_`-style diagnostic (search anyway, log + whether the cutoff would have held) before trusting it live, matching + the existing QSearch-futility calibration pattern. + +4. **EFP-vs-LMP substitution experiment** (supersedes a plain "extend EFP's + tiers" plan -- see "EFP vs LMP: same actuary, fewer dimensions" below for + the full reasoning). Rather than hand-extending EFP's own Heinz-derived + margin schedule deeper, test whether LMP's existing, already-4D-informed + schedule can be given equivalent license at the depths EFP currently + owns (1-2 ply) and simply carried further out, retiring EFP once that's + confirmed. If it can't hold strength even with matched permissiveness, + that's real evidence EFP's margin/value test sees something the table's + current axes don't, and extending EFP's own tiers (calibrated against + measured surprise rate, not hand-picked) becomes the right fallback. + +5. **Eval-variance side study.** See "Eval variance and the shape of the + tree" below for the full reasoning -- short version: minimax doesn't + just pass eval noise through, it amplifies it, and every mechanism in + this plan (EFP, LMP, RFP, `iImprovement`) reads `iEval` directly, so + this isn't a side curiosity, it's a shared input-quality question for + the whole plan. + +6. **Validation protocol.** One variable at a time against the + `head_reference/` baseline, fixed `sd` depth (not `st`, per existing + ECM-comparison guidance), all three curated suites per change (not + just the fastest one), fail-high/surprise rate as the primary signal + on individual knobs and curated-suite solve counts as the final gate. + Rebuild `head_reference/` when a change is kept. + +## EFP vs LMP: same actuary, fewer dimensions + +Working through this live surfaced a cleaner way to see the relationship, +and it changes what "extend EFP" should even mean. + +**EFP is LMP running on 2 of its 4 axes, with a hand-picked margin standing +in for what the table would otherwise estimate directly.** Both start from +the same base eligibility gate (`CanWeReduceOrPruneMove`) and both end up +consulting `g_uJointFHThousandths` for the same move. Where they differ: + +| | EFP | LMP (`ShouldLatePrune`) | +|---|---|---| +| Gap to alpha | Explicit, via a margin test (`iEstimatedMoveValue + margin <= iAlpha`) | Implicit, as one of the table's own 4 trained axes | +| Hung material | Explicit, via `iHangCp` folded into the same margin test | Same `iHangCp`, same table axis -- identical input | +| Remaining depth | Coarse 2-tier switch (`<=1.75 ply` / `<=2.75 ply`), margin hand-picked per tier from Heinz's paper | Smooth, trained schedule (`{200,200,150,75,50,25,12,6,3,1,0}` thousandths, indexed by remaining ply) | +| Move lateness | Not consulted at all | One of the table's 4 trained axes | +| Verdict source | A margin/value threshold, independently derived, then AND-ed with the table's strictest veto tier (`uPredictedFH < 300`) | The table's own calibrated prediction against a depth-indexed threshold, plus this move-identity's dynamic history | + +So EFP doesn't disagree with LMP's calibration -- on gap and hung material +it uses the *identical* inputs, and both schedules get more conservative +(prune less readily) as remaining depth grows, which matches the +uncontroversial "more depth remaining = more chance of recovering" logic +both Crafty and Heinz build on. Measured by hand (gdb breakpoints on real +positions): EFP's working threshold (~0.3%, with 0.1% found to "kill" it) +sits right next to LMP's own shallow-depth allowance (200/1000 = 0.2%) -- +not a coincidence, the same calibrated reality reached two different ways. +The one thing EFP has that LMP doesn't is lateness-blindness: it prunes a +move regardless of how early or late it was tried, something LMP's table +lookup already accounts for. + +**Why the earlier "remove EFP, it backfired" result doesn't settle +anything on its own.** That experiment removed a pruning mechanism without +compensating the other one -- if LMP's shallow-depth threshold was left +exactly as it was (tuned to co-exist alongside EFP owning the 1-2 ply +window), removing EFP just leaves a gap neither mechanism covers, which +is a different, less informative test than "margin-based pruning vs. +table-based pruning, given equal license." It demonstrates removing a +safety net without replacing it is bad, which isn't surprising, not that +EFP's specific mechanism is what's load-bearing. + +**The fair test:** disable EFP, but first widen LMP's allowed-FH% +threshold at remaining depth 1-2 ply to be at least as permissive as EFP +was there, and let that same (already depth-and-lateness-aware) schedule +extend outward toward the depths Crafty/Stockfish prune at, tightening +further out the way the existing array already does. If strength holds, +LMP genuinely subsumes EFP, and "go deeper" becomes "keep widening LMP's +schedule outward," not "invent new EFP tiers." If it doesn't hold even +with matched permissiveness, that's real evidence EFP's margin/value test +catches something the table's current 4 axes can't see on their own -- +worth identifying what, specifically, before extending EFP's own tiers as +the fallback plan (calibrated against measured surprise rate the way +`CALIBRATE_QSEARCH_FUTILITY` already does, not hand-picked the way Heinz's +original schedule was). + +The real, unresolved risk either way -- independent of which mechanism +wins this test -- is the one check-evade-capture/tactical-volatility point +raises: a full-ply search below a "hopeless-looking" move genuinely can +even out the score the deeper it's allowed to run, and that volatility is +exactly what's being traded away for node count the further from the +horizon pruning reaches. Crafty's results say that trade pays off in +aggregate even with real misses; whether Typhoon's calibration machinery +(the table, or LMP's dynamic per-move history) lets the same trade be made +more selectively -- pruning deep only where confidence is actually high, +rather than uniformly the way Crafty's flat schedule does -- is the thing +this whole plan is trying to find out. + +## The 4D FH% table and static nullmove + +### What the existing table answers, and why it doesn't directly apply + +`g_uJointFHThousandths[uShipD][uDumb][uGap][uLate]` (`data.c`) currently +answers: *given remaining depth, hung-material estimate, gap-to-alpha, and +move lateness, what's the (Wilson-bound, pessimistic) percent chance this +specific move raises alpha?* It's built and consumed entirely on the +alpha side (`search.c`'s `uRawGapB = GapCpBucket(iAlpha - iEval)`, used by +`ShouldLatePrune` and to veto EFP). + +A static-nullmove/RFP decision is a different question: *given remaining +depth and how far `iEval` already clears `iBeta`, what's the chance that a +full search of this node would still come in at or above beta?* Two of the +four existing axes don't transfer: + +- **Lateness is meaningless.** RFP is evaluated once per node, before any + move is even selected -- there is no "move number" to bucket by. +- **Hung material (the `uDumb`/`iHangCp` axis) is move-specific** (it's an + estimate of what a *particular* move gives away). RFP has no candidate + move in hand yet when it fires, so this axis doesn't apply either. (It + may still be useful as a *node*-level signal -- see below -- but it would + need to be redefined as "is anything hanging in the current position," + not "what does this move give away," which is a different computation.) + +So this needs a new, smaller table: **FH% predicted by (gap above beta, +remaining depth)** -- the direct beta-side analogue of the existing +alpha-side table, not a re-slice of it. + +### What `return` value RFP should use + +Three choices, in order of how much they trust the margin: + +- **`iBeta`** -- the conservative choice, and what Stockfish effectively + approximates (`(661 * beta + 363 * eval) / 1024`, a weighted blend + biased toward `beta` rather than the raw `eval`). Safe: never returns a + score less plausible than the bound being tested. + Stockfish's blend is worth copying outright rather than re-deriving -- + it's already tuned. +- **`iEval - margin`** (the actual tested quantity) -- more information- + preserving for any caller that inspects the returned score (e.g. a + parent's own improving/stand-pat logic), but injects the raw margin's + uncertainty into the score, not just the boolean pass/fail. +- **A genuine average/expected value**, if the new table's confidence is + high enough to support one (i.e., calibrate not just "prune or don't" + but "what score should this cell predict" directly from the sampled + distribution). This is the most ambitious option and should only be + attempted after the simpler two have been validated -- it's a + significantly larger calibration effort (predicting a continuous score, + not a prune/don't-prune boolean) and shouldn't gate the first working + version of RFP. + +Start with Stockfish's blended-toward-beta return value; revisit once the +new table has enough sample volume to ask whether a sharper value is +justified. + +### Other signals worth considering for the new table + +- **`iImprovement`** (already computed and used by EFP's own estimate) -- + whether static eval has been trending up across recent plies at this + node. Stockfish's own RFP explicitly folds in an `improving`/ + `opponentWorsening` correction to the margin; this is likely the single + highest-value addition beyond the two base axes, and it's already + computed elsewhere in the search, so it's cheap to add as a bucket + dimension. +- **Side-to-move's own king safety / piece activity**, as a coarse proxy + for "how much tactical headroom does the opponent have to claw back the + gap" -- more speculative, more expensive, lower priority than + `iImprovement`. +- **Whether the position is a capture-heavy/tactical node** (recent-ply + volatility) vs. quiet -- RFP is riskiest exactly where the position is + about to get sharp; a coarse "was the last move a capture/check" bucket + may be cheap evidence for that without needing a full tactical-detector. +- **Explicitly not** `iHangCp`/lateness, per above -- they don't have a + node-level equivalent that means the same thing they mean on the alpha + side. + +Recommended starting axis set for the new table: **(gap-above-beta bucket, +remaining-depth bucket, improving bucket)** -- a 3D table, one dimension +richer than the bare 2D minimum, reusing `iImprovement` since it's already +on hand at every relevant call site. Hold the capture/check-volatility +signal and king-safety proxy as "measure once collection infra exists, add +only if the bucket shows a real split" rather than building them in up +front. + +### Two more axes, found by re-examining what's already in the codebase + +**Opponent trend, not just own trend.** `iImprovement` (`search.c:728`, +`iEval - ctx->sPlyInfo[ctx->uPly-2].iEval`) is the *mover's own* eval trend +over the last full move, and it's already folded into EFP's per-move +estimate as one scalar (`iEstimatedMoveValue = iEval + iImprovement - +iHangCp`) -- which means the existing alpha-side joint table never sees it +as an independent axis; a big gap with a flat trend and a smaller gap with +a strongly-improving trend currently bucket identically. Worth adding +`iImprovement` as a genuine 5th axis to the *existing* alpha-side table, +not just the new one -- same collection machinery, one more dimension. +Stockfish's RFP keeps this as **two** separate signals, not one: +`improving` (mover's own trend, what Typhoon already computes) and +`opponentWorsening` (the opponent's own trend, computed from the +opponent's eval history, not a sign-flip of the mover's). Both are +candidates for the new table's axis list -- don't conflate them into one +"negate and reuse" axis. + +**Hung material can transfer, but needs a new consumer of a signal that +already exists live -- correcting an earlier wrong turn in this +discussion.** `_EvalLookForDanger` (`eval.c:6194`) really is `#if 0`'d out +and dead, but it is *not* the only hung-piece detector in `Eval()`. +`_EvalTrappedPieces` (`eval.c`, live, runs every full eval for both +colors) already detects exactly "nowhere safe to move to, and currently +attacked" -- the `bbStillSafe == 0` branch combined with +`_WhoControlsSquareFast(pos, c) == xColor`. When it fires for the side +*not* currently on move, it calls `RecordEnprisePieceAtPly(ctx, +ctx->uPly - 1, c)`. + +Tracing where that's actually consumed today: `generate.c:5342` -- +`s += (mv.cFrom == cEnprise) * (PIECE_VALUE(mv.pMoved) / 4)`, read via +`FindEnprisePiece`. That's a **flee bonus**: it rewards moving a piece +*away from* the square where this check found it endangered. So the live +detector is wired in the same direction as the disabled routine's +docstring (self-protective -- "the mover is about to lose something"), +not the "the opponent left me a free capture" direction RFP would want. +It's a real, already-computed fact (both colors, every full eval) with +only one consumer today (move-ordering's flee nudge) -- using it for RFP +means adding a *second* consumer that reads the same recorded fact in the +other direction (does the opponent, who just moved, have a trapped+ +attacked piece sitting there for the side now on move to take), not +writing new detection code and not re-enabling anything disabled. + +This still matters for RFP the way originally argued: `Eval()` is purely +static/positional and never searches captures, so a free piece sitting on +the board isn't reflected in `iEval` at all. If the existing detector +already flagged one for the side about to move, the true value of the +position is higher than `iEval` says, and RFP should fire more +confidently (accept a smaller gap), not less. The cost question from the +original (disabled) routine -- full-eval-only, judged not worth paying +unconditionally -- doesn't block this, since the detection is already +being paid for and computed; the only new cost is reading it. + +Updated axis set to collect for the new table, reflecting both additions: +**(gap-above-beta, remaining-depth, improving, opponent-has-hung-piece)** +-- four axes. The last one reads `_EvalTrappedPieces`'s already-recorded +`cEnprise`/`cTrapped` data from a new angle (see above) rather than +computing anything new, but it's still only populated when a full eval +actually ran at that node -- gate collection on eval tier, don't force a +full eval just to populate the table during data collection. + +### Checked and ruled out: approximating the beta-side table from the existing one + +Tempting shortcut considered: skip collecting new data entirely and +approximate the beta-side FH% by querying the *existing* alpha-side table +at `(depth, hung=0, gap = iBeta - iEval, lateness=1)` -- i.e. treat the +first (best-ordered) move's own "does it raise alpha" cell, with the sign +of the gap flipped, as a proxy for "does the node's best move clear beta." +The algebra behind it is sound (it's the same mirror-image relationship +discussed above, evaluated at the same node instead of one ply over). + +It doesn't work, though, and the reason is checkable rather than a matter +of judgment: `searchsup.c`'s `GapCpBucket` collapses every non-positive +gap into a single bucket 0 -- + +```c +ULONG GapCpBucket(SCORE iGapCp) +{ + if (iGapCp <= 0) { return 0; } + ... +} +``` + +-- by design, since from the alpha side, once a move already clears alpha +there's nothing left to decide, so the table was never built to +distinguish "cleared by 10cp" from "cleared by 900cp" in that regime. A +beta-side lookup via `iBeta - iEval` (always <= 0 whenever RFP would even +be considered) always lands in that same bucket regardless of magnitude -- +exactly the graded information an RFP margin needs, lost before it was +ever shipped. It's not recoverable from the raw collection logs either: +`search.c`'s own instrumentation (`uRawGapB = GapCpBucket(iAlpha - iEval)`, +`search.c:1275`) applies the identical collapse at collection time, not +just at baking/shipping time, so the magnitude was discarded at the source. + +Conclusion: a genuinely new collection run on the positive side of this +boundary is required; it can't be shortcut from data already on hand. The +one thing worth keeping from this idea: once the new table exists, check +that it roughly agrees with the old table's bucket-0 values near the +gap-is-zero boundary, as a cheap cross-validation between the two tables. diff --git a/src/docs/MOBILITY_BITBOARD_REWRITE.md b/src/docs/MOBILITY_BITBOARD_REWRITE.md new file mode 100644 index 0000000..51bb910 --- /dev/null +++ b/src/docs/MOBILITY_BITBOARD_REWRITE.md @@ -0,0 +1,150 @@ +# Replacing the loopy mobility chain-walks with single-probe magic bitboards + +## Motivation + +`_EvalBishop`/`_EvalRook`/`_EvalQueen` currently compute mobility (and the +king-safety x-ray bitboards) via an unbounded `for (;;)` chain walk that +repeatedly recomputes `_RookAttacksBB`/`_BishopAttacksBB` with newly- +discovered blockers excluded, to handle batteries (friendly R/Q stacked +behind each other) and x-ray-through-high-value-enemy-pieces. Crafty does +this with one magic-bitboard probe and a popcount. Goal: match that for +speed while keeping the signal that actually matters for eval quality. +Approximate parity, not exact parity -- see the tradeoffs below, all +explicitly agreed rather than accidental. + +## What's being thrown away, deliberately + +1. **Battery mobility/connectivity through more than one hop.** Today, a + rook stacked three deep behind two friendly rooks on an open file gets + mobility and connected-rook credit for every square and every blocker + in the stack. After this change, each rook only sees its own first + blocker. The squares/credit "lost" at the near rook are not actually + lost from eval overall -- the rook(s) further out credit those same + squares on their own pass. Confirmed via the king-safety-xray analysis: + the only case where continuing past a friendly blocker added + information nothing else already had was the enemy-GE case below. +2. **Mobility credit for squares beyond an enemy queen/king (or, for + bishop, enemy rook) that the piece is directly attacking.** This was + never "real mobility" in the legal-move sense anyway -- a rook cannot + physically move past a piece it would capture in one ply. Dropping it + is a correctness fix that happens to fall out of the simplification, + not a loss. (Confirmed, no objection -- see open question 6, closed.) +3. **The general multi-layer x-ray bitboard** (`bbMinorXrayAttacks`/ + `bbRookXrayAttacks`/`bbQueenXrayAttacks`), replaced by a narrower, + cheaper mechanism (below) that preserves the one case that matters: + rook/bishop/queen pointed at the enemy king (or its ring) through + exactly one enemy queen/king (or, for bishop, rook) blocker. + +## What's being preserved + +- The existing "real mobility" masks (`bbUnsafeForRook`, `bbUnsafeForMinor`, + `bbUnsafeForQueen` -- squares where a cheaper enemy piece would recapture + for free) are untouched. These already encode "real mobility" exactly as + wanted; nothing new needed here. +- The deferred low-mobility re-derivation in `_EvalTrappedPieces` is + untouched -- it only consumes a `bbMobility` snapshot and + `pos->bbKingAttacks[xColor]`, neither of which cares how the snapshot + was computed. +- The king-ring x-ray signal for `_EvalKingSafety` (the skewer/pin-adjacent + "rook/bishop/queen threatens through the enemy queen/king" case) is + preserved, just computed more cheaply (below). +- Blocking semantics for the x-ray check exactly match what the chain walk + does today: any pawn or minor piece fully blocks (no special-casing of + rammed vs. transient pawns -- that would have been *new* behavior, not + preserved behavior, and we're explicitly not adding it). Friendly R/Q and + the piece's enemy-GE set (queen/king for rook and queen; rook/queen/king + for bishop) are the only things that don't block. + +## Per-piece-type design + +### Bishop / Rook (mobility) + +``` +bbAttack = _{Bishop,Rook}AttacksBB(c, pos->bbOccupied); +bbMobility = (bbAttack & ~bbOccupied & ~bbUnsafe) + | (bbAttack & bbEnemySame) + | (bbAttack & bbEnemyGE) + | (bbAttack & bbEnemyLE & ~bbUnsafe); // bishop's transient-pawn case folds in here +uTotalMobility = FastCountBits(bbMobility); +pos->bb{Minor,Rook}Attacks[uColor] |= bbAttack; +``` + +One magic probe, no loop. Same masks as today, just applied to the single +first-blocker attack set instead of a chain-extended one. + +### Queen (mobility) + +Two probes (rook-direction + bishop-direction), same mask logic, no +per-probe chain: + +``` +bbAttack = _RookAttacksBB(c, occ) | _BishopAttacksBB(c, occ); +bbMobility = (bbAttack & ~occ & ~bbUnsafeForQueen) + | (bbAttack & bbEnemyGE) + | (bbAttack & bbEnemyLE & ~bbUnsafeForQueen); +pos->bbQueenAttacks[uColor] |= bbAttack; +``` + +### Connected-rook bonus + +Direct-attack-only, no chain: + +``` +bbConnected = bbAttack & pos->bbPieces[uColor][ROOK]; +// fire EVAL_TERM per bit, horiz/vert exactly as today (RANK(c)==RANK(cBlocker)) +``` + +### King-ring x-ray (replaces bbXXXXrayAttacks' chain-walk population) + +Computed inside each piece's own eval pass (not moved into +`_EvalKingSafety` -- that function's consumer code at eval.c:5216-5248 is +unchanged), gated so the common case (no enemy GE piece directly attacked) +costs nothing beyond an AND against a bitboard already in hand: + +``` +bbSeenEnemyGE = bbAttack & bbEnemyGE; // already computed above for mobility -- free +bbXray = 0; +while (bbSeenEnemyGE) { + sq = pop a bit; + bbXray |= _{Bishop,Rook}AttacksBB(c, occ & ~BBSQUARE[sq]); +} +pos->bb{Minor,Rook,Queen}XrayAttacks[uColor] |= bbXray; +``` + +- Rook: `bbEnemyGE` = enemy queen | enemy king -- at most 2 extra probes, + almost always 0. +- Bishop: `bbEnemyGE` = enemy rook | queen | king -- at most 3 extra probes, + almost always 0. +- Queen: same as rook, applied per axis. +- One probe per GE piece directly attacked is sufficient -- if removing + that single blocker reveals the king/ring, we're done; we never need a + second layer, since the only thing consumed downstream is ring-square + membership, not an arbitrarily deep battery. +- No `g_VectorDelta`/ring-square enumeration needed anywhere -- the gate is + "was this enemy GE square already in my direct attack bitboard," which is + free, not "check every ring square against every enemy piece," which + would have been the expensive shape. + +## Order / plumbing + +No change to eval call order (pawns -> knights/bishops -> rooks -> queens +-> king, per `Eval()`'s existing sequence) and no change to where results +land on `POSITION` (`bbMinorAttacks`/`bbRookAttacks`/`bbQueenAttacks`/ +`bb{Minor,Rook,Queen}XrayAttacks`, all `[2]`-indexed by color, already +there). Knight needs no change -- it was never loopy to begin with +(single-hop `g_KnightAttacksBB` + mask). + +## Validation plan + +- `gmake TEST=1` + `debug_smoke_test.sh` first (asserts on mobility bounds + like `ASSERT(uTotalMobility <= 14)` still need to hold). +- Then the three curated suites (`ecm_ringers`/`ecm_confident_quick`/ + `ecm_hard_quick`) at fixed `sd`, compared against `head_reference/` -- + expect small, explainable deltas (battery/overcount cases), not a cliff. + If solve counts move more than noise-level, suspect the x-ray gating + logic (an enemy GE piece not being re-detected) before suspecting the + mobility masks (those are a straight port). +- `eval_tune/`'s DNA re-tune is not required before landing this (mobility + *tables* themselves are unchanged, only how the bitboard feeding them is + computed), but worth a sanity check with `evaldna dump` before/after on + a few batteries/pin positions by hand if anything looks off. diff --git a/src/docs/MOVE_GUESSER.md b/src/docs/MOVE_GUESSER.md new file mode 100644 index 0000000..35fd3ca --- /dev/null +++ b/src/docs/MOVE_GUESSER.md @@ -0,0 +1,285 @@ +# Move-prediction ("policy") model -- design notes + +Status: design converged enough to start implementation. Not yet begun. + +## Idea + +Train a small model to predict, given a position, how likely each legal +move is to be the move a strong human player would choose (the +Lc0/Maia-chess "policy head" / human-move-prediction pattern). Candidate +uses: +- Move-ordering hint at the root (`root.c`) -- reorder/bias toward + plausible moves, cheap and low-risk: root is called once per search, + over ~20-40 moves, so cost is a non-issue regardless of model size. +- Possibly extend to the top 1-2 plies of internal search + (`generate.c`'s `_ScoreAllMoves`), not deeper. Cost there is + bounded by how many ply-2 nodes alpha-beta actually visits (root + branching factor many, not root-branching-factor^2), which is cheap + enough for a linear model but rules out anything heavier. +- Outright root move pruning/reduction based on low policy confidence -- + explicitly flagged as the higher-risk version of this idea. A + human-implausible move can still be objectively best (that's part of + how engines out-play strong humans), so a hard prune here risks + capping search quality below what plain search would find. Start with + move-ordering use only; treat pruning as a later, carefully-gated + experiment, not a given. +- Every-node use is explicitly ruled out -- far too expensive given + `eval.c`'s existing sub-microsecond-per-node budget. + +## Model choice + +**Linear (logistic regression) model over handcrafted move features, +not a neural net, for the first version.** Rationale, by comparison +against the alternatives considered: + +| | Linear/logistic (chosen) | Small MLP / NNUE-accumulator head | Conv policy head (AlphaZero/Lc0-style) | +|---|---|---|---| +| Params | hundreds | tens of thousands-~1M | millions+ | +| Root-only inference cost | microseconds total | tens of µs | hundreds of µs-ms | +| Training difficulty | trivial (sklearn, minutes) | moderate (small PyTorch job) | a real project (this is Maia's actual scope) | +| C port | dot product | matrix-vector + activation (same shape as NNUE forward pass) | not realistic hand-rolled, no ML framework at inference | + +Conv-style board embeddings are off the table entirely for this codebase +(no ML framework at inference). An MLP on an NNUE-style board embedding +is a plausible *second* iteration once NNUE's accumulator exists (same +input representation, different output head) but isn't a dependency for +starting now. + +**Don't encode raw move identity as one-hot from/to-square (4096 +categories)** -- a linear model can't generalize across squares that +way (no shared weight helps "knight e4-f6" inform "knight e5-f7"); use +relative/derived features instead (see below). Raw square one-hot only +makes sense once there's a conv/attention architecture to generalize +across squares, which is explicitly not this version. + +## Label and training objective + +Label is categorical ("which legal move was played"), not a per-move +score -- this does **not** require search or eval annotation at all, +unlike the NNUE eval work. The label is extracted directly from PGN +move text. + +**Variable candidate-set problem**: every position has a different +number of legal moves, so there's no fixed class space to hand to a +standard multiclass classifier (sklearn's `multi_class` machinery +assumes a fixed shared label set). Fix: flatten to one training row per +**(position, candidate move)** pair, not one row per position: +- Each legal move at a position becomes its own row with that move's + feature vector. +- Binary label: 1 if this move is the one actually played, 0 otherwise. +- Train one shared binary logistic regression (`class_weight='balanced'` + to handle the ~1-in-30-40 positive rate) across all rows from all + positions pooled together. +- At inference: score every legal move at the *current* position with + the one trained weight vector, then rank/softmax across just that + position's candidates. + +This is the standard "pointwise learning-to-rank" formulation and is +directly compatible with plain `sklearn.linear_model.LogisticRegression` +-- no custom loss, no variable-length output layer needed. + +**Top-k falls out for free, no multi-label training needed.** A +softmax/logistic score is already a full relative-ranking signal over +all legal moves at a position; `argsort` the per-move scores and take +the top 3-4. No architecture change, no different label format -- +exactly the single-label training above already produces this. Note +these are *relative* ranking scores, not calibrated probabilities -- +fine for move-ordering, don't assume "70% confidence" literally means +70% empirical accuracy without a separate calibration check. + +**Held-out validation**: hold aside a slice of positions (not used in +training) to measure model quality -- top-k coverage against the +held-out human moves is the first-pass metric (top-1 "exact match" is +the wrong metric for a move-ordering use case; top-3/top-5 coverage is +what actually matters, since the goal is getting the eventually-best +move searched early, not predicting humans exactly). Once that looks +reasonable, validate the actual move-ordering-hint use case via +`match_play.py`'s existing head-to-head gate (same `LOWER95 >= 0.5` +threshold used for DNA tuning), not just offline accuracy. + +## Features + +### Global (position-level, shared across all candidate moves at a position) +- material balance +- game phase, derived from material-left +- side to move +- castling rights, both sides +- king safety, both sides -- reuse `eval.c`'s `iKingSafety[]`, which is + already computed per side after a full `Eval()` call; no new + computation needed +- in-check flag + +### Per-move (recomputed per candidate move) +Built from the `MOVE` union (`chess.h`): `cFrom`, `cTo`, `pMoved`, +`pCaptured`, `pPromoted`, `bvFlags`. + +- `pMoved` (piece type) +- SEE value, if a capture +- gives-check flag (computed the same way `generate.c` does, via + `ExposesCheck`/attack detection after making the move) +- relative square features instead of raw `cFrom`/`cTo`: file delta, + rank delta, distance, and a centrality-table lookup on `cTo` (reuse + whatever `eval.c` already has for this) +- **explicit disambiguated move-category, one-hot, not the raw + `bvFlags` bit**: `{NORMAL, CAPTURE, DOUBLE_PAWN_PUSH, EN_PASSANT, + PROMOTION, PROMOTION_CAPTURE, CASTLE_KINGSIDE, CASTLE_QUEENSIDE}`. + The raw "special" bit's meaning is context-dependent on `pMoved` + (special+king=castle, special+pawn=double-jump/en-passant/promotion + depending on `pPromoted`/`pCaptured`), so it must be decoded into + explicit disjoint categories *before* being handed to a linear model + -- feeding the raw bit directly would conflate castle-ness with + promotion-ness, etc. Decode mirrors the existing `IS_CASTLE`/ + `IS_PROMOTION`/`IS_ENPASSANT`/`IS_DOUBLE_JUMP` macros exactly; castle + side (kingside/queenside) isn't in the union bits at all and must be + derived from `cTo` (g1/g8 vs. c1/c8). + +**Explicitly excluded**: dynamic move-ordering signals (history table, +countermove table, continuation-history evidence). These don't exist +in this training setting at all -- they're accumulated *during* a +single search and reset between searches, so there's no equivalent +"has this move class worked before" signal minable from a static human +PGN position with no search running. (Could become relevant again if +ever training on self-play search traces instead of human games, but +that's a different, much more expensive data source -- not planned.) + +## Training data pipeline + +Driven by a persistent `typhoon` subprocess via the xboard protocol -- +the same `Engine` class pattern `tune_eval_dna.py` and `match_play.py` +already use (one long-lived process, many `setboard`/command round +trips over stdin/stdout, not one process spawned per position -- the +per-call overhead that pattern avoids is exactly what the earlier +`EvalCommand` persistent-context fix was about). + +**New engine command, `movefeatures`**: given the current position +(after `setboard`), generates the legal move list and dumps the full +feature vector for every legal move via `Trace()` (same style as +`evaldna dump`). Critically, this calls the *same* feature-building +code that real move-ordering will eventually call -- see "Shared +feature code" below -- so there is only ever one implementation of the +`MOVE`-union decode logic, never a second hand-ported copy in Python +that can silently drift from `chess.h` if the union's bit layout ever +changes. + +Python driver loop: `setboard <fen>` / `movefeatures` / parse stdout +into feature rows, across the full training position set, against one +subprocess. Source positions: `../pgn/lichess_elite_*.pgn` and +`../pgn/twic_filtered.pgn`, both directly usable as (position, +move-played) pairs with no search/eval annotation needed. + +**Label extraction correctness check**: for each extracted row, verify +the label-matched move, replayed, produces the same resulting FEN as +the PGN's next recorded position -- catches silent label corruption +from castling-notation/en-passant/promotion-parsing mismatches early, +before training on volume. + +## Deployment in C (no Python/ML framework at inference) + +Logistic regression's output is just a weight vector + bias (dozens of +floats) -- far simpler than NNUE's eventual quantization/loader needs. +Export `sklearn`'s `.coef_`/`.intercept_` as a flat array, baked +directly into a C source file the same way `bake_dna.py` writes tuned +values into `eval.c`'s initializers (simplest option given how small +this is; a runtime-loadable file, `evaldna`-style, is a fallback if +retraining turns out to be frequent enough that recompiling each time +is annoying). + +**Shared feature code is the real risk, not the weight format.** There +must be exactly one function, e.g. `BuildMoveFeatures(ctx, move) -> +float[N]`, called by both the `movefeatures` dump command (training +data generation) and the real move-ordering caller (`root.c`, and +`generate.c` if extended to ply-2). Two independent implementations of +this will eventually drift -- the same bug class this project's own +history has already hit more than once (the `command.c` eval-caching +bugs, the killer-table backfill collision). Inference is then: for +each legal move, call `BuildMoveFeatures`, dot-product against the +baked weight array, add bias, rank/softmax across this position's +legal moves. + +**Guard against silent drift between retrains**: give the baked +feature-weight array a length/version check against +`BuildMoveFeatures`'s actual feature count (same defensive pattern as +`bake_dna.py`'s line-count-mismatch abort), so a future feature-vector +change that forgets to retrain fails loudly at build/startup instead of +silently scoring garbage. + +## Staged implementation plan + +Each stage should produce something checkable before moving to the +next -- don't bundle these into one big change. + +1. **`BuildMoveFeatures` + `movefeatures` command.** Write the shared + feature-building function in C (global features first -- material, + phase, side to move, castling rights, `iKingSafety[]` -- then the + per-move features, including the disambiguated move-category + decode). Wire it into a new `movefeatures` command that dumps every + legal move's feature vector via `Trace()`, driven by `setboard`. + Checkable output: by hand, confirm a few known positions (a castle + available, an en-passant-capturable position, a promotion) produce + the expected one-hot move-category for each special move. + +2. **Python extraction driver.** Persistent-subprocess `Engine` loop + (reuse the `tune_eval_dna.py` pattern) over `lichess_elite_*.pgn` / + `twic_filtered.pgn`: `setboard` + `movefeatures` per position, parse + stdout into `(features, label)` rows. Run the replay-verification + check (label-matched move, replayed, matches the PGN's next FEN) on + a sample before trusting volume. Checkable output: a cached rows + file (parquet/csv) plus a verification-pass rate printed at the end. + +3. **Train + offline validation.** Flatten to per-(position, move) rows + with binary labels, `sklearn.linear_model.LogisticRegression` with + `class_weight='balanced'`, train/held-out split. Checkable output: a + top-1/top-3/top-5 coverage number on the held-out slice. Stop here + and reassess if top-3 coverage looks weak before building any C + deployment path -- no point wiring up inference for a model that + doesn't predict anything. + +4. **Export + C inference wiring, root only.** Bake `.coef_`/ + `.intercept_` into a C array (`bake_dna.py`-style, with a + length/version guard against `BuildMoveFeatures`'s feature count). + Add the scoring loop to `root.c`: for each root move, call + `BuildMoveFeatures`, dot-product, rank. Gate this behind a flag so it + can be toggled off cleanly, not wired in live as the only ordering + path on the first pass. + +5. **Head-to-head validation via `match_play.py`.** Candidate = root + ordering using the model's ranking; baseline = current root + ordering. Gate on `LOWER95 >= 0.5`, same as DNA tuning. This is the + real test -- offline top-k accuracy from stage 3 is a necessary + check, not a sufficient one, since move-ordering quality and human- + move-prediction accuracy aren't the same thing. + +6. **If stage 5 clears the gate: extend to ply-2** in + `generate.c`/`_ScoreAllMoves`, same model, same `BuildMoveFeatures` + call, no architecture change -- just a second call site. Re-run the + `match_play.py` gate for this extension separately; a win at root + doesn't guarantee a win at ply-2 (different node population, deeper + in a quiescent-ish part of the tree). + +7. **Out of scope until the above lands**: root-level pruning based on + low policy confidence (the explicitly higher-risk use case), and the + NNUE-accumulator-based second iteration described in "Future + direction" below. + +## Future direction (not needed to start) + +Once NNUE's accumulator exists (see `NNUE.md`), a second iteration +could replace the handcrafted global+per-move feature vector with a +learned board embedding (same accumulator, second output head) -- +this trades interpretability (today's linear weights are individually +auditable, the way eval constants are) for potentially higher +accuracy. Not a dependency of the first version; revisit after NNUE +lands. + +## Open questions, not yet resolved + +- Exact `BuildMoveFeatures` signature and feature-ordering convention, + shared between the dump command and the real move-ordering caller -- + deferred until implementation starts. +- Whether/how far to extend beyond root into ply-2 internal search -- + cost looks affordable per the analysis above but not yet measured + against real search statistics (how many ply-2 nodes actually survive + alpha-beta pruning in practice). +- Calibration -- not needed for ranking/move-ordering use, but would be + needed if a future use case wants literal confidence thresholds (e.g. + for the higher-risk pruning idea). diff --git a/src/docs/MOVE_ORDERING_IDEAS.md b/src/docs/MOVE_ORDERING_IDEAS.md new file mode 100644 index 0000000..5c65934 --- /dev/null +++ b/src/docs/MOVE_ORDERING_IDEAS.md @@ -0,0 +1,226 @@ +# Move ordering experiment backlog + +Goal: a leaner tree (fewer nodes for the same search quality) via move-ordering +changes, not eval or pruning changes. Methodology: per CLAUDE.md's "Dynamic +move ordering experiments" section -- measure fixed-`sd` node count / +first-move-cutoff-rate on the three curated suites (`ecm_ringers`, +`ecm_confident_quick`, `ecm_hard_quick`) as the primary signal, cross-checked +against a larger `cross_sectional_200k.ep_` sample at fixed `sn` as a tertiary +aggregate sanity check. Avoid adding throwaway per-node instrumentation code +to the tree; prefer compile-time diagnostic swaps (like the old +zero-selection-budget trick) that are never committed, or just the suite-level +before/after comparison. + +Status key: TODO / IN PROGRESS / DONE (kept/discarded) / DROPPED (not pursuing) + +## Queue + +2. **TODO -- Qsearch check generation.** Crafty's `quiesce.c` calls + `GenerateChecks()` (movgen.c) to extend qsearch's first ply with + non-capture checking moves (direct + discovered, via attack-set + bitboard tricks), not just captures. Confirmed (per CLAUDE.md) our + QSearch is capture-only. This is a search-shape change more than a pure + ordering change -- re-confirm current QSearch behavior in search.c + before building, since CLAUDE.md notes may be stale. + +5. **TODO -- Countermove fallback key (coarse, empty-slot-only).** Today + the countermove table is keyed by the *exact* previous move; if that + exact key's slots are empty/unevidenced, no countermove signal fires at + all for that node. Add a second, coarser table keyed by + `[piece-type][to-square]` of the previous move, consulted *only* when + the exact-match slots don't fire -- a genuine fallback, not a + competing signal. Different from the earlier "scaled continuation- + history" experiment (flat across a 256x range), which tested coarse + keying as an always-active competing signal, conflating key-coarseness + with "always on." This isolates key-coarseness alone, active only in + the exact-match's current blind spot. + +## Dropped + +6. **DROPPED -- Flee-to-safety revival.** `FLEE_BONUS`/the commented-out + `cEnprise`-match code in `movesup.c`'s `SelectBestWithHistory` (and + the matching `RETIRED` comment in `generate.c`) was dead -- retired + after the unconditional GOOD_MOVE-tier version it replaced was shown + (evidence-calibration, pre-dates this backlog) to perform at the + plain-leftover baseline for the vast majority of its population. + + Reran with real evidence-bucket resolution (`MOVE_ORDERING_EVIDENCE_ + COLLECTION`, reusing `CM_EVIDENCE_BUCKETS`/`_CMEvidenceBucket` -- + same evidence scale as countermove, no butterfly-only workaround + needed since this legitimately reuses the ply-1-keyed continuation + table unlike move-pair/recapture): `evidence >= 10000` hit 56.32%/ + 62.84%/68.23% FH (ecm_ringers/ecm_confident_quick/ecm_hard_quick) + vs. a ~1.35-1.54% unprivileged-leftover baseline -- comparable to + or exceeding countermove's own calibrated bar. A real signal the + prior session's cruder zero/nonzero split never isolated. + + Implemented as a `GOOD_MOVE`-tier escape at that threshold, same + shape as `COUNTERMOVE_TIER_BONUS`. Result: **no measurable effect + either way** -- 0.00%/-0.05%/0.00% node delta at sd10, zero solve + diffs on any of the three suites, sn10M depth/cutoff flat. Not a + magnitude-tuning problem like move-pair/recapture had -- the + qualifying population (en-prise origin AND evidence >= 10000) was + just too rare (2,444-9,209 tries across an entire suite run, vs. + countermove's tens-to-hundreds-of-thousands) for any mechanism + acting on it to move the aggregate needle. `u64FleeTries/FH` and + the scoring/precompute code were fully reverted. + +4. **DROPPED -- Recapture bonus.** A capture landing on the same square + the opponent's last move landed on (`mv.cTo == mvLast.cTo`), nudged + within the winning/even-capture tier (`generate.c`'s `_ScoreAllMoves`, + `s >= 0` branch) rather than escaping it. + + Built the comparison harness properly this time: `RECAPTURE_BUCKETS` + (2: SEE<0 losing, SEE>=0 even/winning) plus a parallel + `u64CaptureBaselineTries/FH` logging *every* capture (not just + recaptures) at the same SEE-sign buckets, so the recapture rate + could be read against an apples-to-apples baseline instead of in + isolation -- exactly the comparison move-pair's harness lacked at + first. Real signal: at SEE >= 0, recapture FH% ran 64-65% vs. a + 49-52% all-captures baseline, consistent across all three curated + suites, on samples of 300K-4.4M tries. + + Two magnitudes tried as a same-tier nudge, neither produced a + tree-size win: + - `RECAPTURE_BONUS = 50` (first guess, scaled like FLEE_BONUS's + old magnitude): -0.74%/+5.04%/-0.45% nodes (mixed), but a + **confirmed real tactical loss** -- ECM.735 (see `../tests/ECM.md`'s + Legit table: "HEAD's own line is clean and solid"), on + `ecm_hard_quick` (17->14/90 solved). Root cause, found by + actually computing the surrounding arithmetic rather than + guessing: the branch's real MVV/LVA differentiator -- + `PIECE_VALUE_OVER_100(captured) - PIECE_VALUE_OVER_100(mover)`, + pawn=1/knight,bishop=3/rook=5/queen=9 -- only spans ~112-128 + (a ~16-unit band) across *every* capture combination. 50 is + over 3x that whole band -- not a tiebreak, a near-dominant + override capable of vaulting a recapture ahead of an + objectively higher-MVV/LVA non-recapture capture at the same + node, which is exactly what cost ECM.735. + - `RECAPTURE_BONUS = 4` (rescaled inside the ~16-unit band): + fixed the regression (ECM.735 recovered, 17/90 again; the + hard_quick loss/gain pair that remained, ECM.491/ECM.443, are + both independently classified Flaky in `ECM.md` -- net real + change zero) but still no node-count win: +2.23%/+7.95%/-0.81%, + worse on `ecm_confident_quick` than the broken 50 value was. + sn10M depth-reached flat to within noise at both magnitudes + (14.76 vs. flat-history-only's 14.83), same as every move-pair + variant. + + Conclusion: same shape as move-pair -- a real, well-calibrated FH% + edge that MVV/LVA's existing ordering apparently already captures + most of, with no tried mechanism converting the residual signal + into a leaner tree. `RECAPTURE_BUCKETS`/`u64RecaptureTries/FH`/ + `u64CaptureBaselineTries/FH` and the scoring-branch code were fully + reverted; `MOVE_ORDERING_EVIDENCE_COLLECTION` infrastructure kept. + + **Methodology note for next time**: cross-referencing actual solve + losses/gains against `../tests/ECM.md`'s per-position Legit/Flaky + classifications (rather than reading raw suite-level solve-count + deltas) is what separated "real tactical cost" (ECM.735) from "suite + noise" (ECM.491/443/496) here -- worth doing by default for any + future candidate's solve-count changes, not just when something + looks surprising. + +3. **DROPPED -- Ply-2 move-pair table.** Crafty maintains a discrete + 2-slot table (`move_pair`, history.c) keyed on the *side's own* move + two plies back (`curmv[ply-2]`), tried in `next.c` right after the + countermove phase. Implemented as `mvMovePair` (same shape/indexing + as `mvCounter`), evidence-gated same as countermove. + + First attempt (borrowed countermove's `g_HistoryCounters + + g_ContinuationHistory` evidence sum verbatim) measured a complete + no-op: identical node counts to flat-history-only across all three + curated suites at sd10, despite the table itself populating and + predicting fine (55-70% raw hit rate). Root cause: `g_ContinuationHistory` + is only ever written keyed by the move *one* ply back -- there's no + table anywhere keyed by two-plies-back, so the borrowed slot read + unrelated noise. Fixed to use butterfly history (`g_HistoryCounters`) + alone. + + Built a dedicated evidence-bucket harness (`MOVE_ORDERING_EVIDENCE_ + COLLECTION` build flag, independent of `PERF_COUNTERS` per later + request; `MP_EVIDENCE_BUCKETS`/`_MPEvidenceBucket`, denser floors + than countermove's own since butterfly-only evidence is capped at + `HISTORY_MAX`) and calibrated a real threshold from it: top band + (evidence >= 6000) hit 53.41%/64.43%/66.23% FH (ecm_ringers/ + ecm_confident_quick/ecm_hard_quick) on large samples (26K/177K/271K + tries) -- comparable to or exceeding countermove's own calibrated + bar. A real, strong, well-sampled signal. + + Three mechanisms tried to act on that calibrated signal, none + produced a tree-size win at sd10 across the three curated suites: + - GOOD_MOVE-tier escape (same pattern as countermove's own + promotion): +1.6%/+1.4%/+1.5% nodes -- small but consistent + regression across all three suites. + - Same-tier nudge at FLEE_BONUS's old magnitude (100000, a + dominant override far exceeding the leftover pool's actual + range): +19.8%/+4.9%/+2.1% nodes -- worse than the tier escape. + A dominant override apparently inherits the escape's downside + (jumping ahead of comparably-good moves) without its upside + (EFP/pruning exemption). + - Proportionate same-tier nudge (2000, scaled to compete with + rather than override PSQT+history+continuation): mixed, + net flat-to-negative (+20.5%/+2.0%/-0.11%) -- `ecm_hard_quick` + went essentially neutral, but `ecm_ringers` (smallest, noisiest + suite) got *worse* than every other variant, suggesting it + reacts badly to this kind of reordering regardless of magnitude. + sn10M depth-reached on the 30-position cross-sectional sample + (`/tmp/typhoon/smoke30.ep_`) stayed flat to within noise (14.72-14.83 + vs. flat-history-only's 14.83) across every variant. + + Conclusion: `SelectBestWithHistory`'s existing PSQT+history+ + continuation scoring apparently already captures most of what this + signal offers; no mechanism tried converts the calibrated FH% + advantage into fewer nodes. `mvMovePair` and its evidence-bucket + harness were fully reverted (table, update function, nudge/tier + code, bucket counters/print block) -- re-derive from git history if + revisited with a different approach. The general + `MOVE_ORDERING_EVIDENCE_COLLECTION` infrastructure (flag, `MOI_INC` + macro, countermove's own bucket harness) was kept -- reusable for + calibrating the next candidate (recapture, flee-to-safety revival). + +## Done + +1. **DONE (kept) -- Flat (non-depth-scaled) history increment.** Replaced + `g_HistoryCounters`/`g_ContinuationHistory`'s old `(depth+1)^2`-scaled + growth with Crafty's flat/saturating update (`HISTORY_MAX`/ + `HISTORY_UPDATE_SHIFT`, chess.h; `_Increment`/`_DecrementMoveHistoryCounter` + and continuation counterparts, dynamic.c) -- depth-independent on both + growth and decay. Measured against a fresh `fbdcb9d` HEAD baseline + (`head_reference/`, refreshed same session): + - sd10, curated suites: node count -18.9% (`ecm_confident_quick`), + -6.6% (`ecm_hard_quick`), +18.9% (`ecm_ringers`, smallest/noisiest -- + first-move cutoff rate barely moved there, 91.918% -> 91.914%, so + likely a re-search artifact not a real regression); first-move + cutoff rate up +0.18pp / +0.16pp on the two larger suites. + - sn10M, 30-position random sample from `cross_sectional_1.ep_` + (seed 42, `/tmp/typhoon/smoke30.ep_`): avg deepest iteration + 14.52 -> 14.83 ply (+0.31), candidate reached equal-or-deeper depth + on 27/29 parsed positions, never shallower by more than 1 ply; + first-move cutoff rate 89.724% -> 89.967%. + Three independent signals (sd10 node count, sd10/sn10M cutoff rate, + sn10M depth-reached) all agree in direction. Kept, not yet committed. + +## Already covered / not pursuing + +- **History malus (decrement on moves tried before the fail-high move).** + Already implemented -- `UpdateDynamicMoveOrdering` (dynamic.c) calls + `_DecrementMoveHistoryCounter`/`_DecrementContinuationCounter` for every + quiet move considered before the one that beat alpha. Not a gap. +- Crafty's `movgen.c` itself (pure move generation -- captures, checks, + evasions, noncaptures, pin detection) contains no ordering/scoring logic; + the real ordering machinery is `next.c` (phase FSM) + `history.c` (killer/ + countermove/move-pair/history updates). + +## Protocol reminder + +- One variable at a time, against a fixed reference build (see + `head_reference/` convention) -- never "working tree a few hours ago." +- `sd` (fixed depth), not `sn`/`st`, for "does this shrink the tree" + questions -- matches what all five ideas above are actually asking. +- Read node-count and first-move-cutoff-rate off the three curated suites, + not solve-count alone (too noisy at 90-ish positions per CLAUDE.md). +- `cross_sectional_200k.ep_` at fixed `sn` as a larger tertiary cross-check, + not the primary signal, and not on a 100-position sample (too small for + an aggregate depth-at-fixed-nodes metric) -- use at least a few thousand + positions if run. diff --git a/src/docs/NNUE.md b/src/docs/NNUE.md new file mode 100644 index 0000000..0ab46ec --- /dev/null +++ b/src/docs/NNUE.md @@ -0,0 +1,1153 @@ +# Replacing Eval() with an NNUE evaluator + +Status: planning / discussion only. No code written yet. This document +records the decisions made so far and the open items that still need +resolving before implementation starts. See `MOVE_GUESSER.md` for the +related-but-separate move-prediction ("policy") model idea, which is +tracked independently and is not a dependency of this work. + +## Why + +Current `Eval()` is hand-tuned (see `eval_tune/`'s DNA tuning pipeline, +622/879 ECM after a first tuning pass vs 606/879 baseline). NNUE-style +learned evaluation is the next large step up in evaluation quality, +at the cost of a meaningfully larger implementation + retraining project. + +## Architecture decisions made so far + +- **Feature set: HalfKAv2 (with the `_hm` horizontal-mirroring variant as + the likely final choice)**, not HalfKP and not a custom-invented + encoding. HalfKA includes the kings themselves as features (unlike + HalfKP, which excludes them), which captures king-vs-king spatial + relationships -- material for endgame/mating-net precision. `_hm` + roughly halves parameter count by exploiting left-right board symmetry. + Chose a known, documented format over inventing one specifically so + existing reference trainers/tooling (e.g. nnue-pytorch) and existing + write-ups are usable as a sanity check, rather than debugging both a + novel architecture and a novel implementation simultaneously. +- **King bucketing: use HalfKAv2_hm's standard bucketing scheme as-is**, + not a custom bucket count. (Background: bucketing groups king squares + so sub-tables of weights are shared across squares in the same bucket + -- fewer buckets means less king-position nuance but faster training + convergence per bucket and a smaller table; more buckets is the + opposite tradeoff. Not worth re-deriving when a settled default + exists.) +- **Net topology: start from a known, previously-validated small + architecture** (the kind of 256x2-32-32-1-ish shape used in early + nnue-pytorch default configs) rather than choosing dimensions from + scratch. Revisit sizing only after a first working net exists. +- **Output convention: single scalar, side-to-move-relative**, matching + current `Eval()`'s convention. Achieved the standard way: two + king-relative accumulator halves (one per color's perspective) are + concatenated in an order determined by side-to-move (own-perspective + half first) before the hidden layers, so the net's one scalar output + is side-to-move-relative by construction. Explicitly rejected a + two-output (white-score, black-score) design -- that reintroduces a + combination-rule-design problem the standard trick already avoids, + and we already committed to following known designs over inventing + our own. +- **SIMD target: AVX2 + FMA.** Confirmed on the actual build box (AMD + Ryzen 9 3900X, FreeBSD, clang 19.1.7): `clang -march=native -dM -E -` + reports `__AVX2__`, `__FMA__`, `__SSE4_1__`/`__SSE4_2__` defined, no + AVX512 (Zen 2 doesn't have it). Quantization/SIMD inference code + should target AVX2 as the baseline, not assume AVX512 availability. +- **Quantization: int16 for the (sparse) input/accumulator layer, int8 + for the dense hidden layers**, matching why Stockfish's own + implementation splits it this way -- the input layer's cost is + dominated by sparse incremental updates (only ~a handful of features + change per move), where int16's SIMD width cost is small, while the + dense later layers benefit more from int8's 2x SIMD lane packing. + Narrower quantization = faster but lossier; this split is the + standard compromise. Needs a quantization-aware scaling pass per + layer during export, not just truncate-and-ship. +- **Lazy evaluation / fast first-pass filter: reuse a cheap linear + subcomponent of the SAME trained net (its PSQT/embedding-sum term, + no hidden-layer nonlinearity), not a second, separately-trained + smaller net.** Rationale: a separately-trained small net would need + its own calibration against alpha/beta margins, and that calibration + would almost certainly disagree with the full net's (nonlinear, + WDL-sigmoid-derived) output scale -- exactly the problem raised about + mixing material-cp lazy skips with a WDL-calibrated net output (see + below). A cheap *partial evaluation* of the one trained net lives in + the same output space as the full net by construction, so there is + no second calibration curve to reconcile. Only fall through to the + full forward pass (hidden layers) when the cheap linear term is + inconclusive relative to alpha/beta. + - Open experiment, explicitly not decided against: a super-lazy + *material-only* pre-filter ahead of even the cheap linear net + term, purely for search speed. Known risk: current `Eval()`'s + super-lazy/lazy/full three-tier system works because all three + tiers share units (centipawns, roughly linearly related to + material). A material-only prefilter ahead of an NNUE net does + NOT have that guarantee once the net's output has gone through + WDL-sigmoid calibration -- a position could look "way above beta" + in raw material and still not clearly fail high in the net's + (compressed, nonlinear) output space. Worth trying, worth + measuring carefully (fail-high-rate-at-the-margin style + instrumentation, not just aggregate solve counts -- see + CLAUDE.md's "Dynamic move ordering experiments" methodology), + explicitly allowed to fail and be reverted. +- **Training data score space: train against win/draw/loss outcomes via + a sigmoid-squashed WDL-style target**, not raw centipawn regression, + extending the same sigmoid-squared-error approach `tune_eval_dna.py` + already uses. Rationale: raw centipawn MSE misweights the loss (a + +500-vs-+700 error matters far less to the actual game outcome than a + +20-vs-+80 error, but unweighted centipawn MSE penalizes them + similarly) -- WDL-style targets are naturally calibrated to what + actually matters (winning the game) and match the training signal + already available from game outcomes. Requires a calibrated + inverse-sigmoid conversion back to centipawns for use inside + alpha-beta search (search needs additive, mate-aware scores, not raw + probabilities) -- that conversion constant needs its own calibration + pass and is a likely source of subtle bugs if done carelessly. +- **Exact scores (mate distance, EGTB/recognizer hits) are never run + through the WDL-sigmoid conversion, in either direction.** These are + game-theoretically exact, not noisy evidence to fit a sigmoid against. + The net itself never needs to learn "is this a mate" -- that detection + stays entirely in the search/TB-probe layer exactly as it does today; + the net is only ever consulted at ordinary, non-exact nodes. This is a + training-data-labeling concern (don't mix exact-value positions into + the sigmoid-fit loss, or into material-lazy-skip calibration), not an + architecture concern. + - **Resolved**: `<TT>`/`<EGTB>`/`<REC>` are sentinel pseudo-moves + (`HASHMOVE`, `EGTBMOVE`, `RECOGNMOVE` in `chess.h`) inserted into + the PV by `UpdatePV()` (called from `search.c` at the hash-hit, + recognizer, and EGTB-probe sites) to record provenance of the line + that follows. Distinction that matters for training labels: + **`<EGTB>`/`<REC>` are terminal and exact** -- the node's value + literally *is* that recognizer/tablebase result, so a position + whose PV ends in one of these tags should have its score excluded + from the sigmoid-fit WDL loss (treated as ground truth, not noisy + evidence). **`<TT>` is NOT an exactness marker** -- it means a + hash hit occurred and the remainder of the printed PV was + reconstructed by walking the hash table's stored continuation, + which can be a fail-high/fail-low *bound*, not necessarily an + exact score. A `<TT>` appearing anywhere in a PV should be treated + as an ordinary search score for labeling purposes, not specially + excluded. + +## Accumulator maintenance design + +- The NNUE input state lives as a **persistent "current" blob per + thread** (analogous to `POSITION`'s always-current board state), not a + recomputed-per-call structure, stored in `SEARCHER_THREAD_CONTEXT` so + each `MP=1` helper thread has its own copy. +- **Normal (non-king) moves**: MakeMove incrementally adds the changed + feature deltas into the live accumulator; UnmakeMove incrementally + subtracts the same deltas back out. This is correct without any extra + bookkeeping because search's Make/Unmake calls are already properly + nested (a recursive call stack) -- UnmakeMove always undoes exactly the + delta the matching MakeMove applied. +- **King moves**: every feature in HalfKA is king-relative, so a king + move invalidates the entire accumulator, not just a delta. Handling: + **a per-ply backup**, stored in the existing `PLY_INFO sPlyInfo[MAX_PLY_PER_SEARCH+1]` + array already present in `SEARCHER_THREAD_CONTEXT` (`chess.h`) -- + add a field there rather than a new parallel array, following the + same pattern as `PLY_INFO`'s existing sig-stamped, conditionally-valid + fields (`u64FullEvalSig`, `iKingScore`). + - On a king MakeMove: memcpy live accumulator -> `sPlyInfo[uPly]`'s + backup slot, then recompute live from scratch. + - On that same king move's UnmakeMove: memcpy the backup slot back + into live (cheaper than recomputing from scratch -- a sequential + copy of a ~1-2KB blob vs. ~30 per-piece weight-table lookups/adds + across both perspectives). + - **Must be indexed by ply, not a single shared backup slot** -- king + moves can nest (either side's king can move more than once within + a single search line before any of those moves is unmade), and a + single shared slot would silently get overwritten by a second, + deeper king move before the first one's UnmakeMove runs, producing + a wrong-but-undetected restore with no assertion to catch it. Per- + ply indexing (bounded by `MAX_PLY_PER_SEARCH`, in practice far + fewer slots actually used) avoids this by construction. +- **Null move**: moves no piece, so it is a true no-op for the + accumulator -- confirm this explicitly in the implementation rather + than assuming it falls out automatically. +- **Audit required before implementation**: confirm nothing in the + current search bypasses paired MakeMove/UnmakeMove to mutate position + state directly (e.g. any bulk board-state copy used by aspiration + re-search, singular extension, or the `MP=1` split-search machinery -- + see the recent `SearchOneMove`/`SPLIT_INFO` refactor). Any such path + would desync the accumulator silently and is exactly the kind of bug + that would not show up until much later. +- **Performance follow-up, not required for correctness**: eager + accumulator maintenance on every MakeMove wastes work at nodes that + get pruned by a cheap lazy filter before ever consulting the net (see + the super-lazy-prefilter experiment above). A lazy/deferred + accumulator scheme (record "what moved" cheaply per ply, only replay + deltas forward from the nearest materialized ancestor when a node + actually needs a net evaluation) is the standard answer if profiling + shows this matters. Defer until there's a working eager version to + measure against. + +## Breaking external dependencies on Eval()'s internal signals + +`cTrapped`/`cEnprise` (and any other signal currently read outside +`Eval()` by move ordering -- hung-piece-escape logic, `FLEE_BONUS`, the +countermove-table evidence gating from the 2026-08-29 move-ordering +session) must be decoupled from `Eval()`'s computation entirely before +`Eval()` can be replaced, since NNUE has no equivalent intermediate +signals to expose. + +- **Enumeration done, 2026-10-15** (grepped `cTrapped`/`pTrapped`/ + `cEnprise`/`pEnprise` across every live `.c`/`.h` file -- `save/` and + `changes/` hold old snapshots, not live code, and were excluded; then + verified consumers by call site, not just by grep match, since one + memorable-sounding candidate turned out to be dead): + - **`FLEE_BONUS` (the 2026-08-29 session's leftover-move en-prise + nudge, `generate.c:4673`'s doc comment) is dead code, not a live + consumer** -- `chess.h:2949` says so explicitly ("separately dead + code in movesup.c's SelectBestWithHistory (commented out) -- + retired after the GOOD_MOVE-tier version it replaced was shown... + to perform at the plain-leftover baseline"); confirmed in + `movesup.c`, only a commented-out declaration remains + (`movesup.c:1439`). Likewise, `search.c:184-194`'s null-move- + calibration-harness comment confirms en-prise/trapped state *was* + tried as an axis of the joint fail-high prediction table in an + earlier (2026-09-24) revision, found "not a safety signal at all" + for that purpose, and explicitly is **not** bucketed in the + current revision. Neither of these is a real dependency today. + - **Consumers that ARE live** (read-only, via `dynamic.c`'s accessor + functions -- none need to change regardless of who produces the + data): + 1. `generate.c:5342` (QSearch-specific move ordering, a different + function from the dead `FLEE_BONUS` one): nudges a move whose + origin square is the cached `cEnprise` hint, applied only on + top of a move SEE has already classified as winning/even -- + SEE runs first, the hint is a secondary nudge. + 2. **`EstimateHangingMaterialCp`** (`searchsup.c:246-413`, called + from `searchsup.c:1826`/`1905`) -- feeds the "dumb move" axis + of the live joint fail-high prediction table (the LMR/reduction + soundness machinery; see CLAUDE.md's "LMR sd11 vs sn10M + protocol" and search.c:242's "live, in-progress collection run + as of this writing"). Structured as a cascade of checks, most + of them bitboard-based, falling through to a `cTrapped` + half-credit hint only as the last resort (`searchsup.c:403`) -- + see the dedicated finding below, since 3 of its 5 checks turned + out to share `_EvalTrappedPieces`'s dependency, not just the + final fallback. + 3. `ValueOfMaterialInTroubleDespiteMove` (`dynamic.c:619`, called + once, from `search.c:2237`) -- a **binary QSearch stand-pat + gate** ("material in danger -> don't let this side stand pat"), + not an FH%-bucket. Reads `cEnprise` (two-slot) and `cTrapped` + (one-slot) together into one `iInTrouble` value, with no + bitboard check of its own -- trusts the cached hint directly, + validated only by an eval-signature match. + - **`EstimateHangingMaterialCp` finding, 2026-10-15**: its 5 checks + split 2 independent / 3 Eval()-entangled, and the entanglement is + bigger than just the `cTrapped` fallback: + 1. Pawn-attacks-destination -- independent (static attack tables + + raw pawn bitboard). + 2. Enemy-king-adjacency -- independent (static tables + `see.c`'s + on-demand `_WhoAttacksSquareBB`, the same on-demand + board-geometry lookup SEE itself uses -- not an eval-cached + bitboard). + 3. Minor-attacks-destination -- **depends on + `bbMinorAttacksSnapshot`**, which `search.c:606-609` populates + straight from `pos->bbMinorAttacks`, **gated on + `fParentAttackTablesAreFresh`** (`search.c:602-603`: true only + if the parent node's `Eval()` call was a full eval, not lazy). + 4. Rook-attacks-destination -- same dependency, `bbRookAttacks`. + 5. The `cTrapped` half-credit fallback (what prompted this + check) -- depends on `_EvalTrappedPieces`, same as `cTrapped`'s + producer below. + **So the real dependency here isn't just the `cTrapped` fallback + -- it's `bbMinorAttacks`/`bbRookAttacks` themselves** -- but + **this is not a blocker, and not a new condition an eval swap + introduces**: `fParentAttackTablesAreFresh` is already false + today on every node where the parent's `Eval()` call took a + lazy/super-lazy exit (today's normal three-tier behavior, not a + hypothetical) -- `EstimateHangingMaterialCp` already falls back to + just its 2 independent checks on those nodes, routinely, right + now. An NNUE swap just makes that already-live degraded path the + permanent one, not a newly-broken one. The actual effect, per the + function's own return-value contract (cp-loss-or-zero, consumed + only through `HangCpBucket`): losing checks 3-5 means some moves + that would have bucketed as "a little dumb" instead bucket as "not + dumb" (bucket 0) -- a **precision/recall shift in the joint + fail-high prediction table's "dumb move" axis, used for pruning/ + reduction/node-type decisions, not a correctness bug.** Whether + this matters enough to rebuild is an empirical question, not an + a-priori blocker: measure what fraction of real search nodes + already hit the degraded (no-fresh-tables) path today, and how + much predictive power the joint table actually loses if that + fraction goes to 100%, before deciding whether building the + independent bitboard pass is worth it versus just accepting the + less-pessimistic FH% estimate this already-tolerated degraded mode + produces. + - **Producers -- two genuinely different cases, not one**, which + changes the decoupling cost estimate: + 1. **`cEnprise`'s cheap source is already fully decoupled from + `Eval()` today**: `searchsup.c:3032`'s null-move-refutation + path (`RecordEnprisePieceAtPly`, inside + `_VerifyNullMove`/nullmove handling) flags a piece en prise + purely from *the move that refuted a null-move search* + (`mv.pCaptured`) -- no eval attack-table dependency at all, + pure search/move-list logic. This half needs **no change** to + survive an eval swap. + 2. **`cTrapped`'s only source, and `cEnprise`'s *other* source, + are both inside `eval.c`'s `_EvalTrappedPieces`** + (`eval.c:6188`), and this one is genuinely entangled with + `Eval()`'s own term-by-term computation, not cleanly + separable by just moving a function: it reads + `pos->bbPawnAttacks`/`bbMinorAttacks`/`bbRookAttacks`/ + `bbQueenAttacks`/`bbKingAttacks` (built incrementally by + earlier per-piece-type eval passes) and + `pos->cTrapped[uColor][]`/`pos->bbTrappedEscapes[uColor][]` + (a *nomination* list populated earlier still, during + low-mobility-piece detection, specifically deferred there + because full attack information isn't available until later + passes finish -- see `_EvalBishop`'s "carefully" comment). + `_WhoControlsSquareFast` (used both here and at the dead + `_EvalLookForDanger`, `#if 0`'d out) also wants those same + attack bitboards. None of this exists independently of + `Eval()` running its normal term-by-term pass today. + - **Revised estimate, 2026-10-15 -- not a blocker.** `cEnprise`'s + null-move-refutation source is free (already independent) with no + caveats. Everything routed through `_EvalTrappedPieces` -- + `cTrapped`, `cEnprise`'s other source, and 3 of + `EstimateHangingMaterialCp`'s 5 checks -- degrades gracefully + rather than breaking: an eval swap just makes permanent a + low-information fallback path (`fParentAttackTablesAreFresh == + FALSE`) that already fires routinely today on lazy/super-lazy + eval exits. Net effect is a less-pessimistic "dumb move" axis in + the joint fail-high prediction table -- a tuning/precision + question, not correctness. **Decision: do not build the + independent bitboard pass speculatively.** Ship the eval swap + first with these signals simply going permanently degraded, then + measure with the same joint-table instrumentation already in + `search.c` whether the prediction table's actual pruning/reduction + behavior suffers enough to be worth recovering. Revisit building a + slimmed `bbMinorAttacks`/`bbRookAttacks`-plus-trapped-piece pass + only if that measurement says so -- not before. +- Preferred approach, revised: **no independent tactical-signals pass + is planned up front.** The earlier plan here (build a slimmed + bitboard pass before the eval swap, to protect `cTrapped`/`cEnprise` + consumers) is deferred per the decision above -- it's optional, + data-gated follow-up work, not a precondition for replacing + `Eval()`. What *does* still apply: the 2026-08-29 move-ordering + session's countermove-table evidence gating is unaffected by any of + this (doesn't touch `cTrapped`/`cEnprise` at all); `FLEE_BONUS` + itself was already dead code before this investigation started, so + there was never anything of its to protect. The null-move- + refutation `cEnprise` source needs no changes at all, regardless. + +## Feature-indexing parity (Python trainer vs. C inference) + +The single most likely way to silently wreck this project: the Python +training code computes HalfKAv2 feature indices one way, the C inference +code computes them a different way (square-numbering off-by-one, +different king-bucket formula, different perspective ordering), and the +net trains with a perfectly reasonable loss curve but performs randomly +in the engine, because the trained weights are being read back out +against the wrong feature slots. + +- Write one canonical feature-indexing spec (square numbering, + king-bucket formula, per-perspective feature ordering, HalfKAv2_hm's + mirroring rule) **once**, in this document or a linked one, before + either the Python encoder or the C encoder is written. +- Before any real training run: build a small cross-checker that dumps + the feature vector for N known test positions from both the Python + encoder and a standalone C test harness and diffs them bit-for-bit. + Do this before training, not after a training run produces confusing + results. +- Confirm the output sign/perspective convention (side-to-move-relative, + per the architecture decision above) the same way -- a sign flip only + manifests as "plays like it wants to lose," which is a much harder bug + to localize after the fact than a direct feature-vector diff. + +## Canonical HalfKAv2_hm feature-indexing spec (2026-10-15) + +Per the "known, documented format" architecture decision above, this +is **not a reinvented scheme** -- it is Stockfish's own `HalfKAv2_hm` +feature set, pinned to a specific historical implementation +(`src/nnue/features/half_ka_v2_hm.{h,cpp}` at git tag `sf_16`, fetched +directly from `official-stockfish/Stockfish` 2026-10-15, quoted +verbatim below) rather than re-derived from memory or a secondary +description. Current Stockfish `master` has moved on to a different, +newer feature set entirely (`k32q2`/`pp_3wide`/`full_threats`) -- do +not fetch "latest Stockfish" expecting to find `HalfKAv2_hm` there. + +**Square numbering (Stockfish side, the convention both the Python +encoder and this spec's formulas use)**: `Square = file + 8*rank`, +0-indexed, `SQ_A1 = 0` ... `SQ_H8 = 63` (`a1..h1` = 0-7, `a2..h2` = +8-15, ..., `a8..h8` = 56-63). + +**typhoon's own square numbering is different and must be converted**: +typhoon uses a 0x88 mailbox representation (`chess.h`): `A1 = 0x70`, +`RANK(c) = 8 - (c >> 4)` (1-indexed rank), `FILE(c) = c & 0x0F` +(0-indexed file). The C encoder must convert every typhoon `SQUARE` +to the Stockfish-style 0-63 index before applying any formula below: +`sq64 = FILE(c) + 8 * (RANK(c) - 1)`. Getting this one line wrong +(e.g. an off-by-one on `RANK(c) - 1`, or forgetting typhoon's rank +numbering runs opposite to its own nibble ordering) is exactly the +class of silent bug this whole spec exists to prevent -- it must be +the first thing the cross-checker below verifies, in isolation, +before testing any king-bucket/orientation logic that builds on top +of it. + +**Piece-square plane index** (`PieceSquareIndex`, verbatim from +`half_ka_v2_hm.h`) -- `SQUARE_NB = 64`: +``` +PS_NONE=0, PS_W_PAWN=0, PS_B_PAWN=64, PS_W_KNIGHT=128, PS_B_KNIGHT=192, +PS_W_BISHOP=256, PS_B_BISHOP=320, PS_W_ROOK=384, PS_B_ROOK=448, +PS_W_QUEEN=512, PS_B_QUEEN=576, PS_KING=640, PS_NB=704 +``` +Per-perspective piece->plane lookup (own/enemy colors swap when +Perspective=BLACK; **both kings share the single `PS_KING` plane** -- +own-king and enemy-king features differ only by which square within +that one 64-square block is set, not by which plane): +``` +Perspective=WHITE: {pawn:PS_W_PAWN, knight:PS_W_KNIGHT, bishop:PS_W_BISHOP, + rook:PS_W_ROOK, queen:PS_W_QUEEN, king:PS_KING} for White(=own) pieces; + {pawn:PS_B_PAWN, ..., king:PS_KING} for Black(=enemy) pieces. +Perspective=BLACK: own/enemy swapped (own=Black pieces use the PS_W_* + constants' *role*, i.e. the W/B labels in this table mean "us/them", + not literal color -- see the "convention: W-us, B-them" comment in + the source). +``` + +**King buckets** -- 32 buckets per perspective (not 64; this is +where `_hm`'s horizontal mirroring halves the table vs. plain +HalfKAv2). `KingBuckets[Perspective][[Square 0..63]]`, each entry +`bucket_index * PS_NB` (so it's already a feature offset, not a bare +bucket number) -- verbatim from the source, printed one visual board +row per line, **rank 1 first** (i.e. row *i* of this literal = Squares +`8*i .. 8*i+7`, matching Stockfish's own `rank*8+file` ordering, not a +top-down board diagram): +``` +KingBuckets[WHITE] (value = bucket*704): + rank1: 28 29 30 31 31 30 29 28 + rank2: 24 25 26 27 27 26 25 24 + rank3: 20 21 22 23 23 22 21 20 + rank4: 16 17 18 19 19 18 17 16 + rank5: 12 13 14 15 15 14 13 12 + rank6: 8 9 10 11 11 10 9 8 + rank7: 4 5 6 7 7 6 5 4 + rank8: 0 1 2 3 3 2 1 0 +KingBuckets[BLACK] (value = bucket*704) -- mirror image of WHITE's table + (own king's bucket decreases as it advances up the board *from that + perspective*, same intent, opposite physical ranks): + rank1: 0 1 2 3 3 2 1 0 + rank2: 4 5 6 7 7 6 5 4 + rank3: 8 9 10 11 11 10 9 8 + rank4: 12 13 14 15 15 14 13 12 + rank5: 16 17 18 19 19 18 17 16 + rank6: 20 21 22 23 23 22 21 20 + rank7: 24 25 26 27 27 26 25 24 + rank8: 28 29 30 31 31 30 29 28 +``` +Only 4 distinct bucket values per rank (not 8) because of the +horizontal mirror: king on file a/h share a bucket, b/g share one, +c/f share one, d/e share one -- the actual file-level distinction +survives through the *orientation* step below, not the bucket. + +**Orientation / mirroring** (`OrientTBL`, verbatim) -- gives the +XOR mask applied to every piece's square (including the king's own), +keyed by the *perspective's own king's actual square*: +``` +OrientTBL[WHITE][ksq]: SQ_H1 (=7) if ksq's file is a-d (king on the + queenside half) + SQ_A1 (=0) if ksq's file is e-h (kingside half) + (applies to ksq on ANY rank -- only ksq's FILE selects the branch) +OrientTBL[BLACK][ksq]: SQ_H8 (=63) if ksq's file is a-d + SQ_A8 (=56) if ksq's file is e-h +``` +`7 = 0b111` flips only the file bits (pure horizontal mirror); +`56 = 0b111000` flips only the rank bits (vertical flip, i.e. "view +from Black's side" with no horizontal mirror); `63 = 0b111111` flips +both (Black's view, horizontally mirrored too). Net effect matches +the header comment: **the perspective's own king always ends up on +the e-h files** after orientation, regardless of which side it +started on or which color is to move. + +**Feature index formula** (`make_index`, verbatim): +``` +index(Perspective, s, piece_at_s, ksq) = + (s XOR OrientTBL[Perspective][ksq]) + + PieceSquareIndex[Perspective][piece_at_s] + + KingBuckets[Perspective][ksq] +``` +where `s` is every occupied square's Stockfish-style 0-63 index +(including `ksq` itself, and including the *other* king's square), +and `ksq` is fixed per perspective as that perspective's own king's +*actual, unoriented* square (the bucket/orient lookups are keyed by +the real square -- only the piece square `s` gets XORed). + +**Per-perspective ordering / total dimensions**: `Dimensions = +SQUARE_NB * PS_NB / 2 = 64 * 704 / 2 = 22528` features per +perspective (the `/2` is exactly the `_hm` mirror halving 64 king +positions down to 32 buckets). Two independent 22528-wide sparse +input vectors are produced per position, one per color's perspective +(White's accumulator half and Black's accumulator half), concatenated +own-perspective-first per the side-to-move-relative output convention +already decided above. Up to 32 features are active at once per +perspective (`MaxActiveDimensions`) -- one per piece on the board, +including both kings (max 32 pieces total on a legal board). + +**Not yet decided, deliberately deferred**: whether typhoon's trained +net keeps Stockfish's literal `HalfKAv2_hm(Friend)` hash/name +convention or adopts its own -- irrelevant until a `.nnue`-compatible +serialization format is chosen, which isn't needed for this project's +own from-scratch C inference code. The spec above only needs to be +byte-for-byte *internally* consistent between typhoon's Python encoder +and typhoon's C encoder; it does not need to produce files +interchangeable with real Stockfish `.nnue` binaries. + +**Bit-for-bit Python/C cross-checker built and passing, 2026-10-15** +(closes milestone 1): +- **C side**: `src/nnuefeatures.c`, a new `dump nnuefeatures` engine + command (wired into `command.c`'s existing `dump` dispatcher, + declared in `chess.h`, added to `GNUmakefile`'s `OBJS`). + Deliberately **not** wired into eval/search/the accumulator -- + reference-math-only, operating on the already-trusted + `g_RootPosition`/`rgSquare` board state so no new FEN-parsing code + was needed. Implements the 0x88->64 conversion + (`CoorToSq64`), `KingBucket`, `OrientMask`, and the index formula as + small standalone static functions, each matching one piece of the + spec above 1:1. Prints `NNUE_FEATURES white <idx>...` / + `NNUE_FEATURES black <idx>...` for the current position. +- **Python side**: `annotate_pgns/nnue_feature_spec.py`, an + independent from-scratch implementation of the identical formulas + using `python-chess`'s own native 0-63 square numbering (no 0x88 + conversion needed on this side -- that's purely a typhoon-C-side + translation concern). +- **Driver**: `annotate_pgns/cross_check_features.py` spawns the real + engine binary via `--batch --command "setboard <fen>; dump + nnuefeatures"`, diffs its output against the Python encoder, + index-set for index-set, over 9 hand-picked FENs chosen to catch a + perspective/orientation mixup specifically (not just the symmetric + starting position, which can't distinguish a White-perspective bug + from a Black-perspective bug since it's symmetric under exactly the + transform between perspectives): both kings queenside (a-d files), + both kingside (e-h), each combination of one king on each half (the + cases most likely to expose an orientation/bucket bug), a dense + midgame position exercising every piece type, a sparse king-and-pawn + endgame (3 active features, near the low end of the range), and a + multi-queen position. **All 9 positions pass on both + perspectives.** Also re-verified passing against the release binary + rebuilt by `precommit_check.sh` (which itself passed both gates), + not just a one-off debug build. +- Run it again with `python3 cross_check_features.py` from + `annotate_pgns/` any time the C or Python side of this formula + changes -- cheap (a few seconds), should be treated as a required + gate before trusting any future feature-vector output from either + side. + +## Training data pipeline + +- **Score target**: sigmoid-squashed WDL-style, per the decision above. + Mixed with or validated against actual game outcome where available + (standard NNUE bootstrapping practice), with exact + mate/EGTB-recognizer positions excluded from the sigmoid-fit loss (see + above). +- **Move-played-as-successor-inversion technique** (confirmed sound): + to get a score for the move actually played in a human game (no + guarantee the engine's own search would pick the same move), score the + resulting position after the move and negate: + `score_before_move_from_mover's_perspective = -eval_or_search(position_after_move)`. + This is just the standard negamax relationship -- well-posed regardless + of whose move it was, and does not require the engine to agree the + move was objectively best. +- **Data sources and current state** (checked 2026-10-14): + - `../pgn/self_play_games.pgn`: 166,640 total games, but only + **26,347 have score annotations** (started "a couple of weeks + ago" per conversation) -- roughly 2.36M annotated `{ comment }` + positions total. The other ~140k pre-annotation games are also + pre-dating a recent change that scrambles opening positions more, + so they are likely heavily duplicate/near-duplicate and not worth + careful deduplication effort yet. Self-play is ongoing + concurrently with this planning, so this pool grows on its own. + - `../pgn/lichess_elite_*.pgn` (2013-2020, many files) and + `../pgn/twic_filtered.pgn` (642,268 games, >=2400 Elo / >40 ply, + 556MB) -- far larger and more diverse than the current self-play + pool, not sourced from this engine's own play, so free of the + "net just re-learns the current eval's blind spots" risk that + bootstrapping directly off self-play data would have. +- **Recommended phasing** (standard NNUE bootstrapping order, not novel): + 1. **Phase 1 -- bootstrap from human games.** Train the first net + primarily on `twic_filtered.pgn`/`lichess_elite_*` positions. + Training on the current self-play pool first would mostly teach + the net to reproduce the current hand-tuned `Eval()`'s own blind + spots, since that's what generated those games -- not useful + signal for replacing it. + 2. **Phase 2 -- self-play with the new net.** Once a first net + exists, use it (not the old `Eval()`) to generate much larger, + more targeted self-play volume -- this is where self-play data + becomes valuable, since it now reflects the new net's own + tactical gaps rather than the old eval's. +- **Search depth for annotation: `sd 10`, fixed depth (not `st` time-based), + decided 2026-10-14.** Measured on a 300-position sample drawn from + `twic_filtered.pgn` (`/tmp/typhoon/nnue_pilot/`): `sd 10` with a 32MB + hash runs at roughly 7 positions/sec single-threaded (~6/sec at the + default 256MB hash -- smaller hash is faster per-position because + every `setboard` triggers a full `ClearHashTable()` memset via + `SetBoardCommand`'s `PreGameReset(FALSE)` call, so a smaller table is + cheaper to clear). At 16 single-threaded shards (`--cpus 1 --hash 32m + --logfile -` each), 5M positions is projected at ~12.4 hours. + `--logfile -` was confirmed to already skip `BeginLogging`'s + backup-file dance entirely (`main.c`), no code change needed there. + The `script` command's periodic checkpoint-report dump (every 5 + positions, `script.c`'s `CHECKPOINT_GO_INTERVAL`, kept for crash + resilience on long joint-table-collection runs) was measured and found + to cost nothing meaningful at this scale (43.9s vs 42.7s/300 positions + with it disabled) -- not worth adding a flag for. + - Depth-adequacy check: compared `sd 10` vs `sd 16` scores on the + same 21 positions (a subsample of the 300; the full 300-position + `sd 16` comparison was not completed -- it was running at ~14s/ + position, ~100x slower than `sd 10`, confirming `sd 16`+ is not + feasible for bulk annotation at the volumes this project needs + regardless of any quality difference). Result: mean |delta| 19cp, + median 10cp; in win-probability terms (what actually matters for + WDL-sigmoid training labels) mean |delta| was ~2.3%, only 1 of 21 + exceeded a 10% win-probability swing, and that one outlier was + already a clearly-winning position at both depths (didn't change + position *character*). Two apparent "sign flips" were both + near-equal/balanced positions where a small score wobble flips + which side nominally leads -- noise at the margin, not a missed + tactic. Small sample (n=21), not weighted toward tactically sharp + positions the way ECM is, but consistent with the expectation that + ordinary human-game middlegame positions (unlike curated tactical + suites) are not especially depth-sensitive for the net's purpose + -- the net's job is positional judgment on these; finding forced + tactics is the search's job at inference time, not the + training-label-generation search's job to get perfectly right. + Conclusion: proceed with `sd 10` for the bootstrap annotation pass. +- **Annotation harness**: build a TWIC/lichess annotation tool reusing + `eval_tune/tune_eval_dna.py`'s `Engine`/subprocess pattern (xboard + protocol driver, persistent-context `eval` command, SysV-semaphore + pitfalls already solved there -- see that file's existing + `Engine`/`EnginePool` context-manager requirement). This is buildable + and runnable immediately, independent of all engine-internals work + below. +- **Quiet-position filtering, applied at training-set assembly time (not + at annotation time)**: exclude positions where the side to move is in + check, or where the best/played move is an SEE-positive capture + (signals the position is mid-exchange, not settled). Two reasons: (1) + a position dominated by a forced tactical resolution tests "find the + tactic," which is quiescence search's job, not the net's positional + judgment -- training heavily on these wastes net capacity re-deriving + something SEE already gives for free; (2) a forcing capture sequence + sampled at plies N, N+1, N+2 produces several highly correlated + near-duplicate samples of one tactical event, not independent data + points, inflating apparent corpus diversity without adding real + signal. Keep annotating every ply regardless (the raw PV/score corpus + has standalone value for analysis); apply this filter only when + building the actual training set from that corpus. Note this does not + create a train/inference mismatch -- the net still gets invoked at + many non-quiet-looking nodes during `QSearch()` at inference time, but + quiescence search is specifically designed to walk down to a settled, + stand-pat-ish leaf before a position's evaluation is actually trusted, + so the distribution of positions where the score is relied upon is + itself mostly quiet-ish by construction. +- **Held-out test set: resolved, already built.** `split_holdout.py`'s + stratified game-level split (result x length-quartile, 10% sampled + independently per stratum -- see Status below) is the holdout pool: + **26,995 train games / 2,298,648 positions** vs. **3,005 holdout + games / 255,245 positions**, split by *game* (not position, avoiding + cross-split leakage from one game's correlated positions) and never + touched when fitting the WDL-sigmoid K constant. This is a separate + pool from `eval_tune/tuned.best_holdout.dna` (that one is + DNA-tuning's own flat, position-level holdout, built for a different + pipeline) -- the two are not reused across projects; NNUE training/ + validation/gating all draw from this `annotate_pgns/` pool instead. + +## Status (2026-10-14) + +- **Sharded annotation harness built**: `../annotate_pgns/annotate.py`. + Shards an input PGN across N single-threaded engine instances, scores + every position in every game at a fixed `sd <depth>` (depth is a + plain CLI arg, so re-running at `sd 12`/`sd 16` later needs no code + changes, just a different `--depth`/`--outdir`), and reassembles an + annotated PGN matching the self-play comment format (`{ +score/depth + PV }`). Validated end-to-end on an 8-game/4-shard smoke test before + trusting it at scale. `load_games()` skips `Result "*"` (unterminated) + games -- can't derive a WDL label from an unknown outcome. Short + agreed-draw games are not separately filtered: `twic_filtered.pgn`'s + existing `>40 ply` / `>=2400 Elo` filter (`eval_tune/filter_pgn.py`) + already excludes those by construction, confirmed empirically (a + 5000-game sample has a hard floor at 40-49 plies, nothing shorter). +- **Stage A (pipeline-validation) run in progress.** Target: ~2M + annotated positions (~30,000 games from `twic_filtered.pgn`, the + first 30,000 in file order) at `sd 10`, 16 single-threaded shards + (`--cpus 1 --hash 32m --logfile -` each). Launched 2026-10-14 ~15:05. + - Output location: `../annotate_pgns/runs/sd10_twic_2M/annotated.pgn` + (written once all 16 shards finish and the driver reassembles + them). Intermediate per-shard files kept in the same directory: + `shard_NN.script` (generated engine command scripts), + `shard_NN.out` (raw engine stdout per shard). Driver progress log: + `../annotate_pgns/runs/sd10_twic_2M.driver.log`. + - Actual yield came in higher than the small smoke-test estimate: + 30,000 games produced 2,553,893 total positions to score (~85/game + average, vs. the ~67/game estimated from the 8-game smoke sample) + -- expect final usable count after quiet-filtering (applied later, + at training-set-assembly time, not during annotation) to land + somewhat under 2,553,893, comfortably still in/above the ~2M + target. + - Measured throughput came in slower than the single-shard pilot + extrapolation: ~75 positions/sec aggregate across all 16 shards + (vs. ~112/sec extrapolated from the single-shard pilot's 7/sec x + 16), most likely from 16-way contention on this shared box + (cache/memory bandwidth) rather than any bug -- progress checks + showed clean, steady, monotonic counts with no errors/aborts + across all 16 engine processes. + - As of 2026-10-14 16:47: 459,156 / 2,553,893 positions done (18.0%). + Revised ETA: ~7.7 more hours from that point (~9.4 hours total + from the 15:05 launch), finishing roughly 00:30 rather than the + original ~5-hour back-of-envelope guess. +- **Stage A run finished 2026-10-14 ~00:40** (~9.6 hours, close to the + revised ETA): 30,000 games, 2,553,893 positions scored across all 16 + shards with no crashes/aborts. +- **Bug found and fixed in `annotate.py`'s output parser, discovered + only after the full run completed (would have been cheap to catch in + the smoke test -- lesson for next time: smoke-test specifically needs + a position or two that hits a forced mate/EGTB hit, not just ordinary + middlegame positions).** `TELLOTHERS_RE` only matched plain-float + scores; the engine's own `+MATE<N>`/`-MATE<N>` notation (and `<EGTB>`/ + `<REC>` PV-terminal sentinel moves, which can carry an ordinary- + looking float score when real moves precede the sentinel, e.g. + `PV=Kf6 Bc3+ Kxg6 Kf8 Kxh5 <EGTB>`) didn't match, so those result + lines were silently dropped from the parsed list while staying in the + position index. Since `annotate_shard` zips the two lists + *positionally*, one dropped line didn't just lose its own + annotation -- it desynced every position after it in that shard, + attaching the wrong score/PV to the wrong FEN for the rest of the + shard's output. All 16 shards hit this (975-1,265 bad lines each, + 16,994 total, ~0.67% of all scored positions), confirmed via a full + scan of every `tellothers` line across all 16 `shard_NN.out` files + (2,553,893 total lines, 0 that failed to match the outer line format + at all -- the only two non-numeric score tokens that ever appear are + `+MATE<N>` (9,986 occurrences) and `-MATE<N>` (7,008), confirming + there's no other silent-drop source hiding in this run's data). +- **Recovered without re-running the search.** `annotate_pgns/extract_positions.py` + mines `shard_NN.script` (the ordered list of `setboard <fen>` lines + already written for that shard) zipped positionally against + `shard_NN.out`'s `tellothers` results -- the engine processes `go` in + strict script order with no interleaving (see CLAUDE.md), and + `setboard`-count equals `tellothers`-count exactly once mate/EGTB + lines are captured as real results instead of dropped, so this + recovers a clean, correctly-aligned dataset straight from the + existing raw logs. PGN reconstruction (what `annotate.py` was doing) + turned out to be an unnecessary detour for a project that only ever + needed (FEN, score) pairs, not game trees -- it's also what + introduced the misalignment risk in the first place. + Output: `runs/sd10_twic_2M/positions.jsonl`, one JSON object per + position (`fen`, `depth`, `score`, `score_raw`, `pv`, `exact`, + `exact_reason`). Exactness is detected from the PV's *last move + token* being `<EGTB>`/`<REC>` (not the whole-PV string or the score + field -- a real line of moves can precede the sentinel, and the + score alongside it can look like an ordinary centipawn value even + though the position is game-theoretically exact) or the score being + `+-MATE<N>`. Result: 2,553,893 positions total, 48,407 flagged + `exact` (zero `<REC>` occurrences this run, all exact rows were mate + scores or `<EGTB>`-terminated PVs), **2,505,486 usable rows** for + sigmoid-fit training -- matches the original run's full target yield. + `annotate.py` itself still has the original bug (not yet patched -- + low priority now that `positions.jsonl`, not its PGN output, is the + actual training artifact; would still need fixing before trusting any + of its PGN output for other purposes). +- **Related engine-side bug found and fixed while investigating the + `+315.92`-style non-exact-looking EGTB scores above**: `probe.c`'s + `EGTB_WIN_SCORE`/`EGTB_LOSS_SCORE` used to be `NMATE - VALUE_QUEEN` + (deliberately kept below the `NMATE` threshold so it wouldn't be + mistaken for a literal forced mate elsewhere in the engine), which is + why EGTB hits printed as ordinary-looking floats instead of + `+-MATE<N>`. Changed to literal `+-NMATE` so `FormatScore` (`util.c`) + renders every EGTB win/loss as a flat `+-MATE100` uniformly, matching + the already-observed bare-`<EGTB>` case and making all EGTB-terminated + PVs parseable by the same plain `+-MATE\d+` regex going forward -- + no separate PV-suffix check will be needed for *future* runs (this + run still needs it, since it predates the fix). Caveat checked before + trusting this: `ADJUST_MATE_IN_N` (the `#ifdef` in `hash.c`'s + `StoreExactScore` that would ply-adjust a score assuming it encodes + mate distance -- which a flat EGTB sentinel does not) is not defined + anywhere in this build (confirmed via grep across all headers and + `GNUmakefile`), so the hash-store path this change now exercises for + every EGTB hit is the safe `#else` branch (downgrades to a + `HASH_FLAG_LOWER`/`HASH_FLAG_UPPER` bound entry with a flat `+-NMATE` + value, no ply arithmetic) -- confirmed this exercises a **second, + previously-dormant bug** in that same `#else` branch: the `iValue <= + -NMATE` case was pairing `HASH_FLAG_UPPER` with `sNew.iValue = + +NMATE` (should be `-NMATE` -- compare the mirror-image win-case + branch just above it, which correctly pairs `HASH_FLAG_LOWER` with + `+NMATE`). Previously dormant because the old `NMATE - VALUE_QUEEN` + EGTB-loss score never satisfied `iValue <= -NMATE` on its own (only a + literal engine-found forced mate-against could reach this branch); + the `EGTB_LOSS_SCORE = -NMATE` change makes every EGTB-loss position + hit it. In a release build this was harmless (a `+NMATE` upper bound + is vacuously loose, never produces an incorrect cutoff, just wastes a + hash write); in a `DEBUG` build it would trip `search.c`'s + `VERIFY_HASH_HIT` macro's `ASSERT(iScore < +NMATE)` on later + retrieval under the `HASH_FLAG_UPPER` case. Fixed (`hash.c`: + `sNew.iValue = -NMATE`). +- **Four more real engine bugs found and fixed in the same session**, + all via `precommit_check.sh`-verified fixes, committed together as + `ef2c2f6` ("Fix several latent search/move-generation bugs found via + NNUE annotation tooling"): + - **`san.c`**: a checking castle move (`O-O+`/`O-O-O+`) printed as + `+-O`/`+-O-O` -- the check-suffix pointer (`p = szTextMove;`) was + captured *before* `strcpy` filled the buffer with `"O-O"`, so + `*p++ = '+'` overwrote the leading `O` instead of appending after + it. Fixed by recomputing `p` from `strlen(szTextMove)` after the + `strcpy`. + - **`search.c`/`searchsup.c`**: all five PV sentinel-recording sites + (`MATEMOVE`, `DRAWMOVE` x2 -- stalemate and the separate + repetition/50-move site, `RECOGNMOVE`/`EGTBMOVE` x2 -- main + search and QSearch) were gated by the same strict + `(iAlpha < iScore) && (iScore < iBeta)` window check used for + ordinary heuristic PV moves. Traced live (temporary `Trace()` + instrumentation on a real mate-in-2 from the corpus, removed + after): once a forced line's value is already known (from an + earlier, shallower iterative-deepening pass, or the hash table), + later visits to it are typically probed with a PVS scout/null + window built *around* that known value, so the terminal node's + real score *ties* the window rather than strictly beating it -- + failing this gate by equality, not by being out of range. But + these five values are exact/game-theoretic facts, not heuristic + bounds -- unlike an ordinary move, they can't be "wrong" for + having been reached outside the window. Fixed by recording them + unconditionally. This alone did not fully fix `<#>`/`<=>` display, + though -- see the next bullet. + - **`util.c` (`WalkPV`)**: traced further and found the *actual final + move* of a forced mate (one ply shallower than the terminal + sentinel) hits the identical tie, through the completely separate + *ordinary* move-acceptance code path (`iScore > iAlpha` at + `search.c`'s main move loop) -- so the recorded PV is often + missing its own concluding move, not just the sentinel after it. + Rather than touch that hot, per-node path (every interior node of + the entire search tree), `WalkPV` now reconstructs the missing + tail directly from the board when the recorded PV runs out one + ply short of (or exactly at) a forced mate/stalemate/draw: the + position at that point already reflects every printed move (each + was `MakeMove`'d during the walk), so it checks `IsDraw`, or + failing that, finds the one legal move the shrinking mate + distance guarantees must exist. Runs once per improved root + score, not in the search tree, so the extra legal-move + verification is cheap. Verified on mate-in-1 (`Rd8+ <#>`), a real + mate-in-6 (13-move chain, no gaps anywhere in the middle, not + just at the end), and a constructed 50-move-rule draw + (`Kc1 <=>`). + - **`gamelist.c`/`board.c`/`testsup.c`**: `SetRootPosition` used to + parse a `setboard` FEN straight into the global `g_RootPosition`, + so a *rejected* FEN (malformed syntax, or -- newly caught -- + describing an illegal position) still left partially-parsed + garbage in `g_RootPosition`, because `GetRootPosition()` hands + out a stable pointer straight to it with no defensive copy + (removing one was a deliberate, confirmed-correct fix for a + *different*, unrelated bug -- see that function's own comment). + The next `go` searched that garbage regardless of the earlier + rejection, discovered independently two ways: an `InvalidGenerator` + dispatch panic (`generate.c`) and, separately, a debug-build + assert in `_ScoreAllMoves`'s MVV/LVA scoring (`PIECE_VALUE(KING)` + overflowing past `VALUE_KING` -- a pseudo-legal "capture the enemy + king" move, generated because nothing stops the generator from + producing one against an opponent king already left in check). + Root cause of *that*: a hand-built FEN leaving the side not on + move in check is illegal (unreachable through real play), and + nothing validated for it. Fixed: `SetRootPosition` now parses into + a local staging `POSITION` and only commits to `g_RootPosition` if + `FenToPosition` succeeds *and* `IsBoardLegal` (pawns never on + rank 1/8, side not on move never in check) *and* + `VerifyPositionConsistency` (internal struct consistency -- piece + lists, material counts, signatures) both pass. Both checks now + run unconditionally, in every build profile, not just + `TEST`/`DEBUG` -- `IsBoardLegal` had to be moved from `testsup.c` + (only linked under `TEST`/`EVAL_DUMP`) to `board.c` (always + linked) to make this possible; `VerifyPositionConsistency` is + DEBUG-gated at its many *other* (hot, per-move) call sites for + performance, but this one is cold (once per game/`setboard`, not + per node), so the cost is worth it here specifically. + **Important, deliberately NOT a bug**: a separate illegal-FEN + repro that looked like a second instance of the same class + (`8/8/8/4kp2/5P2/4K3/8/8 w - - 99 60`, locked pawn pair adjacent + to both kings) turned out to just be a badly hand-built test FEN + of my own -- White's own pawn on f4 already attacked the square + Black's king sat on while it was White's move, which is the same + "opponent in check" illegality, not a new generator bug. Confirmed + by reverting all of this session's changes (`git stash`) and + reproducing the identical crash on unmodified `master` -- notable + only as a reminder that a hand-built FEN needs to be checked for + basic legality before trusting a crash it causes. +- **`extract_positions.py` exactness-tagging gap found and fixed** (see + the earlier bullet on the `<EGTB>`/`<REC>` MATE-regex bug): it also + missed `<=>` (`DRAWMOVE` -- a claimed repetition/50-move draw) and the + `"only"` sentinel (`root.c`'s single-legal-move-at-the-root shortcut, + which skips searching entirely and hardcodes `score=0`/`depth=0` -- + not a real evaluation, not even indeterminate, just never searched). + A draw-by-repetition/50-move claim is a fact about the *game's move + history*, not the position -- the same position reached a different + way might not be a draw at all -- so it's excluded from training for + the same reason `<EGTB>`/`<REC>` are. Re-running extraction with both + fixed: exact-row count went from 48,407 to **93,415** (47,304 of the + increase is `<=>`), leaving 2,460,478 usable rows (down from + 2,505,486). +- **Quiet filter built and run on the full corpus** (`quiet_filter.py`, + reusing `eval_tune/tune_eval_dna.py`'s pre-existing + `static_exchange_eval`/`has_hanging_material` rather than + reimplementing): excludes in-check positions and positions whose PV's + first move is an SEE-positive (>=50cp) capture. Full-corpus result: + 2,553,893 total rows -> 48,407 exact -> 112,405 excluded (in check) -> + 311,949 excluded (SEE-positive first move) -> **2,080,911 quiet + positions** (`runs/sd10_twic_2M/positions.quiet.jsonl`), ~81.5% + retention. +- **Shared quiet-predicate module factored out** (`quiet_common.py`): + `position_is_quiet(board, move)` plus the SEE-threshold constant, + used by all three consumers below so the exclusion logic (in-check, + SEE >=50cp capture) lives in exactly one place. The two call sites + differ only in *which move* gets SEE-checked, and this is a real, + tracked distinction, not an implementation detail -- the engine's + own PV first move (the two post-hoc filters, below) vs. the move + actually played in the source game (the new pre-filter, below) are + two different training distributions; rows produced under the two + should not be silently mixed into one training run. +- **Quiet filter now also applied to the game-shaped labeled files** + (`quiet_filter_games.py`), closing the gap this file previously + flagged as open -- re-applies the identical PV-based predicate + directly to each position already embedded in a game record, rather + than joining against `positions.quiet.jsonl` (which carries no + row/game identity, and a repeated FEN across different games/plies + would make a content-based join silently misattribute rows). A game + that loses every position to filtering is dropped entirely (none + were, in this corpus). Run on both labeled splits 2026-10-15: + **holdout**: 255,245 -> **215,134** positions (9,476 in-check + + 30,628 SEE-positive-capture + 7 indeterminate excluded), 3,005/3,005 + games kept -- `games.holdout.labeled.quiet.jsonl`. **train**: + 2,298,648 -> **1,932,713** positions (88,278 in-check + 277,593 + SEE-positive-capture + 64 indeterminate excluded), 26,995/26,995 + games kept -- `games.train.labeled.quiet.jsonl`. Combined retention + across both splits is 2,147,847 / 2,553,893 = 84.1%, noticeably + higher than the flat-file filter's 81.5% -- **not noise, a real + behavioral difference**: `quiet_filter.py` (flat) drops exact rows + (mate/EGTB/`<=>`) from its output entirely, while + `quiet_filter_games.py` always keeps them (only the in-check/SEE + criterion applies to non-exact rows; exact rows are real, + game-theoretic ground truth, not something the quiet filter has any + basis to exclude -- NNUE.md already excludes them from the + sigmoid-fit *loss* elsewhere, which is a separate, correct exclusion + from excluding them from the dataset altogether). The game-shaped + filter's behavior is the one intended going forward; the flat file's + stricter exact-row drop was never a deliberate design choice, just + an artifact of how `quiet_filter.py` was first written. +- **Quiet pre-filtering moved upstream into `annotate.py`** + (`--prefilter-quiet` flag; `quiet_common.position_is_quiet` applied + inside `positions_for_game` before a position is ever written into a + shard script), so a *future* annotation run never pays `sd N` search + cost on a position that would only be discarded afterward -- making + the post-hoc game-shaped filter above unnecessary for runs going + forward. **Deliberate semantic difference from the post-hoc + filters**: no PV exists before the search has run, so the pre-filter + SEE-checks the move *actually played* in the source game instead of + a PV's first move (see `quiet_common.py`'s module docstring). + `build_games.py` gained a matching `--prefilter-quiet` flag that + **must match** whatever the originating `annotate.py` run used -- it + recomputes per-game ply counts via the same `positions_for_game` call + to re-slice `positions.jsonl`, and a mismatched flag would silently + misalign every position after the first game that differs. Verified + end-to-end on a 4-game/2-shard/`sd 6` smoke test: pre-filtering wrote + 253 of the games' total plies into the shard scripts, all 253 came + back annotated with no count-mismatch warnings, and `build_games.py + --prefilter-quiet` reproduced the identical per-game slice + boundaries. The already-completed Stage A run below predates this + flag and was not pre-filtered -- this only changes the *next* + annotation run. +- **Python SEE cross-checked against the engine's real `SEE()`** + (`check_see_against_engine.py`, using the bulk `seescores` command): + 2,000 sampled positions, 4,032 actual capture comparisons -- 31.0% + exact match, 92.7% sign match, 7.3% sign mismatch. Investigated one + mismatch in depth (`Bxa6` in a real position, engine said +100, + Python said -230) via temporary `Trace()` instrumentation in + `_SEEFull`/`SEE()`: **not an engine bug** -- the engine correctly + finds that after `Bxa6 Rxa6`, White's queen, freshly revealed by the + bishop's own departure (x-ray down the same diagonal), recaptures the + rook, so Black's actual best reply is to decline the recapture + entirely (net -100, not -230). Python's `static_exchange_eval` + computes all attacker lists once, off the pre-move board, so it never + sees a piece the mover's own departure uncovers -- exactly the + "ignores... discoveries" limitation its own docstring already + confessed. The 7.3% mismatch rate is overwhelmingly (likely entirely) + this same known, accepted limitation, not a new one -- confirms the + cross-check tool did its job, not that anything is broken. +- **Game-container reconstruction built** (`build_games.py`): recovers + per-game grouping that `positions.jsonl` lost (flat list, no + `game_id`) by re-loading the same PGN with the same + `max_games`/`--shards` (pure parsing, no search, deterministic) and + slicing the already-extracted rows back into per-game chunks via each + game's known position count. Output shape -- `{game_id, result, + positions: [...]}`, one line per game -- deliberately matches the + shape a self-play PGN with per-move score/PV comments already has + (self-play annotation is a separate, ongoing effort outside this + project, run every time the engine plays a self-play match), so a + future loader for that source can emit the same schema and both mix + freely at training-set-assembly time. Verified exact: 30,000 games, + 2,553,893 positions, matching `positions.jsonl`'s count precisely. + Output: `runs/sd10_twic_2M/games.jsonl`. +- **Stratified game-level train/holdout split built and run** + (`split_holdout.py`): splits by *game*, not position (every position + from one game lands on the same side -- avoids the leakage a flat + position-level shuffle has, which `eval_tune/tune_eval_dna.py`'s own + holdout split still does). Stratified on two axes so a 10% sample's + composition mirrors the full pool rather than risking (by chance) too + many decisive games, too few draws, or a skew toward short/long games: + result (White win/Black win/Draw) x game-length quartile (quartile + cutpoints derived from the actual corpus: 63/80/102 annotated plies, + not guessed). 10% sampled independently within each of the 12 + resulting strata. Result: **26,995 train games / 2,298,648 positions** + (`games.train.jsonl`), **3,005 holdout games / 255,245 positions** + (`games.holdout.jsonl`). +- **WDL-sigmoid target conversion built and run** (`fit_k_and_label.py`): + per NNUE.md's existing decision, reuses `tune_eval_dna.py`'s + `fit_k`/ternary-search-over-K approach, but K is fit directly against + the already-computed `sd 10` search scores (no engine round-trips + needed, unlike the original `fit_k`, which called live `Eval()`), and + **only on the train split** -- fitting K on holdout data would leak + information into what's supposed to be an untouched generalization + check; the same fitted K is then applied (never refit) to label + holdout positions too. **Caveat hit and fixed**: the first attempt + copied `tune_eval_dna.py`'s own search bracket (`[100, 2000]`, + calibrated for *centipawn*-scale `Eval()` scores) verbatim and got + `K=10.00` -- exactly its lower bound, a clear sign of a clamped, + not-actually-converged fit, not a real answer. Root cause: this + project's own scores are stored in *pawn* units (e.g. `+0.47`, not + `47`), a 100x unit mismatch. Refit with an appropriately-scaled + bracket (`[0.05, 20]`) landed well inside the range: **K = 1.3123** + (pawns), MSE improved from 0.1401 (clamped) to 0.1175 (real optimum). + Sanity-checks as plausible for >=2400-level play: a 1-pawn edge -> + `sigmoid(1/1.3123) ~= 0.68` win probability, a 2-pawn edge -> `~= 0.82` + -- modest material deficits are often held or converted at this + level, not auto-losses. Deliberately does **not** blend `p_search` + (the sigmoid-derived estimate) with `r_outcome` (that position's own + game's actual final result, side-to-move-relative) into a single + target yet -- both are stored separately per position so the blend + weight (lambda in `target = lambda*p_search + (1-lambda)*r_outcome`) + remains a free parameter to sweep later without re-deriving anything + from scratch. Verified correct on spot-checked positions (result + flips correctly to each position's own side-to-move perspective). + **Lambda decided for Phase A: lambda=1.0 (pure `p_search`, no + `r_outcome` blend), 2026-10-15.** Phase A's goal is pipeline/ + framework validation on a small local-CPU training run, not the + best achievable net -- a free parameter with no data yet to tune it + against would just be guessed, so defer the sweep to Phase B/the + larger GPU-trained run, where there's enough data and enough training + iterations for a lambda sweep to produce a real signal rather than + noise. `r_outcome` stays stored per position regardless (already + computed, free to use later) so the sweep needs no re-derivation. + Output: `runs/sd10_twic_2M/games.train.labeled.jsonl` (26,995 games, + 2,298,648 positions, 84,302 exact rows left unlabeled), + `games.holdout.labeled.jsonl` (3,005 games, 255,245 positions, 9,113 + exact rows left unlabeled). +- **Not yet built**: the Python training script itself (the loss + function, quantization-aware export pass), the feature-indexing-spec + cross-checker, any `SEARCHER_THREAD_CONTEXT`/accumulator plumbing + (still correctly gated behind a dedicated branch, not started), and + the lambda sweep/selection for blending `p_search`/`r_outcome`. The + quiet-filter gap previously noted here (not yet applied to the + game-shaped/labeled files) is closed -- see `quiet_filter_games.py` + above -- and a future annotation run can skip the post-hoc step + entirely via `annotate.py --prefilter-quiet`. + +## Training compute + +- No reliable local GPU. Decided against Google Colab's notebook-session + model (session timeouts, no persistent SSH access) in favor of **hourly + GPU rental with real shell access** -- vast.ai, RunPod, or Lambda Cloud. + Rough order-of-magnitude expectation: a Stockfish-sized net (low + millions of parameters) over tens of millions of positions is on the + order of 1-2 hours/epoch on a single consumer-class rented GPU, full + run under a day; CPU-only training of the same workload is roughly + 20-50x slower (no tensor cores/wide SIMD at that scale), making a full + run impractical locally but a small-scale pipeline-smoke-test (a few + million positions, few epochs) plausible in hours-to-a-day on CPU if + needed for early iteration before committing to a paid GPU run. +- **Decided 2026-10-15**: Phase A uses exactly that local-CPU + pipeline-smoke-test path -- PyTorch on CPU, against the ~2M-position + quiet-filtered train/holdout pool already built, purely to prove the + training script/data loader/export pass work end-to-end and to get + fast local iteration when something breaks. GPU rental is deferred + to the real, larger Phase B training run. + +## Integration into search + +- Lazy-eval tier restructuring: today's three-tier super-lazy/lazy/full + `Eval()` system does not map 1:1 onto NNUE, which doesn't have + comparable internal tiers the way the handcrafted eval's term-by-term + structure does. Tentative new tiering: (optional, experimental) + material-only super-lazy prefilter -> cheap linear PSQT-term-of-the-net + first pass -> full net forward pass. Decide how much of `eval.c`'s + existing tiering logic is kept vs. deleted once this is actually + implemented, not before. +- Futility/EFP/LMR/NMP margins are all tuned against the current + `Eval()`'s score distribution and will need full recalibration against + the new net's score distribution -- expect this to be a comparably + sized project to the LMR/NMP tuning already referenced in CLAUDE.md, + not a quick follow-up step. + +## Validation and rollout + +- **Gate using `eval_tune/match_play.py`'s existing LOWER95 methodology** + (already used for DNA-tuning candidates) -- head-to-head match at + fixed search depth, gate on `LOWER95 >= 0.5`, not raw score, for the + same non-determinism reasons already documented for DNA tuning. +- Snapshot a fresh `head_reference`-style baseline before any + search-affecting plumbing (accumulator maintenance, cTrapped/cEnprise + decoupling) lands, per the existing `head_reference/` protocol in + CLAUDE.md -- this is a broad enough change to want a clean before/after + reference point, same as any other search-affecting change. +- Keep both evaluators coexisting behind a toggle for as long as + practical, so regressions can be isolated to "is it the net" vs. "is + it the recalibrated margins" vs. "is it the accumulator plumbing" + rather than landing all three at once. +- Do not retire the handcrafted `Eval()` until NNUE wins convincingly and + consistently under the `match_play.py` gate. + +## Sequencing / milestones + +1. **Done 2026-10-15.** Canonical HalfKAv2_hm feature-indexing spec + written down + Python/C cross-checker built and passing on known + test positions. +2. **Done 2026-10-15, concluded not a blocker.** + `cTrapped`/`cEnprise` consumers and producers enumerated (see + "Breaking external dependencies" above). `cEnprise`'s null-move- + refutation source needs no change. Everything else -- `cTrapped`'s + only source, `cEnprise`'s other source, and 3 of + `EstimateHangingMaterialCp`'s 5 checks -- is entangled in + `_EvalTrappedPieces`'s attack-bitboard logic, but degrades + gracefully (the same low-information fallback path already fires + routinely today on lazy/super-lazy eval exits) rather than breaking + -- a precision shift in the joint fail-high prediction table's + "dumb move" axis, not a correctness bug. **Decision: ship the eval + swap first, measure the actual pruning/reduction impact with + existing joint-table instrumentation, and only build an independent + bitboard pass if that measurement says it's worth it.** No + speculative tactical-signals pass planned up front. +3. MakeMove/UnmakeMove accumulator plumbing (ply-indexed king-move + backup in `PLY_INFO`) built and proven correct/fast **with the old + `Eval()` still live** -- prove the plumbing before any trained net + exists to evaluate against it. **Do this on a separate git branch**, + not on `master`, given how much live search-path code it touches; + snapshot `head_reference` first. +4. TWIC/lichess annotation harness built; `<TT>`/`<EGTB>`/`<REC>` tag + semantics investigated and resolved; first large annotated training + pool assembled (Phase 1 data, per above). **Unblocked now, independent + of steps 1-3** -- no engine-internals risk, can start immediately. +5. First bootstrap net trained (rented GPU), quantization-aware export + pass built. +6. C inference code (forward pass + quantized weight loader) written and + validated against the Python model's output on held-out positions. +7. Wired into search behind a toggle; margins recalibrated. +8. `match_play.py` gate; iterate; retire old `Eval()` only after a + convincing, repeated win. + +## Immediately actionable, unblocked work + +Not gated behind any engine-internals decision above: + +- Build the annotation harness and begin scoring `lichess_elite_*`/ + `twic_filtered.pgn` games (Phase 1 training data). +- Investigate `<TT>`/`<EGTB>`/`<REC>` tag semantics in the self-play PV + printer. +- Finalize the written HalfKAv2_hm feature-indexing spec (design-only, + no code). + +Gated behind opening a dedicated branch (per discussion -- not yet +started): + +- Any `SEARCHER_THREAD_CONTEXT`/`PLY_INFO`/MakeMove/UnmakeMove changes. diff --git a/src/docs/NULLMOVE_SKIP_CALIBRATION.md b/src/docs/NULLMOVE_SKIP_CALIBRATION.md new file mode 100644 index 0000000..50a050e --- /dev/null +++ b/src/docs/NULLMOVE_SKIP_CALIBRATION.md @@ -0,0 +1,213 @@ +# Null-move-skip calibration plan (2026-10-12) + +## Motivation + +Two FH%-prediction tables now exist in `data.c`: + +- `g_uJointFHThousandths` ("alpha table") -- predicts a move's FH% given + [depth][hang][gap-to-alpha][late][improve]. Used today for LMP/EFP/LMR + decisions on individual leftover moves. +- `g_uBetaFHThousandths` ("beta table") -- predicts a node's FH% given + [depth][gap-to-beta][ownImprove][enemyImprove], for nodes where + `iEval > iBeta`. Used today for the static-NM shortcut + (`search.c`'s `fTryBetaPruningHere` block, >95% -> return `iBeta` + without searching). + +Question: is there a *low*-confidence corner in either table reliable +enough to justify **skipping the real null-move search attempt** +entirely at a node (not pruning the node itself -- just not trying the +null probe, falling through to the normal move loop: hash move, +killers, etc.)? Unlike the static-NM shortcut, this is a pure +efficiency play with no score-injection risk: if we're wrong, the real +move loop still runs and finds the correct score. We just forgo the +cheap shortcut on the rare occasion it would have worked. + +## Why the alpha table, not the beta table, for this + +At non-PV nodes, `iAlpha == iBeta - 1`, so `iEval <= iAlpha` and +`iEval >= iBeta` together cover the entire range -- no gap. Checked +both tables' actual baked data (`data.c`) for a usable low-FH corner: + +- **Alpha table**, sliced at hang=0 ("nothing hung"), late=0 ("even the + single best-looking move"): deep remaining depth (shipped buckets + 8-10) x large gap-to-alpha (buckets 8-10) x improve bucket 0 (own + eval declining) gives real, non-fallback predicted FH% as low as + 0.001-0.1% -- two-plus orders of magnitude below the ~0.4-0.9% + baseline leftover-move rate. A strong, well-observed signal. +- **Beta table**: its 2026-10-11 inversion (storing gap-to-100% instead + of FH% directly, to keep resolution near 100% for the static-NM + shortcut) saturates every cell predicting below 34.466% FH to the + same stored value. No resolution exists below that floor in the + current bake -- there is no usable low corner to find here. Would + need a dedicated unsaturated rebake to pursue; out of scope for now. + +So: implement the skip using the alpha table only, for the +`iEval <= iAlpha` side. + +## The EV algebra + +Trying null-move at a node has expected value (in nodes saved): + +``` +EV(try) = p * realLoopCost - nullProbeCost +``` + +where `p` = probability null-move fails high, `nullProbeCost` = the +reduced-depth probe's node cost, `realLoopCost` = what the real move +loop costs to reach its own cutoff (the cost skipped entirely when the +probe succeeds). Skip trying null-move iff +`p < nullProbeCost / realLoopCost`. + +- `nullProbeCost` is trivial to measure: a node-counter delta around + the probe's recursive `Search()` call (same as `TEST_NULL`'s existing + `uNullCalibProbeNodeCost`). +- `realLoopCost` is conceptually simple (it's the cost the node always + pays via the real move loop when null isn't tried) but the + *counterfactual* isolation -- measuring it specifically for the + "null was never attempted" branch, without contamination from + production's own null-move logic firing afterward -- needs a + deliberate harness. Not yet built; see "Future work" below. + +### R-factor / branching-factor estimate + +Using `p* ~= b^(-R)` with `b = 2.5` (assumed average branching) and +`R = 3 + floor(depth/6)` (actual `SelectNullmoveRFactor` formula): + +| remaining depth (plies) | R | p* = 2.5^-R | +|---|---|---| +| 0-5 | 3 | 6.4% | +| 6-11 | 4 | 2.6% | +| 12-17 | 5 | 1.0% | +| 18-23 | 6 | 0.41% | +| 24-29 | 7 | 0.16% | + +The threshold **tightens** (gets stricter) with depth, not looser -- +`realLoopCost` grows with full remaining depth while `nullProbeCost` +only shrinks mildly via R's slow growth. The alpha-table corner found +above (shipped depth 8-10, raw ~8-13 plies, landing in the R=4-5 rows, +`p* ~= 1-2.6%`) showed predicted FH% of 0.001-0.15% there -- clearing +the math-derived bar by one to two orders of magnitude. Good sign the +corner is real, not just noise, even under a crude model. + +## Nested-null-move handling + +Raised explicitly: does a null-move search recursing into *another* +null-move search deeper in its own subtree break any of this? + +- **Inside the probe's own subtree**: not a problem. `fAvoidNullmove` + is a single shared flag, not stacked per-ply -- it only blocks two + *consecutive* nulls, toggled tightly around the probe call and reset + immediately after. Nested null-moves several plies down inside the + probe are real and representative, exactly as they'd behave in + production. This does mean `nullProbeCost` is a high-variance + per-sample quantity (one lucky nested cutoff can collapse most of a + probe's subtree) -- argues for a large sample floor per bucket + (reuse the ~500-try convention `dual_table_run` already uses) + before trusting any bucket average, not a correctness fix. +- **Inside the "skip branch" realLoopCost measurement**: this is where + it matters. If/when a skip-rule is compiled into `Search()`, nested + nodes inside that measured real-loop subtree will also apply the + same rule recursively -- calibration data collected this way depends + on whatever rule is *already* active at calibration time. Circular. + Resolution (same pattern already used for every other baked table + here): **Pass 1** calibrates with no skip-rule compiled in anywhere + (matches `TEST_NULL`'s existing "structural eligibility only, don't + consult the thing being evaluated" design). Ship from that. **Pass + 2**: re-measure/re-splice once live, since nested applications then + change descendant costs too (likely compounding favorably, since a + collapsing-eval signal is autocorrelated down a few plies -- a + hypothesis, not yet verified). + +## What's implemented now (2026-10-12) + +A single flat, compile-time sweep knob -- not yet the full +depth-bucketed table: + +- `search.c:93`: `#define NULLMOVE_SKIP_FH_CEILING_THOUSANDTHS (0)` + -- `0` disables (default/shipped state). +- `search.c`, inside `fTryBetaPruningHere`, opposite-direction check + from the existing static-NM block: when `iEval <= iAlpha`, look up + `_JointTablePredictedFHThousandths(uRawDepthB, 0, GapCpBucket(iAlpha + - iEval), 2, uRawImproveB)` (hang=0, late=2 -- the table's own domain + floor, since collection required `uLegalMoves > 1`, so there's no + literal "move 1" bucket; 2 is the nearest available proxy for "even + the single best-looking move"). `uRawDepthB`/`uRawImproveB` are + reused from the already-computed per-node locals (lines ~551, ~726), + not recomputed. If predicted FH% is below the ceiling, skip calling + `WeShouldTryNullmovePruning` for this node entirely, falling through + to the real move loop. +- `chess.h`: new counter `u64NullMoveSkippedLowFH` (tree counters + struct only, matching `u64QuickNullMoveSuccess`'s precedent -- not + MP-merged, diagnostic-only). + +## Empirical sweep results + +### bench (6 fixed positions, fast but small) + +Baseline (ceiling=0): 2,714,949 nodes. Shallow dip at ceiling 10-30 +(~0.4-0.5% fewer nodes), rising above 100, blowing up by 3000-10000. + +### ecm_ringers (11 positions, `sd 10`) -- too small, non-monotonic + +| ceiling | total nodes | solved/11 | +|---|---|---| +| 0 | 4,293,355 | 3 | +| 10 | 4,435,817 (+3.3%) | 3 | +| 30 | 4,314,830 (+0.5%) | 3 | +| 100 | 4,250,553 (-1.0%) | 3 | +| 300 | 4,089,390 (-4.7%) | 3 | +| 1000 | 4,700,197 (+9.5%) | **2** (lost ECM.783) | +| 3000 | 3,904,085 (-9.1%) | 3 | + +Matches this project's own documented caveat: ringers is too small and +chaotically order-sensitive to trust a single run's shape. + +### ecm_confident_quick + ecm_hard_quick (90+90 positions, `sd 10`) + +Ran as 8 parallel background processes (4 ceilings x 2 suites, 24-core +box, ~idle at the time) using 4 separately-built binaries (ceiling is +a compile-time `#define`, can't be swept at runtime without rebuilding). + +| ceiling | combined nodes (180 pos) | delta | combined solved (baseline 79) | +|---|---|---|---| +| 0 | 48,588,231 | -- | 79 | +| 300 | 47,468,723 | -2.3% | 76 (-3) | +| 1000 | 47,783,919 | -1.7% | 77 (-2) | +| 3000 | 45,907,126 | **-5.5%** | **79 (net 0)** | + +Per-suite: `ecm_confident_quick` solved count degrades monotonically +(62/60/61/59) as ceiling rises; `ecm_hard_quick` is flat/down through +300-1000 (17/16/16) then jumps to **20** at 3000. The jump is what +makes 300/1000 look worse than their neighbors on both sides -- +non-monotonic, not a stable curve. + +**Conclusion so far**: no clean single minimum for a flat ceiling -- +consistent with (not just predicted by) the depth-dependence argument +above. Ceiling=3000 is the one genuinely encouraging point (real node +savings, net-zero solve cost across 180 positions) but a flat knob +can't properly track a threshold that should move with depth; finding +a better flat number isn't the right next move. + +## Future work + +1. **Don't keep sweeping the flat knob.** Build the depth-bucketed + version instead: extend `TEST_NULL` to measure `nullProbeCost` and + `realLoopCost` per (depth, gap, improve) bucket (Pass 1, no live + skip-rule anywhere, per the nested-null discussion above), bake a + per-bucket `p*` the same way `dual_table_run`/`splice_dual.py` + already bakes the other tables, re-measure/re-splice once shipped + (Pass 2). +2. If that proves too complex or doesn't pan out, fall back to a + coarser depth-only piecewise ceiling (a handful of depth bands, + each with its own flat threshold derived from this sweep's + per-depth behavior) rather than one global number. +3. Before committing anything: full `ecm.ep_` run at the chosen + setting(s), `precommit_check.sh`, and the usual `head_reference` + comparison protocol. +4. The beta-side (`iEval > iBeta`) mirror of this idea is still + blocked on a resolution-preserving rebake of `g_uBetaFHThousandths` + below its current 34.466% saturation floor -- not worth pursuing + until the alpha-side version proves out. + +Sweep artifacts (binaries, logs, stdout) are under +`/tmp/typhoon/nullskip_sweep/`, scratch -- not kept. diff --git a/src/docs/PROFILE.md b/src/docs/PROFILE.md new file mode 100644 index 0000000..67106e0 --- /dev/null +++ b/src/docs/PROFILE.md @@ -0,0 +1,222 @@ +# Profiling ./typhoon + +Two environments, two different root situations. Short version: **on +Linux, use `gmake GPROF=1` + gprof(1) -- fully root-free.** On FreeBSD, +`pmcstat`'s hardware-counter sampling is also root-free *in practice* once +`hwpmc.ko` is loaded, which is the one step that does need root (and only +once per boot). + +## Linux (this box): gprof, no root required at all + +`perf` is blocked here -- `kernel.perf_event_paranoid` is `4` (stricter +than the usual 0-3 range; Ubuntu-specific), so `perf record`/`perf stat` +refuse to run for a non-root user with no CAP_PERFMON, and there's no +non-root workaround short of asking someone to lower that sysctl or grant +the capability. `valgrind`/`callgrind` and `uftrace` aren't installed and +installing them needs `apt` (root). `gprof` (part of binutils) is already +installed and needs no privilege of any kind -- it works by compiling the +binary with `-pg`, which makes every function call `mcount()` to accumulate +call-graph edge counts, plus a `SIGPROF`-interval sampler for self-time. +Nothing about that touches hardware counters or kernel permissions. + +A `GPROF=1` gmake flag has been added to `GNUmakefile` for this (folds +`-pg` into `$(PROFILE)`, so it lands in both the compile and link steps +automatically): + +```sh +gmake clean # -pg changes codegen for every TU; must rebuild from scratch +gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 GPROF=1 +``` + +Run the resulting `./typhoon` from whatever directory you want `gmon.out` +written into (it's always named exactly that, in cwd, overwritten on each +run) -- **use `--cpus 1`**, not the usual `MP=1` multi-threaded search, for +this run specifically. See the threading caveat below for why. + +```sh +mkdir -p /tmp/typhoon/profile && cd /tmp/typhoon/profile +/home/scott/typhoon/src/typhoon --cpus 1 --hash 256m --batch --command \ + "force; sd 12; script ../tests/ecm_confident_quick.ep_" +``` + +Then read it with gprof, pointing at the *same* binary that produced it +(symbol addresses are binary-specific): + +```sh +gprof -b /home/scott/typhoon/src/typhoon gmon.out > report.txt +less report.txt +``` + +`report.txt` has two sections: a **flat profile** (self time, sorted +descending -- "what's hot") first, then a **call graph** (who calls whom, +cumulative time) after. The flat profile is usually the right place to +start. Confirmed working end-to-end on this box: a `sd 14` search from the +startpos produced a flat profile correctly topped by `Eval` (16.1% self +time, 74M calls), then `Search`, `SelectBestWithHistory`, +`_GenerateAllMovesBB`, `GenerateMoves`, etc. -- a sane-looking distribution. + +**Don't forget to rebuild the normal release binary afterward** (`gmake +clean` + the standard `GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1` profile, +no `GPROF=1`), same reasoning as `precommit_check.sh`: a `-pg`-instrumented +`typhoon` sitting in the working directory as a side effect of profiling is +easy to mistake for the real release build later. + +### Threading caveat: use `--cpus 1` for gprof runs + +glibc's gprof support predates pervasive multithreading and only reliably +attributes `SIGPROF`-based self-time samples to the thread that set up +profiling (typically the one running `main()`) -- worker threads spawned +under `MP=1`'s multithreaded search can have their CPU time undercounted +or misattributed in the flat profile. Call-graph edge counts (`mcount()` +increments into global memory, not timer-based) aren't subject to this, +but to keep the whole report trustworthy, profile with `--cpus 1` and +treat multi-threaded gprof numbers as unreliable if you ever need to look +at search scaling specifically (use the FreeBSD `pmcstat` path below for +that, or `perf` if the paranoid sysctl ever gets relaxed on this box). + +### Other caveat: inlining + +`-pg` only instruments actual (non-inlined) function calls -- a function +the optimizer inlined into its caller never gets its own `mcount()` call +and shows up folded into the caller's self time, not as its own line. The +flat profile above is still real, just coarser for small/hot helper +functions than it would be with a sampling-based (hardware PMC) profiler. + +## FreeBSD (`wannabe.house`, this engine's native/original platform): pmcstat + gprof + +Verified against the live box 2026-10-05 (session had passwordless ssh to +`[email protected]`, `~/typhoon` there, a separate enlistment from this +one). **The commonly-assumed "needs root" turned out to be only partially +true**: `hwpmc.ko` must be loaded (`kldload hwpmc`, root-only, but +typically already loaded by someone else on a shared box -- check with +`kldstat | grep hwpmc` before assuming you need to ask for it) but once +it's loaded, **process-scoped sampling with `pmcstat -P` and the whole +gprof-conversion pipeline below ran successfully as a plain, non-root user +in this session** -- no `sudo` anywhere. The earlier assumption of +needing root for every run likely came from either testing with `-S` +(system-wide mode, which genuinely does require `PRIV_PMC_SYSTEM` / +root) or from the `hwpmc.ko` load itself. + +This box is AuthenticAMD (`kern.hwpmc.cpuid: AuthenticAMD-23-71-0`), so +Intel-style event names like `CPU_CLK_UNHALTED.THREAD` don't exist. List +available raw events with `pmcstat -L`; the AMD equivalent of "unhalted +cycles" is `ls_not_halted_cyc`. The portable alias `instructions` also +works; the portable alias `cycles` does **not** work for sampling (`-P`/ +`-S`) because it's backed by the TSC, which can't deliver the +overflow interrupt sampling needs -- use `ls_not_halted_cyc` (or another +real hardware event) instead, not the `cycles` alias. + +### 0. Confirm hwpmc is loaded (one-time, needs root if not already) + +```sh +kldstat | grep hwpmc # if this prints a line, skip straight to step 1 +kldload hwpmc # root-only, one-time (until next reboot) +``` + +### 1. Collect samples (process-scoped, not system-wide) + +Use capital `-P` (process-mode sampling, follows only the spawned child) +-- NOT lowercase `-p` (counting mode only, incompatible with `-O`) and NOT +`-S` (system-wide: captures every process on the box, requires root, and +makes symbol resolution in step 2 slow/noisy). + +Append `,<count>` to the event name to set the sampling interval in +cycles -- the default reload count samples far too often and produces +multi-GB output files that take forever to convert. `4000000` (~every 4M +cycles) is a reasonable starting point; increase for a coarser/faster +profile, decrease for finer resolution on a short run. + +```sh +mkdir -p /tmp/typhoon/profile +pmcstat -P ls_not_halted_cyc,4000000 -O /tmp/typhoon/profile/pmc.out -- \ + ./typhoon --cpus 1 --hash 256m --batch --command \ + "force; sd 12; script ../tests/ecm_confident_quick.ep_" +``` + +No `sudo` needed for this step as long as step 0 is satisfied. + +### 2. Convert to a gmon.out-style file, per executable + +`pmcstat -R <file> -g` writes a `*.gmon` file per executable/library seen +in the samples, auto-named, into a subdirectory named after the event +(e.g. `ls_not_halted_cyc/`) under the current directory. (`-O` cannot be +combined with `-R` -- the input file is the positional arg to `-R` +itself.) A `kernel.gmon overflowed` warning is expected/harmless in +process-mode -- ignore it, we only want the typhoon binary's own file. + +```sh +pmcstat -R /tmp/typhoon/profile/pmc.out -g +# -> creates ./ls_not_halted_cyc/typhoon.gmon (plus one per shared lib) +``` + +### 3. View the report with gprof + +FreeBSD's base gprof segfaults on the raw dump produced by `-G <file>` +(a different, two-step raw-dump path) -- skip that entirely. It does +**not** segfault on the proper gmon.out-format file produced by `-g` +above: + +```sh +gprof ./typhoon ls_not_halted_cyc/typhoon.gmon > /tmp/typhoon/profile/report.txt +less /tmp/typhoon/profile/report.txt +``` + +Same two-section report as the Linux path: call graph, then flat profile +(self time, sorted descending) -- the flat profile is usually the more +useful starting point. + +### Gotchas recap + +- `-S` = system-wide sampling, requires root, resolves symbols for + unrelated processes too -- avoid. `-P` = process-scoped, no root needed + once `hwpmc.ko` is loaded -- what we want. `-p` (lowercase) = counting + mode, not sampling, incompatible with `-O`. +- Always set an explicit `,<count>` sampling interval or the output file + balloons to multi-GB and conversion takes forever. Confirmed: an + ~11-second `sd 14` single-position search at the default interval + produced a 300MB+ sample file. +- `-O` and `-R` are mutually exclusive; `-R` takes the input file + directly as its argument. +- `-G` writes a raw dump gprof segfaults on; `-g` writes a proper + gmon.out-format file gprof handles fine. Use `-g`. +- The workload needs to run long enough for `-P` sampling to actually + catch anything -- a single quick `sd 9` search (~1s) produced a fixed + ~3.8KB output file (just the exec/mapping record, zero PC samples) + regardless of the sampling interval used. A `script <suite>.ep_` run + over a whole suite (tens of seconds to minutes, e.g. `ecm_ringers.ep_` + at `sd 10`, ~25s) or a single deep `sd 14`+ search (confirmed: ~11s) is + long enough to get real samples. +- **Not every event supports `-P` sampling** (some AMD PMC events are + counting-only). Branch-misprediction events (`ex_ret_brn_misp`, + `ex_ret_cond_misp`, etc.) work fine in counting mode (`pmcstat -p + ex_ret_brn_misp -- <cmd>`, prints a single total at exit) but produce + the same fixed ~3.8KB/zero-samples output as the "too-short workload" + case even against a long run -- a real per-event limitation, not a + workload-duration problem (confirmed by running the known-good + `ls_not_halted_cyc` event against an identical command/interval/ + duration, which produced immediate real samples). Likely cause: not + every AMD PMC event supports overflow-interrupt (PMI) delivery, which + `-P`/`-S` sampling requires -- some are counting-only, and neither + `pmcstat -L` nor `pmccontrol -L` distinguish which is which. The only + way to find out is to try `-P` against a known-long-enough workload and + check the output file size. If a per-function mispredict breakdown is + ever actually needed, look into whether a PEBS-like interrupt-capable + event exists for this, before assuming it's simply not possible. + +## Which to reach for + +- **Linux, day-to-day**: `gmake GPROF=1` + gprof. No privilege + dependency, no sysadmin step, works today. +- **FreeBSD (`wannabe.house`)**: `pmcstat -P` + gprof, after confirming + `hwpmc.ko` is loaded. Hardware-counter-based (actual cycles/ + instructions, not call-count-overhead-inflated like `-pg`), so more + trustworthy for small/hot inlined-adjacent functions, and not subject + to the multithreading self-time caveat gprof's `SIGPROF` approach has + on Linux -- but still process-scoped, so still only reflects the one + profiled process, not the whole system. +- **If Linux `perf` access ever gets relaxed** (someone lowers + `kernel.perf_event_paranoid` or grants `CAP_PERFMON`), it would be the + best of both: hardware counters *and* correct multithreaded attribution, + without `-pg`'s inlining blind spot. Worth revisiting if a future + profiling question specifically needs multi-threaded search behavior + under `MP=1`. diff --git a/src/docs/TESTING_PLAN.md b/src/docs/TESTING_PLAN.md new file mode 100644 index 0000000..7d2bfc2 --- /dev/null +++ b/src/docs/TESTING_PLAN.md @@ -0,0 +1,298 @@ +# Testing plan: closing coverage gaps in test*.c + +Written 2026-09-16 after an audit of every `src/test*.c` file and how +(or whether) each is wired into `main.c`'s self-test sequence. Goal +per Scott: not more test code for its own sake -- find the +years-old bugs that asserts have been missing. Priorities: search, +eval, generate (not exclusively). Perf testing is also explicitly +in scope. + +## Dead/disconnected infrastructure (fix first -- near-zero cost) + +Several tests already exist but don't run. Recovering this coverage is +strictly cheaper than writing anything new, so do this before anything +else below. + +1. **SEE has zero active correctness testing.** `testsee.c`'s own + header comment admits it: `DebugSEE`/`DebugSEERecursive` (a + brute-force make/unmake exchange-walk oracle meant to check `SEE()` + against) is wrapped in `#ifdef TEST_BROKEN` -- a define nothing + sets. `testsee.c`'s only *called* functions (`TestGetAttacks`, + `TestIsAttackedBB`) test attack generation, not SEE. Given SEE is + what `GetMoveSee`/extensions/QSearch pruning all trust (and is the + exact code reworked in commit c0ac3d7), this is the single + highest-value fix here. Get `TEST_BROKEN` -> live (`TEST`), fix + whatever bitrot surfaces, and confirm it's called from `main.c`. +2. **`TestEvalWithKnownPositions`** (`testeval.c`, ~20 hand-curated + "side X should be better here" positions from chess literature) is + defined but **never called from anywhere** -- not `main.c`, not + `command.c`. Wire it into the self-test sequence. +3. **`TestRecognExhaustiveKNKP`** (`testrecogn.c`, exhaustive KNKP + endgame-recognizer enumeration) is defined and prototyped in + `chess.h` but referenced only in a comment -- never invoked. Only + the smaller table-driven `TestRecogn` runs. Wire it in. +4. **The hash table has no correctness self-test at all.** + `testhash.c` contains `SanityCheckHashEntry` (an entry-shape + validator) and `AnalyzeFullHashTable` -- but the latter is a + post-search *diagnostic dump* called from `root.c`, not a test, and + it's never invoked from the `TEST` self-test sequence. See "New + test: hash correctness" below for what should replace this gap, not + just a wiring fix. +5. **`TestEvalWithSymmetry`** only runs a *single* random symmetric + position per invocation, and only when a human types `eval` at the + prompt in an `EVAL_DUMP` build (`command.c:2203`) -- not part of the + automated suite, never loops. See "New test: eval symmetry fuzz" + below. + +## New tests, by area + +### Search (thinnest area relative to its bug surface) + +- `TestSearch` (`testsearch.c`) runs 20 random positions at fixed + **depth 2** only, checking solely "returned move is legal" and + "didn't blow the time limit." + - **DONE -- hash-consistency test** (`TestSearchHashConsistency`, + testsearch.c): searches a position to depth D twice in a row with + a warm hash table (scores must match), and separately confirms a + depth-D2 search seeded by a shallower depth-D1 warm-up agrees with + a from-scratch depth-D2 search. Wired into the `TEST=1` self-test + sequence (main.c). + - **DONE (partially) -- QSearch sanity test** + (`TestQSearchSanity`, testsearch.c): checks search termination + (no `INVALID_SCORE`/garbage) and a whole-search node-count + ceiling. Caveat found and documented in the function's own header + comment: the self-test suite builds with plain `gmake TEST=1`, + which doesn't define `PERF_COUNTERS` -- every `sCounters` field + (including `u64QNodeCount`, the qsearch-only counter this was + originally meant to check) is wrapped in `chess.h`'s `INC()` macro, + a no-op without that define. Confirmed live (`sd 3; go` on a + `TEST`-only binary reports "0 qnodes" regardless of real qsearch + activity). The test currently checks + `ctx->sCounters.tree.u64TotalNodeCount` instead (root.c's + unconditional per-node `++`, always live) -- a real but strictly + weaker signal, since it can't distinguish "qsearch exploded" from + "full-width search exploded." Building a `PERF_COUNTERS`-enabled + `TEST` profile (or adding an always-live qsearch-only counter) is + the natural follow-up to close this for real. The hand-auditable- + tree idea (`Trace()` every qsearch node on a small position) is + still worth doing but wasn't implemented this session. + - **NOT DONE -- determinism test** (same position/depth searched + twice, PV must match): explicitly descoped -- Scott's call is that + a stable-PV requirement makes the test break every time move + ordering changes for unrelated reasons, which is just noise, not a + real regression signal. A same-*score* (not same-PV) check is + already covered by the hash-consistency test above. + - **NOT DONE -- MP-vs-single-thread agreement test**: still open. + `split.c` has no dedicated correctness test. + - **DONE (as an EPD suite, not a C test) -- null-move/zugzwang + soundness**: `tests/zugzwang.epd` (5 positions, run via the new + `epd` command -- see below) checks actual best-move outcome + instead of an internal counter. An earlier attempt asserted + against `ctx->sCounters.extension.uZugzwang` (the verification- + search hit counter) using the 3 classic literature zugzwang + positions, but none of them structurally reach that code path on + this build: 2 have too little material for + `WeShouldTryNullmovePruning`'s own `> 2` non-pawn-count gate to + attempt null-move pruning at all, and the 3rd (despite landing in + the eligible window) gets resolved directly off this box's local + Syzygy tablebase before search ever touches null-move logic -- + confirmed live via the `TABLEBASE MOVE` trace line. Removed in + favor of the EPD suite, which doesn't care about internal + mechanism, only the final answer. See `testsearch.c`'s comment + where `TestNullMoveZugzwang` used to be for the full writeup. + +**New infrastructure that came out of chasing the above (search.c +touches none of this, but it's now how any future EPD-based search +test should be added):** + +- **`epd <file>` engine command** (`EpdCommand`, script.c) parses and + runs a raw EPD suite file directly -- no perl preprocessing step + first. Backed by `ParseEpdRecord` (fen.c), real per-record EPD + parsing (4 position fields, optional FEN-style halfmove/fullmove + counters, then `;`-separated `bm`/`am`/`id` opcodes). +- **Found and fixed while building this**: `tests/convert_epd.pl` (the + old perl-based EPD-to-script converter) had a real, silent- + corruption bug -- its `bm`/`am` regex used `[^b]+` as the filler + between the board and the opcode, which breaks on *every* + Black-to-move record (the side-to-move field is literally the + character `b`), and since `$fen`/`$id`/`$solution` were unscoped + globals rather than per-line locals, a failed match silently carried + the *previous* record's values forward instead of erroring. + Confirmed live: feeding it a 5-position batch with one Black-to-move + entry produced a byte-for-byte duplicate of the previous record for + that one line, no diagnostic at all. The script is left in place + (marked superseded in its own header) in case something still shells + out to it, but new suites should use `epd` instead. + Cross-checked the new parser against the old pipeline on + `ecm_ringers`: `script` (old `.ep_` format) and `epd` (new + `tests/ecm_ringers.epd`, produced by `ep_to_epd.pl`, see next item) + produced byte-identical node counts and solve stats. +- **`tests/ep_to_epd.pl`**: one-off reverse converter, turns the + existing `setboard`/`solution`/`avoid`/`id`/`go` script-format + suites back into real EPD now that `epd` can read it directly. Run + against every genuine tactical suite in `tests/` (not `bench.ep_` or + `trial.ep_`, which use `sd`/`st`/`easy`/`quit` and aren't EPD + underneath) -- produced `.epd` siblings for `arasan`, + `ecm_confident_quick`, `ecm_confident_sample2`, `ecm_confident`, + `ecm_hard_quick`, `ecm_hard`, `ecm_late`, `ecm_quick`, `ecm_ringers`, + `ecm-gcp`, `ecm-hard`, `ecm`, `ecm016_only`, `LCT2`, `mes400`, + `newiq`, `quick`, `swiss`, `wac`. `ecm.epd` came out to 879 records, + matching the `606/879`/`622/879` solve-count baselines already + quoted elsewhere in CLAUDE.md. + +### Eval + +- Automate `TestEvalWithSymmetry` into a real fuzz loop (N random + symmetric positions, assert `Eval() == 0` for all, as part of the + standard self-test run) instead of the current single-shot + manual-only invocation. +- **Adversarial/degenerate-position term-bounds test**: 0-pawn + positions, 9-queen positions, fully-open-board positions -- extend + `GenerateRandomLegalPosition` or hand-build cases, assert no eval + term produces an out-of-range score (reuse the existing + `EvalTrace`/`EVAL_DUMP` term-tracing machinery for this). +- Audit for any **incrementally-maintained eval state** (pawn hash, + king-safety cache) and add a from-scratch-recompute-vs-incremental + consistency check if any exists -- classic "works until the hash + collides" bug class. +- **Eval speed regression test**: no benchmark currently exists for + eval itself despite it being the named NPS bottleneck vs. Crafty + (see CLAUDE.md). Same interleaved-comparison shape as the existing + `TestGenerate*Speed` benchmarks, broken down by game phase. + +### Generate (best-covered area, but gaps remain) + +- Perft (`TestMoveGenerator`) only checks to **ply 4** even though + `uLeaves[]` already has reference values to ply 6/7 sitting unused -- + extend to a `--perft-deep` slow path, and add perft-**divide** + (per-root-move leaf breakdown) to localize failures to a specific + move instead of just tripping a panic. +- Add perft positions specifically chosen to stress **en passant, + underpromotion, and castling-through-check** individually (the + current 5 FENs give general coverage but weren't chosen for this). +- `TestGenerateAllMovesSpeed` and the per-piece-type speed tests are + **pure microbenchmarks with no correctness assertion** -- they print + cycles and never diff the move *sets* produced by the mailbox vs. + bitboard generators being compared. Add a sorted-move-list-equality + assertion alongside each (both generators already run back-to-back + in the same loop, so this is nearly free) -- otherwise a bitboard + generator silently missing a move class just prints a suspiciously + fast number and passes. +- `GetMoveSee`'s new non-pawn-capture SEE path (c0ac3d7) has no + dedicated test yet -- covered by the SEE oracle fix above once live. + +### Move/Position plumbing + +- Reasonably solid already (`TestMakeUnmakeMove`, `TestExposesCheck`, + `TestIsAttacked`). Add a **long-game fuzz test**: play N random games + of ~200 plies each (not just 1-ply-at-a-time exercises), asserting + `VerifyPositionConsistency` and `ComputeSig`-from-scratch agreement + with the incrementally-maintained signature at every ply. Confirm + `VerifyPositionConsistency` is actually compiled live in the + `TEST`/`DEBUG` build, not just present in source. + +### Hash (currently zero correctness coverage) + +New test needed, not just a wiring fix: +- Store/probe roundtrip correctness. +- Replacement-scheme behavior under forced collisions (fill table past + capacity with distinct signatures, confirm replacement policy + behaves as designed, not just "doesn't crash"). +- `ADJUST_MATE_IN_N` ply-distance adjustment on `HASH_FLAG_EXACT` + entries specifically -- the exact mechanism CLAUDE.md flags as + audited-by-hand but not test-covered after this session's `ASSERT` + removals in `QSearch()`. + +### Recognizer + +- Wire in `TestRecognExhaustiveKNKP` (see dead-infra list above). + Consider the same exhaustive-enumeration treatment for any other + small recognized endgame class in `recogn.c` (KPK, KRK) if one + exists without a test. + +## Perf testing + +Current coverage is narrow: five isolated per-piece-type +mailbox-vs-bitboard microbenchmarks (`TestGenerate*Speed`), nothing for +search, eval, or hash. + +- **Eval NPS/cycles-per-call benchmark**, same interleaved-comparison + shape as the existing generate benchmarks, by game phase -- eval is + the named bottleneck (CLAUDE.md's 1.5M vs 7.5M nps gap vs. Crafty) + and has no dedicated speed test today. +- **Whole-search NPS benchmark**: `bench.c` already exists for this -- + confirm whether it's part of the automated `TEST` self-test pass or + purely a manual `bench` command; if manual-only, wire a + fixed-node-budget version into the automated sequence so a + regression surfaces without someone remembering to run it by hand. +- **Hash-probe-under-load benchmark**: cycles/probe at low occupancy + vs. near-full table -- replacement-scheme cost can regress silently + today with nothing to measure it. +- **Stored baselines with pass/fail thresholds**, not print-only + output: today every perf test prints cycles/call and leaves + interpretation to a human reading the log. Store a last-known-good + baseline (same idea as `head_reference/` for solve-counts) and fail + loud if a call site regresses past some tolerance. + +## Suggested priority order + +1. Un-bitrot the three dead tests (SEE oracle, + `TestEvalWithKnownPositions`, `TestRecognExhaustiveKNKP`) -- + near-zero cost, immediate coverage recovery. +2. Hash-table correctness test (store/probe/replace/mate-adjust) -- + currently the single biggest correctness blind spot. +3. Search determinism + MP-vs-single-thread agreement test -- targets + the exact class of bug most likely to hide in threading/hash/ + extension interactions that per-function unit tests can't see. +4. Extend perft to deeper plies + divide output; add move-set-equality + assertions to the existing generate speed benchmarks (turns + dead-weight microbenchmarks into correctness tests for free). +5. Eval: automate the symmetry fuzz test (loop N, not 1) and add + adversarial/degenerate-position term-bounds checks. +6. Round out perf testing per the list above, with stored baselines + and pass/fail thresholds instead of print-only output. + +## "Fuzz" testing + +**Implemented: `fuzz_test.sh [binary] [depth] [batch_size] [cpus]`** +(defaults: `./_typhoon`, `sd 6`, 200 positions/batch, `--cpus 2`). Loops +forever: each iteration draws a fresh random batch of `setboard`/`go` +records from the full 400,000-line / 200k-position +`tests/cross_sectional_200k.ep_` pool, runs them through a `DEBUG=1` +build under gdb, and scans the gdb output for a crash/assert signature +before starting the next batch. Stops (exit 1) and prints the offending +batch + engine log paths as soon as one is found; otherwise runs +indefinitely (`^C` to stop). Logs land under `/usr/local/tmp/typhoon/fuzz/` +(one sampled `.ep_` batch + one `.gdb.txt` + one engine `--logfile` per +iteration). + +**Two build/invocation bugs found and fixed getting this working, neither +a sandbox limitation despite first appearances:** + +1. **The DEBUG binary must be built with `MP=1` as well as `DEBUG=1`** + (`gmake clean; gmake DEBUG=1 MP=1`) -- `--cpus` is compiled in only + under `#ifdef MP` (`main.c:311`). A `DEBUG=1`-only build silently + treats `--cpus` and its numeric argument as two separate unrecognized + arguments (`main.c:400`'s "Error (unknown argument)" path) and falls + back to the 1-cpu default with no hard failure -- easy to miss, and it + also means `split.c`'s parallel-search code never actually links in, + defeating the point of passing `--cpus 2` at all. +2. **`gdb -batch -ex "handle SIGTRAP nostop noprint pass" -ex run ...` + breaks thread control against this MP/pthread build specifically**, + producing `ptrace (PT_SUSPEND): Operation not permitted` followed by + `Child process unexpectedly missing: No child processes` -- looked + exactly like a sandboxed environment blocking ptrace outright (and was + initially misdiagnosed as that), but reproduced identically outside + any harness/sandbox, and disappeared entirely once that one `-ex` line + was dropped. `fuzz_test.sh` omits it; a plain `run`/`bt full`/ + `info registers` gdb batch invocation debugs the MP binary fine, + including catching a genuine `int3` breakpoint from a failed `ASSERT`. + +Not yet done: no crash has been found by a run so far, so the +crash-detection path (the `grep -qE` in `fuzz_test.sh`) is exercised by +its own logic reading correctly, not yet by a real hit. Worth periodically +running this for stretches (it's unbounded/`^C`-driven, unlike the other +gates) rather than treating it as a one-time pass/fail check like +`debug_smoke_test.sh`. See also: `./debug_smoke_test.sh` (fast, curated +sample, single pass/fail) for the pre-commit-sized version of the same +idea. diff --git a/src/docs/UNIFY_SEARCH.md b/src/docs/UNIFY_SEARCH.md new file mode 100644 index 0000000..f38970b --- /dev/null +++ b/src/docs/UNIFY_SEARCH.md @@ -0,0 +1,406 @@ +# Unifying the per-move search kernel in `search.c` and `split.c` + +Status: **plan only, not started.** Do as the only change in a commit; +test with `precommit_check.sh` plus the `head_reference` node-count gate +below at each step. Line numbers below are current as of this revision +(EFP's explicit per-move pruning block has been removed from `Search()`; +only stale names remain — see "EFP is gone" below). + +## Motivation + +`Search()`'s main move loop (`search.c:909..1780`) and `HelpSearch()` +(`split.c:1058..1283`) independently implement what is supposed to be the +same per-move logic: make the move, compute its extension, decide whether +to prune or reduce it, search it with a PVS window, re-search if a reduced +move fails high, unmake it. They have drifted apart because every +improvement to this logic has been written once, in `search.c`, and not +reliably ported to `split.c`. Concretely, today: + +- `Search()` has LMP (late move pruning, via `ShouldLatePrune`/the joint + FH% table). `HelpSearch()` does not. +- `Search()` gates reduction/pruning eligibility through the single + `CanWeReduceOrPruneMove()` (searchsup.c:783), which the routine's own + comment documents as 9 base checks plus 2 move-identity exemptions + (killer-adjacency, countermove match). `HelpSearch()` (split.c:1187-1193) + has a hand-rolled 7-condition gate that is missing: the SEE-negative- + capture carve-out, the mate-magnitude-alpha check, the `fThreat` check, + and both exemptions. **This is a bigger behavioral gap than "HelpSearch + lacks LMP"** — `HelpSearch()` is currently reducing/pruning-eligible on + moves `Search()` would exempt outright. +- `Search()` passes real `bbMinorAttacksSnapshot`/`bbRookAttacksSnapshot` + attack-bitboard snapshots into `EstimateHangingMaterialCp`. `HelpSearch()` + passes zeroed snapshots (`split.c:1215-1216`), degrading its hang + estimate (and, by extension, `ShouldLatePrune`'s joint-table lookup once + step 4 below gives it one). +- `split.c:1245` and `split.c:1268` write `ctx->sPlyInfo[ctx->uPly] + .iExtensionAmount` — but both writes happen **after** `MakeMove`, i.e. + after `ctx->uPly` has already advanced (confirmed: `search.c:465` binds + `pi = &ctx->sPlyInfo[ctx->uPly]` *before* the move loop's `MakeMove` + calls; `move.c:928` is where `MakeMove` increments `ctx->uPly`). + `split.c:1156`'s own `ComputeMoveExtension` call already compensates + with `uBegin[ctx->uPly - 1]` one line away, so the "-1 after MakeMove" + pattern is known and applied inconsistently, not unknown. **This is two + separate bugs, not one:** + 1. Wrong slot: writes the child's `iExtensionAmount`, not the mover's — + invisible to the null-move history-reduced-bailout check one ply + down (searchsup.c). + 2. Stale-on-skip: `search.c:1386`'s own comment documents that it used + to only write `pi->iExtensionAmount` when `iExtend < 0`, and that was + itself a bug (a later sibling move could read a stale reduction left + by an earlier one) — fixed in `Search()` by writing it + unconditionally every iteration. `split.c:1245`/`1268` still only + write inside `if (iLMR < 0)` / the re-search branch, i.e. still have + the stale-on-skip bug `Search()` already fixed once. + Both must be fixed together: unconditional write, to + `ctx->sPlyInfo[ctx->uPly - 1].iExtensionAmount`. + +## EFP is gone — update, don't carry forward + +A prior session removed EFP's explicit per-move futility-pruning block +from `Search()`'s move loop entirely. Confirmed by search: `ShouldFutilityPrune`/ +`FutilityPrun*`/`StaticNullMove`/`ShouldStaticNull` have zero hits anywhere +in `search.c`/`searchsup.c`/`split.c`. Only inert names remain: +`EFP_FH_MIN_SAMPLES` (search.c:84, unused #define), `_PostReduceDepthBucketEfpAligned` +(a depth-bucketing helper, name is now just vestigial), and +`fAnyMoveEFPPruned` (search.c:1851, read in an `else` branch comment on +the store-upper-bound path — confirm whether this flag can still ever be +set anywhere before relying on it in the extraction; if nothing sets it +any more it's dead and should be dropped in step 0 below, not carried into +the shared kernel). + +This also **resolves the doc's previous closing question** ("unify EFP and +LMP before this project") — there's no EFP left to unify; `ShouldLatePrune`'s +joint FH% table is the sole pruning mechanism now. Drop that requirement. + +It also means the previously-planned "related gap" about `pi->uPVDistance` +never written in `HelpSearch()` is **moot**: `uPVDistance` has zero hits +anywhere in the codebase now (it was EFP-era instrumentation, removed +along with EFP). Nothing to port. + +## What should NOT be extracted + +The two loops differ along three axes that are not just stylistic — they +reflect genuinely different control/data flow, and trying to force them +into one shape will make the result worse, not better. + +1. **Move acquisition.** `Search()` pulls from a staged iterator (hash + move, killers, generated moves, ..., tracked via index `x`). + `HelpSearch()` pulls from a shared move pool via `_GetNextParallelMove`, + which also hands back the *current* shared `iAlpha`/`iBestScore` + (another helper thread may have raised them since this thread's last + iteration). This is loop-header logic. **Stays in each caller.** +2. **Result publication.** `Search()` keeps `iAlpha`/`mvBest` as plain + thread-local variables, calls `UpdatePV()`, and uses `goto end`/`goto + no_more_moves`. `HelpSearch()` publishes through the locked + `_UpdateSplitInfo(ctx, mv, iScore, u)` and checks `g_SplitInfo[u] + .fTerminate` after every move. Different synchronization models. + **Stays in each caller.** +3. **The split-launch decision itself** (`search.c:1082-1173`, + `StartParallelSearch` and the `g_fCanSplit`/`g_uNumHelpersAvailable` + gating). Must stay outside the extracted kernel, or a helper thread + executing the kernel could attempt to launch a nested split, which the + current design does not expect. **Stays in `search.c` only.** + +## What SHOULD be extracted + +The inner per-move kernel, called once per move after it's been pulled off +whichever move source the caller uses — current location +`search.c:1175-1681` (SEE computation through `UnmakeMove` and the +`iCumulativeExtend` restore), immediately before the publication tail that +starts at `search.c:1684`: + +``` +SEE (pre-move, on pos) + -> MakeMove + -> ComputeMoveExtension + -> CanWeReduceOrPruneMove gate + -> ShouldLatePrune (LMP) + -> GetLMRReduction (LMR) + -> PVS search (null-window, then full-window re-search if it raises alpha) + -> re-search to full depth if a reduced move fails high + -> UnmakeMove +``` + +## The instrumentation problem (the design question the old draft missed) + +The span to extract is NOT a clean make/decide/search/unmake block with no +side channels. The publication tail at `search.c:1684-1777` reads several +locals the kernel's interior sets, and `JOINT_EXTRA_VERIFY`'s shadow-search +block sits *between* the PVS re-search and `PRUNE_VERIFY`'s fail-high +check, inside the make/unmake span — the instrumentation can't be +boundary-excluded without duplicating the real move-handling logic around +it. Locals that cross the proposed kernel/caller boundary in both +directions: + +- Caller → kernel (already loop-invariant, computed once outside the + per-move loop; pass as params, don't recompute): `uRawDepthB` + (`search.c:993`), `uRawImproveB`, `fNodeSplitEligible`. +- Kernel → caller (read by the publication tail at 1684+, must survive the + call): `fIsLeftoverMove`, `fMoveIsPruneReduceEligible`, + `uThisMoveHangBucket`/`GapBucket`/`NumBucket`/`DepthBucket` (all + `PERF_COUNTERS`/`FH_TABLE_COLLECTION`-gated), `fThisMoveIsCountermoveMatch`/ + `uCMEvidenceBucket` (`MOVE_ORDERING_EVIDENCE_COLLECTION`-gated), + `fThisMoveJointExtraEligible`/`uThisMoveJointExtraTier` + (`PERF_COUNTERS`-gated), `fVerifyFlagged`/`uVerifyBand`/`uVerifyDepthB`/ + `uVerifyAttB`/`uVerifyRateB` (`PRUNE_VERIFY`-gated), and the `TEST_NULL` + node-cost pair. + +**Decision needed before writing code, not discovered mid-extraction**: +bundle all of this into one `OUT` struct (e.g. `MOVE_KERNEL_DIAGNOSTICS`, +`#ifdef`-sectioned to match each field's existing gate) that the kernel +fills in and the caller's publication tail reads — rather than a long +flat parameter list. The user's note below ("put what you can in +`PLY_INFO`") is the other half of this: some of these (the bucket/tier +values) are genuinely per-move-at-this-ply state and belong as fields on +`ctx->sPlyInfo[ctx->uPly - 1]` (indexed the same way `iExtensionAmount` +already is) rather than threaded through call/return at all — this +removes most of the diagnostics-struct bulk and gives `HelpSearch()`'s +own ply slot the same fields for free. Resolve this split (struct vs. +`PLY_INFO` fields vs. a mix) as part of scoping the extraction, before +step 3. + +## Two sequencing traps that will break the "bit-identical node counts" gate + +- `pf->iCumulativeExtend += iExtend` (search.c:1387) happens *after* the + LMP-prune `continue` (search.c:1313-1315) — the prune path `UnmakeMove`s + but never increments/decrements `iCumulativeExtend` for that move at + all. A `MOVE_KERNEL_PRUNED` return must preserve that asymmetry exactly; + don't "clean up" by making prune and search paths symmetric around + `iCumulativeExtend`. +- `WE_SHOULD_STOP_SEARCHING` (search.c:1684) is checked *after* + `UnmakeMove` and *after* the `iCumulativeExtend -=` restore. A + `MOVE_KERNEL_ABORTED` return must not skip either of those — the kernel + itself must still fully unwind (unmake + restore) before reporting + abort, not bail out of the middle of its own cleanup. + +## Proposed signature + +```c +typedef enum _MOVE_KERNEL_RESULT +{ + MOVE_KERNEL_SEARCHED, // move was made, searched, and unmade; iScore valid + MOVE_KERNEL_PRUNED, // move was late-move-pruned; not made; iScore invalid + MOVE_KERNEL_ABORTED // search was stopped mid-move; iScore is a partial result +} MOVE_KERNEL_RESULT; + +FORCEINLINE MOVE_KERNEL_RESULT +SearchOneMove(SEARCHER_THREAD_CONTEXT *ctx, + MOVE mv, + ULONG uDepth, + SCORE iAlpha, + SCORE iBeta, + SCORE iEval, + ULONG uLegalMoves, + FLAG fIsLeftoverMove, + FLAG fFullWindow, + FLAG fThreat, + ULONG uRawDepthB, + ULONG uRawImproveB, + BITBOARD bbMinorAttacksSnapshot[2], + BITBOARD bbRookAttacksSnapshot[2], + OUT SCORE *piScore, + OUT INT *piExtend); + // Diagnostics (hang/gap/num/depth buckets, countermove-evidence + // bucket, joint-extra tier, prune-verify band) are NOT parameters -- + // see "the instrumentation problem" above. They land in + // ctx->sPlyInfo[ctx->uPly - 1] (written by the kernel before it + // unmakes) wherever that's a clean fit, or an OUT diagnostics struct + // otherwise. Finalize the split while scoping step 3, not here. +``` + +Notes on why each non-obvious parameter is there: + +- **`fFullWindow` is explicit, not inferred.** `Search()`'s current code + infers "is this the first move at this node" from the sentinel + `iBestScore == -INFINITY_SCORE` (search.c:1396). That sentinel is only + true for `Search()`'s actual first move. `HelpSearch()` must never take + that branch — a helper thread's first move off the shared pool is *not* + the node's first move — and today this is only safe because + `_GetNextParallelMove` happens to never hand back `-INFINITY_SCORE`. + Once this logic is shared, that implicit contract becomes load-bearing + in a way that won't assert if violated. Making it an explicit + caller-supplied flag removes the trap. +- **`uLegalMoves` is a parameter, not computed inside.** It feeds + `MoveNumBucket`/`ShouldLatePrune`'s floor and `CanWeReduceOrPruneMove`'s + own `uLegalMoves > 1` check. `search.c` has it as a running counter; + `split.c` must pass `g_SplitInfo[u].uAlreadyDone + uMoveNum + 1` + (matching what it already computes locally as `uSplitLegalMoves` at + split.c:1217 — confirm this matches the `> 5` floor `split.c:1189` + hardcodes today, which should become a call into the shared + `CanWeReduceOrPruneMove` instead of staying hand-rolled). Getting this + wrong silently changes LMP/LMR aggressiveness with no assert to catch + it. +- **`fThreat` must be plumbed through `g_SplitInfo`.** `CanWeReduceOrPruneMove` + takes `fThreat` as a real gating input (searchsup.c:783) but + `chess.h:1612-1662`'s `SPLIT_INFO` struct has no such field today — add + one, populated by the originating thread at split-launch time (same + pattern as `iEval`/`iImprovement`), or `HelpSearch()` can't call the + real gate at all. +- **Attack snapshots are real parameters, not internally zeroed.** This is + the fix for the degraded-hang-estimate gap above: `HelpSearch()` must + start passing real snapshots instead of the zeroed + `bbNoMinorAttacksSnapshot`/`bbNoRookAttacksSnapshot` it uses today + (split.c:1215-1216). Requires extending `g_SplitInfo[]` to carry + `bbMinorAttacksSnapshot`/`bbRookAttacksSnapshot` from the originating + thread (already computed in `Search()` before `StartParallelSearch`). + **Open risk, worth raising with the user explicitly (not just noting):** + once the kernel is shared, `FH_TABLE_COLLECTION`'s sampling + (search.c:1323-1365) fires from helper threads too. If step 4 (unify) + lands before step 5 (real snapshots), those samples get logged with + zeroed snapshots — training the joint FH% table on degraded inputs, not + just giving `HelpSearch()` a degraded *decision*. **Recommendation: land + step 5 together with step 4, not as a follow-up**, or gate + `FH_TABLE_COLLECTION`'s increments to the originating thread only + (`ctx->uPly` vs. some "is this a split helper" check) until step 5 lands. + Ask the user which they'd rather do before step 4. +- **Return value is an enum, not control flow via `goto`.** The kernel + can't `goto end`/`goto no_more_moves`/`continue` — those targets don't + exist in the extracted function. Each caller's own loop dispatches on + the enum: `Search()` on `PRUNED` does `continue` (and must NOT touch + `iCumulativeExtend`, see the sequencing trap above); on `ABORTED` it + does its `goto end` with `iBestScore = iScore`; `HelpSearch()` on + `ABORTED` does `break`. The abort *condition* itself also differs + (`WE_SHOULD_STOP_SEARCHING` vs. `g_SplitInfo[u].fTerminate`) — stays a + caller-side check made around the kernel call, not a predicate pushed + into it. + +## Suggested sequencing + +0. **Housekeeping**: confirm whether `fAnyMoveEFPPruned` (search.c:1851) + can still be set anywhere post-EFP-removal; if dead, delete it rather + than carry it into the shared kernel's scope. Small, standalone. +1. **Standalone fix**: `iExtensionAmount`'s two bugs in `split.c` (wrong + slot + stale-on-skip, see Motivation above) — write + `ctx->sPlyInfo[ctx->uPly - 1].iExtensionAmount = iExtend` unconditionally, + every iteration, matching `search.c:1386`'s already-fixed convention. + Commit on its own, verified via `precommit_check.sh`. +2. **Extraction**: pull the kernel out of `search.c` into a new shared + header (`search_inline.h`, `FORCEINLINE` across translation units + requires a header), make `Search()` call it, with `HelpSearch()` + untouched (still running its own inline copy, including its own + gap-riddled gate). This isolates "did the extraction change + `Search()`'s behavior" from "did unifying the two callers change + `HelpSearch()`'s behavior." Resolve the instrumentation-placement + question (PLY_INFO fields vs. diagnostics struct) as part of this + step, before writing the header. +3. **`g_SplitInfo` plumbing**: add `fThreat` and the two attack-snapshot + bitboards to `SPLIT_INFO` (chess.h), populated at split-launch time. + Do this *before* step 4, not after — see the `FH_TABLE_COLLECTION` + risk above. +4. **Unification**: make `HelpSearch()` call the same kernel via the real + `CanWeReduceOrPruneMove()` gate (replacing its hand-rolled 7-condition + version) and `ShouldLatePrune`, removing its inline copy entirely. This + is the commit that actually changes `HelpSearch()`'s behavior — it + gains LMP, the full 9-check+2-exemption gate, and real attack + snapshots, all at once, since step 3 already landed the plumbing they + need. + +## Verification plan + +Standard gates at every step (`precommit_check.sh` — `TEST=1` self-test +suite + `debug_smoke_test.sh`), plus: + +1. **Isolate the single-threaded path first (after step 2).** Force + `g_uNumHelpersAvailable = 0` (no split ever fires) and confirm **node + counts are bit-identical to HEAD** on the three curated suites + (`ecm_ringers`, `ecm_confident_quick`, `ecm_hard_quick`) at a fixed `sd` + depth, per the `head_reference` protocol in CLAUDE.md. This should be a + pure refactor at this stage — "no change whatsoever," not "no + significant change." Any node-count delta here is a real extraction + bug, not noise. Watch specifically for the two sequencing traps above + (`iCumulativeExtend` on the prune path, `WE_SHOULD_STOP_SEARCHING` + ordering) — those are exactly the kind of thing that produces a subtle + node-count drift instead of a crash. +2. **After step 4, re-enable helpers and compare separately**, expecting a + real difference this time (`HelpSearch()` now has LMP, the full gate, + and real snapshots it didn't have before). Compare solve counts and + fail-high rates across the three curated suites per CLAUDE.md's + existing methodology (measure the pick, not just the game). Rebuild + `head_reference/` as the new baseline once this change is committed. +3. Run the full `precommit_check.sh` gate before every commit in the + sequence above, not just at the end — this is a search/move-generation + change throughout. + +## Performance: `Search()` is the current profiler hotspot + +Confirmed with the user: `Search()` is the single hottest function in the +program right now. That changes the risk profile of this refactor from +"pure correctness/maintainability" to "must not cost NPS" — a few concrete +implications for how to do each step: + +- **`FORCEINLINE` is load-bearing, not decorative, here.** The whole + point of extracting into a header function instead of a normal + `.c`-file one is that the compiler can fully inline `SearchOneMove` at + both call sites and optimize it in the context of each caller's + surrounding code (register allocation across the call, dead-branch + elimination of whichever `#ifdef` blocks aren't compiled in) exactly as + if the code were still written inline. If it isn't actually inlined in + the shipping `GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1` build, this + refactor becomes a real regression (a non-inlined call in the hottest + function in the program), not a neutral one. Don't just assert + `FORCEINLINE` and move on — check post-step-2 that `clang` actually + inlines it (`-Rpass=inline`/`-Rpass-missed=inline`, or inspect + `nm`/disassembly for a surviving `SearchOneMove` symbol) before trusting + the bit-identical-node-count gate alone; a node-count match doesn't + prove NPS didn't drop. +- **Don't let the diagnostics-boundary decision regress the hot path.** + The "instrumentation problem" section above already flags that several + locals need to cross the kernel/caller boundary. For the fields that + move into `ctx->sPlyInfo[ctx->uPly - 1]`: writing them is already in + the hot path today (`iExtensionAmount` already lives there), so this is + likely free. For anything kept as an `OUT` diagnostics struct instead: + every field in that struct gets written on every move in the + `PERF_COUNTERS`/`FH_TABLE_COLLECTION`-eligible path even when the + calling code only reads a couple of them, and the struct itself is a + stack object passed by pointer — prefer the `PLY_INFO`-field approach + wherever a field is genuinely per-ply state (most of them), and reserve + an `OUT` struct only for values with no natural per-ply home. This is + exactly the tradeoff the "instrumentation problem" section says to + resolve before writing code — performance is now part of *why* that + decision matters, not just design cleanliness. +- **Don't widen the non-`PERF_COUNTERS` build's hot path.** The + `SearchOneMove` signature above is sized for the worst case (every + diagnostic input). In a release build with none of those macros + defined, make sure the corresponding parameters either compile away + entirely (same `#ifdef`-sectioning convention the rest of this file + already uses) or are proven by the compiler to be dead weight — don't + let `uRawDepthB`/`uRawImproveB`/the snapshots turn into always-live + arguments on every call when the only consumers are instrumentation + that's compiled out. Verify by comparing generated code size/a quick + NPS run of a `PERF_COUNTERS`-undefined build before and after step 2, + not just the instrumented build used for the suite gates. +- **Opportunistic, in-scope wins to look for while the code is open** + (don't go hunting outside this refactor's diff, but don't ignore an + obvious one either, since the whole per-move span is being rewritten + once regardless): any snapshot/bucket value currently recomputed + per-move that's actually loop-invariant per node (the doc's + `uRawDepthB`/`uRawImproveB` hoist is already one example carried over + from the current code — check for others while rewriting this block, + e.g. whether `GetMoveFailHighPctThousandths(mv, &uFHAttempts)` or the + joint-table lookup have any sub-computation that's the same across + moves at a node and is currently being redone); and whether the + diagnostics-struct-vs-PLY_INFO choice above can also drop a redundant + write (e.g. if a value is written to both a local and `pi->` today, + keeping only the `pi->` copy and reading it back removes a stack + write). Treat these as reviewed, measured changes (node-count + + NPS-on-`head_reference`-suite comparison), not assumed-safe + micro-optimizations — a performance claim needs the same before/after + evidence a correctness claim does here. +- **Verification gate, added to the existing plan**: alongside the + bit-identical node-count check in step 2's isolation gate, also record + wall-clock NPS for the same `sd`-depth suite runs, before and after + each step, on this box (which the project's own CLAUDE.md already + warns is noisy under shared load — do a back-to-back A/B, not a single + run, per the Environment notes). A refactor that's node-count-identical + but measurably slower per node is a regression this plan must not ship + silently. + +## Open question for the user + +Whether to gate `FH_TABLE_COLLECTION`'s sampling to the originating +thread only until step 3's snapshot plumbing lands, or accept a short +window (within the same commit sequence, steps 3+4 landing together) where +no helper-thread sample is ever collected with zeroed snapshots because +the plumbing and the unification land atomically. Recommendation above is +the latter (land 3 and 4 together) — flagging here so it's a decision, not +an assumption. |
