diff options
| author | Scott Gasch <[email protected]> | 2026-10-20 09:26:44 -0700 |
|---|---|---|
| committer | Scott Gasch <[email protected]> | 2026-10-20 09:26:44 -0700 |
| commit | c8c6bbc2c1a103e5b07b079a0f8bdf66c7724d80 (patch) | |
| tree | b6c2137c9ac577fa83b1e948fa0311d53dfe445d /src/docs/NNUE.md | |
| parent | 3b5d3677bb14ae14728a1c4b221176bb83667d92 (diff) | |
Wire NNUE eval into search, build the training pipeline, and add a fast
move/eval-agreement benchmark against Stockfish (tests/lop)
Implements the architecture decisions in src/docs/NNUE.md: HalfKAv2_hm
feature set, int16/int8 quantization, per-ply accumulator backup for
king moves, and a lazy-eval-ready integration into search.c/eval.c.
Adds the full annotate_pgns/ training pipeline (extraction, K-fit/
labeling, quiet filtering, tensor building, training, export) plus an
end-to-end orchestrator (run_nnue_pipeline.py) and validation tooling
(sanity_check_net.py, compare_vs_stockfish_net.py). Adds tests/lop:
a 10k-position list-of-positions benchmark comparing engine eval rank
and move choice against pre-computed Stockfish/lc0 references, much
cheaper than a full self-play match for day-to-day progress checks.
First sd10-matched self-play result against classic HCE: LLR +2.28
(H1=+2.94), -72=119+169 over 360 games, confirming the net is
"smarter per node" at matched depth. A real-time (--st 4) match is
currently weaker (candidate loses ~3 plies of depth to a ~3.3-3.7x
nps gap against an unrecalibrated search), consistent with
NNUE.md's adoption-criterion section -- pruning/reduction constants
still need retuning for the new eval's score scale before a fair
real-time comparison is possible.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Diffstat (limited to 'src/docs/NNUE.md')
| -rw-r--r-- | src/docs/NNUE.md | 1605 |
1 files changed, 1572 insertions, 33 deletions
diff --git a/src/docs/NNUE.md b/src/docs/NNUE.md index 0ab46ec..dbc5d54 100644 --- a/src/docs/NNUE.md +++ b/src/docs/NNUE.md @@ -1034,16 +1034,187 @@ interchangeable with real Stockfish `.nnue` binaries. 2,298,648 positions, 84,302 exact rows left unlabeled), `games.holdout.labeled.jsonl` (3,005 games, 255,245 positions, 9,113 exact rows left unlabeled). -- **Not yet built**: the Python training script itself (the loss - function, quantization-aware export pass), the feature-indexing-spec - cross-checker, any `SEARCHER_THREAD_CONTEXT`/accumulator plumbing - (still correctly gated behind a dedicated branch, not started), and - the lambda sweep/selection for blending `p_search`/`r_outcome`. The +- **Update 2026-10-15: all of the below is now done** (see "Status + (2026-10-15)" further down for the full writeup) -- the training + script, feature-indexing cross-checker, accumulator plumbing, and + the first trained/exported/wired-in/match-played net all exist now. + **Still not done**: the lambda sweep/selection for blending + `p_search`/`r_outcome` (still correctly deferred to Phase B, per the + lambda decision above -- no new information has arrived that would + change that), and quantization (see "Quantization plan"). The quiet-filter gap previously noted here (not yet applied to the game-shaped/labeled files) is closed -- see `quiet_filter_games.py` above -- and a future annotation run can skip the post-hoc step entirely via `annotate.py --prefilter-quiet`. +## Status (2026-10-15): training run finished, net wired into real search, first match-play result + +- **Phase A training run finished.** `annotate_pgns/nnue_train.py` against + the ~2.08M-row quiet-filtered pool above, 20 epochs, PyTorch on CPU + (per the Phase A local-CPU decision below) -- final train MSE 0.0065, + holdout MSE ~0.022. Checkpoints: `runs/sd10_twic_2M/nnue_ckpt/ + ckpt_epoch000.pt` through `ckpt_epoch019.pt`. +- **Feature-indexing cross-checker built and passing** + (`cross_check_features.py` against `nnue_feature_spec.py`'s Python + reference encoder and `src/nnuefeatures.c`'s C reference dump, 9 FENs, + all matching) -- closes milestone 1 below. +- **Weight export pass built** (`export_nnue_weights.py`): flat binary + format, 4-byte `"NNU1"` magic, `<4I` dims header + (`NNUE_DIMENSIONS`/`L1`/`L2`/`L3`), then the eight weight/bias arrays + in a fixed order. `ckpt_epoch019.pt` exported to + `runs/sd10_twic_2M/nnue_ckpt/weights.nnue` (23MB). +- **Accumulator plumbing built, tested, and proven fast** (on the + `nnue-accumulator-plumbing` branch, per milestone 3's branch + requirement) -- `SEARCHER_THREAD_CONTEXT` gained `sNnueAccum[2]`; + all 8 of `move.c`'s Lift/Place/Slide(WithoutSigs) primitives + maintain it incrementally (chose primitive-hooking over re-deriving + move-kind from `mv` after discussion -- these primitives already + couple tightly with Zobrist hashing, and `GenerateMoves`/ + `MakeMove`/`UnmakeMove` are too hot to afford re-branching); king + moves trigger a full `NnueAccumRecompute` for their *own* perspective + only (every feature in that perspective is own-king-relative), with + a `PLY_INFO`-indexed backup/restore for `UnmakeMove`. `testnnue.c`'s + `TestNnueAccumulator` plays real games from 7 curated FENs chosen to + hit captures/promotions/en passant/castling/plain king moves within + 2 plies, comparing incremental maintenance against a from-scratch + `NnueAccumRecompute` after every move -- caught two real bugs on its + first-ever runs (a king-capture backup-ordering bug where the + backup was taken after the capturing `LiftPiece` instead of before + it; and, when the perf-gating fix below landed, the test's own + failure to set `g_fUseNnueEval` before exercising the gated code). + `setboard`'s downstream effect is covered by the same + `ReInitializeSearcherContext` choke point every `go`/`ponder` already + calls -- confirmed this is the one place that needs a hook, not + `SetRootPosition` itself. +- **Load/call infrastructure and real search integration built**: + `NnueLoadWeights` (`nnueeval.c`) loads an exported `.nnue` file, + dimension-checked against the build's own constants; `NnueForward` + runs the real FC head (ReLU(accumulator) -> own-perspective-first + concat by side-to-move -> dense 2L1->L2->L3->1). `Eval()` now + short-circuits to `NnueForward` when `g_fUseNnueEval && + g_fNnueWeightsLoaded`. Reachable via the `nnue load|eval|on|off` + command and a `--nnueweights <file>` CLI flag (the latter needed + because `match_play.py` spawns engines with fixed args -- no chance + to send interactive commands first). +- **Critical performance regression found and fixed.** The accumulator + maintenance added to move.c's primitives ran unconditionally on the + (unverified, wrong) assumption that it was cheap -- cost ~50% of + engine throughput even with NNUE off (673k nps measured vs. the + expected 1.4-1.5M baseline). Gated all of it behind `g_fUseNnueEval`. + Confirmed via `bench`: classic HCE back to ~1.52M nps, NNUE eval a + fair ~410-411k nps. +- **`test_vs_head.sh` extended with `NNUE_WEIGHTS`**: wraps the + candidate binary in a thin exec shim that always injects + `--nnueweights $NNUE_WEIGHTS`, since `match_play.py`'s `Engine` class + passes identical `engine_args` to both sides and spawns the given + path directly via `Popen` (no shell), so a shebang wrapper script in + the candidate's place works unmodified -- no `match_play.py` patch + needed. +- **First real match-play result, stopped by hand at 70 games**: `-45 + =0 +25` (score 0.357), llr trending toward H0 but nowhere near + resolved (far too few games for this SPRT's bounds). At `--st 4`, + the NNUE candidate searched ~3.4-3.8 plies shallower than the HCE + head (avg depth ~10.1-10.5 vs ~13.7-13.9) and at roughly 4.6-4.8x + fewer nps (~285-293k vs ~1.32-1.41M) -- yet still won 25 of 70 games + and held a stable ~0.30-0.36 score across the run rather than + eroding toward 0 as the sample grew, and ran a *tighter* tree despite + running under HCE-tuned pruning margins (cumulative ebf + ~3.8-3.97 vs. head's ~4.59-5.01). Read as: the net's per-position + judgment looks genuinely promising, but is not yet being given a + fair fight -- two known confounds stack against it simultaneously + (raw search-depth handicap from the speed gap, and futility/ + delta-pruning/aspiration-window margins all tuned against HCE's + score distribution, not NNUE's). **Zero draws is not a red flag** -- + expected signature of a large strength/depth mismatch (the weaker + side tends to lose material in sharp lines well before either side + would reach a balanced technical ending), not evidence anything is + broken. +- **Notably, draws were exactly zero** across all 70 games -- see + above, this is explained by the depth/strength gap, not a bug. +- **Quantization investigation started** (not yet implemented -- see + "Quantization plan" below for the concrete next steps): + - AVX2 build support added (`AVX2=1` GNUmakefile option, scoped + opt-in like `GPROF=1`, not folded into the default/`-mtune=native` + set -- same portability reasoning as the existing `-march=native` + rejection). Confirmed clang's `-O3 -ffast-math` already + autovectorizes both `NnueForward` and `NnueAccumAddFeature`/ + `RemoveFeature` to SSE (4-wide) without this flag; `AVX2=1` + widens them to real 256-bit `ymm` ops (verified via `objdump` on + the linked binary, not the `.o` -- the `.o` files are LTO + bitcode, `objdump -d *.o` shows nothing meaningful). + - **AVX2 is a ~3% win for classic HCE but a measured no-op for + NNUE** (`bench`: 410,380 -> 411,402 nps, within noise). This is + a real, useful negative result: it confirms the NNUE bottleneck + is memory-bound (cache misses gathering rows out of the ~23MB + `g_rgfNnueWeight` embedding table on every accumulator + add/remove -- the per-*move* hot path, far more frequent than + the once-per-evaluated-leaf FC head), not FP throughput -- wider + SIMD can't help a stall waiting on a cache-line fetch. + - **Quantization safety-checked against the real checkpoint before + committing to any scale constants**, since this net trains with + plain (unclipped) ReLU, not Stockfish's clipped ReLU that makes + its own int16/int8 scheme near-lossless by construction. Loaded + `ckpt_epoch019.pt` directly: `ft.weight` max|.|=5.01, + std~=1.00; FC weights all max|.| < 0.36. Sampled 2000 trials of + 32 active features (a realistic full-board HalfKAv2_hm count) + summed through `ft.weight` + bias: worst-case accumulator + magnitude ~26. **Conclusion: safe to quantize this checkpoint + directly, no retrain needed** -- even a generous fixed-point + scale (256-512) leaves int16 (+-32767) comfortable headroom + against the realistic worst case (~6,600-13,300), nowhere near + saturation. + +## Quantization plan (not yet implemented) + +Priority order, chosen from the AVX2 experiment above: the embedding +table (per-*move* cost, memory-bound) matters more than the FC head +(per-evaluated-*leaf* cost, compute-bound and already partly +vectorized), even though the FC head is the easier, more isolated +change. Planned sequence: + +1. **Int16-quantize the embedding table first** (`g_rgfNnueWeight`, + `g_rgfNnueBias`) -- halves its footprint (~23MB -> ~11.5MB), + directly targets the measured memory-bound bottleneck via improved + cache hit rate on `NnueAccumAddFeature`/`RemoveFeature`'s per-move + gather. Fixed-point scale TBD from the 256-512 range checked safe + above; `NnueAccumRecompute`'s accumulation itself can likely stay + int32 (cheap, avoids overflow entirely) even once the table it + gathers from is int16. +2. **Int8-quantize the FC head second** (`nnueeval.c`'s `NnueForward`) + -- isolated to one file, runs once per evaluated leaf (not per + move), and gets the largest *relative* SIMD win (int8 dot products + are 32-wide under AVX2 vs. SSE's 4-wide float32) once it's worth + doing at all. +3. **New weight-file format version required.** `NnueLoadWeights` + currently validates magic (`"NNU1"`) + dims but has no dtype/format + field -- an int16/int8 export with the same dims would silently be + read as float32 and produce garbage, exactly the failure class the + existing dim-check exists to prevent. Bump to `"NNU2"` (or add an + explicit format word) before shipping any quantized export. +4. **New correctness test required -- `TestNnueAccumulator` will not + catch quantization error.** It only compares incremental + maintenance against `NnueAccumRecompute`; once both sides are + quantized identically they still agree with each other regardless + of whether quantization degraded the actual eval. Need a separate + check: float `NnueForward` vs. quantized `NnueForward`, same + position, across a few hundred FENs, comparing the resulting + centipawn deltas against some acceptance threshold (TBD). +5. **Side benefit, not the goal**: integer incremental accumulator + add/subtract is exact (no float drift), so quantizing removes both + `_CompareAccumToRecompute`'s 0.01f epsilon tolerance and any + long-move-sequence floating-point drift question entirely. +6. Re-run `test_vs_head.sh` with `NNUE_WEIGHTS` after each step to + measure the real nps delta before moving to the next step -- same + reasoning as checking AVX2 in isolation before quantizing: don't + stack two speed changes in one measurement. +7. Once NNUE's effective search depth is no longer handicapped by + several plies at the standard time control, re-run the + `match_play.py` gate for a real read on judgment quality, largely + decoupled from the raw-speed confound -- and only then decide + whether margin recalibration (see "Integration into search" above) + is worth doing before or after a Phase B (GPU-trained, larger net) + run. + ## Training compute - No reliable local GPU. Decided against Google Colab's notebook-session @@ -1118,36 +1289,1404 @@ interchangeable with real Stockfish `.nnue` binaries. existing joint-table instrumentation, and only build an independent bitboard pass if that measurement says it's worth it.** No speculative tactical-signals pass planned up front. -3. MakeMove/UnmakeMove accumulator plumbing (ply-indexed king-move - backup in `PLY_INFO`) built and proven correct/fast **with the old - `Eval()` still live** -- prove the plumbing before any trained net - exists to evaluate against it. **Do this on a separate git branch**, - not on `master`, given how much live search-path code it touches; - snapshot `head_reference` first. -4. TWIC/lichess annotation harness built; `<TT>`/`<EGTB>`/`<REC>` tag - semantics investigated and resolved; first large annotated training - pool assembled (Phase 1 data, per above). **Unblocked now, independent - of steps 1-3** -- no engine-internals risk, can start immediately. -5. First bootstrap net trained (rented GPU), quantization-aware export - pass built. -6. C inference code (forward pass + quantized weight loader) written and - validated against the Python model's output on held-out positions. -7. Wired into search behind a toggle; margins recalibrated. -8. `match_play.py` gate; iterate; retire old `Eval()` only after a - convincing, repeated win. +3. **Done, on `nnue-accumulator-plumbing`.** MakeMove/UnmakeMove + accumulator plumbing (ply-indexed king-move backup in `PLY_INFO`) + built and proven correct via `testnnue.c`'s `TestNnueAccumulator`, + and proven fast after finding and fixing a real ~50% throughput + regression (see "Status (2026-10-15)" above) -- gated behind + `g_fUseNnueEval` so it costs nothing with NNUE off. `head_reference` + snapshot already existed per the existing protocol. +4. **Done.** TWIC annotation harness built and run (Stage A, 2.55M + positions from 30,000 `twic_filtered.pgn` games); `<TT>`/`<EGTB>`/ + `<REC>`/`<=>`/`"only"` tag semantics investigated and resolved (see + "Status (2026-10-14)" above, including the real engine bugs found + along the way). +5. **Done for Phase A (CPU-trained bootstrap, not the GPU-trained + Phase B net).** First net trained (`nnue_train.py`, 20 epochs, + train MSE 0.0065 / holdout MSE ~0.022) and exported + (`export_nnue_weights.py` -> `weights.nnue`). **Quantization-aware + export pass is explicitly NOT done yet** -- current export is + plain float32; see "Quantization plan" above for the concrete, + not-yet-started next steps. +6. **Done**, modulo quantization. C inference code (`nnueeval.c`'s + `NnueForward` + `NnueLoadWeights`) written and running against real + search positions via the `nnue eval` command and live search + integration. **Not yet validated against the Python model's output + position-by-position** -- worth doing (a batch of FENs run through + both `nnue_model.py`'s `forward()` and `NnueForward`, comparing + scores) before trusting the C side's numerical correctness beyond + "it loads and produces plausible-looking scores," which is all + that's been checked so far. +7. **Done.** Wired into search behind `g_fUseNnueEval`/ + `g_fNnueWeightsLoaded` (`Eval()`'s short-circuit). **Margins NOT + yet recalibrated** -- still running under HCE-tuned futility/EFP/ + LMR/NMP margins, which the first match-play result's analysis + flags as a likely real confound, not yet quantified. +8. **In progress, first result in hand, inconclusive by design (stopped + early).** `match_play.py` gate: 70 games, `-45 =0 +25`, llr nowhere + near either SPRT bound. Not resumed yet -- current priority is + closing the speed gap (quantization) before spending more match + time on a comparison known to be confounded by it. Old `Eval()` + stays live and is the default (`g_fUseNnueEval` defaults `FALSE`) + until NNUE wins convincingly and consistently under this gate. + +## Status (2026-10-15, quantization session): both quantization steps done, validated, and measured + +Both steps of the "Quantization plan" above are now implemented, on +the `nnue-accumulator-plumbing` branch, in this order: + +- **Step 1 -- embedding table int16, done.** `g_rgfNnueWeight`/ + `g_rgfNnueBias` (`chess.h`/`nnueaccum.c`) are now `SHORT`, at a fixed + scale of 4096 (checked against `ft.weight`'s real max|.|=5.01 -- + `5.01*4096 ~= 20502`, well under `MAX_SHORT`, ~1.6x headroom for a + larger Phase B net). `NNUE_ACCUMULATOR.rgiValue` (was `rgfValue`) is + `INT` (int32), not `SHORT` -- summing up to 32 active features' + `SHORT` rows can exceed `SHORT`'s range even though each individual + row fits (measured worst-case real accumulator magnitude ~26 in + float-equivalent units, i.e. ~106,500 at this scale -- 3x past + `MAX_SHORT`). `NnueForward` converts back to float (divides by + `g_fNnueFtScale`) only where the accumulator is read, never where + it's maintained -- incremental `MakeMove`/`UnmakeMove` updates stay + purely additive integer arithmetic, exact, no float drift. + `testnnue.c`'s `TestNnueAccumulator` updated to match (integer test + weights, exact-equality comparison instead of an epsilon tolerance -- + a real correctness upgrade this dtype change enabled, not just a + mechanical port). Passes clean under `DEBUG=1 TEST=1`. +- **Step 2 -- FC head int8, done.** `g_rgfNnueFc1Weight`/ + `g_rgfNnueFc2Weight`/`g_rgfNnueFc3Weight` (`chess.h`/`nnueeval.c`) + are `CHAR` (int8, signed), one weight scale per layer derived + exactly from that layer's own max|.| (no sampling needed, weights + are fixed and known). Biases stay `float32` (tiny arrays, added + post-dequant, not worth quantizing). Activations are UCHAR (uint8, + unsigned -- confirmed all three stages are strictly post-ReLU, + `min=0.0000` on a real 50k-position sample, so no bits wasted on a + sign). `NnueForward` rewritten stage-by-stage: dequantize accumulator + -> ReLU -> requantize to UCHAR at AS1 -> int8x8 dot product + accumulated in `INT` -> dequantize (divide by `AS*WS`) + bias -> ReLU + -> requantize at AS2 -> repeat through fc2/fc3. Matches a Python + numpy simulation of the identical quantized math, built and checked + before touching any C code (see below). + - **Activation scales (AS1/AS2/AS3) are calibrated from real + sampled positions, not a theoretical bound** -- unlike weights, + activation ranges aren't knowable from the checkpoint alone. + `export_nnue_weights.py`'s `CalibrateActivationScales` runs + 50,000 real positions from `tensors_train/` through the float + model with forward hooks, takes the observed max per stage, and + applies a 1.3x headroom multiplier before deriving the scale + (`AS = 255 / (observed_max * 1.3)`). Real observed maxes: + `fc1_in` 27.12, `fc2_in` 19.11, `fc3_in` 7.72 (p99.9 well below + max in each case -- confirms the max is tail-driven, headroom is + doing real work, not padding noise). + - **File format bumped to `"NNU3"`** (`NnueLoadWeights` rejects + both older `"NNU1"` and `"NNU2"` files explicitly, not silently). + Header now carries `ft_scale` (1x float), `act_scale` (3x float: + AS1/AS2/AS3), `w_scale` (3x float: WS1/WS2/WS3) before the + weight arrays themselves. + - **Accuracy measured before touching any C code, same discipline + as step 1's safety check**: a numpy simulation of the exact + quantized int8 math (uint8 activations, int8 weights, int32 + accumulation, per-layer scales) against the real float model, + 5000 real holdout positions: **mean |delta| 1.55cp, median + 0.99cp, max 20.91cp, zero clipping anywhere** (weights or + activations, all three layers, out of 2.56M+160K+1K+32 values + checked). Context: the project's own `sd10`-vs-`sd16` + depth-adequacy check (training-label noise already tolerated) + measured mean |delta| 19cp -- this quantization's error is + **over 10x smaller** than noise already baked into the training + data. Confirmed again directly in C after implementation: `nnue + eval` on 4 hand-picked real FENs vs the Python float reference, + deltas of 1/2/11/0 cp -- consistent with the offline simulation, + no sign of an implementation bug. +- **AVX2+FMA confirmed as a real, separate win once real weights are + loaded** (not the "measured no-op for NNUE" result from the + pre-quantization investigation above -- that measurement predated + knowing which function actually dominated runtime, see the + profiling finding below). `gmake AVX2=1`: `objdump` on the linked + binary confirms `NnueForward`'s dot-product loops widen from 4-wide + `xmm`/`mulps`+`addps` to 8-wide `ymm` with 99 real `vfmadd*` fused + instructions. Measured: int16-table-only build went from ~381-384k + nps (no `AVX2=1`) to ~409-410k nps (`AVX2=1`) -- a real, repeatable + ~7% win, zero retrain, zero risk. +- **Profiling (`docs/PROFILE.md`'s FreeBSD `pmcstat -P` + `gprof` path) + found `NnueForward` -- the FC head, not the embedding table -- + dominates runtime: 34.0% of total self-time, alone**, more than + double the next entry (`SlidePiece`, 12.6%) and roughly on par with + *all* accumulator-maintenance-related functions combined even when + generously over-attributing their non-NNUE move-bookkeeping cost to + NNUE. `NnueAccumAddFeature`/`RemoveFeature` don't even appear as + separate flat-profile line items -- small enough to get inlined into + their `SlidePiece`-family callers. This directly revised the + quantization plan's priority ordering: the embedding table (step 1) + was assumed memory-bound-dominant from the AVX2 experiment's + "no-op for NNUE" result, but that result just meant AVX2 doesn't + help *that* code path -- it didn't establish the table was the + actual bottleneck. `fc1`'s matmul (`2*L1 x L2 = 2*256*32 = 16,384` + multiply-adds per evaluated leaf) being far larger than `fc2` + (1,024) or `fc3` (32) explains why: cost at each layer boundary + scales with the *product* of adjacent layer widths, and the + 512-wide-to-32 transition is the one expensive step in this + topology, more frequently hit (once per evaluated leaf) and more + expensive per-call than the table gather (once per move, but only + a handful of 256-element row reads). +- **Net bench result, chaining every change in this session**: + + | Build state | nps (real weights, `nnue on`) | + |---|---| + | Float32 everywhere (pre-session baseline) | ~411,000 | + | int16 table, no `AVX2=1` | ~381,000-384,000 | + | int16 table + `AVX2=1` | ~409,000-410,000 | + | int16 table + int8 FC head + `AVX2=1` | **~559,000-563,000** | + + Net effect: **~37% faster** than the pre-session float32 baseline, + closing the gap to classic HCE (~1.51M nps) from 3.9x down to 2.7x. + Correctness verified at every step (self-tests, offline Python + simulations before touching C, then direct C-vs-Python spot checks + after). +- **`WS1`/`WS2`/`WS3` derivation note**: unlike the embedding table + (one global scale, since it's one homogeneous array), each FC layer + gets its *own* weight scale, derived from that layer's own max|.| -- + `fc1.weight` max 0.350, `fc2.weight` max 0.361, `fc3.weight` max + 0.334 (all much smaller than `ft.weight`'s 5.01, but the relevant + comparison for int8 resolution isn't the max, it's the *step size* + relative to each layer's std -- at full-range scale this works out + to roughly 18 quantization levels per standard deviation for the FC + weights, versus thousands for the int16 table. The 1.55cp measured + result shows this coarser per-weight resolution doesn't translate + 1:1 into coarser *output* accuracy -- dot-product summation over + many terms (512/32/32) partially cancels independent rounding + errors, and ReLU's sparsity (46-69% exact zeros measured on real + activations) contributes zero error for free on a large fraction of + terms. + +## Lazy-evaluation tiering plan (started 2026-10-15, not yet wired into search) + +Classic `Eval()` has three real exits -- super-lazy (material only), +lazy (material + a defined subset of positional terms), full -- each +gated by a margin derived from a *measured population* of real +positions, not a modeled/assumed distribution. `Eval()`'s current NNUE +short-circuit (added when NNUE was first wired into search, see +"Status (2026-10-15): training run finished" above) bypasses all of +this -- it's an unconditional full `NnueForward()` call, with a comment +explicitly noting the margin machinery below it "has not been +recalibrated against NNUE's score distribution yet." This section +plans closing that gap. + +**The mechanism, read directly out of the live code** (not +reconstructed from memory -- see `eval.c`'s +`iSuperLazyResidualPessimisticByOwnArmy`/`OptimisticByOwnArmy`/ +`P50ByOwnArmy` tables and their use in `Eval()`'s super-lazy exit, +~line 7150): + +- Every one of these tables is **literally empirical percentiles read + off a measured population of real positions** -- the comment blocks + above several of them show raw population stats before the table + (e.g. `n=104187433 min=-68 p1=-10 ... p50=10 ... max=165`). No + distribution shape is assumed; the percentile is read directly off + sorted real data, per material bucket (`_MaterialBucket`/per-side + army-scaler bucket, 8 buckets in the existing tables). +- **Two different statistics for two different jobs, not one margin**: + - **p50 is a bias correction** -- the typical/unbiased value fed to + `iPositional` (a lazy exit's best-guess estimate of the missing + positional terms, consumed downstream by e.g. QSearch's own + futility margins). Current NNUE short-circuit reports a flat + `{0, 0}` here -- the "intentionally least-specific but safe" + placeholder, not a real estimate. A calibrated p50 of the probe's + own residual could improve this once real residual data exists. + - **p2/p98 (not p5/p95) is the soundness bound** -- answers "how far + could the true full-eval score plausibly be from this cheap + estimate, in the realistic tail," and is what actually gates + whether the lazy exit is safe to take. The exact comment: *"p50 is + the right statistic for a bias correction, p2/p98 is the right + statistic for a soundness bound."* + - Direction is asymmetric and combines two terms before a `MAX0()` + floor: checking against `alpha` (could this fail low?) uses the + **optimistic** tail (most generous plausible correction in the + mover's favor); checking against `beta` (could this fail high?) + uses the **pessimistic** tail (least favorable plausible + correction). Getting this sign/direction wrong would silently + invert the safety guarantee -- needs care, not pattern-matching, + when implemented for real. +- This two-statistic pattern (p50 bias, p2/p98 safety bound) is what + the NNUE lazy tier should reuse, not a new invention -- same + methodology the existing calibration harness + (`CALIBRATE_MARGIN_SAFETY`, `RecordSuperLazyMarginSafetySwing`, swing + histograms bucketed by material) already proved out for classic + `Eval()`'s own tiers. + +**The proposed NNUE lazy-tier signal**: not a second, separately +trained net (that reintroduces the calibration-mismatch problem +already rejected in the "Architecture decisions" section above) -- +instead, a **linear probe fit directly on the real net's own +accumulator** (`fc1`'s input: the ReLU'd, concatenated 512-wide +accumulator halves), which is already computed and resident at every +node for free as a side effect of `nnueaccum.c`'s incremental +maintenance, before `NnueForward`'s expensive `fc1->fc2->fc3` pass +(34% of runtime, see profiling finding above) ever runs. + +**Progress so far**: +- **Offline probe fit, done, preliminary.** Ridge regression (512 + features + bias, lambda=1.0) fit against 80,000 real positions from + `tensors_train/`, targeting the real float net's own raw output + (`forward()`, pre-sigmoid). Saved to + `annotate_pgns/probe_weight.npy`/`probe_bias.npy`. +- **Measured residual (full net score minus probe score), 20,000 held-out + positions from the same sample**: mean |residual| 50.10cp, median + 33.09cp, p2=-166.79cp, p50=0.33cp, p98=+169.14cp. **An order of + magnitude coarser than the int8 quantization's 1.55cp** -- expected, + since a flat linear projection discards all of `fc1->fc2->fc3`'s + nonlinear structure, unlike int8 which preserves the same + computation at lower numeric precision. This sets real expectations: + the implied safety margin (~150-200cp) is large relative to typical + eval swings, so the lazy exit will only fire on nodes already + lopsided relative to the search window -- a real but likely narrower + slice of total nodes than hoped going in. +- **Real fit-vs-validation gap observed** (mean 37.85cp on the fit + split vs 50.10cp held out, p98 153cp vs 195cp) -- larger than + expected given 80k examples against 512 features plus ridge + regularization. Not yet root-caused (could be regularization + strength not cross-validated, could be real limits of a flat linear + fit over this representation, could be this offline sample's size). + **Flagged as unresolved, not yet trusted as final** -- this was an + offline, single-position-sample check, explicitly a preliminary + step, not the real calibration. + +**Not yet done -- the actual next steps, in order**: +1. Root-cause the fit/validation gap above (cross-validate `lambda`, + or just use more fitting data) before trusting any residual + numbers from this probe as final. +2. **Build the real calibration pass**: a `CALIBRATE_MARGIN_SAFETY`- + style harness for the NNUE lazy probe specifically -- same pattern + as classic `Eval()`'s existing one (run the full `NnueForward` pass + anyway even when the probe-based exit would fire, record the swing, + restore the real build's behavior before returning) -- against + **real search-node traffic**, not another offline position sample. + This is the measurement that actually matters; the offline check + above was only ever a cheap sanity gate before investing in this. +3. From that real population, derive p50 (bias correction) and p2/p98 + (safety bound) -- possibly bucketed by material/phase the way + HCE's own tables are, possibly not. **Whether NNUE's accumulator + already implicitly encodes phase** (unlike HCE's explicitly + material-gated lazy terms) **is an open, empirically-answerable + question from this same population** -- check whether the residual + distribution actually shifts meaningfully bucket-to-bucket before + committing to that complexity. +4. Wire the gate into `Eval()`'s existing NNUE short-circuit, using + `iAlpha`/`iBeta` already passed into `Eval()` -- no new interface + needed, this is a parallel early-return block next to the current + unconditional bypass. +5. Ship it instrumented-only first (swing recorded, real build + behavior unaffected) before trusting it live, same rollout + discipline as every other change in this project. + +The super-lazy (material-only, no net at all) idea from the +"Architecture decisions" section above remains separate, more +tentative, and explicitly not decided either way -- the calibration +risk there (no shared units with the WDL-sigmoid-calibrated net +output) is structurally different from this linear-probe tier, which +shares the real net's own trained representation and units by +construction. + +## Status (2026-10-16): lazy probe gate shipped live, QSearch futility margins fixed, same-depth ECM win confirmed, overnight match queued + +Picking up from the "Lazy-evaluation tiering plan" section above -- +the fit-vs-validation gap was root-caused as pure sampling noise (more +data + the real `tensors_holdout/` split made it vanish; lambda barely +mattered, the model was never near overfitting), and everything from +"build the real calibration pass" onward is now done. + +- **Fit/validation gap resolved**: 500k-example fit + real `tensors_ + holdout/` evaluation (not another ad-hoc split) showed holdout + *better* than fit (38.68cp vs 41.09cp mean) -- the earlier 50.10cp + number was noise from a too-small single sample, not overfitting. +- **Real calibration done properly, twice**: first pass used the three + curated tactical suites (91M samples) -- too narrow a slice (hand- + picked for sharp tactics), confirmed by a second pass against + `tests/cross_sectional_200k.ep_` (200,000 diverse positions, 16 + parallel single-threaded shards, `sd 6`, 3.09 BILLION real search- + node samples) landing close to but meaningfully more trustworthy + than the first (p98 margins `[180, 200, 190, 180, 170, 160, 150, + 150]` by material bucket vs the tactical-suite-only `[190, 200, 200, + 190, 190, 170, 150, 150]` -- also surfaced real tail outliers (max + 534cp) the narrower sample never saw). +- **A real, separate bug found and fixed along the way**: chaining + multiple commands after `script <file>` in one `--batch --command` + string (e.g. `script a.ep_; script b.ep_; calibrate marginsafety`) + silently truncates every search except the last to a shallower depth + than requested -- `PushNewInput` (`input.c`) splits and queues + *everything* up front, and `searchsup.c`'s mid-search + `NumberOfPendingInputEvents()` check (legitimate for live xboard/ + WinBoard responsiveness) fires on that queued-but-irrelevant trailing + text, treating it as "stop, something wants my attention." Root + cause confirmed by direct repro (`sd 10` silently landing at depth 8, + 8x fewer nodes) -- **not fixed in the engine** (the interactive- + responsiveness behavior is legitimate and deliberately left alone; + the input-queueing semantics are too sensitive to touch for a + testing-only problem that already has a zero-risk workaround). + **Workaround, now the required pattern for any multi-step batch + calibration**: never chain unrelated commands after `script` in one + `--command` string; if a command needs to run after a suite finishes + in the same process (e.g. `calibrate marginsafety`), put it as its + own line *inside* the `.ep_` file instead -- `script.c`'s own + `_ProcessSuiteLine` already dispatches any non-"go" line via the + queue-free `ExecuteCommandLineDirectly`, so it runs in-order with no + truncation risk. For N-way sharded runs (per CLAUDE.md's existing + convention), each shard gets its own single `script` call + its own + `calibrate marginsafety <path>` line inside the file, in its own + process -- then combine the N raw histogram files after the fact. +- **The "super lazy" (material-only) tier was investigated and + rejected as a live feature** -- `pos->iMaterialBalance[color]` is + already in the same cp units as `NnueForward`'s output + (`VALUE_PAWN=100`), so no fit/regression was needed at all (unlike + the accumulator probe) -- straight subtraction, same harness. Real + measured residual against the full net, same 3.09B-sample + cross-sectional pool: mean 124-344cp, p98 510-1230cp by bucket -- + **6-8x larger than the accumulator probe's** (47-77cp mean, 150-200cp + p98). Shape is an inverted-U by material bucket (peaks in the + middle, smaller at both bare-king endgames and near-full-material + positions), not the simple monotonic-opposite-of-the-probe trend + originally guessed. Conclusion: material alone is too crude a proxy + for *this* net's output to fire often enough to be worth it -- NNUE's + accumulator captures far more positional judgment relative to + material than a handcrafted eval does, unlike HCE where material + really is close to the final answer. Not built as a live gate. +- **Lazy probe gate shipped live** (`nnue lazyprobeon`/`lazyprobeoff`, + defaults `FALSE` same as `g_fUseNnueEval`): `Eval()`'s NNUE short- + circuit now actually skips `NnueForward` when + `NnueProbeScore() +/- margin` confidently resolves fail-high/fail-low + against `[iAlpha, iBeta]`, dual-mode via `#ifdef CALIBRATE_MARGIN_ + SAFETY` (calibration build always still runs the full pass for + measurement; release build actually skips). Held-out validation + (`ecm_confident_sample2.ep_`, not used for calibration) showed + exceeded rates mostly at or under the ~2% the p98 choice implies + (buckets 2-7: 0.87-2.24%); buckets 0-1 ran hotter (3.47-4.67%, + consistent with those being the smallest-sample buckets in + calibration, so the noisiest estimates). +- **Measured lazy-probe exit rate on real search traffic** (`twic_ + sample.ep_`, `sd 6`): **14-23% of NNUE eval calls resolve via the + cheap probe**, varying by material bucket (peak ~20-23% in buckets + 3-5, low 8.7% in bucket 7/near-full-material). Most calls still pay + for the full forward pass. +- **Real, clean, sequential (not parallel -- see below) same-depth + measurement, `ecm_hard_quick`, `sd 10`, gate off vs on**: + + | | OFF | ON | + |---|---|---| + | Solved | 32/90 | 33/90 | + | Nodes | 47,790,608 | 50,438,386 | + | Time | 112.4s | 107.1s | + | nps | 425,271 | 470,938 | + + **~10.7% real per-node throughput win, no solve-rate regression** + (if anything +1). Note nodes went *up* while time went *down* -- + the gate changes which nodes get pruned (approximate score at some + nodes shifts alpha-beta cutoffs), so node count isn't held constant + the way `sd` depth is; nps is the cleaner isolated signal. + **Caveat on parallel-run timing noise, confirmed twice this + session**: running gate-off and gate-on as simultaneous background + jobs on a shared box can be contaminated by real external load (cron + jobs, rsyncs, other users) even when each job's own `--cpus 1` leaves + plenty of idle capacity -- the fix was a clean *sequential* run on an + otherwise-idle box, not avoiding parallelism itself (6-way `--cpus 1` + parallel jobs were confirmed fine once the box was actually quiet). +- **Full three-suite combined result, same clean methodology**: + 108/191 (56.5%) solved OFF vs 112/191 (58.6%) ON, total time 247.7s + vs 240.3s (-3.0%) -- consistent with the single-suite sequential + number, no regression anywhere. +- **Independently, at the SAME fixed `sd 10` depth (no lazy gate, + separate from all of the above): NNUE solves substantially more ECM + tactics than classic HCE** -- 108/191 (56.5%) vs HCE's 72/191 + (37.7%) across all three curated suites. Since depth is held + constant, this isn't explained by NNUE's remaining speed deficit. + **Correction, 2026-10-17: do not read this as evidence the net + plays better chess.** Every real head-to-head game match run + since (see "Status (2026-10-17)" below -- five independent + configurations: two different nets, `st 4` and `sd 10`, with and + without the lazy probe) has NNUE losing decisively to classic HCE, + including at matched search depth where this ECM result would + predict an NNUE edge. ECM is a narrow slice (forced-tactic/mate + puzzles, where qsearch/extensions/move-ordering machinery shared + by both evaluators does most of the work) and evidently does not + generalize to real-game strength. As of this writing, solving more + ECM puzzles than HCE is the *only* thing either NNUE net has been + shown to do better than HCE -- it has not won a single head-to-head + match, at any depth or time control tried. +- **QSearch futility-pruning margins needed their own investigation, + separate from the eval-exit-gate margins above** -- `PLY_INFO. + uLastPositionalSource` (the per-ply tag `search.c`'s + `FUTILITY_BASE_MARGIN_BY_SOURCE[]` reads to decide how much slack to + give QSearch's delta/futility pruning) was **never being set at all** + by either NNUE exit -- every NNUE-sourced score was silently + inheriting a stale value from the last classic-`Eval()` call at that + ply slot (or the C zero-init default, `EVAL_POSITIONAL_SOURCE_FULL`, + if none ever ran). Fixed: added `EVAL_POSITIONAL_SOURCE_NNUE_FULL`/ + `NNUE_LAZY`, set at both NNUE return sites, each with its own + `FUTILITY_BASE_MARGIN` entry and always-on exit-rate counters + (`calibrate exitrate` extended to report NNUE's own lazy-vs-full + split, separate from the eval-exit gate's). + - **First real measurement** (`twic_sample.ep_`, 1500 positions, + `sd 6`, the same pool/depth the *original* classic margins were + derived from): `nnue full`'s surprise rate was 7.41% in the + checking-capture/promo gate at the 450 starting placeholder -- + well above anything in classic's documented history. Widening to + 650 tested as a complete no-op (identical 7.41%, identical `n`). + - **Root-caused, not just observed**: added a direct diagnostic + (raw pre-clamp `iFutility` value, before `MAX0`) and confirmed + `nnue full`'s population is **100.00% clamped to the `MAX0` + floor** (mean pre-clamp -589.3) -- the margin constant is never + actually read for this population, so no value could have shown + an effect. Classic `full eval`, measured the identical way on + the identical pool, is clamped 91.05% of the time (not 100%, + and less deeply negative: mean -282.5) -- saturation is a real, + general QSearch structural property (not NNUE-specific: classic + `regular lazy`'s 24.95% clamp rate matches `nnue lazy probe`'s + 24.76% almost exactly), but NNUE's `FULL` tier saturates somewhat + more completely than classic's. + - **Surprise magnitude (not just rate) measured too**, resolving + a live "is NNUE just noisier" vs "does NNUE miss bigger things" + question: `nnue full`'s surprises, when they happen, run + *smaller* than classic `full eval`'s (mean 477.5cp vs 938.7cp, + `>100cp` 9.88% vs 28.74%) -- same pattern for `nnue lazy probe` + vs classic `regular lazy` (5085.3cp vs 11030.2cp mean; 19.45% vs + 69.73% `>100cp`). So the net effect is: NNUE_FULL fails *more + often* but *less expensively* each time -- consistent with more + frequent small eval jitter near the decision boundary, not with + NNUE being tactically blind to bigger things than HCE already + is. One sample verified by hand (a genuine `Qh7+` mate-in-1, + confirmed independently via a real `sd 16` search) to make sure + the diagnostic's rare huge-magnitude entries were real finds, not + a measurement bug. + - **Decision: `FUTILITY_BASE_MARGIN_NNUE_FULL` reverted to 450** + (matching classic `FULL`) rather than kept at the untested 650 -- + the saturation finding means neither value was actually + validated by this data, and 450 is the more defensible default + given NNUE_FULL's measured surprises skew smaller than HCE's own. + **`FUTILITY_BASE_MARGIN_NNUE_LAZY=200` kept** -- this one *is* + genuinely validated (only 24.76% clamped, moved the expected + direction when tightened from 350, landed in a sane 1.3-2.1% + surprise-rate range, smaller surprise magnitudes than classic's + own `regular lazy`). + - **TODO, not yet done**: find and calibrate against the actual + non-saturated `NNUE_FULL` population specifically (the ~0% slice + this test pool never surfaced) before trusting any `NNUE_FULL` + margin as real, NNUE-specific-validated rather than a reasonable + borrowed default. Likely candidate: nodes where `iEval` sits + close to `iAlpha` rather than comfortably above it (the + saturated case is "`iEval` way above `iAlpha`" -- look at the + complement). +- **Match-play CLI wiring added, needed before any NNUE+lazy-gate + match can run at all**: `--nnueweights` already loaded weights and + set `g_fUseNnueEval=TRUE` at startup (for `match_play.py`'s fresh- + process-per-game spawning, no interactive commands available) -- + but there was no equivalent for the probe/lazy gate at all. Added + `--nnueprobeweights <file>` (same convention, sets + `g_fUseNnueLazyProbe=TRUE` automatically), and extended `eval_tune/ + test_vs_head.sh`'s `NNUE_WEIGHTS` wrapper with a new + `NNUE_PROBE_WEIGHTS` env var that injects `--nnueprobeweights` into + the same shim. Verified working end-to-end (both flags load and + enable correctly) before trusting it for an unattended run. + **Gotcha**: `test_vs_head.sh`'s default (no candidate path given) + rebuilds the candidate itself via `gmake clean && gmake release + MP=1 PERF_COUNTERS=1` -- **without `AVX2=1`** -- silently losing the + ~7% AVX2/FMA win if not passed a pre-built candidate path instead. +- **Overnight `match_play.py` run queued** (not yet run/reported) -- + NNUE+lazy-gate candidate vs classic HCE `head_reference`, using the + new `NNUE_WEIGHTS`/`NNUE_PROBE_WEIGHTS` env vars and an explicit + pre-built `AVX2=1` candidate binary to avoid the gotcha above. + +## Next up + +1. Read the overnight match result once it's in -- first real + judgment-quality read with both the quantization work and the lazy + probe gate live together, at a real time control rather than fixed + `sd` depth (so the remaining ~2.7-3.2x speed gap is a live confound + again, same as the original 70-game result, but now with a smaller + gap and the added lazy-gate throughput). +2. The `NNUE_FULL` futility-margin TODO above -- find the non-saturated + population and calibrate against it specifically, rather than + continuing to rely on a borrowed classic-HCE default. +3. Decide whether the ~3-11% nps win from the lazy probe gate + (measured) is worth the complexity on its own, or whether it's more + valuable combined with further quantization work on the FC head's + remaining float32 pieces (biases) or other speed levers -- revisit + once the overnight match gives a real strength read to weigh + against. + +## Status (2026-10-17): material-blindness fixed and verified, but zero +## head-to-head wins against classic HCE in five independent trials + +**Summary, stated plainly up front**: a real, measured defect (the net +could not distinguish losing a pawn from losing a queen in crowded +midgame positions) was found and fixed this session. The fix is +verified to work, at the population level, with a new reusable tool. +But **no NNUE configuration tried today -- old net, new net, with or +without the lazy probe, at `st 4` or at matched `sd 10` depth -- won +or even drew competitively against classic HCE.** The only thing +either net has been shown to do better than HCE is solve more ECM +puzzles at matched depth (see the correction just above). Do not +read anything below as evidence the net plays winning chess yet. + +### The material-blindness bug, found and fixed + +- **Found via `sanity_check_net.py`** (new tool, `annotate_pgns/`): + loads a raw training checkpoint directly (no export/C round-trip + needed -- the earliest point in the pipeline a net can be checked) + and removes one piece at a time from hand-picked and (better) real + sampled positions, reporting the score delta. On the Phase A net + trained with plain `λ=1.0` MSE against WDL-sigmoid targets: removing + a pawn, a knight, a rook, or a queen from the same midgame position + produced nearly indistinguishable score drops (-0.56 to -0.70) -- + directionally correct, magnitude-blind. +- **Root cause, two compounding mechanisms, neither one specific to + which corpus a position came from**: + 1. Sigmoid-gradient argument (later found incomplete on its own -- + see below): training loss is MSE against `sigmoid(raw/K)`, and + sigmoid's gradient vanishes once a position is already clearly + lost/won. + 2. **Feature dilution / shortcut learning in crowded positions** + (the more complete explanation): with ~20+ active embedding + features summed per position, a net can match training labels + well without correctly attributing any single piece's marginal + value, because real games never present a controlled + single-piece-ablation pair to force that attribution. Confirmed + empirically: the bug was severe in crowded midgame positions and + essentially absent in sparse endgames (2-6 pieces), where each + piece dominates a much larger share of the accumulator. +- **Data rebalancing (TWIC downsampled to match self-play 1:1 by game + count) was necessary but not sufficient.** It measurably helped the + sparse-endgame case (which TWIC's human-resignation bias + structurally starves) but did nothing for the crowded-midgame case, + confirming the bug was a representation/optimization problem, not + purely a data-coverage gap. +- **Two cheap levers tried and only partially effective on their + own**: + - `loss_exponent=2.6` (Stockfish's own documented value, reweights + the loss toward large errors): modest effect, spread narrowed but + monotonicity still failed, bishop/rook order still inconsistent + across test positions. + - λ-sweep (blend in `r_outcome`, game result): **made things + actively worse** -- λ=0.5/0.0 flipped the sign of several + deltas (removing a piece appeared to *help* the score). See the + "λ-sweep methodology was weaker than later standards" note below + before trusting this conclusion, though -- it has not been + properly re-tested since. + - **Lesson from this part of the investigation**: a single + hand-picked test position cannot reliably settle a question like + "does the net value a bishop more than a rook" -- one test showed + bishop > rook, a different hand-picked position showed the + reverse. This is per-position confounding (an active bishop vs. + a boxed-in rook on that specific square/position), not + necessarily evidence about the net's general prior. +- **Fix that worked: an auxiliary material-probe loss** + (`nnue_model.py`'s `material_probe`, a plain `Linear(512, 1)` + reading the same concatenated accumulator `fc1` does). Trained + jointly with the main WDL loss: `loss = wdl_loss + + material_loss_weight * mse(material_probe(accumulator), + material_balance)`, where `material_balance` is computed directly + from the FEN (pawn=1..queen=9, mover-relative, no sigmoid -- + deliberately non-saturating). `material_loss_weight=0.01` chosen to + keep the material term's raw contribution smaller than the WDL + loss's throughout training (per the user's explicit concern about + over-fixating on material, HCE-style) -- confirmed: material loss + contributed ~0.019 vs. WDL's ~0.044 at epoch 0, ~0.007 vs. ~0.022 at + epoch 19. + - **Mechanistically why this is safer than adding material to the + output directly**: the probe loss only updates `material_probe` + itself and (via backprop) the shared `ft` embedding table -- it + never touches `fc1`/`fc2`/`fc3`, the actual eval head. The eval + head stays free to deviate from pure material counting for + positional reasons; only the *representation underneath it* is + forced to keep material linearly recoverable. + - Needed more epochs than the baseline run to show its full effect + -- material-probe MSE was still falling at epoch 19 (not + plateaued), extending to epoch 34 took it from a single hand-picked + test's queen delta of -0.815 (already beating rook's -0.727) to + a much larger, cleaner separation. + +### `sanity_check_net.py --cross-sectional`: the real validation tool + +Single hand-picked positions were repeatedly shown to be unreliable +(see above) -- replaced with a population-level check: sample N real +positions from a holdout pool, remove one random piece of each +present type per position, pool deltas by piece type across the whole +sample, report mean/median/std/p25/p75. On the matprobe net +(2000 sampled holdout positions): **population means came out cleanly +monotonic in piece value, both directions** -- pawn -0.747/+0.786, +knight -1.363/+1.581, bishop -1.500/+1.710, rook -2.483/+2.907, queen +-4.160/+5.024 (own-piece-removed / opponent-piece-removed). The +earlier bishop-vs-rook inconsistency vanished entirely at the +population level, confirming it was per-position noise, not a real +defect. Scaling is compressed relative to nominal chess piece values +(queen/pawn ratio ~5.6x here vs. nominal 9x) -- expected and likely +correct, not a bug, since training targets are WDL-sigmoid-calibrated +scores, not raw material counts, and a large material edge's marginal +effect on win probability is genuinely smaller than its point value. + +This tool (`annotate_pgns/sanity_check_net.py`) should be run on any +future checkpoint before trusting a single hand-picked-position read +-- `--cross-sectional --n-positions 2000` against a holdout pool, +`python3 sanity_check_net.py --ckpt <path>` for the quick hand-picked +smoke test. + +### Export and C-side validation: clean + +`export_nnue_weights.py` (already quantizing directly to the `NNU3` +format, no separate float32-only export step in the current pipeline) +ran against the matprobe checkpoint with zero code changes needed -- +it only reads `ft`/`fc1`/`fc2`/`fc3` by name, so the auxiliary +`material_probe` layer is silently and correctly ignored at export +time. 6 real holdout FENs spot-checked, Python float vs. C +int16/int8-quantized: deltas of 0.3-4.7cp, same ballpark as the +1.55cp-mean/20.91cp-max already measured and accepted for the first +net's quantization step. No sign/implementation bug found. + +### Five independent head-to-head trials, all losses + +All via `eval_tune/test_vs_head.sh` with `NNUE_WEIGHTS` set (and, for +two of the five, `NNUE_PROBE_WEIGHTS` pointing at the OLD net's lazy +probe -- see the stale-probe caveat below), against the classic-HCE +`head_reference`-equivalent head binary: + +| # | net | time control | result (at stop) | llr | candidate depth vs head | +|---|---|---|---|---|---| +| 1 | matprobe (new) | `st 4` | -27 =2 +1 (n=30), score 0.067 | -2.07 | 12.16 vs 15.08 | +| 2 | v3 (old) | `st 4` | -29 =1 +0 (n=30), score 0.017 | -13.05 | 10.74 vs 13.51 | +| 3 | v3 (old) | `sd 10` | -28 =2 +0 (n=30), score 0.067 | -1.52 | 9.57 vs 9.23 | +| 4 | matprobe (new) | `sd 10` | -24 =4 +2 (n=30), score 0.133 | -0.97 | 9.81 vs 9.70 | +| 5 | v3 (old) + lazy probe | `sd 10` | -30 =0 +0 (n=30), score 0.000 | (total wipeout) | 9.63 vs 9.25 | + +**Key findings**: +- **The new (matprobe) net consistently loses less badly than the old + net, under every condition tested** (trial 1 vs 2 at `st 4`: score + 0.067 vs 0.017; trial 4 vs 3 at `sd 10`: score 0.133 vs 0.067). The + material fix is a real, reproducible improvement -- just nowhere + near enough to close the gap with HCE. +- **At matched `sd 10` depth (trials 3 and 4), NNUE still loses + decisively** -- this directly contradicts the working hypothesis + (built from the ECM result above) that the real-time losses were + purely a depth/speed-handicap artifact. They are not purely that. + Matching the final depth number does not mean matching search + quality throughout the tree -- EFP/LMR/NMP/futility margins are + still HCE-tuned in every one of these trials (see "Integration into + search" above, still not done) and could still be causing bad + cutoffs specific to NNUE's score distribution even when the + reported depth matches. +- **The lazy probe used in trial 5 is STALE and the result should be + discounted**: `annotate_pgns/nnue_probe.bin` was fit against the OLD + net's (v3) accumulator specifically, before this session's retrain. + It happens to be the correct vintage for trial 5 (v3 + v3's own + probe), so that comparison is internally consistent, but no probe + exists yet for the matprobe net -- `typhoon_nnue`'s wrapper script + was updated this session to point at the matprobe net by default + and to NOT fall back to the stale probe file (previously silent and + wrong). A new probe would need to be fit + calibrated from scratch + against the matprobe net's accumulator before it could be used in a + trial 4-equivalent run. +- **A user-remembered "NNUE competitive or better" result could not + be reproduced under any of these five conditions.** Likely + explanations, unconfirmed: a different match configuration entirely + (not `st 4` or `sd 10` against this head binary), or a misremembered + result. Do not plan further work assuming that result is real until + it can be reproduced under a specific, documented configuration. + +### Pipeline sanity check: ruled out a structural bug + +Given five straight losses, including at matched depth where ECM +predicted an edge, the obvious question was "is the training pipeline +itself broken" (bad feature indexing under real gradient flow, broken +accumulator math, a training-loop bug) rather than a genuine +net-quality gap. Tested directly (`annotate_pgns/ +overfit_sanity_check.py`, new tool): sampled 3000 real positions, +scored each with the engine's own classic `Eval()` (fast +persistent-context `eval` command), and trained the *same* +architecture to directly regress raw output against that score (plain +MSE, no sigmoid, no K, no λ, no quiet filter, no quantization -- the +simplest possible test of "can this architecture memorize a small, +clean, noiseless target at all"). **Result: clean overfit** -- RMSE +collapsed from 307cp to ~3-5cp over 400 epochs on the fixed 3000-point +set (max error 46cp). **No structural bug found.** The five +head-to-head losses are real evidence about net/training quality, not +a pipeline defect. + +### Revised understanding of why: likely bootstrapping off HCE's own +### search sets a ceiling, not (as earlier argued) label noise or +### architecture capacity + +Two explanations floated earlier in this investigation don't hold up +well under scrutiny: +- **Architecture capacity**: the 256x2-32-32-1 shape is the literal + one Stockfish's own early, successful NNUE work used. Not a good + suspect. +- **"Search-score labels are noisy"**: standard practice (`sd 8`-ish + search-score training is exactly what real NNUE projects use); the + depth/source of search isn't inherently suspect. + +**The sharper version of the concern**: every label in the current +training pool -- TWIC and self-play both -- comes from THIS SAME +engine's own classic `Eval()` run through its own `sd 10` search, and +training used `λ=1.0` (pure `p_search`, zero weight on actual game +`r_outcome`). A net trained to perfectly reproduce "HCE's static eval, +distilled to one forward pass" can at best approach HCE's *positional +judgment* -- it has no training signal that could let it exceed HCE's +actual playing strength, which is that judgment plus a mature, +decades-tuned full search. This is single-generation distillation +from the exact opponent being benchmarked against, with no +independent ground truth in the loop. Real NNUE bootstrapping success +stories (including Stockfish's own) iterate through many generations, +each net's own self-play becoming the next round's data, with game +outcomes mattering throughout -- not one shot of pure +distillation-from-one-engine's-shallow-search. + +**This reframes the earlier-rejected λ-sweep as an open question +again, not a settled one** (see the "λ-sweep methodology was weaker +than later standards" note above) -- a properly-controlled λ=0.5 run +(combined with the material-probe fix, trained to real convergence, +judged with the cross-sectional population check, not a single +hand-picked position) is in progress as of this writing to answer +this properly. + +### Next up + +1. **In progress**: λ=0.5 + material-probe-loss combined run, trained + to convergence, judged with `sanity_check_net.py --cross-sectional` + -- the properly-controlled version of the λ-sweep question above. +2. Once a net shows a real head-to-head result against HCE (win, draw, + or a meaningfully-less-bad loss than trial 4's 0.133), THEN invest + in margin recalibration (EFP/LMR/NMP/futility, still 100% HCE-tuned + today) and a new lazy-probe fit for whichever net is being kept -- + not before, per this session's experience chasing search-side + explanations that turned out to not be the dominant factor. +3. Data volume scale-up (TWIC + self-play, more self-play generations + specifically using the matprobe-or-later net to play, per the + original Phase 2 plan) remains the most clearly-actionable lever, + but should be paired with an actual outcome-grounding signal in the + training target, not just more HCE-sd10-labeled positions of the + same shape already in hand. + +## Status (2026-10-18): Stockfish-labeled relabel in progress; feature +## factorization planned as the next architecture-side lever + +Labels are being regenerated using Stockfish (not this engine's own +`Eval()`/`sd 10`) to break the single-generation self-distillation +problem described above -- that's a speed/ground-truth fix, not +believed to be the root cause of the five head-to-head losses by +itself. In parallel, researching Stockfish's own NNUE docs surfaced a +second, independent candidate explanation worth trying once there's +enough Stockfish-labeled data to re-run training: **feature +factorization** (Stockfish's term for a training-only "virtual +feature set" that is coalesced into the real weights post-training; +see https://github.com/official-stockfish/nnue-pytorch/blob/master/docs/nnue.md#feature-factorization, +excerpted into this conversation). + +### Why this looks like a real lever here, not just a ported idea + +The current feature set is HalfKAv2_hm per the canonical spec above: +22528-dim, king-bucketed (32 buckets via `king_bucket()`'s +horizontal-mirror quartet), so every `(piece_square, piece_type, +piece_color)` combination exists as up to 32 separate rows, one per +king bucket, each seen only when the king happens to sit in that +bucket. Checked directly against the current real training cache +(`runs/sd10_twic_2M/tensors_train`, 1,848,411 positions): + +- **23.9% of the 22528 HalfKAv2 rows have zero observations** -- + never fire once in training, never receive a gradient update. +- **60.5% have fewer than 50 observations.** +- Collapsing the same data to the king-bucket-independent "P" feature + set (704 rows: `piece_square x piece_type x piece_color`, no king + info) gives a median of **8375** observations per row -- roughly + 32x denser, as expected, with none left permanently untrained. + +So a substantial fraction of the net's weight space is currently +stuck at random init or barely trained regardless of label quality -- +plausibly part of why two different training attempts "worked" +(loss went down, no crash) but neither net could beat HCE in +head-to-head play. This is Stockfish's own "dusty corners" framing: +sparsity from king-conditioning starves rare (king, piece, square) +combinations of signal, which factorization is specifically designed +to fix during early training. + +### Planned implementation (not yet done -- next after enough +### Stockfish-labeled data exists to re-run training) + +1. **No change to `build_training_tensors.py` / no cache rebuild + needed.** The HalfKAv2_hm index formula is + `idx = (sq ^ mask) + piece_plane + king_bucket * PS_NB` where + `(sq ^ mask) + piece_plane <= 703 < PS_NB (704)`. So the P-factor + index is exactly `real_idx % 704` -- recoverable from the existing + `white_idx`/`black_idx` tensors at train time, in `nnue_model.py`. +2. **`nnue_model.py`**: add a second embedding table, + `self.ft_p = nn.EmbeddingBag(704, l1, mode="sum")`, **zero-initialized** + (matters: both tables sum into the same accumulator, so a + default-init P table would double the accumulator's scale at step + 0, confounding the factorization effect with an effective LR + change), with no bias of its own (`ft_bias` stays as the only + bias, or coalescing below isn't exact). In `accumulator()`, derive + `white_idx % 704` / `black_idx % 704` and sum `ft_p`'s output into + each perspective's accumulator alongside `ft`. +3. **Skip the "K" and "HalfRelativeKP" factors** Stockfish's doc also + describes. HalfKAv2_hm already encodes the king directly (kings + are included in the `piece_map()` loop via `piece_plane()`'s + `PS_KING` case), so the K factor would be redundant here. One + variable at a time, per the incremental-change rule used + throughout this project -- P alone first, measure, then decide. +4. **Coalescing after training, in `export_nnue_weights.py`**: this is + the step most likely to silently fail, because a bad coalesce + produces a net that is *worse* than not factoring at all (the real + HalfKAv2_hm rows learn less because the factor absorbed the shared + signal) and the training loss curve won't show it either way. + `ft.weight += ft_p.weight[torch.arange(22528) % 704]`, then drop + both `ft_p` and `material_probe` from the exported weights (two + training-only heads now, same pattern). **Required check**: assert + the pre-coalesce factored forward pass and the post-coalesce + plain-`ft`-only forward pass agree to ~1e-5 on a real batch before + trusting the export. +5. **Evaluation protocol**: Stockfish's doc is explicit that the + factorizer's effect is concentrated in *early* training and fades + as training progresses -- compare early-epoch loss descent against + a matched-seed, matched-epoch unfactored control, not just final + validation loss. A final-loss-only comparison could read a real + early-training effect as zero. + +### Sequencing + +This factorization work is queued behind the in-progress Stockfish +relabel -- the two are independent, complementary levers (ground-truth +quality vs. training-signal density under sparsity), and per this +project's incremental-change discipline they should be evaluated one +at a time rather than conflated in a single before/after comparison. + +## Status (2026-10-18, continued): Stockfish-labeled corpus built, P-factor +## factorization implemented and measured -- why it's load-bearing, not optional + +Both levers above are now done, in sequence, against the new +`pgn/stockfish_self_play_training_data.pgn` corpus (35,295 Stockfish- +vs-Stockfish games, 4,008,045 positions, labeled via Stockfish's own +search rather than this engine's `Eval()` -- see `extract_selfplay_ +positions.py`'s `MATE_CLAMP_THRESHOLD=90.0`, needed because this +foreign-engine corpus encodes mate as a clamped float, `[99.00,99.99]` +pawns via PolyGlot's UCI-to-xboard bridge, not typhoon's own `+M`/`-M` +sentinel token). Full pipeline run at `lambda=0.75` (`target = +0.75*p_search + 0.25*r_outcome`, chosen over the Phase-A `1.0` default +specifically because pure single-engine-distillation was the leading +suspect for the prior two nets' failure to beat HCE): `split_holdout.py` +-> `fit_k_and_label.py` (K=2.4130, this corpus's own fit, NOT the TWIC- +fit `DEFAULT_K=1.3123` -- the two corpora have different score +distributions and must not share a K) -> `quiet_filter_games.py` -> +`build_training_tensors.py` (now with `--workers`, default 16, +multiprocessing-sharded across positions, verified byte-for-byte +identical output to the old sequential path -- the per-position +feature-index computation has no cross-position dependency, so this +was a clean win on an idle multi-core box). + +**Why factorization is necessary here, not a nice-to-have**: measured +directly on this corpus's real training tensors (not a TWIC estimate), +2,237 of HalfKAv2_hm's 22,528 rows (9.9%) never fire once across all +2.84M training positions -- confirmed, after training, to be **bit- +for-bit identical** to their random pre-training init (diffed the +trained checkpoint's `ft.weight` against a freshly re-seeded, +untrained `NnueNet()` at the same seed: max difference on those rows +is exactly 0.0, vs. a clearly nonzero shift on every observed row). +This 9.9% figure is *better* than TWIC's 23.9% (this corpus's +decisive, played-to-conclusion games naturally cover more king +positions than human-resignation-biased TWIC), but still real, and +**not fixable by collecting more data at any practical scale**: +subsampling the existing pool at increasing sizes and fitting the +shrinkage curve (`zero_count ~= 5529 - 226*ln(N)`, log-linear, fit on +6 points from 10% to 100% of the current 2.84M-position pool) says +halving the dead-cell count to ~1,118 needs ~290M positions (~100x +the current pool), and getting under 500 needs ~4.4B (~1,560x). Brute- +force self-play volume cannot close this gap; it needs an +architecture-side fix, which is exactly what factorization is for. + +**Implementation** (`nnue_model.py`, `export_nnue_weights.py`), per +the plan above, built essentially as specified: a second, zero- +initialized `ft_p` `EmbeddingBag(PS_NB=704, l1)`, summed into the +accumulator alongside `ft` via `idx % PS_NB` (same bag offsets, no +`build_training_tensors.py` changes needed), coalesced into `ft.weight` +at export time (`ft.weight += ft_p.weight[arange(22528) % 704]`) with +a required pre/post-coalesce forward-pass agreement check (measured +3.58e-07 on a random-weighted smoke test, comfortably inside the +~1e-5 tolerance) before `ft_p` is dropped from the exported file. + +**Measured effect, matched-seed/epoch (lambda=0.75, K=2.4130, +material-loss-weight=0.001 -- see the next note on why that changed +from 0.01, 34 epochs, otherwise identical hyperparameters)**: +- Holdout MSE at matched checkpoints: factored beats unfactored at + every logged step, not just early training (contra the "effect + fades" caveat in the plan above, which turned out not to dominate + here) -- e.g. step 4000/epoch 11: 0.01193 (factored) vs. 0.01446 + (unfactored); final epoch 33: 0.01201 (factored) vs. 0.01331 + (unfactored). +- `sanity_check_net.py --cross-sectional` (2000 sampled holdout + positions, final checkpoints): both nets monotonic in piece value, + but factored's queen/pawn ratio is closer to nominal (~8.1x own / + 9.8x opponent-removed, vs. the already-decent unfactored run's + ~8.2x/9.9x -- both well above the old TWIC-trained matprobe net's + ~5.6x). +- Compared against a REAL Stockfish net's NNUE output (see next + section) on 1000 shared holdout positions: factored wins on every + metric -- correlation 0.879 vs 0.861, mean|diff| 0.776 vs 0.852 + pawns raw, 0.718 vs 0.762 pawns after fitting away the overall + scale difference (both nets run at ~0.79-0.83x Stockfish's scale, + expected since ours is WDL-sigmoid-calibrated, not raw-centipawn- + trained). + +**`material-loss-weight` lowered from 0.01 to 0.001 for this corpus**, +before any of the above: the original 0.01 (see "material-blindness +fixed" status above) was tuned against TWIC's homogeneous, balanced- +middlegame material distribution, where the raw material-MSE term +was naturally small and its weighted contribution stayed well below +the WDL loss's. This corpus has far more material-imbalanced +positions by construction (decisive self-play games played to real +conclusions, not human-resignation-biased), so the same 0.01 weight +measurably put the material term at parity with the WDL loss rather +than a gentle nudge -- confirmed by briefly launching training at +0.01 before catching it (checkpoints `ckpt_epoch000-001.pt` from that +run were deleted, not used for anything above). + +**Known residual gap, not yet fixed**: `ft_p`'s zero-init was +deliberate (avoids confounding the factorization effect with an +init-time accumulator-scale change), but `ft` itself still uses +`nn.EmbeddingBag`'s default `N(0,1)`-per-entry init, which is NOT +zero -- so an untouched row's random contribution is the same +magnitude (`norm~16`) as a fully-trained row's, not suppressed by +scale. Today this is masked by `ft_p`'s well-trained, shared +correction dominating the sum at those positions (confirmed: 0/1000 +Stockfish-comparison positions activated any dead cell, so this gap +hasn't tainted any reported result above), but it's a real residual +risk for any future position that activates several dead cells +together. Cheap, not-yet-applied fix: zero-init `ft` the same way +`ft_p` already is, so a dead cell degrades gracefully to "contributes +nothing" rather than "contributes untrained noise." + +## Verifying against a real Stockfish net (one-time diagnostic, not a +## pipeline dependency) + +To sanity-check whether the above was in the right ballpark at all (as +distinct from reusing Stockfish's own weights, which was explicitly +ruled out -- the goal is a from-scratch net strong enough to self-play +and iterate on, "eating its own tail," not a transplant), pulled a +real, period-appropriate Stockfish net (`nn-fdc1d0fe6455.nnue`, added +to Stockfish 2023-06-12 per `github.com/official-stockfish/networks`' +commit history, rev `38e61663d836e062af0bc002814ad5149c4b7729`) and +parsed its `FeatureTransformer::psqtWeights` table directly, against +that exact commit's header-reading code (fetched and read, not +guessed -- a first attempt at parsing by eye got the header layout +wrong and was caught before trusting any conclusion from it). + +- **Confirmed identical feature space**: `HalfKAv2_hm::HashValue = + 0x7f234cb8` and `Dimensions=22528` match our own spec exactly (same + king buckets, same orientation table, same piece-square planes); + this net's stored `ft` hash decodes to `L1=1536` (vs. our 256) via + `ft_hash = HashValue XOR (2*L1)`. +- **`psqtWeights[index][bucket]` for our own 2,237 dead-cell indices + looks completely ordinary** relative to that plane's overall average + (e.g. queen plane: 2487cp mean on dead cells vs. 2541cp overall) -- + i.e. our dusty corners aren't inherently weird positions, just a + sampling gap in this specific corpus, which is exactly the pattern + factorization is built to exploit (the true per-piece value is + roughly king-bucket-independent). +- To get a real NNUE eval from this old net for the comparison above, + built Stockfish from source at the exact matching commit + (`gmake -j$(nproc) build ARCH=x86-64-avx2 COMP=clang` -- note + FreeBSD's default `make` is BSD make and cannot parse this + Makefile's GNU-Make syntax, must use `gmake`), rather than risk + hand-reimplementing its 8-layer-stack quantized forward pass. + Compared its `eval` command's "NNUE evaluation" line (pure NNUE + output, no classical-blend) against our own `forward()`, both + converted to white-relative, over 1000 shared holdout FENs -- see + `annotate_pgns/compare_vs_stockfish_net.py` and the factorization + status note above for the numeric results. + +## Iterating on a net: adding a new generation of self-play data + +The first net to actually beat (or seriously contest) HCE at matched +`sd 10` is the `lambda=0.75`, P-factored checkpoint from the status +above -- the point where self-play with this net starts producing a +genuinely bigger/more-targeted training pool becomes real ("eating its +own tail", per the conversation that drove this project's current +phase). This section is the checklist for folding a new batch of +self-play games into the next training generation -- written down so +it doesn't need re-deriving from scratch (or re-making the same +mistakes) each time. **None of this requires a new architecture or a +new `nnue_model.py` -- same model class, same feature set, same +`--resume` mechanism.** It's a data/calibration discipline problem, +not a code problem. + +### 1. Combine, don't just append + +Pool the new self-play positions together with the existing training +corpus (corpora can mix freely at the games.jsonl level -- see +`extract_selfplay_positions.py`'s docstring) and re-run the **full** +pipeline on the combined set: + +``` +split_holdout.py -> fit_k_and_label.py -> quiet_filter_games.py -> build_training_tensors.py +``` + +Do NOT train on only the new self-play batch by itself, even via +`--resume`. Self-play games cluster around positions the current net +already favors (that's what self-play means) -- training on only the +new batch narrows the effective training distribution and risks +forgetting what the original, more diverse corpus taught. The combined +pool is also exactly how the dusty-corner coverage problem (see the +factorization status above) actually improves over time: new self-play +games sample different king positions/endgame structures than the +corpus that came before, so the pooled coverage count should be +re-measured after each new generation (same `np.bincount` check used +throughout this doc) to see whether it's actually shrinking the dead- +cell count, not just assumed. + +### 2. Refit K on the combined pool -- never carry an old K forward + +`fit_k_and_label.py`'s K is fit to one specific corpus's own score +distribution. **"Fitted K = 2.4130 -- record this; applying a different +K later (e.g. after labeling more data) is not the same model"** (the +tool's own exit message) applies in the other direction too: once the +pool changes, K must be refit on the new combined TRAIN split, not +reused from the previous generation. Training against a target +calibrated for a different score distribution than the one actually +being trained on silently miscalibrates every label, the same class of +problem as DNA-tuning's holdout-leakage concerns elsewhere in this +project. + +### 3. Watch the label source -- this is the one that actually matters + +How were the new self-play games scored? + +- **If an independent, stronger reference (Stockfish, or any engine + not being trained) produced the centipawn labels**, this is safe + regardless of which engine(s) played the moves -- self-play between + two candidates is a great way to source diverse, targeted positions + (specifically ones that stress whatever the current net is weakest + at), as long as the position doesn't then get "approved" by asking + the same net being trained to grade itself. +- **If the candidate net's own search produced the score labels for + its own games**, this reintroduces the single-generation self- + distillation problem already diagnosed as the likely reason the + first two (pre-Stockfish-relabel) nets failed to beat HCE at all + (see the "Revised understanding of why" status note above) -- a net + trained to reproduce its own judgment can at best match that + judgment, never exceed it. Mitigate, don't ignore, by keeping + `lambda < 1.0` in `build_training_tensors.py` so real game outcomes + (`r_outcome`) stay in the training target regardless of where the + centipawn half of the blend came from -- outcomes are ground truth + no self-distillation loop can corrupt, even when the score labels + themselves are suspect. + +### 4. Re-check `material-loss-weight` against the new pool's material + distribution + +`material-loss-weight` was lowered from `0.01` (tuned for TWIC's +balanced-middlegame distribution) to `0.001` specifically because this +project's Stockfish self-play corpus has far more decisive, material- +imbalanced games. A new self-play generation's material distribution +is a new empirical fact, not something to assume stays `0.001` forever +-- if candidate-vs-candidate games turn out far more drawish/balanced +than the original bootstrap corpus (plausible once both sides are +playing well), the same weight could again land at an unintended +fraction of the total loss. Check, don't assume -- same reasoning as +the original 0.01->0.001 change, just potentially in either direction +next time. + +### 5. Warm-start, don't restart + +`--resume <previous checkpoint>` on the new combined-pool tensors is +the right default -- it's strictly faster than random-init-and-train- +from-scratch, and there's no evidence in this project so far that +warm-starting hurts final quality (nothing here has tested the +opposite, to be clear -- if a future generation's holdout metrics look +worse than a from-scratch control at the same epoch count, that's +worth isolating before trusting warm-starting blindly forever). + +### 6. Validate before trusting the new generation + +In order, cheapest/fastest first -- stop at the first one that fails +rather than running all of them blind: + +1. `sanity_check_net.py --cross-sectional` on the new checkpoint -- + monotonic piece-value deltas, reasonable magnitudes (compare + against the queen/pawn ratio of the previous generation as a + baseline, not just "is it monotonic"). +2. Re-run the dead-cell coverage check (bincount over the new combined + training tensors) -- confirm it's trending down, not flat or up. +3. `compare_vs_stockfish_net.py` against the same real Stockfish net + used above, same held-out FEN sample where practical, to see + whether correlation/scale-fit-residual improved or regressed vs. + the previous generation's numbers (0.879 corr / 0.718 pawns resid. + as of the factored lambda=0.75 checkpoint -- the number to beat). +4. `export_nnue_weights.py` (coalesce check must pass, zero clipped + values) then a real matched-depth (`sd 10`, not `st` -- see the + fixed-depth-for-ECM-comparisons convention used throughout this + project) head-to-head via `eval_tune/test_vs_head.sh` with + `NNUE_WEIGHTS` set, against classic HCE. This is the only step that + actually answers "did this generation get better," everything above + is a cheaper proxy that can catch a regression before burning a + multi-hour match run on it. + +## Adoption criterion: sd-equal parity is not the bar, `st`-equal is -- +## and the net is currently ~3.3x slower per node than HCE + +Measured directly in the first real sd10-matched match (the factored +`lambda=0.75` checkpoint vs. classic HCE, 2026-10-18, 400 games): +candidate nps=470-475K vs. head nps=1.5-1.6M -- **a ~3.3x gap**, not a +rounding error. At matched *depth* this is invisible (both sides reach +sd10 regardless of nps), which is exactly why the sd-equal match is +the right first filter -- it isolates "is the net smarter per node" +from the speed handicap. But a `st`-equal (real time control) match +would currently be lopsided in HCE's favor for a reason that has +nothing to do with eval quality: at this project's branching factor +(ebf ~3.8-3.9, measured in the same match), a 3.3x nps disadvantage +costs roughly a full ply of search depth at any fixed time budget, on +top of whatever the eval-quality gap already costs. Testing at `st` +before closing that gap would answer "how bad is an engine with no +lazy-eval fast path," not "is this net any good." + +**Revised understanding (2026-10-18, same conversation): nps parity is +sufficient, not a blocker -- the real Stockfish binary on this exact +box is the proof.** It searches at ~500K nps -- essentially the same +as our net's measured 470-480K, not the 1.5-1.6M HCE achieves -- and +still beats this project's HCE every single recorded game (not even a +draw). Stockfish doesn't out-node HCE; it out-*prunes* it, searching +dramatically trimmer trees at the same or worse raw nps, because its +eval is trustworthy enough to prune aggressively on. This reframes the +priority between the two levers below: **lazy eval (closing the 3.3x +nps gap) is a nice-to-have, not the thing standing between this net +and real strength.** The actual lever is section 3 of the checklist +below -- null-move R, LMR's reduction table and fail-high prediction, +LMP thresholds, futility margins -- all of which currently encode how +*little* HCE's eval can be trusted. A learned eval earning the right to +prune harder, the way Stockfish's does, is plausibly worth more than +recovering the nps gap ever could be. Lazy eval stays worth doing (free, +no downside), just not gated-on-first the way the staged list below +originally implied. + +**Staged gate, in order -- do not skip ahead:** + +1. **Sd-equal match (done, 2026-10-18)**: necessary-but-not-sufficient + filter. If the net can't hold its own at matched depth, it has no + chance at matched time, since real time controls only make the + speed handicap worse for it, never better. +2. **Recalibrate the pruning-aggressiveness constants (checklist + section 3) -- this is the primary lever, per the Stockfish + comparison above, not a secondary cleanup pass.** Not a full + DNA-tuning sweep, just enough that search isn't making prune/ + reduce/extend decisions calibrated for a different eval's scale and + trustworthiness. Plausibly worth more than nps recovery on its own. +3. **Close the free speed gap too** -- lazy eval (checklist section 1), + still worth doing since it's pure engineering with no downside (it + doesn't change what the net *thinks*, only how often it's asked to + think at nodes that get pruned anyway) -- just not the bottleneck + the Stockfish comparison suggested it might be. +4. **Only then, a real `st`-equal SPRT against HCE.** This is the step + that actually answers the adoption question -- everything before it + answers a necessary precondition, not the question itself. + +**The adoption threshold itself shouldn't be pure parity-or-better, +though.** HCE is a mature, hand-tuned eval already deep into +diminishing returns on its own DNA-tuning pipeline. NNUE has an +open-ended improvement path still mostly unexploited (more self-play +generations per the iteration section above, factorization already +landed, a wider L1 if a Phase B GPU run ever happens -- see the real +Stockfish net's L1=1536 vs. our 256 as a sense of the room available). +A net that's close-but-not-quite-parity at a genuinely fair (`st`- +equal, post-calibration) time control may still be worth adopting if +the plan is to keep training it -- the alternative is maintaining two +eval paths indefinitely while one's ceiling keeps rising and the +other's doesn't. Close-and-improving beats even-and-stagnant. + +## Engine calibration checklist: what needs re-tuning for a new eval + +This engine's search has several subsystems whose margins/tables were +fit against classic `Eval()`'s specific output scale and noise +profile. None of this is required for the net to *run* (it already +does, correctly, per the wiring in `search.c`/`nnueeval.c`) -- it's +required for the net to run *well*, since an uncalibrated search is +fighting the new eval with pruning/reduction decisions tuned for a +different one. Confirmed against this engine's actual current search +(2026-10-18 conversation; EFP and ProbCut were previously assumed +present and are not -- corrected below rather than left as dead +checklist items): + +**1. Free speed, no behavior change (do this first, before any of the +below -- see the adoption-criterion note above for why ordering +matters here):** +- Lazy eval fast path -- designed (architecture decisions above: reuse + the net's own cheap linear/PSQT-ish term -- here `ft_p`/ + `material_probe`-shaped -- as a first-pass filter, falling through + to the full forward pass only when inconclusive vs. alpha/beta), not + yet built. This is the single biggest lever on the measured 3.3x nps + gap and touches zero tuning -- it's the right first move specifically + because it can't make the net play worse, only faster. +- Deferred/lazy accumulator maintenance (flagged in the accumulator + design section above as a "performance follow-up, not required for + correctness") -- avoid eager accumulator updates at nodes pruned + before any eval is ever consulted. + +**2. The raw-score-to-cp mapping -- verify this before touching any +margin below, since it's a multiplier on all of them:** +Every margin in section 3 is a flat cp (or near-cp) threshold tuned +against HCE's output scale. The trained net runs at ~0.8x a real +Stockfish net's scale (measured via `compare_vs_stockfish_net.py`) and +is WDL-sigmoid-calibrated (K=2.4), not raw-centipawn-trained. If +search currently treats `forward()`'s raw output as if it were +already in HCE's cp convention, every margin below is off by that +scale factor before even considering noise-profile differences. This +needs the inverse-sigmoid-to-cp conversion the original architecture +decision already called for ("Requires a calibrated inverse-sigmoid +conversion back to centipawns... that conversion constant needs its +own calibration pass") -- confirm it exists and is right once, rather +than chasing a scale error through a dozen separately-miscalibrated +margins. + +**3. Score-scale/noise-dependent pruning and reduction constants -- +this engine's actual current set (corrected 2026-10-18: no EFP in main +search, only LMP; no separate razoring step -- LMR fills that role +here; no ProbCut; QSearch futility and "delta pruning" are the same +mechanism, not two):** +- QSearch futility margins +- Static/reverse null-move pruning margin +- Null-move reduction depth R, and any eval-based skip/verification + gating on the null-move try itself +- LMP move-count table, `data.c` +- LMR reduction table, `data.c`, **and** the joint fail-high + prediction table feeding it (see the LMR sd11/sn10M protocol note + elsewhere in this project) -- this table is a direct statement about + how trustworthy the eval is at a reduced-depth node, fit entirely + against HCE's noise profile +- Root aspiration window initial width and re-widening schedule on + fail-high/fail-low (tuned against HCE's iteration-to-iteration score + noise -- a good catch, easy to overlook since it's at the root, not + buried in the main search loop) + +**4. Eval-component-dependent heuristics (lower priority -- already +confirmed to degrade gracefully under a lazy/missing full eval per the +"Breaking external dependencies" section above; re-measure how often +they actually fire once NNUE lazy eval ships, don't re-derive from +scratch):** +- `EstimateHangingMaterialCp`'s `cTrapped`-fallback "dumb move" axis +- `ValueOfMaterialInTroubleDespiteMove`'s QSearch stand-pat gate + +**5. Currently dormant, no action needed yet:** draw/contempt scoring +is built but always returns 0 right now -- no live miscalibration risk +today, but if it's ever turned on, the same cp-scale concern in +section 2 applies to whatever constant drives it. -## Immediately actionable, unblocked work +## Status (2026-10-18, continued): pipeline-scaling audit, three gaps fixed, end-to-end driver added -Not gated behind any engine-internals decision above: +Prompted by generating a much larger volume of annotated PGN than +before and wanting to confirm the post-annotation Python pipeline +(`annotate_pgns/`) actually uses all available parallelism rather than +silently bottlenecking on a single-threaded stage. `annotate.py` +itself (the sd-N search stage) was already confirmed fine -- sharded +across N single-threaded engine subprocesses -- and +`build_training_tensors.py` already had a `--workers 16` +`multiprocessing.Pool`. Three real gaps found and fixed; one +unrelated question (an end-to-end PGN-to-`.nnue` driver) answered by +adding a new orchestrator script. -- Build the annotation harness and begin scoring `lichess_elite_*`/ - `twic_filtered.pgn` games (Phase 1 training data). -- Investigate `<TT>`/`<EGTB>`/`<REC>` tag semantics in the self-play PV - printer. -- Finalize the written HalfKAv2_hm feature-indexing spec (design-only, - no code). +- **Gap: `quiet_filter.py` had no parallelism at all.** It's a + single-threaded Python loop doing `chess.Board()` construction + SEE + per row over the entire corpus (2.5M+ rows for the Stage A run) -- + the one CPU-bound step in the non-annotation part of the pipeline + with zero parallelism, unlike `build_training_tensors.py`'s `Pool` + right next to it. **Fixed via a new wrapper, `shard_quiet_filter.py`**, + rather than rewriting `quiet_filter.py` itself: round-robin-splits + the input across `--workers` (default 16) temp shard files, runs + that many `quiet_filter.py` subprocesses concurrently, concatenates + their outputs. Deliberately reuses `quiet_filter.py` unmodified -- + one definition of the filter, reused as a subprocess, the wrapper + only owns sharding/recombining. Row order in the combined output + does not match the input (shards are concatenated shard-by-shard, + not interleaved back to original order) -- harmless here since + `positions.quiet.jsonl` is a flat, order-independent training-set + input. Verified end-to-end on a synthetic 40-row fixture: 40 in, 40 + out, across 4 shards. +- **Gap: `fit_k_and_label.py`'s ternary-search K-fit was unvectorized + Python.** `fit_k()`'s 40 ternary-search iterations each call `mse()` + twice, and the old `mse()` was a plain Python `for` loop over every + (score, outcome) pair -- roughly 80 full passes over ~2M+ pairs in + pure Python for a step that does very little actual compute. + **Rewritten to numpy** (`sigmoid`/`mse` as vectorized array ops over + the full pair set) **and sharded** across `--workers` (default 16) + via `multiprocessing.Pool`, for *both* halves of the file: the + calibration-pair collection (`collect_calibration_pairs`, now + `collect_chunk` workers each vectorizing one chunk of games with + numpy, combined via `np.concatenate`) and the labeling pass + (`label_file`, now `label_chunk` workers, each vectorizing one + game's non-exact positions' sigmoid computation per chunk) -- + `Pool.imap`-based, order-preserving, mirroring + `build_training_tensors.py`'s existing chunking pattern. Verified + end-to-end on a synthetic 50-game fixture (4 workers, chunk-size 7): + correct game/position counts, K fit, both splits labeled. +- **Gap: `nnue_train.py`'s CPU thread default was lower than the rest + of the pipeline's convention.** `--threads` default bumped 8 -> 16, + matching the `--workers 16` convention used everywhere else in this + pipeline (kept below this box's 24 cores -- see below -- rather than + maxed out, so other jobs on the shared box keep some breathing + room). +- **Core-count note, not acted on:** this box has 24 cores + (`nproc`), so every one of these `--workers`/`--threads`/ + `annotate.py --shards` defaults (16) intentionally leaves some + headroom rather than using all 24 -- a deliberate choice to leave + the box's other jobs breathing room, not an oversight. `annotate.py` + Stage A's own measured throughput (NNUE.md's "Status (2026-10-14)" + section: ~75 pos/sec aggregate at 16 shards vs. ~112/sec + extrapolated from one shard x16, attributed to contention) is a + reminder that this box's parallelism doesn't scale linearly past a + point anyway -- blindly raising shard/worker counts further is not + assumed to help without measuring first. +- **New: `run_nnue_pipeline.py`**, an end-to-end orchestrator from a + glob of input PGN to a `.nnue` weights file, requested directly as + "can one script run the whole training process end to end." Targets + **already-scored PGN** specifically (match_play.py's + `"{+-}NN.NN/DEPTH PV..."` per-move comment format -- e.g. + `pgn/stockfish_self_play_training_data.pgn`, `pgn/self_play_games.pgn`), + not raw human-game PGN needing a fresh sd-N search -- that's + `annotate.py`'s job and is a different, not-yet-automated entry + point. Chains, in order: `extract_selfplay_positions.py` (once per + input file, since that script takes a single PGN; outputs + concatenated) -> `split_holdout.py` -> `fit_k_and_label.py` -> + `quiet_filter_games.py` (both splits) -> `build_training_tensors.py` + (both splits) -> `nnue_train.py` -> `export_nnue_weights.py` (against + the latest `ckpt_epochNNN.pt` written -- `nnue_train.py` has no + separate "best checkpoint" selection yet, so "latest" is the only + option today). Every stage checks whether its expected output + already exists and skips it unless `--force` is passed -- the later + stages here can run for hours, so a crash/interrupt partway through + should not mean restarting from stage 1. Passes `--workers`/ + `--threads` through at the same default of 16 used everywhere else. + A thin orchestrator by design: shells out to the existing, + individually-tested scripts via `subprocess` rather than + reimplementing any of their logic. Argument parsing, `--help`, and + the glob-miss error path were smoke-tested; the full multi-stage run + (which needs the engine binary, real PGNs, and hours of wall clock) + was not run end-to-end as part of this change. -Gated behind opening a dedicated branch (per discussion -- not yet -started): +## TODO (not yet implemented) -- Any `SEARCHER_THREAD_CONTEXT`/`PLY_INFO`/MakeMove/UnmakeMove changes. +- **Horizontal-flip data augmentation, in the position loader + (`build_training_tensors.py`).** HalfKAv2_hm's king-bucket mirroring + does NOT tie piece-square weights across the a/h files -- the + `OrientTBL` XOR step re-orients the board onto the king's own half + but a piece's *file within that half* still maps to a distinct + feature row, so a↔h-mirrored positions are not already folded + together by the architecture. Randomly mirroring positions + left-right at load time (flip every square's file, recompute both + perspectives' indices, swap castling rights symmetrically where + applicable) is a legitimate, standard way to roughly double the + effective sample count per feature row without changing dataset + size on disk. Directly relevant given the measured sparsity + (23.9% of HalfKAv2 rows with zero observations, per the + 2026-10-18 factorization status above) -- should be evaluated + alongside P-factorization, not instead of it, since the two + address the same sparsity symptom through different mechanisms + (more effective samples per row vs. a denser auxiliary feature + set) and their effects may not be additive. +- **L2 weight decay in `nnue_train.py`'s optimizer (AdamW).** Pulls + rarely-updated weights (the same sparse/underexposed rows the + factorization work above targets) toward zero instead of leaving + them to drift on whatever noisy gradient they happened to last + receive. Cheap -- a single optimizer hyperparameter, no + architecture or export-format change -- so there's no reason to + gate it behind the factorization/augmentation experiments above; + can be added and swept independently. |