# Replacing Eval() with an NNUE evaluator Status: planning / discussion only. No code written yet. This document records the decisions made so far and the open items that still need resolving before implementation starts. See `MOVE_GUESSER.md` for the related-but-separate move-prediction ("policy") model idea, which is tracked independently and is not a dependency of this work. ## Why Current `Eval()` is hand-tuned (see `eval_tune/`'s DNA tuning pipeline, 622/879 ECM after a first tuning pass vs 606/879 baseline). NNUE-style learned evaluation is the next large step up in evaluation quality, at the cost of a meaningfully larger implementation + retraining project. ## Architecture decisions made so far - **Feature set: HalfKAv2 (with the `_hm` horizontal-mirroring variant as the likely final choice)**, not HalfKP and not a custom-invented encoding. HalfKA includes the kings themselves as features (unlike HalfKP, which excludes them), which captures king-vs-king spatial relationships -- material for endgame/mating-net precision. `_hm` roughly halves parameter count by exploiting left-right board symmetry. Chose a known, documented format over inventing one specifically so existing reference trainers/tooling (e.g. nnue-pytorch) and existing write-ups are usable as a sanity check, rather than debugging both a novel architecture and a novel implementation simultaneously. - **King bucketing: use HalfKAv2_hm's standard bucketing scheme as-is**, not a custom bucket count. (Background: bucketing groups king squares so sub-tables of weights are shared across squares in the same bucket -- fewer buckets means less king-position nuance but faster training convergence per bucket and a smaller table; more buckets is the opposite tradeoff. Not worth re-deriving when a settled default exists.) - **Net topology: start from a known, previously-validated small architecture** (the kind of 256x2-32-32-1-ish shape used in early nnue-pytorch default configs) rather than choosing dimensions from scratch. Revisit sizing only after a first working net exists. - **Output convention: single scalar, side-to-move-relative**, matching current `Eval()`'s convention. Achieved the standard way: two king-relative accumulator halves (one per color's perspective) are concatenated in an order determined by side-to-move (own-perspective half first) before the hidden layers, so the net's one scalar output is side-to-move-relative by construction. Explicitly rejected a two-output (white-score, black-score) design -- that reintroduces a combination-rule-design problem the standard trick already avoids, and we already committed to following known designs over inventing our own. - **SIMD target: AVX2 + FMA.** Confirmed on the actual build box (AMD Ryzen 9 3900X, FreeBSD, clang 19.1.7): `clang -march=native -dM -E -` reports `__AVX2__`, `__FMA__`, `__SSE4_1__`/`__SSE4_2__` defined, no AVX512 (Zen 2 doesn't have it). Quantization/SIMD inference code should target AVX2 as the baseline, not assume AVX512 availability. - **Quantization: int16 for the (sparse) input/accumulator layer, int8 for the dense hidden layers**, matching why Stockfish's own implementation splits it this way -- the input layer's cost is dominated by sparse incremental updates (only ~a handful of features change per move), where int16's SIMD width cost is small, while the dense later layers benefit more from int8's 2x SIMD lane packing. Narrower quantization = faster but lossier; this split is the standard compromise. Needs a quantization-aware scaling pass per layer during export, not just truncate-and-ship. - **Lazy evaluation / fast first-pass filter: reuse a cheap linear subcomponent of the SAME trained net (its PSQT/embedding-sum term, no hidden-layer nonlinearity), not a second, separately-trained smaller net.** Rationale: a separately-trained small net would need its own calibration against alpha/beta margins, and that calibration would almost certainly disagree with the full net's (nonlinear, WDL-sigmoid-derived) output scale -- exactly the problem raised about mixing material-cp lazy skips with a WDL-calibrated net output (see below). A cheap *partial evaluation* of the one trained net lives in the same output space as the full net by construction, so there is no second calibration curve to reconcile. Only fall through to the full forward pass (hidden layers) when the cheap linear term is inconclusive relative to alpha/beta. - Open experiment, explicitly not decided against: a super-lazy *material-only* pre-filter ahead of even the cheap linear net term, purely for search speed. Known risk: current `Eval()`'s super-lazy/lazy/full three-tier system works because all three tiers share units (centipawns, roughly linearly related to material). A material-only prefilter ahead of an NNUE net does NOT have that guarantee once the net's output has gone through WDL-sigmoid calibration -- a position could look "way above beta" in raw material and still not clearly fail high in the net's (compressed, nonlinear) output space. Worth trying, worth measuring carefully (fail-high-rate-at-the-margin style instrumentation, not just aggregate solve counts -- see CLAUDE.md's "Dynamic move ordering experiments" methodology), explicitly allowed to fail and be reverted. - **Training data score space: train against win/draw/loss outcomes via a sigmoid-squashed WDL-style target**, not raw centipawn regression, extending the same sigmoid-squared-error approach `tune_eval_dna.py` already uses. Rationale: raw centipawn MSE misweights the loss (a +500-vs-+700 error matters far less to the actual game outcome than a +20-vs-+80 error, but unweighted centipawn MSE penalizes them similarly) -- WDL-style targets are naturally calibrated to what actually matters (winning the game) and match the training signal already available from game outcomes. Requires a calibrated inverse-sigmoid conversion back to centipawns for use inside alpha-beta search (search needs additive, mate-aware scores, not raw probabilities) -- that conversion constant needs its own calibration pass and is a likely source of subtle bugs if done carelessly. - **Exact scores (mate distance, EGTB/recognizer hits) are never run through the WDL-sigmoid conversion, in either direction.** These are game-theoretically exact, not noisy evidence to fit a sigmoid against. The net itself never needs to learn "is this a mate" -- that detection stays entirely in the search/TB-probe layer exactly as it does today; the net is only ever consulted at ordinary, non-exact nodes. This is a training-data-labeling concern (don't mix exact-value positions into the sigmoid-fit loss, or into material-lazy-skip calibration), not an architecture concern. - **Resolved**: ``/``/`` are sentinel pseudo-moves (`HASHMOVE`, `EGTBMOVE`, `RECOGNMOVE` in `chess.h`) inserted into the PV by `UpdatePV()` (called from `search.c` at the hash-hit, recognizer, and EGTB-probe sites) to record provenance of the line that follows. Distinction that matters for training labels: **``/`` are terminal and exact** -- the node's value literally *is* that recognizer/tablebase result, so a position whose PV ends in one of these tags should have its score excluded from the sigmoid-fit WDL loss (treated as ground truth, not noisy evidence). **`` is NOT an exactness marker** -- it means a hash hit occurred and the remainder of the printed PV was reconstructed by walking the hash table's stored continuation, which can be a fail-high/fail-low *bound*, not necessarily an exact score. A `` appearing anywhere in a PV should be treated as an ordinary search score for labeling purposes, not specially excluded. ## Accumulator maintenance design - The NNUE input state lives as a **persistent "current" blob per thread** (analogous to `POSITION`'s always-current board state), not a recomputed-per-call structure, stored in `SEARCHER_THREAD_CONTEXT` so each `MP=1` helper thread has its own copy. - **Normal (non-king) moves**: MakeMove incrementally adds the changed feature deltas into the live accumulator; UnmakeMove incrementally subtracts the same deltas back out. This is correct without any extra bookkeeping because search's Make/Unmake calls are already properly nested (a recursive call stack) -- UnmakeMove always undoes exactly the delta the matching MakeMove applied. - **King moves**: every feature in HalfKA is king-relative, so a king move invalidates the entire accumulator, not just a delta. Handling: **a per-ply backup**, stored in the existing `PLY_INFO sPlyInfo[MAX_PLY_PER_SEARCH+1]` array already present in `SEARCHER_THREAD_CONTEXT` (`chess.h`) -- add a field there rather than a new parallel array, following the same pattern as `PLY_INFO`'s existing sig-stamped, conditionally-valid fields (`u64FullEvalSig`, `iKingScore`). - On a king MakeMove: memcpy live accumulator -> `sPlyInfo[uPly]`'s backup slot, then recompute live from scratch. - On that same king move's UnmakeMove: memcpy the backup slot back into live (cheaper than recomputing from scratch -- a sequential copy of a ~1-2KB blob vs. ~30 per-piece weight-table lookups/adds across both perspectives). - **Must be indexed by ply, not a single shared backup slot** -- king moves can nest (either side's king can move more than once within a single search line before any of those moves is unmade), and a single shared slot would silently get overwritten by a second, deeper king move before the first one's UnmakeMove runs, producing a wrong-but-undetected restore with no assertion to catch it. Per- ply indexing (bounded by `MAX_PLY_PER_SEARCH`, in practice far fewer slots actually used) avoids this by construction. - **Null move**: moves no piece, so it is a true no-op for the accumulator -- confirm this explicitly in the implementation rather than assuming it falls out automatically. - **Audit required before implementation**: confirm nothing in the current search bypasses paired MakeMove/UnmakeMove to mutate position state directly (e.g. any bulk board-state copy used by aspiration re-search, singular extension, or the `MP=1` split-search machinery -- see the recent `SearchOneMove`/`SPLIT_INFO` refactor). Any such path would desync the accumulator silently and is exactly the kind of bug that would not show up until much later. - **Performance follow-up, not required for correctness**: eager accumulator maintenance on every MakeMove wastes work at nodes that get pruned by a cheap lazy filter before ever consulting the net (see the super-lazy-prefilter experiment above). A lazy/deferred accumulator scheme (record "what moved" cheaply per ply, only replay deltas forward from the nearest materialized ancestor when a node actually needs a net evaluation) is the standard answer if profiling shows this matters. Defer until there's a working eager version to measure against. ## Breaking external dependencies on Eval()'s internal signals `cTrapped`/`cEnprise` (and any other signal currently read outside `Eval()` by move ordering -- hung-piece-escape logic, `FLEE_BONUS`, the countermove-table evidence gating from the 2026-08-29 move-ordering session) must be decoupled from `Eval()`'s computation entirely before `Eval()` can be replaced, since NNUE has no equivalent intermediate signals to expose. - **Enumeration done, 2026-10-15** (grepped `cTrapped`/`pTrapped`/ `cEnprise`/`pEnprise` across every live `.c`/`.h` file -- `save/` and `changes/` hold old snapshots, not live code, and were excluded; then verified consumers by call site, not just by grep match, since one memorable-sounding candidate turned out to be dead): - **`FLEE_BONUS` (the 2026-08-29 session's leftover-move en-prise nudge, `generate.c:4673`'s doc comment) is dead code, not a live consumer** -- `chess.h:2949` says so explicitly ("separately dead code in movesup.c's SelectBestWithHistory (commented out) -- retired after the GOOD_MOVE-tier version it replaced was shown... to perform at the plain-leftover baseline"); confirmed in `movesup.c`, only a commented-out declaration remains (`movesup.c:1439`). Likewise, `search.c:184-194`'s null-move- calibration-harness comment confirms en-prise/trapped state *was* tried as an axis of the joint fail-high prediction table in an earlier (2026-09-24) revision, found "not a safety signal at all" for that purpose, and explicitly is **not** bucketed in the current revision. Neither of these is a real dependency today. - **Consumers that ARE live** (read-only, via `dynamic.c`'s accessor functions -- none need to change regardless of who produces the data): 1. `generate.c:5342` (QSearch-specific move ordering, a different function from the dead `FLEE_BONUS` one): nudges a move whose origin square is the cached `cEnprise` hint, applied only on top of a move SEE has already classified as winning/even -- SEE runs first, the hint is a secondary nudge. 2. **`EstimateHangingMaterialCp`** (`searchsup.c:246-413`, called from `searchsup.c:1826`/`1905`) -- feeds the "dumb move" axis of the live joint fail-high prediction table (the LMR/reduction soundness machinery; see CLAUDE.md's "LMR sd11 vs sn10M protocol" and search.c:242's "live, in-progress collection run as of this writing"). Structured as a cascade of checks, most of them bitboard-based, falling through to a `cTrapped` half-credit hint only as the last resort (`searchsup.c:403`) -- see the dedicated finding below, since 3 of its 5 checks turned out to share `_EvalTrappedPieces`'s dependency, not just the final fallback. 3. `ValueOfMaterialInTroubleDespiteMove` (`dynamic.c:619`, called once, from `search.c:2237`) -- a **binary QSearch stand-pat gate** ("material in danger -> don't let this side stand pat"), not an FH%-bucket. Reads `cEnprise` (two-slot) and `cTrapped` (one-slot) together into one `iInTrouble` value, with no bitboard check of its own -- trusts the cached hint directly, validated only by an eval-signature match. - **`EstimateHangingMaterialCp` finding, 2026-10-15**: its 5 checks split 2 independent / 3 Eval()-entangled, and the entanglement is bigger than just the `cTrapped` fallback: 1. Pawn-attacks-destination -- independent (static attack tables + raw pawn bitboard). 2. Enemy-king-adjacency -- independent (static tables + `see.c`'s on-demand `_WhoAttacksSquareBB`, the same on-demand board-geometry lookup SEE itself uses -- not an eval-cached bitboard). 3. Minor-attacks-destination -- **depends on `bbMinorAttacksSnapshot`**, which `search.c:606-609` populates straight from `pos->bbMinorAttacks`, **gated on `fParentAttackTablesAreFresh`** (`search.c:602-603`: true only if the parent node's `Eval()` call was a full eval, not lazy). 4. Rook-attacks-destination -- same dependency, `bbRookAttacks`. 5. The `cTrapped` half-credit fallback (what prompted this check) -- depends on `_EvalTrappedPieces`, same as `cTrapped`'s producer below. **So the real dependency here isn't just the `cTrapped` fallback -- it's `bbMinorAttacks`/`bbRookAttacks` themselves** -- but **this is not a blocker, and not a new condition an eval swap introduces**: `fParentAttackTablesAreFresh` is already false today on every node where the parent's `Eval()` call took a lazy/super-lazy exit (today's normal three-tier behavior, not a hypothetical) -- `EstimateHangingMaterialCp` already falls back to just its 2 independent checks on those nodes, routinely, right now. An NNUE swap just makes that already-live degraded path the permanent one, not a newly-broken one. The actual effect, per the function's own return-value contract (cp-loss-or-zero, consumed only through `HangCpBucket`): losing checks 3-5 means some moves that would have bucketed as "a little dumb" instead bucket as "not dumb" (bucket 0) -- a **precision/recall shift in the joint fail-high prediction table's "dumb move" axis, used for pruning/ reduction/node-type decisions, not a correctness bug.** Whether this matters enough to rebuild is an empirical question, not an a-priori blocker: measure what fraction of real search nodes already hit the degraded (no-fresh-tables) path today, and how much predictive power the joint table actually loses if that fraction goes to 100%, before deciding whether building the independent bitboard pass is worth it versus just accepting the less-pessimistic FH% estimate this already-tolerated degraded mode produces. - **Producers -- two genuinely different cases, not one**, which changes the decoupling cost estimate: 1. **`cEnprise`'s cheap source is already fully decoupled from `Eval()` today**: `searchsup.c:3032`'s null-move-refutation path (`RecordEnprisePieceAtPly`, inside `_VerifyNullMove`/nullmove handling) flags a piece en prise purely from *the move that refuted a null-move search* (`mv.pCaptured`) -- no eval attack-table dependency at all, pure search/move-list logic. This half needs **no change** to survive an eval swap. 2. **`cTrapped`'s only source, and `cEnprise`'s *other* source, are both inside `eval.c`'s `_EvalTrappedPieces`** (`eval.c:6188`), and this one is genuinely entangled with `Eval()`'s own term-by-term computation, not cleanly separable by just moving a function: it reads `pos->bbPawnAttacks`/`bbMinorAttacks`/`bbRookAttacks`/ `bbQueenAttacks`/`bbKingAttacks` (built incrementally by earlier per-piece-type eval passes) and `pos->cTrapped[uColor][]`/`pos->bbTrappedEscapes[uColor][]` (a *nomination* list populated earlier still, during low-mobility-piece detection, specifically deferred there because full attack information isn't available until later passes finish -- see `_EvalBishop`'s "carefully" comment). `_WhoControlsSquareFast` (used both here and at the dead `_EvalLookForDanger`, `#if 0`'d out) also wants those same attack bitboards. None of this exists independently of `Eval()` running its normal term-by-term pass today. - **Revised estimate, 2026-10-15 -- not a blocker.** `cEnprise`'s null-move-refutation source is free (already independent) with no caveats. Everything routed through `_EvalTrappedPieces` -- `cTrapped`, `cEnprise`'s other source, and 3 of `EstimateHangingMaterialCp`'s 5 checks -- degrades gracefully rather than breaking: an eval swap just makes permanent a low-information fallback path (`fParentAttackTablesAreFresh == FALSE`) that already fires routinely today on lazy/super-lazy eval exits. Net effect is a less-pessimistic "dumb move" axis in the joint fail-high prediction table -- a tuning/precision question, not correctness. **Decision: do not build the independent bitboard pass speculatively.** Ship the eval swap first with these signals simply going permanently degraded, then measure with the same joint-table instrumentation already in `search.c` whether the prediction table's actual pruning/reduction behavior suffers enough to be worth recovering. Revisit building a slimmed `bbMinorAttacks`/`bbRookAttacks`-plus-trapped-piece pass only if that measurement says so -- not before. - Preferred approach, revised: **no independent tactical-signals pass is planned up front.** The earlier plan here (build a slimmed bitboard pass before the eval swap, to protect `cTrapped`/`cEnprise` consumers) is deferred per the decision above -- it's optional, data-gated follow-up work, not a precondition for replacing `Eval()`. What *does* still apply: the 2026-08-29 move-ordering session's countermove-table evidence gating is unaffected by any of this (doesn't touch `cTrapped`/`cEnprise` at all); `FLEE_BONUS` itself was already dead code before this investigation started, so there was never anything of its to protect. The null-move- refutation `cEnprise` source needs no changes at all, regardless. ## Feature-indexing parity (Python trainer vs. C inference) The single most likely way to silently wreck this project: the Python training code computes HalfKAv2 feature indices one way, the C inference code computes them a different way (square-numbering off-by-one, different king-bucket formula, different perspective ordering), and the net trains with a perfectly reasonable loss curve but performs randomly in the engine, because the trained weights are being read back out against the wrong feature slots. - Write one canonical feature-indexing spec (square numbering, king-bucket formula, per-perspective feature ordering, HalfKAv2_hm's mirroring rule) **once**, in this document or a linked one, before either the Python encoder or the C encoder is written. - Before any real training run: build a small cross-checker that dumps the feature vector for N known test positions from both the Python encoder and a standalone C test harness and diffs them bit-for-bit. Do this before training, not after a training run produces confusing results. - Confirm the output sign/perspective convention (side-to-move-relative, per the architecture decision above) the same way -- a sign flip only manifests as "plays like it wants to lose," which is a much harder bug to localize after the fact than a direct feature-vector diff. ## Canonical HalfKAv2_hm feature-indexing spec (2026-10-15) Per the "known, documented format" architecture decision above, this is **not a reinvented scheme** -- it is Stockfish's own `HalfKAv2_hm` feature set, pinned to a specific historical implementation (`src/nnue/features/half_ka_v2_hm.{h,cpp}` at git tag `sf_16`, fetched directly from `official-stockfish/Stockfish` 2026-10-15, quoted verbatim below) rather than re-derived from memory or a secondary description. Current Stockfish `master` has moved on to a different, newer feature set entirely (`k32q2`/`pp_3wide`/`full_threats`) -- do not fetch "latest Stockfish" expecting to find `HalfKAv2_hm` there. **Square numbering (Stockfish side, the convention both the Python encoder and this spec's formulas use)**: `Square = file + 8*rank`, 0-indexed, `SQ_A1 = 0` ... `SQ_H8 = 63` (`a1..h1` = 0-7, `a2..h2` = 8-15, ..., `a8..h8` = 56-63). **typhoon's own square numbering is different and must be converted**: typhoon uses a 0x88 mailbox representation (`chess.h`): `A1 = 0x70`, `RANK(c) = 8 - (c >> 4)` (1-indexed rank), `FILE(c) = c & 0x0F` (0-indexed file). The C encoder must convert every typhoon `SQUARE` to the Stockfish-style 0-63 index before applying any formula below: `sq64 = FILE(c) + 8 * (RANK(c) - 1)`. Getting this one line wrong (e.g. an off-by-one on `RANK(c) - 1`, or forgetting typhoon's rank numbering runs opposite to its own nibble ordering) is exactly the class of silent bug this whole spec exists to prevent -- it must be the first thing the cross-checker below verifies, in isolation, before testing any king-bucket/orientation logic that builds on top of it. **Piece-square plane index** (`PieceSquareIndex`, verbatim from `half_ka_v2_hm.h`) -- `SQUARE_NB = 64`: ``` PS_NONE=0, PS_W_PAWN=0, PS_B_PAWN=64, PS_W_KNIGHT=128, PS_B_KNIGHT=192, PS_W_BISHOP=256, PS_B_BISHOP=320, PS_W_ROOK=384, PS_B_ROOK=448, PS_W_QUEEN=512, PS_B_QUEEN=576, PS_KING=640, PS_NB=704 ``` Per-perspective piece->plane lookup (own/enemy colors swap when Perspective=BLACK; **both kings share the single `PS_KING` plane** -- own-king and enemy-king features differ only by which square within that one 64-square block is set, not by which plane): ``` Perspective=WHITE: {pawn:PS_W_PAWN, knight:PS_W_KNIGHT, bishop:PS_W_BISHOP, rook:PS_W_ROOK, queen:PS_W_QUEEN, king:PS_KING} for White(=own) pieces; {pawn:PS_B_PAWN, ..., king:PS_KING} for Black(=enemy) pieces. Perspective=BLACK: own/enemy swapped (own=Black pieces use the PS_W_* constants' *role*, i.e. the W/B labels in this table mean "us/them", not literal color -- see the "convention: W-us, B-them" comment in the source). ``` **King buckets** -- 32 buckets per perspective (not 64; this is where `_hm`'s horizontal mirroring halves the table vs. plain HalfKAv2). `KingBuckets[Perspective][[Square 0..63]]`, each entry `bucket_index * PS_NB` (so it's already a feature offset, not a bare bucket number) -- verbatim from the source, printed one visual board row per line, **rank 1 first** (i.e. row *i* of this literal = Squares `8*i .. 8*i+7`, matching Stockfish's own `rank*8+file` ordering, not a top-down board diagram): ``` KingBuckets[WHITE] (value = bucket*704): rank1: 28 29 30 31 31 30 29 28 rank2: 24 25 26 27 27 26 25 24 rank3: 20 21 22 23 23 22 21 20 rank4: 16 17 18 19 19 18 17 16 rank5: 12 13 14 15 15 14 13 12 rank6: 8 9 10 11 11 10 9 8 rank7: 4 5 6 7 7 6 5 4 rank8: 0 1 2 3 3 2 1 0 KingBuckets[BLACK] (value = bucket*704) -- mirror image of WHITE's table (own king's bucket decreases as it advances up the board *from that perspective*, same intent, opposite physical ranks): rank1: 0 1 2 3 3 2 1 0 rank2: 4 5 6 7 7 6 5 4 rank3: 8 9 10 11 11 10 9 8 rank4: 12 13 14 15 15 14 13 12 rank5: 16 17 18 19 19 18 17 16 rank6: 20 21 22 23 23 22 21 20 rank7: 24 25 26 27 27 26 25 24 rank8: 28 29 30 31 31 30 29 28 ``` Only 4 distinct bucket values per rank (not 8) because of the horizontal mirror: king on file a/h share a bucket, b/g share one, c/f share one, d/e share one -- the actual file-level distinction survives through the *orientation* step below, not the bucket. **Orientation / mirroring** (`OrientTBL`, verbatim) -- gives the XOR mask applied to every piece's square (including the king's own), keyed by the *perspective's own king's actual square*: ``` OrientTBL[WHITE][ksq]: SQ_H1 (=7) if ksq's file is a-d (king on the queenside half) SQ_A1 (=0) if ksq's file is e-h (kingside half) (applies to ksq on ANY rank -- only ksq's FILE selects the branch) OrientTBL[BLACK][ksq]: SQ_H8 (=63) if ksq's file is a-d SQ_A8 (=56) if ksq's file is e-h ``` `7 = 0b111` flips only the file bits (pure horizontal mirror); `56 = 0b111000` flips only the rank bits (vertical flip, i.e. "view from Black's side" with no horizontal mirror); `63 = 0b111111` flips both (Black's view, horizontally mirrored too). Net effect matches the header comment: **the perspective's own king always ends up on the e-h files** after orientation, regardless of which side it started on or which color is to move. **Feature index formula** (`make_index`, verbatim): ``` index(Perspective, s, piece_at_s, ksq) = (s XOR OrientTBL[Perspective][ksq]) + PieceSquareIndex[Perspective][piece_at_s] + KingBuckets[Perspective][ksq] ``` where `s` is every occupied square's Stockfish-style 0-63 index (including `ksq` itself, and including the *other* king's square), and `ksq` is fixed per perspective as that perspective's own king's *actual, unoriented* square (the bucket/orient lookups are keyed by the real square -- only the piece square `s` gets XORed). **Per-perspective ordering / total dimensions**: `Dimensions = SQUARE_NB * PS_NB / 2 = 64 * 704 / 2 = 22528` features per perspective (the `/2` is exactly the `_hm` mirror halving 64 king positions down to 32 buckets). Two independent 22528-wide sparse input vectors are produced per position, one per color's perspective (White's accumulator half and Black's accumulator half), concatenated own-perspective-first per the side-to-move-relative output convention already decided above. Up to 32 features are active at once per perspective (`MaxActiveDimensions`) -- one per piece on the board, including both kings (max 32 pieces total on a legal board). **Not yet decided, deliberately deferred**: whether typhoon's trained net keeps Stockfish's literal `HalfKAv2_hm(Friend)` hash/name convention or adopts its own -- irrelevant until a `.nnue`-compatible serialization format is chosen, which isn't needed for this project's own from-scratch C inference code. The spec above only needs to be byte-for-byte *internally* consistent between typhoon's Python encoder and typhoon's C encoder; it does not need to produce files interchangeable with real Stockfish `.nnue` binaries. **Bit-for-bit Python/C cross-checker built and passing, 2026-10-15** (closes milestone 1): - **C side**: `src/nnuefeatures.c`, a new `dump nnuefeatures` engine command (wired into `command.c`'s existing `dump` dispatcher, declared in `chess.h`, added to `GNUmakefile`'s `OBJS`). Deliberately **not** wired into eval/search/the accumulator -- reference-math-only, operating on the already-trusted `g_RootPosition`/`rgSquare` board state so no new FEN-parsing code was needed. Implements the 0x88->64 conversion (`CoorToSq64`), `KingBucket`, `OrientMask`, and the index formula as small standalone static functions, each matching one piece of the spec above 1:1. Prints `NNUE_FEATURES white ...` / `NNUE_FEATURES black ...` for the current position. - **Python side**: `annotate_pgns/nnue_feature_spec.py`, an independent from-scratch implementation of the identical formulas using `python-chess`'s own native 0-63 square numbering (no 0x88 conversion needed on this side -- that's purely a typhoon-C-side translation concern). - **Driver**: `annotate_pgns/cross_check_features.py` spawns the real engine binary via `--batch --command "setboard ; dump nnuefeatures"`, diffs its output against the Python encoder, index-set for index-set, over 9 hand-picked FENs chosen to catch a perspective/orientation mixup specifically (not just the symmetric starting position, which can't distinguish a White-perspective bug from a Black-perspective bug since it's symmetric under exactly the transform between perspectives): both kings queenside (a-d files), both kingside (e-h), each combination of one king on each half (the cases most likely to expose an orientation/bucket bug), a dense midgame position exercising every piece type, a sparse king-and-pawn endgame (3 active features, near the low end of the range), and a multi-queen position. **All 9 positions pass on both perspectives.** Also re-verified passing against the release binary rebuilt by `precommit_check.sh` (which itself passed both gates), not just a one-off debug build. - Run it again with `python3 cross_check_features.py` from `annotate_pgns/` any time the C or Python side of this formula changes -- cheap (a few seconds), should be treated as a required gate before trusting any future feature-vector output from either side. ## Training data pipeline - **Score target**: sigmoid-squashed WDL-style, per the decision above. Mixed with or validated against actual game outcome where available (standard NNUE bootstrapping practice), with exact mate/EGTB-recognizer positions excluded from the sigmoid-fit loss (see above). - **Move-played-as-successor-inversion technique** (confirmed sound): to get a score for the move actually played in a human game (no guarantee the engine's own search would pick the same move), score the resulting position after the move and negate: `score_before_move_from_mover's_perspective = -eval_or_search(position_after_move)`. This is just the standard negamax relationship -- well-posed regardless of whose move it was, and does not require the engine to agree the move was objectively best. - **Data sources and current state** (checked 2026-10-14): - `../pgn/self_play_games.pgn`: 166,640 total games, but only **26,347 have score annotations** (started "a couple of weeks ago" per conversation) -- roughly 2.36M annotated `{ comment }` positions total. The other ~140k pre-annotation games are also pre-dating a recent change that scrambles opening positions more, so they are likely heavily duplicate/near-duplicate and not worth careful deduplication effort yet. Self-play is ongoing concurrently with this planning, so this pool grows on its own. - `../pgn/lichess_elite_*.pgn` (2013-2020, many files) and `../pgn/twic_filtered.pgn` (642,268 games, >=2400 Elo / >40 ply, 556MB) -- far larger and more diverse than the current self-play pool, not sourced from this engine's own play, so free of the "net just re-learns the current eval's blind spots" risk that bootstrapping directly off self-play data would have. - **Recommended phasing** (standard NNUE bootstrapping order, not novel): 1. **Phase 1 -- bootstrap from human games.** Train the first net primarily on `twic_filtered.pgn`/`lichess_elite_*` positions. Training on the current self-play pool first would mostly teach the net to reproduce the current hand-tuned `Eval()`'s own blind spots, since that's what generated those games -- not useful signal for replacing it. 2. **Phase 2 -- self-play with the new net.** Once a first net exists, use it (not the old `Eval()`) to generate much larger, more targeted self-play volume -- this is where self-play data becomes valuable, since it now reflects the new net's own tactical gaps rather than the old eval's. - **Search depth for annotation: `sd 10`, fixed depth (not `st` time-based), decided 2026-10-14.** Measured on a 300-position sample drawn from `twic_filtered.pgn` (`/tmp/typhoon/nnue_pilot/`): `sd 10` with a 32MB hash runs at roughly 7 positions/sec single-threaded (~6/sec at the default 256MB hash -- smaller hash is faster per-position because every `setboard` triggers a full `ClearHashTable()` memset via `SetBoardCommand`'s `PreGameReset(FALSE)` call, so a smaller table is cheaper to clear). At 16 single-threaded shards (`--cpus 1 --hash 32m --logfile -` each), 5M positions is projected at ~12.4 hours. `--logfile -` was confirmed to already skip `BeginLogging`'s backup-file dance entirely (`main.c`), no code change needed there. The `script` command's periodic checkpoint-report dump (every 5 positions, `script.c`'s `CHECKPOINT_GO_INTERVAL`, kept for crash resilience on long joint-table-collection runs) was measured and found to cost nothing meaningful at this scale (43.9s vs 42.7s/300 positions with it disabled) -- not worth adding a flag for. - Depth-adequacy check: compared `sd 10` vs `sd 16` scores on the same 21 positions (a subsample of the 300; the full 300-position `sd 16` comparison was not completed -- it was running at ~14s/ position, ~100x slower than `sd 10`, confirming `sd 16`+ is not feasible for bulk annotation at the volumes this project needs regardless of any quality difference). Result: mean |delta| 19cp, median 10cp; in win-probability terms (what actually matters for WDL-sigmoid training labels) mean |delta| was ~2.3%, only 1 of 21 exceeded a 10% win-probability swing, and that one outlier was already a clearly-winning position at both depths (didn't change position *character*). Two apparent "sign flips" were both near-equal/balanced positions where a small score wobble flips which side nominally leads -- noise at the margin, not a missed tactic. Small sample (n=21), not weighted toward tactically sharp positions the way ECM is, but consistent with the expectation that ordinary human-game middlegame positions (unlike curated tactical suites) are not especially depth-sensitive for the net's purpose -- the net's job is positional judgment on these; finding forced tactics is the search's job at inference time, not the training-label-generation search's job to get perfectly right. Conclusion: proceed with `sd 10` for the bootstrap annotation pass. - **Annotation harness**: build a TWIC/lichess annotation tool reusing `eval_tune/tune_eval_dna.py`'s `Engine`/subprocess pattern (xboard protocol driver, persistent-context `eval` command, SysV-semaphore pitfalls already solved there -- see that file's existing `Engine`/`EnginePool` context-manager requirement). This is buildable and runnable immediately, independent of all engine-internals work below. - **Quiet-position filtering, applied at training-set assembly time (not at annotation time)**: exclude positions where the side to move is in check, or where the best/played move is an SEE-positive capture (signals the position is mid-exchange, not settled). Two reasons: (1) a position dominated by a forced tactical resolution tests "find the tactic," which is quiescence search's job, not the net's positional judgment -- training heavily on these wastes net capacity re-deriving something SEE already gives for free; (2) a forcing capture sequence sampled at plies N, N+1, N+2 produces several highly correlated near-duplicate samples of one tactical event, not independent data points, inflating apparent corpus diversity without adding real signal. Keep annotating every ply regardless (the raw PV/score corpus has standalone value for analysis); apply this filter only when building the actual training set from that corpus. Note this does not create a train/inference mismatch -- the net still gets invoked at many non-quiet-looking nodes during `QSearch()` at inference time, but quiescence search is specifically designed to walk down to a settled, stand-pat-ish leaf before a position's evaluation is actually trusted, so the distribution of positions where the score is relied upon is itself mostly quiet-ish by construction. - **Held-out test set: resolved, already built.** `split_holdout.py`'s stratified game-level split (result x length-quartile, 10% sampled independently per stratum -- see Status below) is the holdout pool: **26,995 train games / 2,298,648 positions** vs. **3,005 holdout games / 255,245 positions**, split by *game* (not position, avoiding cross-split leakage from one game's correlated positions) and never touched when fitting the WDL-sigmoid K constant. This is a separate pool from `eval_tune/tuned.best_holdout.dna` (that one is DNA-tuning's own flat, position-level holdout, built for a different pipeline) -- the two are not reused across projects; NNUE training/ validation/gating all draw from this `annotate_pgns/` pool instead. ## Status (2026-10-14) - **Sharded annotation harness built**: `../annotate_pgns/annotate.py`. Shards an input PGN across N single-threaded engine instances, scores every position in every game at a fixed `sd ` (depth is a plain CLI arg, so re-running at `sd 12`/`sd 16` later needs no code changes, just a different `--depth`/`--outdir`), and reassembles an annotated PGN matching the self-play comment format (`{ +score/depth PV }`). Validated end-to-end on an 8-game/4-shard smoke test before trusting it at scale. `load_games()` skips `Result "*"` (unterminated) games -- can't derive a WDL label from an unknown outcome. Short agreed-draw games are not separately filtered: `twic_filtered.pgn`'s existing `>40 ply` / `>=2400 Elo` filter (`eval_tune/filter_pgn.py`) already excludes those by construction, confirmed empirically (a 5000-game sample has a hard floor at 40-49 plies, nothing shorter). - **Stage A (pipeline-validation) run in progress.** Target: ~2M annotated positions (~30,000 games from `twic_filtered.pgn`, the first 30,000 in file order) at `sd 10`, 16 single-threaded shards (`--cpus 1 --hash 32m --logfile -` each). Launched 2026-10-14 ~15:05. - Output location: `../annotate_pgns/runs/sd10_twic_2M/annotated.pgn` (written once all 16 shards finish and the driver reassembles them). Intermediate per-shard files kept in the same directory: `shard_NN.script` (generated engine command scripts), `shard_NN.out` (raw engine stdout per shard). Driver progress log: `../annotate_pgns/runs/sd10_twic_2M.driver.log`. - Actual yield came in higher than the small smoke-test estimate: 30,000 games produced 2,553,893 total positions to score (~85/game average, vs. the ~67/game estimated from the 8-game smoke sample) -- expect final usable count after quiet-filtering (applied later, at training-set-assembly time, not during annotation) to land somewhat under 2,553,893, comfortably still in/above the ~2M target. - Measured throughput came in slower than the single-shard pilot extrapolation: ~75 positions/sec aggregate across all 16 shards (vs. ~112/sec extrapolated from the single-shard pilot's 7/sec x 16), most likely from 16-way contention on this shared box (cache/memory bandwidth) rather than any bug -- progress checks showed clean, steady, monotonic counts with no errors/aborts across all 16 engine processes. - As of 2026-10-14 16:47: 459,156 / 2,553,893 positions done (18.0%). Revised ETA: ~7.7 more hours from that point (~9.4 hours total from the 15:05 launch), finishing roughly 00:30 rather than the original ~5-hour back-of-envelope guess. - **Stage A run finished 2026-10-14 ~00:40** (~9.6 hours, close to the revised ETA): 30,000 games, 2,553,893 positions scored across all 16 shards with no crashes/aborts. - **Bug found and fixed in `annotate.py`'s output parser, discovered only after the full run completed (would have been cheap to catch in the smoke test -- lesson for next time: smoke-test specifically needs a position or two that hits a forced mate/EGTB hit, not just ordinary middlegame positions).** `TELLOTHERS_RE` only matched plain-float scores; the engine's own `+MATE`/`-MATE` notation (and ``/ `` PV-terminal sentinel moves, which can carry an ordinary- looking float score when real moves precede the sentinel, e.g. `PV=Kf6 Bc3+ Kxg6 Kf8 Kxh5 `) didn't match, so those result lines were silently dropped from the parsed list while staying in the position index. Since `annotate_shard` zips the two lists *positionally*, one dropped line didn't just lose its own annotation -- it desynced every position after it in that shard, attaching the wrong score/PV to the wrong FEN for the rest of the shard's output. All 16 shards hit this (975-1,265 bad lines each, 16,994 total, ~0.67% of all scored positions), confirmed via a full scan of every `tellothers` line across all 16 `shard_NN.out` files (2,553,893 total lines, 0 that failed to match the outer line format at all -- the only two non-numeric score tokens that ever appear are `+MATE` (9,986 occurrences) and `-MATE` (7,008), confirming there's no other silent-drop source hiding in this run's data). - **Recovered without re-running the search.** `annotate_pgns/extract_positions.py` mines `shard_NN.script` (the ordered list of `setboard ` lines already written for that shard) zipped positionally against `shard_NN.out`'s `tellothers` results -- the engine processes `go` in strict script order with no interleaving (see CLAUDE.md), and `setboard`-count equals `tellothers`-count exactly once mate/EGTB lines are captured as real results instead of dropped, so this recovers a clean, correctly-aligned dataset straight from the existing raw logs. PGN reconstruction (what `annotate.py` was doing) turned out to be an unnecessary detour for a project that only ever needed (FEN, score) pairs, not game trees -- it's also what introduced the misalignment risk in the first place. Output: `runs/sd10_twic_2M/positions.jsonl`, one JSON object per position (`fen`, `depth`, `score`, `score_raw`, `pv`, `exact`, `exact_reason`). Exactness is detected from the PV's *last move token* being ``/`` (not the whole-PV string or the score field -- a real line of moves can precede the sentinel, and the score alongside it can look like an ordinary centipawn value even though the position is game-theoretically exact) or the score being `+-MATE`. Result: 2,553,893 positions total, 48,407 flagged `exact` (zero `` occurrences this run, all exact rows were mate scores or ``-terminated PVs), **2,505,486 usable rows** for sigmoid-fit training -- matches the original run's full target yield. `annotate.py` itself still has the original bug (not yet patched -- low priority now that `positions.jsonl`, not its PGN output, is the actual training artifact; would still need fixing before trusting any of its PGN output for other purposes). - **Related engine-side bug found and fixed while investigating the `+315.92`-style non-exact-looking EGTB scores above**: `probe.c`'s `EGTB_WIN_SCORE`/`EGTB_LOSS_SCORE` used to be `NMATE - VALUE_QUEEN` (deliberately kept below the `NMATE` threshold so it wouldn't be mistaken for a literal forced mate elsewhere in the engine), which is why EGTB hits printed as ordinary-looking floats instead of `+-MATE`. Changed to literal `+-NMATE` so `FormatScore` (`util.c`) renders every EGTB win/loss as a flat `+-MATE100` uniformly, matching the already-observed bare-`` case and making all EGTB-terminated PVs parseable by the same plain `+-MATE\d+` regex going forward -- no separate PV-suffix check will be needed for *future* runs (this run still needs it, since it predates the fix). Caveat checked before trusting this: `ADJUST_MATE_IN_N` (the `#ifdef` in `hash.c`'s `StoreExactScore` that would ply-adjust a score assuming it encodes mate distance -- which a flat EGTB sentinel does not) is not defined anywhere in this build (confirmed via grep across all headers and `GNUmakefile`), so the hash-store path this change now exercises for every EGTB hit is the safe `#else` branch (downgrades to a `HASH_FLAG_LOWER`/`HASH_FLAG_UPPER` bound entry with a flat `+-NMATE` value, no ply arithmetic) -- confirmed this exercises a **second, previously-dormant bug** in that same `#else` branch: the `iValue <= -NMATE` case was pairing `HASH_FLAG_UPPER` with `sNew.iValue = +NMATE` (should be `-NMATE` -- compare the mirror-image win-case branch just above it, which correctly pairs `HASH_FLAG_LOWER` with `+NMATE`). Previously dormant because the old `NMATE - VALUE_QUEEN` EGTB-loss score never satisfied `iValue <= -NMATE` on its own (only a literal engine-found forced mate-against could reach this branch); the `EGTB_LOSS_SCORE = -NMATE` change makes every EGTB-loss position hit it. In a release build this was harmless (a `+NMATE` upper bound is vacuously loose, never produces an incorrect cutoff, just wastes a hash write); in a `DEBUG` build it would trip `search.c`'s `VERIFY_HASH_HIT` macro's `ASSERT(iScore < +NMATE)` on later retrieval under the `HASH_FLAG_UPPER` case. Fixed (`hash.c`: `sNew.iValue = -NMATE`). - **Four more real engine bugs found and fixed in the same session**, all via `precommit_check.sh`-verified fixes, committed together as `ef2c2f6` ("Fix several latent search/move-generation bugs found via NNUE annotation tooling"): - **`san.c`**: a checking castle move (`O-O+`/`O-O-O+`) printed as `+-O`/`+-O-O` -- the check-suffix pointer (`p = szTextMove;`) was captured *before* `strcpy` filled the buffer with `"O-O"`, so `*p++ = '+'` overwrote the leading `O` instead of appending after it. Fixed by recomputing `p` from `strlen(szTextMove)` after the `strcpy`. - **`search.c`/`searchsup.c`**: all five PV sentinel-recording sites (`MATEMOVE`, `DRAWMOVE` x2 -- stalemate and the separate repetition/50-move site, `RECOGNMOVE`/`EGTBMOVE` x2 -- main search and QSearch) were gated by the same strict `(iAlpha < iScore) && (iScore < iBeta)` window check used for ordinary heuristic PV moves. Traced live (temporary `Trace()` instrumentation on a real mate-in-2 from the corpus, removed after): once a forced line's value is already known (from an earlier, shallower iterative-deepening pass, or the hash table), later visits to it are typically probed with a PVS scout/null window built *around* that known value, so the terminal node's real score *ties* the window rather than strictly beating it -- failing this gate by equality, not by being out of range. But these five values are exact/game-theoretic facts, not heuristic bounds -- unlike an ordinary move, they can't be "wrong" for having been reached outside the window. Fixed by recording them unconditionally. This alone did not fully fix `<#>`/`<=>` display, though -- see the next bullet. - **`util.c` (`WalkPV`)**: traced further and found the *actual final move* of a forced mate (one ply shallower than the terminal sentinel) hits the identical tie, through the completely separate *ordinary* move-acceptance code path (`iScore > iAlpha` at `search.c`'s main move loop) -- so the recorded PV is often missing its own concluding move, not just the sentinel after it. Rather than touch that hot, per-node path (every interior node of the entire search tree), `WalkPV` now reconstructs the missing tail directly from the board when the recorded PV runs out one ply short of (or exactly at) a forced mate/stalemate/draw: the position at that point already reflects every printed move (each was `MakeMove`'d during the walk), so it checks `IsDraw`, or failing that, finds the one legal move the shrinking mate distance guarantees must exist. Runs once per improved root score, not in the search tree, so the extra legal-move verification is cheap. Verified on mate-in-1 (`Rd8+ <#>`), a real mate-in-6 (13-move chain, no gaps anywhere in the middle, not just at the end), and a constructed 50-move-rule draw (`Kc1 <=>`). - **`gamelist.c`/`board.c`/`testsup.c`**: `SetRootPosition` used to parse a `setboard` FEN straight into the global `g_RootPosition`, so a *rejected* FEN (malformed syntax, or -- newly caught -- describing an illegal position) still left partially-parsed garbage in `g_RootPosition`, because `GetRootPosition()` hands out a stable pointer straight to it with no defensive copy (removing one was a deliberate, confirmed-correct fix for a *different*, unrelated bug -- see that function's own comment). The next `go` searched that garbage regardless of the earlier rejection, discovered independently two ways: an `InvalidGenerator` dispatch panic (`generate.c`) and, separately, a debug-build assert in `_ScoreAllMoves`'s MVV/LVA scoring (`PIECE_VALUE(KING)` overflowing past `VALUE_KING` -- a pseudo-legal "capture the enemy king" move, generated because nothing stops the generator from producing one against an opponent king already left in check). Root cause of *that*: a hand-built FEN leaving the side not on move in check is illegal (unreachable through real play), and nothing validated for it. Fixed: `SetRootPosition` now parses into a local staging `POSITION` and only commits to `g_RootPosition` if `FenToPosition` succeeds *and* `IsBoardLegal` (pawns never on rank 1/8, side not on move never in check) *and* `VerifyPositionConsistency` (internal struct consistency -- piece lists, material counts, signatures) both pass. Both checks now run unconditionally, in every build profile, not just `TEST`/`DEBUG` -- `IsBoardLegal` had to be moved from `testsup.c` (only linked under `TEST`/`EVAL_DUMP`) to `board.c` (always linked) to make this possible; `VerifyPositionConsistency` is DEBUG-gated at its many *other* (hot, per-move) call sites for performance, but this one is cold (once per game/`setboard`, not per node), so the cost is worth it here specifically. **Important, deliberately NOT a bug**: a separate illegal-FEN repro that looked like a second instance of the same class (`8/8/8/4kp2/5P2/4K3/8/8 w - - 99 60`, locked pawn pair adjacent to both kings) turned out to just be a badly hand-built test FEN of my own -- White's own pawn on f4 already attacked the square Black's king sat on while it was White's move, which is the same "opponent in check" illegality, not a new generator bug. Confirmed by reverting all of this session's changes (`git stash`) and reproducing the identical crash on unmodified `master` -- notable only as a reminder that a hand-built FEN needs to be checked for basic legality before trusting a crash it causes. - **`extract_positions.py` exactness-tagging gap found and fixed** (see the earlier bullet on the ``/`` MATE-regex bug): it also missed `<=>` (`DRAWMOVE` -- a claimed repetition/50-move draw) and the `"only"` sentinel (`root.c`'s single-legal-move-at-the-root shortcut, which skips searching entirely and hardcodes `score=0`/`depth=0` -- not a real evaluation, not even indeterminate, just never searched). A draw-by-repetition/50-move claim is a fact about the *game's move history*, not the position -- the same position reached a different way might not be a draw at all -- so it's excluded from training for the same reason ``/`` are. Re-running extraction with both fixed: exact-row count went from 48,407 to **93,415** (47,304 of the increase is `<=>`), leaving 2,460,478 usable rows (down from 2,505,486). - **Quiet filter built and run on the full corpus** (`quiet_filter.py`, reusing `eval_tune/tune_eval_dna.py`'s pre-existing `static_exchange_eval`/`has_hanging_material` rather than reimplementing): excludes in-check positions and positions whose PV's first move is an SEE-positive (>=50cp) capture. Full-corpus result: 2,553,893 total rows -> 48,407 exact -> 112,405 excluded (in check) -> 311,949 excluded (SEE-positive first move) -> **2,080,911 quiet positions** (`runs/sd10_twic_2M/positions.quiet.jsonl`), ~81.5% retention. - **Shared quiet-predicate module factored out** (`quiet_common.py`): `position_is_quiet(board, move)` plus the SEE-threshold constant, used by all three consumers below so the exclusion logic (in-check, SEE >=50cp capture) lives in exactly one place. The two call sites differ only in *which move* gets SEE-checked, and this is a real, tracked distinction, not an implementation detail -- the engine's own PV first move (the two post-hoc filters, below) vs. the move actually played in the source game (the new pre-filter, below) are two different training distributions; rows produced under the two should not be silently mixed into one training run. - **Quiet filter now also applied to the game-shaped labeled files** (`quiet_filter_games.py`), closing the gap this file previously flagged as open -- re-applies the identical PV-based predicate directly to each position already embedded in a game record, rather than joining against `positions.quiet.jsonl` (which carries no row/game identity, and a repeated FEN across different games/plies would make a content-based join silently misattribute rows). A game that loses every position to filtering is dropped entirely (none were, in this corpus). Run on both labeled splits 2026-10-15: **holdout**: 255,245 -> **215,134** positions (9,476 in-check + 30,628 SEE-positive-capture + 7 indeterminate excluded), 3,005/3,005 games kept -- `games.holdout.labeled.quiet.jsonl`. **train**: 2,298,648 -> **1,932,713** positions (88,278 in-check + 277,593 SEE-positive-capture + 64 indeterminate excluded), 26,995/26,995 games kept -- `games.train.labeled.quiet.jsonl`. Combined retention across both splits is 2,147,847 / 2,553,893 = 84.1%, noticeably higher than the flat-file filter's 81.5% -- **not noise, a real behavioral difference**: `quiet_filter.py` (flat) drops exact rows (mate/EGTB/`<=>`) from its output entirely, while `quiet_filter_games.py` always keeps them (only the in-check/SEE criterion applies to non-exact rows; exact rows are real, game-theoretic ground truth, not something the quiet filter has any basis to exclude -- NNUE.md already excludes them from the sigmoid-fit *loss* elsewhere, which is a separate, correct exclusion from excluding them from the dataset altogether). The game-shaped filter's behavior is the one intended going forward; the flat file's stricter exact-row drop was never a deliberate design choice, just an artifact of how `quiet_filter.py` was first written. - **Quiet pre-filtering moved upstream into `annotate.py`** (`--prefilter-quiet` flag; `quiet_common.position_is_quiet` applied inside `positions_for_game` before a position is ever written into a shard script), so a *future* annotation run never pays `sd N` search cost on a position that would only be discarded afterward -- making the post-hoc game-shaped filter above unnecessary for runs going forward. **Deliberate semantic difference from the post-hoc filters**: no PV exists before the search has run, so the pre-filter SEE-checks the move *actually played* in the source game instead of a PV's first move (see `quiet_common.py`'s module docstring). `build_games.py` gained a matching `--prefilter-quiet` flag that **must match** whatever the originating `annotate.py` run used -- it recomputes per-game ply counts via the same `positions_for_game` call to re-slice `positions.jsonl`, and a mismatched flag would silently misalign every position after the first game that differs. Verified end-to-end on a 4-game/2-shard/`sd 6` smoke test: pre-filtering wrote 253 of the games' total plies into the shard scripts, all 253 came back annotated with no count-mismatch warnings, and `build_games.py --prefilter-quiet` reproduced the identical per-game slice boundaries. The already-completed Stage A run below predates this flag and was not pre-filtered -- this only changes the *next* annotation run. - **Python SEE cross-checked against the engine's real `SEE()`** (`check_see_against_engine.py`, using the bulk `seescores` command): 2,000 sampled positions, 4,032 actual capture comparisons -- 31.0% exact match, 92.7% sign match, 7.3% sign mismatch. Investigated one mismatch in depth (`Bxa6` in a real position, engine said +100, Python said -230) via temporary `Trace()` instrumentation in `_SEEFull`/`SEE()`: **not an engine bug** -- the engine correctly finds that after `Bxa6 Rxa6`, White's queen, freshly revealed by the bishop's own departure (x-ray down the same diagonal), recaptures the rook, so Black's actual best reply is to decline the recapture entirely (net -100, not -230). Python's `static_exchange_eval` computes all attacker lists once, off the pre-move board, so it never sees a piece the mover's own departure uncovers -- exactly the "ignores... discoveries" limitation its own docstring already confessed. The 7.3% mismatch rate is overwhelmingly (likely entirely) this same known, accepted limitation, not a new one -- confirms the cross-check tool did its job, not that anything is broken. - **Game-container reconstruction built** (`build_games.py`): recovers per-game grouping that `positions.jsonl` lost (flat list, no `game_id`) by re-loading the same PGN with the same `max_games`/`--shards` (pure parsing, no search, deterministic) and slicing the already-extracted rows back into per-game chunks via each game's known position count. Output shape -- `{game_id, result, positions: [...]}`, one line per game -- deliberately matches the shape a self-play PGN with per-move score/PV comments already has (self-play annotation is a separate, ongoing effort outside this project, run every time the engine plays a self-play match), so a future loader for that source can emit the same schema and both mix freely at training-set-assembly time. Verified exact: 30,000 games, 2,553,893 positions, matching `positions.jsonl`'s count precisely. Output: `runs/sd10_twic_2M/games.jsonl`. - **Stratified game-level train/holdout split built and run** (`split_holdout.py`): splits by *game*, not position (every position from one game lands on the same side -- avoids the leakage a flat position-level shuffle has, which `eval_tune/tune_eval_dna.py`'s own holdout split still does). Stratified on two axes so a 10% sample's composition mirrors the full pool rather than risking (by chance) too many decisive games, too few draws, or a skew toward short/long games: result (White win/Black win/Draw) x game-length quartile (quartile cutpoints derived from the actual corpus: 63/80/102 annotated plies, not guessed). 10% sampled independently within each of the 12 resulting strata. Result: **26,995 train games / 2,298,648 positions** (`games.train.jsonl`), **3,005 holdout games / 255,245 positions** (`games.holdout.jsonl`). - **WDL-sigmoid target conversion built and run** (`fit_k_and_label.py`): per NNUE.md's existing decision, reuses `tune_eval_dna.py`'s `fit_k`/ternary-search-over-K approach, but K is fit directly against the already-computed `sd 10` search scores (no engine round-trips needed, unlike the original `fit_k`, which called live `Eval()`), and **only on the train split** -- fitting K on holdout data would leak information into what's supposed to be an untouched generalization check; the same fitted K is then applied (never refit) to label holdout positions too. **Caveat hit and fixed**: the first attempt copied `tune_eval_dna.py`'s own search bracket (`[100, 2000]`, calibrated for *centipawn*-scale `Eval()` scores) verbatim and got `K=10.00` -- exactly its lower bound, a clear sign of a clamped, not-actually-converged fit, not a real answer. Root cause: this project's own scores are stored in *pawn* units (e.g. `+0.47`, not `47`), a 100x unit mismatch. Refit with an appropriately-scaled bracket (`[0.05, 20]`) landed well inside the range: **K = 1.3123** (pawns), MSE improved from 0.1401 (clamped) to 0.1175 (real optimum). Sanity-checks as plausible for >=2400-level play: a 1-pawn edge -> `sigmoid(1/1.3123) ~= 0.68` win probability, a 2-pawn edge -> `~= 0.82` -- modest material deficits are often held or converted at this level, not auto-losses. Deliberately does **not** blend `p_search` (the sigmoid-derived estimate) with `r_outcome` (that position's own game's actual final result, side-to-move-relative) into a single target yet -- both are stored separately per position so the blend weight (lambda in `target = lambda*p_search + (1-lambda)*r_outcome`) remains a free parameter to sweep later without re-deriving anything from scratch. Verified correct on spot-checked positions (result flips correctly to each position's own side-to-move perspective). **Lambda decided for Phase A: lambda=1.0 (pure `p_search`, no `r_outcome` blend), 2026-10-15.** Phase A's goal is pipeline/ framework validation on a small local-CPU training run, not the best achievable net -- a free parameter with no data yet to tune it against would just be guessed, so defer the sweep to Phase B/the larger GPU-trained run, where there's enough data and enough training iterations for a lambda sweep to produce a real signal rather than noise. `r_outcome` stays stored per position regardless (already computed, free to use later) so the sweep needs no re-derivation. Output: `runs/sd10_twic_2M/games.train.labeled.jsonl` (26,995 games, 2,298,648 positions, 84,302 exact rows left unlabeled), `games.holdout.labeled.jsonl` (3,005 games, 255,245 positions, 9,113 exact rows left unlabeled). - **Update 2026-10-15: all of the below is now done** (see "Status (2026-10-15)" further down for the full writeup) -- the training script, feature-indexing cross-checker, accumulator plumbing, and the first trained/exported/wired-in/match-played net all exist now. **Still not done**: the lambda sweep/selection for blending `p_search`/`r_outcome` (still correctly deferred to Phase B, per the lambda decision above -- no new information has arrived that would change that), and quantization (see "Quantization plan"). The quiet-filter gap previously noted here (not yet applied to the game-shaped/labeled files) is closed -- see `quiet_filter_games.py` above -- and a future annotation run can skip the post-hoc step entirely via `annotate.py --prefilter-quiet`. ## Status (2026-10-15): training run finished, net wired into real search, first match-play result - **Phase A training run finished.** `annotate_pgns/nnue_train.py` against the ~2.08M-row quiet-filtered pool above, 20 epochs, PyTorch on CPU (per the Phase A local-CPU decision below) -- final train MSE 0.0065, holdout MSE ~0.022. Checkpoints: `runs/sd10_twic_2M/nnue_ckpt/ ckpt_epoch000.pt` through `ckpt_epoch019.pt`. - **Feature-indexing cross-checker built and passing** (`cross_check_features.py` against `nnue_feature_spec.py`'s Python reference encoder and `src/nnuefeatures.c`'s C reference dump, 9 FENs, all matching) -- closes milestone 1 below. - **Weight export pass built** (`export_nnue_weights.py`): flat binary format, 4-byte `"NNU1"` magic, `<4I` dims header (`NNUE_DIMENSIONS`/`L1`/`L2`/`L3`), then the eight weight/bias arrays in a fixed order. `ckpt_epoch019.pt` exported to `runs/sd10_twic_2M/nnue_ckpt/weights.nnue` (23MB). - **Accumulator plumbing built, tested, and proven fast** (on the `nnue-accumulator-plumbing` branch, per milestone 3's branch requirement) -- `SEARCHER_THREAD_CONTEXT` gained `sNnueAccum[2]`; all 8 of `move.c`'s Lift/Place/Slide(WithoutSigs) primitives maintain it incrementally (chose primitive-hooking over re-deriving move-kind from `mv` after discussion -- these primitives already couple tightly with Zobrist hashing, and `GenerateMoves`/ `MakeMove`/`UnmakeMove` are too hot to afford re-branching); king moves trigger a full `NnueAccumRecompute` for their *own* perspective only (every feature in that perspective is own-king-relative), with a `PLY_INFO`-indexed backup/restore for `UnmakeMove`. `testnnue.c`'s `TestNnueAccumulator` plays real games from 7 curated FENs chosen to hit captures/promotions/en passant/castling/plain king moves within 2 plies, comparing incremental maintenance against a from-scratch `NnueAccumRecompute` after every move -- caught two real bugs on its first-ever runs (a king-capture backup-ordering bug where the backup was taken after the capturing `LiftPiece` instead of before it; and, when the perf-gating fix below landed, the test's own failure to set `g_fUseNnueEval` before exercising the gated code). `setboard`'s downstream effect is covered by the same `ReInitializeSearcherContext` choke point every `go`/`ponder` already calls -- confirmed this is the one place that needs a hook, not `SetRootPosition` itself. - **Load/call infrastructure and real search integration built**: `NnueLoadWeights` (`nnueeval.c`) loads an exported `.nnue` file, dimension-checked against the build's own constants; `NnueForward` runs the real FC head (ReLU(accumulator) -> own-perspective-first concat by side-to-move -> dense 2L1->L2->L3->1). `Eval()` now short-circuits to `NnueForward` when `g_fUseNnueEval && g_fNnueWeightsLoaded`. Reachable via the `nnue load|eval|on|off` command and a `--nnueweights ` CLI flag (the latter needed because `match_play.py` spawns engines with fixed args -- no chance to send interactive commands first). - **Critical performance regression found and fixed.** The accumulator maintenance added to move.c's primitives ran unconditionally on the (unverified, wrong) assumption that it was cheap -- cost ~50% of engine throughput even with NNUE off (673k nps measured vs. the expected 1.4-1.5M baseline). Gated all of it behind `g_fUseNnueEval`. Confirmed via `bench`: classic HCE back to ~1.52M nps, NNUE eval a fair ~410-411k nps. - **`test_vs_head.sh` extended with `NNUE_WEIGHTS`**: wraps the candidate binary in a thin exec shim that always injects `--nnueweights $NNUE_WEIGHTS`, since `match_play.py`'s `Engine` class passes identical `engine_args` to both sides and spawns the given path directly via `Popen` (no shell), so a shebang wrapper script in the candidate's place works unmodified -- no `match_play.py` patch needed. - **First real match-play result, stopped by hand at 70 games**: `-45 =0 +25` (score 0.357), llr trending toward H0 but nowhere near resolved (far too few games for this SPRT's bounds). At `--st 4`, the NNUE candidate searched ~3.4-3.8 plies shallower than the HCE head (avg depth ~10.1-10.5 vs ~13.7-13.9) and at roughly 4.6-4.8x fewer nps (~285-293k vs ~1.32-1.41M) -- yet still won 25 of 70 games and held a stable ~0.30-0.36 score across the run rather than eroding toward 0 as the sample grew, and ran a *tighter* tree despite running under HCE-tuned pruning margins (cumulative ebf ~3.8-3.97 vs. head's ~4.59-5.01). Read as: the net's per-position judgment looks genuinely promising, but is not yet being given a fair fight -- two known confounds stack against it simultaneously (raw search-depth handicap from the speed gap, and futility/ delta-pruning/aspiration-window margins all tuned against HCE's score distribution, not NNUE's). **Zero draws is not a red flag** -- expected signature of a large strength/depth mismatch (the weaker side tends to lose material in sharp lines well before either side would reach a balanced technical ending), not evidence anything is broken. - **Notably, draws were exactly zero** across all 70 games -- see above, this is explained by the depth/strength gap, not a bug. - **Quantization investigation started** (not yet implemented -- see "Quantization plan" below for the concrete next steps): - AVX2 build support added (`AVX2=1` GNUmakefile option, scoped opt-in like `GPROF=1`, not folded into the default/`-mtune=native` set -- same portability reasoning as the existing `-march=native` rejection). Confirmed clang's `-O3 -ffast-math` already autovectorizes both `NnueForward` and `NnueAccumAddFeature`/ `RemoveFeature` to SSE (4-wide) without this flag; `AVX2=1` widens them to real 256-bit `ymm` ops (verified via `objdump` on the linked binary, not the `.o` -- the `.o` files are LTO bitcode, `objdump -d *.o` shows nothing meaningful). - **AVX2 is a ~3% win for classic HCE but a measured no-op for NNUE** (`bench`: 410,380 -> 411,402 nps, within noise). This is a real, useful negative result: it confirms the NNUE bottleneck is memory-bound (cache misses gathering rows out of the ~23MB `g_rgfNnueWeight` embedding table on every accumulator add/remove -- the per-*move* hot path, far more frequent than the once-per-evaluated-leaf FC head), not FP throughput -- wider SIMD can't help a stall waiting on a cache-line fetch. - **Quantization safety-checked against the real checkpoint before committing to any scale constants**, since this net trains with plain (unclipped) ReLU, not Stockfish's clipped ReLU that makes its own int16/int8 scheme near-lossless by construction. Loaded `ckpt_epoch019.pt` directly: `ft.weight` max|.|=5.01, std~=1.00; FC weights all max|.| < 0.36. Sampled 2000 trials of 32 active features (a realistic full-board HalfKAv2_hm count) summed through `ft.weight` + bias: worst-case accumulator magnitude ~26. **Conclusion: safe to quantize this checkpoint directly, no retrain needed** -- even a generous fixed-point scale (256-512) leaves int16 (+-32767) comfortable headroom against the realistic worst case (~6,600-13,300), nowhere near saturation. ## Quantization plan (not yet implemented) Priority order, chosen from the AVX2 experiment above: the embedding table (per-*move* cost, memory-bound) matters more than the FC head (per-evaluated-*leaf* cost, compute-bound and already partly vectorized), even though the FC head is the easier, more isolated change. Planned sequence: 1. **Int16-quantize the embedding table first** (`g_rgfNnueWeight`, `g_rgfNnueBias`) -- halves its footprint (~23MB -> ~11.5MB), directly targets the measured memory-bound bottleneck via improved cache hit rate on `NnueAccumAddFeature`/`RemoveFeature`'s per-move gather. Fixed-point scale TBD from the 256-512 range checked safe above; `NnueAccumRecompute`'s accumulation itself can likely stay int32 (cheap, avoids overflow entirely) even once the table it gathers from is int16. 2. **Int8-quantize the FC head second** (`nnueeval.c`'s `NnueForward`) -- isolated to one file, runs once per evaluated leaf (not per move), and gets the largest *relative* SIMD win (int8 dot products are 32-wide under AVX2 vs. SSE's 4-wide float32) once it's worth doing at all. 3. **New weight-file format version required.** `NnueLoadWeights` currently validates magic (`"NNU1"`) + dims but has no dtype/format field -- an int16/int8 export with the same dims would silently be read as float32 and produce garbage, exactly the failure class the existing dim-check exists to prevent. Bump to `"NNU2"` (or add an explicit format word) before shipping any quantized export. 4. **New correctness test required -- `TestNnueAccumulator` will not catch quantization error.** It only compares incremental maintenance against `NnueAccumRecompute`; once both sides are quantized identically they still agree with each other regardless of whether quantization degraded the actual eval. Need a separate check: float `NnueForward` vs. quantized `NnueForward`, same position, across a few hundred FENs, comparing the resulting centipawn deltas against some acceptance threshold (TBD). 5. **Side benefit, not the goal**: integer incremental accumulator add/subtract is exact (no float drift), so quantizing removes both `_CompareAccumToRecompute`'s 0.01f epsilon tolerance and any long-move-sequence floating-point drift question entirely. 6. Re-run `test_vs_head.sh` with `NNUE_WEIGHTS` after each step to measure the real nps delta before moving to the next step -- same reasoning as checking AVX2 in isolation before quantizing: don't stack two speed changes in one measurement. 7. Once NNUE's effective search depth is no longer handicapped by several plies at the standard time control, re-run the `match_play.py` gate for a real read on judgment quality, largely decoupled from the raw-speed confound -- and only then decide whether margin recalibration (see "Integration into search" above) is worth doing before or after a Phase B (GPU-trained, larger net) run. ## Training compute - No reliable local GPU. Decided against Google Colab's notebook-session model (session timeouts, no persistent SSH access) in favor of **hourly GPU rental with real shell access** -- vast.ai, RunPod, or Lambda Cloud. Rough order-of-magnitude expectation: a Stockfish-sized net (low millions of parameters) over tens of millions of positions is on the order of 1-2 hours/epoch on a single consumer-class rented GPU, full run under a day; CPU-only training of the same workload is roughly 20-50x slower (no tensor cores/wide SIMD at that scale), making a full run impractical locally but a small-scale pipeline-smoke-test (a few million positions, few epochs) plausible in hours-to-a-day on CPU if needed for early iteration before committing to a paid GPU run. - **Decided 2026-10-15**: Phase A uses exactly that local-CPU pipeline-smoke-test path -- PyTorch on CPU, against the ~2M-position quiet-filtered train/holdout pool already built, purely to prove the training script/data loader/export pass work end-to-end and to get fast local iteration when something breaks. GPU rental is deferred to the real, larger Phase B training run. ## Integration into search - Lazy-eval tier restructuring: today's three-tier super-lazy/lazy/full `Eval()` system does not map 1:1 onto NNUE, which doesn't have comparable internal tiers the way the handcrafted eval's term-by-term structure does. Tentative new tiering: (optional, experimental) material-only super-lazy prefilter -> cheap linear PSQT-term-of-the-net first pass -> full net forward pass. Decide how much of `eval.c`'s existing tiering logic is kept vs. deleted once this is actually implemented, not before. - Futility/EFP/LMR/NMP margins are all tuned against the current `Eval()`'s score distribution and will need full recalibration against the new net's score distribution -- expect this to be a comparably sized project to the LMR/NMP tuning already referenced in CLAUDE.md, not a quick follow-up step. ## Validation and rollout - **Gate using `eval_tune/match_play.py`'s existing LOWER95 methodology** (already used for DNA-tuning candidates) -- head-to-head match at fixed search depth, gate on `LOWER95 >= 0.5`, not raw score, for the same non-determinism reasons already documented for DNA tuning. - Snapshot a fresh `head_reference`-style baseline before any search-affecting plumbing (accumulator maintenance, cTrapped/cEnprise decoupling) lands, per the existing `head_reference/` protocol in CLAUDE.md -- this is a broad enough change to want a clean before/after reference point, same as any other search-affecting change. - Keep both evaluators coexisting behind a toggle for as long as practical, so regressions can be isolated to "is it the net" vs. "is it the recalibrated margins" vs. "is it the accumulator plumbing" rather than landing all three at once. - Do not retire the handcrafted `Eval()` until NNUE wins convincingly and consistently under the `match_play.py` gate. ## Sequencing / milestones 1. **Done 2026-10-15.** Canonical HalfKAv2_hm feature-indexing spec written down + Python/C cross-checker built and passing on known test positions. 2. **Done 2026-10-15, concluded not a blocker.** `cTrapped`/`cEnprise` consumers and producers enumerated (see "Breaking external dependencies" above). `cEnprise`'s null-move- refutation source needs no change. Everything else -- `cTrapped`'s only source, `cEnprise`'s other source, and 3 of `EstimateHangingMaterialCp`'s 5 checks -- is entangled in `_EvalTrappedPieces`'s attack-bitboard logic, but degrades gracefully (the same low-information fallback path already fires routinely today on lazy/super-lazy eval exits) rather than breaking -- a precision shift in the joint fail-high prediction table's "dumb move" axis, not a correctness bug. **Decision: ship the eval swap first, measure the actual pruning/reduction impact with existing joint-table instrumentation, and only build an independent bitboard pass if that measurement says it's worth it.** No speculative tactical-signals pass planned up front. 3. **Done, on `nnue-accumulator-plumbing`.** MakeMove/UnmakeMove accumulator plumbing (ply-indexed king-move backup in `PLY_INFO`) built and proven correct via `testnnue.c`'s `TestNnueAccumulator`, and proven fast after finding and fixing a real ~50% throughput regression (see "Status (2026-10-15)" above) -- gated behind `g_fUseNnueEval` so it costs nothing with NNUE off. `head_reference` snapshot already existed per the existing protocol. 4. **Done.** TWIC annotation harness built and run (Stage A, 2.55M positions from 30,000 `twic_filtered.pgn` games); ``/``/ ``/`<=>`/`"only"` tag semantics investigated and resolved (see "Status (2026-10-14)" above, including the real engine bugs found along the way). 5. **Done for Phase A (CPU-trained bootstrap, not the GPU-trained Phase B net).** First net trained (`nnue_train.py`, 20 epochs, train MSE 0.0065 / holdout MSE ~0.022) and exported (`export_nnue_weights.py` -> `weights.nnue`). **Quantization-aware export pass is explicitly NOT done yet** -- current export is plain float32; see "Quantization plan" above for the concrete, not-yet-started next steps. 6. **Done**, modulo quantization. C inference code (`nnueeval.c`'s `NnueForward` + `NnueLoadWeights`) written and running against real search positions via the `nnue eval` command and live search integration. **Not yet validated against the Python model's output position-by-position** -- worth doing (a batch of FENs run through both `nnue_model.py`'s `forward()` and `NnueForward`, comparing scores) before trusting the C side's numerical correctness beyond "it loads and produces plausible-looking scores," which is all that's been checked so far. 7. **Done.** Wired into search behind `g_fUseNnueEval`/ `g_fNnueWeightsLoaded` (`Eval()`'s short-circuit). **Margins NOT yet recalibrated** -- still running under HCE-tuned futility/EFP/ LMR/NMP margins, which the first match-play result's analysis flags as a likely real confound, not yet quantified. 8. **In progress, first result in hand, inconclusive by design (stopped early).** `match_play.py` gate: 70 games, `-45 =0 +25`, llr nowhere near either SPRT bound. Not resumed yet -- current priority is closing the speed gap (quantization) before spending more match time on a comparison known to be confounded by it. Old `Eval()` stays live and is the default (`g_fUseNnueEval` defaults `FALSE`) until NNUE wins convincingly and consistently under this gate. ## Status (2026-10-15, quantization session): both quantization steps done, validated, and measured Both steps of the "Quantization plan" above are now implemented, on the `nnue-accumulator-plumbing` branch, in this order: - **Step 1 -- embedding table int16, done.** `g_rgfNnueWeight`/ `g_rgfNnueBias` (`chess.h`/`nnueaccum.c`) are now `SHORT`, at a fixed scale of 4096 (checked against `ft.weight`'s real max|.|=5.01 -- `5.01*4096 ~= 20502`, well under `MAX_SHORT`, ~1.6x headroom for a larger Phase B net). `NNUE_ACCUMULATOR.rgiValue` (was `rgfValue`) is `INT` (int32), not `SHORT` -- summing up to 32 active features' `SHORT` rows can exceed `SHORT`'s range even though each individual row fits (measured worst-case real accumulator magnitude ~26 in float-equivalent units, i.e. ~106,500 at this scale -- 3x past `MAX_SHORT`). `NnueForward` converts back to float (divides by `g_fNnueFtScale`) only where the accumulator is read, never where it's maintained -- incremental `MakeMove`/`UnmakeMove` updates stay purely additive integer arithmetic, exact, no float drift. `testnnue.c`'s `TestNnueAccumulator` updated to match (integer test weights, exact-equality comparison instead of an epsilon tolerance -- a real correctness upgrade this dtype change enabled, not just a mechanical port). Passes clean under `DEBUG=1 TEST=1`. - **Step 2 -- FC head int8, done.** `g_rgfNnueFc1Weight`/ `g_rgfNnueFc2Weight`/`g_rgfNnueFc3Weight` (`chess.h`/`nnueeval.c`) are `CHAR` (int8, signed), one weight scale per layer derived exactly from that layer's own max|.| (no sampling needed, weights are fixed and known). Biases stay `float32` (tiny arrays, added post-dequant, not worth quantizing). Activations are UCHAR (uint8, unsigned -- confirmed all three stages are strictly post-ReLU, `min=0.0000` on a real 50k-position sample, so no bits wasted on a sign). `NnueForward` rewritten stage-by-stage: dequantize accumulator -> ReLU -> requantize to UCHAR at AS1 -> int8x8 dot product accumulated in `INT` -> dequantize (divide by `AS*WS`) + bias -> ReLU -> requantize at AS2 -> repeat through fc2/fc3. Matches a Python numpy simulation of the identical quantized math, built and checked before touching any C code (see below). - **Activation scales (AS1/AS2/AS3) are calibrated from real sampled positions, not a theoretical bound** -- unlike weights, activation ranges aren't knowable from the checkpoint alone. `export_nnue_weights.py`'s `CalibrateActivationScales` runs 50,000 real positions from `tensors_train/` through the float model with forward hooks, takes the observed max per stage, and applies a 1.3x headroom multiplier before deriving the scale (`AS = 255 / (observed_max * 1.3)`). Real observed maxes: `fc1_in` 27.12, `fc2_in` 19.11, `fc3_in` 7.72 (p99.9 well below max in each case -- confirms the max is tail-driven, headroom is doing real work, not padding noise). - **File format bumped to `"NNU3"`** (`NnueLoadWeights` rejects both older `"NNU1"` and `"NNU2"` files explicitly, not silently). Header now carries `ft_scale` (1x float), `act_scale` (3x float: AS1/AS2/AS3), `w_scale` (3x float: WS1/WS2/WS3) before the weight arrays themselves. - **Accuracy measured before touching any C code, same discipline as step 1's safety check**: a numpy simulation of the exact quantized int8 math (uint8 activations, int8 weights, int32 accumulation, per-layer scales) against the real float model, 5000 real holdout positions: **mean |delta| 1.55cp, median 0.99cp, max 20.91cp, zero clipping anywhere** (weights or activations, all three layers, out of 2.56M+160K+1K+32 values checked). Context: the project's own `sd10`-vs-`sd16` depth-adequacy check (training-label noise already tolerated) measured mean |delta| 19cp -- this quantization's error is **over 10x smaller** than noise already baked into the training data. Confirmed again directly in C after implementation: `nnue eval` on 4 hand-picked real FENs vs the Python float reference, deltas of 1/2/11/0 cp -- consistent with the offline simulation, no sign of an implementation bug. - **AVX2+FMA confirmed as a real, separate win once real weights are loaded** (not the "measured no-op for NNUE" result from the pre-quantization investigation above -- that measurement predated knowing which function actually dominated runtime, see the profiling finding below). `gmake AVX2=1`: `objdump` on the linked binary confirms `NnueForward`'s dot-product loops widen from 4-wide `xmm`/`mulps`+`addps` to 8-wide `ymm` with 99 real `vfmadd*` fused instructions. Measured: int16-table-only build went from ~381-384k nps (no `AVX2=1`) to ~409-410k nps (`AVX2=1`) -- a real, repeatable ~7% win, zero retrain, zero risk. - **Profiling (`docs/PROFILE.md`'s FreeBSD `pmcstat -P` + `gprof` path) found `NnueForward` -- the FC head, not the embedding table -- dominates runtime: 34.0% of total self-time, alone**, more than double the next entry (`SlidePiece`, 12.6%) and roughly on par with *all* accumulator-maintenance-related functions combined even when generously over-attributing their non-NNUE move-bookkeeping cost to NNUE. `NnueAccumAddFeature`/`RemoveFeature` don't even appear as separate flat-profile line items -- small enough to get inlined into their `SlidePiece`-family callers. This directly revised the quantization plan's priority ordering: the embedding table (step 1) was assumed memory-bound-dominant from the AVX2 experiment's "no-op for NNUE" result, but that result just meant AVX2 doesn't help *that* code path -- it didn't establish the table was the actual bottleneck. `fc1`'s matmul (`2*L1 x L2 = 2*256*32 = 16,384` multiply-adds per evaluated leaf) being far larger than `fc2` (1,024) or `fc3` (32) explains why: cost at each layer boundary scales with the *product* of adjacent layer widths, and the 512-wide-to-32 transition is the one expensive step in this topology, more frequently hit (once per evaluated leaf) and more expensive per-call than the table gather (once per move, but only a handful of 256-element row reads). - **Net bench result, chaining every change in this session**: | Build state | nps (real weights, `nnue on`) | |---|---| | Float32 everywhere (pre-session baseline) | ~411,000 | | int16 table, no `AVX2=1` | ~381,000-384,000 | | int16 table + `AVX2=1` | ~409,000-410,000 | | int16 table + int8 FC head + `AVX2=1` | **~559,000-563,000** | Net effect: **~37% faster** than the pre-session float32 baseline, closing the gap to classic HCE (~1.51M nps) from 3.9x down to 2.7x. Correctness verified at every step (self-tests, offline Python simulations before touching C, then direct C-vs-Python spot checks after). - **`WS1`/`WS2`/`WS3` derivation note**: unlike the embedding table (one global scale, since it's one homogeneous array), each FC layer gets its *own* weight scale, derived from that layer's own max|.| -- `fc1.weight` max 0.350, `fc2.weight` max 0.361, `fc3.weight` max 0.334 (all much smaller than `ft.weight`'s 5.01, but the relevant comparison for int8 resolution isn't the max, it's the *step size* relative to each layer's std -- at full-range scale this works out to roughly 18 quantization levels per standard deviation for the FC weights, versus thousands for the int16 table. The 1.55cp measured result shows this coarser per-weight resolution doesn't translate 1:1 into coarser *output* accuracy -- dot-product summation over many terms (512/32/32) partially cancels independent rounding errors, and ReLU's sparsity (46-69% exact zeros measured on real activations) contributes zero error for free on a large fraction of terms. ## Lazy-evaluation tiering plan (started 2026-10-15, not yet wired into search) Classic `Eval()` has three real exits -- super-lazy (material only), lazy (material + a defined subset of positional terms), full -- each gated by a margin derived from a *measured population* of real positions, not a modeled/assumed distribution. `Eval()`'s current NNUE short-circuit (added when NNUE was first wired into search, see "Status (2026-10-15): training run finished" above) bypasses all of this -- it's an unconditional full `NnueForward()` call, with a comment explicitly noting the margin machinery below it "has not been recalibrated against NNUE's score distribution yet." This section plans closing that gap. **The mechanism, read directly out of the live code** (not reconstructed from memory -- see `eval.c`'s `iSuperLazyResidualPessimisticByOwnArmy`/`OptimisticByOwnArmy`/ `P50ByOwnArmy` tables and their use in `Eval()`'s super-lazy exit, ~line 7150): - Every one of these tables is **literally empirical percentiles read off a measured population of real positions** -- the comment blocks above several of them show raw population stats before the table (e.g. `n=104187433 min=-68 p1=-10 ... p50=10 ... max=165`). No distribution shape is assumed; the percentile is read directly off sorted real data, per material bucket (`_MaterialBucket`/per-side army-scaler bucket, 8 buckets in the existing tables). - **Two different statistics for two different jobs, not one margin**: - **p50 is a bias correction** -- the typical/unbiased value fed to `iPositional` (a lazy exit's best-guess estimate of the missing positional terms, consumed downstream by e.g. QSearch's own futility margins). Current NNUE short-circuit reports a flat `{0, 0}` here -- the "intentionally least-specific but safe" placeholder, not a real estimate. A calibrated p50 of the probe's own residual could improve this once real residual data exists. - **p2/p98 (not p5/p95) is the soundness bound** -- answers "how far could the true full-eval score plausibly be from this cheap estimate, in the realistic tail," and is what actually gates whether the lazy exit is safe to take. The exact comment: *"p50 is the right statistic for a bias correction, p2/p98 is the right statistic for a soundness bound."* - Direction is asymmetric and combines two terms before a `MAX0()` floor: checking against `alpha` (could this fail low?) uses the **optimistic** tail (most generous plausible correction in the mover's favor); checking against `beta` (could this fail high?) uses the **pessimistic** tail (least favorable plausible correction). Getting this sign/direction wrong would silently invert the safety guarantee -- needs care, not pattern-matching, when implemented for real. - This two-statistic pattern (p50 bias, p2/p98 safety bound) is what the NNUE lazy tier should reuse, not a new invention -- same methodology the existing calibration harness (`CALIBRATE_MARGIN_SAFETY`, `RecordSuperLazyMarginSafetySwing`, swing histograms bucketed by material) already proved out for classic `Eval()`'s own tiers. **The proposed NNUE lazy-tier signal**: not a second, separately trained net (that reintroduces the calibration-mismatch problem already rejected in the "Architecture decisions" section above) -- instead, a **linear probe fit directly on the real net's own accumulator** (`fc1`'s input: the ReLU'd, concatenated 512-wide accumulator halves), which is already computed and resident at every node for free as a side effect of `nnueaccum.c`'s incremental maintenance, before `NnueForward`'s expensive `fc1->fc2->fc3` pass (34% of runtime, see profiling finding above) ever runs. **Progress so far**: - **Offline probe fit, done, preliminary.** Ridge regression (512 features + bias, lambda=1.0) fit against 80,000 real positions from `tensors_train/`, targeting the real float net's own raw output (`forward()`, pre-sigmoid). Saved to `annotate_pgns/probe_weight.npy`/`probe_bias.npy`. - **Measured residual (full net score minus probe score), 20,000 held-out positions from the same sample**: mean |residual| 50.10cp, median 33.09cp, p2=-166.79cp, p50=0.33cp, p98=+169.14cp. **An order of magnitude coarser than the int8 quantization's 1.55cp** -- expected, since a flat linear projection discards all of `fc1->fc2->fc3`'s nonlinear structure, unlike int8 which preserves the same computation at lower numeric precision. This sets real expectations: the implied safety margin (~150-200cp) is large relative to typical eval swings, so the lazy exit will only fire on nodes already lopsided relative to the search window -- a real but likely narrower slice of total nodes than hoped going in. - **Real fit-vs-validation gap observed** (mean 37.85cp on the fit split vs 50.10cp held out, p98 153cp vs 195cp) -- larger than expected given 80k examples against 512 features plus ridge regularization. Not yet root-caused (could be regularization strength not cross-validated, could be real limits of a flat linear fit over this representation, could be this offline sample's size). **Flagged as unresolved, not yet trusted as final** -- this was an offline, single-position-sample check, explicitly a preliminary step, not the real calibration. **Not yet done -- the actual next steps, in order**: 1. Root-cause the fit/validation gap above (cross-validate `lambda`, or just use more fitting data) before trusting any residual numbers from this probe as final. 2. **Build the real calibration pass**: a `CALIBRATE_MARGIN_SAFETY`- style harness for the NNUE lazy probe specifically -- same pattern as classic `Eval()`'s existing one (run the full `NnueForward` pass anyway even when the probe-based exit would fire, record the swing, restore the real build's behavior before returning) -- against **real search-node traffic**, not another offline position sample. This is the measurement that actually matters; the offline check above was only ever a cheap sanity gate before investing in this. 3. From that real population, derive p50 (bias correction) and p2/p98 (safety bound) -- possibly bucketed by material/phase the way HCE's own tables are, possibly not. **Whether NNUE's accumulator already implicitly encodes phase** (unlike HCE's explicitly material-gated lazy terms) **is an open, empirically-answerable question from this same population** -- check whether the residual distribution actually shifts meaningfully bucket-to-bucket before committing to that complexity. 4. Wire the gate into `Eval()`'s existing NNUE short-circuit, using `iAlpha`/`iBeta` already passed into `Eval()` -- no new interface needed, this is a parallel early-return block next to the current unconditional bypass. 5. Ship it instrumented-only first (swing recorded, real build behavior unaffected) before trusting it live, same rollout discipline as every other change in this project. The super-lazy (material-only, no net at all) idea from the "Architecture decisions" section above remains separate, more tentative, and explicitly not decided either way -- the calibration risk there (no shared units with the WDL-sigmoid-calibrated net output) is structurally different from this linear-probe tier, which shares the real net's own trained representation and units by construction. ## Status (2026-10-16): lazy probe gate shipped live, QSearch futility margins fixed, same-depth ECM win confirmed, overnight match queued Picking up from the "Lazy-evaluation tiering plan" section above -- the fit-vs-validation gap was root-caused as pure sampling noise (more data + the real `tensors_holdout/` split made it vanish; lambda barely mattered, the model was never near overfitting), and everything from "build the real calibration pass" onward is now done. - **Fit/validation gap resolved**: 500k-example fit + real `tensors_ holdout/` evaluation (not another ad-hoc split) showed holdout *better* than fit (38.68cp vs 41.09cp mean) -- the earlier 50.10cp number was noise from a too-small single sample, not overfitting. - **Real calibration done properly, twice**: first pass used the three curated tactical suites (91M samples) -- too narrow a slice (hand- picked for sharp tactics), confirmed by a second pass against `tests/cross_sectional_200k.ep_` (200,000 diverse positions, 16 parallel single-threaded shards, `sd 6`, 3.09 BILLION real search- node samples) landing close to but meaningfully more trustworthy than the first (p98 margins `[180, 200, 190, 180, 170, 160, 150, 150]` by material bucket vs the tactical-suite-only `[190, 200, 200, 190, 190, 170, 150, 150]` -- also surfaced real tail outliers (max 534cp) the narrower sample never saw). - **A real, separate bug found and fixed along the way**: chaining multiple commands after `script ` in one `--batch --command` string (e.g. `script a.ep_; script b.ep_; calibrate marginsafety`) silently truncates every search except the last to a shallower depth than requested -- `PushNewInput` (`input.c`) splits and queues *everything* up front, and `searchsup.c`'s mid-search `NumberOfPendingInputEvents()` check (legitimate for live xboard/ WinBoard responsiveness) fires on that queued-but-irrelevant trailing text, treating it as "stop, something wants my attention." Root cause confirmed by direct repro (`sd 10` silently landing at depth 8, 8x fewer nodes) -- **not fixed in the engine** (the interactive- responsiveness behavior is legitimate and deliberately left alone; the input-queueing semantics are too sensitive to touch for a testing-only problem that already has a zero-risk workaround). **Workaround, now the required pattern for any multi-step batch calibration**: never chain unrelated commands after `script` in one `--command` string; if a command needs to run after a suite finishes in the same process (e.g. `calibrate marginsafety`), put it as its own line *inside* the `.ep_` file instead -- `script.c`'s own `_ProcessSuiteLine` already dispatches any non-"go" line via the queue-free `ExecuteCommandLineDirectly`, so it runs in-order with no truncation risk. For N-way sharded runs (per CLAUDE.md's existing convention), each shard gets its own single `script` call + its own `calibrate marginsafety ` line inside the file, in its own process -- then combine the N raw histogram files after the fact. - **The "super lazy" (material-only) tier was investigated and rejected as a live feature** -- `pos->iMaterialBalance[color]` is already in the same cp units as `NnueForward`'s output (`VALUE_PAWN=100`), so no fit/regression was needed at all (unlike the accumulator probe) -- straight subtraction, same harness. Real measured residual against the full net, same 3.09B-sample cross-sectional pool: mean 124-344cp, p98 510-1230cp by bucket -- **6-8x larger than the accumulator probe's** (47-77cp mean, 150-200cp p98). Shape is an inverted-U by material bucket (peaks in the middle, smaller at both bare-king endgames and near-full-material positions), not the simple monotonic-opposite-of-the-probe trend originally guessed. Conclusion: material alone is too crude a proxy for *this* net's output to fire often enough to be worth it -- NNUE's accumulator captures far more positional judgment relative to material than a handcrafted eval does, unlike HCE where material really is close to the final answer. Not built as a live gate. - **Lazy probe gate shipped live** (`nnue lazyprobeon`/`lazyprobeoff`, defaults `FALSE` same as `g_fUseNnueEval`): `Eval()`'s NNUE short- circuit now actually skips `NnueForward` when `NnueProbeScore() +/- margin` confidently resolves fail-high/fail-low against `[iAlpha, iBeta]`, dual-mode via `#ifdef CALIBRATE_MARGIN_ SAFETY` (calibration build always still runs the full pass for measurement; release build actually skips). Held-out validation (`ecm_confident_sample2.ep_`, not used for calibration) showed exceeded rates mostly at or under the ~2% the p98 choice implies (buckets 2-7: 0.87-2.24%); buckets 0-1 ran hotter (3.47-4.67%, consistent with those being the smallest-sample buckets in calibration, so the noisiest estimates). - **Measured lazy-probe exit rate on real search traffic** (`twic_ sample.ep_`, `sd 6`): **14-23% of NNUE eval calls resolve via the cheap probe**, varying by material bucket (peak ~20-23% in buckets 3-5, low 8.7% in bucket 7/near-full-material). Most calls still pay for the full forward pass. - **Real, clean, sequential (not parallel -- see below) same-depth measurement, `ecm_hard_quick`, `sd 10`, gate off vs on**: | | OFF | ON | |---|---|---| | Solved | 32/90 | 33/90 | | Nodes | 47,790,608 | 50,438,386 | | Time | 112.4s | 107.1s | | nps | 425,271 | 470,938 | **~10.7% real per-node throughput win, no solve-rate regression** (if anything +1). Note nodes went *up* while time went *down* -- the gate changes which nodes get pruned (approximate score at some nodes shifts alpha-beta cutoffs), so node count isn't held constant the way `sd` depth is; nps is the cleaner isolated signal. **Caveat on parallel-run timing noise, confirmed twice this session**: running gate-off and gate-on as simultaneous background jobs on a shared box can be contaminated by real external load (cron jobs, rsyncs, other users) even when each job's own `--cpus 1` leaves plenty of idle capacity -- the fix was a clean *sequential* run on an otherwise-idle box, not avoiding parallelism itself (6-way `--cpus 1` parallel jobs were confirmed fine once the box was actually quiet). - **Full three-suite combined result, same clean methodology**: 108/191 (56.5%) solved OFF vs 112/191 (58.6%) ON, total time 247.7s vs 240.3s (-3.0%) -- consistent with the single-suite sequential number, no regression anywhere. - **Independently, at the SAME fixed `sd 10` depth (no lazy gate, separate from all of the above): NNUE solves substantially more ECM tactics than classic HCE** -- 108/191 (56.5%) vs HCE's 72/191 (37.7%) across all three curated suites. Since depth is held constant, this isn't explained by NNUE's remaining speed deficit. **Correction, 2026-10-17: do not read this as evidence the net plays better chess.** Every real head-to-head game match run since (see "Status (2026-10-17)" below -- five independent configurations: two different nets, `st 4` and `sd 10`, with and without the lazy probe) has NNUE losing decisively to classic HCE, including at matched search depth where this ECM result would predict an NNUE edge. ECM is a narrow slice (forced-tactic/mate puzzles, where qsearch/extensions/move-ordering machinery shared by both evaluators does most of the work) and evidently does not generalize to real-game strength. As of this writing, solving more ECM puzzles than HCE is the *only* thing either NNUE net has been shown to do better than HCE -- it has not won a single head-to-head match, at any depth or time control tried. - **QSearch futility-pruning margins needed their own investigation, separate from the eval-exit-gate margins above** -- `PLY_INFO. uLastPositionalSource` (the per-ply tag `search.c`'s `FUTILITY_BASE_MARGIN_BY_SOURCE[]` reads to decide how much slack to give QSearch's delta/futility pruning) was **never being set at all** by either NNUE exit -- every NNUE-sourced score was silently inheriting a stale value from the last classic-`Eval()` call at that ply slot (or the C zero-init default, `EVAL_POSITIONAL_SOURCE_FULL`, if none ever ran). Fixed: added `EVAL_POSITIONAL_SOURCE_NNUE_FULL`/ `NNUE_LAZY`, set at both NNUE return sites, each with its own `FUTILITY_BASE_MARGIN` entry and always-on exit-rate counters (`calibrate exitrate` extended to report NNUE's own lazy-vs-full split, separate from the eval-exit gate's). - **First real measurement** (`twic_sample.ep_`, 1500 positions, `sd 6`, the same pool/depth the *original* classic margins were derived from): `nnue full`'s surprise rate was 7.41% in the checking-capture/promo gate at the 450 starting placeholder -- well above anything in classic's documented history. Widening to 650 tested as a complete no-op (identical 7.41%, identical `n`). - **Root-caused, not just observed**: added a direct diagnostic (raw pre-clamp `iFutility` value, before `MAX0`) and confirmed `nnue full`'s population is **100.00% clamped to the `MAX0` floor** (mean pre-clamp -589.3) -- the margin constant is never actually read for this population, so no value could have shown an effect. Classic `full eval`, measured the identical way on the identical pool, is clamped 91.05% of the time (not 100%, and less deeply negative: mean -282.5) -- saturation is a real, general QSearch structural property (not NNUE-specific: classic `regular lazy`'s 24.95% clamp rate matches `nnue lazy probe`'s 24.76% almost exactly), but NNUE's `FULL` tier saturates somewhat more completely than classic's. - **Surprise magnitude (not just rate) measured too**, resolving a live "is NNUE just noisier" vs "does NNUE miss bigger things" question: `nnue full`'s surprises, when they happen, run *smaller* than classic `full eval`'s (mean 477.5cp vs 938.7cp, `>100cp` 9.88% vs 28.74%) -- same pattern for `nnue lazy probe` vs classic `regular lazy` (5085.3cp vs 11030.2cp mean; 19.45% vs 69.73% `>100cp`). So the net effect is: NNUE_FULL fails *more often* but *less expensively* each time -- consistent with more frequent small eval jitter near the decision boundary, not with NNUE being tactically blind to bigger things than HCE already is. One sample verified by hand (a genuine `Qh7+` mate-in-1, confirmed independently via a real `sd 16` search) to make sure the diagnostic's rare huge-magnitude entries were real finds, not a measurement bug. - **Decision: `FUTILITY_BASE_MARGIN_NNUE_FULL` reverted to 450** (matching classic `FULL`) rather than kept at the untested 650 -- the saturation finding means neither value was actually validated by this data, and 450 is the more defensible default given NNUE_FULL's measured surprises skew smaller than HCE's own. **`FUTILITY_BASE_MARGIN_NNUE_LAZY=200` kept** -- this one *is* genuinely validated (only 24.76% clamped, moved the expected direction when tightened from 350, landed in a sane 1.3-2.1% surprise-rate range, smaller surprise magnitudes than classic's own `regular lazy`). - **TODO, not yet done**: find and calibrate against the actual non-saturated `NNUE_FULL` population specifically (the ~0% slice this test pool never surfaced) before trusting any `NNUE_FULL` margin as real, NNUE-specific-validated rather than a reasonable borrowed default. Likely candidate: nodes where `iEval` sits close to `iAlpha` rather than comfortably above it (the saturated case is "`iEval` way above `iAlpha`" -- look at the complement). - **Match-play CLI wiring added, needed before any NNUE+lazy-gate match can run at all**: `--nnueweights` already loaded weights and set `g_fUseNnueEval=TRUE` at startup (for `match_play.py`'s fresh- process-per-game spawning, no interactive commands available) -- but there was no equivalent for the probe/lazy gate at all. Added `--nnueprobeweights ` (same convention, sets `g_fUseNnueLazyProbe=TRUE` automatically), and extended `eval_tune/ test_vs_head.sh`'s `NNUE_WEIGHTS` wrapper with a new `NNUE_PROBE_WEIGHTS` env var that injects `--nnueprobeweights` into the same shim. Verified working end-to-end (both flags load and enable correctly) before trusting it for an unattended run. **Gotcha**: `test_vs_head.sh`'s default (no candidate path given) rebuilds the candidate itself via `gmake clean && gmake release MP=1 PERF_COUNTERS=1` -- **without `AVX2=1`** -- silently losing the ~7% AVX2/FMA win if not passed a pre-built candidate path instead. - **Overnight `match_play.py` run queued** (not yet run/reported) -- NNUE+lazy-gate candidate vs classic HCE `head_reference`, using the new `NNUE_WEIGHTS`/`NNUE_PROBE_WEIGHTS` env vars and an explicit pre-built `AVX2=1` candidate binary to avoid the gotcha above. ## Next up 1. Read the overnight match result once it's in -- first real judgment-quality read with both the quantization work and the lazy probe gate live together, at a real time control rather than fixed `sd` depth (so the remaining ~2.7-3.2x speed gap is a live confound again, same as the original 70-game result, but now with a smaller gap and the added lazy-gate throughput). 2. The `NNUE_FULL` futility-margin TODO above -- find the non-saturated population and calibrate against it specifically, rather than continuing to rely on a borrowed classic-HCE default. 3. Decide whether the ~3-11% nps win from the lazy probe gate (measured) is worth the complexity on its own, or whether it's more valuable combined with further quantization work on the FC head's remaining float32 pieces (biases) or other speed levers -- revisit once the overnight match gives a real strength read to weigh against. ## Status (2026-10-17): material-blindness fixed and verified, but zero ## head-to-head wins against classic HCE in five independent trials **Summary, stated plainly up front**: a real, measured defect (the net could not distinguish losing a pawn from losing a queen in crowded midgame positions) was found and fixed this session. The fix is verified to work, at the population level, with a new reusable tool. But **no NNUE configuration tried today -- old net, new net, with or without the lazy probe, at `st 4` or at matched `sd 10` depth -- won or even drew competitively against classic HCE.** The only thing either net has been shown to do better than HCE is solve more ECM puzzles at matched depth (see the correction just above). Do not read anything below as evidence the net plays winning chess yet. ### The material-blindness bug, found and fixed - **Found via `sanity_check_net.py`** (new tool, `annotate_pgns/`): loads a raw training checkpoint directly (no export/C round-trip needed -- the earliest point in the pipeline a net can be checked) and removes one piece at a time from hand-picked and (better) real sampled positions, reporting the score delta. On the Phase A net trained with plain `λ=1.0` MSE against WDL-sigmoid targets: removing a pawn, a knight, a rook, or a queen from the same midgame position produced nearly indistinguishable score drops (-0.56 to -0.70) -- directionally correct, magnitude-blind. - **Root cause, two compounding mechanisms, neither one specific to which corpus a position came from**: 1. Sigmoid-gradient argument (later found incomplete on its own -- see below): training loss is MSE against `sigmoid(raw/K)`, and sigmoid's gradient vanishes once a position is already clearly lost/won. 2. **Feature dilution / shortcut learning in crowded positions** (the more complete explanation): with ~20+ active embedding features summed per position, a net can match training labels well without correctly attributing any single piece's marginal value, because real games never present a controlled single-piece-ablation pair to force that attribution. Confirmed empirically: the bug was severe in crowded midgame positions and essentially absent in sparse endgames (2-6 pieces), where each piece dominates a much larger share of the accumulator. - **Data rebalancing (TWIC downsampled to match self-play 1:1 by game count) was necessary but not sufficient.** It measurably helped the sparse-endgame case (which TWIC's human-resignation bias structurally starves) but did nothing for the crowded-midgame case, confirming the bug was a representation/optimization problem, not purely a data-coverage gap. - **Two cheap levers tried and only partially effective on their own**: - `loss_exponent=2.6` (Stockfish's own documented value, reweights the loss toward large errors): modest effect, spread narrowed but monotonicity still failed, bishop/rook order still inconsistent across test positions. - λ-sweep (blend in `r_outcome`, game result): **made things actively worse** -- λ=0.5/0.0 flipped the sign of several deltas (removing a piece appeared to *help* the score). See the "λ-sweep methodology was weaker than later standards" note below before trusting this conclusion, though -- it has not been properly re-tested since. - **Lesson from this part of the investigation**: a single hand-picked test position cannot reliably settle a question like "does the net value a bishop more than a rook" -- one test showed bishop > rook, a different hand-picked position showed the reverse. This is per-position confounding (an active bishop vs. a boxed-in rook on that specific square/position), not necessarily evidence about the net's general prior. - **Fix that worked: an auxiliary material-probe loss** (`nnue_model.py`'s `material_probe`, a plain `Linear(512, 1)` reading the same concatenated accumulator `fc1` does). Trained jointly with the main WDL loss: `loss = wdl_loss + material_loss_weight * mse(material_probe(accumulator), material_balance)`, where `material_balance` is computed directly from the FEN (pawn=1..queen=9, mover-relative, no sigmoid -- deliberately non-saturating). `material_loss_weight=0.01` chosen to keep the material term's raw contribution smaller than the WDL loss's throughout training (per the user's explicit concern about over-fixating on material, HCE-style) -- confirmed: material loss contributed ~0.019 vs. WDL's ~0.044 at epoch 0, ~0.007 vs. ~0.022 at epoch 19. - **Mechanistically why this is safer than adding material to the output directly**: the probe loss only updates `material_probe` itself and (via backprop) the shared `ft` embedding table -- it never touches `fc1`/`fc2`/`fc3`, the actual eval head. The eval head stays free to deviate from pure material counting for positional reasons; only the *representation underneath it* is forced to keep material linearly recoverable. - Needed more epochs than the baseline run to show its full effect -- material-probe MSE was still falling at epoch 19 (not plateaued), extending to epoch 34 took it from a single hand-picked test's queen delta of -0.815 (already beating rook's -0.727) to a much larger, cleaner separation. ### `sanity_check_net.py --cross-sectional`: the real validation tool Single hand-picked positions were repeatedly shown to be unreliable (see above) -- replaced with a population-level check: sample N real positions from a holdout pool, remove one random piece of each present type per position, pool deltas by piece type across the whole sample, report mean/median/std/p25/p75. On the matprobe net (2000 sampled holdout positions): **population means came out cleanly monotonic in piece value, both directions** -- pawn -0.747/+0.786, knight -1.363/+1.581, bishop -1.500/+1.710, rook -2.483/+2.907, queen -4.160/+5.024 (own-piece-removed / opponent-piece-removed). The earlier bishop-vs-rook inconsistency vanished entirely at the population level, confirming it was per-position noise, not a real defect. Scaling is compressed relative to nominal chess piece values (queen/pawn ratio ~5.6x here vs. nominal 9x) -- expected and likely correct, not a bug, since training targets are WDL-sigmoid-calibrated scores, not raw material counts, and a large material edge's marginal effect on win probability is genuinely smaller than its point value. This tool (`annotate_pgns/sanity_check_net.py`) should be run on any future checkpoint before trusting a single hand-picked-position read -- `--cross-sectional --n-positions 2000` against a holdout pool, `python3 sanity_check_net.py --ckpt ` for the quick hand-picked smoke test. ### Export and C-side validation: clean `export_nnue_weights.py` (already quantizing directly to the `NNU3` format, no separate float32-only export step in the current pipeline) ran against the matprobe checkpoint with zero code changes needed -- it only reads `ft`/`fc1`/`fc2`/`fc3` by name, so the auxiliary `material_probe` layer is silently and correctly ignored at export time. 6 real holdout FENs spot-checked, Python float vs. C int16/int8-quantized: deltas of 0.3-4.7cp, same ballpark as the 1.55cp-mean/20.91cp-max already measured and accepted for the first net's quantization step. No sign/implementation bug found. ### Five independent head-to-head trials, all losses All via `eval_tune/test_vs_head.sh` with `NNUE_WEIGHTS` set (and, for two of the five, `NNUE_PROBE_WEIGHTS` pointing at the OLD net's lazy probe -- see the stale-probe caveat below), against the classic-HCE `head_reference`-equivalent head binary: | # | net | time control | result (at stop) | llr | candidate depth vs head | |---|---|---|---|---|---| | 1 | matprobe (new) | `st 4` | -27 =2 +1 (n=30), score 0.067 | -2.07 | 12.16 vs 15.08 | | 2 | v3 (old) | `st 4` | -29 =1 +0 (n=30), score 0.017 | -13.05 | 10.74 vs 13.51 | | 3 | v3 (old) | `sd 10` | -28 =2 +0 (n=30), score 0.067 | -1.52 | 9.57 vs 9.23 | | 4 | matprobe (new) | `sd 10` | -24 =4 +2 (n=30), score 0.133 | -0.97 | 9.81 vs 9.70 | | 5 | v3 (old) + lazy probe | `sd 10` | -30 =0 +0 (n=30), score 0.000 | (total wipeout) | 9.63 vs 9.25 | **Key findings**: - **The new (matprobe) net consistently loses less badly than the old net, under every condition tested** (trial 1 vs 2 at `st 4`: score 0.067 vs 0.017; trial 4 vs 3 at `sd 10`: score 0.133 vs 0.067). The material fix is a real, reproducible improvement -- just nowhere near enough to close the gap with HCE. - **At matched `sd 10` depth (trials 3 and 4), NNUE still loses decisively** -- this directly contradicts the working hypothesis (built from the ECM result above) that the real-time losses were purely a depth/speed-handicap artifact. They are not purely that. Matching the final depth number does not mean matching search quality throughout the tree -- EFP/LMR/NMP/futility margins are still HCE-tuned in every one of these trials (see "Integration into search" above, still not done) and could still be causing bad cutoffs specific to NNUE's score distribution even when the reported depth matches. - **The lazy probe used in trial 5 is STALE and the result should be discounted**: `annotate_pgns/nnue_probe.bin` was fit against the OLD net's (v3) accumulator specifically, before this session's retrain. It happens to be the correct vintage for trial 5 (v3 + v3's own probe), so that comparison is internally consistent, but no probe exists yet for the matprobe net -- `typhoon_nnue`'s wrapper script was updated this session to point at the matprobe net by default and to NOT fall back to the stale probe file (previously silent and wrong). A new probe would need to be fit + calibrated from scratch against the matprobe net's accumulator before it could be used in a trial 4-equivalent run. - **A user-remembered "NNUE competitive or better" result could not be reproduced under any of these five conditions.** Likely explanations, unconfirmed: a different match configuration entirely (not `st 4` or `sd 10` against this head binary), or a misremembered result. Do not plan further work assuming that result is real until it can be reproduced under a specific, documented configuration. ### Pipeline sanity check: ruled out a structural bug Given five straight losses, including at matched depth where ECM predicted an edge, the obvious question was "is the training pipeline itself broken" (bad feature indexing under real gradient flow, broken accumulator math, a training-loop bug) rather than a genuine net-quality gap. Tested directly (`annotate_pgns/ overfit_sanity_check.py`, new tool): sampled 3000 real positions, scored each with the engine's own classic `Eval()` (fast persistent-context `eval` command), and trained the *same* architecture to directly regress raw output against that score (plain MSE, no sigmoid, no K, no λ, no quiet filter, no quantization -- the simplest possible test of "can this architecture memorize a small, clean, noiseless target at all"). **Result: clean overfit** -- RMSE collapsed from 307cp to ~3-5cp over 400 epochs on the fixed 3000-point set (max error 46cp). **No structural bug found.** The five head-to-head losses are real evidence about net/training quality, not a pipeline defect. ### Revised understanding of why: likely bootstrapping off HCE's own ### search sets a ceiling, not (as earlier argued) label noise or ### architecture capacity Two explanations floated earlier in this investigation don't hold up well under scrutiny: - **Architecture capacity**: the 256x2-32-32-1 shape is the literal one Stockfish's own early, successful NNUE work used. Not a good suspect. - **"Search-score labels are noisy"**: standard practice (`sd 8`-ish search-score training is exactly what real NNUE projects use); the depth/source of search isn't inherently suspect. **The sharper version of the concern**: every label in the current training pool -- TWIC and self-play both -- comes from THIS SAME engine's own classic `Eval()` run through its own `sd 10` search, and training used `λ=1.0` (pure `p_search`, zero weight on actual game `r_outcome`). A net trained to perfectly reproduce "HCE's static eval, distilled to one forward pass" can at best approach HCE's *positional judgment* -- it has no training signal that could let it exceed HCE's actual playing strength, which is that judgment plus a mature, decades-tuned full search. This is single-generation distillation from the exact opponent being benchmarked against, with no independent ground truth in the loop. Real NNUE bootstrapping success stories (including Stockfish's own) iterate through many generations, each net's own self-play becoming the next round's data, with game outcomes mattering throughout -- not one shot of pure distillation-from-one-engine's-shallow-search. **This reframes the earlier-rejected λ-sweep as an open question again, not a settled one** (see the "λ-sweep methodology was weaker than later standards" note above) -- a properly-controlled λ=0.5 run (combined with the material-probe fix, trained to real convergence, judged with the cross-sectional population check, not a single hand-picked position) is in progress as of this writing to answer this properly. ### Next up 1. **In progress**: λ=0.5 + material-probe-loss combined run, trained to convergence, judged with `sanity_check_net.py --cross-sectional` -- the properly-controlled version of the λ-sweep question above. 2. Once a net shows a real head-to-head result against HCE (win, draw, or a meaningfully-less-bad loss than trial 4's 0.133), THEN invest in margin recalibration (EFP/LMR/NMP/futility, still 100% HCE-tuned today) and a new lazy-probe fit for whichever net is being kept -- not before, per this session's experience chasing search-side explanations that turned out to not be the dominant factor. 3. Data volume scale-up (TWIC + self-play, more self-play generations specifically using the matprobe-or-later net to play, per the original Phase 2 plan) remains the most clearly-actionable lever, but should be paired with an actual outcome-grounding signal in the training target, not just more HCE-sd10-labeled positions of the same shape already in hand. ## Status (2026-10-18): Stockfish-labeled relabel in progress; feature ## factorization planned as the next architecture-side lever Labels are being regenerated using Stockfish (not this engine's own `Eval()`/`sd 10`) to break the single-generation self-distillation problem described above -- that's a speed/ground-truth fix, not believed to be the root cause of the five head-to-head losses by itself. In parallel, researching Stockfish's own NNUE docs surfaced a second, independent candidate explanation worth trying once there's enough Stockfish-labeled data to re-run training: **feature factorization** (Stockfish's term for a training-only "virtual feature set" that is coalesced into the real weights post-training; see https://github.com/official-stockfish/nnue-pytorch/blob/master/docs/nnue.md#feature-factorization, excerpted into this conversation). ### Why this looks like a real lever here, not just a ported idea The current feature set is HalfKAv2_hm per the canonical spec above: 22528-dim, king-bucketed (32 buckets via `king_bucket()`'s horizontal-mirror quartet), so every `(piece_square, piece_type, piece_color)` combination exists as up to 32 separate rows, one per king bucket, each seen only when the king happens to sit in that bucket. Checked directly against the current real training cache (`runs/sd10_twic_2M/tensors_train`, 1,848,411 positions): - **23.9% of the 22528 HalfKAv2 rows have zero observations** -- never fire once in training, never receive a gradient update. - **60.5% have fewer than 50 observations.** - Collapsing the same data to the king-bucket-independent "P" feature set (704 rows: `piece_square x piece_type x piece_color`, no king info) gives a median of **8375** observations per row -- roughly 32x denser, as expected, with none left permanently untrained. So a substantial fraction of the net's weight space is currently stuck at random init or barely trained regardless of label quality -- plausibly part of why two different training attempts "worked" (loss went down, no crash) but neither net could beat HCE in head-to-head play. This is Stockfish's own "dusty corners" framing: sparsity from king-conditioning starves rare (king, piece, square) combinations of signal, which factorization is specifically designed to fix during early training. ### Planned implementation (not yet done -- next after enough ### Stockfish-labeled data exists to re-run training) 1. **No change to `build_training_tensors.py` / no cache rebuild needed.** The HalfKAv2_hm index formula is `idx = (sq ^ mask) + piece_plane + king_bucket * PS_NB` where `(sq ^ mask) + piece_plane <= 703 < PS_NB (704)`. So the P-factor index is exactly `real_idx % 704` -- recoverable from the existing `white_idx`/`black_idx` tensors at train time, in `nnue_model.py`. 2. **`nnue_model.py`**: add a second embedding table, `self.ft_p = nn.EmbeddingBag(704, l1, mode="sum")`, **zero-initialized** (matters: both tables sum into the same accumulator, so a default-init P table would double the accumulator's scale at step 0, confounding the factorization effect with an effective LR change), with no bias of its own (`ft_bias` stays as the only bias, or coalescing below isn't exact). In `accumulator()`, derive `white_idx % 704` / `black_idx % 704` and sum `ft_p`'s output into each perspective's accumulator alongside `ft`. 3. **Skip the "K" and "HalfRelativeKP" factors** Stockfish's doc also describes. HalfKAv2_hm already encodes the king directly (kings are included in the `piece_map()` loop via `piece_plane()`'s `PS_KING` case), so the K factor would be redundant here. One variable at a time, per the incremental-change rule used throughout this project -- P alone first, measure, then decide. 4. **Coalescing after training, in `export_nnue_weights.py`**: this is the step most likely to silently fail, because a bad coalesce produces a net that is *worse* than not factoring at all (the real HalfKAv2_hm rows learn less because the factor absorbed the shared signal) and the training loss curve won't show it either way. `ft.weight += ft_p.weight[torch.arange(22528) % 704]`, then drop both `ft_p` and `material_probe` from the exported weights (two training-only heads now, same pattern). **Required check**: assert the pre-coalesce factored forward pass and the post-coalesce plain-`ft`-only forward pass agree to ~1e-5 on a real batch before trusting the export. 5. **Evaluation protocol**: Stockfish's doc is explicit that the factorizer's effect is concentrated in *early* training and fades as training progresses -- compare early-epoch loss descent against a matched-seed, matched-epoch unfactored control, not just final validation loss. A final-loss-only comparison could read a real early-training effect as zero. ### Sequencing This factorization work is queued behind the in-progress Stockfish relabel -- the two are independent, complementary levers (ground-truth quality vs. training-signal density under sparsity), and per this project's incremental-change discipline they should be evaluated one at a time rather than conflated in a single before/after comparison. ## Status (2026-10-18, continued): Stockfish-labeled corpus built, P-factor ## factorization implemented and measured -- why it's load-bearing, not optional Both levers above are now done, in sequence, against the new `pgn/stockfish_self_play_training_data.pgn` corpus (35,295 Stockfish- vs-Stockfish games, 4,008,045 positions, labeled via Stockfish's own search rather than this engine's `Eval()` -- see `extract_selfplay_ positions.py`'s `MATE_CLAMP_THRESHOLD=90.0`, needed because this foreign-engine corpus encodes mate as a clamped float, `[99.00,99.99]` pawns via PolyGlot's UCI-to-xboard bridge, not typhoon's own `+M`/`-M` sentinel token). Full pipeline run at `lambda=0.75` (`target = 0.75*p_search + 0.25*r_outcome`, chosen over the Phase-A `1.0` default specifically because pure single-engine-distillation was the leading suspect for the prior two nets' failure to beat HCE): `split_holdout.py` -> `fit_k_and_label.py` (K=2.4130, this corpus's own fit, NOT the TWIC- fit `DEFAULT_K=1.3123` -- the two corpora have different score distributions and must not share a K) -> `quiet_filter_games.py` -> `build_training_tensors.py` (now with `--workers`, default 16, multiprocessing-sharded across positions, verified byte-for-byte identical output to the old sequential path -- the per-position feature-index computation has no cross-position dependency, so this was a clean win on an idle multi-core box). **Why factorization is necessary here, not a nice-to-have**: measured directly on this corpus's real training tensors (not a TWIC estimate), 2,237 of HalfKAv2_hm's 22,528 rows (9.9%) never fire once across all 2.84M training positions -- confirmed, after training, to be **bit- for-bit identical** to their random pre-training init (diffed the trained checkpoint's `ft.weight` against a freshly re-seeded, untrained `NnueNet()` at the same seed: max difference on those rows is exactly 0.0, vs. a clearly nonzero shift on every observed row). This 9.9% figure is *better* than TWIC's 23.9% (this corpus's decisive, played-to-conclusion games naturally cover more king positions than human-resignation-biased TWIC), but still real, and **not fixable by collecting more data at any practical scale**: subsampling the existing pool at increasing sizes and fitting the shrinkage curve (`zero_count ~= 5529 - 226*ln(N)`, log-linear, fit on 6 points from 10% to 100% of the current 2.84M-position pool) says halving the dead-cell count to ~1,118 needs ~290M positions (~100x the current pool), and getting under 500 needs ~4.4B (~1,560x). Brute- force self-play volume cannot close this gap; it needs an architecture-side fix, which is exactly what factorization is for. **Implementation** (`nnue_model.py`, `export_nnue_weights.py`), per the plan above, built essentially as specified: a second, zero- initialized `ft_p` `EmbeddingBag(PS_NB=704, l1)`, summed into the accumulator alongside `ft` via `idx % PS_NB` (same bag offsets, no `build_training_tensors.py` changes needed), coalesced into `ft.weight` at export time (`ft.weight += ft_p.weight[arange(22528) % 704]`) with a required pre/post-coalesce forward-pass agreement check (measured 3.58e-07 on a random-weighted smoke test, comfortably inside the ~1e-5 tolerance) before `ft_p` is dropped from the exported file. **Measured effect, matched-seed/epoch (lambda=0.75, K=2.4130, material-loss-weight=0.001 -- see the next note on why that changed from 0.01, 34 epochs, otherwise identical hyperparameters)**: - Holdout MSE at matched checkpoints: factored beats unfactored at every logged step, not just early training (contra the "effect fades" caveat in the plan above, which turned out not to dominate here) -- e.g. step 4000/epoch 11: 0.01193 (factored) vs. 0.01446 (unfactored); final epoch 33: 0.01201 (factored) vs. 0.01331 (unfactored). - `sanity_check_net.py --cross-sectional` (2000 sampled holdout positions, final checkpoints): both nets monotonic in piece value, but factored's queen/pawn ratio is closer to nominal (~8.1x own / 9.8x opponent-removed, vs. the already-decent unfactored run's ~8.2x/9.9x -- both well above the old TWIC-trained matprobe net's ~5.6x). - Compared against a REAL Stockfish net's NNUE output (see next section) on 1000 shared holdout positions: factored wins on every metric -- correlation 0.879 vs 0.861, mean|diff| 0.776 vs 0.852 pawns raw, 0.718 vs 0.762 pawns after fitting away the overall scale difference (both nets run at ~0.79-0.83x Stockfish's scale, expected since ours is WDL-sigmoid-calibrated, not raw-centipawn- trained). **`material-loss-weight` lowered from 0.01 to 0.001 for this corpus**, before any of the above: the original 0.01 (see "material-blindness fixed" status above) was tuned against TWIC's homogeneous, balanced- middlegame material distribution, where the raw material-MSE term was naturally small and its weighted contribution stayed well below the WDL loss's. This corpus has far more material-imbalanced positions by construction (decisive self-play games played to real conclusions, not human-resignation-biased), so the same 0.01 weight measurably put the material term at parity with the WDL loss rather than a gentle nudge -- confirmed by briefly launching training at 0.01 before catching it (checkpoints `ckpt_epoch000-001.pt` from that run were deleted, not used for anything above). **Known residual gap, not yet fixed**: `ft_p`'s zero-init was deliberate (avoids confounding the factorization effect with an init-time accumulator-scale change), but `ft` itself still uses `nn.EmbeddingBag`'s default `N(0,1)`-per-entry init, which is NOT zero -- so an untouched row's random contribution is the same magnitude (`norm~16`) as a fully-trained row's, not suppressed by scale. Today this is masked by `ft_p`'s well-trained, shared correction dominating the sum at those positions (confirmed: 0/1000 Stockfish-comparison positions activated any dead cell, so this gap hasn't tainted any reported result above), but it's a real residual risk for any future position that activates several dead cells together. Cheap, not-yet-applied fix: zero-init `ft` the same way `ft_p` already is, so a dead cell degrades gracefully to "contributes nothing" rather than "contributes untrained noise." ## Verifying against a real Stockfish net (one-time diagnostic, not a ## pipeline dependency) To sanity-check whether the above was in the right ballpark at all (as distinct from reusing Stockfish's own weights, which was explicitly ruled out -- the goal is a from-scratch net strong enough to self-play and iterate on, "eating its own tail," not a transplant), pulled a real, period-appropriate Stockfish net (`nn-fdc1d0fe6455.nnue`, added to Stockfish 2023-06-12 per `github.com/official-stockfish/networks`' commit history, rev `38e61663d836e062af0bc002814ad5149c4b7729`) and parsed its `FeatureTransformer::psqtWeights` table directly, against that exact commit's header-reading code (fetched and read, not guessed -- a first attempt at parsing by eye got the header layout wrong and was caught before trusting any conclusion from it). - **Confirmed identical feature space**: `HalfKAv2_hm::HashValue = 0x7f234cb8` and `Dimensions=22528` match our own spec exactly (same king buckets, same orientation table, same piece-square planes); this net's stored `ft` hash decodes to `L1=1536` (vs. our 256) via `ft_hash = HashValue XOR (2*L1)`. - **`psqtWeights[index][bucket]` for our own 2,237 dead-cell indices looks completely ordinary** relative to that plane's overall average (e.g. queen plane: 2487cp mean on dead cells vs. 2541cp overall) -- i.e. our dusty corners aren't inherently weird positions, just a sampling gap in this specific corpus, which is exactly the pattern factorization is built to exploit (the true per-piece value is roughly king-bucket-independent). - To get a real NNUE eval from this old net for the comparison above, built Stockfish from source at the exact matching commit (`gmake -j$(nproc) build ARCH=x86-64-avx2 COMP=clang` -- note FreeBSD's default `make` is BSD make and cannot parse this Makefile's GNU-Make syntax, must use `gmake`), rather than risk hand-reimplementing its 8-layer-stack quantized forward pass. Compared its `eval` command's "NNUE evaluation" line (pure NNUE output, no classical-blend) against our own `forward()`, both converted to white-relative, over 1000 shared holdout FENs -- see `annotate_pgns/compare_vs_stockfish_net.py` and the factorization status note above for the numeric results. ## Iterating on a net: adding a new generation of self-play data The first net to actually beat (or seriously contest) HCE at matched `sd 10` is the `lambda=0.75`, P-factored checkpoint from the status above -- the point where self-play with this net starts producing a genuinely bigger/more-targeted training pool becomes real ("eating its own tail", per the conversation that drove this project's current phase). This section is the checklist for folding a new batch of self-play games into the next training generation -- written down so it doesn't need re-deriving from scratch (or re-making the same mistakes) each time. **None of this requires a new architecture or a new `nnue_model.py` -- same model class, same feature set, same `--resume` mechanism.** It's a data/calibration discipline problem, not a code problem. ### 1. Combine, don't just append Pool the new self-play positions together with the existing training corpus (corpora can mix freely at the games.jsonl level -- see `extract_selfplay_positions.py`'s docstring) and re-run the **full** pipeline on the combined set: ``` split_holdout.py -> fit_k_and_label.py -> quiet_filter_games.py -> build_training_tensors.py ``` Do NOT train on only the new self-play batch by itself, even via `--resume`. Self-play games cluster around positions the current net already favors (that's what self-play means) -- training on only the new batch narrows the effective training distribution and risks forgetting what the original, more diverse corpus taught. The combined pool is also exactly how the dusty-corner coverage problem (see the factorization status above) actually improves over time: new self-play games sample different king positions/endgame structures than the corpus that came before, so the pooled coverage count should be re-measured after each new generation (same `np.bincount` check used throughout this doc) to see whether it's actually shrinking the dead- cell count, not just assumed. ### 2. Refit K on the combined pool -- never carry an old K forward `fit_k_and_label.py`'s K is fit to one specific corpus's own score distribution. **"Fitted K = 2.4130 -- record this; applying a different K later (e.g. after labeling more data) is not the same model"** (the tool's own exit message) applies in the other direction too: once the pool changes, K must be refit on the new combined TRAIN split, not reused from the previous generation. Training against a target calibrated for a different score distribution than the one actually being trained on silently miscalibrates every label, the same class of problem as DNA-tuning's holdout-leakage concerns elsewhere in this project. ### 3. Watch the label source -- this is the one that actually matters How were the new self-play games scored? - **If an independent, stronger reference (Stockfish, or any engine not being trained) produced the centipawn labels**, this is safe regardless of which engine(s) played the moves -- self-play between two candidates is a great way to source diverse, targeted positions (specifically ones that stress whatever the current net is weakest at), as long as the position doesn't then get "approved" by asking the same net being trained to grade itself. - **If the candidate net's own search produced the score labels for its own games**, this reintroduces the single-generation self- distillation problem already diagnosed as the likely reason the first two (pre-Stockfish-relabel) nets failed to beat HCE at all (see the "Revised understanding of why" status note above) -- a net trained to reproduce its own judgment can at best match that judgment, never exceed it. Mitigate, don't ignore, by keeping `lambda < 1.0` in `build_training_tensors.py` so real game outcomes (`r_outcome`) stay in the training target regardless of where the centipawn half of the blend came from -- outcomes are ground truth no self-distillation loop can corrupt, even when the score labels themselves are suspect. ### 4. Re-check `material-loss-weight` against the new pool's material distribution `material-loss-weight` was lowered from `0.01` (tuned for TWIC's balanced-middlegame distribution) to `0.001` specifically because this project's Stockfish self-play corpus has far more decisive, material- imbalanced games. A new self-play generation's material distribution is a new empirical fact, not something to assume stays `0.001` forever -- if candidate-vs-candidate games turn out far more drawish/balanced than the original bootstrap corpus (plausible once both sides are playing well), the same weight could again land at an unintended fraction of the total loss. Check, don't assume -- same reasoning as the original 0.01->0.001 change, just potentially in either direction next time. ### 5. Warm-start, don't restart `--resume ` on the new combined-pool tensors is the right default -- it's strictly faster than random-init-and-train- from-scratch, and there's no evidence in this project so far that warm-starting hurts final quality (nothing here has tested the opposite, to be clear -- if a future generation's holdout metrics look worse than a from-scratch control at the same epoch count, that's worth isolating before trusting warm-starting blindly forever). ### 6. Validate before trusting the new generation In order, cheapest/fastest first -- stop at the first one that fails rather than running all of them blind: 1. `sanity_check_net.py --cross-sectional` on the new checkpoint -- monotonic piece-value deltas, reasonable magnitudes (compare against the queen/pawn ratio of the previous generation as a baseline, not just "is it monotonic"). 2. Re-run the dead-cell coverage check (bincount over the new combined training tensors) -- confirm it's trending down, not flat or up. 3. `compare_vs_stockfish_net.py` against the same real Stockfish net used above, same held-out FEN sample where practical, to see whether correlation/scale-fit-residual improved or regressed vs. the previous generation's numbers (0.879 corr / 0.718 pawns resid. as of the factored lambda=0.75 checkpoint -- the number to beat). 4. `export_nnue_weights.py` (coalesce check must pass, zero clipped values) then a real matched-depth (`sd 10`, not `st` -- see the fixed-depth-for-ECM-comparisons convention used throughout this project) head-to-head via `eval_tune/test_vs_head.sh` with `NNUE_WEIGHTS` set, against classic HCE. This is the only step that actually answers "did this generation get better," everything above is a cheaper proxy that can catch a regression before burning a multi-hour match run on it. ## Adoption criterion: sd-equal parity is not the bar, `st`-equal is -- ## and the net is currently ~3.3x slower per node than HCE Measured directly in the first real sd10-matched match (the factored `lambda=0.75` checkpoint vs. classic HCE, 2026-10-18, 400 games): candidate nps=470-475K vs. head nps=1.5-1.6M -- **a ~3.3x gap**, not a rounding error. At matched *depth* this is invisible (both sides reach sd10 regardless of nps), which is exactly why the sd-equal match is the right first filter -- it isolates "is the net smarter per node" from the speed handicap. But a `st`-equal (real time control) match would currently be lopsided in HCE's favor for a reason that has nothing to do with eval quality: at this project's branching factor (ebf ~3.8-3.9, measured in the same match), a 3.3x nps disadvantage costs roughly a full ply of search depth at any fixed time budget, on top of whatever the eval-quality gap already costs. Testing at `st` before closing that gap would answer "how bad is an engine with no lazy-eval fast path," not "is this net any good." **Revised understanding (2026-10-18, same conversation): nps parity is sufficient, not a blocker -- the real Stockfish binary on this exact box is the proof.** It searches at ~500K nps -- essentially the same as our net's measured 470-480K, not the 1.5-1.6M HCE achieves -- and still beats this project's HCE every single recorded game (not even a draw). Stockfish doesn't out-node HCE; it out-*prunes* it, searching dramatically trimmer trees at the same or worse raw nps, because its eval is trustworthy enough to prune aggressively on. This reframes the priority between the two levers below: **lazy eval (closing the 3.3x nps gap) is a nice-to-have, not the thing standing between this net and real strength.** The actual lever is section 3 of the checklist below -- null-move R, LMR's reduction table and fail-high prediction, LMP thresholds, futility margins -- all of which currently encode how *little* HCE's eval can be trusted. A learned eval earning the right to prune harder, the way Stockfish's does, is plausibly worth more than recovering the nps gap ever could be. Lazy eval stays worth doing (free, no downside), just not gated-on-first the way the staged list below originally implied. **Staged gate, in order -- do not skip ahead:** 1. **Sd-equal match (done, 2026-10-18)**: necessary-but-not-sufficient filter. If the net can't hold its own at matched depth, it has no chance at matched time, since real time controls only make the speed handicap worse for it, never better. 2. **Recalibrate the pruning-aggressiveness constants (checklist section 3) -- this is the primary lever, per the Stockfish comparison above, not a secondary cleanup pass.** Not a full DNA-tuning sweep, just enough that search isn't making prune/ reduce/extend decisions calibrated for a different eval's scale and trustworthiness. Plausibly worth more than nps recovery on its own. 3. **Close the free speed gap too** -- lazy eval (checklist section 1), still worth doing since it's pure engineering with no downside (it doesn't change what the net *thinks*, only how often it's asked to think at nodes that get pruned anyway) -- just not the bottleneck the Stockfish comparison suggested it might be. 4. **Only then, a real `st`-equal SPRT against HCE.** This is the step that actually answers the adoption question -- everything before it answers a necessary precondition, not the question itself. **The adoption threshold itself shouldn't be pure parity-or-better, though.** HCE is a mature, hand-tuned eval already deep into diminishing returns on its own DNA-tuning pipeline. NNUE has an open-ended improvement path still mostly unexploited (more self-play generations per the iteration section above, factorization already landed, a wider L1 if a Phase B GPU run ever happens -- see the real Stockfish net's L1=1536 vs. our 256 as a sense of the room available). A net that's close-but-not-quite-parity at a genuinely fair (`st`- equal, post-calibration) time control may still be worth adopting if the plan is to keep training it -- the alternative is maintaining two eval paths indefinitely while one's ceiling keeps rising and the other's doesn't. Close-and-improving beats even-and-stagnant. ## Engine calibration checklist: what needs re-tuning for a new eval This engine's search has several subsystems whose margins/tables were fit against classic `Eval()`'s specific output scale and noise profile. None of this is required for the net to *run* (it already does, correctly, per the wiring in `search.c`/`nnueeval.c`) -- it's required for the net to run *well*, since an uncalibrated search is fighting the new eval with pruning/reduction decisions tuned for a different one. Confirmed against this engine's actual current search (2026-10-18 conversation; EFP and ProbCut were previously assumed present and are not -- corrected below rather than left as dead checklist items): **1. Free speed, no behavior change (do this first, before any of the below -- see the adoption-criterion note above for why ordering matters here):** - Lazy eval fast path -- designed (architecture decisions above: reuse the net's own cheap linear/PSQT-ish term -- here `ft_p`/ `material_probe`-shaped -- as a first-pass filter, falling through to the full forward pass only when inconclusive vs. alpha/beta), not yet built. This is the single biggest lever on the measured 3.3x nps gap and touches zero tuning -- it's the right first move specifically because it can't make the net play worse, only faster. - Deferred/lazy accumulator maintenance (flagged in the accumulator design section above as a "performance follow-up, not required for correctness") -- avoid eager accumulator updates at nodes pruned before any eval is ever consulted. **2. The raw-score-to-cp mapping -- verify this before touching any margin below, since it's a multiplier on all of them:** Every margin in section 3 is a flat cp (or near-cp) threshold tuned against HCE's output scale. The trained net runs at ~0.8x a real Stockfish net's scale (measured via `compare_vs_stockfish_net.py`) and is WDL-sigmoid-calibrated (K=2.4), not raw-centipawn-trained. If search currently treats `forward()`'s raw output as if it were already in HCE's cp convention, every margin below is off by that scale factor before even considering noise-profile differences. This needs the inverse-sigmoid-to-cp conversion the original architecture decision already called for ("Requires a calibrated inverse-sigmoid conversion back to centipawns... that conversion constant needs its own calibration pass") -- confirm it exists and is right once, rather than chasing a scale error through a dozen separately-miscalibrated margins. **3. Score-scale/noise-dependent pruning and reduction constants -- this engine's actual current set (corrected 2026-10-18: no EFP in main search, only LMP; no separate razoring step -- LMR fills that role here; no ProbCut; QSearch futility and "delta pruning" are the same mechanism, not two):** - QSearch futility margins - Static/reverse null-move pruning margin - Null-move reduction depth R, and any eval-based skip/verification gating on the null-move try itself - LMP move-count table, `data.c` - LMR reduction table, `data.c`, **and** the joint fail-high prediction table feeding it (see the LMR sd11/sn10M protocol note elsewhere in this project) -- this table is a direct statement about how trustworthy the eval is at a reduced-depth node, fit entirely against HCE's noise profile - Root aspiration window initial width and re-widening schedule on fail-high/fail-low (tuned against HCE's iteration-to-iteration score noise -- a good catch, easy to overlook since it's at the root, not buried in the main search loop) **4. Eval-component-dependent heuristics (lower priority -- already confirmed to degrade gracefully under a lazy/missing full eval per the "Breaking external dependencies" section above; re-measure how often they actually fire once NNUE lazy eval ships, don't re-derive from scratch):** - `EstimateHangingMaterialCp`'s `cTrapped`-fallback "dumb move" axis - `ValueOfMaterialInTroubleDespiteMove`'s QSearch stand-pat gate **5. Currently dormant, no action needed yet:** draw/contempt scoring is built but always returns 0 right now -- no live miscalibration risk today, but if it's ever turned on, the same cp-scale concern in section 2 applies to whatever constant drives it. ## Status (2026-10-18, continued): pipeline-scaling audit, three gaps fixed, end-to-end driver added Prompted by generating a much larger volume of annotated PGN than before and wanting to confirm the post-annotation Python pipeline (`annotate_pgns/`) actually uses all available parallelism rather than silently bottlenecking on a single-threaded stage. `annotate.py` itself (the sd-N search stage) was already confirmed fine -- sharded across N single-threaded engine subprocesses -- and `build_training_tensors.py` already had a `--workers 16` `multiprocessing.Pool`. Three real gaps found and fixed; one unrelated question (an end-to-end PGN-to-`.nnue` driver) answered by adding a new orchestrator script. - **Gap: `quiet_filter.py` had no parallelism at all.** It's a single-threaded Python loop doing `chess.Board()` construction + SEE per row over the entire corpus (2.5M+ rows for the Stage A run) -- the one CPU-bound step in the non-annotation part of the pipeline with zero parallelism, unlike `build_training_tensors.py`'s `Pool` right next to it. **Fixed via a new wrapper, `shard_quiet_filter.py`**, rather than rewriting `quiet_filter.py` itself: round-robin-splits the input across `--workers` (default 16) temp shard files, runs that many `quiet_filter.py` subprocesses concurrently, concatenates their outputs. Deliberately reuses `quiet_filter.py` unmodified -- one definition of the filter, reused as a subprocess, the wrapper only owns sharding/recombining. Row order in the combined output does not match the input (shards are concatenated shard-by-shard, not interleaved back to original order) -- harmless here since `positions.quiet.jsonl` is a flat, order-independent training-set input. Verified end-to-end on a synthetic 40-row fixture: 40 in, 40 out, across 4 shards. - **Gap: `fit_k_and_label.py`'s ternary-search K-fit was unvectorized Python.** `fit_k()`'s 40 ternary-search iterations each call `mse()` twice, and the old `mse()` was a plain Python `for` loop over every (score, outcome) pair -- roughly 80 full passes over ~2M+ pairs in pure Python for a step that does very little actual compute. **Rewritten to numpy** (`sigmoid`/`mse` as vectorized array ops over the full pair set) **and sharded** across `--workers` (default 16) via `multiprocessing.Pool`, for *both* halves of the file: the calibration-pair collection (`collect_calibration_pairs`, now `collect_chunk` workers each vectorizing one chunk of games with numpy, combined via `np.concatenate`) and the labeling pass (`label_file`, now `label_chunk` workers, each vectorizing one game's non-exact positions' sigmoid computation per chunk) -- `Pool.imap`-based, order-preserving, mirroring `build_training_tensors.py`'s existing chunking pattern. Verified end-to-end on a synthetic 50-game fixture (4 workers, chunk-size 7): correct game/position counts, K fit, both splits labeled. - **Gap: `nnue_train.py`'s CPU thread default was lower than the rest of the pipeline's convention.** `--threads` default bumped 8 -> 16, matching the `--workers 16` convention used everywhere else in this pipeline (kept below this box's 24 cores -- see below -- rather than maxed out, so other jobs on the shared box keep some breathing room). - **Core-count note, not acted on:** this box has 24 cores (`nproc`), so every one of these `--workers`/`--threads`/ `annotate.py --shards` defaults (16) intentionally leaves some headroom rather than using all 24 -- a deliberate choice to leave the box's other jobs breathing room, not an oversight. `annotate.py` Stage A's own measured throughput (NNUE.md's "Status (2026-10-14)" section: ~75 pos/sec aggregate at 16 shards vs. ~112/sec extrapolated from one shard x16, attributed to contention) is a reminder that this box's parallelism doesn't scale linearly past a point anyway -- blindly raising shard/worker counts further is not assumed to help without measuring first. - **New: `run_nnue_pipeline.py`**, an end-to-end orchestrator from a glob of input PGN to a `.nnue` weights file, requested directly as "can one script run the whole training process end to end." Targets **already-scored PGN** specifically (match_play.py's `"{+-}NN.NN/DEPTH PV..."` per-move comment format -- e.g. `pgn/stockfish_self_play_training_data.pgn`, `pgn/self_play_games.pgn`), not raw human-game PGN needing a fresh sd-N search -- that's `annotate.py`'s job and is a different, not-yet-automated entry point. Chains, in order: `extract_selfplay_positions.py` (once per input file, since that script takes a single PGN; outputs concatenated) -> `split_holdout.py` -> `fit_k_and_label.py` -> `quiet_filter_games.py` (both splits) -> `build_training_tensors.py` (both splits) -> `nnue_train.py` -> `export_nnue_weights.py` (against the latest `ckpt_epochNNN.pt` written -- `nnue_train.py` has no separate "best checkpoint" selection yet, so "latest" is the only option today). Every stage checks whether its expected output already exists and skips it unless `--force` is passed -- the later stages here can run for hours, so a crash/interrupt partway through should not mean restarting from stage 1. Passes `--workers`/ `--threads` through at the same default of 16 used everywhere else. A thin orchestrator by design: shells out to the existing, individually-tested scripts via `subprocess` rather than reimplementing any of their logic. Argument parsing, `--help`, and the glob-miss error path were smoke-tested; the full multi-stage run (which needs the engine binary, real PGNs, and hours of wall clock) was not run end-to-end as part of this change. ## TODO (not yet implemented) - **Horizontal-flip data augmentation, in the position loader (`build_training_tensors.py`).** HalfKAv2_hm's king-bucket mirroring does NOT tie piece-square weights across the a/h files -- the `OrientTBL` XOR step re-orients the board onto the king's own half but a piece's *file within that half* still maps to a distinct feature row, so a↔h-mirrored positions are not already folded together by the architecture. Randomly mirroring positions left-right at load time (flip every square's file, recompute both perspectives' indices, swap castling rights symmetrically where applicable) is a legitimate, standard way to roughly double the effective sample count per feature row without changing dataset size on disk. Directly relevant given the measured sparsity (23.9% of HalfKAv2 rows with zero observations, per the 2026-10-18 factorization status above) -- should be evaluated alongside P-factorization, not instead of it, since the two address the same sparsity symptom through different mechanisms (more effective samples per row vs. a denser auxiliary feature set) and their effects may not be additive. - **L2 weight decay in `nnue_train.py`'s optimizer (AdamW).** Pulls rarely-updated weights (the same sparse/underexposed rows the factorization work above targets) toward zero instead of leaving them to drift on whatever noisy gradient they happened to last receive. Cheap -- a single optimizer hyperparameter, no architecture or export-format change -- so there's no reason to gate it behind the factorization/augmentation experiments above; can be added and swept independently. ## Status (2026-10-20): losing-position score compression fixed (K ## recalibration), per-generation score calibration automated, lazy ## probe refit wired into the pipeline, margin-safety recalibrated ## against real search traffic, LOP-based promotion gate added Prompted by a direct observation: the production net's score in clearly-losing positions barely distinguished "lost" (-2) from "about to be mated" (-3) -- no meaningful magnitude gradation at the losing extreme, despite real Stockfish scores in the same game ranging to -13 or worse just before the actual mate. **Root cause, confirmed analytically and empirically**: `raw` (the net's pre-sigmoid scalar) is only ever supervised through `sigmoid(raw/K)` against a WDL-probability target -- never regressed against a centipawn/pawn target directly. That sigmoid saturates hard once a position is clearly decided; with this corpus's originally fitted `K=2.0659`, `sigmoid(8/K)=0.9796` vs `sigmoid(13/K)=0.9982` -- a target spread of under 0.02 for a 5-pawn difference in the real score, leaving almost no training gradient to differentiate them. No value of `lambda` (the `p_search`/`r_outcome` blend) fixes this, since both blend candidates are already post-sigmoid probabilities. **What did NOT fix it, tried in order, each measured at a medium scale (500k positions) before committing further**: 1. An auxiliary `score_probe` head (nnue_model.py, same mechanism as `material_probe` -- a separate `Linear` on the shared accumulator, trained with its own MSE loss against the real Stockfish score). Modest, inconsistent improvement -- its gradient only ever reaches the *shared embedding tables*, never `fc1`/`fc2`/`fc3`, so nothing forces the actually-exported path to widen its range. 2. A direct `MSE(raw, clipped_score)` term added to the main loss (replacing `score_probe` -- puts gradient straight on `fc1-3`). Also modest/inconsistent across weights (0.001, 0.005) and loss shapes (plain MSE vs. Huber) -- sometimes better on the winning side, not reliably better on the losing side that actually prompted this investigation. 3. **The actual fix: widen K itself.** Measured `raw` vs. real score at two K values (2.07, 5.0) on the same held-out sample; the ratio `slope/K` came out nearly identical at both points (0.3577 vs. 0.3574) -- strong confirmation that `raw` scales linearly with K as theory predicts, letting the right K be solved for directly: `K* = 1/0.3575 ≈ 2.8`. Confirmed by actually training at K=2.8: aggregate slope landed at 1.0005 (vs. 0.74 at the original fit), with per-bucket magnitude tracking the real score far more closely across the whole severity range, no overshoot the way K=5.0 produced. Cross-checked against Stockfish's own documented NNUE training write-up (`scaling_factor≈410cp=4.1 pawns`) -- our corpus's own re-fit (2.8) came in lower, a reminder that this project's corpus-specific fit isn't automatically wrong just for disagreeing with a different engine's number, but also isn't assumed right just for being "ours." 4. Also adopted: `--loss-exponent 2.5` (Stockfish's own documented practice, already wired as `powered_loss()`'s `--loss-exponent`, just not previously used) -- `|error|^2.5` instead of plain MSE in WDL-probability space, which shrinks every error (since errors are <1) but shrinks small ones proportionally more, concentrating gradient on the worst-wrong predictions. Small, not dramatic, improvement on top of the K fix; adopted since it's free. **Real residual tradeoff, not a flawless fix**: after K=2.8/exponent 2.5, aggregate calibration is essentially exact (slope ~1.0), but there's genuine non-linearity left (overshoot at the extremes, undershoot mid-range) and per-position noise went up slightly (mean abs error 1.78 vs. 1.57 pawns on the full-scale retrain's held-out sample) -- widening K to fix systematic compression also amplifies whatever per-position imprecision the net already had, since the same logit-space noise now maps to a bigger pawn-space error. Not treated as disqualifying, but recorded so a future reader doesn't expect this to be perfect. **Keeping this stable across future retrains -- the real answer isn't "just reuse the same K"**: tested directly, holding K fixed within one run still left measurable residual non-linearity/noise -- a fixed *recipe* (K/lambda/loss-exponent) is not the same as a fixed *output scale*, since training variance (data, convergence) shifts the empirical raw-to-score mapping even with an identical recipe. Built instead: - `export_nnue_weights.py`'s new `CalibrateScoreMapping`: fits a fresh affine correction `(a, b)` via OLS, `real_score ≈ a + b*raw`, against the HOLDOUT tensor dir (not train -- an honest fit needs positions the checkpoint didn't see), every single export. Baked into a new file format version, `NNU4` (magic bump from `NNU3`): two extra float32 fields, `CALIB_A`/`CALIB_B`, read by `nnueeval.c`'s `NnueLoadWeights` and applied in both `NnueForward` (`(CALIB_A + CALIB_B*fOut) * 100.0f`, replacing the old bare `fOut * 100.0f`) and `NnueProbeScore` (same correction, since that function's own doc comment promises it stays directly comparable to `NnueForward`). This decouples engine search-tuning from needing the raw net's internal scale to stay perfectly constant forever -- the centipawn numbers search actually consumes stay calibrated by construction, regenerated fresh every generation, regardless of how much the underlying raw net drifts. - Target for this fit is deliberately Stockfish's own score (the same ground truth used throughout this project), not game outcome and not HCE's scale -- outcome is noisier per-position by design (lambda's whole point), and matching HCE's scale would just reintroduce a version of the original compression problem in a different flavor. - **Real, previously-undiscovered bug found and fixed in the same pass**: `run_nnue_pipeline.py` never threaded `--k` through to `nnue_train.py` at all -- every prior pipeline-driven production run (`sf_20261019`, `sf_20261020`, `sf98k`/the then-current production net) silently trained its loss against `nnue_train.py`'s own `DEFAULT_K=1.3123` (a Phase-A placeholder), completely independent of whatever K `fit_k_and_label.py` actually fit for that corpus's own `p_search` labels. `--k` is now a required argument to `run_nnue_pipeline.py` (no silent default) specifically so this can't happen again unnoticed. **Lazy-eval probe now refit automatically too**: new `annotate_pgns/fit_lazy_probe.py` (ridge regression on the trained net's own accumulator, targeting that same checkpoint's raw `forward()` output pre-calibration, matching `NnueProbeScore`'s own convention) wired into `run_nnue_pipeline.py` as stage 8, run unconditionally after export -- every retrain now produces a fresh, matching probe file alongside the `.nnue` weights, written to `_probe.bin`. The real search-node-traffic margin-safety calibration (next item) is deliberately **not** automated the same way -- too expensive to justify every generation, and not yet established that it needs to be redone every generation either (see below). **Full retrain done with the fixed recipe**: pooled all three Stockfish self-play PGN files (`stockfish_self_play_training_data{,.0, .1}.pgn`, ~120k games total), `lambda=0.99`, `K=2.8`, `loss-exponent=2.5`, `score-loss-weight=0` (the score_probe/direct-term experiments above were abandoned in favor of the K fix). 10,484,835 training positions, 1,163,529 held out. Early-stopped at epoch 20 (5 epochs past the best, epoch 15, per `--patience 5`), `holdout_mse= 0.00457`. Output: `annotate_pgns/runs/sf_20261020_recal/ nnue_weights.nnue` (+ `nnue_weights_probe.bin`). **Margin-safety calibration re-run for real** (not the offline-sample proxy that was all that existed before): 16 parallel shards of 12,500 positions each (200,000 total) from `tests/cross_sectional_200k.ep_` at `sd 6` -- same protocol as the original 2026-10-15 calibration, re-run because that one was fit against an earlier net generation, before this recalibration widened the net's raw output scale; the old margins (`{180, 200, 190, 180, 170, 160, 150, 150}`) would have been unsafe (too tight) against the new, wider-scale net's real swings. New `iNnueLazyMarginByMatBucket` (p98 per material bucket, ~100-435M real search-node swing samples per bucket): `{300, 320, 410, 640, 770, 770, 540, 230}`, now live in `eval.c`. New tooling: `annotate_pgns/aggregate_marginsafety.py` sums the raw per-shard histograms (`eval.c`'s `PersistMarginSafetyHistogramsToFile` format) bucket-and-bin-wise and computes exact combined percentiles -- shard percentiles don't combine validly on their own, only the raw counts do. - **Side finding**: NNUE super-lazy (material-only, no net at all) measured much more competitive than the earlier "too wild to use" verdict suggested -- p98 `{420, 450, 520, 730, 860, 850, 640, 290}`, consistently ~30-40% looser than the real probe but the same order of magnitude, not wildly worse. Likely explanation: the earlier verdict was measured against the *old, uncalibrated* `NnueForward` output, so comparing a stable-scale quantity (material) against a compressed/wrong-scale one (the old net) spuriously inflated the apparent disagreement. Not re-enabled by default -- `g_fUseNnueLazyProbe` and `g_fUseNnueSuperLazy` both remain opt-in flags; flipping either on by default is a strength/speed tradeoff decision that needs its own match-based validation, not something to bundle into a calibration-table update. - **Open question, deliberately not resolved**: whether this expensive live-search-traffic pass needs re-running every future generation, or whether a cheap offline-residual proxy (same mechanism as `CalibrateScoreMapping`, just also recording percentiles) tracks it closely enough to trust between occasional live re-checks. Not yet tested head-to-head; treat the live pass as the trusted source until that comparison is actually done. **Validation: real, independent movement on the LOP benchmark** (`tests/lop/` -- eval/move-agreement against a real Stockfish reference on a fixed 10,000-position set, much cheaper than a match -- see that section's own history below). Prior checkpoints (pre-recalibration): 87.089%/87.076% eval correlation, 46.800% move correlation -- statistically indistinguishable from classic HCE's own 87.064%/46.830% on the same benchmark. This recalibrated net: **90.319% eval correlation, 48.550% move correlation** -- the first real movement on this benchmark all session, and the basis on which this net was promoted to production (see below). **Status correction on match results -- the "has not won a single head-to-head match" note elsewhere in this document is now stale**: against the *pre-recalibration* net, a real `sd 10`-matched self-play run (`--sd 10`, isolating "is the net smarter per node" from the nps handicap) produced LLR=+2.28 against an H1 bound of +2.94, score 0.635 over 360 games (`-72 =119 +169`), climbing every update, halted early as conclusive. NNUE now soundly beats classic HCE at matched search depth -- this was not true when the adoption-criterion section below was written. At matched real time (`--st 4`), the same pre-recalibration net was close to even (score 0.447, LLR -0.37, inconclusive, consistent with the known ~3.3x nps gap and not-yet-recalibrated search margins costing it ~3 plies of depth) -- **the recalibrated net in this status update has not been st4-tested yet**; that is the planned next step, separate from this work. **Promotion policy changed, 2026-10-20**: `head_reference/nnue/ promote.sh` now accepts LOP evidence (`--lop-baseline-log`/ `--lop-candidate-log`, parsing `compare_lop`'s own stdout) as an alternative to a match log, gated on the candidate's eval correlation strictly exceeding the baseline's (no `--allow-negative` equivalent -- if LOP doesn't even show improvement, it isn't evidence of anything). Decision, not just convenience: requiring a full self-play match for every incremental retrain (more data, same recipe) is impractical -- a genuine few-Elo improvement is often not reliably resolvable in a match without a very large game count, and this project's own history (ECM solve-rate improving while early NNUE nets still lost every real match, see below) already shows a cheap static benchmark isn't *proof* of match strength either -- match evidence has its own noise problems at this scale, it isn't pure ground truth being traded away for convenience. LOP is accepted as the *routine* gate specifically because requiring a match every time is the impractical alternative, not because LOP has been shown to track match strength reliably yet. An occasional real match is still recommended periodically to confirm LOP-based promotions haven't quietly drifted away from actual playing strength -- this recalibrated net was itself promoted on LOP evidence alone (see `head_reference/nnue/current/metadata.json`), with a real match against the previous baseline still outstanding. ## Status (2026-10-20, continued): lazy probe measured net-negative on ## real search traffic, super-lazy confirmed a clean win and now ## defaults on Follow-up to the "Engine calibration checklist" section above, picking up item 1's lazy-eval fast path -- the goal was to actually turn the two gates on (both were built and opt-in per the recalibration section just above) and measure real before/after numbers, not just trust the earlier margin-safety calibration as sufficient justification on its own. **Methodology**: `ecm_ringers`/`ecm_confident_quick`/`ecm_hard_quick` (the three curated suites), each at both `sd 10` (fixed depth) and `sn 5000000` (fixed node budget), run as a clean baseline (gates off), then with only the lazy probe on, then with only super-lazy on, all against the same `eval.nnue` weights -- never mixed/parallel-vs- sequential in a way that could introduce load-noise into the comparison (confirmed box load ~4.3 on a lightly-loaded multi-core machine throughout; `sd`/`sn` hold depth/node-budget fixed by construction regardless of load, so nps and solve-count differences here are real, not noise). **Baseline** (gates off): 108/191 solved across all three suites at `sd 10` (7/11, 73/90, 27/90), 117/191 at `sn 5000000` (8/11, 78/90, 31/90); nps 479k-539k depending on suite/mode. **Lazy probe alone, at the real p98 margin table (`{300, 320, 410, 640, 770, 770, 540, 230}` by material bucket, from the real `cross_sectional_200k.ep_`/16-shard run)**: a clear net LOSS -- **nps down 3-7% in every configuration**, and fewer solves in 4 of 6 configs (ringers 6/11 and 7/11 vs baseline's 7/11 and 8/11; confident_quick 70/90 flat/down; hard_quick 24/90 and 28/90 vs 27/90 and 31/90). Not explained by the probe being expensive -- see the latency measurement below. **Direct per-call latency measurement** (new instrumentation, piled into the existing `EVAL_TIME` build profile rather than a new flag -- `TIMED_EVAL_CALL` wraps the `NnueProbeScore()`/`NnueForward()` call sites in `eval.c`'s production NNUE branch, accumulating into two new always-present-but-only-`EVAL_TIME`-populated `COUNTERS.tree` fields, `u64CyclesNnueProbeCall`/`u64CyclesNnueForwardCall`, plus a new `u64NnueProbeAttempts` counter -- unlike `u64LazyEvals`, which only counts *hits*, this counts every attempt, hit or miss, since the gate pays its cost on every attempt and only recoups it on a hit). Reported in `root.c` right after the (NNUE-irrelevant, classic-HCE- only) `LAZY_EVAL` exit-breakdown block, with the measurement's own ~38-cycle rdtsc-pair overhead printed alongside rather than silently subtracted, so the raw numbers can be sanity-checked against it: ``` NnueProbeScore: ~272 cyc/call (net ~235 after overhead) NnueForward: ~4100 cyc/call (net ~4060 after overhead) ``` **A clean 15:1 ratio, extremely consistent across all 11 `ecm_ringers` positions (272±3 cyc every time)** -- the probe itself is cheap and behaving exactly as designed. Implied breakeven hit rate: `272/4100 ≈ 6.6%`. The docs' earlier 2026-10-16 measurement (before the K/lambda rescale) claimed a 14-23% real hit rate, comfortably above breakeven. **But the real hit rate on this suite, at the current (post-rescale) p98 table, is ~1.0% overall** (71,022 hits / 7,100,023 attempts on `ecm_ringers`, per-position rates ranging 0.03%-6.98%) -- an order of magnitude under breakeven. Expected per-call cost with the gate on: `272 + 0.99*4100 ≈ 4331` cyc vs. `4100` cyc with it off -- a ~5.6% per-call slowdown, matching the measured nps regression almost exactly. **Root cause, not just a symptom**: the p98 margin table is calibrated purely for *safety* (bounding `|NnueForward() - NnueProbeScore()|` error to the 98th percentile), a completely different statistic from *how close typical nodes sit to their own alpha/beta window*, which is what actually determines hit rate. A margin can be a perfectly correct p98 safety bound and still almost never let the gate fire if real windows at that material bucket are typically narrower than the error the probe needs room for -- which is apparently true for most of this table's buckets (only bucket 7's 230 happens to land in useful territory; the large buckets, 640-770, are far out past where any real window would resolve a cutoff). **Empirical margin-vs-hit-rate sweep** (flat override across all buckets via `nnue lazymargin `, `ecm_ringers`/`sd 10` only -- fast enough to run live rather than needing a real calibration pass): | flat margin | hit rate | solved/11 | nps | |---|---|---|---| | 770 | 0.00% | 6 | 509k | | 400 | 1.80% | 6 | 510k | | 230 | 16.0% | 6 | 538k | | 150 | 34.9% | 5 | 582k | | 100 | 45.0% | 4 | 632k | | 70 | 61.6% | 4 | 696k | | 50 | 67.9% | 3 | 775k | | 30 | 78.9% | 2 | 896k | | 15 | 87.9% | 2 | 1024k | Smooth tradeoff curve, no cliff -- the knee sits around margin≈200-230, where nps is clearly improving (538k, a real win over baseline) with no solve cost yet; solves start falling steadily from 150 down. This is genuine evidence a per-bucket table *could* be built to win cleanly -- it just isn't the p98 table, which was never tuned for this objective at all. **p95 recalibration attempt -- inconclusive, not a real retest**: tried to avoid a full hour+ `cross_sectional_200k.ep_` re-collection by recomputing percentiles from already-persisted histogram data. Found `eval.c`'s `PersistMarginSafetyHistogramsToFile` writes to `/var/tmp/typhoon/marginsafety_raw_histograms.txt` by default, and `annotate_pgns/aggregate_marginsafety.py` keeps the full binned histogram (not just the percentiles it prints), so an arbitrary percentile is cheap to recompute *if the real per-shard files still exist*. **They don't** -- only one small leftover file was found (~1.9M total `nnue_lazy` samples across buckets vs. the real run's ~100-435M/bucket; almost certainly one shard or an early smoke test, not the aggregated dataset), and a project-wide `find -name "*hist*"` turned up nothing else NNUE-related (one old, unrelated classic-HCE residual-calibration file from a `guess2_30k` run). Computed p95 from this small/wrong-provenance file anyway as a fast, low-confidence stand-in (`{180, 240, 520, 530, 670, 720, 480, 310}` by bucket) and re-ran all three suites, both depths, with the lazy probe on this table. **Result was noisier than the p98 run, not clearly better**: some solve counts recovered (ringers back to 8/11 at `sn5M`, hard_quick up to 26/90 and 33/90), but nps actually *dropped* in several configs relative to p98 (`sn5M` confident_quick 504k vs p98's 537k; hard_quick 483k vs 512k) -- backwards from what a looser margin should do, strongly suggesting the small sample's per-bucket numbers don't reliably reflect the real distribution. **Not trusted as a real retune** -- a legitimate hit-rate-targeted per-bucket calibration still needs the full `cross_sectional_200k.ep_` collection (or equivalent), this time explicitly measuring node-level alpha/beta proximity (what determines hit rate), not just probe-vs-forward error (what the existing harness measures, which only bounds safety). **Super-lazy alone, at the real p98 margin table (newly built this session -- previously a single flat runtime override with no per-bucket fit at all, `g_iNnueSuperLazyMargin`)**: measured a clean win across the board. `sd 10`: ringers 5/11 (down from 7, but ringers is only 11 positions -- noisy), confident_quick 70/90 (down from 73, flat vs lazy's own 70), hard_quick 26/90 (close to baseline's 27); nps 478k-504k, roughly matching or beating baseline (479k-502k). `sn 5000000`: ringers 6/11 (down from 8), confident_quick **79/90** (up from 78), hard_quick **33/90** (up from 31); nps **540k-601k**, a genuine 1-12% win over baseline (514k-539k) in every suite. Net read: real nps win, solve-count effect mixed but mostly neutral-to- positive outside the small `ecm_ringers` suite (where single-position flips swing the ratio a lot) -- per the user's own judgment, ECM solve-count noise at this suite size isn't treated as a correctness signal worth chasing into `tests/ECM.md` lookups; the nps win is the real, load-bearing result. **Same-bucketing confirmed**: super-lazy's own margin-safety calibration (`RecordNnueSuperLazyMarginSafetySwing`, `g_uNnueSuperLazyMarginSwingMax[BASE_MARGIN_MAT_BUCKETS]` etc.) uses the identical `_MaterialBucket()` (8 buckets, combined-army-scaler width 8) as the lazy probe's own table. **No bucket excluded outright** (unlike classic HCE's own super-lazy, which structurally excludes material bucket 0 -- bare-king/pawn- dominated endgames -- per a specific *exceeded-rate* finding, 6.8% blow-through from pawn-structure blindness, since HCE's super-lazy runs before `_EvalPawns` entirely). NNUE's measured p98 swing by bucket, `{420, 450, 520, 730, 860, 850, 640, 290}`, has bucket 0 as the *second-lowest* value, not an outlier -- nothing in the data says to disable it. Caveat: no exceeded-rate-specific check (as opposed to plain p98 magnitude) has been run for NNUE's super-lazy the way it was for HCE's, so this is "nothing visible says disable a bucket," not "confirmed safe at every bucket." **Decision**: lazy probe gate code kept live (cheap to leave dormant, and the latency math + margin sweep both say a real win is plausible with the right margins -- just not demonstrated yet), but `g_fUseNnueLazyProbe` stays `FALSE` by default (`nnueeval.c`), margin table reverted to the real p98 values (the p95 guess is preserved only as a commented-out, documented-as-unreliable alternative). Super-lazy promoted to default-on (`g_fUseNnueSuperLazy = TRUE`, `nnueeval.c`), using its own new per-bucket `iNnueSuperLazyMarginByMatBucket` table (`eval.c`) rather than the old flat-only override, which remains available via `g_iNnueSuperLazyMargin >= 0` for experimentation. Both gates only take effect when `g_fUseNnueEval` is also on. **Follow-up, not yet done**: a real hit-rate-targeted per-bucket lazy probe retune, against ordinary (non-adversarial-tactical) search traffic such as `twic_sample.ep_` rather than ECM -- the current margin-vs-hit-rate sweep and the docs' original 14-23%-hit-rate/ 10.7%-nps-win measurement were on different corpora, and ECM's narrower forcing-line alpha/beta windows may themselves suppress hit rate in a way that doesn't generalize to normal play. Also worth measuring: an exceeded-rate check for NNUE super-lazy's bucket 0, matching the specific diagnostic that led HCE to exclude it, rather than relying on p98 magnitude alone.